planet/docs/plans/earth-bgp-region-aggregation-plan.md

BGP Region Aggregation Plan

Goal

This document refines the current BGP activity layer into an implementation-ready regional aggregation design.

Primary product goal:

• turn sparse prefix-level observations, anomalies, and incidents into a readable regional observability layer
• keep Earth visually alive during low-incident periods
• make incident markers remain the highest-confidence foreground layer instead of replacing them

This layer is not a new collector, detector, or raw storage table. It is an aggregation/view-model layer:

observations -> enrichment -> anomalies/incidents -> geography mapping -> region aggregation -> Earth/UI activity layer

Why This Layer Exists

Current product gap from bgp-context.md:

• incident density is naturally low
• anomaly density is higher, but still not enough to keep the globe expressive all the time
• collector presence alone proves coverage, but does not communicate where routing is currently active or noisy

So the missing middle layer is:

• collectors show that observation exists
• regions show where activity is building up
• incidents show the specific high-confidence focus events

Scope

This plan is specifically for:

• a backend aggregation service
• a summary API for console/stats
• a GeoJSON API for Earth rendering
• an Earth background activity layer that supports, but does not replace, incident markers

This plan does not attempt to solve:

• exact prefix geolocation quality
• polygon-heavy geopolitical visualization
• persistent materialized region tables in v1

Region Layer Definition

Recommended conceptual model:

• region layer = background situational awareness
• incident layer = focal event markers

That means:

• region activity should answer where is routing behavior currently active or abnormal
• incident markers should answer which concrete event should the user click

Recommended Output Model

Suggested backend output object:

BGPRegionActivity

{
  "region_key": "sea",
  "region_name": "Southeast Asia",
  "center_lat": 1.3521,
  "center_lon": 103.8198,
  "observation_count": 128,
  "anomaly_count": 9,
  "incident_count": 2,
  "activity_score": 17.6,
  "status": "incident",
  "affected_prefix_count": 14,
  "affected_asn_count": 6,
  "collector_count": 5,
  "first_seen_at": "2026-04-02T10:00:00Z",
  "last_seen_at": "2026-04-02T10:12:00Z"
}

Fields To Keep In MVP

• region_key
• region_name
• center_lat
• center_lon
• observation_count
• anomaly_count
• incident_count
• activity_score
• status
• affected_prefix_count
• affected_asn_count
• collector_count
• first_seen_at
• last_seen_at

Fields To Delay

These are useful, but not required for the first implementation:

• bounding_box
• top_incident_types
• top_prefixes
• polygon geometry

Region Definition Strategy

Recommendation

Use a static region-definition table first.

Examples:

• north_america
• south_america
• western_europe
• eastern_europe
• east_asia
• southeast_asia
• south_asia
• middle_east
• north_africa
• sub_saharan_africa
• oceania

Why this is the right v1 choice:

• stable UI semantics
• strong readability on Earth
• easier debugging and explanation
• lower implementation cost than geohash or H3 grids

Not Recommended For V1

• geohash cell aggregation
• H3 aggregation
• fine-grained lat/lon bucket maps

Those are more flexible, but they make the map feel fragmented and less explainable.

Geography Mapping Strategy

Do not reduce the implementation to only prefix -> exact geo.

The region layer should follow the same geography-priority logic already implied by the current BGP direction:

1. prefix_geography
2. prefix_scope
3. ASN organization region
4. collector centroid fallback

This matters because exact prefix geography will often be incomplete or approximate. The region layer should stay robust even when only partial enrichment is available.

Backend Design

Recommended new service file:

• backend/app/services/bgp_regions.py

Suggested responsibilities:

• map_record_to_region(...)
• aggregate_region_activity(...)
• build_region_geojson(...)
• resolve_activity_status(...)
• compute_activity_score(...)

Data Source Inputs

Use a recent rolling window, default 15 minutes, and aggregate from:

• BGPObservation
• BGPAnomaly
• active BGPIncident

Aggregation Flow

1. query observations in the time window
2. query anomalies in the same window
3. query active incidents in the same window or active status set
4. resolve each record to a best-effort region
5. accumulate per-region counters
6. compute score and status
7. return region activity list

Status Model

Recommended status buckets:

• idle
• observing
• anomaly
• incident

Suggested rule:

if incident_count > 0: incident
elif anomaly_count > 0: anomaly
elif observation_count > 0: observing
else: idle

This aligns well with the current Earth status language and keeps the visual mapping simple.

Activity Score

The score should be a tunable heuristic, not a fixed truth model.

Recommended v1 formula:

activity_score =
  min(observation_count, 50) * 0.03
  + anomaly_count * 1.2
  + incident_count * 5.0

Why cap observations:

• observation volume is usually much larger than anomaly or incident volume
• uncapped observation counts would overwhelm the score
• capped observation counts preserve baseline presence without drowning real abnormality

Practical Guidance

• treat coefficients as configuration-like constants
• expect to retune after looking at real data
• keep incident weight dominant

API Design

1. Summary/List API

Suggested endpoint:

• /api/v1/bgp/regions/activity

Response shape:

{
  "window_minutes": 15,
  "regions": []
}

Use cases:

• BGP console summaries
• right-side Earth stats
• future region list panels

2. GeoJSON API

Suggested endpoint:

• /api/v1/visualization/geo/bgp-regions

Response shape:

{
  "type": "FeatureCollection",
  "features": []
}

Each feature should include:

• geometry
  • v1: Point
  • later: optional Polygon
• properties
  • region_key
  • region_name
  • status
  • activity_score
  • observation_count
  • anomaly_count
  • incident_count
  • affected_prefix_count
  • affected_asn_count
  • collector_count

Earth Rendering Plan

Detailed visual layering guidance is expanded in bgp-earth-rendering-plan.md.

Layer Relationship

• region layer = ambient background activity
• incident marker = focal event object

Do not replace incident markers with region markers.

Region Visual Rules

Suggested mapping:

• observing
  • weak glow
  • low pulse or no pulse
• anomaly
  • stronger glow
  • more visible pulse
• incident
  • strongest regional emphasis
  • but still visually secondary to the incident marker itself

Region Labels

Good v2 enhancement:

• show region name
• show counts like 2 incidents / 5 anomalies

This is useful, but should come after the core aggregation and Earth glow layer are working.

Interaction Model

Click Region

Recommended detail payload:

• region name
• observation/anomaly/incident counts in the selected window
• affected prefix count
• affected ASN count
• collector count
• recent incidents in the region

Click Incident

Keep the current incident-detail flow.

Interaction should feel hierarchical:

1. region gives situational context
2. incident gives event focus

MVP Implementation Order

Step 1

Define static REGIONS in code or config.

Step 2

Map geography-enriched BGP records into regions using the fallback chain.

Step 3

Aggregate recent window counts:

• observation_count
• anomaly_count
• incident_count

Step 4

Compute activity_score and status.

Step 5

Expose:

• /api/v1/bgp/regions/activity
• /api/v1/visualization/geo/bgp-regions

Step 6

Render region glows on Earth behind incident markers.

Out Of Scope For MVP

• persistent materialized region tables
• geohash or H3 support
• polygon-filled regional overlays
• detailed top-prefix ranking in the first release
• complicated scoring personalization

Risks And Constraints

Geography Quality

Prefix geography is approximate and incomplete. The region layer must tolerate fallback-based placement.

Query Cost

Dynamic aggregation is the right v1 choice, but repeated short-window queries may eventually need:

• in-process caching
• scheduled pre-aggregation
• materialized summaries

UI Overcrowding

If region glow, collector activity, and incidents all become too strong at once, Earth readability will regress. The region layer must remain supportive, not dominant.

Final Recommendation

The current BGP roadmap should explicitly add:

• region aggregation as the concrete implementation of the missing activity layer

The recommended product interpretation is:

• collectors prove observation coverage
• regions communicate live routing activity and abnormality
• incidents remain the clearest high-confidence event objects

In one sentence:

region aggregation is not a replacement for incidents; it is the situational background that makes sparse incidents feel legible on Earth.