planet/docs/technical/en/earth-bgp-context.md

BGP Context

Current Goal

The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:

raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization

The practical product goal is no longer just to "show incidents on the globe". The current product objective is:

1. keep BGP visually present on Earth even when incident density is low
2. make incidents clearly feel like a higher-confidence layer than anomalies
3. show that the observation network is still active even when there are no active incidents

In practice, that means Earth should behave like an observability surface, not only an incident map:

• collectors show that observation is happening
• activity shows where routing state is currently active or noisy
• incidents become the highest-confidence focus layer

Current Backend Architecture

Data Layers

1. BGPObservation

   • File: backend/app/models/bgp_observation.py
   • Purpose: store normalized raw routing observations from live/history sources.
   • Typical fields:
     • source
     • collector
     • peer_asn
     • peer_ip
     • prefix
     • event_type
     • as_path
     • origin_asn
     • next_hop
     • communities
     • observed_at
     • raw_payload
     • collector_geo
     • ingest_batch_id

2. BGPAnomaly

   • File: backend/app/models/bgp_anomaly.py
   • Purpose: hold atomic detector outputs.
   • Current detector output types include:
     • origin_change
     • more_specific_burst
     • mass_withdrawal

3. BGPIncident

   • File: backend/app/models/bgp_incident.py
   • Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.

Pipeline

Main flow is currently anchored in:

• backend/app/services/collectors/bgp_common.py
• backend/app/services/bgp_enrichment.py
• backend/app/services/bgp_detectors.py
• backend/app/services/bgp_incidents.py

Operational flow:

1. collectors fetch raw BGP data
2. normalize_bgp_event() standardizes payloads
3. observations are persisted to bgp_observations
4. enrichment augments events with analysis context
5. detectors create bgp_anomalies
6. incident aggregation rolls anomalies up into bgp_incidents

Current Ingest Sources

1. RIPE RIS Live

   • Collector file: backend/app/services/collectors/ris_live.py
   • Used for realtime observation flow.

2. CAIDA BGPStream Backfill

   • Collector file: backend/app/services/collectors/bgpstream.py
   • Used as history/backfill entry point.

Current Enrichment Status

Implemented enrichment skeleton in:

• backend/app/services/bgp_enrichment.py

Current enrichments:

• prefix family / prefix length
• supernet / more-specific derivation
• deduplicated AS path
• path prepending hints
• collector region info
• prefix baseline hints
• new-origin detection
• ASN organization profile from PeeringDB where available
• prefix scope / impacted region hints
• prefix geography source priority:
  • OpenGeoFeed (override/high confidence)
  • IPtoASN (country-range baseline)
  • NRO delegated stats (registry-allocation fallback)

Current limitation:

• RPKI is still placeholder-only and returns unknown
• no real ROA validation source is integrated yet
• inetnum / inet6num whois fallback is still pending

Current API Surface

Primary API file:

• backend/app/api/v1/bgp.py

Available endpoints:

• /api/v1/bgp/events
• /api/v1/bgp/events/summary
• /api/v1/bgp/events/{id}
• /api/v1/bgp/anomalies
• /api/v1/bgp/anomalies/summary
• /api/v1/bgp/anomalies/{id}
• /api/v1/bgp/incidents
• /api/v1/bgp/incidents/summary
• /api/v1/bgp/incidents/{id}

Visualization GeoJSON endpoints:

• backend/app/api/v1/visualization.py
• /api/v1/visualization/geo/bgp-collectors
• /api/v1/visualization/geo/bgp-anomalies
• /api/v1/visualization/geo/bgp-incidents

Current Earth Behavior

Relevant files:

• frontend/public/earth/js/bgp.js
• frontend/public/earth/js/main.js
• frontend/public/earth/js/info-card.js
• frontend/public/earth/js/constants.js
• frontend/public/earth/index.html

Current design:

1. Collectors are always shown when BGP is enabled.
2. Incident markers are now the primary Earth BGP markers.
3. If there are no incidents, Earth falls back to anomaly markers.
4. If there are no anomalies either, collectors still provide presence.
5. A dedicated activity layer now adds:
   • per-collector recent 15-minute activity halos
   • clustered regional activity hints derived from active collectors
6. Incident markers now use:
   • symbol-driven event cores
   • outward ring pulses
   • reduced diffuse glow compared with older Earth builds
   • event region halos, collector coverage halos, and radar pulses derive a lighter tint from their own icon color instead of using a fixed teal; red high-severity events, orange active collectors, and blue idle collectors keep their hue family while broad halos stay softer than the icon
7. The right-side stats now show:
   • BGP events
   • collector count
   • BGP status summary

This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated activity layer between raw observation and incident focus.

Current BGP status strategy:

• incidents present: show active incident count
• no incidents but anomalies present: show active anomaly count, plus active observation regions when available
• no incidents/anomalies but activity present: show 观测网络运行中
• no incidents/anomalies but collectors present: show 观测网络运行中 · 当前未发现聚合级事件
• no BGP data at all: show 暂无观测数据

Earth info-card strategy:

• bgp card is now incident-centric in wording
• bgp_collector card shows collector location and current event count

Current Product Gap

The main product gap is not architecture correctness. It is low-density visualization strategy.

Current reality:

• incident count is naturally much lower than anomaly count
• that is expected, because incidents are aggregated and de-noised
• but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer

Implementation detail for the recommended activity layer is kept in the repository path docs/plans/earth-bgp-region-aggregation-plan.md.

So the immediate next milestone is:

event map -> observability map

That means Earth needs three simultaneously readable layers:

1. observation layer
   • collectors
   • recent collector activity
   • baseline coverage
2. activity layer
   • recent event density
   • anomaly/noise hotspots
   • regional activity scoring
   • incident presence bonus
3. incident layer
   • sparse but highly legible, high-confidence event objects
   • symbol-driven markers
   • outward ring pulse instead of broad diffuse glow

Incident Visual Direction

The Earth incident layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.

Design principles:

1. incident markers should use a strong primary symbol

   • the symbol shape should carry type meaning where possible
   • examples:
     • origin_change: triangle-like warning marker
     • mass_withdrawal: alert/exclamation-style marker
     • more_specific_burst: split/radiating marker

2. emphasis should come from outward ring pulses, not area flooding

   • use a compact hot core
   • use one or more expanding ring pulses
   • avoid broad luminous blobs that make the event center feel vague

3. collector and incident must stay visually distinct

   • collectors are observation infrastructure
   • incidents are extracted event focus
   • collector activity should stay quieter than incident pulse language

4. calm periods still need observability presence

   • collectors and activity layers should keep the map alive
   • once incidents appear, they should clearly dominate nearby BGP visuals

5. incident geography should become prefix-centric

   • collectors should remain evidence sources, not the primary event location
   • preferred geography priority:
     • prefix_geography
     • prefix_scope
     • ASN organization region
     • collector centroid as final fallback
   • prefix_scope should remain an observation-derived scope hint
   • a new prefix_geography layer should be introduced for actual prefix-centric placement

Reference inspiration:

• World Monitor
  • sparse event symbols
  • compact centers
  • ring-like outward pulses
  • stronger incident legibility than diffuse glow

Current Console Behavior

Relevant page:

• frontend/src/admin/pages/PlainResourcePages.tsx

Current BGP console page has three levels:

1. observation summary

   • total events
   • collector count
   • prefix count

2. incident summary and incident table

3. anomaly detail table plus recent observation events

This means the BGP page still has useful signal even when there are zero anomalies.

Known Product/Engineering Boundaries

1. The current system is still closer to an event board than a full BGP sensing platform.
2. RIS coverage still needs to expand beyond narrow subscription scope.
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
4. Collector geography still depends heavily on static RIPE RIS mappings.
5. Incident-to-cable/IXP/region association is still weak and early-stage.
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.

Test Status

BGP-specific tests live in:

• backend/tests/test_bgp.py

Verified status at this point:

• 25 passed for backend/tests/test_bgp.py
• 62 passed for backend/tests

Covered areas include:

• normalization
• observation serialization
• enrichment
• detectors, including route leak candidate and path flap
• incident aggregation
• batch anomaly creation
• BGP events/incidents API
• summary endpoints

Most Relevant Files

Backend:

• backend/app/models/bgp_observation.py
• backend/app/models/bgp_anomaly.py
• backend/app/models/bgp_incident.py
• backend/app/services/collectors/bgp_common.py
• backend/app/services/bgp_enrichment.py
• backend/app/services/bgp_detectors.py
• backend/app/services/bgp_incidents.py
• backend/app/api/v1/bgp.py
• backend/app/api/v1/visualization.py

Frontend:

• frontend/src/admin/pages/PlainResourcePages.tsx
• frontend/public/earth/js/bgp.js
• frontend/public/earth/js/main.js
• frontend/public/earth/js/info-card.js
• frontend/public/earth/js/constants.js
• frontend/public/earth/index.html

Recommended Next Steps

Next Backend / Detection Priority

1. Integrate real RPKI validation data.
2. Expand realtime collector coverage and include withdrawals more broadly.
3. Continue refining route leak and path instability detectors with stronger heuristics.

Next Correlation / Storytelling Priority

4. Strengthen incident aggregation semantics and titles.
5. Add weak correlation from incidents to:
   • cable corridors
   • landing points
   • IXPs
   • other traffic anomaly sources
6. Refine Earth hover/click handoff between collectors and incidents.

Next Visualization Priority

7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
8. Add more incident symbol types as new detectors land.
9. Add a real prefix geography source:
   • IPtoASN / IPtoCountry as the first practical dataset
   • OpenGeoFeed as a higher-quality override layer
   • registry/whois only as fallback