planet/docs/technical/earth-bgp-context.md

# BGP Context

## Current Goal

The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:

`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`

The practical product goal is no longer just to "show incidents on the globe". The current product objective is:

1. keep BGP visually present on Earth even when incident density is low
2. make incidents clearly feel like a higher-confidence layer than anomalies
3. show that the observation network is still active even when there are no active incidents

In practice, that means Earth should behave like an observability surface, not only an incident map:

- `collectors` show that observation is happening
- `activity` shows where routing state is currently active or noisy
- `incidents` become the highest-confidence focus layer

## Current Backend Architecture

### Data Layers

1. `BGPObservation`
   - File: `backend/app/models/bgp_observation.py`
   - Purpose: store normalized raw routing observations from live/history sources.
   - Typical fields:
     - `source`
     - `collector`
     - `peer_asn`
     - `peer_ip`
     - `prefix`
     - `event_type`
     - `as_path`
     - `origin_asn`
     - `next_hop`
     - `communities`
     - `observed_at`
     - `raw_payload`
     - `collector_geo`
     - `ingest_batch_id`

2. `BGPAnomaly`
   - File: `backend/app/models/bgp_anomaly.py`
   - Purpose: hold atomic detector outputs.
   - Current detector output types include:
     - `origin_change`
     - `more_specific_burst`
     - `mass_withdrawal`

3. `BGPIncident`
   - File: `backend/app/models/bgp_incident.py`
   - Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.

### Pipeline

Main flow is currently anchored in:

- `backend/app/services/collectors/bgp_common.py`
- `backend/app/services/bgp_enrichment.py`
- `backend/app/services/bgp_detectors.py`
- `backend/app/services/bgp_incidents.py`

Operational flow:

1. collectors fetch raw BGP data
2. `normalize_bgp_event()` standardizes payloads
3. observations are persisted to `bgp_observations`
4. enrichment augments events with analysis context
5. detectors create `bgp_anomalies`
6. incident aggregation rolls anomalies up into `bgp_incidents`

### Current Ingest Sources

1. `RIPE RIS Live`
   - Collector file: `backend/app/services/collectors/ris_live.py`
   - Used for realtime observation flow.

2. `CAIDA BGPStream Backfill`
   - Collector file: `backend/app/services/collectors/bgpstream.py`
   - Used as history/backfill entry point.

## Current Enrichment Status

Implemented enrichment skeleton in:

- `backend/app/services/bgp_enrichment.py`

Current enrichments:

- prefix family / prefix length
- supernet / more-specific derivation
- deduplicated AS path
- path prepending hints
- collector region info
- prefix baseline hints
- new-origin detection
- ASN organization profile from PeeringDB where available
- prefix scope / impacted region hints
- prefix geography source priority:
  - `OpenGeoFeed` (override/high confidence)
  - `IPtoASN` (country-range baseline)
  - `NRO delegated stats` (registry-allocation fallback)

Current limitation:

- `RPKI` is still placeholder-only and returns `unknown`
- no real ROA validation source is integrated yet
- `inetnum` / `inet6num` whois fallback is still pending

## Current API Surface

Primary API file:

- `backend/app/api/v1/bgp.py`

Available endpoints:

- `/api/v1/bgp/events`
- `/api/v1/bgp/events/summary`
- `/api/v1/bgp/events/{id}`
- `/api/v1/bgp/anomalies`
- `/api/v1/bgp/anomalies/summary`
- `/api/v1/bgp/anomalies/{id}`
- `/api/v1/bgp/incidents`
- `/api/v1/bgp/incidents/summary`
- `/api/v1/bgp/incidents/{id}`

Visualization GeoJSON endpoints:

- `backend/app/api/v1/visualization.py`
- `/api/v1/visualization/geo/bgp-collectors`
- `/api/v1/visualization/geo/bgp-anomalies`
- `/api/v1/visualization/geo/bgp-incidents`

## Current Earth Behavior

Relevant files:

- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`
- `frontend/public/earth/js/constants.js`
- `frontend/public/earth/index.html`

Current design:

1. Collectors are always shown when BGP is enabled.
2. Incident markers are now the primary Earth BGP markers.
3. If there are no incidents, Earth falls back to anomaly markers.
4. If there are no anomalies either, collectors still provide presence.
5. A dedicated `activity layer` now adds:
   - per-collector recent 15-minute activity halos
   - clustered regional activity hints derived from active collectors
6. Incident markers now use:
   - symbol-driven event cores
   - outward ring pulses
   - reduced diffuse glow compared with older Earth builds
5. The right-side stats now show:
   - BGP events
   - collector count
   - BGP status summary

This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus.

Current BGP status strategy:

- incidents present: show active incident count
- no incidents but anomalies present: show active anomaly count, plus active observation regions when available
- no incidents/anomalies but activity present: show `观测网络运行中`
- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件`
- no BGP data at all: show `暂无观测数据`

Earth info-card strategy:

- `bgp` card is now incident-centric in wording
- `bgp_collector` card shows collector location and current event count

## Current Product Gap

The main product gap is not architecture correctness. It is low-density visualization strategy.

Current reality:

- incident count is naturally much lower than anomaly count
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer

Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).

So the immediate next milestone is:

`event map -> observability map`

That means Earth needs three simultaneously readable layers:

1. `observation layer`
   - collectors
   - recent collector activity
   - baseline coverage
2. `activity layer`
   - recent event density
   - anomaly/noise hotspots
   - regional activity scoring
   - incident presence bonus
3. `incident layer`
   - sparse but highly legible, high-confidence event objects
   - symbol-driven markers
   - outward ring pulse instead of broad diffuse glow

## Incident Visual Direction

The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.

Design principles:

1. `incident` markers should use a strong primary symbol
   - the symbol shape should carry type meaning where possible
   - examples:
     - `origin_change`: triangle-like warning marker
     - `mass_withdrawal`: alert/exclamation-style marker
     - `more_specific_burst`: split/radiating marker

2. emphasis should come from outward ring pulses, not area flooding
   - use a compact hot core
   - use one or more expanding ring pulses
   - avoid broad luminous blobs that make the event center feel vague

3. `collector` and `incident` must stay visually distinct
   - collectors are observation infrastructure
   - incidents are extracted event focus
   - collector activity should stay quieter than incident pulse language

4. calm periods still need observability presence
   - collectors and activity layers should keep the map alive
   - once incidents appear, they should clearly dominate nearby BGP visuals

5. incident geography should become `prefix-centric`
   - collectors should remain evidence sources, not the primary event location
   - preferred geography priority:
     - `prefix_geography`
     - `prefix_scope`
     - `ASN organization region`
     - `collector centroid` as final fallback
   - `prefix_scope` should remain an observation-derived scope hint
   - a new `prefix_geography` layer should be introduced for actual prefix-centric placement

Reference inspiration:

- `World Monitor`
  - sparse event symbols
  - compact centers
  - ring-like outward pulses
  - stronger incident legibility than diffuse glow

## Current Console Behavior

Relevant page:

- `frontend/src/pages/BGP/BGP.tsx`

Current BGP console page has three levels:

1. observation summary
   - total events
   - collector count
   - prefix count

2. incident summary and incident table

3. anomaly detail table plus recent observation events

This means the BGP page still has useful signal even when there are zero anomalies.

## Known Product/Engineering Boundaries

1. The current system is still closer to an event board than a full BGP sensing platform.
2. RIS coverage still needs to expand beyond narrow subscription scope.
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
4. Collector geography still depends heavily on static RIPE RIS mappings.
5. Incident-to-cable/IXP/region association is still weak and early-stage.
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.

## Test Status

BGP-specific tests live in:

- `backend/tests/test_bgp.py`

Verified status at this point:

- `25 passed` for `backend/tests/test_bgp.py`
- `62 passed` for `backend/tests`

Covered areas include:

- normalization
- observation serialization
- enrichment
- detectors, including route leak candidate and path flap
- incident aggregation
- batch anomaly creation
- BGP events/incidents API
- summary endpoints

## Most Relevant Files

Backend:

- `backend/app/models/bgp_observation.py`
- `backend/app/models/bgp_anomaly.py`
- `backend/app/models/bgp_incident.py`
- `backend/app/services/collectors/bgp_common.py`
- `backend/app/services/bgp_enrichment.py`
- `backend/app/services/bgp_detectors.py`
- `backend/app/services/bgp_incidents.py`
- `backend/app/api/v1/bgp.py`
- `backend/app/api/v1/visualization.py`

Frontend:

- `frontend/src/pages/BGP/BGP.tsx`
- `frontend/public/earth/js/bgp.js`
- `frontend/public/earth/js/main.js`
- `frontend/public/earth/js/info-card.js`
- `frontend/public/earth/js/constants.js`
- `frontend/public/earth/index.html`

## Recommended Next Steps

### Next Backend / Detection Priority

1. Integrate real RPKI validation data.
2. Expand realtime collector coverage and include withdrawals more broadly.
3. Continue refining route leak and path instability detectors with stronger heuristics.

### Next Correlation / Storytelling Priority

4. Strengthen incident aggregation semantics and titles.
5. Add weak correlation from incidents to:
   - cable corridors
   - landing points
   - IXPs
   - other traffic anomaly sources
6. Refine Earth hover/click handoff between collectors and incidents.

### Next Visualization Priority

7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
8. Add more incident symbol types as new detectors land.
9. Add a real prefix geography source:
   - `IPtoASN / IPtoCountry` as the first practical dataset
   - `OpenGeoFeed` as a higher-quality override layer
   - registry/whois only as fallback