release: bump version to 0.31.1
This commit is contained in:
rayd1o
355
docs/technical/earth-bgp-context.md
Normal file
355
docs/technical/earth-bgp-context.md
Normal file
@@ -0,0 +1,355 @@
|
||||
# BGP Context
|
||||
|
||||
## Current Goal
|
||||
|
||||
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:
|
||||
|
||||
`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`
|
||||
|
||||
The practical product goal is no longer just to "show incidents on the globe". The current product objective is:
|
||||
|
||||
1. keep BGP visually present on Earth even when incident density is low
|
||||
2. make incidents clearly feel like a higher-confidence layer than anomalies
|
||||
3. show that the observation network is still active even when there are no active incidents
|
||||
|
||||
In practice, that means Earth should behave like an observability surface, not only an incident map:
|
||||
|
||||
- `collectors` show that observation is happening
|
||||
- `activity` shows where routing state is currently active or noisy
|
||||
- `incidents` become the highest-confidence focus layer
|
||||
|
||||
## Current Backend Architecture
|
||||
|
||||
### Data Layers
|
||||
|
||||
1. `BGPObservation`
|
||||
- File: `backend/app/models/bgp_observation.py`
|
||||
- Purpose: store normalized raw routing observations from live/history sources.
|
||||
- Typical fields:
|
||||
- `source`
|
||||
- `collector`
|
||||
- `peer_asn`
|
||||
- `peer_ip`
|
||||
- `prefix`
|
||||
- `event_type`
|
||||
- `as_path`
|
||||
- `origin_asn`
|
||||
- `next_hop`
|
||||
- `communities`
|
||||
- `observed_at`
|
||||
- `raw_payload`
|
||||
- `collector_geo`
|
||||
- `ingest_batch_id`
|
||||
|
||||
2. `BGPAnomaly`
|
||||
- File: `backend/app/models/bgp_anomaly.py`
|
||||
- Purpose: hold atomic detector outputs.
|
||||
- Current detector output types include:
|
||||
- `origin_change`
|
||||
- `more_specific_burst`
|
||||
- `mass_withdrawal`
|
||||
|
||||
3. `BGPIncident`
|
||||
- File: `backend/app/models/bgp_incident.py`
|
||||
- Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.
|
||||
|
||||
### Pipeline
|
||||
|
||||
Main flow is currently anchored in:
|
||||
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
|
||||
Operational flow:
|
||||
|
||||
1. collectors fetch raw BGP data
|
||||
2. `normalize_bgp_event()` standardizes payloads
|
||||
3. observations are persisted to `bgp_observations`
|
||||
4. enrichment augments events with analysis context
|
||||
5. detectors create `bgp_anomalies`
|
||||
6. incident aggregation rolls anomalies up into `bgp_incidents`
|
||||
|
||||
### Current Ingest Sources
|
||||
|
||||
1. `RIPE RIS Live`
|
||||
- Collector file: `backend/app/services/collectors/ris_live.py`
|
||||
- Used for realtime observation flow.
|
||||
|
||||
2. `CAIDA BGPStream Backfill`
|
||||
- Collector file: `backend/app/services/collectors/bgpstream.py`
|
||||
- Used as history/backfill entry point.
|
||||
|
||||
## Current Enrichment Status
|
||||
|
||||
Implemented enrichment skeleton in:
|
||||
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
|
||||
Current enrichments:
|
||||
|
||||
- prefix family / prefix length
|
||||
- supernet / more-specific derivation
|
||||
- deduplicated AS path
|
||||
- path prepending hints
|
||||
- collector region info
|
||||
- prefix baseline hints
|
||||
- new-origin detection
|
||||
- ASN organization profile from PeeringDB where available
|
||||
- prefix scope / impacted region hints
|
||||
- prefix geography source priority:
|
||||
- `OpenGeoFeed` (override/high confidence)
|
||||
- `IPtoASN` (country-range baseline)
|
||||
- `NRO delegated stats` (registry-allocation fallback)
|
||||
|
||||
Current limitation:
|
||||
|
||||
- `RPKI` is still placeholder-only and returns `unknown`
|
||||
- no real ROA validation source is integrated yet
|
||||
- `inetnum` / `inet6num` whois fallback is still pending
|
||||
|
||||
## Current API Surface
|
||||
|
||||
Primary API file:
|
||||
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
|
||||
Available endpoints:
|
||||
|
||||
- `/api/v1/bgp/events`
|
||||
- `/api/v1/bgp/events/summary`
|
||||
- `/api/v1/bgp/events/{id}`
|
||||
- `/api/v1/bgp/anomalies`
|
||||
- `/api/v1/bgp/anomalies/summary`
|
||||
- `/api/v1/bgp/anomalies/{id}`
|
||||
- `/api/v1/bgp/incidents`
|
||||
- `/api/v1/bgp/incidents/summary`
|
||||
- `/api/v1/bgp/incidents/{id}`
|
||||
|
||||
Visualization GeoJSON endpoints:
|
||||
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
- `/api/v1/visualization/geo/bgp-collectors`
|
||||
- `/api/v1/visualization/geo/bgp-anomalies`
|
||||
- `/api/v1/visualization/geo/bgp-incidents`
|
||||
|
||||
## Current Earth Behavior
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
Current design:
|
||||
|
||||
1. Collectors are always shown when BGP is enabled.
|
||||
2. Incident markers are now the primary Earth BGP markers.
|
||||
3. If there are no incidents, Earth falls back to anomaly markers.
|
||||
4. If there are no anomalies either, collectors still provide presence.
|
||||
5. A dedicated `activity layer` now adds:
|
||||
- per-collector recent 15-minute activity halos
|
||||
- clustered regional activity hints derived from active collectors
|
||||
6. Incident markers now use:
|
||||
- symbol-driven event cores
|
||||
- outward ring pulses
|
||||
- reduced diffuse glow compared with older Earth builds
|
||||
5. The right-side stats now show:
|
||||
- BGP events
|
||||
- collector count
|
||||
- BGP status summary
|
||||
|
||||
This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus.
|
||||
|
||||
Current BGP status strategy:
|
||||
|
||||
- incidents present: show active incident count
|
||||
- no incidents but anomalies present: show active anomaly count, plus active observation regions when available
|
||||
- no incidents/anomalies but activity present: show `观测网络运行中`
|
||||
- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件`
|
||||
- no BGP data at all: show `暂无观测数据`
|
||||
|
||||
Earth info-card strategy:
|
||||
|
||||
- `bgp` card is now incident-centric in wording
|
||||
- `bgp_collector` card shows collector location and current event count
|
||||
|
||||
## Current Product Gap
|
||||
|
||||
The main product gap is not architecture correctness. It is low-density visualization strategy.
|
||||
|
||||
Current reality:
|
||||
|
||||
- incident count is naturally much lower than anomaly count
|
||||
- that is expected, because incidents are aggregated and de-noised
|
||||
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
|
||||
|
||||
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
|
||||
|
||||
So the immediate next milestone is:
|
||||
|
||||
`event map -> observability map`
|
||||
|
||||
That means Earth needs three simultaneously readable layers:
|
||||
|
||||
1. `observation layer`
|
||||
- collectors
|
||||
- recent collector activity
|
||||
- baseline coverage
|
||||
2. `activity layer`
|
||||
- recent event density
|
||||
- anomaly/noise hotspots
|
||||
- regional activity scoring
|
||||
- incident presence bonus
|
||||
3. `incident layer`
|
||||
- sparse but highly legible, high-confidence event objects
|
||||
- symbol-driven markers
|
||||
- outward ring pulse instead of broad diffuse glow
|
||||
|
||||
## Incident Visual Direction
|
||||
|
||||
The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.
|
||||
|
||||
Design principles:
|
||||
|
||||
1. `incident` markers should use a strong primary symbol
|
||||
- the symbol shape should carry type meaning where possible
|
||||
- examples:
|
||||
- `origin_change`: triangle-like warning marker
|
||||
- `mass_withdrawal`: alert/exclamation-style marker
|
||||
- `more_specific_burst`: split/radiating marker
|
||||
|
||||
2. emphasis should come from outward ring pulses, not area flooding
|
||||
- use a compact hot core
|
||||
- use one or more expanding ring pulses
|
||||
- avoid broad luminous blobs that make the event center feel vague
|
||||
|
||||
3. `collector` and `incident` must stay visually distinct
|
||||
- collectors are observation infrastructure
|
||||
- incidents are extracted event focus
|
||||
- collector activity should stay quieter than incident pulse language
|
||||
|
||||
4. calm periods still need observability presence
|
||||
- collectors and activity layers should keep the map alive
|
||||
- once incidents appear, they should clearly dominate nearby BGP visuals
|
||||
|
||||
5. incident geography should become `prefix-centric`
|
||||
- collectors should remain evidence sources, not the primary event location
|
||||
- preferred geography priority:
|
||||
- `prefix_geography`
|
||||
- `prefix_scope`
|
||||
- `ASN organization region`
|
||||
- `collector centroid` as final fallback
|
||||
- `prefix_scope` should remain an observation-derived scope hint
|
||||
- a new `prefix_geography` layer should be introduced for actual prefix-centric placement
|
||||
|
||||
Reference inspiration:
|
||||
|
||||
- `World Monitor`
|
||||
- sparse event symbols
|
||||
- compact centers
|
||||
- ring-like outward pulses
|
||||
- stronger incident legibility than diffuse glow
|
||||
|
||||
## Current Console Behavior
|
||||
|
||||
Relevant page:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
|
||||
Current BGP console page has three levels:
|
||||
|
||||
1. observation summary
|
||||
- total events
|
||||
- collector count
|
||||
- prefix count
|
||||
|
||||
2. incident summary and incident table
|
||||
|
||||
3. anomaly detail table plus recent observation events
|
||||
|
||||
This means the BGP page still has useful signal even when there are zero anomalies.
|
||||
|
||||
## Known Product/Engineering Boundaries
|
||||
|
||||
1. The current system is still closer to an event board than a full BGP sensing platform.
|
||||
2. RIS coverage still needs to expand beyond narrow subscription scope.
|
||||
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
|
||||
4. Collector geography still depends heavily on static RIPE RIS mappings.
|
||||
5. Incident-to-cable/IXP/region association is still weak and early-stage.
|
||||
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.
|
||||
|
||||
## Test Status
|
||||
|
||||
BGP-specific tests live in:
|
||||
|
||||
- `backend/tests/test_bgp.py`
|
||||
|
||||
Verified status at this point:
|
||||
|
||||
- `25 passed` for `backend/tests/test_bgp.py`
|
||||
- `62 passed` for `backend/tests`
|
||||
|
||||
Covered areas include:
|
||||
|
||||
- normalization
|
||||
- observation serialization
|
||||
- enrichment
|
||||
- detectors, including route leak candidate and path flap
|
||||
- incident aggregation
|
||||
- batch anomaly creation
|
||||
- BGP events/incidents API
|
||||
- summary endpoints
|
||||
|
||||
## Most Relevant Files
|
||||
|
||||
Backend:
|
||||
|
||||
- `backend/app/models/bgp_observation.py`
|
||||
- `backend/app/models/bgp_anomaly.py`
|
||||
- `backend/app/models/bgp_incident.py`
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
|
||||
Frontend:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
### Next Backend / Detection Priority
|
||||
|
||||
1. Integrate real RPKI validation data.
|
||||
2. Expand realtime collector coverage and include withdrawals more broadly.
|
||||
3. Continue refining route leak and path instability detectors with stronger heuristics.
|
||||
|
||||
### Next Correlation / Storytelling Priority
|
||||
|
||||
4. Strengthen incident aggregation semantics and titles.
|
||||
5. Add weak correlation from incidents to:
|
||||
- cable corridors
|
||||
- landing points
|
||||
- IXPs
|
||||
- other traffic anomaly sources
|
||||
6. Refine Earth hover/click handoff between collectors and incidents.
|
||||
|
||||
### Next Visualization Priority
|
||||
|
||||
7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
|
||||
8. Add more incident symbol types as new detectors land.
|
||||
9. Add a real prefix geography source:
|
||||
- `IPtoASN / IPtoCountry` as the first practical dataset
|
||||
- `OpenGeoFeed` as a higher-quality override layer
|
||||
- registry/whois only as fallback
|
||||
Reference in New Issue
Block a user