feat: improve bgp incident visibility
This commit is contained in:
linkong
238
docs/bgp-context.md
Normal file
238
docs/bgp-context.md
Normal file
@@ -0,0 +1,238 @@
|
||||
# BGP Context
|
||||
|
||||
## Current Goal
|
||||
|
||||
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:
|
||||
|
||||
`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`
|
||||
|
||||
The practical product goal is to turn low-level BGP control-plane changes into understandable network situation events with collector coverage, impact regions, and incident-centric visualization.
|
||||
|
||||
## Current Backend Architecture
|
||||
|
||||
### Data Layers
|
||||
|
||||
1. `BGPObservation`
|
||||
- File: `backend/app/models/bgp_observation.py`
|
||||
- Purpose: store normalized raw routing observations from live/history sources.
|
||||
- Typical fields:
|
||||
- `source`
|
||||
- `collector`
|
||||
- `peer_asn`
|
||||
- `peer_ip`
|
||||
- `prefix`
|
||||
- `event_type`
|
||||
- `as_path`
|
||||
- `origin_asn`
|
||||
- `next_hop`
|
||||
- `communities`
|
||||
- `observed_at`
|
||||
- `raw_payload`
|
||||
- `collector_geo`
|
||||
- `ingest_batch_id`
|
||||
|
||||
2. `BGPAnomaly`
|
||||
- File: `backend/app/models/bgp_anomaly.py`
|
||||
- Purpose: hold atomic detector outputs.
|
||||
- Current detector output types include:
|
||||
- `origin_change`
|
||||
- `more_specific_burst`
|
||||
- `mass_withdrawal`
|
||||
|
||||
3. `BGPIncident`
|
||||
- File: `backend/app/models/bgp_incident.py`
|
||||
- Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI.
|
||||
|
||||
### Pipeline
|
||||
|
||||
Main flow is currently anchored in:
|
||||
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
|
||||
Operational flow:
|
||||
|
||||
1. collectors fetch raw BGP data
|
||||
2. `normalize_bgp_event()` standardizes payloads
|
||||
3. observations are persisted to `bgp_observations`
|
||||
4. enrichment augments events with analysis context
|
||||
5. detectors create `bgp_anomalies`
|
||||
6. incident aggregation rolls anomalies up into `bgp_incidents`
|
||||
|
||||
### Current Ingest Sources
|
||||
|
||||
1. `RIPE RIS Live`
|
||||
- Collector file: `backend/app/services/collectors/ris_live.py`
|
||||
- Used for realtime observation flow.
|
||||
|
||||
2. `CAIDA BGPStream Backfill`
|
||||
- Collector file: `backend/app/services/collectors/bgpstream.py`
|
||||
- Used as history/backfill entry point.
|
||||
|
||||
## Current Enrichment Status
|
||||
|
||||
Implemented enrichment skeleton in:
|
||||
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
|
||||
Current enrichments:
|
||||
|
||||
- prefix family / prefix length
|
||||
- supernet / more-specific derivation
|
||||
- deduplicated AS path
|
||||
- path prepending hints
|
||||
- collector region info
|
||||
- prefix baseline hints
|
||||
- new-origin detection
|
||||
- ASN organization profile from PeeringDB where available
|
||||
- prefix scope / impacted region hints
|
||||
|
||||
Current limitation:
|
||||
|
||||
- `RPKI` is still placeholder-only and returns `unknown`
|
||||
- no real ROA validation source is integrated yet
|
||||
|
||||
## Current API Surface
|
||||
|
||||
Primary API file:
|
||||
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
|
||||
Available endpoints:
|
||||
|
||||
- `/api/v1/bgp/events`
|
||||
- `/api/v1/bgp/events/summary`
|
||||
- `/api/v1/bgp/events/{id}`
|
||||
- `/api/v1/bgp/anomalies`
|
||||
- `/api/v1/bgp/anomalies/summary`
|
||||
- `/api/v1/bgp/anomalies/{id}`
|
||||
- `/api/v1/bgp/incidents`
|
||||
- `/api/v1/bgp/incidents/summary`
|
||||
- `/api/v1/bgp/incidents/{id}`
|
||||
|
||||
Visualization GeoJSON endpoints:
|
||||
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
- `/api/v1/visualization/geo/bgp-collectors`
|
||||
- `/api/v1/visualization/geo/bgp-anomalies`
|
||||
- `/api/v1/visualization/geo/bgp-incidents`
|
||||
|
||||
## Current Earth Behavior
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
Current design:
|
||||
|
||||
1. Collectors are always shown when BGP is enabled.
|
||||
2. Incident markers are now the primary Earth BGP markers.
|
||||
3. If there are no incidents, Earth falls back to anomaly markers.
|
||||
4. If there are no anomalies either, collectors still provide presence.
|
||||
5. The right-side stats now show:
|
||||
- BGP events
|
||||
- collector count
|
||||
- BGP status summary
|
||||
|
||||
Current BGP status strategy:
|
||||
|
||||
- incidents present: show active incident count
|
||||
- no incidents but anomalies present: show active anomaly count
|
||||
- no incidents/anomalies but collectors present: show `当前无活跃事件`
|
||||
- no BGP data at all: show `暂无观测数据`
|
||||
|
||||
Earth info-card strategy:
|
||||
|
||||
- `bgp` card is now incident-centric in wording
|
||||
- `bgp_collector` card shows collector location and current event count
|
||||
|
||||
## Current Console Behavior
|
||||
|
||||
Relevant page:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
|
||||
Current BGP console page has three levels:
|
||||
|
||||
1. observation summary
|
||||
- total events
|
||||
- collector count
|
||||
- prefix count
|
||||
|
||||
2. incident summary and incident table
|
||||
|
||||
3. anomaly detail table plus recent observation events
|
||||
|
||||
This means the BGP page still has useful signal even when there are zero anomalies.
|
||||
|
||||
## Known Product/Engineering Boundaries
|
||||
|
||||
1. The current system is still closer to an event board than a full BGP sensing platform.
|
||||
2. RIS coverage still needs to expand beyond narrow subscription scope.
|
||||
3. BGPStream history is still not full MRT-to-prefix decoded analytics.
|
||||
4. Collector geography still depends heavily on static RIPE RIS mappings.
|
||||
5. Incident-to-cable/IXP/region association is still weak and early-stage.
|
||||
6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths.
|
||||
|
||||
## Test Status
|
||||
|
||||
BGP-specific tests live in:
|
||||
|
||||
- `backend/tests/test_bgp.py`
|
||||
|
||||
Verified status at this point:
|
||||
|
||||
- `17 passed`
|
||||
|
||||
Covered areas include:
|
||||
|
||||
- normalization
|
||||
- observation serialization
|
||||
- enrichment
|
||||
- detectors
|
||||
- incident aggregation
|
||||
- batch anomaly creation
|
||||
- BGP events/incidents API
|
||||
- summary endpoints
|
||||
|
||||
## Most Relevant Files
|
||||
|
||||
Backend:
|
||||
|
||||
- `backend/app/models/bgp_observation.py`
|
||||
- `backend/app/models/bgp_anomaly.py`
|
||||
- `backend/app/models/bgp_incident.py`
|
||||
- `backend/app/services/collectors/bgp_common.py`
|
||||
- `backend/app/services/bgp_enrichment.py`
|
||||
- `backend/app/services/bgp_detectors.py`
|
||||
- `backend/app/services/bgp_incidents.py`
|
||||
- `backend/app/api/v1/bgp.py`
|
||||
- `backend/app/api/v1/visualization.py`
|
||||
|
||||
Frontend:
|
||||
|
||||
- `frontend/src/pages/BGP/BGP.tsx`
|
||||
- `frontend/public/earth/js/bgp.js`
|
||||
- `frontend/public/earth/js/main.js`
|
||||
- `frontend/public/earth/js/info-card.js`
|
||||
- `frontend/public/earth/js/constants.js`
|
||||
- `frontend/public/earth/index.html`
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
1. Expand realtime collector coverage and include withdrawals more broadly.
|
||||
2. Integrate real RPKI validation data.
|
||||
3. Improve route leak and path instability detectors.
|
||||
4. Strengthen incident aggregation semantics and titles.
|
||||
5. Add weak correlation from incidents to:
|
||||
- cable corridors
|
||||
- landing points
|
||||
- IXPs
|
||||
- other traffic anomaly sources
|
||||
6. Refine Earth hover/click handoff between collectors and incidents.
|
||||
Reference in New Issue
Block a user