release: bump version to 0.31.1
This commit is contained in:
rayd1o
422
docs/plans/earth-bgp-region-aggregation-plan.md
Normal file
422
docs/plans/earth-bgp-region-aggregation-plan.md
Normal file
@@ -0,0 +1,422 @@
|
||||
# BGP Region Aggregation Plan
|
||||
|
||||
## Goal
|
||||
|
||||
This document refines the current BGP `activity layer` into an implementation-ready regional aggregation design.
|
||||
|
||||
Primary product goal:
|
||||
|
||||
- turn sparse prefix-level observations, anomalies, and incidents into a readable `regional observability layer`
|
||||
- keep Earth visually alive during low-incident periods
|
||||
- make `incident markers` remain the highest-confidence foreground layer instead of replacing them
|
||||
|
||||
This layer is not a new collector, detector, or raw storage table.
|
||||
It is an aggregation/view-model layer:
|
||||
|
||||
`observations -> enrichment -> anomalies/incidents -> geography mapping -> region aggregation -> Earth/UI activity layer`
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
- collector presence alone proves coverage, but does not communicate `where routing is currently active or noisy`
|
||||
|
||||
So the missing middle layer is:
|
||||
|
||||
- `collectors` show that observation exists
|
||||
- `regions` show where activity is building up
|
||||
- `incidents` show the specific high-confidence focus events
|
||||
|
||||
## Scope
|
||||
|
||||
This plan is specifically for:
|
||||
|
||||
- a backend aggregation service
|
||||
- a summary API for console/stats
|
||||
- a GeoJSON API for Earth rendering
|
||||
- an Earth background activity layer that supports, but does not replace, incident markers
|
||||
|
||||
This plan does not attempt to solve:
|
||||
|
||||
- exact prefix geolocation quality
|
||||
- polygon-heavy geopolitical visualization
|
||||
- persistent materialized region tables in v1
|
||||
|
||||
## Region Layer Definition
|
||||
|
||||
Recommended conceptual model:
|
||||
|
||||
- `region layer` = background situational awareness
|
||||
- `incident layer` = focal event markers
|
||||
|
||||
That means:
|
||||
|
||||
- region activity should answer `where is routing behavior currently active or abnormal`
|
||||
- incident markers should answer `which concrete event should the user click`
|
||||
|
||||
## Recommended Output Model
|
||||
|
||||
Suggested backend output object:
|
||||
|
||||
## `BGPRegionActivity`
|
||||
|
||||
```json
|
||||
{
|
||||
"region_key": "sea",
|
||||
"region_name": "Southeast Asia",
|
||||
"center_lat": 1.3521,
|
||||
"center_lon": 103.8198,
|
||||
"observation_count": 128,
|
||||
"anomaly_count": 9,
|
||||
"incident_count": 2,
|
||||
"activity_score": 17.6,
|
||||
"status": "incident",
|
||||
"affected_prefix_count": 14,
|
||||
"affected_asn_count": 6,
|
||||
"collector_count": 5,
|
||||
"first_seen_at": "2026-04-02T10:00:00Z",
|
||||
"last_seen_at": "2026-04-02T10:12:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Fields To Keep In MVP
|
||||
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `center_lat`
|
||||
- `center_lon`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `activity_score`
|
||||
- `status`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
- `first_seen_at`
|
||||
- `last_seen_at`
|
||||
|
||||
### Fields To Delay
|
||||
|
||||
These are useful, but not required for the first implementation:
|
||||
|
||||
- `bounding_box`
|
||||
- `top_incident_types`
|
||||
- `top_prefixes`
|
||||
- polygon geometry
|
||||
|
||||
## Region Definition Strategy
|
||||
|
||||
### Recommendation
|
||||
|
||||
Use a static region-definition table first.
|
||||
|
||||
Examples:
|
||||
|
||||
- `north_america`
|
||||
- `south_america`
|
||||
- `western_europe`
|
||||
- `eastern_europe`
|
||||
- `east_asia`
|
||||
- `southeast_asia`
|
||||
- `south_asia`
|
||||
- `middle_east`
|
||||
- `north_africa`
|
||||
- `sub_saharan_africa`
|
||||
- `oceania`
|
||||
|
||||
Why this is the right v1 choice:
|
||||
|
||||
- stable UI semantics
|
||||
- strong readability on Earth
|
||||
- easier debugging and explanation
|
||||
- lower implementation cost than geohash or H3 grids
|
||||
|
||||
### Not Recommended For V1
|
||||
|
||||
- geohash cell aggregation
|
||||
- H3 aggregation
|
||||
- fine-grained lat/lon bucket maps
|
||||
|
||||
Those are more flexible, but they make the map feel fragmented and less explainable.
|
||||
|
||||
## Geography Mapping Strategy
|
||||
|
||||
Do not reduce the implementation to only `prefix -> exact geo`.
|
||||
|
||||
The region layer should follow the same geography-priority logic already implied by the current BGP direction:
|
||||
|
||||
1. `prefix_geography`
|
||||
2. `prefix_scope`
|
||||
3. `ASN organization region`
|
||||
4. `collector centroid` fallback
|
||||
|
||||
This matters because exact prefix geography will often be incomplete or approximate.
|
||||
The region layer should stay robust even when only partial enrichment is available.
|
||||
|
||||
## Backend Design
|
||||
|
||||
Recommended new service file:
|
||||
|
||||
- [backend/app/services/bgp_regions.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_regions.py)
|
||||
|
||||
Suggested responsibilities:
|
||||
|
||||
- `map_record_to_region(...)`
|
||||
- `aggregate_region_activity(...)`
|
||||
- `build_region_geojson(...)`
|
||||
- `resolve_activity_status(...)`
|
||||
- `compute_activity_score(...)`
|
||||
|
||||
### Data Source Inputs
|
||||
|
||||
Use a recent rolling window, default `15 minutes`, and aggregate from:
|
||||
|
||||
- `BGPObservation`
|
||||
- `BGPAnomaly`
|
||||
- active `BGPIncident`
|
||||
|
||||
### Aggregation Flow
|
||||
|
||||
1. query observations in the time window
|
||||
2. query anomalies in the same window
|
||||
3. query active incidents in the same window or active status set
|
||||
4. resolve each record to a best-effort region
|
||||
5. accumulate per-region counters
|
||||
6. compute score and status
|
||||
7. return region activity list
|
||||
|
||||
## Status Model
|
||||
|
||||
Recommended status buckets:
|
||||
|
||||
- `idle`
|
||||
- `observing`
|
||||
- `anomaly`
|
||||
- `incident`
|
||||
|
||||
Suggested rule:
|
||||
|
||||
```text
|
||||
if incident_count > 0: incident
|
||||
elif anomaly_count > 0: anomaly
|
||||
elif observation_count > 0: observing
|
||||
else: idle
|
||||
```
|
||||
|
||||
This aligns well with the current Earth status language and keeps the visual mapping simple.
|
||||
|
||||
## Activity Score
|
||||
|
||||
The score should be a tunable heuristic, not a fixed truth model.
|
||||
|
||||
Recommended v1 formula:
|
||||
|
||||
```text
|
||||
activity_score =
|
||||
min(observation_count, 50) * 0.03
|
||||
+ anomaly_count * 1.2
|
||||
+ incident_count * 5.0
|
||||
```
|
||||
|
||||
Why cap observations:
|
||||
|
||||
- observation volume is usually much larger than anomaly or incident volume
|
||||
- uncapped observation counts would overwhelm the score
|
||||
- capped observation counts preserve baseline presence without drowning real abnormality
|
||||
|
||||
### Practical Guidance
|
||||
|
||||
- treat coefficients as configuration-like constants
|
||||
- expect to retune after looking at real data
|
||||
- keep `incident` weight dominant
|
||||
|
||||
## API Design
|
||||
|
||||
### 1. Summary/List API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"window_minutes": 15,
|
||||
"regions": []
|
||||
}
|
||||
```
|
||||
|
||||
Use cases:
|
||||
|
||||
- BGP console summaries
|
||||
- right-side Earth stats
|
||||
- future region list panels
|
||||
|
||||
### 2. GeoJSON API
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "FeatureCollection",
|
||||
"features": []
|
||||
}
|
||||
```
|
||||
|
||||
Each feature should include:
|
||||
|
||||
- `geometry`
|
||||
- v1: `Point`
|
||||
- later: optional `Polygon`
|
||||
- `properties`
|
||||
- `region_key`
|
||||
- `region_name`
|
||||
- `status`
|
||||
- `activity_score`
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
- `affected_prefix_count`
|
||||
- `affected_asn_count`
|
||||
- `collector_count`
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
- `region layer` = ambient background activity
|
||||
- `incident marker` = focal event object
|
||||
|
||||
Do not replace incident markers with region markers.
|
||||
|
||||
### Region Visual Rules
|
||||
|
||||
Suggested mapping:
|
||||
|
||||
- `observing`
|
||||
- weak glow
|
||||
- low pulse or no pulse
|
||||
- `anomaly`
|
||||
- stronger glow
|
||||
- more visible pulse
|
||||
- `incident`
|
||||
- strongest regional emphasis
|
||||
- but still visually secondary to the incident marker itself
|
||||
|
||||
### Region Labels
|
||||
|
||||
Good v2 enhancement:
|
||||
|
||||
- show region name
|
||||
- show counts like `2 incidents / 5 anomalies`
|
||||
|
||||
This is useful, but should come after the core aggregation and Earth glow layer are working.
|
||||
|
||||
## Interaction Model
|
||||
|
||||
### Click Region
|
||||
|
||||
Recommended detail payload:
|
||||
|
||||
- region name
|
||||
- observation/anomaly/incident counts in the selected window
|
||||
- affected prefix count
|
||||
- affected ASN count
|
||||
- collector count
|
||||
- recent incidents in the region
|
||||
|
||||
### Click Incident
|
||||
|
||||
Keep the current incident-detail flow.
|
||||
|
||||
Interaction should feel hierarchical:
|
||||
|
||||
1. region gives situational context
|
||||
2. incident gives event focus
|
||||
|
||||
## MVP Implementation Order
|
||||
|
||||
### Step 1
|
||||
|
||||
Define static `REGIONS` in code or config.
|
||||
|
||||
### Step 2
|
||||
|
||||
Map geography-enriched BGP records into regions using the fallback chain.
|
||||
|
||||
### Step 3
|
||||
|
||||
Aggregate recent window counts:
|
||||
|
||||
- `observation_count`
|
||||
- `anomaly_count`
|
||||
- `incident_count`
|
||||
|
||||
### Step 4
|
||||
|
||||
Compute `activity_score` and `status`.
|
||||
|
||||
### Step 5
|
||||
|
||||
Expose:
|
||||
|
||||
- `/api/v1/bgp/regions/activity`
|
||||
- `/api/v1/visualization/geo/bgp-regions`
|
||||
|
||||
### Step 6
|
||||
|
||||
Render region glows on Earth behind incident markers.
|
||||
|
||||
## Out Of Scope For MVP
|
||||
|
||||
- persistent materialized region tables
|
||||
- geohash or H3 support
|
||||
- polygon-filled regional overlays
|
||||
- detailed top-prefix ranking in the first release
|
||||
- complicated scoring personalization
|
||||
|
||||
## Risks And Constraints
|
||||
|
||||
### Geography Quality
|
||||
|
||||
Prefix geography is approximate and incomplete.
|
||||
The region layer must tolerate fallback-based placement.
|
||||
|
||||
### Query Cost
|
||||
|
||||
Dynamic aggregation is the right v1 choice, but repeated short-window queries may eventually need:
|
||||
|
||||
- in-process caching
|
||||
- scheduled pre-aggregation
|
||||
- materialized summaries
|
||||
|
||||
### UI Overcrowding
|
||||
|
||||
If region glow, collector activity, and incidents all become too strong at once, Earth readability will regress.
|
||||
The region layer must remain supportive, not dominant.
|
||||
|
||||
## Final Recommendation
|
||||
|
||||
The current BGP roadmap should explicitly add:
|
||||
|
||||
- `region aggregation` as the concrete implementation of the missing `activity layer`
|
||||
|
||||
The recommended product interpretation is:
|
||||
|
||||
- `collectors` prove observation coverage
|
||||
- `regions` communicate live routing activity and abnormality
|
||||
- `incidents` remain the clearest high-confidence event objects
|
||||
|
||||
In one sentence:
|
||||
|
||||
`region aggregation is not a replacement for incidents; it is the situational background that makes sparse incidents feel legible on Earth.`
|
||||
Reference in New Issue
Block a user