linkong/planet
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix: stabilize earth bgp geography and rendering

Browse Source
This commit is contained in:
linkong
2026-04-02 15:36:20 +08:00
parent 07e4f519a1
commit e5fec8ba3d
26 changed files with 1788 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
docs/CHANGELOG.md
View File

@@ -7,6 +7,49 @@ This project follows the repository versioning rule:
- `feature` -> `+0.1.0`
- `bugfix` -> `+0.0.1`
## 0.22.10
Released: 2026-04-02
### Highlights
- Recovered the Earth-side BGP experience after a failed cache-busting / asset-loading refactor temporarily broke the globe runtime, removed textures, and made the BGP layer disappear when one backend endpoint timed out.
- Added a first usable `prefix_geography` data layer backed by `IPtoASN / IP-to-Country` ingestion so BGP geography can start moving away from pure collector-centric placement.
- Reworked BGP Earth rendering to keep collectors visible under degraded backend conditions, restore symbol-based incident markers, and split icon pulse from outward event-ring animation.
### Added
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/prefix-geography-plan.md).
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
### Improved
- Improved BGP enrichment in [bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) so events now attach `prefix_geography`, `prefix_scope`, ASN profile context, and country-centroid-backed geography hints in one place.
- Improved incident aggregation in [bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) so existing incidents refresh their regions and geography metadata instead of remaining pinned to stale first-generation evidence forever.
- Improved anomaly generation flow in [bgp_common.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/bgp_common.py) by cleaning up duplicate incident-seeding paths and only feeding newly created or refreshed anomalies forward.
- Improved Earth BGP loading in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) so collectors are now the mandatory baseline layer while anomalies and incidents can fail independently without blanking the whole BGP surface.
- Improved Earth BGP marker language in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) and [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) by restoring typed event symbols, reducing additive white blowout, and making the incident ring animation read as an outward pulse instead of a generic glow blob.
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
### Fixed
- Fixed a failed Earth asset-versioning route where hand-applied cache-busting and a parallel Vite multi-entry experiment introduced duplicate module instances, broken `/earth` boot paths, missing textures, and severe runtime instability; the globe has now been restored to the stable `frontend/public/earth` runtime instead of the abandoned refactor path.
- Fixed Earth cloud and terrain loading regressions by restoring the old static Earth entrypoint and ensuring local cloud and 8K day-map assets resolve again from `public/earth/assets`.
- Fixed a full-layer BGP disappearance regression where `bgp-anomalies` or `bgp-incidents` timeouts caused the entire BGP layer to show `0` collectors and `0` events even though collector data still existed.
- Fixed `prefix_geography` lookups in [bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) that previously failed because JSON metadata access mixed SQL column names and ORM property names.
- Fixed stale anomaly and incident geography reuse so pre-existing records can now absorb refreshed evidence instead of staying locked to older Amsterdam-centric geography forever.
- Fixed a wrong optimization path in [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py) where live `prefix_geography` lookups were pushed directly into Earth visualization endpoints, causing `bgp-anomalies` and `bgp-incidents` to time out under load; the visualization layer now prefers cached evidence again so Earth remains responsive.
- Fixed Earth-side BGP fallback rendering so anomaly fallback no longer collapses into a single undifferentiated glow layer when incidents are unavailable.
- Fixed extreme incident brightness and same-coordinate blowout in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by removing additive blending from incident cores, reducing ring intensity, and deduplicating incident rendering at the coordinate level.
- Fixed the confusing floating BGP event hub in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by removing the suspended off-surface glow anchor and its arc links, leaving Earth incidents grounded on the globe surface with regional halo context instead.
## 0.22.9
Released: 2026-04-02

126
docs/bgp-context.md
View File

@@ -6,7 +6,17 @@ The BGP module is being evolved from an anomaly-only demo into a layered observa
`raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization`
The practical product goal is to turn low-level BGP control-plane changes into understandable network situation events with collector coverage, impact regions, and incident-centric visualization.
The practical product goal is no longer just to "show incidents on the globe". The current product objective is:
1. keep BGP visually present on Earth even when incident density is low
2. make incidents clearly feel like a higher-confidence layer than anomalies
3. show that the observation network is still active even when there are no active incidents
In practice, that means Earth should behave like an observability surface, not only an incident map:
- `collectors` show that observation is happening
- `activity` shows where routing state is currently active or noisy
- `incidents` become the highest-confidence focus layer
## Current Backend Architecture
@@ -135,16 +145,26 @@ Current design:
2. Incident markers are now the primary Earth BGP markers.
3. If there are no incidents, Earth falls back to anomaly markers.
4. If there are no anomalies either, collectors still provide presence.
5. A dedicated `activity layer` now adds:
- per-collector recent 15-minute activity halos
- clustered regional activity hints derived from active collectors
6. Incident markers now use:
- symbol-driven event cores
- outward ring pulses
- reduced diffuse glow compared with older Earth builds
5. The right-side stats now show:
- BGP events
- collector count
- BGP status summary
This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus.
Current BGP status strategy:
- incidents present: show active incident count
- no incidents but anomalies present: show active anomaly count
- no incidents/anomalies but collectors present: show `当前无活跃事件`
- no incidents but anomalies present: show active anomaly count, plus active observation regions when available
- no incidents/anomalies but activity present: show `观测网络运行中`
- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件`
- no BGP data at all: show `暂无观测数据`
Earth info-card strategy:
@@ -152,6 +172,81 @@ Earth info-card strategy:
- `bgp` card is now incident-centric in wording
- `bgp_collector` card shows collector location and current event count
## Current Product Gap
The main product gap is not architecture correctness. It is low-density visualization strategy.
Current reality:
- incident count is naturally much lower than anomaly count
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
So the immediate next milestone is:
`event map -> observability map`
That means Earth needs three simultaneously readable layers:
1. `observation layer`
- collectors
- recent collector activity
- baseline coverage
2. `activity layer`
- recent event density
- anomaly/noise hotspots
- regional activity scoring
- incident presence bonus
3. `incident layer`
- sparse but highly legible, high-confidence event objects
- symbol-driven markers
- outward ring pulse instead of broad diffuse glow
## Incident Visual Direction
The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus.
Design principles:
1. `incident` markers should use a strong primary symbol
- the symbol shape should carry type meaning where possible
- examples:
- `origin_change`: triangle-like warning marker
- `mass_withdrawal`: alert/exclamation-style marker
- `more_specific_burst`: split/radiating marker
2. emphasis should come from outward ring pulses, not area flooding
- use a compact hot core
- use one or more expanding ring pulses
- avoid broad luminous blobs that make the event center feel vague
3. `collector` and `incident` must stay visually distinct
- collectors are observation infrastructure
- incidents are extracted event focus
- collector activity should stay quieter than incident pulse language
4. calm periods still need observability presence
- collectors and activity layers should keep the map alive
- once incidents appear, they should clearly dominate nearby BGP visuals
5. incident geography should become `prefix-centric`
- collectors should remain evidence sources, not the primary event location
- preferred geography priority:
- `prefix_geography`
- `prefix_scope`
- `ASN organization region`
- `collector centroid` as final fallback
- `prefix_scope` should remain an observation-derived scope hint
- a new `prefix_geography` layer should be introduced for actual prefix-centric placement
Reference inspiration:
- `World Monitor`
- sparse event symbols
- compact centers
- ring-like outward pulses
- stronger incident legibility than diffuse glow
## Current Console Behavior
Relevant page:
@@ -188,14 +283,15 @@ BGP-specific tests live in:
Verified status at this point:
- `17 passed`
- `25 passed` for `backend/tests/test_bgp.py`
- `62 passed` for `backend/tests`
Covered areas include:
- normalization
- observation serialization
- enrichment
- detectors
- detectors, including route leak candidate and path flap
- incident aggregation
- batch anomaly creation
- BGP events/incidents API
@@ -226,9 +322,14 @@ Frontend:
## Recommended Next Steps
1. Expand realtime collector coverage and include withdrawals more broadly.
2. Integrate real RPKI validation data.
3. Improve route leak and path instability detectors.
### Next Backend / Detection Priority
1. Integrate real RPKI validation data.
2. Expand realtime collector coverage and include withdrawals more broadly.
3. Continue refining route leak and path instability detectors with stronger heuristics.
### Next Correlation / Storytelling Priority
4. Strengthen incident aggregation semantics and titles.
5. Add weak correlation from incidents to:
- cable corridors
@@ -236,3 +337,12 @@ Frontend:
- IXPs
- other traffic anomaly sources
6. Refine Earth hover/click handoff between collectors and incidents.
### Next Visualization Priority
7. Refine regional activity scoring so the activity layer is informative without becoming noisy.
8. Add more incident symbol types as new detectors land.
9. Add a real prefix geography source:
- `IPtoASN / IPtoCountry` as the first practical dataset
- `OpenGeoFeed` as a higher-quality override layer
- registry/whois only as fallback

216
docs/prefix-geography-plan.md Normal file
View File

@@ -0,0 +1,216 @@
# Prefix Geography Plan
## Goal
Make Earth BGP incidents `prefix-centric` instead of `collector-centric`.
The map should primarily answer:
- where a prefix-related event is likely centered
- which regions the prefix is likely associated with
- which collectors observed the event as evidence
It should not continue to imply that the event is located at the collector itself unless no better geography is available.
## Why Current Geography Is Not Enough
Current incident geography can still collapse back to collector-derived regions because:
1. `prefix_scope` is currently built mostly from observed collector regions and historical observation regions.
2. `origin_asn_profile` currently comes from `peeringdb_network`, which is useful for ASN footprint hints but not sufficient as a primary prefix location source.
3. `collector centroid` is still a common fallback and therefore dominates sparse incidents.
This makes Earth feel like a collector map with event decorations instead of a prefix impact map.
## Data Source Layers
Prefix geography should be built from four layers, ordered by confidence.
### Layer 1. Prefix-to-country / prefix-to-region
This is the primary source layer and the current missing piece.
Recommended sources:
1. `IPtoASN / IPtoCountry`
- URL: <https://iptoasn.com/>
- Good fit for this project because it provides downloadable IPv4/IPv6 range-to-ASN and range-to-country mappings.
- Best use:
- map a prefix to country code
- enrich prefixes with coarse regional placement
2. `OpenGeoFeed`
- URL: <https://opengeofeed.org/faq/>
- Best use:
- override coarse country mappings when the prefix holder publishes a geofeed
- provide a more realistic deployment/service region than whois-style registration country
### Layer 2. Registry allocation fallback
Use these only as fallback signals, not as a ground-truth physical location.
Candidate inputs:
- RIR delegated stats
- `inetnum` / `inet6num` whois
Best use:
- detect registration country / allocation region
- provide fallback when no direct prefix geolocation dataset is available
### Layer 3. ASN footprint hints
Existing in this project:
- `peeringdb_network`
- `peeringdb_facility`
- `peeringdb_ixp`
Best use:
- derive ASN city/country footprint
- identify likely exchange/facility regions
- act as secondary evidence when prefix-specific geography is unavailable
### Layer 4. Observation evidence
Existing in this project:
- `RIPE RIS Live`
- `CAIDA BGPStream Backfill`
Best use:
- prove who observed the event
- derive affected observation regions
- support impact evidence
This should remain the final fallback and evidence layer, not the primary event geography.
## Recommended Geography Priority
The backend should compute incident geography with this order:
1. `prefix_geography`
- prefix-to-country / region / geofeed-backed result
2. `asn_region`
- ASN organization / facility / IXP footprint
3. `collector_centroid`
- observed collector regions only as final fallback
Returned GeoJSON should keep exposing the selected mode through:
- `geography_mode = prefix_geography | asn_region | collector_centroid`
## Proposed Backend Changes
### 1. Add a dedicated prefix geography dataset
New datasource candidates:
- `ip2asn_prefix_geo`
- optionally `opengeofeed_prefix_geo`
Suggested storage model:
- keep downloaded rows in `CollectedData` first for speed of integration
- later move to a dedicated table if lookup volume grows
Minimum normalized fields:
- `range_start`
- `range_end`
- `prefix`
- `country`
- `continent`
- `asn`
- `as_name`
- `source`
- `confidence`
### 2. Add prefix geography enrichment
Extend:
- `backend/app/services/bgp_enrichment.py`
New enrichment payload should include:
- `prefix_geography`
- `country`
- `continent`
- `regions`
- `source`
- `confidence`
This should be separate from the current `prefix_scope`.
Suggested distinction:
- `prefix_scope`
- observation-derived scope hint
- `prefix_geography`
- prefix-centric geography estimate
### 3. Update incident visualization geography selection
Extend:
- `backend/app/api/v1/visualization.py`
Selection order:
1. `prefix_geography.regions`
2. ASN geography hints from PeeringDB-derived profile
3. observation-derived `affected_regions`
### 4. Keep evidence visible in the frontend
Earth should distinguish:
- event center = prefix geography estimate
- evidence lines / collectors = observation proof
This keeps the event meaningful for non-expert users without losing collector evidence.
## Earth UX Result
After this change, a user should see:
- an incident marker near the estimated affected prefix region
- collectors as supporting evidence, not as the event center itself
- cables / landing points / nearby infrastructure as weak correlation around the estimated region
This makes BGP incidents readable as “where the event is likely happening or affecting”, instead of “which station saw it”.
## Implementation Order
### Phase 1
1. Add `IPtoASN / IPtoCountry` datasource support
2. Normalize rows into lookup-friendly format
3. Enrich BGP events with `prefix_geography`
4. Switch incident geography priority to prefer `prefix_geography`
### Phase 2
5. Add `OpenGeoFeed` support
6. Let geofeed override coarse country-level prefix geography
7. Add confidence scoring per geography source
### Phase 3
8. Add RIR / whois fallback
9. Add better ASN regional footprint from PeeringDB facilities / IXPs
10. Refine Earth visual semantics for prefix geography vs observation evidence
## Recommendation
The best next engineering move is:
1. integrate `IPtoASN / IPtoCountry`
2. model `prefix_geography` separately from `prefix_scope`
3. only then continue refining incident map placement
Without this layer, any further Earth tuning will still be constrained by collector-centric data.