planet/docs/technical/en/platform-data-flows.md

Business Architecture and Data Flows

This document is the business entry point for Intelligent Planet data products. It explains why each Earth data type exists, where it is collected from, which fact or derived tables it uses, and how cache invalidation plus WebSocket hints reach Earth. Frontend, backend, and Earth docs should focus on implementation details; start here when you need the cross-system data flow.

Overview

Intelligent Planet data moves through three stages:

1. Collect and normalize: built-in collectors, admin actions, or the location pipeline write PostgreSQL. Generic raw output lands in collected_data; layer-ready projections land in derived tables.
2. Project and broadcast: database triggers write fact changes to the earth_data_change_events outbox and wake the backend listener with LISTEN/NOTIFY. The listener maps the change through layer adapters, invalidates cache, and broadcasts earth_updates.
3. Reload and render in Earth: Earth receives layer-level refresh hints, applies clear_then_reload, reload, or delta, then refetches /api/v1/visualization/....

flowchart LR
  Source["External source / admin action"] --> Collector["Collector or data job"]
  Collector --> Facts["collected_data"]
  Collector --> Derived["Derived tables"]
  Facts --> Outbox["earth_data_change_events"]
  Derived --> Outbox
  Outbox --> Listener["DB change listener"]
  Listener --> Cache["Earth cache invalidation"]
  Listener --> WS["earth_updates"]
  WS --> Earth["Earth layer reload"]
  Earth --> API["Visualization APIs"]

LISTEN/NOTIFY is only the low-latency wakeup path. The reliable source is the outbox. A real zero-data state is valid and APIs should return HTTP 200 with an empty collection; only actual endpoint failures should return 5xx.

Data Products

flowchart TB
  Satellites["Satellite TLE"] --> SatelliteLayer["satellites layer"]
  Cables["Cables + landing points"] --> CableLayer["cables layer"]
  Compute["TOP500 / AI GPU / HF"] --> ComputeLocations["compute_center_locations"]
  ComputeLocations --> ComputeLayer["computeCenters layer"]
  BgpRaw["RIS Live / BGPStream / Prefix"] --> BgpDerived["bgp_observations / anomalies / incidents"]
  BgpDerived --> BgpLayer["bgp layer"]
  VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
  VesselDerived --> VesselLayer["vessels layer"]
  Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
  NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
  NewsItems --> NewsLayer["news / media layer"]

Data product	Business use	Fact sources	Derived / dimension tables	Earth layer	Strategy
Satellites	In-orbit objects, tracks, coverage, and cruise targets	celestrak_tle, spacetrack_tle	No long-lived projection; APIs convert TLE on demand	satellites	clear_then_reload
Cables and landing points	Submarine connectivity, landing cities, cable detail, and search	arcgis_cables, arcgis_landing_points, TeleGeography / FAO landing sources	Cable relation and landing aggregation data	cables	clear_then_reload
Compute centers	TOP500, AI GPU, model platform facilities, and location-completion state	top500, epoch_ai_gpu, HuggingFace sources	compute_center_locations	computeCenters	reload
BGP context	Collectors, anomalies, incidents, route events, and regional context	ris_live_bgp, bgpstream_bgp, prefix geography sources	bgp_observations, bgp_anomalies, bgp_incidents, bgp_collector_locations	bgp	clear_then_reload
Vessels	AIS vessels, positions, tracks, legend, and source health	AIS sources, barentswatch_vessels	vessel_static, vessel_position, ais_raw_observations, ais_source_health	vessels	clear_then_reload
Interactables	Generic surface icons, manual objects, and future small layers	earth_interactables	None	interactables	delta
News and media	Earth news, live streams, cruise summaries, and situation content	news sources	earth_news_items	news / media	reload

Satellites

Satellite data powers online satellite markers, tracks, footprint policy, and cruise lists. CelesTrak or Space-Track collectors fetch TLE and write collected_data. The visualization API converts TLE into current positions and trails on request.

• Collection entry: CelesTrak TLE, Space-Track TLE.
• Fact table: collected_data.source IN ('celestrak_tle', 'spacetrack_tle').
• API: satellite visualization API reads TLE and builds the Earth payload.
• Delete semantics: deleting the source broadcasts satellites with clear_then_reload; the frontend clears satellite markers and trails before refetching. With no TLE, the API returns an empty list.
• Common failure: summary is zero but Earth still shows satellites. Usually WS did not fire, the adapter did not cover the source, or the frontend did not clear existing Three.js objects.

Cables and Landing Points

Cables and landing points show global network connectivity, landing cities, cable details, and search targets. Cable lines and landing points are one business layer; deleting either side must refresh cables.

• Collection entry: ArcGIS cables, ArcGIS landing points, and TeleGeography / FAO landing related sources.
• Fact table: cable and landing point sources in collected_data.
• Derived data: cable relation tables, landing point aggregation, interface cache.
• API: cable visualization API returns cables, landing points, and relation data.
• Delete semantics: deleting cable or landing source clears owned projections and broadcasts cables with clear_then_reload. Empty data returns HTTP 200, not 404.
• Common failure: cables disappear for a second and return. Usually a collection replacement or cache refresh window reuses old cache; check duplicate broadcasts and cache patterns.

Compute Centers

Compute centers represent supercomputers, AI GPU sites, model-platform facilities, and unresolved-location state. Raw sources often only provide organization, country, site, or fuzzy location, so Earth rendering depends on compute_center_locations.

• Collection entry: TOP500, Epoch AI GPU, HuggingFace related sources.
• Fact table: compute sources in collected_data.
• Dimension table: compute_center_locations stores adopted coordinate candidates.
• API: compute center visualization API merges raw records with location rows.
• Delete semantics: deleting TOP500 or similar sources refreshes computeCenters; changing the location table also refreshes the layer. The default is reload, because location updates do not always need an immediate clear.
• Common failure: collection finished but Earth count did not change. Check visualization cache, location rows, and adapter coverage.

BGP

BGP data shows route collectors, anomalies, incidents, diffusion rings, and situation summaries. Earth does not render raw BGP rows directly; it renders projected observations, anomalies, and incidents. This is the most common path where deleting raw data can leave old objects visible.

sequenceDiagram
  participant UI as Admin delete button
  participant Job as clear_data job
  participant DB as PostgreSQL
  participant Outbox as earth_data_change_events
  participant Listener as DB change listener
  participant Earth as Earth frontend

  UI->>Job: Submit delete for ris_live_bgp / bgpstream_bgp
  Job->>DB: Delete collected_data raw rows
  Job->>DB: Delete BGP-owned derived tables
  DB->>Outbox: trigger writes bgp layer changed
  DB-->>Listener: LISTEN/NOTIFY wakeup
  Listener->>Listener: merge events and clear bgp cache
  Listener-->>Earth: broadcast earth_updates clear_then_reload
  Earth->>Earth: clear BGP objects
  Earth->>DB: refetch projections through visualization API

• Collection entry: RIPE RIS Live BGP, CAIDA BGPStream Backfill, IPtoASN / OpenGeoFeed / NRO prefix geography.
• Fact table: BGP and prefix sources in collected_data.
• Derived tables: bgp_observations, bgp_anomalies, bgp_incidents, bgp_collector_locations.
• API: BGP visualization API reads projections and combines stored or candidate locations.
• Delete semantics: deleting RIS Live or BGPStream raw sources must also clear BGP-owned projections and broadcast bgp with clear_then_reload. Direct derived-table deletes also write outbox events.
• Common failure: collectors or incidents remain after raw rows are removed. Check derived tables first, not only collected_data.

Vessels

Vessel data shows AIS vessels, navigation state, vessel-type legend, and source health. Earth rendering uses position snapshots and static vessel information; it should not render every raw AIS observation.

• Collection entry: AIS sources, BarentsWatch vessels.
• Fact table: collected_data or AIS raw observation tables.
• Derived tables: vessel_static, vessel_position, ais_raw_observations, ais_source_health.
• API: vessel visualization API returns current vessel markers and detail fields.
• Delete semantics: deleting any vessel source lets owned projections broadcast vessels with clear_then_reload.
• Common failure: the count panel changes but vessels remain. Summary and layer data are separate; Earth should clear objects based on layer updates.

Interactables

earth_interactables is the generic surface-icon capability for manual objects, extensions, and future small layers. Unlike most layers, it keeps object-level delta.

• Fact table: earth_interactables.
• API: interactable API and Earth generic icon interface.
• Strategy: upsert for create/update and removeItem for delete; no full layer reload.
• Delete semantics: delete payloads must include a stable id so the frontend can remove the object.
• Common failure: an object cannot be removed. Usually the stable id is missing, the delta handler did not match the object type, or an older duplicate render path still exists.

News and Media

News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to reload.

• Collection entry: RSS, live streams, news sources.
• Fact table: news source rows in collected_data.
• Derived table: earth_news_items.
• API: news, live stream, and media content APIs.
• Delete semantics: deleting news sources or earth_news_items broadcasts news / media reload; empty responses hide the corresponding content.
• Common failure: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.

Adding a New Layer

Add a new Earth data product in this order:

1. Define the business purpose and Earth layer name.
2. Identify fact sources, fact tables, and derived tables.
3. Register source/table coverage, cache patterns, owned derived cleanup, and default strategy in the backend layer adapter.
4. Ensure the visualization API returns HTTP 200 with an empty collection for zero data.
5. Keep the Earth frontend driven only by layer and strategy, not database table names.
6. Update this document with the data product flow, then document implementation details in the relevant frontend, backend, or Earth technical doc.