planet/docs/technical/en/backend-collectors.md

Data Collectors

I. System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Data Collection Architecture                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐       │
│   │  TOP500     │    │  Epoch AI   │    │ HuggingFace │       │
│   │  Collector  │    │  Collector  │    │  Collector  │       │
│   └──────┬──────┘    └──────┬──────┘    └──────┬──────┘       │
│          │                   │                   │              │
│          └───────────────────┼───────────────────┘              │
│                              ▼                                  │
│                  ┌─────────────────────┐                        │
│                  │   BaseCollector    │◄── Base class (unified) │
│                  │   run() method     │                        │
│                  └─────────┬───────────┘                        │
│                            │                                     │
│          ┌─────────────────┼─────────────────┐                   │
│          ▼                 ▼                 ▼                   │
│   ┌───────────┐    ┌───────────┐    ┌───────────┐              │
│   │ fetch()   │    │transform()│    │ _save_data│              │
│   │ raw data  │    │ transform │    │ save to DB│              │
│   └───────────┘    └───────────┘    └───────────┘              │
│                              │                                  │
│                              ▼                                  │
│                  ┌─────────────────────┐                        │
│                  │  CollectedData table│◄── Unified storage     │
│                  └─────────────────────┘                        │
│                                                                 │
│   ┌─────────────────────────────────────────────────────────┐  │
│   │                    Scheduler (APScheduler)               │  │
│   │   Scheduled tasks: every 4h/6h/12h/1d auto-execute      │  │
│   └─────────────────────────────────────────────────────────┘  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

II. Pipeline

# 1. Scheduler triggers (scheduled or manual)
#    ↓

# 2. run() executes the full pipeline
async def run(self, db):
    # 2.1 Check if collector is enabled
    if not collector_registry.is_active(self.name):
        return {"status": "skipped"}

    # 2.2 Record task start
    task = CollectionTask(status="running")
    db.add(task)
    await db.commit()

    # 2.3 FETCH — get raw data (implemented by subclass)
    raw_data = await self.fetch()

    # 2.4 TRANSFORM — convert to unified format
    data = self.transform(raw_data)

    # 2.5 SAVE — persist to database
    records_count = await self._save_data(db, data)

    # 2.6 Record task completion
    task.status = "success"
    task.records_processed = records_count
    await db.commit()

Core file: backend/app/services/collectors/base.py

III. Collector List

Collector	Data type	Content	Frequency
TOP500	supercomputer	Global supercomputer rankings (compute, performance)	4 hours
Epoch AI	gpu_cluster	GPU compute cluster info	6 hours
HuggingFace Models	model	AI model information	12 hours
HuggingFace Datasets	dataset	Dataset information	12 hours
HuggingFace Spaces	space	Demo applications	1 day
PeeringDB	ixp/network/facility	Internet exchange points / networks / facilities	1-2 days
TeleGeography	submarine_cable	Submarine cable information	7 days
BarentsWatch AIS	vessel	AIS vessel positions, speed, heading, MMSI, and related fields	Collector settings
AISStream Vessels	vessel_ais	AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation	Collector settings

AIS vessel collectors use a different persistence path from regular CollectedData collectors. BarentsWatch, AISStream, and custom vessel_ais sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.

Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and /api/v1/earth/boundaries/* builds frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to CollectedData.

TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or compute_center_locations dimension-table coordinates during the main map startup path; records without coordinates are returned as unresolved instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into compute_center_locations and rendered from that table on the next layer refresh.

Admin Next collection management follows the business hierarchy instead of flattening every endpoint into one table:

• Collectors: endpoint, authentication, headers, base parameters, enabled state, and credential guides.
• Collection Schedule: scheduler state and task controls.
• Collection History / Snapshots: history grouped by collector, with a detail-side snapshot selector for versions.

Snapshot lists should not show every snapshot of the same collector as separate top-level records. The top-level list selects a collector; the detail area switches between time versions.

Credential guides are maintained by backend/app/services/credential_guides.py. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.

IV. Data Format (stored in CollectedData table)

# Each collector's parse_response() return format
{
    "source_id": "top500_1",          # Original system ID (required)
    "name": "El Capitan",             # Name (required)
    "description": "System desc...",  # Description
    "country": "United States",       # Country
    "city": "Livermore, CA",          # City
    "latitude": "37.6819",            # Latitude (string)
    "longitude": "-121.7681",         # Longitude (string)
    "value": "1742.00",               # Performance value (e.g. compute)
    "unit": "PFlop/s",                # Unit
    "metadata": {                     # Extra data (JSON)
        "rank": 1,
        "r_peak": 2746.38,
        "cores": 11039616
    },
    "reference_date": "2025-11-01"    # Data reference date
}

V. Database Schema

CollectedData table (collected_data)

Field	Type	Description
id	SERIAL	Primary key
source	VARCHAR(100)	Data source name (top500, huggingface, etc.)
source_id	VARCHAR(100)	Original data ID
data_type	VARCHAR(50)	Data type (supercomputer, model, etc.)
name	VARCHAR(500)	Name
title	VARCHAR(500)	Title
description	TEXT	Description
country	VARCHAR(100)	Country
city	VARCHAR(100)	City
latitude	VARCHAR(50)	Latitude
longitude	VARCHAR(50)	Longitude
value	VARCHAR(100)	Performance value
unit	VARCHAR(20)	Unit
metadata	JSONB	Extra metadata
collected_at	TIMESTAMP	Collection time
reference_date	TIMESTAMP	Data reference date
is_valid	INTEGER	Whether valid

Core file: backend/app/models/collected_data.py

VI. TOP500 Collector Example (full pipeline)

# 1. fetch() — get HTML from the web
async def fetch(self):
    url = "https://top500.org/lists/top500/list/2025/11/"
    response = await client.get(url)
    return response.text  # returns HTML

# 2. parse_response() — parse HTML into unified format
def parse_response(self, html):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.find("table")

    for row in table.find_all("tr")[1:]:  # skip header
        cells = row.find_all("td")

        entry = {
            "source_id": f"top500_{cells[0].text}",
            "name": cells[1].text.strip(),
            "country": cells[2].text.strip(),
            "city": "",
            "latitude": "",
            "longitude": "",
            "value": "1742.00",
            "unit": "PFlop/s",
            "metadata": {
                "rank": 1,
                "cores": "11340000"
            },
            "reference_date": "2025-11-01"
        }
        data.append(entry)

    return data

# 3. run() automatically calls _save_data() to save to database

Core file: backend/app/services/collectors/top500.py

VII. Scheduler

# Register all collectors into scheduled tasks at startup
def start_scheduler():
    for name, collector in collectors.items():
        if collector_registry.is_active(name):
            scheduler.add_job(
                run_collector_task,
                trigger=IntervalTrigger(hours=collector.frequency_hours),
                id=name,
                name=name
            )

Collector	Frequency
TOP500	Every 4 hours
Epoch AI	Every 6 hours
HuggingFace	Every 12 hours
PeeringDB	Every 1-2 days
TeleGeography	Every 7 days

Core file: backend/app/services/scheduler.py

VIII. Code Files

backend/app/services/collectors/
├── base.py              # Base class: run() pipeline, _save_data() persistence
├── registry.py          # Collector registry
├── scheduler.py         # Scheduled task dispatch (APScheduler)
├── top500.py            # TOP500 collector
├── epoch_ai.py          # Epoch AI collector
├── huggingface.py       # HuggingFace collector
├── peeringdb.py         # PeeringDB collector
├── telegeraphy.py       # TeleGeography submarine cable collector
├── vessel_ais.py        # BarentsWatch AIS vessel collector
├── aisstream.py         # AISStream WebSocket vessel collector
└── earth_boundaries.py  # Earth boundary source verification and static tile artifact collector

backend/app/services/
├── custom_datasource_runtime.py      # Custom REST / WebSocket mapping runtime
├── datasource_mapping.py             # Deterministic field mapping and target writes
├── vessel_ais_aggregation.py         # AIS raw observation writes and aggregate reads
├── vessel_aggregation_strategy.py    # Multi-source field selection, freshness fallback, and conflict records
└── vessel_enrichment.py              # Vessel profile enrichment cache

backend/app/models/
├── collected_data.py           # Unified data model
└── vessel_enrichment.py        # Vessel enrichment cache

IX. Credentialed Collectors

Some collectors require external service credentials:

Collector	Credential provider	Credential sources
barentswatch_vessels	barentswatch	Console collector settings, environment variables, ~/.zshrc
aisstream_vessels	aisstream	Console collector settings, environment variables, ~/.zshrc for connectivity checks; save it in collector settings or inject it into the backend environment for collection
spacetrack_tle	spacetrack	Environment variables, ~/.zshrc

BarentsWatch AIS

BarentsWatch AIS credential resolution is centralized in:

• barentswatch.py

VesselAISCollector only collects and transforms AIS data. It no longer reads environment variables or builds token requests directly. It uses:

• resolve_barentswatch_config()
• fetch_barentswatch_access_token()

Resolution priority:

1. DataSourceConfig.auth_config
2. DataSourceConfig.config
3. Environment variables
4. ~/.zshrc

Supported variables:

export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."

Historical misspellings are also supported:

export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."

Connectivity validation requests https://id.barentswatch.no/connect/token for an access token with scope=ais, then requests the AIS endpoint with Authorization: Bearer <token>.

AISStream Realtime Vessels

AISStream uses the wss://stream.aisstream.io/v0/stream WebSocket endpoint. Its default runtime is a long-lived realtime collector rather than the traditional REST pattern of one request, progress to 100%, then completion.

Runtime configuration:

• api_key: read first from DataSourceConfig.auth_config.api_key or config.api_key; it can also come from the backend process environment variable AISSTREAM_API_KEY.
• bounding_boxes: AISStream subscription bounds. The default example is global [[[-90, -180], [90, 180]]]; demos and production runs should usually start with a smaller area.
• message_types: defaults to PositionReport and ShipStaticData.
• streaming_enabled: enables long-lived streaming by default; disabling it falls back to batch-style fetch -> transform -> save.
• streaming_max_messages: test-only stop limit. Non-zero values stop the stream after the requested number of messages.
• reconnect_delay_seconds and receive_timeout_seconds: control reconnect delay and idle receive waits.

State semantics:

• connecting: connecting to AISStream.
• streaming: receiving realtime messages; records_processed means messages seen, usually without a fixed total or percentage.
• reconnecting: upstream or network interruption; the collector records AISSourceHealth and waits before reconnecting.
• stopped / cancelled: stopped by a test limit or user action.

AISStream connectivity validation reads the saved collector configuration, environment variables, and AISSTREAM_API_KEY in ~/.zshrc through datasource_connectivity.py. For actual collection, the most reliable path is saving the API key in Collection Management -> Collectors -> AISStream Vessels; if the key only lives in ~/.zshrc, confirm that the backend process inherited it.

The console manages AISStream from /datasources -> Realtime Streams, not from the normal finite collection progress bar. The realtime stream API aggregates runtime state, health, configuration preview, and raw observation counters:

GET /api/v1/realtime-sources
POST /api/v1/realtime-sources/{source}/start
POST /api/v1/realtime-sources/{source}/stop
POST /api/v1/realtime-sources/{source}/restart

aisstream_vessels and custom source_type=websocket sources appear in that API. They do not participate in one-click collection percentages; the UI interprets them as long-lived services with message counters, lag, last success, and last error.

AIS Raw Observations And Aggregation

AIS observations do not directly replace final vessel records. They are first saved as raw observations:

• source records the origin, such as barentswatch_vessels, aisstream_vessels, or a custom source name.
• delivery_mode captures realtime quality; realtime_stream outranks polling.
• transport records websocket or http.
• Dynamic fields such as position, speed, and course are selected by freshness and source priority.
• Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.

Earth vessel rendering now consumes the bounded snapshot endpoint and realtime delta channel:

GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts

/api/v1/vessels/snapshot requires bbox and zoom, defaults to limit=1000, and caps limit at 5000. It prefers aggregated ais_raw_observations; when the current raw window is empty, it can fall back to the latest legacy vessel_position / vessel_static rows and marks that path with diagnostics.legacy_fallback_used. The old /api/v1/visualization/geo/vessels route has been removed.

Realtime deltas are sent through the /ws vessels channel. Clients must subscribe with the current viewport:

{
  "type": "subscribe",
  "data": {
    "channel": "vessels",
    "bbox": [120.8, 30.7, 122.1, 31.8],
    "zoom": 12,
    "limit": 1000
  }
}

The backend stores lightweight subscription filters per connection and only sends vessel updates that match the subscriber bbox. Collector broadcasts enter a 1-second throttle queue; within each flush window, only the latest update per MMSI is retained.

Layer APIs And Global Stats

Earth is moving to two API families:

GET /api/v1/data-products
GET /api/v1/data-products/{product_id}/status
GET /api/v1/layers/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/cables?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/landing-points?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/satellites?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/bgp/anomalies?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/bgp/incidents?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
GET /api/v1/layers/bgp/collectors?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000

/api/v1/data-products/* is for aggregate panels and keeps a global statistics scope independent of the map bbox. /api/v1/layers/* is for map rendering, requires bbox and zoom, defaults to limit=1000, and caps limit at 5000; low zoom falls back to a smaller response cap and reports degraded, truncated, limit_clamped, and stats_scope=viewport in diagnostics. Non-vessel layers currently reuse the existing GeoJSON converters before the guard layer; future product-specific queries can push bbox filtering deeper.

X. Collectors And Connectivity Validation

The console "Collectors" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:

• endpoint
• auth type
• headers
• config
• credential provider
• credential fingerprint

Related APIs:

GET  /api/v1/datasources/configs/all
POST /api/v1/datasources/configs/builtin/connection-status
POST /api/v1/datasources/configs/builtin/connect
POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset

See Collectors and Connectivity Validation for the full flow.

XI. Data Usage

Collected data ultimately:

1. Visualization — displays supercomputers, GPU clusters, and submarine cables' geographic positions
2. Situational analysis — global compute distribution statistics and growth trends
3. Alert system — detects changes to important nodes

XII. Collector Registration

Collectors are automatically registered at application startup:

# backend/app/services/collectors/__init__.py

collector_registry.register(TOP500Collector())
collector_registry.register(EpochAIGPUCollector())
collector_registry.register(HuggingFaceModelCollector())
collector_registry.register(HuggingFaceDatasetCollector())
collector_registry.register(HuggingFaceSpacesCollector())
collector_registry.register(PeeringDBIXPCollector())
collector_registry.register(PeeringDBNetworkCollector())
collector_registry.register(PeeringDBFacilityCollector())
collector_registry.register(TeleGeographyCableCollector())
collector_registry.register(TeleGeographyLandingPointCollector())
collector_registry.register(TeleGeographyCableSystemCollector())

Core file: backend/app/services/collectors/registry.py

XIII. Triggering Collection

Method 1: Scheduled

At startup, APScheduler automatically creates scheduled tasks based on each collector's frequency_hours setting.

Method 2: Manual API trigger

# Trigger TOP500 collection
curl -X POST http://localhost:8000/api/v1/datasources/1/trigger \
  -H "Authorization: Bearer <token>"

Batch collection uses:

POST /api/v1/datasources/trigger-batch

The request body may pass source_ids for selected rows. Without source_ids, the backend filters by product, module, is_active, run_status, collected, credential_status, and q. The endpoint skips disabled sources, sources already running without force, and sources still inside their frequency window, then returns triggered, skipped, and failed groups.

Core file: backend/app/api/v1/datasources.py