planet/docs/technical/en/backend-collectors.md

Data Collectors

I. System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Data Collection Architecture                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐       │
│   │  TOP500     │    │  Epoch AI   │    │ HuggingFace │       │
│   │  Collector  │    │  Collector  │    │  Collector  │       │
│   └──────┬──────┘    └──────┬──────┘    └──────┬──────┘       │
│          │                   │                   │              │
│          └───────────────────┼───────────────────┘              │
│                              ▼                                  │
│                  ┌─────────────────────┐                        │
│                  │   BaseCollector    │◄── Base class (unified) │
│                  │   run() method     │                        │
│                  └─────────┬───────────┘                        │
│                            │                                     │
│          ┌─────────────────┼─────────────────┐                   │
│          ▼                 ▼                 ▼                   │
│   ┌───────────┐    ┌───────────┐    ┌───────────┐              │
│   │ fetch()   │    │transform()│    │ _save_data│              │
│   │ raw data  │    │ transform │    │ save to DB│              │
│   └───────────┘    └───────────┘    └───────────┘              │
│                              │                                  │
│                              ▼                                  │
│                  ┌─────────────────────┐                        │
│                  │  CollectedData table│◄── Unified storage     │
│                  └─────────────────────┘                        │
│                                                                 │
│   ┌─────────────────────────────────────────────────────────┐  │
│   │                    Scheduler (APScheduler)               │  │
│   │   Scheduled tasks: every 4h/6h/12h/1d auto-execute      │  │
│   └─────────────────────────────────────────────────────────┘  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

II. Pipeline

# 1. Scheduler triggers (scheduled or manual)
#    ↓

# 2. run() executes the full pipeline
async def run(self, db):
    # 2.1 Check if collector is enabled
    if not collector_registry.is_active(self.name):
        return {"status": "skipped"}

    # 2.2 Record task start
    task = CollectionTask(status="running")
    db.add(task)
    await db.commit()

    # 2.3 FETCH — get raw data (implemented by subclass)
    raw_data = await self.fetch()

    # 2.4 TRANSFORM — convert to unified format
    data = self.transform(raw_data)

    # 2.5 SAVE — persist to database
    records_count = await self._save_data(db, data)

    # 2.6 Record task completion
    task.status = "success"
    task.records_processed = records_count
    await db.commit()

Core file: backend/app/services/collectors/base.py

III. Collector List

Collector	Data type	Content	Frequency
TOP500	supercomputer	Global supercomputer rankings (compute, performance)	4 hours
Epoch AI	gpu_cluster	GPU compute cluster info	6 hours
HuggingFace Models	model	AI model information	12 hours
HuggingFace Datasets	dataset	Dataset information	12 hours
HuggingFace Spaces	space	Demo applications	1 day
PeeringDB	ixp/network/facility	Internet exchange points / networks / facilities	1-2 days
TeleGeography	submarine_cable	Submarine cable information	7 days
BarentsWatch AIS	vessel	AIS vessel positions, speed, heading, MMSI, and related fields	Collector settings
AISStream Vessels	vessel_ais	AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation	Collector settings

AIS vessel collectors use a different persistence path from regular CollectedData collectors. BarentsWatch, AISStream, and custom vessel_ais sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.

TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or compute_center_locations dimension-table coordinates during the main map startup path; records without coordinates are returned as unresolved instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into compute_center_locations and rendered from that table on the next layer refresh.

IV. Data Format (stored in CollectedData table)

# Each collector's parse_response() return format
{
    "source_id": "top500_1",          # Original system ID (required)
    "name": "El Capitan",             # Name (required)
    "description": "System desc...",  # Description
    "country": "United States",       # Country
    "city": "Livermore, CA",          # City
    "latitude": "37.6819",            # Latitude (string)
    "longitude": "-121.7681",         # Longitude (string)
    "value": "1742.00",               # Performance value (e.g. compute)
    "unit": "PFlop/s",                # Unit
    "metadata": {                     # Extra data (JSON)
        "rank": 1,
        "r_peak": 2746.38,
        "cores": 11039616
    },
    "reference_date": "2025-11-01"    # Data reference date
}

V. Database Schema

CollectedData table (collected_data)

Field	Type	Description
id	SERIAL	Primary key
source	VARCHAR(100)	Data source name (top500, huggingface, etc.)
source_id	VARCHAR(100)	Original data ID
data_type	VARCHAR(50)	Data type (supercomputer, model, etc.)
name	VARCHAR(500)	Name
title	VARCHAR(500)	Title
description	TEXT	Description
country	VARCHAR(100)	Country
city	VARCHAR(100)	City
latitude	VARCHAR(50)	Latitude
longitude	VARCHAR(50)	Longitude
value	VARCHAR(100)	Performance value
unit	VARCHAR(20)	Unit
metadata	JSONB	Extra metadata
collected_at	TIMESTAMP	Collection time
reference_date	TIMESTAMP	Data reference date
is_valid	INTEGER	Whether valid

Core file: backend/app/models/collected_data.py

VI. TOP500 Collector Example (full pipeline)

# 1. fetch() — get HTML from the web
async def fetch(self):
    url = "https://top500.org/lists/top500/list/2025/11/"
    response = await client.get(url)
    return response.text  # returns HTML

# 2. parse_response() — parse HTML into unified format
def parse_response(self, html):
    soup = BeautifulSoup(html, "html.parser")
    table = soup.find("table")

    for row in table.find_all("tr")[1:]:  # skip header
        cells = row.find_all("td")

        entry = {
            "source_id": f"top500_{cells[0].text}",
            "name": cells[1].text.strip(),
            "country": cells[2].text.strip(),
            "city": "",
            "latitude": "",
            "longitude": "",
            "value": "1742.00",
            "unit": "PFlop/s",
            "metadata": {
                "rank": 1,
                "cores": "11340000"
            },
            "reference_date": "2025-11-01"
        }
        data.append(entry)

    return data

# 3. run() automatically calls _save_data() to save to database

Core file: backend/app/services/collectors/top500.py

VII. Scheduler

# Register all collectors into scheduled tasks at startup
def start_scheduler():
    for name, collector in collectors.items():
        if collector_registry.is_active(name):
            scheduler.add_job(
                run_collector_task,
                trigger=IntervalTrigger(hours=collector.frequency_hours),
                id=name,
                name=name
            )

Collector	Frequency
TOP500	Every 4 hours
Epoch AI	Every 6 hours
HuggingFace	Every 12 hours
PeeringDB	Every 1-2 days
TeleGeography	Every 7 days

Core file: backend/app/services/scheduler.py

VIII. Code Files

backend/app/services/collectors/
├── base.py              # Base class: run() pipeline, _save_data() persistence
├── registry.py          # Collector registry
├── scheduler.py         # Scheduled task dispatch (APScheduler)
├── top500.py            # TOP500 collector
├── epoch_ai.py          # Epoch AI collector
├── huggingface.py       # HuggingFace collector
├── peeringdb.py         # PeeringDB collector
├── telegeraphy.py       # TeleGeography submarine cable collector
├── vessel_ais.py        # BarentsWatch AIS vessel collector
└── aisstream.py         # AISStream WebSocket vessel collector

backend/app/services/
├── custom_datasource_runtime.py      # Custom REST / WebSocket mapping runtime
├── datasource_mapping.py             # Deterministic field mapping and target writes
├── vessel_ais_aggregation.py         # AIS raw observation writes and aggregate reads
├── vessel_aggregation_strategy.py    # Multi-source field selection, freshness fallback, and conflict records
└── vessel_enrichment.py              # Vessel profile enrichment cache

backend/app/models/
├── collected_data.py           # Unified data model
└── vessel_enrichment.py        # Vessel enrichment cache

IX. Credentialed Collectors

Some collectors require external service credentials:

Collector	Credential provider	Credential sources
barentswatch_vessels	barentswatch	Console collector settings, environment variables, ~/.zshrc
aisstream_vessels	aisstream	Console collector settings, environment variables, ~/.zshrc for connectivity checks; save it in collector settings or inject it into the backend environment for collection
spacetrack_tle	spacetrack	Environment variables, ~/.zshrc

BarentsWatch AIS

BarentsWatch AIS credential resolution is centralized in:

• barentswatch.py

VesselAISCollector only collects and transforms AIS data. It no longer reads environment variables or builds token requests directly. It uses:

• resolve_barentswatch_config()
• fetch_barentswatch_access_token()

Resolution priority:

1. DataSourceConfig.auth_config
2. DataSourceConfig.config
3. Environment variables
4. ~/.zshrc

Supported variables:

export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."

Historical misspellings are also supported:

export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."

Connectivity validation requests https://id.barentswatch.no/connect/token for an access token with scope=ais, then requests the AIS endpoint with Authorization: Bearer <token>.

AISStream Realtime Vessels

AISStream uses the wss://stream.aisstream.io/v0/stream WebSocket endpoint. Its default runtime is a long-lived realtime collector rather than the traditional REST pattern of one request, progress to 100%, then completion.

Runtime configuration:

• api_key: read first from DataSourceConfig.auth_config.api_key or config.api_key; it can also come from the backend process environment variable AISSTREAM_API_KEY.
• bounding_boxes: AISStream subscription bounds. The default example is global [[[-90, -180], [90, 180]]]; demos and production runs should usually start with a smaller area.
• message_types: defaults to PositionReport and ShipStaticData.
• streaming_enabled: enables long-lived streaming by default; disabling it falls back to batch-style fetch -> transform -> save.
• streaming_max_messages: test-only stop limit. Non-zero values stop the stream after the requested number of messages.
• reconnect_delay_seconds and receive_timeout_seconds: control reconnect delay and idle receive waits.

State semantics:

• connecting: connecting to AISStream.
• streaming: receiving realtime messages; records_processed means messages seen, usually without a fixed total or percentage.
• reconnecting: upstream or network interruption; the collector records AISSourceHealth and waits before reconnecting.
• stopped / cancelled: stopped by a test limit or user action.

AISStream connectivity validation reads the saved collector configuration, environment variables, and AISSTREAM_API_KEY in ~/.zshrc through datasource_connectivity.py. For actual collection, the most reliable path is saving the API key in Settings -> Collector Settings -> AISStream Vessels; if the key only lives in ~/.zshrc, confirm that the backend process inherited it.

AIS Raw Observations And Aggregation

AIS observations do not directly replace final vessel records. They are first saved as raw observations:

• source records the origin, such as barentswatch_vessels, aisstream_vessels, or a custom source name.
• delivery_mode captures realtime quality; realtime_stream outranks polling.
• transport records websocket or http.
• Dynamic fields such as position, speed, and course are selected by freshness and source priority.
• Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.

Earth still reads vessel data from:

GET /api/v1/visualization/geo/vessels
GET /api/v1/visualization/vessels/{mmsi}
GET /api/v1/visualization/vessels/{mmsi}/track
GET /api/v1/visualization/vessels/{mmsi}/conflicts
GET /api/v1/visualization/vessels/aggregation/diagnostics

/geo/vessels merges raw observation aggregation with the legacy BarentsWatch latest-position tables so adding AISStream does not hide historical BarentsWatch-only vessels.

X. Collector Settings And Connectivity Validation

The console "Collector Settings" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:

• endpoint
• auth type
• headers
• config
• credential provider
• credential fingerprint

Related APIs:

GET  /api/v1/datasources/configs/all
POST /api/v1/datasources/configs/builtin/connection-status
POST /api/v1/datasources/configs/builtin/connect
POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset

See Collector Settings and Connectivity Validation for the full flow.

XI. Data Usage

Collected data ultimately:

1. Visualization — displays supercomputers, GPU clusters, and submarine cables' geographic positions
2. Situational analysis — global compute distribution statistics and growth trends
3. Alert system — detects changes to important nodes

XII. Collector Registration

Collectors are automatically registered at application startup:

# backend/app/services/collectors/__init__.py

collector_registry.register(TOP500Collector())
collector_registry.register(EpochAIGPUCollector())
collector_registry.register(HuggingFaceModelCollector())
collector_registry.register(HuggingFaceDatasetCollector())
collector_registry.register(HuggingFaceSpacesCollector())
collector_registry.register(PeeringDBIXPCollector())
collector_registry.register(PeeringDBNetworkCollector())
collector_registry.register(PeeringDBFacilityCollector())
collector_registry.register(TeleGeographyCableCollector())
collector_registry.register(TeleGeographyLandingPointCollector())
collector_registry.register(TeleGeographyCableSystemCollector())

Core file: backend/app/services/collectors/registry.py

XIII. Triggering Collection

Method 1: Scheduled

At startup, APScheduler automatically creates scheduled tasks based on each collector's frequency_hours setting.

Method 2: Manual API trigger

# Trigger TOP500 collection
curl -X POST http://localhost:8000/api/v1/datasources/1/trigger \
  -H "Authorization: Bearer <token>"

Core file: backend/app/api/v1/datasources.py