93eb41a9f7
Release 0.58.0 includes the Earth high-precision boundary PMTiles/MVT pipeline, standardized Earth boundary source collectors, China POV boundary configuration templates, and removal of the legacy low-precision GeoJSON fallback. It also adds Earth news target-location queueing/archive support, fixes datasource task status visibility, documents the Earth surface depth-spacing rules that prevent far-zoom z-fighting snow/black blocks, and updates bilingual operations/developer docs.
455 lines
23 KiB
Markdown
455 lines
23 KiB
Markdown
# Data Collectors
|
|
|
|
## I. System Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Data Collection Architecture │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ TOP500 │ │ Epoch AI │ │ HuggingFace │ │
|
|
│ │ Collector │ │ Collector │ │ Collector │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
|
|
│ │ │ │ │
|
|
│ └───────────────────┼───────────────────┘ │
|
|
│ ▼ │
|
|
│ ┌─────────────────────┐ │
|
|
│ │ BaseCollector │◄── Base class (unified) │
|
|
│ │ run() method │ │
|
|
│ └─────────┬───────────┘ │
|
|
│ │ │
|
|
│ ┌─────────────────┼─────────────────┐ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
|
|
│ │ fetch() │ │transform()│ │ _save_data│ │
|
|
│ │ raw data │ │ transform │ │ save to DB│ │
|
|
│ └───────────┘ └───────────┘ └───────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌─────────────────────┐ │
|
|
│ │ CollectedData table│◄── Unified storage │
|
|
│ └─────────────────────┘ │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
|
│ │ Scheduler (APScheduler) │ │
|
|
│ │ Scheduled tasks: every 4h/6h/12h/1d auto-execute │ │
|
|
│ └─────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## II. Pipeline
|
|
|
|
```python
|
|
# 1. Scheduler triggers (scheduled or manual)
|
|
# ↓
|
|
|
|
# 2. run() executes the full pipeline
|
|
async def run(self, db):
|
|
# 2.1 Check if collector is enabled
|
|
if not collector_registry.is_active(self.name):
|
|
return {"status": "skipped"}
|
|
|
|
# 2.2 Record task start
|
|
task = CollectionTask(status="running")
|
|
db.add(task)
|
|
await db.commit()
|
|
|
|
# 2.3 FETCH — get raw data (implemented by subclass)
|
|
raw_data = await self.fetch()
|
|
|
|
# 2.4 TRANSFORM — convert to unified format
|
|
data = self.transform(raw_data)
|
|
|
|
# 2.5 SAVE — persist to database
|
|
records_count = await self._save_data(db, data)
|
|
|
|
# 2.6 Record task completion
|
|
task.status = "success"
|
|
task.records_processed = records_count
|
|
await db.commit()
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/base.py`
|
|
|
|
## III. Collector List
|
|
|
|
| Collector | Data type | Content | Frequency |
|
|
|-----------|-----------|---------|-----------|
|
|
| TOP500 | supercomputer | Global supercomputer rankings (compute, performance) | 4 hours |
|
|
| Epoch AI | gpu_cluster | GPU compute cluster info | 6 hours |
|
|
| HuggingFace Models | model | AI model information | 12 hours |
|
|
| HuggingFace Datasets | dataset | Dataset information | 12 hours |
|
|
| HuggingFace Spaces | space | Demo applications | 1 day |
|
|
| PeeringDB | ixp/network/facility | Internet exchange points / networks / facilities | 1-2 days |
|
|
| TeleGeography | submarine_cable | Submarine cable information | 7 days |
|
|
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
|
|
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
|
|
| Earth Admin-0 Boundaries | earth_admin0_boundaries | Downloads the configured country-boundary source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
|
|
| Earth Coastline | earth_coastline | Downloads the configured coastline source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
|
|
| Earth Claim Lines | earth_claim_lines | Downloads the configured claim-line source, saves an artifact, and writes an `earth_boundary_source` manifest record | Collector settings |
|
|
| Earth PMTiles Builder | earth_boundary_tiles | Reads the three Earth boundary source records and builds / registers the PMTiles artifact | Collector settings |
|
|
|
|
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.
|
|
|
|
Earth boundaries are now split into three real source collectors plus one downstream builder. `earth_admin0_boundaries`, `earth_coastline`, and `earth_claim_lines` read endpoint, headers, auth, and `config.target_schema=earth_boundary_source` from Collector Settings. Triggering them requests the configured endpoint, writes the full response to `data/earth-boundary-sources/<collector>/<sha256>.*`, and stores sha256, feature count, license, artifact path, sample properties, and mapping metadata in `CollectedData`.
|
|
|
|
`earth_boundary_tiles` no longer means source-data collection. It reads the latest successful records from those three source collectors; if any source is missing, the task fails as "not ready" and does not register "4 high-precision tile" records. Once all sources exist, it uses `tippecanoe` / `pmtiles` to build `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`; if those tools are missing, the task fails with the missing-tool message. There is no legacy low-precision fallback for country boundaries.
|
|
|
|
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
|
|
|
## IV. Data Format (stored in CollectedData table)
|
|
|
|
```python
|
|
# Each collector's parse_response() return format
|
|
{
|
|
"source_id": "top500_1", # Original system ID (required)
|
|
"name": "El Capitan", # Name (required)
|
|
"description": "System desc...", # Description
|
|
"country": "United States", # Country
|
|
"city": "Livermore, CA", # City
|
|
"latitude": "37.6819", # Latitude (string)
|
|
"longitude": "-121.7681", # Longitude (string)
|
|
"value": "1742.00", # Performance value (e.g. compute)
|
|
"unit": "PFlop/s", # Unit
|
|
"metadata": { # Extra data (JSON)
|
|
"rank": 1,
|
|
"r_peak": 2746.38,
|
|
"cores": 11039616
|
|
},
|
|
"reference_date": "2025-11-01" # Data reference date
|
|
}
|
|
```
|
|
|
|
## V. Database Schema
|
|
|
|
**CollectedData table** (`collected_data`)
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| id | SERIAL | Primary key |
|
|
| source | VARCHAR(100) | Data source name (top500, huggingface, etc.) |
|
|
| source_id | VARCHAR(100) | Original data ID |
|
|
| data_type | VARCHAR(50) | Data type (supercomputer, model, etc.) |
|
|
| name | VARCHAR(500) | Name |
|
|
| title | VARCHAR(500) | Title |
|
|
| description | TEXT | Description |
|
|
| country | VARCHAR(100) | Country |
|
|
| city | VARCHAR(100) | City |
|
|
| latitude | VARCHAR(50) | Latitude |
|
|
| longitude | VARCHAR(50) | Longitude |
|
|
| value | VARCHAR(100) | Performance value |
|
|
| unit | VARCHAR(20) | Unit |
|
|
| metadata | JSONB | Extra metadata |
|
|
| collected_at | TIMESTAMP | Collection time |
|
|
| reference_date | TIMESTAMP | Data reference date |
|
|
| is_valid | INTEGER | Whether valid |
|
|
|
|
**Core file**: `backend/app/models/collected_data.py`
|
|
|
|
## VI. TOP500 Collector Example (full pipeline)
|
|
|
|
```python
|
|
# 1. fetch() — get HTML from the web
|
|
async def fetch(self):
|
|
url = "https://top500.org/lists/top500/list/2025/11/"
|
|
response = await client.get(url)
|
|
return response.text # returns HTML
|
|
|
|
# 2. parse_response() — parse HTML into unified format
|
|
def parse_response(self, html):
|
|
soup = BeautifulSoup(html, "html.parser")
|
|
table = soup.find("table")
|
|
|
|
for row in table.find_all("tr")[1:]: # skip header
|
|
cells = row.find_all("td")
|
|
|
|
entry = {
|
|
"source_id": f"top500_{cells[0].text}",
|
|
"name": cells[1].text.strip(),
|
|
"country": cells[2].text.strip(),
|
|
"city": "",
|
|
"latitude": "",
|
|
"longitude": "",
|
|
"value": "1742.00",
|
|
"unit": "PFlop/s",
|
|
"metadata": {
|
|
"rank": 1,
|
|
"cores": "11340000"
|
|
},
|
|
"reference_date": "2025-11-01"
|
|
}
|
|
data.append(entry)
|
|
|
|
return data
|
|
|
|
# 3. run() automatically calls _save_data() to save to database
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/top500.py`
|
|
|
|
## VII. Scheduler
|
|
|
|
```python
|
|
# Register all collectors into scheduled tasks at startup
|
|
def start_scheduler():
|
|
for name, collector in collectors.items():
|
|
if collector_registry.is_active(name):
|
|
scheduler.add_job(
|
|
run_collector_task,
|
|
trigger=IntervalTrigger(hours=collector.frequency_hours),
|
|
id=name,
|
|
name=name
|
|
)
|
|
```
|
|
|
|
| Collector | Frequency |
|
|
|-----------|-----------|
|
|
| TOP500 | Every 4 hours |
|
|
| Epoch AI | Every 6 hours |
|
|
| HuggingFace | Every 12 hours |
|
|
| PeeringDB | Every 1-2 days |
|
|
| TeleGeography | Every 7 days |
|
|
|
|
**Core file**: `backend/app/services/scheduler.py`
|
|
|
|
## VIII. Code Files
|
|
|
|
```
|
|
backend/app/services/collectors/
|
|
├── base.py # Base class: run() pipeline, _save_data() persistence
|
|
├── registry.py # Collector registry
|
|
├── scheduler.py # Scheduled task dispatch (APScheduler)
|
|
├── top500.py # TOP500 collector
|
|
├── epoch_ai.py # Epoch AI collector
|
|
├── huggingface.py # HuggingFace collector
|
|
├── peeringdb.py # PeeringDB collector
|
|
├── telegeraphy.py # TeleGeography submarine cable collector
|
|
├── vessel_ais.py # BarentsWatch AIS vessel collector
|
|
├── aisstream.py # AISStream WebSocket vessel collector
|
|
└── earth_boundaries.py # Earth boundary source verification and static tile artifact collector
|
|
|
|
backend/app/services/
|
|
├── custom_datasource_runtime.py # Custom REST / WebSocket mapping runtime
|
|
├── datasource_mapping.py # Deterministic field mapping and target writes
|
|
├── vessel_ais_aggregation.py # AIS raw observation writes and aggregate reads
|
|
├── vessel_aggregation_strategy.py # Multi-source field selection, freshness fallback, and conflict records
|
|
└── vessel_enrichment.py # Vessel profile enrichment cache
|
|
|
|
backend/app/models/
|
|
├── collected_data.py # Unified data model
|
|
└── vessel_enrichment.py # Vessel enrichment cache
|
|
```
|
|
|
|
## IX. Credentialed Collectors
|
|
|
|
Some collectors require external service credentials:
|
|
|
|
| Collector | Credential provider | Credential sources |
|
|
| --- | --- | --- |
|
|
| `barentswatch_vessels` | `barentswatch` | Console collector settings, environment variables, `~/.zshrc` |
|
|
| `aisstream_vessels` | `aisstream` | Console collector settings, environment variables, `~/.zshrc` for connectivity checks; save it in collector settings or inject it into the backend environment for collection |
|
|
| `spacetrack_tle` | `spacetrack` | Environment variables, `~/.zshrc` |
|
|
|
|
### BarentsWatch AIS
|
|
|
|
BarentsWatch AIS credential resolution is centralized in:
|
|
|
|
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
|
|
|
`VesselAISCollector` only collects and transforms AIS data. It no longer reads environment variables or builds token requests directly. It uses:
|
|
|
|
- `resolve_barentswatch_config()`
|
|
- `fetch_barentswatch_access_token()`
|
|
|
|
Resolution priority:
|
|
|
|
1. `DataSourceConfig.auth_config`
|
|
2. `DataSourceConfig.config`
|
|
3. Environment variables
|
|
4. `~/.zshrc`
|
|
|
|
Supported variables:
|
|
|
|
```bash
|
|
export BARENTSWATCH_CLIENT_ID="..."
|
|
export BARENTSWATCH_CLIENT_SECRET="..."
|
|
```
|
|
|
|
Historical misspellings are also supported:
|
|
|
|
```bash
|
|
export BARRENTSWATCH_CLIENT_ID="..."
|
|
export BARRENTSWATCH_CLIENT_SECRET="..."
|
|
```
|
|
|
|
Connectivity validation requests `https://id.barentswatch.no/connect/token` for an access token with `scope=ais`, then requests the AIS endpoint with `Authorization: Bearer <token>`.
|
|
|
|
### AISStream Realtime Vessels
|
|
|
|
AISStream uses the `wss://stream.aisstream.io/v0/stream` WebSocket endpoint. Its default runtime is a long-lived realtime collector rather than the traditional REST pattern of one request, progress to 100%, then completion.
|
|
|
|
Runtime configuration:
|
|
|
|
- `api_key`: read first from `DataSourceConfig.auth_config.api_key` or `config.api_key`; it can also come from the backend process environment variable `AISSTREAM_API_KEY`.
|
|
- `bounding_boxes`: AISStream subscription bounds. The default example is global `[[[-90, -180], [90, 180]]]`; demos and production runs should usually start with a smaller area.
|
|
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
|
|
- `streaming_enabled`: enables long-lived streaming by default; disabling it falls back to batch-style `fetch -> transform -> save`.
|
|
- `streaming_max_messages`: test-only stop limit. Non-zero values stop the stream after the requested number of messages.
|
|
- `reconnect_delay_seconds` and `receive_timeout_seconds`: control reconnect delay and idle receive waits.
|
|
|
|
State semantics:
|
|
|
|
- `connecting`: connecting to AISStream.
|
|
- `streaming`: receiving realtime messages; `records_processed` means messages seen, usually without a fixed total or percentage.
|
|
- `reconnecting`: upstream or network interruption; the collector records `AISSourceHealth` and waits before reconnecting.
|
|
- `stopped` / `cancelled`: stopped by a test limit or user action.
|
|
|
|
AISStream connectivity validation reads the saved collector configuration, environment variables, and `AISSTREAM_API_KEY` in `~/.zshrc` through `datasource_connectivity.py`. For actual collection, the most reliable path is saving the API key in `Settings -> Collector Settings -> AISStream Vessels`; if the key only lives in `~/.zshrc`, confirm that the backend process inherited it.
|
|
|
|
The console manages AISStream from `/datasources -> Realtime Streams`, not from the normal finite collection progress bar. The realtime stream API aggregates runtime state, health, configuration preview, and raw observation counters:
|
|
|
|
```http
|
|
GET /api/v1/realtime-sources
|
|
POST /api/v1/realtime-sources/{source}/start
|
|
POST /api/v1/realtime-sources/{source}/stop
|
|
POST /api/v1/realtime-sources/{source}/restart
|
|
```
|
|
|
|
`aisstream_vessels` and custom `source_type=websocket` sources appear in that API. They do not participate in one-click collection percentages; the UI interprets them as long-lived services with message counters, lag, last success, and last error.
|
|
|
|
### AIS Raw Observations And Aggregation
|
|
|
|
AIS observations do not directly replace final vessel records. They are first saved as raw observations:
|
|
|
|
- `source` records the origin, such as `barentswatch_vessels`, `aisstream_vessels`, or a custom source name.
|
|
- `delivery_mode` captures realtime quality; `realtime_stream` outranks `polling`.
|
|
- `transport` records `websocket` or `http`.
|
|
- Dynamic fields such as position, speed, and course are selected by freshness and source priority.
|
|
- Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.
|
|
|
|
Earth vessel rendering now consumes the bounded snapshot endpoint and realtime delta channel:
|
|
|
|
```http
|
|
GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/visualization/vessels/{mmsi}
|
|
GET /api/v1/visualization/vessels/{mmsi}/track
|
|
GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
|
```
|
|
|
|
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, defaults to `limit=1000`, and caps `limit` at `5000`. It prefers aggregated `ais_raw_observations`; when the current raw window is empty, it can fall back to the latest legacy `vessel_position` / `vessel_static` rows and marks that path with `diagnostics.legacy_fallback_used`. The old `/api/v1/visualization/geo/vessels` route has been removed.
|
|
|
|
Realtime deltas are sent through the `/ws` `vessels` channel. Clients must subscribe with the current viewport:
|
|
|
|
```json
|
|
{
|
|
"type": "subscribe",
|
|
"data": {
|
|
"channel": "vessels",
|
|
"bbox": [120.8, 30.7, 122.1, 31.8],
|
|
"zoom": 12,
|
|
"limit": 1000
|
|
}
|
|
}
|
|
```
|
|
|
|
The backend stores lightweight subscription filters per connection and only sends vessel updates that match the subscriber bbox. Collector broadcasts enter a 1-second throttle queue; within each flush window, only the latest update per MMSI is retained.
|
|
|
|
### Layer APIs And Global Stats
|
|
|
|
Earth is moving to two API families:
|
|
|
|
```http
|
|
GET /api/v1/data-products
|
|
GET /api/v1/data-products/{product_id}/status
|
|
GET /api/v1/layers/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/cables?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/landing-points?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/satellites?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/anomalies?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/incidents?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
GET /api/v1/layers/bgp/collectors?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000
|
|
```
|
|
|
|
`/api/v1/data-products/*` is for aggregate panels and keeps a global statistics scope independent of the map bbox. `/api/v1/layers/*` is for map rendering, requires `bbox` and `zoom`, defaults to `limit=1000`, and caps `limit` at `5000`; low zoom falls back to a smaller response cap and reports `degraded`, `truncated`, `limit_clamped`, and `stats_scope=viewport` in `diagnostics`. Non-vessel layers currently reuse the existing GeoJSON converters before the guard layer; future product-specific queries can push bbox filtering deeper.
|
|
|
|
## X. Collector Settings And Connectivity Validation
|
|
|
|
The console "Collector Settings" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:
|
|
|
|
- endpoint
|
|
- auth type
|
|
- headers
|
|
- config
|
|
- credential provider
|
|
- credential fingerprint
|
|
|
|
Related APIs:
|
|
|
|
```http
|
|
GET /api/v1/datasources/configs/all
|
|
POST /api/v1/datasources/configs/builtin/connection-status
|
|
POST /api/v1/datasources/configs/builtin/connect
|
|
POST /api/v1/settings/integrations/barentswatch/connect
|
|
GET /api/v1/settings/credential-guides/{provider}
|
|
POST /api/v1/settings/credential-guides/{provider}/generate
|
|
POST /api/v1/settings/credential-guides/{provider}/reset
|
|
```
|
|
|
|
See [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md) for the full flow.
|
|
|
|
## XI. Data Usage
|
|
|
|
Collected data ultimately:
|
|
|
|
1. **Visualization** — displays supercomputers, GPU clusters, and submarine cables' geographic positions
|
|
2. **Situational analysis** — global compute distribution statistics and growth trends
|
|
3. **Alert system** — detects changes to important nodes
|
|
|
|
## XII. Collector Registration
|
|
|
|
Collectors are automatically registered at application startup:
|
|
|
|
```python
|
|
# backend/app/services/collectors/__init__.py
|
|
|
|
collector_registry.register(TOP500Collector())
|
|
collector_registry.register(EpochAIGPUCollector())
|
|
collector_registry.register(HuggingFaceModelCollector())
|
|
collector_registry.register(HuggingFaceDatasetCollector())
|
|
collector_registry.register(HuggingFaceSpacesCollector())
|
|
collector_registry.register(PeeringDBIXPCollector())
|
|
collector_registry.register(PeeringDBNetworkCollector())
|
|
collector_registry.register(PeeringDBFacilityCollector())
|
|
collector_registry.register(TeleGeographyCableCollector())
|
|
collector_registry.register(TeleGeographyLandingPointCollector())
|
|
collector_registry.register(TeleGeographyCableSystemCollector())
|
|
```
|
|
|
|
**Core file**: `backend/app/services/collectors/registry.py`
|
|
|
|
## XIII. Triggering Collection
|
|
|
|
### Method 1: Scheduled
|
|
|
|
At startup, APScheduler automatically creates scheduled tasks based on each collector's `frequency_hours` setting.
|
|
|
|
### Method 2: Manual API trigger
|
|
|
|
```bash
|
|
# Trigger TOP500 collection
|
|
curl -X POST http://localhost:8000/api/v1/datasources/1/trigger \
|
|
-H "Authorization: Bearer <token>"
|
|
```
|
|
|
|
Batch collection uses:
|
|
|
|
```http
|
|
POST /api/v1/datasources/trigger-batch
|
|
```
|
|
|
|
The request body may pass `source_ids` for selected rows. Without `source_ids`, the backend filters by `product`, `module`, `is_active`, `run_status`, `collected`, `credential_status`, and `q`. The endpoint skips disabled sources, sources already running without `force`, and sources still inside their frequency window, then returns `triggered`, `skipped`, and `failed` groups.
|
|
|
|
**Core file**: `backend/app/api/v1/datasources.py`
|