release: bump version to 0.49.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
linkong
@@ -87,6 +87,10 @@ async def run(self, db):
|
||||
| BarentsWatch AIS | vessel | AIS vessel positions, speed, heading, MMSI, and related fields | Collector settings |
|
||||
| AISStream Vessels | vessel_ais | AIS WebSocket realtime stream, written to the raw observation layer and displayed through aggregation | Collector settings |
|
||||
|
||||
AIS vessel collectors use a different persistence path from regular `CollectedData` collectors. BarentsWatch, AISStream, and custom `vessel_ais` sources write into the AIS raw observation layer first, then the aggregation service merges those observations into the GeoJSON and detail payloads used by the Earth vessel layer. This preserves source, transport, field conflicts, and observation time instead of letting one realtime source overwrite the final display table.
|
||||
|
||||
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
|
||||
|
||||
## IV. Data Format (stored in CollectedData table)
|
||||
|
||||
```python
|
||||
@@ -213,13 +217,136 @@ backend/app/services/collectors/
|
||||
├── epoch_ai.py # Epoch AI collector
|
||||
├── huggingface.py # HuggingFace collector
|
||||
├── peeringdb.py # PeeringDB collector
|
||||
└── telegeraphy.py # TeleGeography submarine cable collector
|
||||
├── telegeraphy.py # TeleGeography submarine cable collector
|
||||
├── vessel_ais.py # BarentsWatch AIS vessel collector
|
||||
└── aisstream.py # AISStream WebSocket vessel collector
|
||||
|
||||
backend/app/services/
|
||||
├── custom_datasource_runtime.py # Custom REST / WebSocket mapping runtime
|
||||
├── datasource_mapping.py # Deterministic field mapping and target writes
|
||||
├── vessel_ais_aggregation.py # AIS raw observation writes and aggregate reads
|
||||
├── vessel_aggregation_strategy.py # Multi-source field selection, freshness fallback, and conflict records
|
||||
└── vessel_enrichment.py # Vessel profile enrichment cache
|
||||
|
||||
backend/app/models/
|
||||
└── collected_data.py # Unified data model
|
||||
├── collected_data.py # Unified data model
|
||||
└── vessel_enrichment.py # Vessel enrichment cache
|
||||
```
|
||||
|
||||
## IX. Data Usage
|
||||
## IX. Credentialed Collectors
|
||||
|
||||
Some collectors require external service credentials:
|
||||
|
||||
| Collector | Credential provider | Credential sources |
|
||||
| --- | --- | --- |
|
||||
| `barentswatch_vessels` | `barentswatch` | Console collector settings, environment variables, `~/.zshrc` |
|
||||
| `aisstream_vessels` | `aisstream` | Console collector settings, environment variables, `~/.zshrc` for connectivity checks; save it in collector settings or inject it into the backend environment for collection |
|
||||
| `spacetrack_tle` | `spacetrack` | Environment variables, `~/.zshrc` |
|
||||
|
||||
### BarentsWatch AIS
|
||||
|
||||
BarentsWatch AIS credential resolution is centralized in:
|
||||
|
||||
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
||||
|
||||
`VesselAISCollector` only collects and transforms AIS data. It no longer reads environment variables or builds token requests directly. It uses:
|
||||
|
||||
- `resolve_barentswatch_config()`
|
||||
- `fetch_barentswatch_access_token()`
|
||||
|
||||
Resolution priority:
|
||||
|
||||
1. `DataSourceConfig.auth_config`
|
||||
2. `DataSourceConfig.config`
|
||||
3. Environment variables
|
||||
4. `~/.zshrc`
|
||||
|
||||
Supported variables:
|
||||
|
||||
```bash
|
||||
export BARENTSWATCH_CLIENT_ID="..."
|
||||
export BARENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
Historical misspellings are also supported:
|
||||
|
||||
```bash
|
||||
export BARRENTSWATCH_CLIENT_ID="..."
|
||||
export BARRENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
Connectivity validation requests `https://id.barentswatch.no/connect/token` for an access token with `scope=ais`, then requests the AIS endpoint with `Authorization: Bearer <token>`.
|
||||
|
||||
### AISStream Realtime Vessels
|
||||
|
||||
AISStream uses the `wss://stream.aisstream.io/v0/stream` WebSocket endpoint. Its default runtime is a long-lived realtime collector rather than the traditional REST pattern of one request, progress to 100%, then completion.
|
||||
|
||||
Runtime configuration:
|
||||
|
||||
- `api_key`: read first from `DataSourceConfig.auth_config.api_key` or `config.api_key`; it can also come from the backend process environment variable `AISSTREAM_API_KEY`.
|
||||
- `bounding_boxes`: AISStream subscription bounds. The default example is global `[[[-90, -180], [90, 180]]]`; demos and production runs should usually start with a smaller area.
|
||||
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
|
||||
- `streaming_enabled`: enables long-lived streaming by default; disabling it falls back to batch-style `fetch -> transform -> save`.
|
||||
- `streaming_max_messages`: test-only stop limit. Non-zero values stop the stream after the requested number of messages.
|
||||
- `reconnect_delay_seconds` and `receive_timeout_seconds`: control reconnect delay and idle receive waits.
|
||||
|
||||
State semantics:
|
||||
|
||||
- `connecting`: connecting to AISStream.
|
||||
- `streaming`: receiving realtime messages; `records_processed` means messages seen, usually without a fixed total or percentage.
|
||||
- `reconnecting`: upstream or network interruption; the collector records `AISSourceHealth` and waits before reconnecting.
|
||||
- `stopped` / `cancelled`: stopped by a test limit or user action.
|
||||
|
||||
AISStream connectivity validation reads the saved collector configuration, environment variables, and `AISSTREAM_API_KEY` in `~/.zshrc` through `datasource_connectivity.py`. For actual collection, the most reliable path is saving the API key in `Settings -> Collector Settings -> AISStream Vessels`; if the key only lives in `~/.zshrc`, confirm that the backend process inherited it.
|
||||
|
||||
### AIS Raw Observations And Aggregation
|
||||
|
||||
AIS observations do not directly replace final vessel records. They are first saved as raw observations:
|
||||
|
||||
- `source` records the origin, such as `barentswatch_vessels`, `aisstream_vessels`, or a custom source name.
|
||||
- `delivery_mode` captures realtime quality; `realtime_stream` outranks `polling`.
|
||||
- `transport` records `websocket` or `http`.
|
||||
- Dynamic fields such as position, speed, and course are selected by freshness and source priority.
|
||||
- Static fields prefer non-empty values; conflicting candidates are recorded for detail and diagnostics views.
|
||||
|
||||
Earth still reads vessel data from:
|
||||
|
||||
```http
|
||||
GET /api/v1/visualization/geo/vessels
|
||||
GET /api/v1/visualization/vessels/{mmsi}
|
||||
GET /api/v1/visualization/vessels/{mmsi}/track
|
||||
GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
||||
GET /api/v1/visualization/vessels/aggregation/diagnostics
|
||||
```
|
||||
|
||||
`/geo/vessels` merges raw observation aggregation with the legacy BarentsWatch latest-position tables so adding AISStream does not hide historical BarentsWatch-only vessels.
|
||||
|
||||
## X. Collector Settings And Connectivity Validation
|
||||
|
||||
The console "Collector Settings" page owns endpoint, headers, timeouts, retries, and credentials for all built-in collectors. Connectivity is derived by the backend checksum rather than by frontend button styling:
|
||||
|
||||
- endpoint
|
||||
- auth type
|
||||
- headers
|
||||
- config
|
||||
- credential provider
|
||||
- credential fingerprint
|
||||
|
||||
Related APIs:
|
||||
|
||||
```http
|
||||
GET /api/v1/datasources/configs/all
|
||||
POST /api/v1/datasources/configs/builtin/connection-status
|
||||
POST /api/v1/datasources/configs/builtin/connect
|
||||
POST /api/v1/settings/integrations/barentswatch/connect
|
||||
GET /api/v1/settings/credential-guides/{provider}
|
||||
POST /api/v1/settings/credential-guides/{provider}/generate
|
||||
POST /api/v1/settings/credential-guides/{provider}/reset
|
||||
```
|
||||
|
||||
See [Collector Settings and Connectivity Validation](/home/ray/dev/linkong/planet/docs/technical/en/datasource-collector-settings-connectivity.md) for the full flow.
|
||||
|
||||
## XI. Data Usage
|
||||
|
||||
Collected data ultimately:
|
||||
|
||||
@@ -227,7 +354,7 @@ Collected data ultimately:
|
||||
2. **Situational analysis** — global compute distribution statistics and growth trends
|
||||
3. **Alert system** — detects changes to important nodes
|
||||
|
||||
## X. Collector Registration
|
||||
## XII. Collector Registration
|
||||
|
||||
Collectors are automatically registered at application startup:
|
||||
|
||||
@@ -249,7 +376,7 @@ collector_registry.register(TeleGeographyCableSystemCollector())
|
||||
|
||||
**Core file**: `backend/app/services/collectors/registry.py`
|
||||
|
||||
## XI. Triggering Collection
|
||||
## XIII. Triggering Collection
|
||||
|
||||
### Method 1: Scheduled
|
||||
|
||||
|
||||
Reference in New Issue
Block a user