release: bump version to 0.49.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -62,7 +62,7 @@ File:
|
||||
Current behavior:
|
||||
|
||||
- The `collector_credentials` tab is displayed as "Collector Settings".
|
||||
- A select lists all built-in collectors.
|
||||
- A select lists built-in collectors and supports maintaining custom supplemental sources that merge into built-in data.
|
||||
- The only button beside the select is a plug icon for health checks.
|
||||
- Status tags below the select show:
|
||||
- `Credentials required` / `No credentials required`
|
||||
@@ -72,6 +72,8 @@ Current behavior:
|
||||
- Whether the endpoint is overridden
|
||||
- Collectors that require credentials place the credential card above base configuration.
|
||||
- Collectors without credentials only show base configuration.
|
||||
- The AISStream collector uses WebSocket semantics: connecting, streaming, reconnecting, or stopped. It does not use a fixed completion percentage.
|
||||
- Custom source editing lives in collector settings. The data source catalog keeps overview, run controls, and read-only drawers.
|
||||
|
||||
The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.
|
||||
|
||||
@@ -284,6 +286,100 @@ Normalization:
|
||||
- Vessel type usually comes from lower-frequency `ShipStaticData.Type`; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
|
||||
- If a vessel has not yet produced a static message, its aggregated type can still be `Other`; v5 vessel profile enrichment is planned to fill that gap.
|
||||
|
||||
Connectivity validation reads saved configuration, environment variables, and `AISSTREAM_API_KEY` from `~/.zshrc`. For actual collection, prefer saving the API key in collector settings. If the key only lives in `~/.zshrc`, confirm that the backend process inherited it; otherwise validation may pass while the collector runtime cannot read the key.
|
||||
|
||||
## Custom REST / WebSocket Mapping Runtime
|
||||
|
||||
Files:
|
||||
|
||||
- [custom_datasource_runtime.py](/home/ray/dev/linkong/planet/backend/app/services/custom_datasource_runtime.py)
|
||||
- [datasource_mapping.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_mapping.py)
|
||||
|
||||
Custom sources are supplemental inputs for existing target schemas, not isolated data islands. The most complete target today is `vessel_ais`: a custom REST or WebSocket source is mapped deterministically, written into AIS raw observations, and then pushed to Earth through the `vessels` WebSocket channel.
|
||||
|
||||
### Configuration Semantics
|
||||
|
||||
Important fields:
|
||||
|
||||
- `source_type`: `rest` / `http` / `websocket` / `ws`.
|
||||
- `endpoint`: REST uses `http(s)://`; WebSocket uses `ws(s)://`.
|
||||
- `auth_type`: `none`, `bearer`, `api_key`, or `basic`.
|
||||
- `headers`: static request headers.
|
||||
- `auth_config`: token, API key, or basic username/password; API keys can be sent by header or query.
|
||||
- `config.target_schema`: for example `vessel_ais`.
|
||||
- `config.delivery_mode`: REST defaults to `polling`; WebSocket defaults to `realtime_stream`.
|
||||
- `config.merge_target_source`: records which built-in source this custom source supplements, such as `barentswatch_vessels`.
|
||||
|
||||
The REST runner supports:
|
||||
|
||||
- `GET` / `POST`
|
||||
- query params
|
||||
- JSON body
|
||||
- headers and auth injection
|
||||
- active mapping writes into the target schema
|
||||
|
||||
The WebSocket runner supports:
|
||||
|
||||
- endpoint format validation
|
||||
- headers and auth injection
|
||||
- optional `ws_subscribe_message`
|
||||
- `ws_message_path` / `ws_items_path` extraction
|
||||
- reconnects
|
||||
- `debug_max_messages` debug limits
|
||||
- background stream start / stop / status
|
||||
|
||||
Related APIs:
|
||||
|
||||
```http
|
||||
POST /api/v1/datasources/custom/sample
|
||||
GET /api/v1/datasources/target-schemas
|
||||
POST /api/v1/datasources/{config_id}/run-mapped
|
||||
POST /api/v1/datasources/{config_id}/stop-mapped
|
||||
GET /api/v1/datasources/{config_id}/mapped-status
|
||||
DELETE /api/v1/datasources/configs/{config_id}?delete_mappings=true&delete_source_data=true
|
||||
```
|
||||
|
||||
`run-mapped?background=true` only matters for WebSocket sources and starts a background stream. REST sources remain one-shot collection runs.
|
||||
|
||||
### Delete And Data Cleanup
|
||||
|
||||
Deleting a custom source has three levels:
|
||||
|
||||
- Delete configuration only: preserve mapping and historical data.
|
||||
- Delete configuration and mapping: also delete mapping templates for that config.
|
||||
- Delete configuration, mapping, and source data: delete that source's `collected_data`, `ais_raw_observations`, and `ais_source_health`.
|
||||
|
||||
When deleted `vessel_ais` source data affects Earth, the backend broadcasts `reload_required` on the `vessels` channel so Earth reloads aggregated vessels. Legacy `vessel_position` rows are not deleted by custom source because that table cannot safely attribute rows back to a custom source.
|
||||
|
||||
### Local AIS Mock WebSocket
|
||||
|
||||
File:
|
||||
|
||||
- [mock-ais-ws-server.ts](/home/ray/dev/linkong/planet/scripts/mock-ais-ws-server.ts)
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bun run mock:ais-ws
|
||||
```
|
||||
|
||||
The mock service continuously sends AIS-like JSON to validate the chain: WebSocket custom source -> mapping -> AIS raw observation -> `vessels` channel -> Earth vessel upsert. Typical config:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_type": "websocket",
|
||||
"endpoint": "ws://localhost:8787",
|
||||
"config": {
|
||||
"target_schema": "vessel_ais",
|
||||
"delivery_mode": "realtime_stream",
|
||||
"merge_target_source": "barentswatch_vessels",
|
||||
"ws_message_path": "$.data",
|
||||
"ws_items_path": "$.vessels[*]",
|
||||
"ws_reconnect": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Credential Guide
|
||||
|
||||
File:
|
||||
@@ -345,6 +441,7 @@ Added coverage:
|
||||
Credential providers currently supported:
|
||||
|
||||
- `barentswatch`
|
||||
- `aisstream`
|
||||
- `spacetrack`
|
||||
|
||||
Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.
|
||||
|
||||
Reference in New Issue
Block a user