planet/docs/technical/en/datasource-collector-settings-connectivity.md

Collector Settings and Connectivity Validation

Background

The console now separates the "data source catalog" from "collector configuration":

• /datasources
  • Lists all data sources, including built-in and custom sources.
  • Clicking a name only opens an information drawer.
  • Focuses on status, manual collection, and running collection tasks.
• /collection-management?tab=collector_credentials
  • Displays as "Collectors".
  • Owns endpoint, headers, base parameters, and credentials.
  • Every collector exposes a connection button for health checks.

This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to Collectors instead of being scattered across the data source list and system settings.

User-Facing Rules

Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.

A built-in collector is considered "connected" when either condition is true:

• The current configuration has successfully collected data.
• The user clicked the connection button for the current configuration and backend validation succeeded.

If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".

Frontend Entry Points

Data Source Catalog

Files:

• DataSources.tsx
• index.css

Current behavior:

• Built-in and custom data sources are merged into a UnifiedDataSource list.
• The table only keeps view, collect, and status actions.
• Clicking the name opens a read-only drawer.
• The drawer shows:
  • Whether the source is built in
  • Whether it is enabled
  • Module, priority, and frequency
  • Endpoint
  • Headers
  • Base configuration
  • Whether credentials are required
• When tasks are running, the top progress area shows a clickable Collecting N pill.
• Clicking Collecting N opens a task list modal with per-task progress.

data-source-bulk-toolbar__running-pill is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.

Collectors

File:

• Settings.tsx

Current behavior:

• The collector_credentials tab is displayed as "Collectors" under /collection-management.
• A select lists built-in collectors and supports maintaining custom supplemental sources that merge into built-in data.
• The only button beside the select is a plug icon for health checks.
• Status tags below the select show:
  • Credentials required / No credentials required
  • Module
  • Enabled / Disabled
  • Unchecked / Available / Unavailable
  • Whether the endpoint is overridden
• Collectors that require credentials place the credential card above base configuration.
• Collectors without credentials only show base configuration.
• The AISStream collector uses WebSocket semantics: connecting, streaming, reconnecting, or stopped. It does not use a fixed completion percentage.
• Custom source editing lives in collector settings. The data source catalog keeps overview, run controls, and read-only drawers.

The connection button uses an inline Tabler-style plug icon with plug-connected semantics, avoiding the older refresh icon for a connection action.

Backend APIs

Data Source Configuration List

GET /api/v1/datasources/configs/all

Returns a merged view of YAML default data sources and database overrides. This route must be declared before /configs/{config_id}; otherwise FastAPI treats all as a path parameter and returns 422.

Returned fields include:

• name
• default_url
• endpoint
• is_overridden
• is_active
• source_type
• auth_type
• headers
• config
• config_id
• description

Before returning config, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.

Built-In Collector Connection Status

POST /api/v1/datasources/configs/builtin/connection-status

Purpose:

• Accept a candidate configuration.
• Compute its checksum.
• Determine whether the current configuration is already connected.

The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.

Built-In Collector Connectivity Validation

POST /api/v1/datasources/configs/builtin/connect

Purpose:

• Free collectors request the endpoint directly.
• Credentialed collectors go through their credential provider.
• Successful validation writes a system-level connection record.

Successful responses include:

• success
• connected
• checksum
• stage
• message
• response_time_ms
• credential_provider
• credential_source

BarentsWatch AIS Connectivity Validation

POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/integrations/barentswatch/connectivity

BarentsWatch uses separate endpoints because draft credentials must be validated before saving:

• Use draft client_id / client_secret to fetch a token.
• Use that token to request the AIS endpoint.
• After success, write a built-in collector connection record using the draft credential fingerprint.

Connectivity Validation Service

File:

• datasource_connectivity.py

Core responsibilities:

• Compute built-in collector configuration checksums.
• Read credentials from environment variables and ~/.zshrc.
• Determine whether the current configuration is already connected.
• Run endpoint health checks.
• Save successful connection records.

Checksum Inputs

The checksum includes:

• Collector name
• Endpoint
• Auth type
• Headers
• Config after removing internal validation fields
• Credential provider
• Credential fingerprint

The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.

Connection Records

Successful connection records are written to SystemSetting:

category = datasource_connectivity_validations

The payload uses collector source as the key:

{
  "barentswatch_vessels": {
    "checksum": "...",
    "status": "success",
    "validated_at": "2026-04-29T00:00:00+00:00",
    "status_code": 200,
    "credential_source": "datasource_config",
    "connected_by": "connection_button"
  }
}

connected_by currently has two sources:

• connection_button: the user manually clicked the connection button.
• collection: a collection task completed successfully, so the system recorded the current effective configuration as connected.

Successful Collection Means Connected

After a successful collection, the scheduler writes a connection record:

• scheduler.py

This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.

BarentsWatch AIS Credential Chain

Files:

• barentswatch.py
• vessel_ais.py

Resolution priority:

1. DataSourceConfig.auth_config
2. DataSourceConfig.config
3. Environment variables
4. ~/.zshrc

Supported environment variables:

export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."

Historical misspellings are also supported:

export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."

Token request rules:

• Token URL: https://id.barentswatch.no/connect/token
• Content-Type: application/x-www-form-urlencoded
• Body:
  • grant_type=client_credentials
  • client_id
  • client_secret
  • scope=ais

AIS request rules:

• Default endpoint: https://live.ais.barentswatch.no/v1/latest/combined
• Header: Authorization: Bearer <access_token>

VesselAISCollector no longer reads environment variables directly. It goes through resolve_barentswatch_config() and fetch_barentswatch_access_token() so settings, connectivity validation, and collection do not fork into three credential flows.

AISStream Collector Chain

Files:

• aisstream.py
• vessel_ais_aggregation.py

AISStream uses a WebSocket realtime stream. The collector writes only to the ais_raw_observations raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.

Configuration:

• api_key: stored in DataSourceConfig.auth_config, or provided through AISSTREAM_API_KEY.
• endpoint: defaults to wss://stream.aisstream.io/v0/stream.
• message_types: defaults to PositionReport and ShipStaticData.
• bounding_boxes: AISStream format is [[[lat_min, lon_min], [lat_max, lon_max]]]; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
• max_messages and receive_timeout_seconds: control the batch-style WebSocket collection window.

Normalization:

• PositionReport mainly provides position, speed, course, heading, and navigation status.
• Vessel names can be filled from MetaData.ShipName even when the message body has no name.
• Vessel type usually comes from lower-frequency ShipStaticData.Type; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
• If a vessel has not yet produced a static message, its aggregated type can still be Other; v5 vessel profile enrichment is planned to fill that gap.

Connectivity validation reads saved configuration, environment variables, and AISSTREAM_API_KEY from ~/.zshrc. For actual collection, prefer saving the API key in collector settings. If the key only lives in ~/.zshrc, confirm that the backend process inherited it; otherwise validation may pass while the collector runtime cannot read the key.

Connectivity validation and actual collection are separate actions. A banner such as AISStream credentials configured, WebSocket endpoint format valid only means the saved settings can be used for a connection attempt; runtime status may still be disconnected. Global AIS data is written locally only while aisstream_vessels is streaming / connected and its realtime stream counters plus last_seen_at keep advancing. Start, stop, reconnect, health, and counters are exposed through /datasources -> Realtime Streams and /api/v1/realtime-sources; AISStream is not counted in normal one-click collection progress.

The new vessel list entry point is no longer the legacy /api/v1/visualization/geo/vessels route. Earth initial state should call:

GET /api/v1/vessels/snapshot?bbox=lon_min,lat_min,lon_max,lat_max&zoom=12&limit=1000

That endpoint prefers local aggregated ais_raw_observations; when the current raw window is empty, it can fall back to the latest legacy vessel_position / vessel_static rows and exposes that through diagnostics.legacy_fallback_used. Realtime updates use the /ws vessels channel; subscriptions must include bbox, zoom, and limit. The server filters updates per connection and merges collector broadcasts every second, keeping only the latest position per MMSI.

Custom REST / WebSocket Mapping Runtime

Files:

• custom_datasource_runtime.py
• datasource_mapping.py

Custom sources are supplemental inputs for existing target schemas, not isolated data islands. The most complete target today is vessel_ais: a custom REST or WebSocket source is mapped deterministically, written into AIS raw observations, and then pushed to Earth through the vessels WebSocket channel.

Earth high-precision boundaries no longer use custom-source target schemas. Boundaries are Earth static assets: the Earth Assets settings panel saves local source configuration and triggers PMTiles builds without writing records to CollectedData.

Configuration Semantics

Important fields:

• source_type: rest / http / websocket / ws.
• endpoint: REST uses http(s)://; WebSocket uses ws(s)://.
• auth_type: none, bearer, api_key, or basic.
• headers: static request headers.
• auth_config: token, API key, or basic username/password; API keys can be sent by header or query.
• config.target_schema: for example vessel_ais, geo_points, or generic_records.
• config.delivery_mode: REST defaults to polling; WebSocket defaults to realtime_stream.
• config.merge_target_source: records which built-in source this custom source supplements, such as barentswatch_vessels.

The REST runner supports:

• GET / POST
• query params
• JSON body
• headers and auth injection
• active mapping writes into the target schema

The WebSocket runner supports:

• endpoint format validation
• headers and auth injection
• optional ws_subscribe_message
• ws_message_path / ws_items_path extraction
• reconnects
• debug_max_messages debug limits
• background stream start / stop / status

Related APIs:

POST /api/v1/datasources/custom/sample
GET  /api/v1/datasources/target-schemas
POST /api/v1/datasources/{config_id}/run-mapped
POST /api/v1/datasources/{config_id}/stop-mapped
GET  /api/v1/datasources/{config_id}/mapped-status
DELETE /api/v1/datasources/configs/{config_id}?delete_mappings=true&delete_source_data=true

run-mapped?background=true only matters for WebSocket sources and starts a background stream. REST sources remain one-shot collection runs.

Delete And Data Cleanup

Deleting a custom source has three levels:

• Delete configuration only: preserve mapping and historical data.
• Delete configuration and mapping: also delete mapping templates for that config.
• Delete configuration, mapping, and source data: delete that source's collected_data, ais_raw_observations, and ais_source_health.

When deleted vessel_ais source data affects Earth, the backend broadcasts reload_required on the vessels channel so Earth reloads aggregated vessels. Legacy vessel_position rows are not deleted by custom source because that table cannot safely attribute rows back to a custom source.

Local AIS Mock WebSocket

File:

• mock-ais-ws-server.ts

Run:

bun run mock:ais-ws

The mock service continuously sends AIS-like JSON to validate the chain: WebSocket custom source -> mapping -> AIS raw observation -> vessels channel -> Earth vessel upsert. Typical config:

{
  "source_type": "websocket",
  "endpoint": "ws://localhost:8787",
  "config": {
    "target_schema": "vessel_ais",
    "delivery_mode": "realtime_stream",
    "merge_target_source": "barentswatch_vessels",
    "ws_message_path": "$.data",
    "ws_items_path": "$.vessels[*]",
    "ws_reconnect": true
  }
}

Credential Guide

File:

• credential_guides.py

APIs:

GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset

Currently supported:

• barentswatch
• aisstream

The default guide includes the official BarentsWatch tutorial:

https://developer.barentswatch.no/docs/tutorial

If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to SystemSetting:

category = collector_credential_guides

Reset deletes the custom tutorial and restores the default guide.

Save Rules

When built-in collector configuration is saved, the internal connectivity_validation field is removed so validation state does not mix with user configuration.

BarentsWatch client_secret has special handling:

• The input shows a masked preview.
• If the submitted value still matches the masked preview, the backend keeps the old secret.
• If a new value is submitted, the secret is replaced.
• The previous separate "clear current secret" checkbox is no longer provided.

Test Coverage

Related tests:

• test_vessels.py

Added coverage:

• BarentsWatch credentials can be parsed from ~/.zshrc.
• When environment variables are empty, resolve_barentswatch_config() can fall back to ~/.zshrc.
• Vessel data conversion and GeoJSON output remain compatible.

Current Provider Coverage

Credential providers currently supported:

• barentswatch
• aisstream
• spacetrack

Other collectors with requires_credentials=true return that their credential chain has not been wired yet, and the frontend shows Unavailable.