planet/docs/technical/en/datasource-collector-settings-connectivity.md

Collector Settings and Connectivity Validation

Background

The console now separates the "data source catalog" from "collector configuration":

• /datasources
  • Lists all data sources, including built-in and custom sources.
  • Clicking a name only opens an information drawer.
  • Focuses on status, manual collection, and running collection tasks.
• /settings?tab=collector_credentials
  • Displays as "Collector Settings".
  • Owns endpoint, headers, base parameters, and credentials.
  • Every collector exposes a connection button for health checks.

This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.

User-Facing Rules

Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.

A built-in collector is considered "connected" when either condition is true:

• The current configuration has successfully collected data.
• The user clicked the connection button for the current configuration and backend validation succeeded.

If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".

Frontend Entry Points

Data Source Catalog

Files:

• DataSources.tsx
• index.css

Current behavior:

• Built-in and custom data sources are merged into a UnifiedDataSource list.
• The table only keeps view, collect, and status actions.
• Clicking the name opens a read-only drawer.
• The drawer shows:
  • Whether the source is built in
  • Whether it is enabled
  • Module, priority, and frequency
  • Endpoint
  • Headers
  • Base configuration
  • Whether credentials are required
• When tasks are running, the top progress area shows a clickable Collecting N pill.
• Clicking Collecting N opens a task list modal with per-task progress.

data-source-bulk-toolbar__running-pill is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.

Collector Settings

File:

• Settings.tsx

Current behavior:

• The collector_credentials tab is displayed as "Collector Settings".
• A select lists all built-in collectors.
• The only button beside the select is a plug icon for health checks.
• Status tags below the select show:
  • Credentials required / No credentials required
  • Module
  • Enabled / Disabled
  • Unchecked / Available / Unavailable
  • Whether the endpoint is overridden
• Collectors that require credentials place the credential card above base configuration.
• Collectors without credentials only show base configuration.

The connection button uses an inline Tabler-style plug icon with plug-connected semantics, avoiding the older refresh icon for a connection action.

Backend APIs

Data Source Configuration List

GET /api/v1/datasources/configs/all

Returns a merged view of YAML default data sources and database overrides. This route must be declared before /configs/{config_id}; otherwise FastAPI treats all as a path parameter and returns 422.

Returned fields include:

• name
• default_url
• endpoint
• is_overridden
• is_active
• source_type
• auth_type
• headers
• config
• config_id
• description

Before returning config, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.

Built-In Collector Connection Status

POST /api/v1/datasources/configs/builtin/connection-status

Purpose:

• Accept a candidate configuration.
• Compute its checksum.
• Determine whether the current configuration is already connected.

The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.

Built-In Collector Connectivity Validation

POST /api/v1/datasources/configs/builtin/connect

Purpose:

• Free collectors request the endpoint directly.
• Credentialed collectors go through their credential provider.
• Successful validation writes a system-level connection record.

Successful responses include:

• success
• connected
• checksum
• stage
• message
• response_time_ms
• credential_provider
• credential_source

BarentsWatch AIS Connectivity Validation

POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/integrations/barentswatch/connectivity

BarentsWatch uses separate endpoints because draft credentials must be validated before saving:

• Use draft client_id / client_secret to fetch a token.
• Use that token to request the AIS endpoint.
• After success, write a built-in collector connection record using the draft credential fingerprint.

Connectivity Validation Service

File:

• datasource_connectivity.py

Core responsibilities:

• Compute built-in collector configuration checksums.
• Read credentials from environment variables and ~/.zshrc.
• Determine whether the current configuration is already connected.
• Run endpoint health checks.
• Save successful connection records.

Checksum Inputs

The checksum includes:

• Collector name
• Endpoint
• Auth type
• Headers
• Config after removing internal validation fields
• Credential provider
• Credential fingerprint

The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.

Connection Records

Successful connection records are written to SystemSetting:

category = datasource_connectivity_validations

The payload uses collector source as the key:

{
  "barentswatch_vessels": {
    "checksum": "...",
    "status": "success",
    "validated_at": "2026-04-29T00:00:00+00:00",
    "status_code": 200,
    "credential_source": "datasource_config",
    "connected_by": "connection_button"
  }
}

connected_by currently has two sources:

• connection_button: the user manually clicked the connection button.
• collection: a collection task completed successfully, so the system recorded the current effective configuration as connected.

Successful Collection Means Connected

After a successful collection, the scheduler writes a connection record:

• scheduler.py

This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.

BarentsWatch AIS Credential Chain

Files:

• barentswatch.py
• vessel_ais.py

Resolution priority:

1. DataSourceConfig.auth_config
2. DataSourceConfig.config
3. Environment variables
4. ~/.zshrc

Supported environment variables:

export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."

Historical misspellings are also supported:

export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."

Token request rules:

• Token URL: https://id.barentswatch.no/connect/token
• Content-Type: application/x-www-form-urlencoded
• Body:
  • grant_type=client_credentials
  • client_id
  • client_secret
  • scope=ais

AIS request rules:

• Default endpoint: https://live.ais.barentswatch.no/v1/latest/combined
• Header: Authorization: Bearer <access_token>

VesselAISCollector no longer reads environment variables directly. It goes through resolve_barentswatch_config() and fetch_barentswatch_access_token() so settings, connectivity validation, and collection do not fork into three credential flows.

AISStream Collector Chain

Files:

• aisstream.py
• vessel_ais_aggregation.py

AISStream uses a WebSocket realtime stream. The collector writes only to the ais_raw_observations raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.

Configuration:

• api_key: stored in DataSourceConfig.auth_config, or provided through AISSTREAM_API_KEY.
• endpoint: defaults to wss://stream.aisstream.io/v0/stream.
• message_types: defaults to PositionReport and ShipStaticData.
• bounding_boxes: AISStream format is [[[lat_min, lon_min], [lat_max, lon_max]]]; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
• max_messages and receive_timeout_seconds: control the batch-style WebSocket collection window.

Normalization:

• PositionReport mainly provides position, speed, course, heading, and navigation status.
• Vessel names can be filled from MetaData.ShipName even when the message body has no name.
• Vessel type usually comes from lower-frequency ShipStaticData.Type; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
• If a vessel has not yet produced a static message, its aggregated type can still be Other; v5 vessel profile enrichment is planned to fill that gap.

Credential Guide

File:

• credential_guides.py

APIs:

GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset

Currently supported:

• barentswatch
• aisstream

The default guide includes the official BarentsWatch tutorial:

https://developer.barentswatch.no/docs/tutorial

If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to SystemSetting:

category = collector_credential_guides

Reset deletes the custom tutorial and restores the default guide.

Save Rules

When built-in collector configuration is saved, the internal connectivity_validation field is removed so validation state does not mix with user configuration.

BarentsWatch client_secret has special handling:

• The input shows a masked preview.
• If the submitted value still matches the masked preview, the backend keeps the old secret.
• If a new value is submitted, the secret is replaced.
• The previous separate "clear current secret" checkbox is no longer provided.

Test Coverage

Related tests:

• test_vessels.py

Added coverage:

• BarentsWatch credentials can be parsed from ~/.zshrc.
• When environment variables are empty, resolve_barentswatch_config() can fall back to ~/.zshrc.
• Vessel data conversion and GeoJSON output remain compatible.

Current Provider Coverage

Credential providers currently supported:

• barentswatch
• spacetrack

Other collectors with requires_credentials=true return that their credential chain has not been wired yet, and the frontend shows Unavailable.