planet/docs/technical/en/datasource-collector-settings-connectivity.md

# Collector Settings and Connectivity Validation

## Background

The console now separates the "data source catalog" from "collector configuration":

- `/datasources`
  - Lists all data sources, including built-in and custom sources.
  - Clicking a name only opens an information drawer.
  - Focuses on status, manual collection, and running collection tasks.
- `/settings?tab=collector_credentials`
  - Displays as "Collector Settings".
  - Owns endpoint, headers, base parameters, and credentials.
  - Every collector exposes a connection button for health checks.

This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.

## User-Facing Rules

Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.

A built-in collector is considered "connected" when either condition is true:

- The current configuration has successfully collected data.
- The user clicked the connection button for the current configuration and backend validation succeeded.

If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".

## Frontend Entry Points

### Data Source Catalog

Files:

- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)

Current behavior:

- Built-in and custom data sources are merged into a `UnifiedDataSource` list.
- The table only keeps view, collect, and status actions.
- Clicking the name opens a read-only drawer.
- The drawer shows:
  - Whether the source is built in
  - Whether it is enabled
  - Module, priority, and frequency
  - Endpoint
  - Headers
  - Base configuration
  - Whether credentials are required
- When tasks are running, the top progress area shows a clickable `Collecting N` pill.
- Clicking `Collecting N` opens a task list modal with per-task progress.

`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.

### Collector Settings

File:

- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)

Current behavior:

- The `collector_credentials` tab is displayed as "Collector Settings".
- A select lists all built-in collectors.
- The only button beside the select is a plug icon for health checks.
- Status tags below the select show:
  - `Credentials required` / `No credentials required`
  - Module
  - `Enabled` / `Disabled`
  - `Unchecked` / `Available` / `Unavailable`
  - Whether the endpoint is overridden
- Collectors that require credentials place the credential card above base configuration.
- Collectors without credentials only show base configuration.

The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.

## Backend APIs

### Data Source Configuration List

```http
GET /api/v1/datasources/configs/all
```

Returns a merged view of YAML default data sources and database overrides. This route must be declared before `/configs/{config_id}`; otherwise FastAPI treats `all` as a path parameter and returns 422.

Returned fields include:

- `name`
- `default_url`
- `endpoint`
- `is_overridden`
- `is_active`
- `source_type`
- `auth_type`
- `headers`
- `config`
- `config_id`
- `description`

Before returning `config`, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.

### Built-In Collector Connection Status

```http
POST /api/v1/datasources/configs/builtin/connection-status
```

Purpose:

- Accept a candidate configuration.
- Compute its checksum.
- Determine whether the current configuration is already connected.

The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.

### Built-In Collector Connectivity Validation

```http
POST /api/v1/datasources/configs/builtin/connect
```

Purpose:

- Free collectors request the endpoint directly.
- Credentialed collectors go through their credential provider.
- Successful validation writes a system-level connection record.

Successful responses include:

- `success`
- `connected`
- `checksum`
- `stage`
- `message`
- `response_time_ms`
- `credential_provider`
- `credential_source`

### BarentsWatch AIS Connectivity Validation

```http
POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/integrations/barentswatch/connectivity
```

BarentsWatch uses separate endpoints because draft credentials must be validated before saving:

- Use draft `client_id` / `client_secret` to fetch a token.
- Use that token to request the AIS endpoint.
- After success, write a built-in collector connection record using the draft credential fingerprint.

## Connectivity Validation Service

File:

- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)

Core responsibilities:

- Compute built-in collector configuration checksums.
- Read credentials from environment variables and `~/.zshrc`.
- Determine whether the current configuration is already connected.
- Run endpoint health checks.
- Save successful connection records.

### Checksum Inputs

The checksum includes:

- Collector name
- Endpoint
- Auth type
- Headers
- Config after removing internal validation fields
- Credential provider
- Credential fingerprint

The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.

### Connection Records

Successful connection records are written to `SystemSetting`:

```text
category = datasource_connectivity_validations
```

The payload uses collector source as the key:

```json
{
  "barentswatch_vessels": {
    "checksum": "...",
    "status": "success",
    "validated_at": "2026-04-29T00:00:00+00:00",
    "status_code": 200,
    "credential_source": "datasource_config",
    "connected_by": "connection_button"
  }
}
```

`connected_by` currently has two sources:

- `connection_button`: the user manually clicked the connection button.
- `collection`: a collection task completed successfully, so the system recorded the current effective configuration as connected.

### Successful Collection Means Connected

After a successful collection, the scheduler writes a connection record:

- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)

This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.

## BarentsWatch AIS Credential Chain

Files:

- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)

Resolution priority:

1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. Environment variables
4. `~/.zshrc`

Supported environment variables:

```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```

Historical misspellings are also supported:

```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```

Token request rules:

- Token URL: `https://id.barentswatch.no/connect/token`
- `Content-Type`: `application/x-www-form-urlencoded`
- Body:
  - `grant_type=client_credentials`
  - `client_id`
  - `client_secret`
  - `scope=ais`

AIS request rules:

- Default endpoint: `https://live.ais.barentswatch.no/v1/latest/combined`
- Header: `Authorization: Bearer <access_token>`

`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.

## AISStream Collector Chain

Files:

- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)

AISStream uses a WebSocket realtime stream. The collector writes only to the `ais_raw_observations` raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.

Configuration:

- `api_key`: stored in `DataSourceConfig.auth_config`, or provided through `AISSTREAM_API_KEY`.
- `endpoint`: defaults to `wss://stream.aisstream.io/v0/stream`.
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
- `bounding_boxes`: AISStream format is `[[[lat_min, lon_min], [lat_max, lon_max]]]`; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
- `max_messages` and `receive_timeout_seconds`: control the batch-style WebSocket collection window.

Normalization:

- `PositionReport` mainly provides position, speed, course, heading, and navigation status.
- Vessel names can be filled from `MetaData.ShipName` even when the message body has no `name`.
- Vessel type usually comes from lower-frequency `ShipStaticData.Type`; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
- If a vessel has not yet produced a static message, its aggregated type can still be `Other`; v5 vessel profile enrichment is planned to fill that gap.

## Credential Guide

File:

- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)

APIs:

```http
GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```

Currently supported:

- `barentswatch`
- `aisstream`

The default guide includes the official BarentsWatch tutorial:

```text
https://developer.barentswatch.no/docs/tutorial
```

If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to `SystemSetting`:

```text
category = collector_credential_guides
```

Reset deletes the custom tutorial and restores the default guide.

## Save Rules

When built-in collector configuration is saved, the internal `connectivity_validation` field is removed so validation state does not mix with user configuration.

BarentsWatch `client_secret` has special handling:

- The input shows a masked preview.
- If the submitted value still matches the masked preview, the backend keeps the old secret.
- If a new value is submitted, the secret is replaced.
- The previous separate "clear current secret" checkbox is no longer provided.

## Test Coverage

Related tests:

- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)

Added coverage:

- BarentsWatch credentials can be parsed from `~/.zshrc`.
- When environment variables are empty, `resolve_barentswatch_config()` can fall back to `~/.zshrc`.
- Vessel data conversion and GeoJSON output remain compatible.

## Current Provider Coverage

Credential providers currently supported:

- `barentswatch`
- `spacetrack`

Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.