planet/docs/technical/en/datasource-collector-settings-connectivity.md

# Collector Settings and Connectivity Validation

## Background

The console now separates the "data source catalog" from "collector configuration":

- `/datasources`
  - Lists all data sources, including built-in and custom sources.
  - Clicking a name only opens an information drawer.
  - Focuses on status, manual collection, and running collection tasks.
- `/collection-management?tab=collector_credentials`
  - Displays as "Collectors".
  - Owns endpoint, headers, base parameters, and credentials.
  - Every collector exposes a connection button for health checks.

This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to Collectors instead of being scattered across the data source list and system settings.

## User-Facing Rules

Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.

A built-in collector is considered "connected" when either condition is true:

- The current configuration has successfully collected data.
- The user clicked the connection button for the current configuration and backend validation succeeded.

If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".

## Frontend Entry Points

### Data Source Catalog

Files:

- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)
- [AdminRoutes.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/AdminRoutes.tsx)

Current behavior:

- Built-in and custom data sources are merged into a `UnifiedDataSource` list.
- The table only keeps view, collect, and status actions.
- Clicking the name opens a read-only drawer.
- The drawer shows:
  - Whether the source is built in
  - Whether it is enabled
  - Module, priority, and frequency
  - Endpoint
  - Headers
  - Base configuration
  - Whether credentials are required
- When tasks are running, the top progress area shows a clickable `Collecting N` pill.
- Clicking `Collecting N` opens a task list modal with per-task progress.

`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.

### Collection Management

File:

- [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx)

Current behavior:

- `/collection-management` follows the legacy hierarchy: `Collectors`, `Collection Schedule`, and `Collection History / Snapshots`.
- `Collectors` is the configuration page. The left list shows collector configs; the right form edits endpoint, authentication, headers, collection parameters, and enabled state.
- New collectors and target schemas use draft detail pages instead of transparent JSON modals; save persists the draft, while cancel destroys it.
- The connection button only tests connectivity and does not save. The save button only persists the form.
- Credential guides open in a draggable Markdown modal. When no guide exists, the modal still opens and offers a generate action. Generation shows a centered waiting state; reset confirmation must render above the guide modal.
- `Collection History / Snapshots` groups by collector. The list must not repeat every snapshot for the same datasource; detail view switches versions through a Time Capsule / Time Machine style selector.
- The AISStream collector uses WebSocket semantics: connecting, streaming, reconnecting, or stopped. It does not use a fixed completion percentage.
- Custom source editing lives in collector settings. The data source catalog keeps overview, run controls, and read-only drawers.

The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.

### Mapping Templates and Target Schemas

Mapping remains part of collection management, but not the main collector tab:

- `Mapping Templates` owns sample payload, AI propose, preview, create/update, and activate.
- `Target Schemas` maintains writable target structures.
- `run-mapped`, `stop-mapped`, and `stream-status` run mapped custom collectors.

The new UI should keep these paths form-first. Only advanced fields should collapse into JSON. Do not flatten templates, schemas, runtime status, and collector configuration into one table.

## Backend APIs

### Data Source Configuration List

```http
GET /api/v1/datasources/configs/all
```

Returns a merged view of YAML default data sources and database overrides. This route must be declared before `/configs/{config_id}`; otherwise FastAPI treats `all` as a path parameter and returns 422.

Returned fields include:

- `name`
- `default_url`
- `endpoint`
- `is_overridden`
- `is_active`
- `source_type`
- `auth_type`
- `headers`
- `config`
- `config_id`
- `description`

Before returning `config`, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.

### Built-In Collector Connection Status

```http
POST /api/v1/datasources/configs/builtin/connection-status
```

Purpose:

- Accept a candidate configuration.
- Compute its checksum.
- Determine whether the current configuration is already connected.

The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.

### Built-In Collector Connectivity Validation

```http
POST /api/v1/datasources/configs/builtin/connect
```

Purpose:

- Free collectors request the endpoint directly.
- Credentialed collectors go through their credential provider.
- Successful validation writes a system-level connection record.

Successful responses include:

- `success`
- `connected`
- `checksum`
- `stage`
- `message`
- `response_time_ms`
- `credential_provider`
- `credential_source`

### Credential Guides

```http
GET  /api/v1/datasources/credential-guides/{provider}
POST /api/v1/datasources/credential-guides/{provider}/generate
POST /api/v1/datasources/credential-guides/{provider}/reset
```

Purpose:

- Read the Markdown credential guide for a provider.
- Generate a guide from collector metadata when none exists.
- Reset back to the backend default guide.

Frontend rules:

- Render Markdown, not backend prompts or metadata.
- Generate/reset actions belong inside the guide modal, not in the collector configuration toolbar.
- Missing guides still open a modal so the user can generate one from there.

### BarentsWatch AIS Connectivity Validation

```http
POST /api/v1/settings/integrations/barentswatch/connect
GET  /api/v1/settings/integrations/barentswatch/connectivity
```

BarentsWatch uses separate endpoints because draft credentials must be validated before saving:

- Use draft `client_id` / `client_secret` to fetch a token.
- Use that token to request the AIS endpoint.
- After success, write a built-in collector connection record using the draft credential fingerprint.

## Connectivity Validation Service

File:

- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)

Core responsibilities:

- Compute built-in collector configuration checksums.
- Read credentials from environment variables and `~/.zshrc`.
- Determine whether the current configuration is already connected.
- Run endpoint health checks.
- Save successful connection records.

### Checksum Inputs

The checksum includes:

- Collector name
- Endpoint
- Auth type
- Headers
- Config after removing internal validation fields
- Credential provider
- Credential fingerprint

The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.

### Connection Records

Successful connection records are written to `SystemSetting`:

```text
category = datasource_connectivity_validations
```

The payload uses collector source as the key:

```json
{
  "barentswatch_vessels": {
    "checksum": "...",
    "status": "success",
    "validated_at": "2026-04-29T00:00:00+00:00",
    "status_code": 200,
    "credential_source": "datasource_config",
    "connected_by": "connection_button"
  }
}
```

`connected_by` currently has two sources:

- `connection_button`: the user manually clicked the connection button.
- `collection`: a collection task completed successfully, so the system recorded the current effective configuration as connected.

### Successful Collection Means Connected

After a successful collection, the scheduler writes a connection record:

- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)

This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.

## BarentsWatch AIS Credential Chain

Files:

- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)

Resolution priority:

1. `DataSourceConfig.auth_config`
2. `DataSourceConfig.config`
3. Environment variables
4. `~/.zshrc`

Supported environment variables:

```bash
export BARENTSWATCH_CLIENT_ID="..."
export BARENTSWATCH_CLIENT_SECRET="..."
```

Historical misspellings are also supported:

```bash
export BARRENTSWATCH_CLIENT_ID="..."
export BARRENTSWATCH_CLIENT_SECRET="..."
```

Token request rules:

- Token URL: `https://id.barentswatch.no/connect/token`
- `Content-Type`: `application/x-www-form-urlencoded`
- Body:
  - `grant_type=client_credentials`
  - `client_id`
  - `client_secret`
  - `scope=ais`

AIS request rules:

- Default endpoint: `https://live.ais.barentswatch.no/v1/latest/combined`
- Header: `Authorization: Bearer <access_token>`

`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.

## AISStream Collector Chain

Files:

- [aisstream.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/aisstream.py)
- [vessel_ais_aggregation.py](/home/ray/dev/linkong/planet/backend/app/services/vessel_ais_aggregation.py)

AISStream uses a WebSocket realtime stream. The collector writes only to the `ais_raw_observations` raw observation layer; it does not directly overwrite the final vessel display table. The aggregation API handles multi-source deduplication, field selection, and conflict records.

Configuration:

- `api_key`: stored in `DataSourceConfig.auth_config`, or provided through `AISSTREAM_API_KEY`.
- `endpoint`: defaults to `wss://stream.aisstream.io/v0/stream`.
- `message_types`: defaults to `PositionReport` and `ShipStaticData`.
- `bounding_boxes`: AISStream format is `[[[lat_min, lon_min], [lat_max, lon_max]]]`; the settings page provides global, Norway / North Sea, Europe coast, East Asia, and North America coast presets.
- `max_messages` and `receive_timeout_seconds`: control the batch-style WebSocket collection window.

Normalization:

- `PositionReport` mainly provides position, speed, course, heading, and navigation status.
- Vessel names can be filled from `MetaData.ShipName` even when the message body has no `name`.
- Vessel type usually comes from lower-frequency `ShipStaticData.Type`; the backend maps AIS numeric type codes to Cargo / Tanker / Passenger / Fishing / Military.
- If a vessel has not yet produced a static message, its aggregated type can still be `Other`; v5 vessel profile enrichment is planned to fill that gap.

Connectivity validation reads saved configuration, environment variables, and `AISSTREAM_API_KEY` from `~/.zshrc`. For actual collection, prefer saving the API key in collector settings. If the key only lives in `~/.zshrc`, confirm that the backend process inherited it; otherwise validation may pass while the collector runtime cannot read the key.

Connectivity validation and actual collection are separate actions. A banner such as `AISStream credentials configured, WebSocket endpoint format valid` only means the saved settings can be used for a connection attempt; runtime status may still be `disconnected`. Global AIS data is written locally only while `aisstream_vessels` is `streaming` / `connected` and its realtime stream counters plus `last_seen_at` keep advancing. Start, stop, reconnect, health, and counters are exposed through `/datasources -> Realtime Streams` and `/api/v1/realtime-sources`; AISStream is not counted in normal one-click collection progress.

The new vessel list entry point is no longer the legacy `/api/v1/visualization/geo/vessels` route. Earth initial state should call:

```http
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
```

That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. If the `/ws` `vessels` channel is connected, it should act as a reload/dirty hint for merged refreshes, not as a per-AIS-delta full-layer rebuild path.

## Custom REST / WebSocket Mapping Runtime

Files:

- [custom_datasource_runtime.py](/home/ray/dev/linkong/planet/backend/app/services/custom_datasource_runtime.py)
- [datasource_mapping.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_mapping.py)

Custom sources are supplemental inputs for existing target schemas, not isolated data islands. The most complete target today is `vessel_ais`: a custom REST or WebSocket source is mapped deterministically, written into AIS raw observations, and then pushed to Earth through the `vessels` WebSocket channel.

Earth high-precision boundaries no longer use custom-source target schemas. Boundaries are Earth static assets: the Earth Assets settings panel saves local source configuration and triggers PMTiles builds without writing records to `CollectedData`.

### Configuration Semantics

Important fields:

- `source_type`: `rest` / `http` / `websocket` / `ws`.
- `endpoint`: REST uses `http(s)://`; WebSocket uses `ws(s)://`.
- `auth_type`: `none`, `bearer`, `api_key`, or `basic`.
- `headers`: static request headers.
- `auth_config`: token, API key, or basic username/password; API keys can be sent by header or query.
- `config.target_schema`: for example `vessel_ais`, `geo_points`, or `generic_records`.
- `config.delivery_mode`: REST defaults to `polling`; WebSocket defaults to `realtime_stream`.
- `config.merge_target_source`: records which built-in source this custom source supplements, such as `barentswatch_vessels`.

The REST runner supports:

- `GET` / `POST`
- query params
- JSON body
- headers and auth injection
- active mapping writes into the target schema

The WebSocket runner supports:

- endpoint format validation
- headers and auth injection
- optional `ws_subscribe_message`
- `ws_message_path` / `ws_items_path` extraction
- reconnects
- `debug_max_messages` debug limits
- background stream start / stop / status

Related APIs:

```http
POST /api/v1/datasources/custom/sample
GET  /api/v1/datasources/target-schemas
POST /api/v1/datasources/{config_id}/run-mapped
POST /api/v1/datasources/{config_id}/stop-mapped
GET  /api/v1/datasources/{config_id}/mapped-status
DELETE /api/v1/datasources/configs/{config_id}?delete_mappings=true&delete_source_data=true
```

`run-mapped?background=true` only matters for WebSocket sources and starts a background stream. REST sources remain one-shot collection runs.

### Delete And Data Cleanup

Deleting a custom source has three levels:

- Delete configuration only: preserve mapping and historical data.
- Delete configuration and mapping: also delete mapping templates for that config.
- Delete configuration, mapping, and source data: delete that source's `collected_data`, `ais_raw_observations`, and `ais_source_health`.

When deleted `vessel_ais` source data affects Earth, the backend broadcasts `reload_required` on the `vessels` channel so Earth reloads aggregated vessels. Legacy `vessel_position` rows are not deleted by custom source because that table cannot safely attribute rows back to a custom source.

### Local AIS Mock WebSocket

File:

- [mock-ais-ws-server.ts](/home/ray/dev/linkong/planet/scripts/mock-ais-ws-server.ts)

Run:

```bash
bun run mock:ais-ws
```

The mock service continuously sends AIS-like JSON to validate the chain: WebSocket custom source -> mapping -> AIS raw observation -> `vessels` channel -> Earth vessel upsert. Typical config:

```json
{
  "source_type": "websocket",
  "endpoint": "ws://localhost:8787",
  "config": {
    "target_schema": "vessel_ais",
    "delivery_mode": "realtime_stream",
    "merge_target_source": "barentswatch_vessels",
    "ws_message_path": "$.data",
    "ws_items_path": "$.vessels[*]",
    "ws_reconnect": true
  }
}
```

## Credential Guide

File:

- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)

APIs:

```http
GET  /api/v1/settings/credential-guides/{provider}
POST /api/v1/settings/credential-guides/{provider}/generate
POST /api/v1/settings/credential-guides/{provider}/reset
```

Currently supported:

- `barentswatch`
- `aisstream`

The default guide includes the official BarentsWatch tutorial:

```text
https://developer.barentswatch.no/docs/tutorial
```

If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to `SystemSetting`:

```text
category = collector_credential_guides
```

Reset deletes the custom tutorial and restores the default guide.

## Save Rules

When built-in collector configuration is saved, the internal `connectivity_validation` field is removed so validation state does not mix with user configuration.

BarentsWatch `client_secret` has special handling:

- The input shows a masked preview.
- If the submitted value still matches the masked preview, the backend keeps the old secret.
- If a new value is submitted, the secret is replaced.
- The previous separate "clear current secret" checkbox is no longer provided.

## Test Coverage

Related tests:

- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)

Added coverage:

- BarentsWatch credentials can be parsed from `~/.zshrc`.
- When environment variables are empty, `resolve_barentswatch_config()` can fall back to `~/.zshrc`.
- Vessel data conversion and GeoJSON output remain compatible.

## Current Provider Coverage

Credential providers currently supported:

- `barentswatch`
- `aisstream`
- `spacetrack`

Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.