release: bump version to 0.46.1
This commit is contained in:
linkong
325
docs/technical/en/datasource-collector-settings-connectivity.md
Normal file
325
docs/technical/en/datasource-collector-settings-connectivity.md
Normal file
@@ -0,0 +1,325 @@
|
||||
# Collector Settings and Connectivity Validation
|
||||
|
||||
## Background
|
||||
|
||||
The console now separates the "data source catalog" from "collector configuration":
|
||||
|
||||
- `/datasources`
|
||||
- Lists all data sources, including built-in and custom sources.
|
||||
- Clicking a name only opens an information drawer.
|
||||
- Focuses on status, manual collection, and running collection tasks.
|
||||
- `/settings?tab=collector_credentials`
|
||||
- Displays as "Collector Settings".
|
||||
- Owns endpoint, headers, base parameters, and credentials.
|
||||
- Every collector exposes a connection button for health checks.
|
||||
|
||||
This reduces first-use confusion: API endpoints, headers, credentials, and custom source configuration all belong to collector settings instead of being scattered across the data source list and system settings.
|
||||
|
||||
## User-Facing Rules
|
||||
|
||||
Connection state is not a frontend styling state. The backend derives it from the current configuration checksum and previously validated records.
|
||||
|
||||
A built-in collector is considered "connected" when either condition is true:
|
||||
|
||||
- The current configuration has successfully collected data.
|
||||
- The user clicked the connection button for the current configuration and backend validation succeeded.
|
||||
|
||||
If endpoint, headers, base configuration, or credential fingerprint changes after the last successful validation, the state returns to "needs reconnection".
|
||||
|
||||
## Frontend Entry Points
|
||||
|
||||
### Data Source Catalog
|
||||
|
||||
Files:
|
||||
|
||||
- [DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx)
|
||||
- [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
Current behavior:
|
||||
|
||||
- Built-in and custom data sources are merged into a `UnifiedDataSource` list.
|
||||
- The table only keeps view, collect, and status actions.
|
||||
- Clicking the name opens a read-only drawer.
|
||||
- The drawer shows:
|
||||
- Whether the source is built in
|
||||
- Whether it is enabled
|
||||
- Module, priority, and frequency
|
||||
- Endpoint
|
||||
- Headers
|
||||
- Base configuration
|
||||
- Whether credentials are required
|
||||
- When tasks are running, the top progress area shows a clickable `Collecting N` pill.
|
||||
- Clicking `Collecting N` opens a task list modal with per-task progress.
|
||||
|
||||
`data-source-bulk-toolbar__running-pill` is the styling entry point for the "Collecting" pill. It is aligned with other status tags, while hover treatment, arrow affordance, and blue outline indicate interactivity.
|
||||
|
||||
### Collector Settings
|
||||
|
||||
File:
|
||||
|
||||
- [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx)
|
||||
|
||||
Current behavior:
|
||||
|
||||
- The `collector_credentials` tab is displayed as "Collector Settings".
|
||||
- A select lists all built-in collectors.
|
||||
- The only button beside the select is a plug icon for health checks.
|
||||
- Status tags below the select show:
|
||||
- `Credentials required` / `No credentials required`
|
||||
- Module
|
||||
- `Enabled` / `Disabled`
|
||||
- `Unchecked` / `Available` / `Unavailable`
|
||||
- Whether the endpoint is overridden
|
||||
- Collectors that require credentials place the credential card above base configuration.
|
||||
- Collectors without credentials only show base configuration.
|
||||
|
||||
The connection button uses an inline Tabler-style plug icon with `plug-connected` semantics, avoiding the older refresh icon for a connection action.
|
||||
|
||||
## Backend APIs
|
||||
|
||||
### Data Source Configuration List
|
||||
|
||||
```http
|
||||
GET /api/v1/datasources/configs/all
|
||||
```
|
||||
|
||||
Returns a merged view of YAML default data sources and database overrides. This route must be declared before `/configs/{config_id}`; otherwise FastAPI treats `all` as a path parameter and returns 422.
|
||||
|
||||
Returned fields include:
|
||||
|
||||
- `name`
|
||||
- `default_url`
|
||||
- `endpoint`
|
||||
- `is_overridden`
|
||||
- `is_active`
|
||||
- `source_type`
|
||||
- `auth_type`
|
||||
- `headers`
|
||||
- `config`
|
||||
- `config_id`
|
||||
- `description`
|
||||
|
||||
Before returning `config`, internal connectivity validation fields are removed so the frontend does not display validation metadata as user configuration.
|
||||
|
||||
### Built-In Collector Connection Status
|
||||
|
||||
```http
|
||||
POST /api/v1/datasources/configs/builtin/connection-status
|
||||
```
|
||||
|
||||
Purpose:
|
||||
|
||||
- Accept a candidate configuration.
|
||||
- Compute its checksum.
|
||||
- Determine whether the current configuration is already connected.
|
||||
|
||||
The current frontend mostly performs an immediate check through the connection button and does not strongly depend on this endpoint. It remains the backend basis for future save-button disabling and restoring initial page state.
|
||||
|
||||
### Built-In Collector Connectivity Validation
|
||||
|
||||
```http
|
||||
POST /api/v1/datasources/configs/builtin/connect
|
||||
```
|
||||
|
||||
Purpose:
|
||||
|
||||
- Free collectors request the endpoint directly.
|
||||
- Credentialed collectors go through their credential provider.
|
||||
- Successful validation writes a system-level connection record.
|
||||
|
||||
Successful responses include:
|
||||
|
||||
- `success`
|
||||
- `connected`
|
||||
- `checksum`
|
||||
- `stage`
|
||||
- `message`
|
||||
- `response_time_ms`
|
||||
- `credential_provider`
|
||||
- `credential_source`
|
||||
|
||||
### BarentsWatch AIS Connectivity Validation
|
||||
|
||||
```http
|
||||
POST /api/v1/settings/integrations/barentswatch/connect
|
||||
GET /api/v1/settings/integrations/barentswatch/connectivity
|
||||
```
|
||||
|
||||
BarentsWatch uses separate endpoints because draft credentials must be validated before saving:
|
||||
|
||||
- Use draft `client_id` / `client_secret` to fetch a token.
|
||||
- Use that token to request the AIS endpoint.
|
||||
- After success, write a built-in collector connection record using the draft credential fingerprint.
|
||||
|
||||
## Connectivity Validation Service
|
||||
|
||||
File:
|
||||
|
||||
- [datasource_connectivity.py](/home/ray/dev/linkong/planet/backend/app/services/datasource_connectivity.py)
|
||||
|
||||
Core responsibilities:
|
||||
|
||||
- Compute built-in collector configuration checksums.
|
||||
- Read credentials from environment variables and `~/.zshrc`.
|
||||
- Determine whether the current configuration is already connected.
|
||||
- Run endpoint health checks.
|
||||
- Save successful connection records.
|
||||
|
||||
### Checksum Inputs
|
||||
|
||||
The checksum includes:
|
||||
|
||||
- Collector name
|
||||
- Endpoint
|
||||
- Auth type
|
||||
- Headers
|
||||
- Config after removing internal validation fields
|
||||
- Credential provider
|
||||
- Credential fingerprint
|
||||
|
||||
The credential fingerprint is a hash of credential content. Plaintext credentials are not written into connection records.
|
||||
|
||||
### Connection Records
|
||||
|
||||
Successful connection records are written to `SystemSetting`:
|
||||
|
||||
```text
|
||||
category = datasource_connectivity_validations
|
||||
```
|
||||
|
||||
The payload uses collector source as the key:
|
||||
|
||||
```json
|
||||
{
|
||||
"barentswatch_vessels": {
|
||||
"checksum": "...",
|
||||
"status": "success",
|
||||
"validated_at": "2026-04-29T00:00:00+00:00",
|
||||
"status_code": 200,
|
||||
"credential_source": "datasource_config",
|
||||
"connected_by": "connection_button"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`connected_by` currently has two sources:
|
||||
|
||||
- `connection_button`: the user manually clicked the connection button.
|
||||
- `collection`: a collection task completed successfully, so the system recorded the current effective configuration as connected.
|
||||
|
||||
### Successful Collection Means Connected
|
||||
|
||||
After a successful collection, the scheduler writes a connection record:
|
||||
|
||||
- [scheduler.py](/home/ray/dev/linkong/planet/backend/app/services/scheduler.py)
|
||||
|
||||
This prevents collectors that already have data from asking the user to validate again. Reconnection is only required when the configuration checksum changes.
|
||||
|
||||
## BarentsWatch AIS Credential Chain
|
||||
|
||||
Files:
|
||||
|
||||
- [barentswatch.py](/home/ray/dev/linkong/planet/backend/app/services/barentswatch.py)
|
||||
- [vessel_ais.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
|
||||
|
||||
Resolution priority:
|
||||
|
||||
1. `DataSourceConfig.auth_config`
|
||||
2. `DataSourceConfig.config`
|
||||
3. Environment variables
|
||||
4. `~/.zshrc`
|
||||
|
||||
Supported environment variables:
|
||||
|
||||
```bash
|
||||
export BARENTSWATCH_CLIENT_ID="..."
|
||||
export BARENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
Historical misspellings are also supported:
|
||||
|
||||
```bash
|
||||
export BARRENTSWATCH_CLIENT_ID="..."
|
||||
export BARRENTSWATCH_CLIENT_SECRET="..."
|
||||
```
|
||||
|
||||
Token request rules:
|
||||
|
||||
- Token URL: `https://id.barentswatch.no/connect/token`
|
||||
- `Content-Type`: `application/x-www-form-urlencoded`
|
||||
- Body:
|
||||
- `grant_type=client_credentials`
|
||||
- `client_id`
|
||||
- `client_secret`
|
||||
- `scope=ais`
|
||||
|
||||
AIS request rules:
|
||||
|
||||
- Default endpoint: `https://live.ais.barentswatch.no/v1/latest/combined`
|
||||
- Header: `Authorization: Bearer <access_token>`
|
||||
|
||||
`VesselAISCollector` no longer reads environment variables directly. It goes through `resolve_barentswatch_config()` and `fetch_barentswatch_access_token()` so settings, connectivity validation, and collection do not fork into three credential flows.
|
||||
|
||||
## Credential Guide
|
||||
|
||||
File:
|
||||
|
||||
- [credential_guides.py](/home/ray/dev/linkong/planet/backend/app/services/credential_guides.py)
|
||||
|
||||
APIs:
|
||||
|
||||
```http
|
||||
GET /api/v1/settings/credential-guides/{provider}
|
||||
POST /api/v1/settings/credential-guides/{provider}/generate
|
||||
POST /api/v1/settings/credential-guides/{provider}/reset
|
||||
```
|
||||
|
||||
Currently supported:
|
||||
|
||||
- `barentswatch`
|
||||
|
||||
The default guide includes the official BarentsWatch tutorial:
|
||||
|
||||
```text
|
||||
https://developer.barentswatch.no/docs/tutorial
|
||||
```
|
||||
|
||||
If the user clicks that the tutorial is not useful, the backend sends the default prompt to AI Provider, generates a new Chinese tutorial, and saves it to `SystemSetting`:
|
||||
|
||||
```text
|
||||
category = collector_credential_guides
|
||||
```
|
||||
|
||||
Reset deletes the custom tutorial and restores the default guide.
|
||||
|
||||
## Save Rules
|
||||
|
||||
When built-in collector configuration is saved, the internal `connectivity_validation` field is removed so validation state does not mix with user configuration.
|
||||
|
||||
BarentsWatch `client_secret` has special handling:
|
||||
|
||||
- The input shows a masked preview.
|
||||
- If the submitted value still matches the masked preview, the backend keeps the old secret.
|
||||
- If a new value is submitted, the secret is replaced.
|
||||
- The previous separate "clear current secret" checkbox is no longer provided.
|
||||
|
||||
## Test Coverage
|
||||
|
||||
Related tests:
|
||||
|
||||
- [test_vessels.py](/home/ray/dev/linkong/planet/backend/tests/test_vessels.py)
|
||||
|
||||
Added coverage:
|
||||
|
||||
- BarentsWatch credentials can be parsed from `~/.zshrc`.
|
||||
- When environment variables are empty, `resolve_barentswatch_config()` can fall back to `~/.zshrc`.
|
||||
- Vessel data conversion and GeoJSON output remain compatible.
|
||||
|
||||
## Current Provider Coverage
|
||||
|
||||
Credential providers currently supported:
|
||||
|
||||
- `barentswatch`
|
||||
- `spacetrack`
|
||||
|
||||
Other collectors with `requires_credentials=true` return that their credential chain has not been wired yet, and the frontend shows `Unavailable`.
|
||||
Reference in New Issue
Block a user