planet/docs/plans/integration-config-schema-system-plan.md

# 统一集成配置 Schema 系统计划

Last updated: 2026-05-20

## Summary

Planet 的采集器、AI Provider 和工具调用配置需要从“页面各自硬编码字段”收敛到同一套低代码 schema 系统。系统负责两件事：

- 用后台可编辑 schema 生成配置表单。
- 按字段 target 把表单值组装成后端运行时需要的请求、凭证和 JSON 配置。

这套 schema 不替代 `target_schema_registry`。`target_schema_registry` 继续负责采集结果映射和校验；本计划中的 `integration_config_schemas` 负责“怎么配置一个集成”。

## Current Problems

- Admin Next 的采集器配置曾把不同凭证形态压成通用 `api_key`，导致 `barentswatch_vessels` 这种 OAuth client credentials 丢失 `client_id` 字段。
- AI Provider、Web Search、OCR 和 DataSource 配置各自维护表单字段、secret 处理和 payload 组装逻辑，重复且容易漂移。
- 新增字段时需要改前端字段列表和保存逻辑，无法做到后台配置化扩展。

## Target Model

新增统一 registry：`integration_config_schemas`，存储在 `SystemSetting.payload`。

Registry 包含：

- `fragments`：可复用字段片段，例如 endpoint、API Key、OAuth Client、HTTP 请求、WebSocket 订阅、AI Provider 基础字段、工具超时字段。
- `auth_schemas`：可复用认证编排，例如 API Key、Bearer Token、Basic、OAuth2 Client Credentials、OAuth2 Authorization Code、Session Cookie Login。
- `schemas`：具体配置对象使用的 schema，例如 `datasource:barentswatch_vessels`、`ai_provider:minimax`、`tool:web_search:tavily`。
- `defaults`：每类集成的默认 schema，例如 `datasource`、`ai_provider`、`tool`。

Schema 必须声明 `kind`：

- `datasource`
- `ai_provider`
- `tool`

字段定义统一使用：

```json
{
  "key": "client_secret",
  "label": "Client Secret",
  "type": "secret",
  "secret": true,
  "target": "auth_config.client_secret",
  "required": true,
  "wide": true
}
```

字段 target 支持写入：

- DataSource：`root.*`、`auth_config.*`、`headers.*`、`config.*`
- AI Provider：`ai_provider.*`、`ai_provider.providers.{provider}.*`
- Tool：`web_search.*`、`web_search.providers.{provider}.*`、`ocr.*`

## Auth Schema Model

认证必须成为 schema 系统的一等能力，不能再把所有凭证强行压成 `api_key`。每个配置 schema 可以引用一个 `auth_schema`，也可以内联声明认证编排。

Auth schema 描述：

- 凭证字段：哪些字段是 secret、是否必填、写入哪个 target。
- 凭证来源：DB、env fallback、运行时草稿、用户授权回调。
- 预认证请求：例如登录接口、token endpoint、OAuth callback。
- 凭证注入方式：header、query、form、JSON body、cookie jar、WebSocket subscription payload。
- reveal 策略：管理员可 reveal 并写 audit log；无 DB/env 值时显示空。
- 测试策略：连接测试必须使用当前表单草稿优先，再 fallback 到已保存/env。

v1 需要支持的认证类型：

- `none`：无认证。
- `api_key`：API Key 写入 header/query/form/body。
- `bearer_token`：Bearer token header。
- `basic`：username/password，支持直接 Basic header 或 provider 特定登录。
- `oauth2_client_credentials`：client_id/client_secret 换 access_token。
- `oauth2_authorization_code`：第三方登录授权，包含 authorize URL、callback、token exchange、refresh。
- `session_cookie_login`：用户名密码登录后保存 cookie jar，再访问数据接口。
- `custom_auth_preflight`：无法归类时，用声明式 preflight 请求生成后续请求上下文。

Auth schema 示例：

```json
{
  "key": "spacetrack_session",
  "type": "session_cookie_login",
  "fields": [
    {
      "key": "username",
      "label": "Username",
      "target": "auth_config.username",
      "required": true
    },
    {
      "key": "password",
      "label": "Password",
      "type": "secret",
      "secret": true,
      "target": "auth_config.password",
      "required": true
    }
  ],
  "preflight": {
    "method": "POST",
    "url": "https://www.space-track.org/ajaxauth/login",
    "body_type": "form",
    "body": {
      "identity": "{{auth_config.username}}",
      "password": "{{auth_config.password}}"
    },
    "success": {
      "type": "cookie"
    }
  },
  "inject": {
    "type": "cookie_jar"
  }
}
```

OAuth Authorization Code 示例：

```json
{
  "key": "github_oauth",
  "type": "oauth2_authorization_code",
  "fields": [
    { "key": "client_id", "target": "auth_config.client_id", "required": true },
    { "key": "client_secret", "type": "secret", "secret": true, "target": "auth_config.client_secret", "required": true },
    { "key": "scopes", "type": "tags", "target": "auth_config.scopes" }
  ],
  "authorization": {
    "url": "https://github.com/login/oauth/authorize",
    "client_id": "{{auth_config.client_id}}",
    "scopes": "{{auth_config.scopes}}",
    "redirect_uri": "{{system.callback_base_url}}/api/v1/integrations/oauth/github/callback"
  },
  "token": {
    "method": "POST",
    "url": "https://github.com/login/oauth/access_token",
    "body_type": "form",
    "body": {
      "client_id": "{{auth_config.client_id}}",
      "client_secret": "{{auth_config.client_secret}}",
      "code": "{{oauth.code}}",
      "redirect_uri": "{{oauth.redirect_uri}}"
    }
  },
  "inject": {
    "type": "bearer_header",
    "token_path": "access_token"
  }
}
```

## Request And Runtime Assembly

这套系统本质是面向 Planet 集成的低代码 Postman，但目标不是临时发请求，而是沉淀成可保存、可测试、可调度、可审计的运行配置。

Schema 支持描述：

- HTTP method：`GET`、`POST`
- endpoint
- headers
- query params
- JSON body / form body
- auth schema / auth config / preflight auth flow
- WebSocket endpoint 和 subscription payload
- AI Provider 的 `provider_api`、`base_url`、`model`、`api_key`、`service_token`
- Tool 的 provider、base_url、api_key、timeout 和工具专属参数

请求执行顺序：

1. 从 schema 字段 target 组装 root/auth_config/headers/config。
2. 如果存在 auth schema，先解析凭证来源和草稿覆盖。
3. 需要 preflight 时执行认证请求，例如 token exchange 或 login。
4. 把认证结果注入正式请求，例如 bearer header、cookie jar、query token。
5. 执行连接测试、采样、采集、AI Provider connect 或 tool connect。

## Default Schemas

### DataSource

默认采集器：

- endpoint
- method
- headers JSON
- query/body/config JSON
- API Key auth
- advanced JSON

`barentswatch_vessels`：

- endpoint
- Client ID -> `auth_config.client_id`
- Client Secret -> `auth_config.client_secret`
- fixed `auth_type = oauth_client`

`aisstream_vessels`：

- WebSocket endpoint
- API Key -> `auth_config.api_key`
- subscription / bounding boxes config

`spacetrack_tle`：

- API Base URL / endpoint
- Username -> `auth_config.username`
- Password -> `auth_config.password`
- auth schema = `session_cookie_login`
- login endpoint = `https://www.space-track.org/ajaxauth/login`
- login body fields：`identity` / `password`
- run/test request uses returned session cookie

### AI Provider

默认字段：

- provider
- provider_api
- base_url
- model
- api_key
- max_tokens
- anthropic_version
- service_url
- service_token
- timeout_seconds
- retry_attempts
- model_provider_apis

Provider presets supply initial defaults, but the editable schema controls which fields appear and where values are saved.

AI Provider auth variants：

- OpenAI-compatible providers：`api_key` or `bearer_token`。
- Local/sidecar services：`none`、`service_token` or custom header。
- OAuth-backed providers：`oauth2_authorization_code`，适用于需要用户授权登录的 provider。
- CLI/session-backed tools such as Codex：优先作为 `tool` 或本机 runner 集成；如果作为 provider，必须显式声明会话来源、权限边界、不可多用户复用的限制。

### Tool

`web_search`：

- enabled
- provider / default_provider
- base_url
- api_key
- max_results
- timeout_seconds
- endpoint_path
- search_depth
- engine
- include_answer / include_raw_content / include_text
- search_path / scrape_path / scrape_formats

`ocr`：

- enabled
- provider
- base_url
- api_key
- model
- languages
- timeout_seconds
- max_file_size_mb
- output_format

Tool auth variants：

- GitHub PAT：`bearer_token`。
- GitHub OAuth App：`oauth2_authorization_code`，适合用户授权登录和代表用户访问。
- GitHub App：`app_installation`，需要 app id/private key/installation id，并通过 schema 声明 installation token exchange。
- Browser/session tools：必须显式标记为 `session_local_only`，不能作为后台多用户稳定凭证。

## API Plan

- `GET /api/v1/integration-config-schemas`
  - Return the full registry.
- `PUT /api/v1/integration-config-schemas`
  - Save the registry. Admin only.
- `POST /api/v1/integration-config-schemas/validate`
  - Validate full registry or one schema.
- `POST /api/v1/integration-config-schemas/auth/test`
  - Test auth schema with draft credentials without saving.
- `GET /api/v1/integration-config-schemas/auth/secrets`
  - Reveal stored/env-backed secret fields for admins; write audit log.
- `POST /api/v1/integrations/oauth/{provider}/start`
  - Start OAuth Authorization Code flow.
- `GET /api/v1/integrations/oauth/{provider}/callback`
  - Complete OAuth callback and store token material according to schema.
- `GET /api/v1/datasources/configs/all`
  - Add `form_schema` to each row.
- `GET /api/v1/settings/integrations`
  - Add `form_schema` for AI Provider and tools.

Existing save APIs remain compatible:

- DataSource saves to `DataSourceConfig`.
- AI Provider, Web Search and OCR save to `external_integrations`.

Secret fields never return plaintext through list/config endpoints. They return configured state and masked preview only.

Secret reveal endpoints return plaintext only on explicit administrator action and must log target, actor, source, result and timestamp. List/config endpoints must never leak secret plaintext.

## Frontend Plan

Admin Next extracts a reusable `SchemaForm`:

- Render fields from `form_schema.fields`.
- Build payload by writing values to each field `target`.
- Preserve masked secret semantics: unchanged masked values do not overwrite stored secrets.
- Reveal secrets through a schema-aware reveal action; if neither DB nor env has a value, show an empty editable input.
- Support text, secret, number, boolean, select, textarea, JSON and tags controls.
- Support auth controls for API key, username/password, OAuth connect/disconnect, session cookie login and custom preflight status.
- Validate schema before saving registry changes.

Pages migrated in v1:

- Collection Management / collector configs.
- AI / Provider configuration.
- AI / Tools configuration for Web Search and OCR.

Each detail page gets a schema editing action for admins. The editor saves registry JSON after validation.

## Test Plan

- Default registry initializes with datasource, AI Provider and tool schemas.
- Validation rejects duplicate keys, illegal targets, illegal field types and secret plaintext defaults.
- `barentswatch_vessels` renders and saves `client_id` / `client_secret`.
- `spacetrack_tle` renders username/password, tests with draft credentials, and does not require API Key.
- `aisstream_vessels` renders API Key and WebSocket subscription fields.
- Session cookie login auth executes preflight before sample/run and uses the resulting cookie jar.
- OAuth Authorization Code schema can start callback flow, store token metadata and inject bearer token.
- Secret reveal returns DB value, env fallback or empty value according to source, and writes audit log.
- Connection tests always prefer current draft credentials over saved/env credentials.
- AI Provider renders and saves `provider_api`, `base_url`, `model`, `api_key` and `service_token`.
- AI Provider and tool schemas can reuse the same auth schema primitives as DataSource.
- Web Search and OCR render and save provider-specific tool fields.
- Adding a schema field in the registry makes it appear in Admin Next without frontend code changes.
- Existing connection tests, datasource sampling, datasource run, AI Provider connect/reveal/refresh, and Web Search connect keep working.

## Assumptions

- `target_schema_registry` remains separate because it describes collected result shape, not configuration forms.
- The first implementation stores schema registry in `SystemSetting`; no new database table is required.
- Old AntD Settings pages stay compatible but are not migrated in v1.
- Tool scope in v1 is Web Search and OCR.