planet/docs/plans/integration-config-schema-system-plan.md

统一集成配置 Schema 系统计划

Last updated: 2026-05-20

Summary

Planet 的采集器、AI Provider 和工具调用配置需要从“页面各自硬编码字段”收敛到同一套低代码 schema 系统。系统负责两件事：

• 用后台可编辑 schema 生成配置表单。
• 按字段 target 把表单值组装成后端运行时需要的请求、凭证和 JSON 配置。

这套 schema 不替代 target_schema_registry。target_schema_registry 继续负责采集结果映射和校验；本计划中的 integration_config_schemas 负责“怎么配置一个集成”。

Current Problems

• Admin Next 的采集器配置曾把不同凭证形态压成通用 api_key，导致 barentswatch_vessels 这种 OAuth client credentials 丢失 client_id 字段。
• AI Provider、Web Search、OCR 和 DataSource 配置各自维护表单字段、secret 处理和 payload 组装逻辑，重复且容易漂移。
• 新增字段时需要改前端字段列表和保存逻辑，无法做到后台配置化扩展。

Target Model

新增统一 registry：integration_config_schemas，存储在 SystemSetting.payload。

Registry 包含：

• fragments：可复用字段片段，例如 endpoint、API Key、OAuth Client、HTTP 请求、WebSocket 订阅、AI Provider 基础字段、工具超时字段。
• auth_schemas：可复用认证编排，例如 API Key、Bearer Token、Basic、OAuth2 Client Credentials、OAuth2 Authorization Code、Session Cookie Login。
• schemas：具体配置对象使用的 schema，例如 datasource:barentswatch_vessels、ai_provider:minimax、tool:web_search:tavily。
• defaults：每类集成的默认 schema，例如 datasource、ai_provider、tool。

Schema 必须声明 kind：

• datasource
• ai_provider
• tool

字段定义统一使用：

{
  "key": "client_secret",
  "label": "Client Secret",
  "type": "secret",
  "secret": true,
  "target": "auth_config.client_secret",
  "required": true,
  "wide": true
}

字段 target 支持写入：

• DataSource：root.*、auth_config.*、headers.*、config.*
• AI Provider：ai_provider.*、ai_provider.providers.{provider}.*
• Tool：web_search.*、web_search.providers.{provider}.*、ocr.*

Auth Schema Model

认证必须成为 schema 系统的一等能力，不能再把所有凭证强行压成 api_key。每个配置 schema 可以引用一个 auth_schema，也可以内联声明认证编排。

Auth schema 描述：

• 凭证字段：哪些字段是 secret、是否必填、写入哪个 target。
• 凭证来源：DB、env fallback、运行时草稿、用户授权回调。
• 预认证请求：例如登录接口、token endpoint、OAuth callback。
• 凭证注入方式：header、query、form、JSON body、cookie jar、WebSocket subscription payload。
• reveal 策略：管理员可 reveal 并写 audit log；无 DB/env 值时显示空。
• 测试策略：连接测试必须使用当前表单草稿优先，再 fallback 到已保存/env。

v1 需要支持的认证类型：

• none：无认证。
• api_key：API Key 写入 header/query/form/body。
• bearer_token：Bearer token header。
• basic：username/password，支持直接 Basic header 或 provider 特定登录。
• oauth2_client_credentials：client_id/client_secret 换 access_token。
• oauth2_authorization_code：第三方登录授权，包含 authorize URL、callback、token exchange、refresh。
• session_cookie_login：用户名密码登录后保存 cookie jar，再访问数据接口。
• custom_auth_preflight：无法归类时，用声明式 preflight 请求生成后续请求上下文。

Auth schema 示例：

{
  "key": "spacetrack_session",
  "type": "session_cookie_login",
  "fields": [
    {
      "key": "username",
      "label": "Username",
      "target": "auth_config.username",
      "required": true
    },
    {
      "key": "password",
      "label": "Password",
      "type": "secret",
      "secret": true,
      "target": "auth_config.password",
      "required": true
    }
  ],
  "preflight": {
    "method": "POST",
    "url": "https://www.space-track.org/ajaxauth/login",
    "body_type": "form",
    "body": {
      "identity": "{{auth_config.username}}",
      "password": "{{auth_config.password}}"
    },
    "success": {
      "type": "cookie"
    }
  },
  "inject": {
    "type": "cookie_jar"
  }
}

OAuth Authorization Code 示例：

{
  "key": "github_oauth",
  "type": "oauth2_authorization_code",
  "fields": [
    { "key": "client_id", "target": "auth_config.client_id", "required": true },
    { "key": "client_secret", "type": "secret", "secret": true, "target": "auth_config.client_secret", "required": true },
    { "key": "scopes", "type": "tags", "target": "auth_config.scopes" }
  ],
  "authorization": {
    "url": "https://github.com/login/oauth/authorize",
    "client_id": "{{auth_config.client_id}}",
    "scopes": "{{auth_config.scopes}}",
    "redirect_uri": "{{system.callback_base_url}}/api/v1/integrations/oauth/github/callback"
  },
  "token": {
    "method": "POST",
    "url": "https://github.com/login/oauth/access_token",
    "body_type": "form",
    "body": {
      "client_id": "{{auth_config.client_id}}",
      "client_secret": "{{auth_config.client_secret}}",
      "code": "{{oauth.code}}",
      "redirect_uri": "{{oauth.redirect_uri}}"
    }
  },
  "inject": {
    "type": "bearer_header",
    "token_path": "access_token"
  }
}

Request And Runtime Assembly

这套系统本质是面向 Planet 集成的低代码 Postman，但目标不是临时发请求，而是沉淀成可保存、可测试、可调度、可审计的运行配置。

Schema 支持描述：

• HTTP method：GET、POST
• endpoint
• headers
• query params
• JSON body / form body
• auth schema / auth config / preflight auth flow
• WebSocket endpoint 和 subscription payload
• AI Provider 的 provider_api、base_url、model、api_key、service_token
• Tool 的 provider、base_url、api_key、timeout 和工具专属参数

请求执行顺序：

1. 从 schema 字段 target 组装 root/auth_config/headers/config。
2. 如果存在 auth schema，先解析凭证来源和草稿覆盖。
3. 需要 preflight 时执行认证请求，例如 token exchange 或 login。
4. 把认证结果注入正式请求，例如 bearer header、cookie jar、query token。
5. 执行连接测试、采样、采集、AI Provider connect 或 tool connect。

Default Schemas

DataSource

默认采集器：

• endpoint
• method
• headers JSON
• query/body/config JSON
• API Key auth
• advanced JSON

barentswatch_vessels：

• endpoint
• Client ID -> auth_config.client_id
• Client Secret -> auth_config.client_secret
• fixed auth_type = oauth_client

aisstream_vessels：

• WebSocket endpoint
• API Key -> auth_config.api_key
• subscription / bounding boxes config

spacetrack_tle：

• API Base URL / endpoint
• Username -> auth_config.username
• Password -> auth_config.password
• auth schema = session_cookie_login
• login endpoint = https://www.space-track.org/ajaxauth/login
• login body fields：identity / password
• run/test request uses returned session cookie

AI Provider

默认字段：

• provider
• provider_api
• base_url
• model
• api_key
• max_tokens
• anthropic_version
• service_url
• service_token
• timeout_seconds
• retry_attempts
• model_provider_apis

Provider presets supply initial defaults, but the editable schema controls which fields appear and where values are saved.

AI Provider auth variants：

• OpenAI-compatible providers：api_key or bearer_token。
• Local/sidecar services：none、service_token or custom header。
• OAuth-backed providers：oauth2_authorization_code，适用于需要用户授权登录的 provider。
• CLI/session-backed tools such as Codex：优先作为 tool 或本机 runner 集成；如果作为 provider，必须显式声明会话来源、权限边界、不可多用户复用的限制。

Tool

web_search：

• enabled
• provider / default_provider
• base_url
• api_key
• max_results
• timeout_seconds
• endpoint_path
• search_depth
• engine
• include_answer / include_raw_content / include_text
• search_path / scrape_path / scrape_formats

ocr：

• enabled
• provider
• base_url
• api_key
• model
• languages
• timeout_seconds
• max_file_size_mb
• output_format

Tool auth variants：

• GitHub PAT：bearer_token。
• GitHub OAuth App：oauth2_authorization_code，适合用户授权登录和代表用户访问。
• GitHub App：app_installation，需要 app id/private key/installation id，并通过 schema 声明 installation token exchange。
• Browser/session tools：必须显式标记为 session_local_only，不能作为后台多用户稳定凭证。

API Plan

• GET /api/v1/integration-config-schemas
  • Return the full registry.
• PUT /api/v1/integration-config-schemas
  • Save the registry. Admin only.
• POST /api/v1/integration-config-schemas/validate
  • Validate full registry or one schema.
• POST /api/v1/integration-config-schemas/auth/test
  • Test auth schema with draft credentials without saving.
• GET /api/v1/integration-config-schemas/auth/secrets
  • Reveal stored/env-backed secret fields for admins; write audit log.
• POST /api/v1/integrations/oauth/{provider}/start
  • Start OAuth Authorization Code flow.
• GET /api/v1/integrations/oauth/{provider}/callback
  • Complete OAuth callback and store token material according to schema.
• GET /api/v1/datasources/configs/all
  • Add form_schema to each row.
• GET /api/v1/settings/integrations
  • Add form_schema for AI Provider and tools.

Existing save APIs remain compatible:

• DataSource saves to DataSourceConfig.
• AI Provider, Web Search and OCR save to external_integrations.

Secret fields never return plaintext through list/config endpoints. They return configured state and masked preview only.

Secret reveal endpoints return plaintext only on explicit administrator action and must log target, actor, source, result and timestamp. List/config endpoints must never leak secret plaintext.

Frontend Plan

Admin Next extracts a reusable SchemaForm:

• Render fields from form_schema.fields.
• Build payload by writing values to each field target.
• Preserve masked secret semantics: unchanged masked values do not overwrite stored secrets.
• Reveal secrets through a schema-aware reveal action; if neither DB nor env has a value, show an empty editable input.
• Support text, secret, number, boolean, select, textarea, JSON and tags controls.
• Support auth controls for API key, username/password, OAuth connect/disconnect, session cookie login and custom preflight status.
• Validate schema before saving registry changes.

Pages migrated in v1:

• Collection Management / collector configs.
• AI / Provider configuration.
• AI / Tools configuration for Web Search and OCR.

Each detail page gets a schema editing action for admins. The editor saves registry JSON after validation.

Test Plan

• Default registry initializes with datasource, AI Provider and tool schemas.
• Validation rejects duplicate keys, illegal targets, illegal field types and secret plaintext defaults.
• barentswatch_vessels renders and saves client_id / client_secret.
• spacetrack_tle renders username/password, tests with draft credentials, and does not require API Key.
• aisstream_vessels renders API Key and WebSocket subscription fields.
• Session cookie login auth executes preflight before sample/run and uses the resulting cookie jar.
• OAuth Authorization Code schema can start callback flow, store token metadata and inject bearer token.
• Secret reveal returns DB value, env fallback or empty value according to source, and writes audit log.
• Connection tests always prefer current draft credentials over saved/env credentials.
• AI Provider renders and saves provider_api, base_url, model, api_key and service_token.
• AI Provider and tool schemas can reuse the same auth schema primitives as DataSource.
• Web Search and OCR render and save provider-specific tool fields.
• Adding a schema field in the registry makes it appear in Admin Next without frontend code changes.
• Existing connection tests, datasource sampling, datasource run, AI Provider connect/reveal/refresh, and Web Search connect keep working.

Assumptions

• target_schema_registry remains separate because it describes collected result shape, not configuration forms.
• The first implementation stores schema registry in SystemSetting; no new database table is required.
• Old AntD Settings pages stay compatible but are not migrated in v1.
• Tool scope in v1 is Web Search and OCR.