diff --git a/.claude/commands/cleanup.md b/.claude/commands/cleanup.md index 68c17323..97e0d0b1 100644 --- a/.claude/commands/cleanup.md +++ b/.claude/commands/cleanup.md @@ -12,6 +12,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"] 若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。 +## 节省上下文规则 + +优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文： + +```bash +git diff --name-only HEAD +git diff --unified=0 HEAD -- +git diff --check +rg -n "TODO|FIXME|console\.log|debugger|print\(" +``` + +只有 focused diff 不足以安全判断或修改时，才读取完整文件。 + ## 审查清单 按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）： @@ -59,8 +72,13 @@ git diff HEAD --name-only ### Step 2 — 逐文件阅读并分析 -- 用 Read 工具读取完整文件（不只读 diff） -- 对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式 +先从 focused diff 开始： + +```bash +git diff --unified=0 HEAD -- +``` + +用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。 ### Step 3 — 报告问题清单 @@ -95,6 +113,7 @@ git diff HEAD --name-only - 只改在审查清单中发现的问题，不做额外优化 - 每次 Edit 只修改确实有问题的行，保持 diff 最小 - 改完后用 `grep` 验证旧的坏代码已消失 +- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具 ### Step 5 — 输出总结 diff --git a/.claude/commands/docs.md b/.claude/commands/docs.md new file mode 100644 index 00000000..bb776ac8 --- /dev/null +++ b/.claude/commands/docs.md @@ -0,0 +1,151 @@ +--- +description: 分析本次 git 变更，在 docs/technical/zh/ 中新建或更新对应的技术文档 +argument-hint: 可选：指定要记录的主题，或留空自动从 git diff 推断 +allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"] +--- + +# /docs — 技术文档写入工作流 + +## 目标 + +根据当前 git 变更（或用户指定主题）在 `docs/technical/zh/` 中写入或更新技术文档，记录**为什么**这样做，而不只是记录做了什么。 + +## 执行步骤 + +### Step 1 — 理解变更范围 + +```bash +git diff HEAD --stat # 变更文件一览 +git diff HEAD --name-only # 变更文件列表 +git log --oneline -10 # 近期 commit 上下文 +``` + +若 `$ARGUMENTS` 指定了主题，优先聚焦该主题；否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff；只对决定文档主题所需的文件读取 focused diff： + +```bash +git diff HEAD -- +rg -n "class |def |function |export |router|@router|interface |type " +``` + +### Step 2 — 确认文档范围 + +分析变更，判断： + +1. **应写几篇文档**：单一主题写一篇，跨领域变更可拆分（如后端性能优化 + 运维启动脚本分开写） +2. **是新建还是更新**：检查 `docs/technical/zh/` 中是否已有相关文档 +3. **文档命名**：按 `领域-主题-副题.md` 格式，全小写，用连字符，如： + - `backend-datasources-api-performance.md` + - `ops-planet-sh-startup.md` + - `earth-bgp-context.md` + +```bash +ls docs/technical/zh/ # 查看现有文档 +``` + +**先输出写作计划供用户确认**（若变更明确且范围小，可直接执行）： + +``` +文档计划： + 新建：docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化 + 更新：docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节 +``` + +### Step 3 — 写文档 + +遵循以下原则： + +**记录 WHY，不只记录 WHAT** +- 好：`将戳文件从 /tmp 移到 ~/.cache/planet/，因为 WSL 重启后 /tmp 被清空` +- 差：`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值` + +**必须包含的内容**： +- 背景/问题：改动之前存在什么问题，为什么要改 +- 核心设计决策及其理由 +- 关键代码片段（用 diff 或 before/after 展示） +- 相关文件列表 + +**格式要求**： +- 使用 `##` 和 `###` 分级，不要超过三级 +- 代码块注明语言（python / bash / typescript / sql） +- 表格用于对比多个选项或列出参数 +- 中文写作，技术术语保留英文原文 +- `docs/technical/zh/` 中的文档不得用英文原文占位；如果存在 `docs/technical/en/` 对应文件，禁止逐字复制成中文文件 +- 中文文档内部链接应指向 `docs/technical/zh/...`，除非明确引用英文专属文档 + +**文档结构模板**： + +```markdown +# 标题（说明做了什么） + +## 背景 + +为什么要做这个改动，改动前存在什么问题。 + +## 核心变更 + +### 子主题一 + +before/after 或决策说明 + 关键代码 + +### 子主题二 + +... + +## 相关文件 + +- `path/to/file.py` — 简短说明 +``` + +### Step 4 — 验证 + +- 读一遍写好的文档，确认逻辑清晰、代码片段无明显错误 +- 用 `rg --files` 或 `test -e` 确认文档中的文件路径在项目中真实存在，避免凭记忆判断： +- 检查中文文档没有误复制英文版： + +```bash +python - <<'PY' +from pathlib import Path +same = [] +for en in sorted(Path("docs/technical/en").glob("*.md")): + zh = Path("docs/technical/zh") / en.name + if zh.exists() and en.read_text() == zh.read_text(): + same.append(en.name) +if same: + raise SystemExit("identical en/zh docs: " + ", ".join(same)) +print("no identical en/zh docs") +PY +``` + +- 检查中文文档内部链接没有继续指向无语言目录： + +```bash +rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2 +``` + +```bash +# 对文档中提到的关键路径做快速验证 +ls +``` + +如需检查大量链接，优先用确定性提取： + +```bash +rg -n "\]\(([^)]+)\)" docs/technical/zh/.md +``` + +### Step 5 — 完成确认 + +输出摘要： + +``` +✓ 新建：docs/technical/zh/ops-planet-sh-startup.md（约 xxx 字） +✓ 更新：docs/technical/zh/backend-datasources-api-performance.md +``` + +## 注意事项 + +- 不要写流水账式的"改了 A、改了 B、改了 C"，要写改动背后的约束和权衡 +- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效 +- 代码片段保持简洁，只保留说明问题的关键部分，省略无关样板代码 +- 如果某个变更已有文档记录，优先在原文档中追加，而不是新建 +- 文档是给未来的开发者看的，假设读者熟悉项目但不了解这次改动的背景 diff --git a/.claude/commands/goal-driven.md b/.claude/commands/goal-driven.md index bfae193c..4757a94c 100644 --- a/.claude/commands/goal-driven.md +++ b/.claude/commands/goal-driven.md @@ -72,6 +72,8 @@ Verification ## 执行风格 - 重证据，轻口头判断 +- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- `、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式 +- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据 - 重验收，轻自我感觉 - 优先用测试、日志、产物、对比结果来证明完成 - 对长期任务保持“未达标就继续”的节奏 diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 3708ccd2..43ff3813 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -28,6 +28,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"] - `docs/CHANGELOG.md` - `docs/version-history.md` +## 节省上下文规则 + +发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取： + +```bash +git status --short +git diff --stat HEAD +git diff --name-only HEAD +rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md +``` + +除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。 + ## 执行步骤 ### Step 1 — 环境检查 @@ -45,7 +58,7 @@ cat VERSION # 读取当前版本 ### Step 2 — 确定发版类型与新版本号 - 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用 -- 否则根据当前 `git diff HEAD` 和 `git log` 推断 +- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断 - 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`） - **先输出发版计划供用户确认**： @@ -91,12 +104,13 @@ cat VERSION # 读取当前版本 针对本次变更范围做最小验证： -- Python 文件有修改：`python3 -m py_compile ` -- Frontend 文件有修改：运行项目标准检查（若无则跳过并说明） +- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile ` +- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明） - 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致 ```bash -grep -h "version" VERSION frontend/package.json pyproject.toml +cat VERSION +rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock ``` ### Step 7 — 提交前预览 diff --git a/.codex/skills/cleanup/SKILL.md b/.codex/skills/cleanup/SKILL.md index 07359c6c..778c16e8 100644 --- a/.codex/skills/cleanup/SKILL.md +++ b/.codex/skills/cleanup/SKILL.md @@ -21,6 +21,19 @@ If the user specifies a file or directory, check only that. Otherwise check all Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code. +## Token-Saving Rule + +Prefer deterministic CLI checks before reading files into model context: + +```bash +git diff --name-only HEAD +git diff --unified=0 HEAD -- +git diff --check +rg -n "TODO|FIXME|console\.log|debugger|print\(" +``` + +Read full files only when the focused diff does not provide enough surrounding context to make a safe edit. + ## Checklist ### 1. Duplicate Logic @@ -64,7 +77,13 @@ Filter to the user-specified path if one was provided. ### Step 2 — Read and analyze each file -Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix. +Start with focused diffs: + +```bash +git diff --unified=0 HEAD -- +``` + +Use `rg`, `git diff --check`, and compiler/linter output for deterministic findings. Read the full file only for files that need surrounding context. For each issue found, record filename, line number, category, and suggested fix. ### Step 3 — Report findings before touching anything @@ -99,6 +118,7 @@ Principles: - Only fix issues identified in the checklist — no extra improvements - Keep each Edit as small as possible - After fixing, verify the old bad pattern is gone with grep +- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type ### Step 5 — Summary diff --git a/.codex/skills/docs/SKILL.md b/.codex/skills/docs/SKILL.md new file mode 100644 index 00000000..182322bb --- /dev/null +++ b/.codex/skills/docs/SKILL.md @@ -0,0 +1,117 @@ +--- +name: docs +description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex. +--- + +# Docs + +Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`. + +## Goal + +Write or update technical docs that explain why a change exists, not only what files changed. + +Default target directory: + +- `docs/technical/zh/` + +## Workflow + +1. Gather change context: + +```bash +git diff HEAD --stat +git diff HEAD --name-only +git log --oneline -10 +ls docs/technical/zh/ +``` + +If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic: + +```bash +git diff HEAD -- +rg -n "class |def |function |export |router|@router|interface |type " +``` + +2. Decide document scope: + +- Use one document for one coherent topic. +- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior. +- Prefer updating an existing relevant doc over creating a duplicate. +- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example: + - `backend-datasources-api-performance.md` + - `ops-planet-sh-startup.md` + - `earth-bgp-context.md` + +3. Write the doc in Chinese: + +- Write Chinese prose for `docs/technical/zh/`. +- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate. +- Use `##` and `###` headings; avoid going deeper than three levels. +- Use fenced code blocks with language tags. +- Use tables when comparing options or listing parameters. + +4. Required content: + +- Background/problem: what was wrong before and why the change was needed. +- Core design decisions and rationale. +- Key code snippets, preferably before/after or focused excerpts. +- Related files and what each file contributes. + +5. Verification: + +- Read the completed doc and check that the reasoning is clear. +- Verify important referenced paths exist. +- Use `rg --files` or `test -e` for path existence instead of relying on memory. +- Run a quick duplicate-language check when editing bilingual docs: + +```bash +python - <<'PY' +from pathlib import Path +same = [] +for en in sorted(Path("docs/technical/en").glob("*.md")): + zh = Path("docs/technical/zh") / en.name + if zh.exists() and en.read_text() == zh.read_text(): + same.append(en.name) +if same: + raise SystemExit("identical en/zh docs: " + ", ".join(same)) +print("no identical en/zh docs") +PY +``` + +Also check that Chinese docs do not link to the old language-less technical docs path: + +```bash +rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2 +``` + +This command should return no matches. + +If checking many links, prefer deterministic extraction: + +```bash +rg -n "\]\(([^)]+)\)" docs/technical/zh/.md +``` + +## Hard Constraints + +- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder. +- Do not leave a Chinese doc with only an English title and English first-screen content. +- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`. +- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file. +- Do not reference PR numbers, issue numbers, or the current conversation. +- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes. +- Keep code snippets concise and relevant. + +## Recommended Output + +After editing, summarize: + +```md +Updated: +- docs/technical/zh/example.md — what changed + +Verified: +- no identical en/zh docs +- no language-less docs/technical links in zh docs +``` diff --git a/.codex/skills/goal-driven/SKILL.md b/.codex/skills/goal-driven/SKILL.md index 443b9d15..4f410deb 100755 --- a/.codex/skills/goal-driven/SKILL.md +++ b/.codex/skills/goal-driven/SKILL.md @@ -72,6 +72,8 @@ In Codex, only use actual subagents when the user explicitly asks for delegation ## Operating Rules - Prefer objective checks over self-reported completion. +- Prefer deterministic tool evidence over long model summaries: use `rg`, `git diff --stat`, targeted `git diff -- `, tests, builds, linters, `curl`, or database queries when they can prove a criterion. +- Do not paste large command output into the conversation; summarize the evidence and keep raw output in tool calls. - Do not confuse progress with completion. - If the worker says "done", verify it. - If verification fails, continue from the gap instead of restarting blindly. diff --git a/.codex/skills/release/SKILL.md b/.codex/skills/release/SKILL.md index 070903d1..e5eff680 100644 --- a/.codex/skills/release/SKILL.md +++ b/.codex/skills/release/SKILL.md @@ -35,6 +35,19 @@ Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative - `docs/CHANGELOG.md` - `docs/version-history.md` +## Token-Saving Rule + +Release work should be driven by deterministic CLI evidence. Prefer compact commands and targeted file reads: + +```bash +git status --short +git diff --stat HEAD +git diff --name-only HEAD +rg -n "version|^## |^Released:|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md +``` + +Do not inspect full diffs unless deciding whether changed code belongs in the release. + ## Workflow ### Step 1 — Environment check @@ -52,7 +65,7 @@ If unrelated uncommitted changes exist, list them and ask the user whether to in ### Step 2 — Determine release type and next version - If the user provided an explicit type (`feature` / `bugfix`), use it -- Otherwise infer from `git diff HEAD` and recent `git log` +- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log` - Compute the next version: - `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`) - `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`) @@ -106,12 +119,13 @@ Get today's date with `date +%Y-%m-%d`. Run the smallest relevant validation for the changes in scope: -- Python files changed: `python3 -m py_compile ` -- Frontend files changed: run the project-standard check if available; otherwise skip and say so +- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile ` +- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so - Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version ```bash -grep -h "version" VERSION frontend/package.json pyproject.toml +cat VERSION +rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock ``` ### Step 7 — Pre-commit preview diff --git a/VERSION b/VERSION index f386df22..8298bb08 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.2 +0.43.0 diff --git a/aiprovider/main.py b/aiprovider/main.py index afcc71af..b35670b9 100644 --- a/aiprovider/main.py +++ b/aiprovider/main.py @@ -37,8 +37,26 @@ def verify_service_token(x_provider_token: str | None = Header(default=None)) -> ) -def get_provider_service() -> ProviderService: - return ProviderService() +def get_provider_service( + x_ai_provider: str | None = Header(default=None), + x_ai_provider_api: str | None = Header(default=None), + x_ai_base_url: str | None = Header(default=None), + x_ai_api_key: str | None = Header(default=None), + x_ai_model: str | None = Header(default=None), + x_ai_max_tokens: str | None = Header(default=None), + x_ai_anthropic_version: str | None = Header(default=None), +) -> ProviderService: + overrides = { + "provider": x_ai_provider, + "provider_api": x_ai_provider_api, + "base_url": x_ai_base_url, + "api_key": x_ai_api_key, + "model": x_ai_model, + "anthropic_version": x_ai_anthropic_version, + } + if x_ai_max_tokens: + overrides["max_tokens"] = x_ai_max_tokens + return ProviderService({key: value for key, value in overrides.items() if value not in (None, "")}) @app.get("/health") diff --git a/aiprovider/provider_service.py b/aiprovider/provider_service.py index 282506da..19f75925 100644 --- a/aiprovider/provider_service.py +++ b/aiprovider/provider_service.py @@ -46,19 +46,22 @@ def _resolve_provider_api(provider: str, configured_api: str) -> str: class ProviderService: - def __init__(self) -> None: - self.provider = _normalize_provider(settings.AI_PROVIDER) + def __init__(self, overrides: dict[str, Any] | None = None) -> None: + overrides = overrides or {} + self.provider = _normalize_provider(overrides.get("provider") or settings.AI_PROVIDER) self.provider_api = _resolve_provider_api( self.provider, - _normalize_provider_api(settings.AI_PROVIDER_API), + _normalize_provider_api(overrides.get("provider_api") or settings.AI_PROVIDER_API), ) - self.base_url = settings.AI_BASE_URL.rstrip("/") - self.api_key = settings.AI_API_KEY - self.default_model = settings.AI_MODEL + self.base_url = str(overrides.get("base_url") or settings.AI_BASE_URL).rstrip("/") + self.api_key = str(overrides.get("api_key") or settings.AI_API_KEY) + self.default_model = str(overrides.get("model") or settings.AI_MODEL) self.timeout = settings.AI_TIMEOUT_SECONDS self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1) - self.max_tokens = settings.AI_MAX_TOKENS - self.anthropic_version = settings.AI_ANTHROPIC_VERSION + self.max_tokens = int(overrides.get("max_tokens") or settings.AI_MAX_TOKENS) + self.anthropic_version = str( + overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION + ) self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT def get_status(self) -> AIProviderStatusResponse: diff --git a/backend/app/api/v1/datasource_config.py b/backend/app/api/v1/datasource_config.py index edb72286..cb3e2952 100644 --- a/backend/app/api/v1/datasource_config.py +++ b/backend/app/api/v1/datasource_config.py @@ -1,20 +1,34 @@ """DataSourceConfig API for user-defined data sources""" -from typing import Optional +from typing import Any, Optional from datetime import datetime import base64 +import json +import re from fastapi import APIRouter, Depends, HTTPException, status from sqlalchemy import select, func from sqlalchemy.ext.asyncio import AsyncSession from pydantic import BaseModel, Field import httpx +from app.core.target_schema_registry import get_target_schema, list_target_schemas from app.db.session import get_db from app.models.user import User from app.models.datasource_config import DataSourceConfig +from app.models.datasource_mapping import DataSourceMappingTemplate from app.core.security import get_current_user from app.core.cache import cache from app.core.time import to_iso8601_utc +from app.schemas.ai import SituationalAnalysisRequest +from app.services.ai_client import AIProviderClient, get_ai_provider_client +from app.services.datasource_mapping import ( + MappingError, + build_heuristic_mapping, + execute_mapping, + persist_mapped_records, + redact_for_llm, + stable_payload_hash, +) router = APIRouter() @@ -59,6 +73,44 @@ class DataSourceConfigResponse(BaseModel): from_attributes = True +class CustomSampleRequest(BaseModel): + datasource_config_id: Optional[int] = None + config: Optional[DataSourceConfigCreate] = None + limit_bytes: int = Field(default=200000, ge=1000, le=1000000) + + +class MappingProposeRequest(BaseModel): + sample_payload: Any + target_schema: str + use_ai: bool = True + + +class MappingPreviewRequest(BaseModel): + sample_payload: Any + target_schema: str + mapping_json: dict + limit: int = Field(default=20, ge=1, le=100) + + +class MappingTemplateCreate(BaseModel): + datasource_config_id: int + target_schema: str + mapping_json: dict + sample_payload: Any | None = None + sample_payload_hash: Optional[str] = None + validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$") + is_active: bool = False + + +class MappingTemplateUpdate(BaseModel): + target_schema: Optional[str] = None + mapping_json: Optional[dict] = None + sample_payload: Any | None = None + sample_payload_hash: Optional[str] = None + validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$") + is_active: Optional[bool] = None + + async def test_endpoint( endpoint: str, auth_type: str, @@ -96,6 +148,134 @@ async def test_endpoint( } +def _build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]: + request_headers = {str(key): str(value) for key, value in (headers or {}).items()} + auth_type = str(auth_type or "none").lower() + auth_config = auth_config or {} + + if auth_type == "bearer" and auth_config.get("token"): + request_headers["Authorization"] = f"Bearer {auth_config['token']}" + elif auth_type == "api_key" and auth_config.get("api_key"): + location = str(auth_config.get("in") or auth_config.get("location") or "header").lower() + if location != "query": + key_name = auth_config.get("key_name", "X-API-Key") + request_headers[str(key_name)] = str(auth_config["api_key"]) + elif auth_type == "basic": + username = auth_config.get("username", "") + password = auth_config.get("password", "") + credentials = f"{username}:{password}" + encoded = base64.b64encode(credentials.encode()).decode() + request_headers["Authorization"] = f"Basic {encoded}" + return request_headers + + +def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]: + params = {} + candidate = (config or {}).get("params") or (config or {}).get("query_params") + if isinstance(candidate, dict): + params.update(candidate) + + auth_type = str(auth_type or "none").lower() + auth_config = auth_config or {} + if auth_type == "api_key" and auth_config.get("api_key"): + location = str(auth_config.get("in") or auth_config.get("location") or "header").lower() + if location == "query": + key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key" + params[str(key_name)] = auth_config["api_key"] + return params + + +async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any: + request_config = config.config or {} + method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper() + if method not in {"GET", "POST"}: + raise HTTPException(status_code=400, detail="Only GET and POST sample requests are supported.") + + headers = _build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {}) + params = _build_query_params(config.auth_type, config.auth_config or {}, request_config) + timeout = float(request_config.get("timeout", 30)) + json_body = request_config.get("json_body") + if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}: + candidate = request_config.get("body") + if isinstance(candidate, (dict, list)): + json_body = candidate + + async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client: + response = await client.request( + method, + config.endpoint, + headers=headers, + params=params or None, + json=json_body, + ) + response.raise_for_status() + content = response.content[:limit_bytes] + if "application/json" in response.headers.get("content-type", ""): + return json.loads(content.decode(response.encoding or "utf-8")) + return {"text": content.decode(response.encoding or "utf-8", errors="replace")} + + +def _parse_mapping_from_ai_text(content: str) -> dict[str, Any] | None: + if not content: + return None + + candidates = [content] + fenced = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", content, flags=re.DOTALL) + candidates = fenced + candidates + for candidate in candidates: + try: + parsed = json.loads(candidate) + except json.JSONDecodeError: + continue + if isinstance(parsed, dict) and isinstance(parsed.get("fields"), dict): + return parsed + return None + + +async def _get_config_for_sample( + payload: CustomSampleRequest, + db: AsyncSession, +) -> DataSourceConfig: + if payload.datasource_config_id is not None: + result = await db.execute( + select(DataSourceConfig).where(DataSourceConfig.id == payload.datasource_config_id) + ) + config = result.scalar_one_or_none() + if not config: + raise HTTPException(status_code=404, detail="Configuration not found") + return config + + if payload.config is None: + raise HTTPException(status_code=400, detail="datasource_config_id or config is required") + + config_data = payload.config + return DataSourceConfig( + name=config_data.name, + description=config_data.description, + source_type=config_data.source_type, + endpoint=config_data.endpoint, + auth_type=config_data.auth_type, + auth_config=config_data.auth_config, + headers=config_data.headers, + config=config_data.config, + ) + + +def serialize_mapping_template(template: DataSourceMappingTemplate) -> dict[str, Any]: + return { + "id": template.id, + "datasource_config_id": template.datasource_config_id, + "target_schema": template.target_schema, + "mapping_json": template.mapping_json, + "sample_payload_hash": template.sample_payload_hash, + "validation_status": template.validation_status, + "version": template.version, + "is_active": template.is_active, + "created_at": to_iso8601_utc(template.created_at), + "updated_at": to_iso8601_utc(template.updated_at), + } + + @router.get("/configs") async def list_configs( active_only: bool = False, @@ -345,3 +525,311 @@ async def list_all_datasources( ) return {"total": len(result), "data": result} + + +@router.post("/custom/sample") +async def fetch_custom_sample( + payload: CustomSampleRequest, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + """Fetch a sample payload for a saved or draft custom data source.""" + config = await _get_config_for_sample(payload, db) + try: + sample = await fetch_custom_sample_from_config(config, payload.limit_bytes) + except httpx.HTTPStatusError as exc: + raise HTTPException( + status_code=exc.response.status_code, + detail=f"Sample request failed: HTTP {exc.response.status_code}", + ) from exc + except httpx.HTTPError as exc: + raise HTTPException(status_code=502, detail=f"Sample request failed: {exc}") from exc + + return { + "success": True, + "sample_payload": sample, + "sample_payload_hash": stable_payload_hash(sample), + "redacted_preview": redact_for_llm(sample), + } + + +@router.get("/target-schemas") +async def get_datasource_target_schemas( + current_user: User = Depends(get_current_user), +): + """List target schemas available for custom datasource mapping.""" + return {"data": list_target_schemas()} + + +@router.post("/mappings/propose") +async def propose_datasource_mapping( + payload: MappingProposeRequest, + current_user: User = Depends(get_current_user), + ai_client: AIProviderClient = Depends(get_ai_provider_client), +): + """Generate a mapping draft for a sample payload and target schema.""" + schema = get_target_schema(payload.target_schema) + redacted_sample = redact_for_llm(payload.sample_payload) + fallback_mapping = build_heuristic_mapping(redacted_sample, payload.target_schema) + + ai_error: str | None = None + mapping = fallback_mapping + generated_by = "heuristic" + if payload.use_ai: + try: + response = await ai_client.analyze( + SituationalAnalysisRequest( + title=f"Generate datasource mapping for {schema.key}", + objective=( + "Return only JSON for a deterministic mapping DSL. " + "The JSON must contain source.items_path and fields. " + "Do not include prose or code." + ), + context={ + "target_schema": schema.to_dict(), + "sample_payload": redacted_sample, + "mapping_dsl_example": fallback_mapping, + }, + observations=[ + "Use JSONPath-like paths beginning with $.", + "Never generate executable code.", + "Use field types from the target schema.", + ], + constraints=[ + "Return a single JSON object.", + "Do not include credentials or secrets.", + "Mark uncertain optional fields with default null.", + ], + ) + ) + parsed = _parse_mapping_from_ai_text(response.content) + if parsed: + mapping = parsed + generated_by = "ai_provider" + else: + ai_error = "AI provider did not return a valid mapping JSON object." + except HTTPException as exc: + ai_error = str(exc.detail) + + mapping.setdefault("meta", {}) + if isinstance(mapping["meta"], dict): + mapping["meta"].update( + { + "generated_by": generated_by, + "requires_review": True, + "ai_error": ai_error, + } + ) + + return { + "target_schema": schema.to_dict(), + "mapping_json": mapping, + "sample_payload_hash": stable_payload_hash(payload.sample_payload), + "redacted_sample_payload": redacted_sample, + } + + +@router.post("/mappings/preview") +async def preview_datasource_mapping( + payload: MappingPreviewRequest, + current_user: User = Depends(get_current_user), +): + """Preview deterministic mapping output for a sample payload.""" + try: + preview = execute_mapping( + payload.sample_payload, + payload.mapping_json, + payload.target_schema, + limit=payload.limit, + ) + except (MappingError, ValueError) as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + + return { + "success": preview["failed_count"] == 0, + "preview": preview, + "sample_payload_hash": stable_payload_hash(payload.sample_payload), + } + + +@router.get("/mappings") +async def list_datasource_mappings( + datasource_config_id: Optional[int] = None, + active_only: bool = False, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + """List saved mapping templates.""" + query = select(DataSourceMappingTemplate).order_by( + DataSourceMappingTemplate.datasource_config_id, + DataSourceMappingTemplate.version.desc(), + ) + if datasource_config_id is not None: + query = query.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id) + if active_only: + query = query.where(DataSourceMappingTemplate.is_active.is_(True)) + + result = await db.execute(query) + mappings = result.scalars().all() + return {"total": len(mappings), "data": [serialize_mapping_template(item) for item in mappings]} + + +@router.post("/mappings") +async def create_datasource_mapping( + payload: MappingTemplateCreate, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + """Save a mapping template for a datasource config.""" + get_target_schema(payload.target_schema) + datasource = await db.get(DataSourceConfig, payload.datasource_config_id) + if not datasource: + raise HTTPException(status_code=404, detail="Configuration not found") + + if payload.sample_payload is not None: + try: + execute_mapping(payload.sample_payload, payload.mapping_json, payload.target_schema, limit=100) + except (MappingError, ValueError) as exc: + raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc + + result = await db.execute( + select(func.max(DataSourceMappingTemplate.version)).where( + DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id, + DataSourceMappingTemplate.target_schema == payload.target_schema, + ) + ) + next_version = int(result.scalar() or 0) + 1 + + if payload.is_active: + await db.execute( + DataSourceMappingTemplate.__table__.update() + .where(DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id) + .values(is_active=False) + ) + + template = DataSourceMappingTemplate( + datasource_config_id=payload.datasource_config_id, + target_schema=payload.target_schema, + mapping_json=payload.mapping_json, + sample_payload_hash=payload.sample_payload_hash + or (stable_payload_hash(payload.sample_payload) if payload.sample_payload is not None else None), + validation_status=payload.validation_status, + version=next_version, + is_active=payload.is_active, + ) + db.add(template) + await db.commit() + await db.refresh(template) + return {"message": "Mapping template saved successfully", "data": serialize_mapping_template(template)} + + +@router.put("/mappings/{mapping_id}") +async def update_datasource_mapping( + mapping_id: int, + payload: MappingTemplateUpdate, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + """Update a mapping template in place.""" + template = await db.get(DataSourceMappingTemplate, mapping_id) + if not template: + raise HTTPException(status_code=404, detail="Mapping template not found") + + target_schema = payload.target_schema or template.target_schema + mapping_json = payload.mapping_json or template.mapping_json + get_target_schema(target_schema) + if payload.sample_payload is not None: + try: + execute_mapping(payload.sample_payload, mapping_json, target_schema, limit=100) + except (MappingError, ValueError) as exc: + raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc + + if payload.is_active is True: + await db.execute( + DataSourceMappingTemplate.__table__.update() + .where(DataSourceMappingTemplate.datasource_config_id == template.datasource_config_id) + .where(DataSourceMappingTemplate.id != template.id) + .values(is_active=False) + ) + + template.target_schema = target_schema + template.mapping_json = mapping_json + if payload.sample_payload_hash is not None: + template.sample_payload_hash = payload.sample_payload_hash + elif payload.sample_payload is not None: + template.sample_payload_hash = stable_payload_hash(payload.sample_payload) + if payload.validation_status is not None: + template.validation_status = payload.validation_status + if payload.is_active is not None: + template.is_active = payload.is_active + + await db.commit() + await db.refresh(template) + return {"message": "Mapping template updated successfully", "data": serialize_mapping_template(template)} + + +@router.post("/{config_id}/run-mapped") +async def run_mapped_datasource( + config_id: int, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + """Run a saved custom datasource through its active deterministic mapping.""" + datasource = await db.get(DataSourceConfig, config_id) + if not datasource: + raise HTTPException(status_code=404, detail="Configuration not found") + + result = await db.execute( + select(DataSourceMappingTemplate) + .where(DataSourceMappingTemplate.datasource_config_id == config_id) + .where(DataSourceMappingTemplate.is_active.is_(True)) + .order_by(DataSourceMappingTemplate.version.desc()) + .limit(1) + ) + mapping = result.scalar_one_or_none() + if not mapping: + raise HTTPException(status_code=404, detail="No active mapping template found") + + try: + sample = await fetch_custom_sample_from_config(datasource, 5_000_000) + mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema) + except httpx.HTTPStatusError as exc: + raise HTTPException( + status_code=exc.response.status_code, + detail=f"Datasource request failed: HTTP {exc.response.status_code}", + ) from exc + except httpx.HTTPError as exc: + raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc + except (MappingError, ValueError) as exc: + raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc + + if mapped["failed_count"] > 0: + return { + "status": "failed", + "datasource_config_id": config_id, + "mapping_id": mapping.id, + "mapping_version": mapping.version, + "target_schema": mapping.target_schema, + "mapped_count": mapped["mapped_count"], + "failed_count": mapped["failed_count"], + "errors": mapped["errors"][:20], + } + + written_count = await persist_mapped_records( + db, + datasource_name=datasource.name, + datasource_config_id=datasource.id, + target_schema=mapping.target_schema, + records=mapped["records"], + mapping_version=mapping.version, + ) + return { + "status": "success", + "datasource_config_id": config_id, + "mapping_id": mapping.id, + "mapping_version": mapping.version, + "target_schema": mapping.target_schema, + "fetched_count": mapped["total_items"], + "mapped_count": mapped["mapped_count"], + "written_count": written_count, + } diff --git a/backend/app/api/v1/datasources.py b/backend/app/api/v1/datasources.py index fedcad0c..653117b6 100644 --- a/backend/app/api/v1/datasources.py +++ b/backend/app/api/v1/datasources.py @@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.core.time import to_iso8601_utc from app.core.security import get_current_user from app.core.data_sources import get_data_sources_config +from app.core.datasource_defaults import DEFAULT_DATASOURCES from app.db.session import get_db from app.models.collected_data import CollectedData from app.models.data_snapshot import DataSnapshot @@ -35,6 +36,17 @@ def format_frequency_label(minutes: int) -> str: return f"{minutes}m" +def datasource_metadata(source: str) -> dict: + info = DEFAULT_DATASOURCES.get(source, {}) + return { + "display_name": info.get("display_name") or info.get("name") or source, + "is_free": bool(info.get("is_free", True)), + "requires_credentials": bool(info.get("requires_credentials", False)), + "credential_provider": info.get("credential_provider"), + "credential_status": info.get("credential_status", "none"), + } + + def is_due_for_collection(datasource: DataSource, now: datetime) -> bool: if datasource.last_run_at is None: return True @@ -72,31 +84,6 @@ async def _load_latest_running_tasks( return {task.datasource_id: task for task in result.scalars().all()} -async def _load_latest_completed_tasks( - db: AsyncSession, - datasource_ids: list[int], -) -> dict[int, CollectionTask]: - if not datasource_ids: - return {} - - ranked_tasks = ( - select( - CollectionTask.id.label("task_id"), - _task_rank_column(CollectionTask.completed_at), - ) - .where(CollectionTask.datasource_id.in_(datasource_ids)) - .where(CollectionTask.completed_at.isnot(None)) - .where(CollectionTask.status.in_(("success", "failed", "cancelled"))) - .subquery() - ) - result = await db.execute( - select(CollectionTask) - .join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id) - .where(ranked_tasks.c.row_num == 1) - ) - return {task.datasource_id: task for task in result.scalars().all()} - - async def _load_latest_task_ids( db: AsyncSession, datasource_ids: list[int], @@ -123,21 +110,6 @@ async def _load_latest_task_ids( return {datasource_id: task_id for datasource_id, task_id in result.all()} -async def _load_datasource_data_counts( - db: AsyncSession, - sources: list[str], -) -> dict[str, int]: - if not sources: - return {} - - result = await db.execute( - select(CollectedData.source, func.count(CollectedData.id)) - .where(CollectedData.source.in_(sources)) - .group_by(CollectedData.source) - ) - return {source: count for source, count in result.all()} - - async def _load_datasource_endpoint_overrides( db: AsyncSession, sources: list[str], @@ -161,7 +133,7 @@ async def _load_datasource_endpoint_overrides( async def _load_datasource_list_context( db: AsyncSession, datasources: list[DataSource], -) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]: +) -> tuple[dict[int, CollectionTask], dict[str, str]]: datasource_ids = [datasource.id for datasource in datasources] sources = [datasource.source for datasource in datasources] @@ -185,10 +157,8 @@ async def _load_datasource_list_context( if stale_datasource_ids: running_tasks = await _load_latest_running_tasks(db, datasource_ids) - completed_tasks = await _load_latest_completed_tasks(db, datasource_ids) - data_counts = await _load_datasource_data_counts(db, sources) endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources) - return running_tasks, completed_tasks, data_counts, endpoint_overrides + return running_tasks, endpoint_overrides async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]: @@ -401,27 +371,19 @@ async def list_datasources( collector_list = [] config = get_data_sources_config() - running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context( - db, - datasources, - ) + running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources) for datasource in datasources: running_task = running_tasks.get(datasource.id) - last_task = completed_tasks.get(datasource.id) - endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url( - datasource.source, - ) - data_count = data_counts.get(datasource.source, 0) - - last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None) - last_run = to_iso8601_utc(last_run_at) - last_status = datasource.last_status or (last_task.status if last_task else None) + endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source) + last_run_at = datasource.last_run_at + last_status = datasource.last_status collector_list.append( { "id": datasource.id, "source": datasource.source, "name": datasource.name, + **datasource_metadata(datasource.source), "module": datasource.module, "priority": datasource.priority, "frequency": format_frequency_label(datasource.frequency_minutes), @@ -429,11 +391,9 @@ async def list_datasources( "is_active": datasource.is_active, "collector_class": datasource.collector_class, "endpoint": endpoint, - "last_run": last_run, + "last_run": to_iso8601_utc(last_run_at), "last_run_at": to_iso8601_utc(last_run_at), "last_status": last_status, - "last_records_processed": last_task.records_processed if last_task else None, - "data_count": data_count, "is_running": running_task is not None, "task_id": running_task.id if running_task else None, "progress": running_task.progress if running_task else None, @@ -576,6 +536,7 @@ async def get_datasource( return { "id": datasource.id, "name": datasource.name, + **datasource_metadata(datasource.source), "module": datasource.module, "priority": datasource.priority, "frequency": format_frequency_label(datasource.frequency_minutes), diff --git a/backend/app/api/v1/settings.py b/backend/app/api/v1/settings.py index ff580a20..7ab528b8 100644 --- a/backend/app/api/v1/settings.py +++ b/backend/app/api/v1/settings.py @@ -9,10 +9,19 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.core.security import get_current_user from app.core.time import to_iso8601_utc +from app.core.config import settings as app_settings +from app.core.data_sources import get_data_sources_config +from app.core.datasource_defaults import DEFAULT_DATASOURCES from app.db.session import get_db from app.models.datasource import DataSource +from app.models.datasource_config import DataSourceConfig from app.models.system_setting import SystemSetting from app.models.user import User +from app.services.llm_provider_catalog import ( + get_fallback_llm_provider_preset, + list_fallback_llm_provider_presets, + refresh_llm_provider_preset, +) from app.services.scheduler import sync_datasource_job from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings @@ -39,6 +48,21 @@ DEFAULT_SETTINGS = { "password_policy": "medium", }, "tv": DEFAULT_TV_SETTINGS, + "external_integrations": { + "ai_provider": { + "service_url": "", + "service_token": "", + "provider": "minimax", + "provider_api": "anthropic-messages", + "base_url": "https://api.minimaxi.com/anthropic", + "model": "MiniMax-M2.7", + "api_key": "", + "max_tokens": 1200, + "anthropic_version": "2023-06-01", + "timeout_seconds": 60, + "retry_attempts": 2, + } + }, } @@ -96,6 +120,34 @@ class TVSettingsUpdate(BaseModel): sources: list[TVStreamSourceUpdate] = Field(default_factory=list) +class AIProviderIntegrationUpdate(BaseModel): + service_url: str = "" + service_token: Optional[str] = None + provider: str = Field(default="minimax", max_length=80) + provider_api: str = Field(default="anthropic-messages", max_length=80) + base_url: str = Field(default="", max_length=500) + model: str = Field(default="", max_length=200) + api_key: Optional[str] = None + max_tokens: int = Field(default=1200, ge=1, le=200000) + anthropic_version: str = Field(default="2023-06-01", max_length=40) + timeout_seconds: int = Field(default=60, ge=5, le=600) + retry_attempts: int = Field(default=2, ge=1, le=10) + clear_service_token: bool = False + clear_api_key: bool = False + + +class BarentsWatchIntegrationUpdate(BaseModel): + endpoint: str = "" + client_id: str = "" + client_secret: Optional[str] = None + clear_client_secret: bool = False + + +class ExternalIntegrationsUpdate(BaseModel): + ai_provider: AIProviderIntegrationUpdate + barentswatch: BarentsWatchIntegrationUpdate + + def merge_with_defaults(category: str, payload: Optional[dict]) -> dict: merged = deepcopy(DEFAULT_SETTINGS[category]) if payload: @@ -146,6 +198,158 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) - return merge_with_defaults(category, record.payload) +def _mask_secret(value: Optional[str]) -> dict: + if not value: + return {"configured": False, "preview": ""} + text = str(value) + if "-" in text: + prefix = text.split("-", 1)[0] + "-" + preview = prefix + ("*" * max(len(text) - len(prefix), 1)) + else: + prefix_len = min(4, len(text)) + preview = text[:prefix_len] + ("*" * max(len(text) - prefix_len, 1)) + return {"configured": True, "preview": preview} + + +async def get_runtime_ai_provider_config(db: AsyncSession) -> dict: + runtime_record = await get_setting_record(db, "external_integrations") + payload = merge_with_defaults( + "external_integrations", + runtime_record.payload if runtime_record else None, + ) + ai_payload = payload.get("ai_provider") or {} + has_runtime_llm_config = bool( + runtime_record + and isinstance(runtime_record.payload, dict) + and isinstance(runtime_record.payload.get("ai_provider"), dict) + ) + return { + "service_url": ai_payload.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL, + "service_token": ai_payload.get("service_token") or app_settings.AI_PROVIDER_SERVICE_TOKEN, + "timeout_seconds": int( + ai_payload.get("timeout_seconds") or app_settings.AI_PROVIDER_TIMEOUT_SECONDS + ), + "retry_attempts": int( + ai_payload.get("retry_attempts") or app_settings.AI_PROVIDER_RETRY_ATTEMPTS + ), + "llm_config": { + "provider": ai_payload.get("provider") or "minimax", + "provider_api": ai_payload.get("provider_api") or "anthropic-messages", + "base_url": ai_payload.get("base_url") or "https://api.minimaxi.com/anthropic", + "model": ai_payload.get("model") or "MiniMax-M2.7", + "api_key": ai_payload.get("api_key") or "", + "max_tokens": int(ai_payload.get("max_tokens") or 1200), + "anthropic_version": ai_payload.get("anthropic_version") or "2023-06-01", + } if has_runtime_llm_config else {}, + } + + +async def get_barentswatch_config_record(db: AsyncSession) -> Optional[DataSourceConfig]: + result = await db.execute( + select(DataSourceConfig) + .where(DataSourceConfig.name == "barentswatch_vessels") + .where(DataSourceConfig.is_active.is_(True)) + ) + return result.scalar_one_or_none() + + +async def serialize_external_integrations(db: AsyncSession) -> dict: + ai_config = await get_runtime_ai_provider_config(db) + runtime_setting = await get_setting_record(db, "external_integrations") + display_llm_config = ai_config["llm_config"] or DEFAULT_SETTINGS["external_integrations"]["ai_provider"] + barentswatch_record = await get_barentswatch_config_record(db) + yaml_config = get_data_sources_config() + barentswatch_auth = barentswatch_record.auth_config if barentswatch_record else {} + barentswatch_auth = barentswatch_auth or {} + return { + "ai_provider": { + "service_url": ai_config["service_url"], + "service_token": _mask_secret(ai_config["service_token"]), + "provider": display_llm_config.get("provider") or "minimax", + "provider_api": display_llm_config.get("provider_api") or "anthropic-messages", + "base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic", + "model": display_llm_config.get("model") or "MiniMax-M2.7", + "api_key": _mask_secret(display_llm_config.get("api_key")), + "max_tokens": int(display_llm_config.get("max_tokens") or 1200), + "anthropic_version": display_llm_config.get("anthropic_version") or "2023-06-01", + "timeout_seconds": ai_config["timeout_seconds"], + "retry_attempts": ai_config["retry_attempts"], + "source": "runtime" if runtime_setting else "env", + }, + "barentswatch": { + "endpoint": ( + barentswatch_record.endpoint + if barentswatch_record and barentswatch_record.endpoint + else yaml_config.get_yaml_url("barentswatch_vessels") + ), + "client_id": barentswatch_auth.get("client_id") or "", + "client_secret": _mask_secret(barentswatch_auth.get("client_secret")), + "source": "datasource_config" if barentswatch_record else "default", + }, + } + + +async def save_external_integrations_payload( + db: AsyncSession, + update: ExternalIntegrationsUpdate, +) -> dict: + current_payload = await get_setting_payload(db, "external_integrations") + current_ai = current_payload.get("ai_provider") or {} + ai_payload = { + "service_url": update.ai_provider.service_url.strip() + or app_settings.AI_PROVIDER_SERVICE_URL, + "service_token": current_ai.get("service_token") or "", + "provider": update.ai_provider.provider.strip() or "minimax", + "provider_api": update.ai_provider.provider_api.strip() or "anthropic-messages", + "base_url": update.ai_provider.base_url.strip(), + "model": update.ai_provider.model.strip(), + "api_key": current_ai.get("api_key") or "", + "max_tokens": update.ai_provider.max_tokens, + "anthropic_version": update.ai_provider.anthropic_version.strip() or "2023-06-01", + "timeout_seconds": update.ai_provider.timeout_seconds, + "retry_attempts": update.ai_provider.retry_attempts, + } + if update.ai_provider.clear_service_token: + ai_payload["service_token"] = "" + elif update.ai_provider.service_token not in (None, ""): + ai_payload["service_token"] = update.ai_provider.service_token + if update.ai_provider.clear_api_key: + ai_payload["api_key"] = "" + elif update.ai_provider.api_key not in (None, ""): + ai_payload["api_key"] = update.ai_provider.api_key + + await save_setting_payload(db, "external_integrations", {"ai_provider": ai_payload}) + + default_endpoint = get_data_sources_config().get_yaml_url("barentswatch_vessels") + barentswatch_record = await get_barentswatch_config_record(db) + if barentswatch_record is None: + barentswatch_record = DataSourceConfig( + name="barentswatch_vessels", + description="BarentsWatch Live AIS credentials", + source_type="api", + endpoint=update.barentswatch.endpoint.strip() or default_endpoint, + auth_type="oauth_client_credentials", + auth_config={}, + headers={}, + config={}, + is_active=True, + ) + db.add(barentswatch_record) + + current_auth = dict(barentswatch_record.auth_config or {}) + if update.barentswatch.clear_client_secret: + current_auth.pop("client_secret", None) + elif update.barentswatch.client_secret not in (None, ""): + current_auth["client_secret"] = update.barentswatch.client_secret + current_auth["client_id"] = update.barentswatch.client_id.strip() + barentswatch_record.endpoint = update.barentswatch.endpoint.strip() or default_endpoint + barentswatch_record.auth_type = "oauth_client_credentials" + barentswatch_record.auth_config = current_auth + await db.commit() + + return await serialize_external_integrations(db) + + def format_frequency_label(minutes: int) -> str: if minutes % 1440 == 0: return f"{minutes // 1440}d" @@ -155,9 +359,11 @@ def format_frequency_label(minutes: int) -> str: def serialize_collector(datasource: DataSource) -> dict: + defaults = DEFAULT_DATASOURCES.get(datasource.source, {}) return { "id": datasource.id, "name": datasource.name, + "display_name": defaults.get("display_name") or datasource.name, "source": datasource.source, "module": datasource.module, "priority": datasource.priority, @@ -167,6 +373,10 @@ def serialize_collector(datasource: DataSource) -> dict: "last_run_at": to_iso8601_utc(datasource.last_run_at), "last_status": datasource.last_status, "next_run_at": to_iso8601_utc(datasource.next_run_at), + "is_free": bool(defaults.get("is_free", True)), + "requires_credentials": bool(defaults.get("requires_credentials", False)), + "credential_provider": defaults.get("credential_provider"), + "credential_status": defaults.get("credential_status", "none"), } @@ -243,6 +453,46 @@ async def update_tv_settings( return {"status": "updated", "tv": normalize_tv_settings(saved)} +@router.get("/integrations") +async def get_external_integrations( + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + return {"integrations": await serialize_external_integrations(db)} + + +@router.get("/integrations/ai-provider/presets") +async def get_ai_provider_presets( + current_user: User = Depends(get_current_user), +): + return {"data": list_fallback_llm_provider_presets()} + + +@router.post("/integrations/ai-provider/presets/{provider}/refresh") +async def refresh_ai_provider_preset( + provider: str, + current_user: User = Depends(get_current_user), +): + try: + return {"data": await refresh_llm_provider_preset(provider)} + except ValueError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + except Exception as exc: + fallback = get_fallback_llm_provider_preset(provider) + fallback["refresh_error"] = str(exc) + return {"data": fallback} + + +@router.put("/integrations") +async def update_external_integrations( + payload: ExternalIntegrationsUpdate, + current_user: User = Depends(get_current_user), + db: AsyncSession = Depends(get_db), +): + saved = await save_external_integrations_payload(db, payload) + return {"status": "updated", "integrations": saved} + + @router.get("/collectors") async def get_collector_settings( current_user: User = Depends(get_current_user), @@ -289,6 +539,7 @@ async def get_all_settings( "notifications": setting_payloads["notifications"], "security": setting_payloads["security"], "tv": await get_tv_settings_payload(db), + "integrations": await serialize_external_integrations(db), "collectors": [serialize_collector(datasource) for datasource in datasources], "generated_at": to_iso8601_utc(datetime.now(UTC)), } diff --git a/backend/app/api/v1/visualization.py b/backend/app/api/v1/visualization.py index b147f5e0..e82bff5d 100644 --- a/backend/app/api/v1/visualization.py +++ b/backend/app/api/v1/visualization.py @@ -4,7 +4,7 @@ Unified API for all visualization data sources. Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium. """ -from datetime import UTC, datetime +from datetime import UTC, datetime, timedelta import math import httpx from fastapi import APIRouter, HTTPException, Depends, Query, Response @@ -20,6 +20,7 @@ from app.db.session import get_db from app.models.bgp_anomaly import BGPAnomaly from app.models.bgp_incident import BGPIncident from app.models.collected_data import CollectedData +from app.models.vessel import VesselPosition, VesselStatic from app.services.bgp_collectors import build_bgp_collector_coverage from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS @@ -511,7 +512,7 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str if unit in {"pflop/s", "pflops", "pflop"}: normalized_tflops = capacity_value * 1000 elif unit in {"gflop/s", "gflops", "gflop"}: - normalized_tflops = capacity_value / 1000 + normalized_tflops = capacity_value else: normalized_tflops = capacity_value @@ -609,6 +610,108 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str return {"type": "FeatureCollection", "features": features} +VESSEL_TYPE_FILTERS = { + "cargo": lambda props: str(props.get("vessel_type_name", "")).lower() == "cargo" + or 70 <= int(props.get("vessel_type") or -1) <= 79, + "tanker": lambda props: str(props.get("vessel_type_name", "")).lower() == "tanker" + or 80 <= int(props.get("vessel_type") or -1) <= 89, + "passenger": lambda props: str(props.get("vessel_type_name", "")).lower() == "passenger" + or 60 <= int(props.get("vessel_type") or -1) <= 69, + "fishing": lambda props: str(props.get("vessel_type_name", "")).lower() == "fishing" + or int(props.get("vessel_type") or -1) == 30, + "military": lambda props: str(props.get("vessel_type_name", "")).lower() == "military" + or int(props.get("vessel_type") or -1) == 35, + "other": lambda props: str(props.get("vessel_type_name", "")).lower() + not in {"cargo", "tanker", "passenger", "fishing", "military"}, +} + + +def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]: + features = [] + for position, static in rows: + if position.lat is None or position.lon is None: + continue + props = { + "mmsi": position.mmsi, + "name": getattr(static, "name", None) or f"MMSI {position.mmsi}", + "callsign": getattr(static, "callsign", None), + "imo": getattr(static, "imo", None), + "vessel_type": getattr(static, "vessel_type", None), + "vessel_type_name": getattr(static, "vessel_type_name", None) or "Other", + "flag": getattr(static, "flag", None), + "length": getattr(static, "length", None), + "width": getattr(static, "width", None), + "draught": getattr(static, "draught", None), + "sog": position.sog, + "cog": position.cog, + "heading": position.heading, + "nav_status": position.nav_status, + "received_at": to_iso8601_utc(position.received_at), + "data_type": "vessel", + } + features.append( + { + "type": "Feature", + "id": position.mmsi, + "geometry": { + "type": "Point", + "coordinates": [position.lon, position.lat], + }, + "properties": props, + } + ) + + return {"type": "FeatureCollection", "features": features} + + +def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None: + if not value: + return None + parts = [part.strip() for part in value.split(",")] + if len(parts) != 4: + raise HTTPException(status_code=400, detail="bbox must be lon_min,lat_min,lon_max,lat_max") + try: + lon_min, lat_min, lon_max, lat_max = [float(part) for part in parts] + except ValueError as exc: + raise HTTPException(status_code=400, detail="bbox values must be numbers") from exc + if lat_min > lat_max: + lat_min, lat_max = lat_max, lat_min + if lon_min > lon_max: + lon_min, lon_max = lon_max, lon_min + return lon_min, lat_min, lon_max, lat_max + + +def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bool: + if not requested_types: + return True + for requested_type in requested_types: + predicate = VESSEL_TYPE_FILTERS.get(requested_type) + if predicate and predicate(props): + return True + return False + + +def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]: + by_type: dict[str, int] = {} + underway = 0 + anchored_or_moored = 0 + for feature in features: + props = feature.get("properties", {}) + vessel_type = str(props.get("vessel_type_name") or "Other") + by_type[vessel_type] = by_type.get(vessel_type, 0) + 1 + nav_status = props.get("nav_status") + if nav_status in (1, 5): + anchored_or_moored += 1 + else: + underway += 1 + return { + "total": len(features), + "by_type": by_type, + "underway": underway, + "anchored_or_moored": anchored_or_moored, + } + + def convert_bgp_anomalies_to_geojson( records: List[BGPAnomaly], geography_hints: Optional[Dict[str, Dict[str, Any]]] = None, @@ -1298,6 +1401,137 @@ async def get_compute_centers_geojson( } +@router.get("/geo/vessels") +async def get_vessels_geojson( + bbox: Optional[str] = Query( + None, + description="Viewport bbox as lon_min,lat_min,lon_max,lat_max", + ), + type: Optional[str] = Query( + None, + description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other", + ), + limit: int = Query(5000, ge=1, le=50000), + db: AsyncSession = Depends(get_db), +): + """Return latest vessel positions as GeoJSON points.""" + latest_times = ( + select( + VesselPosition.mmsi.label("mmsi"), + func.max(VesselPosition.received_at).label("received_at"), + ) + .group_by(VesselPosition.mmsi) + .subquery() + ) + stmt = ( + select(VesselPosition, VesselStatic) + .join( + latest_times, + (VesselPosition.mmsi == latest_times.c.mmsi) + & (VesselPosition.received_at == latest_times.c.received_at), + ) + .outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi) + .order_by(VesselPosition.received_at.desc()) + .limit(limit) + ) + + parsed_bbox = _parse_bbox(bbox) + if parsed_bbox is not None: + lon_min, lat_min, lon_max, lat_max = parsed_bbox + stmt = stmt.where( + VesselPosition.lon >= lon_min, + VesselPosition.lon <= lon_max, + VesselPosition.lat >= lat_min, + VesselPosition.lat <= lat_max, + ) + + result = await db.execute(stmt) + rows = list(result.all()) + geojson = convert_vessels_to_geojson(rows) + requested_types = { + item.strip().lower() + for item in (type or "").split(",") + if item.strip() + } + if requested_types: + geojson["features"] = [ + feature + for feature in geojson.get("features", []) + if _matches_vessel_type(feature.get("properties", {}), requested_types) + ] + + features = geojson.get("features", []) + return { + **geojson, + "count": len(features), + "stats": _build_vessel_stats(features), + } + + +@router.get("/vessels/{mmsi}") +async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)): + latest_position_stmt = ( + select(VesselPosition) + .where(VesselPosition.mmsi == mmsi) + .order_by(VesselPosition.received_at.desc()) + .limit(1) + ) + static = await db.get(VesselStatic, mmsi) + result = await db.execute(latest_position_stmt) + position = result.scalar_one_or_none() + if position is None: + raise HTTPException(status_code=404, detail="Vessel not found") + geojson = convert_vessels_to_geojson([(position, static)]) + return { + **(geojson["features"][0]["properties"]), + "latitude": position.lat, + "longitude": position.lon, + } + + +@router.get("/vessels/{mmsi}/track") +async def get_vessel_track( + mmsi: int, + hours: int = Query(6, ge=1, le=24), + db: AsyncSession = Depends(get_db), +): + cutoff = datetime.now(UTC) - timedelta(hours=hours) + result = await db.execute( + select(VesselPosition) + .where(VesselPosition.mmsi == mmsi) + .where(VesselPosition.received_at >= cutoff) + .order_by(VesselPosition.received_at.asc()) + ) + positions = list(result.scalars().all()) + if not positions: + return { + "type": "FeatureCollection", + "features": [], + "count": 0, + } + + return { + "type": "FeatureCollection", + "features": [ + { + "type": "Feature", + "geometry": { + "type": "LineString", + "coordinates": [[position.lon, position.lat] for position in positions], + }, + "properties": { + "mmsi": mmsi, + "hours": hours, + "point_count": len(positions), + "start_at": to_iso8601_utc(positions[0].received_at), + "end_at": to_iso8601_utc(positions[-1].received_at), + }, + } + ], + "count": 1, + } + + @router.get("/geo/bgp-anomalies") async def get_bgp_anomalies_geojson( severity: Optional[str] = Query(None), @@ -1394,6 +1628,10 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)): db, source_filter=("ris_live_bgp", "bgpstream_bgp"), ) + vessel_count_result = await db.execute( + select(func.count(func.distinct(VesselPosition.mmsi))), + ) + vessel_count = int(vessel_count_result.scalar() or 0) return { "generated_at": to_iso8601_utc(datetime.now(UTC)), @@ -1402,6 +1640,7 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)): "landing_point_count": len(landing_points.get("features", [])), "satellite_count": len(satellites.get("features", [])), "compute_center_count": len(compute_features), + "vessel_count": vessel_count, "supercomputer_count": sum( 1 for feature in compute_features if feature.get("properties", {}).get("site_type") == "supercomputer" diff --git a/backend/app/core/data_sources.py b/backend/app/core/data_sources.py index 8a2a7669..9df98ef3 100644 --- a/backend/app/core/data_sources.py +++ b/backend/app/core/data_sources.py @@ -31,6 +31,7 @@ COLLECTOR_URL_KEYS = { "opengeofeed_prefix_geo": "opengeofeed.public_csv_url", "nro_delegated_prefix_geo": "nro.delegated_stats_url", "news_live_streams": "news_live_streams.channels_url", + "barentswatch_vessels": "barentswatch_vessels.url", } diff --git a/backend/app/core/data_sources.yaml b/backend/app/core/data_sources.yaml index 17e5658e..b1627e2a 100644 --- a/backend/app/core/data_sources.yaml +++ b/backend/app/core/data_sources.yaml @@ -94,3 +94,7 @@ news_live_streams: streams_url: "https://iptv-org.github.io/api/streams.json" # IPTV-org 台标 JSON logos_url: "https://iptv-org.github.io/api/logos.json" + +barentswatch_vessels: + # BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token. + url: "https://live.ais.barentswatch.no/v1/latest/combined" diff --git a/backend/app/core/datasource_defaults.py b/backend/app/core/datasource_defaults.py index b1fb18f7..ee7a70b0 100644 --- a/backend/app/core/datasource_defaults.py +++ b/backend/app/core/datasource_defaults.py @@ -4,163 +4,246 @@ DEFAULT_DATASOURCES = { "top500": { "id": 1, "name": "TOP500 Supercomputers", + "display_name": "TOP500 超算榜单", "module": "L1", "priority": "P0", "frequency_minutes": 240, + "is_free": True, + "requires_credentials": False, }, "epoch_ai_gpu": { "id": 2, "name": "Epoch AI GPU Clusters", + "display_name": "Epoch AI GPU 集群", "module": "L1", "priority": "P0", "frequency_minutes": 360, + "is_free": True, + "requires_credentials": False, }, "huggingface_models": { "id": 3, "name": "HuggingFace Models", + "display_name": "Hugging Face 模型", "module": "L2", "priority": "P1", "frequency_minutes": 720, + "is_free": True, + "requires_credentials": False, }, "huggingface_datasets": { "id": 4, "name": "HuggingFace Datasets", + "display_name": "Hugging Face 数据集", "module": "L2", "priority": "P1", "frequency_minutes": 720, + "is_free": True, + "requires_credentials": False, }, "huggingface_spaces": { "id": 5, "name": "HuggingFace Spaces", + "display_name": "Hugging Face Spaces", "module": "L2", "priority": "P2", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "peeringdb_ixp": { "id": 6, "name": "PeeringDB IXP", + "display_name": "PeeringDB 交换中心", "module": "L2", "priority": "P1", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "peeringdb_network": { "id": 7, "name": "PeeringDB Networks", + "display_name": "PeeringDB 网络", "module": "L2", "priority": "P2", "frequency_minutes": 2880, + "is_free": True, + "requires_credentials": False, }, "peeringdb_facility": { "id": 8, "name": "PeeringDB Facilities", + "display_name": "PeeringDB 设施", "module": "L2", "priority": "P2", "frequency_minutes": 2880, + "is_free": True, + "requires_credentials": False, }, "telegeography_cables": { "id": 9, "name": "Submarine Cables", + "display_name": "海底光缆", "module": "L2", "priority": "P1", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "telegeography_landing": { "id": 10, "name": "Cable Landing Points", + "display_name": "光缆登陆点", "module": "L2", "priority": "P2", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "telegeography_systems": { "id": 11, "name": "Cable Systems", + "display_name": "光缆系统", "module": "L2", "priority": "P2", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "arcgis_cables": { "id": 15, "name": "ArcGIS Submarine Cables", + "display_name": "ArcGIS 海底光缆", "module": "L2", "priority": "P1", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "arcgis_landing_points": { "id": 16, "name": "ArcGIS Landing Points", + "display_name": "ArcGIS 登陆点", "module": "L2", "priority": "P1", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "arcgis_cable_landing_relation": { "id": 17, "name": "ArcGIS Cable-Landing Relations", + "display_name": "ArcGIS 光缆登陆关系", "module": "L2", "priority": "P1", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "fao_landing_points": { "id": 18, "name": "FAO Landing Points", + "display_name": "FAO 登陆点", "module": "L2", "priority": "P1", "frequency_minutes": 10080, + "is_free": True, + "requires_credentials": False, }, "spacetrack_tle": { "id": 19, "name": "Space-Track TLE", + "display_name": "Space-Track 轨道根数", "module": "L3", "priority": "P2", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": True, + "credential_provider": "spacetrack", + "credential_status": "planned", }, "celestrak_tle": { "id": 20, "name": "CelesTrak TLE", + "display_name": "CelesTrak 轨道根数", "module": "L3", "priority": "P2", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "ris_live_bgp": { "id": 21, "name": "RIPE RIS Live BGP", + "display_name": "RIPE RIS 实时 BGP", "module": "L3", "priority": "P1", "frequency_minutes": 15, + "is_free": True, + "requires_credentials": False, }, "bgpstream_bgp": { "id": 22, "name": "CAIDA BGPStream Backfill", + "display_name": "CAIDA BGPStream 回填", "module": "L3", "priority": "P1", "frequency_minutes": 360, + "is_free": True, + "requires_credentials": False, }, "iptoasn_prefix_geo": { "id": 23, "name": "IPtoASN Prefix Geography", + "display_name": "IPtoASN 前缀地理", "module": "L3", "priority": "P1", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "opengeofeed_prefix_geo": { "id": 24, "name": "OpenGeoFeed Prefix Geography", + "display_name": "OpenGeoFeed 前缀地理", "module": "L3", "priority": "P1", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "nro_delegated_prefix_geo": { "id": 25, "name": "NRO Delegated Prefix Geography", + "display_name": "NRO 分配前缀地理", "module": "L3", "priority": "P1", "frequency_minutes": 1440, + "is_free": True, + "requires_credentials": False, }, "news_live_streams": { "id": 26, "name": "News Live Streams", + "display_name": "新闻直播源", "module": "L4", "priority": "P2", "frequency_minutes": 720, + "is_free": True, + "requires_credentials": False, + }, + "barentswatch_vessels": { + "id": 27, + "name": "BarentsWatch AIS Vessels", + "display_name": "BarentsWatch AIS 船舶", + "module": "L4", + "priority": "P1", + "frequency_minutes": 1, + "is_free": True, + "requires_credentials": True, + "credential_provider": "barentswatch", + "credential_status": "supported", }, } diff --git a/backend/app/core/target_schema_registry.py b/backend/app/core/target_schema_registry.py new file mode 100644 index 00000000..0b2aba7e --- /dev/null +++ b/backend/app/core/target_schema_registry.py @@ -0,0 +1,151 @@ +"""Registry of target schemas supported by mapped custom data sources.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Any + +from pydantic import BaseModel, Field, ValidationError, field_validator + + +class VesselAISRecord(BaseModel): + mmsi: int = Field(ge=100000000, le=999999999) + lat: float = Field(ge=-90, le=90) + lon: float = Field(ge=-180, le=180) + sog: float | None = None + cog: float | None = Field(default=None, ge=0, le=360) + heading: int | None = Field(default=None, ge=0, le=511) + name: str | None = None + vessel_type: str | int | None = None + received_at: datetime | None = None + + +class GeoPointRecord(BaseModel): + lat: float = Field(ge=-90, le=90) + lon: float = Field(ge=-180, le=180) + name: str | None = None + type: str | None = None + source_id: str | None = None + observed_at: datetime | None = None + metadata: dict[str, Any] = Field(default_factory=dict) + + +class GenericRecord(BaseModel): + data: dict[str, Any] = Field(default_factory=dict) + source_id: str | None = None + observed_at: datetime | None = None + + @field_validator("data") + @classmethod + def require_payload(cls, value: dict[str, Any]) -> dict[str, Any]: + if not value: + raise ValueError("generic_records requires a non-empty data object") + return value + + +@dataclass(frozen=True) +class TargetField: + name: str + type: str + required: bool = False + description: str = "" + example: Any = None + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "type": self.type, + "required": self.required, + "description": self.description, + "example": self.example, + } + + +@dataclass(frozen=True) +class TargetSchema: + key: str + label: str + description: str + fields: tuple[TargetField, ...] + model: type[BaseModel] + destination: str + + def to_dict(self) -> dict[str, Any]: + return { + "key": self.key, + "label": self.label, + "description": self.description, + "destination": self.destination, + "fields": [field.to_dict() for field in self.fields], + } + + def validate_record(self, record: dict[str, Any]) -> tuple[dict[str, Any] | None, list[str]]: + try: + return self.model.model_validate(record).model_dump(mode="json"), [] + except ValidationError as exc: + return None, [ + ".".join(str(part) for part in error["loc"]) + f": {error['msg']}" + for error in exc.errors() + ] + + +TARGET_SCHEMAS: dict[str, TargetSchema] = { + "vessel_ais": TargetSchema( + key="vessel_ais", + label="船舶 AIS", + description="船只位置、航速、航向、MMSI 等 AIS 数据。", + destination="vessel_position", + model=VesselAISRecord, + fields=( + TargetField("mmsi", "integer", True, "MMSI 九位船舶标识", 257123000), + TargetField("lat", "float", True, "纬度", 59.91), + TargetField("lon", "float", True, "经度", 10.75), + TargetField("sog", "float", False, "对地航速，单位节", 12.4), + TargetField("cog", "float", False, "对地航向，0-360 度", 184.5), + TargetField("heading", "integer", False, "船首向，0-511", 186), + TargetField("name", "string", False, "船名", "OSLO EXPRESS"), + TargetField("vessel_type", "string", False, "船型", "cargo"), + TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"), + ), + ), + "geo_points": TargetSchema( + key="geo_points", + label="通用地理点", + description="带经纬度的通用实体或事件点位。", + destination="generic_geo_points", + model=GeoPointRecord, + fields=( + TargetField("lat", "float", True, "纬度", 1.3), + TargetField("lon", "float", True, "经度", 103.8), + TargetField("name", "string", False, "点位名称", "Singapore"), + TargetField("type", "string", False, "点位类型", "datacenter"), + TargetField("source_id", "string", False, "来源侧 ID", "sg-1"), + TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"), + TargetField("metadata", "object", False, "扩展字段", {"provider": "example"}), + ), + ), + "generic_records": TargetSchema( + key="generic_records", + label="通用结构化记录", + description="未知结构数据沉淀，不直接进入 Earth 图层。", + destination="collected_data", + model=GenericRecord, + fields=( + TargetField("data", "object", True, "结构化记录主体", {"raw": "value"}), + TargetField("source_id", "string", False, "来源侧 ID", "record-1"), + TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"), + ), + ), +} + + +def list_target_schemas() -> list[dict[str, Any]]: + return [schema.to_dict() for schema in TARGET_SCHEMAS.values()] + + +def get_target_schema(key: str) -> TargetSchema: + try: + return TARGET_SCHEMAS[key] + except KeyError as exc: + raise ValueError(f"Unsupported target schema: {key}") from exc diff --git a/backend/app/db/session.py b/backend/app/db/session.py index ae30a1e3..29484719 100644 --- a/backend/app/db/session.py +++ b/backend/app/db/session.py @@ -110,6 +110,8 @@ async def init_db(): import app.models.playground_session # noqa: F401 import app.models.playground_message # noqa: F401 import app.models.system_log # noqa: F401 + import app.models.vessel # noqa: F401 + import app.models.datasource_mapping # noqa: F401 logger.warning_event( "Database pool settings active", diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index 1757eab5..45f15f8f 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -12,6 +12,8 @@ from app.models.system_setting import SystemSetting from app.models.playground_session import PlaygroundSession from app.models.playground_message import PlaygroundMessage from app.models.system_log import SystemLog, AuditLog +from app.models.vessel import VesselPosition, VesselStatic +from app.models.datasource_mapping import DataSourceMappingTemplate __all__ = [ "User", @@ -29,4 +31,7 @@ __all__ = [ "BGPObservation", "SystemLog", "AuditLog", + "VesselPosition", + "VesselStatic", + "DataSourceMappingTemplate", ] diff --git a/backend/app/models/datasource_mapping.py b/backend/app/models/datasource_mapping.py new file mode 100644 index 00000000..eee2c269 --- /dev/null +++ b/backend/app/models/datasource_mapping.py @@ -0,0 +1,32 @@ +"""Mapping templates for user-defined data source payloads.""" + +from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String +from sqlalchemy.sql import func + +from app.db.session import Base + + +class DataSourceMappingTemplate(Base): + __tablename__ = "datasource_mapping_templates" + + id = Column(Integer, primary_key=True, autoincrement=True) + datasource_config_id = Column( + Integer, + ForeignKey("datasource_configs.id"), + nullable=False, + index=True, + ) + target_schema = Column(String(80), nullable=False, index=True) + mapping_json = Column(JSON, nullable=False, default={}) + sample_payload_hash = Column(String(64), nullable=True) + validation_status = Column(String(30), nullable=False, default="draft") + version = Column(Integer, nullable=False, default=1) + is_active = Column(Boolean, nullable=False, default=False, index=True) + created_at = Column(DateTime(timezone=True), server_default=func.now()) + updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now()) + + def __repr__(self): + return ( + f"" + ) diff --git a/backend/app/models/vessel.py b/backend/app/models/vessel.py new file mode 100644 index 00000000..665b4d0d --- /dev/null +++ b/backend/app/models/vessel.py @@ -0,0 +1,75 @@ +"""Vessel AIS models for live maritime tracking.""" + +from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String +from sqlalchemy.sql import func + +from app.core.time import to_iso8601_utc +from app.db.session import Base + + +class VesselStatic(Base): + """Slow-changing vessel identity and dimensions.""" + + __tablename__ = "vessel_static" + + mmsi = Column(BigInteger, primary_key=True) + name = Column(String(128), nullable=True) + callsign = Column(String(16), nullable=True) + vessel_type = Column(SmallInteger, nullable=True, index=True) + vessel_type_name = Column(String(64), nullable=True, index=True) + flag = Column(String(4), nullable=True, index=True) + length = Column(Float, nullable=True) + width = Column(Float, nullable=True) + draught = Column(Float, nullable=True) + imo = Column(BigInteger, nullable=True) + updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) + + def to_dict(self) -> dict: + return { + "mmsi": self.mmsi, + "name": self.name, + "callsign": self.callsign, + "vessel_type": self.vessel_type, + "vessel_type_name": self.vessel_type_name, + "flag": self.flag, + "length": self.length, + "width": self.width, + "draught": self.draught, + "imo": self.imo, + "updated_at": to_iso8601_utc(self.updated_at), + } + + +class VesselPosition(Base): + """Append-only AIS positions retained for short history windows.""" + + __tablename__ = "vessel_position" + + id = Column(Integer, primary_key=True, autoincrement=True) + mmsi = Column(BigInteger, nullable=False, index=True) + lat = Column(Float, nullable=False) + lon = Column(Float, nullable=False) + sog = Column(Float, nullable=True) + cog = Column(Float, nullable=True) + heading = Column(SmallInteger, nullable=True) + nav_status = Column(SmallInteger, nullable=True, index=True) + received_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True) + + __table_args__ = ( + Index("idx_vessel_pos_mmsi_time", "mmsi", "received_at"), + Index("idx_vessel_pos_time", "received_at"), + Index("idx_vessel_pos_lat_lon", "lat", "lon"), + ) + + def to_dict(self) -> dict: + return { + "id": self.id, + "mmsi": self.mmsi, + "lat": self.lat, + "lon": self.lon, + "sog": self.sog, + "cog": self.cog, + "heading": self.heading, + "nav_status": self.nav_status, + "received_at": to_iso8601_utc(self.received_at), + } diff --git a/backend/app/services/ai_client.py b/backend/app/services/ai_client.py index 916a02d1..ed32bb09 100644 --- a/backend/app/services/ai_client.py +++ b/backend/app/services/ai_client.py @@ -3,9 +3,11 @@ from __future__ import annotations import asyncio import httpx -from fastapi import HTTPException, status +from fastapi import Depends, HTTPException, status +from sqlalchemy.ext.asyncio import AsyncSession from app.core.config import settings +from app.db.session import get_db from app.schemas.ai import ( AIProviderStatusResponse, SituationalAnalysisRequest, @@ -14,11 +16,27 @@ from app.schemas.ai import ( class AIProviderClient: - def __init__(self) -> None: - self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/") - self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN - self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS - self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1) + def __init__( + self, + *, + service_url: str | None = None, + service_token: str | None = None, + timeout: int | None = None, + retry_attempts: int | None = None, + llm_config: dict | None = None, + ) -> None: + self.service_url = ( + service_url if service_url is not None else settings.AI_PROVIDER_SERVICE_URL + ).rstrip("/") + self.service_token = ( + service_token if service_token is not None else settings.AI_PROVIDER_SERVICE_TOKEN + ) + self.timeout = timeout if timeout is not None else settings.AI_PROVIDER_TIMEOUT_SECONDS + self.retry_attempts = max( + retry_attempts if retry_attempts is not None else settings.AI_PROVIDER_RETRY_ATTEMPTS, + 1, + ) + self.llm_config = llm_config or {} def _headers(self, request_id: str | None = None) -> dict[str, str]: headers = {"Content-Type": "application/json"} @@ -26,6 +44,19 @@ class AIProviderClient: headers["X-Provider-Token"] = self.service_token if request_id: headers["X-Request-ID"] = request_id + llm_header_map = { + "provider": "X-AI-Provider", + "provider_api": "X-AI-Provider-API", + "base_url": "X-AI-Base-URL", + "api_key": "X-AI-API-Key", + "model": "X-AI-Model", + "max_tokens": "X-AI-Max-Tokens", + "anthropic_version": "X-AI-Anthropic-Version", + } + for key, header_name in llm_header_map.items(): + value = self.llm_config.get(key) + if value not in (None, ""): + headers[header_name] = str(value) return headers async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse: @@ -105,5 +136,14 @@ class AIProviderClient: ) -def get_ai_provider_client() -> AIProviderClient: - return AIProviderClient() +async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient: + from app.api.v1.settings import get_runtime_ai_provider_config + + runtime_config = await get_runtime_ai_provider_config(db) + return AIProviderClient( + service_url=runtime_config["service_url"], + service_token=runtime_config["service_token"], + timeout=runtime_config["timeout_seconds"], + retry_attempts=runtime_config["retry_attempts"], + llm_config=runtime_config.get("llm_config") or {}, + ) diff --git a/backend/app/services/collectors/__init__.py b/backend/app/services/collectors/__init__.py index add854b0..ebe4507d 100644 --- a/backend/app/services/collectors/__init__.py +++ b/backend/app/services/collectors/__init__.py @@ -36,6 +36,7 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector from app.services.collectors.news_live_streams import NewsLiveStreamsCollector +from app.services.collectors.vessel_ais import VesselAISCollector collector_registry.register(TOP500Collector()) collector_registry.register(EpochAIGPUCollector()) @@ -63,3 +64,4 @@ collector_registry.register(IPtoASNPrefixGeoCollector()) collector_registry.register(OpenGeoFeedPrefixGeoCollector()) collector_registry.register(NRODelegatedPrefixGeoCollector()) collector_registry.register(NewsLiveStreamsCollector()) +collector_registry.register(VesselAISCollector()) diff --git a/backend/app/services/collectors/vessel_ais.py b/backend/app/services/collectors/vessel_ais.py new file mode 100644 index 00000000..601f5bb6 --- /dev/null +++ b/backend/app/services/collectors/vessel_ais.py @@ -0,0 +1,326 @@ +"""BarentsWatch AIS collector for vessel tracking.""" + +from datetime import UTC, datetime, timedelta +import os +from typing import Any + +import httpx +from sqlalchemy import delete, select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.models.vessel import VesselPosition, VesselStatic +from app.services.collectors.base import BaseCollector + + +BARENTSWATCH_LATEST_URL = "https://live.ais.barentswatch.no/v1/latest/combined" +BARENTSWATCH_TOKEN_URL = "https://id.barentswatch.no/connect/token" + + +VESSEL_TYPE_NAMES = { + 30: "Fishing", + 35: "Military", + 60: "Passenger", + 70: "Cargo", + 80: "Tanker", +} + + +class VesselAISCollector(BaseCollector): + """Collect latest AIS positions and append them to vessel tables.""" + + name = "barentswatch_vessels" + priority = "P1" + module = "L4" + frequency_hours = 1 + data_type = "vessel_ais" + + @property + def base_url(self) -> str: + return self._resolved_url or BARENTSWATCH_LATEST_URL + + async def _load_datasource_config(self) -> dict[str, Any]: + if not self._db_session: + return {} + try: + from sqlalchemy import select + from app.models.datasource_config import DataSourceConfig + + result = await self._db_session.execute( + select(DataSourceConfig) + .where(DataSourceConfig.name == self.name) + .where(DataSourceConfig.is_active.is_(True)) + ) + datasource_config = result.scalar_one_or_none() + except Exception: + return {} + + if not datasource_config: + return {} + return { + "auth_config": datasource_config.auth_config or {}, + "config": datasource_config.config or {}, + } + + async def _get_access_token(self, client: httpx.AsyncClient) -> str | None: + datasource_config = await self._load_datasource_config() + auth_config = datasource_config.get("auth_config") or {} + config = datasource_config.get("config") or {} + client_id = ( + auth_config.get("client_id") + or config.get("client_id") + or os.getenv("BARENTSWATCH_CLIENT_ID") + or os.getenv("BARRENTSWATCH_CLIENT_ID") + ) + client_secret = ( + auth_config.get("client_secret") + or config.get("client_secret") + or os.getenv("BARENTSWATCH_CLIENT_SECRET") + or os.getenv("BARRENTSWATCH_CLIENT_SECRET") + ) + if not client_id or not client_secret: + return None + + response = await client.post( + BARENTSWATCH_TOKEN_URL, + data={ + "client_id": client_id, + "client_secret": client_secret, + "scope": "ais", + "grant_type": "client_credentials", + }, + headers={"Content-Type": "application/x-www-form-urlencoded"}, + ) + response.raise_for_status() + payload = response.json() + token = payload.get("access_token") + return str(token) if token else None + + async def fetch(self) -> list[dict[str, Any]]: + async with httpx.AsyncClient(timeout=60.0) as client: + headers: dict[str, str] = {} + token = await self._get_access_token(client) + if token: + headers["Authorization"] = f"Bearer {token}" + + response = await client.get(self.base_url, headers=headers) + if response.status_code == 401 and not token: + return self._get_sample_data() + response.raise_for_status() + payload = response.json() + + if isinstance(payload, list): + return [item for item in payload if isinstance(item, dict)] + if isinstance(payload, dict): + for key in ("features", "data", "items", "vessels"): + value = payload.get(key) + if isinstance(value, list): + if key == "features": + return [ + { + **(item.get("properties") or {}), + "geometry": item.get("geometry"), + } + for item in value + if isinstance(item, dict) + ] + return [item for item in value if isinstance(item, dict)] + return self._get_sample_data() + + def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]: + transformed = [] + for item in raw_data: + record = self._normalize_record(item) + if record: + transformed.append(record) + return transformed + + async def _save_data( + self, + db: AsyncSession, + data: list[dict[str, Any]], + task_id: int | None = None, + snapshot_id: int | None = None, + ) -> int: + now = datetime.now(UTC) + records_added = 0 + + for index, item in enumerate(data): + static = await db.get(VesselStatic, item["mmsi"]) + if static is None: + static = VesselStatic(mmsi=item["mmsi"]) + db.add(static) + + for field in ( + "name", + "callsign", + "vessel_type", + "vessel_type_name", + "flag", + "length", + "width", + "draught", + "imo", + ): + value = item.get(field) + if value not in (None, ""): + setattr(static, field, value) + static.updated_at = now + + db.add( + VesselPosition( + mmsi=item["mmsi"], + lat=item["lat"], + lon=item["lon"], + sog=item.get("sog"), + cog=item.get("cog"), + heading=item.get("heading"), + nav_status=item.get("nav_status"), + received_at=item.get("received_at") or now, + ) + ) + records_added += 1 + + if (index + 1) % 1000 == 0: + await self.update_progress(index + 1, commit=True) + + await db.execute( + delete(VesselPosition).where(VesselPosition.received_at < now - timedelta(hours=24)) + ) + await db.commit() + await self.update_progress(records_added, force=True) + return records_added + + def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None: + mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi")) + lat = _as_float(_pick(item, "lat", "latitude", "Latitude")) + lon = _as_float(_pick(item, "lon", "lng", "longitude", "Longitude")) + + geometry = item.get("geometry") + coordinates = geometry.get("coordinates") if isinstance(geometry, dict) else None + if (lat is None or lon is None) and isinstance(coordinates, list) and len(coordinates) >= 2: + lon = _as_float(coordinates[0]) + lat = _as_float(coordinates[1]) + + if mmsi is None or lat is None or lon is None: + return None + if not (-90 <= lat <= 90 and -180 <= lon <= 180): + return None + + vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType")) + vessel_type_name = ( + _pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName") + or _vessel_type_name(vessel_type) + ) + received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime")) + + return { + "mmsi": mmsi, + "name": _pick(item, "name", "shipName", "ship_name", "Name"), + "callsign": _pick(item, "callsign", "callSign", "CallSign"), + "vessel_type": vessel_type, + "vessel_type_name": vessel_type_name, + "flag": _pick(item, "flag", "country", "Flag"), + "length": _as_float(_pick(item, "length", "shipLength", "Length")), + "width": _as_float(_pick(item, "width", "shipWidth", "Width")), + "draught": _as_float(_pick(item, "draught", "draft", "Draught")), + "imo": _as_int(_pick(item, "imo", "IMO", "imoNumber")), + "lat": lat, + "lon": lon, + "sog": _as_float(_pick(item, "sog", "speedOverGround", "SOG")), + "cog": _as_float(_pick(item, "cog", "courseOverGround", "COG")), + "heading": _as_int(_pick(item, "heading", "trueHeading", "Heading")), + "nav_status": _as_int(_pick(item, "nav_status", "navStatus", "NavigationalStatus")), + "received_at": received_at, + } + + def _get_sample_data(self) -> list[dict[str, Any]]: + return [ + { + "mmsi": 257123000, + "name": "OSLO TRADER", + "lat": 59.91, + "lon": 10.73, + "sog": 12.4, + "cog": 214, + "heading": 215, + "nav_status": 0, + "vessel_type": 70, + "vessel_type_name": "Cargo", + "flag": "NO", + "length": 185, + }, + { + "mmsi": 257456000, + "name": "NORDIC FJORD", + "lat": 60.39, + "lon": 5.32, + "sog": 0.2, + "cog": 82, + "heading": 80, + "nav_status": 1, + "vessel_type": 60, + "vessel_type_name": "Passenger", + "flag": "NO", + "length": 126, + }, + ] + + +def _pick(item: dict[str, Any], *keys: str) -> Any: + for key in keys: + if key in item and item[key] not in (None, ""): + return item[key] + return None + + +def _as_float(value: Any) -> float | None: + try: + if value in (None, ""): + return None + return float(value) + except (TypeError, ValueError): + return None + + +def _as_int(value: Any) -> int | None: + try: + if value in (None, ""): + return None + return int(float(value)) + except (TypeError, ValueError): + return None + + +def _parse_datetime(value: Any) -> datetime | None: + if isinstance(value, datetime): + return value if value.tzinfo else value.replace(tzinfo=UTC) + if not value: + return None + if isinstance(value, (int, float)): + timestamp = float(value) + if timestamp > 10_000_000_000: + timestamp /= 1000 + return datetime.fromtimestamp(timestamp, UTC) + if isinstance(value, str): + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC) + except ValueError: + return None + return None + + +def _vessel_type_name(vessel_type: int | None) -> str: + if vessel_type is None: + return "Other" + if 70 <= vessel_type <= 79: + return "Cargo" + if 80 <= vessel_type <= 89: + return "Tanker" + if 60 <= vessel_type <= 69: + return "Passenger" + if vessel_type == 30: + return "Fishing" + if vessel_type == 35: + return "Military" + return VESSEL_TYPE_NAMES.get(vessel_type, "Other") diff --git a/backend/app/services/datasource_mapping.py b/backend/app/services/datasource_mapping.py new file mode 100644 index 00000000..6877f266 --- /dev/null +++ b/backend/app/services/datasource_mapping.py @@ -0,0 +1,358 @@ +"""Deterministic mapping support for custom data sources.""" + +from __future__ import annotations + +import hashlib +import json +import re +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy.ext.asyncio import AsyncSession + +from app.core.target_schema_registry import TargetSchema, get_target_schema + +SECRET_KEY_PATTERN = re.compile( + r"(token|secret|password|passwd|authorization|api[_-]?key|client[_-]?secret)", + re.IGNORECASE, +) + + +class MappingError(ValueError): + """Raised when a mapping definition cannot be executed.""" + + +def stable_payload_hash(payload: Any) -> str: + encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode() + return hashlib.sha256(encoded).hexdigest() + + +def redact_for_llm(value: Any) -> Any: + if isinstance(value, dict): + redacted = {} + for key, item in value.items(): + if SECRET_KEY_PATTERN.search(str(key)): + redacted[key] = "[REDACTED]" + else: + redacted[key] = redact_for_llm(item) + return redacted + if isinstance(value, list): + return [redact_for_llm(item) for item in value[:20]] + return value + + +def extract_path(payload: Any, path: str | None) -> Any: + if not path or path == "$": + return payload + + normalized = path.strip() + if normalized.startswith("$."): + normalized = normalized[2:] + elif normalized.startswith("$"): + normalized = normalized[1:] + normalized = normalized.strip(".") + if not normalized: + return payload + + current = payload + for raw_segment in normalized.split("."): + segment = raw_segment.strip() + if not segment: + continue + + list_all = segment.endswith("[*]") + if list_all: + segment = segment[:-3] + + index = None + match = re.fullmatch(r"(.+)\[(\d+)\]", segment) + if match: + segment = match.group(1) + index = int(match.group(2)) + + if segment: + if isinstance(current, dict): + current = current.get(segment) + else: + return None + + if list_all: + return current if isinstance(current, list) else [] + + if index is not None: + if not isinstance(current, list) or index >= len(current): + return None + current = current[index] + + return current + + +def _convert_value(value: Any, target_type: str | None) -> Any: + if value is None or target_type in (None, "", "any"): + return value + + if target_type == "string": + return str(value) + if target_type == "integer": + return int(value) + if target_type == "float": + return float(value) + if target_type == "boolean": + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "y", "on"} + return bool(value) + if target_type == "datetime": + if isinstance(value, datetime): + return value + if isinstance(value, (int, float)): + return datetime.fromtimestamp(value) + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")) + return value + if target_type == "object": + if isinstance(value, dict): + return value + raise ValueError("expected object") + if target_type == "array": + if isinstance(value, list): + return value + raise ValueError("expected array") + + return value + + +def _apply_enum(value: Any, enum_map: Any) -> Any: + if not isinstance(enum_map, dict): + return value + key = str(value) + return enum_map.get(key, enum_map.get(value, value)) + + +def _map_one(item: Any, field_mapping: dict[str, Any]) -> tuple[dict[str, Any], list[str]]: + output: dict[str, Any] = {} + errors: list[str] = [] + + for field_name, rule in field_mapping.items(): + if isinstance(rule, str): + rule = {"path": rule} + if not isinstance(rule, dict): + errors.append(f"{field_name}: mapping rule must be an object or path string") + continue + + value = extract_path(item, rule.get("path")) + if value is None and "default" in rule: + value = rule.get("default") + value = _apply_enum(value, rule.get("enum")) + + try: + value = _convert_value(value, rule.get("type")) + except (TypeError, ValueError) as exc: + errors.append(f"{field_name}: failed to convert value {value!r}: {exc}") + continue + + if value is not None or rule.get("include_null", False): + output[field_name] = value + + return output, errors + + +def execute_mapping( + payload: Any, + mapping_json: dict[str, Any], + target_schema: str | TargetSchema, + *, + limit: int | None = None, +) -> dict[str, Any]: + schema = get_target_schema(target_schema) if isinstance(target_schema, str) else target_schema + source = mapping_json.get("source") or {} + fields = mapping_json.get("fields") + if not isinstance(fields, dict) or not fields: + raise MappingError("mapping_json.fields must be a non-empty object") + + items_path = source.get("items_path") or mapping_json.get("items_path") or "$" + items = extract_path(payload, items_path) + if isinstance(items, dict): + items = [items] + elif not isinstance(items, list): + items = [] + + if limit is not None: + items = items[:limit] + + mapped_records: list[dict[str, Any]] = [] + errors: list[dict[str, Any]] = [] + for index, item in enumerate(items): + mapped, mapping_errors = _map_one(item, fields) + validated, validation_errors = schema.validate_record(mapped) + all_errors = mapping_errors + validation_errors + if all_errors: + errors.append({"index": index, "errors": all_errors, "record": mapped}) + continue + if validated is not None: + mapped_records.append(validated) + + return { + "target_schema": schema.key, + "total_items": len(items), + "mapped_count": len(mapped_records), + "failed_count": len(errors), + "records": mapped_records, + "errors": errors, + } + + +def build_heuristic_mapping(sample_payload: Any, target_schema_key: str) -> dict[str, Any]: + schema = get_target_schema(target_schema_key) + items_path = "$" + sample_item = sample_payload + if isinstance(sample_payload, dict): + for key in ("data", "items", "results", "features", "vessels"): + candidate = sample_payload.get(key) + if isinstance(candidate, list) and candidate: + items_path = f"$.{key}[*]" + sample_item = candidate[0] + break + elif isinstance(sample_payload, list) and sample_payload: + items_path = "$" + sample_item = sample_payload[0] + + available = _flatten_keys(sample_item if isinstance(sample_item, dict) else {}) + fields: dict[str, Any] = {} + for field in schema.fields: + candidate = _best_field_match(field.name, available) + if candidate: + fields[field.name] = {"path": f"$.{candidate}", "type": field.type} + elif field.name == "data" and target_schema_key == "generic_records": + fields[field.name] = {"path": "$", "type": "object"} + elif not field.required: + fields[field.name] = {"path": f"$.{field.name}", "type": field.type, "default": None} + + return { + "source": {"items_path": items_path}, + "fields": fields, + "meta": { + "generated_by": "heuristic", + "requires_review": True, + }, + } + + +def _flatten_keys(payload: dict[str, Any], prefix: str = "") -> list[str]: + keys: list[str] = [] + for key, value in payload.items(): + dotted = f"{prefix}.{key}" if prefix else str(key) + keys.append(dotted) + if isinstance(value, dict): + keys.extend(_flatten_keys(value, dotted)) + return keys + + +def _best_field_match(field_name: str, candidates: list[str]) -> str | None: + aliases = { + "lat": ("lat", "latitude", "y"), + "lon": ("lon", "lng", "longitude", "x"), + "mmsi": ("mmsi",), + "sog": ("sog", "speed", "speedOverGround"), + "cog": ("cog", "course", "courseOverGround"), + "received_at": ("received_at", "timestamp", "time", "updated_at"), + "observed_at": ("observed_at", "timestamp", "time", "updated_at"), + "source_id": ("id", "source_id", "uuid"), + }.get(field_name, (field_name,)) + + lowered = {candidate.lower(): candidate for candidate in candidates} + for alias in aliases: + if alias.lower() in lowered: + return lowered[alias.lower()] + for candidate in candidates: + tail = candidate.split(".")[-1].lower() + if tail in {alias.lower() for alias in aliases}: + return candidate + return None + + +def _parse_datetime(value: Any) -> datetime | None: + if value is None: + return None + if isinstance(value, datetime): + return value + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")) + return None + + +async def persist_mapped_records( + db: AsyncSession, + *, + datasource_name: str, + datasource_config_id: int, + target_schema: str, + records: list[dict[str, Any]], + mapping_version: int, +) -> int: + """Persist validated mapped records to the destination for a target schema.""" + if target_schema == "vessel_ais": + from app.models.vessel import VesselPosition + + for record in records: + db.add( + VesselPosition( + mmsi=record["mmsi"], + lat=record["lat"], + lon=record["lon"], + sog=record.get("sog"), + cog=record.get("cog"), + heading=record.get("heading"), + received_at=_parse_datetime(record.get("received_at")) or datetime.now(UTC), + ) + ) + await db.commit() + return len(records) + + from app.models.collected_data import CollectedData + + collected_at = datetime.now(UTC) + for index, record in enumerate(records): + if target_schema == "geo_points": + source_id = record.get("source_id") or f"{datasource_config_id}:{index}" + name = record.get("name") + metadata = { + "latitude": record.get("lat"), + "longitude": record.get("lon"), + "type": record.get("type"), + "mapping_version": mapping_version, + "target_schema": target_schema, + **(record.get("metadata") or {}), + } + reference_date = _parse_datetime(record.get("observed_at")) + else: + source_id = record.get("source_id") or f"{datasource_config_id}:{index}" + name = None + metadata = { + "data": record.get("data") or {}, + "mapping_version": mapping_version, + "target_schema": target_schema, + } + reference_date = _parse_datetime(record.get("observed_at")) + + db.add( + CollectedData( + source=datasource_name, + source_id=str(source_id), + entity_key=f"{datasource_name}:{source_id}", + data_type=target_schema, + name=name, + title=name, + extra_data=metadata, + collected_at=collected_at, + reference_date=reference_date, + is_valid=1, + is_current=True, + change_type="created", + change_summary={}, + ) + ) + + await db.commit() + return len(records) diff --git a/backend/app/services/llm_provider_catalog.py b/backend/app/services/llm_provider_catalog.py new file mode 100644 index 00000000..89da1a2d --- /dev/null +++ b/backend/app/services/llm_provider_catalog.py @@ -0,0 +1,150 @@ +"""LLM provider presets used by Settings and the runtime AI provider bridge.""" + +from __future__ import annotations + +from typing import Any + +import httpx + +MODELS_DEV_URL = "https://models.dev/api.json" + + +FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = { + "minimax": { + "provider": "minimax", + "label": "MiniMax", + "provider_api": "anthropic-messages", + "base_url": "https://api.minimaxi.com/anthropic", + "model": "MiniMax-M2.7", + "models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"], + "api_key_env": "MINIMAX_API_KEY", + "source": "fallback", + }, + "openai": { + "provider": "openai", + "label": "OpenAI", + "provider_api": "openai-completions", + "base_url": "https://api.openai.com/v1", + "model": "gpt-5.1", + "models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"], + "api_key_env": "OPENAI_API_KEY", + "source": "fallback", + }, + "anthropic": { + "provider": "anthropic", + "label": "Anthropic", + "provider_api": "anthropic-messages", + "base_url": "https://api.anthropic.com/v1", + "model": "claude-sonnet-4-6", + "models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"], + "api_key_env": "ANTHROPIC_API_KEY", + "source": "fallback", + }, + "deepseek": { + "provider": "deepseek", + "label": "DeepSeek", + "provider_api": "openai-completions", + "base_url": "https://api.deepseek.com/v1", + "model": "deepseek-chat", + "models": ["deepseek-chat", "deepseek-reasoner"], + "api_key_env": "DEEPSEEK_API_KEY", + "source": "fallback", + }, + "alibaba": { + "provider": "alibaba", + "label": "Alibaba Qwen / DashScope", + "provider_api": "openai-completions", + "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1", + "model": "qwen3-max", + "models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"], + "api_key_env": "DASHSCOPE_API_KEY", + "source": "fallback", + }, + "moonshotai": { + "provider": "moonshotai", + "label": "Moonshot AI / Kimi", + "provider_api": "openai-completions", + "base_url": "https://api.moonshot.ai/v1", + "model": "kimi-k2.5", + "models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"], + "api_key_env": "MOONSHOT_API_KEY", + "source": "fallback", + }, + "openrouter": { + "provider": "openrouter", + "label": "OpenRouter", + "provider_api": "openai-completions", + "base_url": "https://openrouter.ai/api/v1", + "model": "openai/gpt-5.1", + "models": ["openai/gpt-5.1", "anthropic/claude-sonnet-4.5", "qwen/qwen3-max"], + "api_key_env": "OPENROUTER_API_KEY", + "source": "fallback", + }, + "ollama": { + "provider": "ollama", + "label": "Ollama Local", + "provider_api": "ollama-generate", + "base_url": "http://127.0.0.1:11434", + "model": "qwen2.5:7b", + "models": ["qwen2.5:7b", "llama3.1:8b", "mistral:7b"], + "api_key_env": "", + "source": "fallback", + }, +} + +MODELS_DEV_PROVIDER_KEYS = { + "minimax": "minimax", + "openai": "openai", + "anthropic": "anthropic", + "deepseek": "deepseek", + "alibaba": "alibaba", + "moonshotai": "moonshotai", + "openrouter": "openrouter", +} + + +def list_fallback_llm_provider_presets() -> list[dict[str, Any]]: + return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()] + + +def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]: + key = provider.strip().lower() + if key not in FALLBACK_LLM_PROVIDER_PRESETS: + raise ValueError(f"Unsupported LLM provider preset: {provider}") + return dict(FALLBACK_LLM_PROVIDER_PRESETS[key]) + + +async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]: + fallback = get_fallback_llm_provider_preset(provider) + models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"]) + if not models_dev_key: + return fallback + + async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client: + response = await client.get( + MODELS_DEV_URL, + headers={"User-Agent": "Planet/1.0"}, + ) + response.raise_for_status() + catalog = response.json() + + upstream = catalog.get(models_dev_key) + if not isinstance(upstream, dict): + return fallback + + upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {} + model_ids = list(upstream_models.keys())[:80] + base_url = upstream.get("api") or fallback["base_url"] + if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com": + base_url = "https://api.deepseek.com/v1" + + refreshed = { + **fallback, + "label": upstream.get("name") or fallback["label"], + "base_url": base_url, + "model": model_ids[0] if model_ids else fallback["model"], + "models": model_ids or fallback["models"], + "api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0], + "source": MODELS_DEV_URL, + } + return refreshed diff --git a/backend/tests/test_datasource_mapping.py b/backend/tests/test_datasource_mapping.py new file mode 100644 index 00000000..9ccb8c4e --- /dev/null +++ b/backend/tests/test_datasource_mapping.py @@ -0,0 +1,199 @@ +from types import SimpleNamespace + +import pytest +from httpx import ASGITransport, AsyncClient + +from app.api.v1.datasource_config import get_ai_provider_client +from app.core.security import get_current_user +from app.core.target_schema_registry import get_target_schema, list_target_schemas +from app.main import app +from app.models.user import User +from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm + + +SAMPLE_AIS = { + "data": [ + { + "mmsi": "257123000", + "latitude": "59.91", + "longitude": "10.75", + "speedOverGround": "12.4", + "timestamp": "2026-04-28T00:00:00Z", + "api_token": "secret-value", + } + ] +} + + +def test_registry_exposes_v1_target_schemas(): + keys = {schema["key"] for schema in list_target_schemas()} + + assert {"vessel_ais", "geo_points", "generic_records"}.issubset(keys) + assert get_target_schema("vessel_ais").destination == "vessel_position" + + +def test_mapping_engine_maps_and_validates_vessel_ais(): + mapping = { + "source": {"items_path": "$.data[*]"}, + "fields": { + "mmsi": {"path": "$.mmsi", "type": "integer"}, + "lat": {"path": "$.latitude", "type": "float"}, + "lon": {"path": "$.longitude", "type": "float"}, + "sog": {"path": "$.speedOverGround", "type": "float"}, + "received_at": {"path": "$.timestamp", "type": "datetime"}, + }, + } + + result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais") + + assert result["mapped_count"] == 1 + assert result["failed_count"] == 0 + assert result["records"][0]["mmsi"] == 257123000 + assert result["records"][0]["lat"] == 59.91 + + +def test_mapping_engine_reports_schema_errors(): + mapping = { + "source": {"items_path": "$.data[*]"}, + "fields": { + "mmsi": {"path": "$.mmsi", "type": "integer"}, + "lat": {"path": "$.missing_lat", "type": "float"}, + "lon": {"path": "$.longitude", "type": "float"}, + }, + } + + result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais") + + assert result["mapped_count"] == 0 + assert result["failed_count"] == 1 + assert any("lat" in error for error in result["errors"][0]["errors"]) + + +def test_redact_for_llm_masks_secret_like_fields(): + redacted = redact_for_llm(SAMPLE_AIS) + + assert redacted["data"][0]["api_token"] == "[REDACTED]" + + +@pytest.mark.asyncio +async def test_persist_mapped_records_writes_generic_records(): + class FakeDB: + def __init__(self): + self.added = [] + self.committed = False + + def add(self, value): + self.added.append(value) + + async def commit(self): + self.committed = True + + db = FakeDB() + + count = await persist_mapped_records( + db, + datasource_name="custom_weather", + datasource_config_id=42, + target_schema="generic_records", + records=[{"source_id": "row-1", "data": {"temp": 25}}], + mapping_version=3, + ) + + assert count == 1 + assert db.committed is True + assert db.added[0].source == "custom_weather" + assert db.added[0].data_type == "generic_records" + assert db.added[0].extra_data["mapping_version"] == 3 + + +@pytest.mark.asyncio +async def test_mapping_preview_api_uses_deterministic_engine(): + def override_get_current_user(): + return User( + id=1, + username="testuser", + email="test@example.com", + password_hash="hashed", + role="admin", + is_active=True, + ) + + app.dependency_overrides = {get_current_user: override_get_current_user} + transport = ASGITransport(app=app) + try: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.post( + "/api/v1/datasources/mappings/preview", + json={ + "sample_payload": SAMPLE_AIS, + "target_schema": "vessel_ais", + "mapping_json": { + "source": {"items_path": "$.data[*]"}, + "fields": { + "mmsi": {"path": "$.mmsi", "type": "integer"}, + "lat": {"path": "$.latitude", "type": "float"}, + "lon": {"path": "$.longitude", "type": "float"}, + }, + }, + }, + ) + finally: + app.dependency_overrides.clear() + + assert response.status_code == 200 + payload = response.json() + assert payload["success"] is True + assert payload["preview"]["records"][0]["mmsi"] == 257123000 + + +@pytest.mark.asyncio +async def test_mapping_propose_api_redacts_sample_before_ai(): + seen_context = {} + + class FakeAIClient: + async def analyze(self, request, request_id=None): + seen_context.update(request.context) + return SimpleNamespace( + content=( + '{"source":{"items_path":"$.data[*]"},"fields":{' + '"mmsi":{"path":"$.mmsi","type":"integer"},' + '"lat":{"path":"$.latitude","type":"float"},' + '"lon":{"path":"$.longitude","type":"float"}}}' + ) + ) + + def override_get_current_user(): + return User( + id=1, + username="testuser", + email="test@example.com", + password_hash="hashed", + role="admin", + is_active=True, + ) + + def override_ai_client(): + return FakeAIClient() + + app.dependency_overrides = { + get_current_user: override_get_current_user, + get_ai_provider_client: override_ai_client, + } + transport = ASGITransport(app=app) + try: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.post( + "/api/v1/datasources/mappings/propose", + json={ + "sample_payload": SAMPLE_AIS, + "target_schema": "vessel_ais", + "use_ai": True, + }, + ) + finally: + app.dependency_overrides.clear() + + assert response.status_code == 200 + payload = response.json() + assert payload["mapping_json"]["meta"]["generated_by"] == "ai_provider" + assert seen_context["sample_payload"]["data"][0]["api_token"] == "[REDACTED]" diff --git a/backend/tests/test_vessels.py b/backend/tests/test_vessels.py new file mode 100644 index 00000000..2ab28aef --- /dev/null +++ b/backend/tests/test_vessels.py @@ -0,0 +1,105 @@ +from datetime import datetime, timedelta, timezone + +import pytest +from httpx import ASGITransport, AsyncClient + +from app.api.v1.visualization import convert_vessels_to_geojson +from app.db.session import get_db +from app.main import app +from app.models.vessel import VesselPosition, VesselStatic +from app.services.collectors.vessel_ais import VesselAISCollector + + +def test_vessel_collector_transforms_barentswatch_like_records(): + collector = VesselAISCollector() + records = collector.transform( + [ + { + "mmsi": "257123000", + "lat": "59.91", + "lon": "10.73", + "sog": 12.4, + "cog": 214, + "nav_status": 0, + "shipType": 70, + "name": "OSLO TRADER", + }, + {"mmsi": "bad", "lat": 120, "lon": 10}, + ] + ) + + assert len(records) == 1 + assert records[0]["mmsi"] == 257123000 + assert records[0]["vessel_type_name"] == "Cargo" + assert records[0]["lat"] == pytest.approx(59.91) + + +def test_convert_vessels_to_geojson(): + position = VesselPosition( + mmsi=257123000, + lat=59.91, + lon=10.73, + sog=12.4, + cog=214, + heading=215, + nav_status=0, + received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc), + ) + static = VesselStatic( + mmsi=257123000, + name="OSLO TRADER", + vessel_type=70, + vessel_type_name="Cargo", + flag="NO", + length=185, + ) + + payload = convert_vessels_to_geojson([(position, static)]) + + assert payload["type"] == "FeatureCollection" + assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91] + assert payload["features"][0]["properties"]["mmsi"] == 257123000 + assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo" + + +@pytest.mark.asyncio +async def test_vessels_geojson_endpoint_filters_type_and_bbox(): + now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc) + rows = [ + ( + VesselPosition(mmsi=1, lat=59.9, lon=10.7, received_at=now), + VesselStatic(mmsi=1, name="Cargo Ship", vessel_type=70, vessel_type_name="Cargo"), + ), + ( + VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now - timedelta(minutes=1)), + VesselStatic(mmsi=2, name="Passenger Ship", vessel_type=60, vessel_type_name="Passenger"), + ), + ] + + class _Result: + def all(self): + return rows + + class _FakeSession: + async def execute(self, _query): + return _Result() + + async def override_get_db(): + yield _FakeSession() + + app.dependency_overrides[get_db] = override_get_db + transport = ASGITransport(app=app) + try: + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get( + "/api/v1/visualization/geo/vessels", + params={"bbox": "0,50,20,70", "type": "cargo"}, + ) + + assert response.status_code == 200 + data = response.json() + assert data["count"] == 1 + assert data["features"][0]["properties"]["name"] == "Cargo Ship" + assert data["stats"]["by_type"]["Cargo"] == 1 + finally: + app.dependency_overrides.clear() diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 607cad24..92947652 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -8,6 +8,21 @@ This project follows the repository versioning rule: - `improvement` -> `+0.0.1`（bugfix + 小功能混合） - `bugfix` -> `+0.0.1` +## [0.43.0] — 2026-04-28 + +### ✨ Highlights +- 新增 Earth 船舶追踪链路，接入 BarentsWatch AIS 凭证配置、采集器、后端 vessel 模型/API 与前端 Earth 船舶图层 +- 新增自定义数据源映射流程，支持样本抓取、目标 schema、AI 辅助生成映射、预览校验和映射执行 +- Settings 拆分 AI Provider 与采集器凭证配置，DataSources 只保留采集状态、运行参数和必要引导 + +### 🔧 Improvements +- AI Provider 支持运行时 LLM 配置、provider preset 下拉与刷新，并在 Playground 中引导到 AI 配置页 +- Markdown 渲染器补齐代码块复制按钮、语言标签、任务列表、图片、自动链接、删除线和文档主题样式 +- Docs 公开导航改为显式元数据白名单，避免开发任务文档自动出现在“其他”分组 +- 将 Codex/Claude cleanup、docs、goal-driven、release 流程补充 CLI-first 约束，并把 `rules.md` 整理成可按模块加载的工程规则 + +--- + ## [0.42.2] — 2026-04-28 ### 🐛 Fixes diff --git a/docs/plans/datasource-custom-api-mapping-plan.md b/docs/plans/datasource-custom-api-mapping-plan.md new file mode 100644 index 00000000..f4aacc25 --- /dev/null +++ b/docs/plans/datasource-custom-api-mapping-plan.md @@ -0,0 +1,424 @@ +# 自定义 API 数据源与 LLM 映射系统 — 实施计划 + +**状态**：规划中 +**创建日期**：2026-04-28 +**核心原则**：LLM 辅助生成映射配置；生产采集使用确定性转换引擎 + +## 已确认决策 + +| 项目 | 决策 | +|-----|------| +| 自定义 API 的定位 | 作为内置数据源的补充入口，不直接等同于 Earth 新功能 | +| LLM 的职责 | 探索未知 API、分析样本 JSON、生成 mapping 草案 | +| 采集时是否调用 LLM | 不调用；采集链路必须确定性、可审计、可复现 | +| 自定义数据如何进入 Earth | 必须映射到已支持的目标 schema，或先进入通用数据沉淀 | +| 外部凭证放置位置 | Settings / 外部集成统一管理 provider token；DataSources 引用 provider profile | +| TimescaleDB | 放入 TODO；高频时序数据稳定后再评估迁移 | + +--- + +## 一、背景与问题 + +当前系统已经有 `datasource_configs`，可以配置自定义数据源的 endpoint、auth、headers、config，也已经有部分 collector 会读取这些配置。但这只能解决“怎么请求数据”，还没有解决以下问题： + +- API 返回 JSON 后，如何转换成系统已有领域模型。 +- 自定义数据源是补充已有能力，还是全新数据沉淀。 +- 转换规则由谁生成、谁校验、谁执行。 +- 未知数据是否能自动在 Earth 上展示。 +- 外部 token 是放在全局配置中心，还是放在每个 datasource 下。 + +专业做法是把“请求配置”“外部凭证”“目标 schema”“字段映射”“采集执行”拆开： + +- Settings 管外部集成凭证，例如 AI Provider、BarentsWatch、未来付费 AIS API。 +- DataSources 管具体数据源实例，例如 endpoint、调度频率、目标 schema、mapping 版本。 +- LLM 只在配置阶段辅助生成 mapping，不进入生产采集链路。 +- Earth 只消费明确 schema 的数据，不消费任意未知 JSON。 + +--- + +## 二、目标架构 + +### 2.1 自定义 API 数据源生命周期 + +```mermaid +flowchart LR + A[配置 endpoint/auth/request] --> B[抓取 sample JSON] + B --> C[选择目标 schema] + C --> D[LLM 生成 mapping 草案] + D --> E[确定性 mapping engine 预览] + E --> F[schema validation] + F --> G[保存 mapping version] + G --> H[scheduler 执行 mapped collector] + H --> I[写入目标表或 generic_records] +``` + +### 2.2 目标 schema 分层 + +| schema | 用途 | Earth 可视化 | +|-------|------|-------------| +| `vessel_ais` | 船只 AIS 位置、航速、航向、MMSI 等 | 进入船舶图层 | +| `geo_points` | 通用点位数据，包含经纬度、名称、类型、时间 | 进入通用 geo layer（TODO） | +| `news_events` | 新闻/事件类数据，带时间、地点、摘要、来源 | 复用新闻/事件链路 | +| `compute_centers` | 算力中心、机房、数据中心数据 | 复用算力中心图层 | +| `generic_records` | 未知结构化数据沉淀 | 不直接展示 | + +v1 建议优先实现： + +- `vessel_ais` +- `geo_points` +- `generic_records` + +其他 schema 可先在 registry 中预留名称，但不承诺完整落库与可视化。 + +### 2.3 LLM 的边界 + +LLM 可以做： + +- 根据 API 文档或 sample JSON 解释字段含义。 +- 推荐目标 schema。 +- 生成 mapping JSON 草案。 +- 给出字段置信度和需要人工确认的字段。 +- 帮用户发现分页、数组路径、时间字段、坐标字段。 + +LLM 不应该做： + +- 在正式采集时参与每批数据转换。 +- 生成并执行 Python/JavaScript 代码。 +- 接触 API key、bearer token、basic auth password。 +- 自动创建新的 Earth 图层或数据库表。 + +--- + +## 三、后端实施计划 + +### Phase 1 — Target Schema Registry + +新增代码级 registry，统一描述系统支持的目标数据类型。 + +每个 target schema 至少包含： + +- `key`：例如 `vessel_ais`。 +- `label`：前端展示名称。 +- `description`：适用场景。 +- `fields`：字段名、类型、是否必填、说明、示例。 +- `validator`：Pydantic 或等价校验器。 +- `destination`：写入目标，例如 vessel 表、generic_records、future geo layer。 + +示例概念： + +```json +{ + "key": "vessel_ais", + "fields": [ + {"name": "mmsi", "type": "integer", "required": true}, + {"name": "lat", "type": "float", "required": true}, + {"name": "lon", "type": "float", "required": true}, + {"name": "sog", "type": "float", "required": false}, + {"name": "cog", "type": "float", "required": false}, + {"name": "received_at", "type": "datetime", "required": false} + ] +} +``` + +### Phase 2 — Mapping Template Model + +新增 mapping 配置持久化表，建议命名为 `datasource_mapping_templates`。 + +关键字段： + +- `id` +- `datasource_config_id` +- `target_schema` +- `mapping_json` +- `sample_payload_hash` +- `validation_status` +- `version` +- `is_active` +- `created_at` +- `updated_at` + +`mapping_json` 是声明式 DSL，不允许任意代码执行。 + +示例： + +```json +{ + "source": { + "items_path": "$.data.vessels[*]" + }, + "fields": { + "mmsi": {"path": "$.mmsi", "type": "integer"}, + "lat": {"path": "$.latitude", "type": "float"}, + "lon": {"path": "$.longitude", "type": "float"}, + "sog": {"path": "$.speedOverGround", "type": "float", "default": null}, + "received_at": {"path": "$.timestamp", "type": "datetime"} + } +} +``` + +### Phase 3 — Deterministic Mapping Engine + +实现独立 mapping engine，输入 sample/raw payload 和 mapping JSON，输出目标 schema 记录。 + +v1 支持能力： + +- JSONPath/JMESPath 风格路径提取。 +- 数组展开。 +- 默认值。 +- 基础类型转换：string、integer、float、boolean、datetime。 +- 坐标范围校验。 +- 简单枚举映射。 +- 错误收集：缺字段、类型转换失败、路径不存在。 + +明确不支持： + +- 任意表达式执行。 +- 用户提交脚本。 +- LLM runtime 修复。 + +### Phase 4 — LLM Mapping Assistant API + +新增配置阶段 API： + +- `POST /api/v1/datasources/custom/sample` + - 按 datasource 请求配置抓取 sample JSON。 +- `GET /api/v1/datasources/target-schemas` + - 返回可选目标 schema 和字段说明。 +- `POST /api/v1/datasources/mappings/propose` + - 输入 sample JSON + target schema，调用 AI provider 生成 mapping 草案。 +- `POST /api/v1/datasources/mappings/preview` + - 使用确定性 mapping engine 预览转换结果。 +- `POST /api/v1/datasources/mappings` + - 保存 mapping 版本。 +- `PUT /api/v1/datasources/mappings/{id}` + - 更新 mapping，生成新版本或覆盖草稿。 +- `POST /api/v1/datasources/{id}/run-mapped` + - 手动触发一次 mapped collector。 + +安全要求： + +- `propose` 请求发送给 LLM 前必须脱敏 sample。 +- auth headers、token、password 不进入 prompt。 +- LLM 返回结果必须再经过 mapping schema 校验。 + +### Phase 5 — Generic Mapped HTTP Collector + +新增通用 collector： + +- 读取 `DataSourceConfig` 请求配置。 +- 读取 active mapping template。 +- 拉取 API 数据。 +- 使用 mapping engine 转换。 +- 使用 target schema validator 校验。 +- 调用 destination handler 写入目标表或 generic storage。 +- 将失败记录写入错误日志或 dead-letter 结构。 + +对于 `generic_records`： + +- 保存 datasource id。 +- 保存 target schema。 +- 保存 normalized JSON。 +- 保存 raw payload 摘要或 raw reference。 +- 保存采集时间、source timestamp、mapping version。 + +--- + +## 四、前端实施计划 + +### Phase 1 — Settings 外部集成 + +Settings 中保留统一外部集成配置： + +- AI Provider：base URL、model、API key。 +- BarentsWatch：client id/client secret 或 bearer token。 +- 未来付费接口：AISHub、MarineTraffic、VesselFinder 等 provider profile。 + +DataSources 不直接管理全局 secret，只引用 provider profile。 + +### Phase 2 — DataSources 自定义源向导 + +自定义数据源配置改成向导或右侧 drawer： + +1. Request + - endpoint + - method + - auth profile + - headers + - query/body config + - schedule +2. Sample + - 点击抓取 sample + - 展示 JSON tree + - 支持选择数组根路径 +3. Target Schema + - 选择 `vessel_ais`、`geo_points`、`generic_records` + - 展示该 schema 必填字段 +4. Mapping Proposal + - 调用 LLM 生成 mapping 草案 + - 显示字段匹配置信度 + - 标出需要人工确认的字段 +5. Preview + - 用确定性 engine 预览前 N 条转换结果 + - 展示校验错误 +6. Save & Enable + - 保存 mapping version + - 启用调度或仅保存草稿 + +### Phase 3 — 运维视图 + +为 mapped datasource 展示： + +- 上次运行时间。 +- 成功记录数。 +- 失败记录数。 +- 当前 mapping version。 +- 目标 schema。 +- 最近错误。 +- 手动运行按钮。 + +--- + +## 五、数据库与存储策略 + +### v1：继续使用 PostgreSQL + +PostgreSQL 可以承载当前规模的采集、关系查询、JSONB 沉淀和基础时序查询。v1 不必因为“时序数据”立刻引入 TimescaleDB。 + +适合继续用 PostgreSQL 的场景： + +- 数据量可控。 +- 最近状态查询为主。 +- 历史保留窗口较短。 +- 查询模式还没稳定。 +- 需要快速迭代 schema 与 mapping。 + +### TODO：TimescaleDB + +以下条件满足后，再评估 TimescaleDB： + +- AIS、遥测、轨迹类数据达到高频持续写入。 +- 需要按时间窗口做聚合、降采样、retention policy。 +- 单表时间序列查询明显成为瓶颈。 +- 历史轨迹保留从 24h 扩展到数周或数月。 + +候选迁移对象： + +- `vessel_position` +- future telemetry tables +- future generic time-series records + +备选方案： + +- PostgreSQL 原生按天/月分区。 +- TimescaleDB hypertable。 +- 热数据 PostgreSQL，冷数据对象存储。 + +--- + +## 六、安全与治理 + +### Secret 管理 + +- Settings 中保存 provider credentials。 +- API 返回配置时必须 mask secret。 +- LLM prompt 只能包含脱敏 sample 和 schema 说明。 +- 后续 TODO：引入字段级加密或 KMS。 + +### Mapping 治理 + +- 每次 mapping 变更保留版本。 +- active mapping 只能有一个。 +- 允许保存 draft mapping。 +- 运行记录关联 mapping version。 +- 校验失败不能自动启用。 + +### 错误处理 + +常见错误类型： + +- API 401/403：凭证错误或过期。 +- API 429：限流，需要调整 schedule。 +- JSON path 不存在：上游结构变化。 +- 类型转换失败：mapping 规则错误。 +- schema validation failed：转换结果不满足目标模型。 + +每次运行需要记录： + +- datasource id。 +- mapping version。 +- started_at / finished_at。 +- fetched count。 +- mapped count。 +- written count。 +- failed count。 +- error summary。 + +--- + +## 七、测试计划 + +### Backend Unit Tests + +- mapping engine： + - path 提取。 + - 数组展开。 + - 默认值。 + - 类型转换。 + - datetime parse。 + - 枚举映射。 + - 缺字段错误。 +- target schema registry： + - `vessel_ais` 必填字段校验。 + - `geo_points` 经纬度范围校验。 + - `generic_records` 接受未知结构。 +- LLM assistant： + - mock provider 返回 mapping。 + - 验证 secret 不进入 prompt。 + - 验证非法 mapping 被拒绝。 + +### Backend Integration Tests + +- sample JSON -> propose mapping -> preview -> save mapping。 +- mapped collector 使用保存的 mapping 写入 `generic_records`。 +- `vessel_ais` sample 写入船舶相关目标结构。 +- 上游 JSON 结构变化时，运行失败并记录错误。 + +### Frontend Tests + +- 自定义数据源向导完整流程。 +- 未配置 AI Provider 时，提示去 Settings 配置，但允许手写 mapping。 +- LLM 返回不完整 mapping 时，Preview 阶段显示校验错误。 +- 保存 mapping 后展示 active version 和运行状态。 + +--- + +## 八、分期工作量 + +| 阶段 | 内容 | 估算 | +|-----|------|------| +| Phase 0 | 完成本规划、确认 schema registry 设计 | 0.5 天 | +| Phase 1 | target schema registry + mapping template model | 1–2 天 | +| Phase 2 | deterministic mapping engine | 2–3 天 | +| Phase 3 | sample/propose/preview/save API | 2–3 天 | +| Phase 4 | DataSources 自定义源向导 | 3–5 天 | +| Phase 5 | generic mapped collector + run history | 2–4 天 | +| Phase 6 | vessel_ais / geo_points destination handler | 2–4 天 | + +--- + +## 九、当前差距与下一步 + +当前差距： + +- `datasource_configs` 只描述请求配置，不描述目标 schema 和 mapping。 +- 自定义源没有 sample -> schema -> mapping -> preview -> save 的闭环。 +- 生产采集还没有通用 mapped collector。 +- Settings 与 DataSources 的职责边界需要在 UI 上进一步明确。 +- Earth 还没有通用 `geo_points` 图层。 + +下一步建议： + +1. 先实现 target schema registry 和 mapping engine，不急着接 LLM。 +2. 用固定 sample JSON 做 `vessel_ais` 和 `generic_records` 的单元测试。 +3. 再接 LLM propose API，让 LLM 产出的只是 mapping 草案。 +4. 最后做前端向导，把人工确认和 preview 放到启用之前。 diff --git a/docs/plans/earth-vessel-tracking-plan.md b/docs/plans/earth-vessel-tracking-plan.md new file mode 100644 index 00000000..c164dc1e --- /dev/null +++ b/docs/plans/earth-vessel-tracking-plan.md @@ -0,0 +1,272 @@ +# 实时船只监控系统 — 实施计划 + +**状态**：规划中 +**创建日期**：2026-04-27 +**优先数据源**：BarentsWatch（免费）→ AISHub / MarineTraffic（TODO，付费） + +## 已确认决策 + +| 项目 | 决策 | +|-----|------| +| 数据源 | BarentsWatch 先行；AISHub / MarineTraffic TODO | +| 船只规模 | BarentsWatch 阶段全部显示；全球数据接入后按需加船型过滤（默认 Cargo + Tanker + Passenger） | +| 更新频率 | 准实时：前端 5 分钟轮询，后端 Collector 每分钟拉取写库 | +| 历史轨迹 | 保留（`vessel_position` 表保留 24h，后期按需扩展） | +| 推送方式 | HTTP 轮询（不用 WebSocket）；换实时数据源后再评估升级 | + +--- + +## 一、技术背景 + +船只通过 AIS（自动识别系统）每 2–10 秒广播位置、航速、航向、目的地等信息。全球约 50 万艘持证船只在线，实时数据通过以下方式获取： + +| 来源类型 | 典型服务 | 覆盖范围 | 成本 | 状态 | +|---------|---------|---------|------|------| +| **BarentsWatch Open API** | live.ais.barentswatch.no | 挪威海域实时 | 完全免费 | **当前使用** | +| **AISHub** | aishub.net | 全球实时 | 免费/小额 | TODO：付费接入 | +| **MarineTraffic API** | marinetraffic.com | 全球实时 | $50–$500/月 | TODO：评估 tier | +| **VesselFinder API** | vesselfinder.com | 全球实时 | $50–$300/月 | TODO：备选 | +| **自建 SDR 接收** | RTL-SDR + AIS-catcher | 仅本地 30–50km | 硬件 $30 | 不考虑 | +| **NOAA 历史数据** | Marine Cadastre | 美国近海历史 | 免费 | 可用于冷启动 | + +### BarentsWatch API + +- 端点：`https://live.ais.barentswatch.no/v1/latest/combined` +- 无需注册，直接 GET，返回挪威近海 2000–5000 艘船只 JSON +- 字段：mmsi, lat, lon, sog, cog, heading, nav_status, name, vessel_type, flag +- 刷新频率：数据约 30–60s 更新一次，可随意轮询 + +### TODO：付费数据源接入 + +- [ ] 评估 AISHub 订阅（全球覆盖，约 $30/月），接入全球实时流 +- [ ] 评估 MarineTraffic API tier，对比 AISHub 数据质量与成本 +- [ ] 实现多数据源适配器，通过 `datasource_config` 切换 +- [ ] 真实高频 AIS 稳定接入后，评估将 `vessel_position` 迁移为 TimescaleDB hypertable（保留 Postgres 原生分区作为备选） + +--- + +## 二、实施计划 + +### Phase 0 — 数据源验证与链路打通（1–2 天） + +- 接入 BarentsWatch Open API，验证数据格式与字段 +- 构建全球 mock 数据生成器（用于前端渲染压测，补充 BarentsWatch 的地域限制） +- 确认前端可渲染船只点，整条链路走通 + +### Phase 1 — 后端基础设施（3–4 天） + +#### 1.1 数据库 Schema + +```sql +-- 船只静态信息（每 6h 刷新一次） +CREATE TABLE vessel_static ( + mmsi BIGINT PRIMARY KEY, + name VARCHAR(128), + callsign VARCHAR(16), + vessel_type SMALLINT, + vessel_type_name VARCHAR(64), + flag VARCHAR(4), -- ISO 国家码 + length FLOAT, + width FLOAT, + draught FLOAT, + imo BIGINT, + updated_at TIMESTAMPTZ +); + +-- 船只实时位置（高频写入，保留 24h 轨迹） +CREATE TABLE vessel_position ( + id BIGSERIAL PRIMARY KEY, + mmsi BIGINT NOT NULL, + lat FLOAT NOT NULL, + lon FLOAT NOT NULL, + sog FLOAT, -- Speed over ground（节） + cog FLOAT, -- Course over ground（度） + heading SMALLINT, -- 真北航向 + nav_status SMALLINT, -- 0=航行 1=锚泊 5=停靠 ... + received_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_vessel_pos_mmsi_time ON vessel_position(mmsi, received_at DESC); +CREATE INDEX idx_vessel_pos_time ON vessel_position(received_at DESC); + +-- 最新位置物化视图（地图渲染主数据源，避免全表扫描） +CREATE MATERIALIZED VIEW vessel_latest AS +SELECT DISTINCT ON (mmsi) + vp.*, vs.name, vs.vessel_type_name, vs.flag, vs.length +FROM vessel_position vp +LEFT JOIN vessel_static vs USING (mmsi) +ORDER BY mmsi, received_at DESC; + +CREATE UNIQUE INDEX ON vessel_latest(mmsi); +``` + +> 后期如需完整历史轨迹查询，迁移 `vessel_position` 到 TimescaleDB 或按天分区。 + +#### 1.2 Collector：VesselAISCollector + +文件：`backend/app/services/collectors/vessel_ais.py` + +- 继承 `BaseCollector`，注册到 `collector_registry` +- 轮询间隔：30–60s（由数据源限速决定） +- 支持多数据源切换，通过 `datasource_config` 配置 URL + API Key +- 写入逻辑：upsert `vessel_latest`，append `vessel_position` +- 接入现有调度系统（`scheduler.py`） + +#### 1.3 API 端点 + +``` +GET /api/v1/visualization/geo/vessels + ?bbox=lon_min,lat_min,lon_max,lat_max # 视口裁剪 + ?type=cargo,tanker,passenger # 船型过滤 + ?limit=5000 +→ GeoJSON FeatureCollection（Point） + +GET /api/v1/visualization/vessels/{mmsi} # 单船详情 +GET /api/v1/visualization/vessels/{mmsi}/track # 历史轨迹（默认 6h） + ?hours=6 +→ GeoJSON LineString +``` + +GeoJSON Feature 格式： + +```json +{ + "type": "Feature", + "geometry": { "type": "Point", "coordinates": [lon, lat] }, + "properties": { + "mmsi": 123456789, + "name": "EVER GIVEN", + "vessel_type": 70, + "vessel_type_name": "Cargo", + "flag": "PA", + "sog": 12.4, + "cog": 247.0, + "heading": 245, + "nav_status": 0, + "length": 400, + "received_at": "2026-04-27T10:00:00Z" + } +} +``` + +#### 1.4 更新机制 + +**HTTP 轮询**（不使用 WebSocket）： + +- 前端 `setInterval(fetchVessels, 5 * 60 * 1000)` 定期拉取最新快照 +- 后端 Collector 每 60s 从 BarentsWatch 拉取并写库，`vessel_latest` 物化视图随时可查 +- WebSocket 留给告警/事件驱动场景（BGP、系统通知），不混入周期性位置刷新 +- 换用 AISHub / MarineTraffic 实时流后，届时再评估是否升级为 WebSocket delta push + +--- + +### Phase 2 — 前端渲染（3–4 天） + +文件：`frontend/public/earth/js/vessels.js` + +#### 2.1 渲染方案 + +参考现有卫星系统（`satellites.js`）的 InstancedMesh 模式： + +- `THREE.InstancedMesh`：每个实例 = 一艘船，矩阵包含位置 + 旋转（朝向 COG） +- 行进船：三角箭头图标，朝向 COG 方向 +- 静止/锚泊船：圆点图标 +- SVG 图标输出到 `frontend/public/earth/assets/icons/vessel-arrow.svg` 和 `vessel-dot.svg` + +#### 2.2 船型颜色规范 + +| 船型 | 颜色 | +|-----|------| +| 货轮 Cargo | `#4A90D9` 蓝 | +| 油轮 Tanker | `#E85D04` 橙红 | +| 客船 Passenger | `#06D6A0` 绿 | +| 渔船 Fishing | `#FFD166` 黄 | +| 军舰 Military | `#73797E` 灰 | +| 其他 | `#9B9B9B` 浅灰 | +| 锚泊/停靠 | 降低饱和度 0.4x | + +#### 2.3 LOD（相机距离细节层次） + +| 相机距离 | 渲染策略 | +|---------|---------| +| > 400 | 仅渲染 top 1000 艘（按数据新鲜度 + 船型优先级） | +| 200–400 | 渲染 top 5000 艘 | +| < 200 | 渲染当前视口 bbox 内全部船只 | + +前端根据相机位置动态计算 bbox，附加到 API 请求中。 + +#### 2.4 图层集成 + +接入现有图层系统，新增"船只"图层项，支持： +- 图层开/关，状态持久化 +- 子过滤（按船型选择显示哪类，可在图例或设置面板中配置） +- 与海缆、BGP、卫星层级共存（renderOrder 待定，参考现有层级文档） + +#### 2.5 Info Card + +复用 `showInfoCard` 机制，点击船只弹出： + +``` +EVER GIVEN 🚢 +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +MMSI 123456789 +IMO 9811000 +旗帜 巴拿马 🇵🇦 +船型 散货轮 +当前航速 12.4 kn +航向 247° +状态 航行中 +目的地 ROTTERDAM +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ +[ 查看轨迹 ] [ MarineTraffic ↗ ] +``` + +#### 2.6 轨迹可视化 + +点击"查看轨迹" → 请求 `/vessels/{mmsi}/track` → 用 `THREE.CatmullRomCurve3` 渲染插值轨迹线，风格与海缆一致。 + +--- + +### Phase 3 — 功能完善（2–3 天） + +| 功能 | 说明 | +|-----|------| +| **船只搜索** | 接入现有搜索面板，按名称 / MMSI 搜索 | +| **统计 HUD** | 显示当前在线船只数、各类型分布 | +| **密度热图** | 超低 zoom 时切换为 hex-bin 热力图（避免点云爆炸） | +| **港口标注** | 加载 WorldPorts 数据集，显示主要港口标记 | +| **关键水道监控** | 马六甲、霍尔木兹、苏伊士等高亮 + 流量统计 | + +--- + +### Phase 4 — 性能与生产化（2–3 天） + +- `vessel_position` 按天分区，7 天自动清理 +- TODO：真实数据量达到百万级/日后，将 `vessel_position` 升级为 TimescaleDB hypertable，配置 retention policy 与压缩策略 +- GeoJSON endpoint 用 Redis 缓存 15s +- 若需 bbox 精确查询，引入 PostGIS `geography` + `ST_DWithin` +- InstancedMesh + frustum culling，目标 5 万船只 60fps + +--- + +## 三、工作量估算 + +| Phase | 内容 | 估计时间 | +|-------|-----|---------| +| Phase 0 | 数据源验证、mock | 1–2 天 | +| Phase 1 | 后端 Schema + Collector + API | 3–4 天 | +| Phase 2 | 前端渲染（InstancedMesh + 图层 + Info Card） | 3–4 天 | +| Phase 3 | 搜索 + 统计 + 轨迹 | 2–3 天 | +| Phase 4 | 性能优化 + 生产数据源接入 | 2–3 天 | +| **合计** | | **约 2–3 周** | + +--- + +## 四、参考资料 + +- BarentsWatch AIS API 文档：https://www.barentswatch.no/en/developer/ais-api/ +- MarineTraffic API：https://www.marinetraffic.com/en/ais-api-services +- AISHub：https://www.aishub.net/api +- AIS 导航状态码：ITU-R M.1371-5 +- 船型编码（vessel_type）：ITU/IMO AIS Message 5 Type and Cargo +- WorldPorts 数据集：https://msi.nga.mil/Publications/WPI diff --git a/docs/plans/frontend-markdown-renderer-plan.md b/docs/plans/frontend-markdown-renderer-plan.md new file mode 100644 index 00000000..2cea1a70 --- /dev/null +++ b/docs/plans/frontend-markdown-renderer-plan.md @@ -0,0 +1,97 @@ +# Markdown 渲染器完善计划 + +## 背景 + +Planet 控制台当前有三类主要 Markdown 使用场景： + +- 文档中心：技术文档、计划文档、运行手册。 +- AI Playground：模型回复、分析结果、代码片段。 +- BGP 简报：由系统生成并保存的态势报告。 + +这些场景都复用 `frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx`。因此 Markdown 能力应该集中在共享渲染器内完成，页面只负责传入内容、链接转换和布局约束，不能让每篇文档或每个页面手写复制按钮、表格样式、列表样式等交互细节。 + +## 目标 + +建设一个稳定、可复用、适合技术文档和 AI 输出的 Markdown 渲染器，优先覆盖常用语法、代码块操作和清晰的阅读样式，并为后续语法高亮、锚点导航、内容安全策略留出接口。 + +## 成功标准 + +- 代码块支持 fenced language、语言标签、复制按钮、复制成功状态和横向滚动。 +- 常用块语法稳定渲染：标题 1-6、段落、引用、分割线、表格、无序列表、有序列表、任务列表。 +- 常用行内语法稳定渲染：链接、自动链接、图片、行内代码、粗体、斜体、删除线。 +- 文档中心、AI Playground、BGP 简报继续复用同一个组件，不出现页面级重复实现。 +- 样式在普通业务面板和文档中心都有合理表现，文档中心可以通过 `.docs-markdown` 覆盖主题变量。 +- 前端 TypeScript build 通过，`git diff --check` 无空白错误。 + +## 当前实施范围 + +### 第一阶段：共享渲染器补齐 + +- 在 `MarkdownRenderer` 内解析 fenced code block 的语言信息。 +- 引入 `MarkdownCodeBlock` 子组件，负责语言标签、复制按钮和复制状态。 +- 保留现有 `Scrollbar` 横向滚动能力，避免长代码撑破页面。 +- 扩展标题渲染到 h1-h6，并保留 `getHeadingId` 对文档目录的支持。 +- 扩展列表解析，支持 `-`、`*`、`+`、`1.`、`1)` 和 GitHub 风格任务列表。 +- 扩展行内解析，支持图片、自动链接、删除线。 + +### 第二阶段：样式统一 + +- 全局 Markdown 样式覆盖业务场景，保持紧凑、清晰、可扫描。 +- 文档中心用 `.docs-markdown` 适配主题变量，避免硬编码颜色破坏明暗主题。 +- 代码块 toolbar 和 copy button 不依赖具体页面。 +- 图片默认响应式展示，避免超出内容区域。 + +### 第三阶段：验证 + +- 使用前端 build 验证 TypeScript 和 Vite 构建。 +- 使用 `git diff --check` 验证补丁格式。 +- 手动检查至少一个文档页中代码块复制按钮、语言标签和表格滚动是否出现。 + +## 后续增强项 + +### 语法高亮 + +当前不新增高亮依赖，避免一次性引入过重运行时代码。后续可以在以下方案中二选一： + +- `shiki`：适合文档中心，视觉质量高，但包体和初始化成本更高。 +- `highlight.js`：接入简单，覆盖语言广，但样式控制需要额外约束。 + +建议当文档代码块数量稳定增加后再引入，并做按需加载或懒加载。 + +### 更完整 CommonMark 支持 + +当前渲染器覆盖 Planet 常见内容，不追求完整 CommonMark 兼容。后续如果需要完整规范，建议切换到成熟生态： + +- `react-markdown` +- `remark-gfm` +- `rehype-sanitize` +- `rehype-slug` + +切换前需要评估：链接转换、目录 ID、现有样式、AI 输出安全策略和包体影响。 + +### 安全策略 + +目前渲染器不解析原始 HTML，这是正确默认值。后续如需支持 HTML，必须先明确： + +- 是否允许用户输入 Markdown。 +- 是否需要 HTML 白名单。 +- 是否需要 `rehype-sanitize`。 +- 图片和链接是否需要域名策略。 + +### 文档页能力 + +可继续补齐： + +- 标题锚点悬浮复制。 +- Mermaid 图表。 +- 代码块折叠。 +- 文档内搜索结果定位到代码块。 +- 复制按钮埋点，用于判断文档片段是否真正被使用。 + +## 维护约束 + +- Markdown 语法能力优先放在共享渲染器，不在具体文档页面散落实现。 +- 文档内容只表达内容，不承载 UI 行为。 +- 新增 Markdown 能力必须同时考虑文档中心、AI Playground、BGP 简报三个调用方。 +- 不解析原始 HTML，除非同步引入明确的 sanitize 策略。 +- 与主题相关的样式优先走页面容器变量覆盖，不在组件内写死文档中心颜色。 diff --git a/docs/technical/zh/README.md b/docs/technical/zh/README.md index 74ed215a..d943faca 100644 --- a/docs/technical/zh/README.md +++ b/docs/technical/zh/README.md @@ -1,4 +1,4 @@ -# Technical Docs +# 技术文档 这里放“当前实现和当前结构”的文档，重点回答： @@ -9,24 +9,24 @@ 适合放入这里的内容： -- Quickstart 和使用手册 +- 快速开始和使用手册 - 前端上下文 - Earth 前端结构 -- Earth 卫星 footprint 策略 +- Earth 卫星覆盖策略 - Earth 渲染图层顺序 - Earth 图层样式属性索引 - 后端运行控制 -- collector 现状 +- 采集器现状 - 采集格式约定 ## 使用入口 -- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md)：从零启动 Planet 的最短路径 -- [manual.md](/home/ray/dev/linkong/planet/docs/technical/manual.md)：控制台、`planet.sh`、Earth 和 Docs 的完整使用手册 +- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)：从零启动 Planet 的最短路径 +- [manual.md](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md)：控制台、`planet.sh`、Earth 和 Docs 的完整使用手册 不适合放入这里的内容： -- 尚未完成的 roadmap +- 尚未完成的路线图 - 未来迭代方案 - 大范围重构计划 diff --git a/docs/technical/zh/agents-aiprovider.md b/docs/technical/zh/agents-aiprovider.md index d82d75e7..a85b8aea 100644 --- a/docs/technical/zh/agents-aiprovider.md +++ b/docs/technical/zh/agents-aiprovider.md @@ -1,108 +1,108 @@ -# AI Provider Guide +# AI Provider 指南 -## Overview +## 概览 -`aiprovider` is the model-adapter service for Planet. +`aiprovider` 是 Planet 的模型适配服务。 -It isolates model-vendor details from the main backend so the rest of the system can call a stable business API: +它把模型厂商差异隔离在主后端之外，让系统其它部分可以调用稳定的业务 API： -- Caller service -> `planet backend` +- 调用方服务 -> `planet backend` - `planet backend` -> `aiprovider` -- `aiprovider` -> concrete model provider +- `aiprovider` -> 具体模型提供方 -The recommended default is: +推荐默认方式： -- External and cross-service callers use `planet backend` -- Only infrastructure-grade internal jobs call `aiprovider` directly +- 外部调用方和跨服务调用方统一调用 `planet backend` +- 只有基础设施级内部任务才直接调用 `aiprovider` -## Responsibilities +## 职责边界 -`backend` is responsible for: +`backend` 负责： -- authentication and authorization -- business-level request shaping -- stable `/api/v1/ai/...` endpoints -- internal service-to-service authentication toward `aiprovider` +- 身份认证和权限控制 +- 业务层请求整理 +- 稳定的 `/api/v1/ai/...` 接口 +- 面向 `aiprovider` 的内部服务认证 -`aiprovider` is responsible for: +`aiprovider` 负责： -- model protocol adaptation -- provider selection by `.env` -- timeout and lightweight retry -- request tracing via `X-Request-ID` +- 模型协议适配 +- 基于 `.env` 选择 provider +- 超时和轻量重试 +- 通过 `X-Request-ID` 串联请求追踪 -This now follows an OpenClaw-like seam: +当前配置采用类似 OpenClaw 的拆分方式： -- `AI_PROVIDER` identifies the vendor or logical provider -- `AI_PROVIDER_API` identifies the wire adapter +- `AI_PROVIDER` 标识厂商或逻辑 provider +- `AI_PROVIDER_API` 标识实际请求协议适配器 -That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field. +这个拆分能更清楚地表达 MiniMax、Claude 兼容网关、自托管 OpenAI 兼容服务等情况，避免把所有含义塞进一个配置项。 -## Supported Providers +## 支持的 Provider -`aiprovider` currently supports these provider identities: +`aiprovider` 当前支持以下 provider 标识： - `openai` - `anthropic` - `minimax` - `ollama` -Supported request adapters: +支持的请求适配器： - `openai-completions` - `anthropic-messages` - `ollama-generate` -Backward-compatible aliases still accepted: +仍然兼容的历史别名： - `openai_compatible` - `anthropic_compatible` - `claude_compatible` -Provider mapping: +推荐映射关系： -- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions` -- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages` -- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages` -- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate` +- `vLLM`、`LM Studio`、`One API`：`AI_PROVIDER=openai`，`AI_PROVIDER_API=openai-completions` +- `MiniMax`：`AI_PROVIDER=minimax`，`AI_PROVIDER_API=anthropic-messages` +- Claude 兼容网关：`AI_PROVIDER=anthropic`，`AI_PROVIDER_API=anthropic-messages` +- `Ollama`：`AI_PROVIDER=ollama`，`AI_PROVIDER_API=ollama-generate` -## API Surfaces +## API 面 -### Main backend API +### 主后端 API -Preferred stable entrypoints: +推荐使用的稳定入口： - `GET /api/v1/ai/provider/status` - `POST /api/v1/ai/situational-awareness/analyze` -Authentication: +认证方式： - `Authorization: Bearer ` -Optional tracing header: +可选追踪头： - `X-Request-ID: ` -The backend will propagate `X-Request-ID` to `aiprovider` and return the same header in the response. +后端会把 `X-Request-ID` 透传给 `aiprovider`，并在响应中返回同一个 header。 -### AI provider internal API +### AI Provider 内部 API -Internal-only endpoints: +仅供内部调用的接口： - `GET /v1/provider/status` - `POST /v1/analyze` -Authentication: +认证方式： - `X-Provider-Token: ` -Optional tracing header: +可选追踪头： - `X-Request-ID: ` -## Request Example +## 请求示例 -### Call through backend +### 通过后端调用 ```bash curl -X POST http://localhost:8000/api/v1/ai/situational-awareness/analyze \ @@ -127,7 +127,7 @@ curl -X POST http://localhost:8000/api/v1/ai/situational-awareness/analyze \ }' ``` -### Call `aiprovider` directly +### 直接调用 `aiprovider` ```bash curl -X POST http://localhost:8010/v1/analyze \ @@ -149,9 +149,9 @@ curl -X POST http://localhost:8010/v1/analyze \ }' ``` -## Response Shape +## 响应结构 -Both backend and `aiprovider` return the same payload shape: +后端和 `aiprovider` 返回相同的 payload 结构： ```json { @@ -166,15 +166,15 @@ Both backend and `aiprovider` return the same payload shape: } ``` -Both services also return: +两个服务都会返回： - `X-Request-ID: ` -## Configuration +## 配置 -### Backend +### 后端 -Recommended backend `.env`: +推荐的后端 `.env`： ```env AI_PROVIDER_SERVICE_URL=http://localhost:8010 @@ -183,21 +183,21 @@ AI_PROVIDER_TIMEOUT_SECONDS=60 AI_PROVIDER_RETRY_ATTEMPTS=2 ``` -Reference file: +参考文件： - [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example) ### AI Provider -Reference file: +参考文件： - [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) -Frontend local reference: +前端本地参考： - [frontend/.env.example](/home/ray/dev/linkong/planet/frontend/.env.example) -Common settings: +通用配置： ```env SERVICE_NAME=planet-ai-provider @@ -208,7 +208,7 @@ AI_HTTP_RETRY_ATTEMPTS=2 AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束，输出结构化、克制、可执行的分析。 ``` -### OpenAI-compatible example +### OpenAI 兼容示例 ```env AI_PROVIDER=openai @@ -218,7 +218,7 @@ AI_API_KEY=local-key AI_MODEL=your-local-model ``` -### MiniMax CN example +### MiniMax 中国区示例 ```env AI_PROVIDER=minimax @@ -230,13 +230,13 @@ AI_MAX_TOKENS=1200 AI_ANTHROPIC_VERSION=2023-06-01 ``` -MiniMax note: +MiniMax 说明： -- This follows the same Anthropic Messages request shape as the official MiniMax examples. -- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object. -- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior. +- 这里使用官方 MiniMax 示例中的 Anthropic Messages 请求结构。 +- 对 MiniMax，`aiprovider` 默认不会开启 `thinking`，除非调用方显式传入 `thinking` 对象。 +- 这个行为和 OpenClaw 对 MiniMax Anthropic 兼容接口的谨慎处理保持一致。 -### Anthropic-compatible example +### Anthropic 兼容示例 ```env AI_PROVIDER=anthropic @@ -248,7 +248,7 @@ AI_MAX_TOKENS=1200 AI_ANTHROPIC_VERSION=2023-06-01 ``` -### Ollama example +### Ollama 示例 ```env AI_PROVIDER=ollama @@ -258,36 +258,36 @@ AI_API_KEY= AI_MODEL=qwen2.5:7b ``` -## Deployment Modes +## 部署模式 -### Single machine +### 单机部署 -Recommended local flow: +推荐的本地流程： -- `backend` on `localhost:8000` -- `aiprovider` on `localhost:8010` -- local model gateway on `localhost:11434` or another local port +- `backend` 运行在 `localhost:8000` +- `aiprovider` 运行在 `localhost:8010` +- 本地模型网关运行在 `localhost:11434` 或其它本地端口 -Helpers already included: +仓库内已包含辅助入口： - [planet.sh](/home/ray/dev/linkong/planet/planet.sh) - [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) -### Multi-machine +### 多机部署 -Example topology: +示例拓扑： -- app machine: `backend` -- AI gateway machine: `aiprovider` -- model machine: local model service or cloud proxy +- 应用机器：`backend` +- AI 网关机器：`aiprovider` +- 模型机器：本地模型服务或云代理 -In that case, this becomes service-to-service HTTP RPC: +此时链路变成服务间 HTTP RPC： - caller -> backend - backend -> `http://10.0.0.12:8010` -- `aiprovider` -> model endpoint +- `aiprovider` -> 模型端点 -Recommended cross-machine backend config: +推荐的跨机器后端配置： ```env AI_PROVIDER_SERVICE_URL=http://10.0.0.12:8010 @@ -296,38 +296,38 @@ AI_PROVIDER_TIMEOUT_SECONDS=60 AI_PROVIDER_RETRY_ATTEMPTS=2 ``` -Recommended operating rules: +推荐运行规则： -- keep `aiprovider` on a private network -- protect it with `X-Provider-Token` at minimum -- always send `X-Request-ID` -- keep callers on the backend API unless they are infrastructure jobs +- 将 `aiprovider` 放在私有网络内 +- 至少用 `X-Provider-Token` 保护它 +- 始终发送 `X-Request-ID` +- 除基础设施任务外，调用方优先走后端 API -## Retry And Failure Behavior +## 重试和失败行为 -`backend -> aiprovider`: +`backend -> aiprovider`： -- retries lightweight network / 5xx failures -- returns `502` when the provider service is unavailable +- 对轻量网络错误和 5xx 失败进行重试 +- provider 服务不可用时返回 `502` -`aiprovider -> model provider`: +`aiprovider -> model provider`： -- retries lightweight network / 5xx failures -- returns `502` when the model provider is unavailable +- 对轻量网络错误和 5xx 失败进行重试 +- 模型提供方不可用时返回 `502` -This is intentionally conservative. It avoids masking persistent errors while still absorbing short hiccups. +这个策略故意保持保守：它能吸收短暂抖动，但不会掩盖持续性错误。 -## Operational Notes +## 运维说明 -- `./planet.sh start` now starts `aiprovider` automatically -- `./planet.sh restart -a` restarts only `aiprovider` -- `./planet.sh log -a` tails `aiprovider` logs -- `./planet.sh health` reports `aiprovider` health +- `./planet.sh start` 会自动启动 `aiprovider` +- `./planet.sh restart -a` 只重启 `aiprovider` +- `./planet.sh log -a` 跟随查看 `aiprovider` 日志 +- `./planet.sh health` 会报告 `aiprovider` 健康状态 -## Recommended Calling Policy +## 推荐调用策略 -- Frontend and application services: call `backend` -- Scheduled infra jobs and diagnostics: optionally call `aiprovider` -- Do not let multiple business services integrate model vendors independently +- 前端和应用服务：调用 `backend` +- 定时基础设施任务和诊断任务：可选直接调用 `aiprovider` +- 不要让多个业务服务分别接入模型厂商 -That keeps provider switching centralized and avoids model-specific drift across the system. +这样可以集中管理 provider 切换，避免模型相关差异在系统里四处扩散。 diff --git a/docs/technical/zh/backend-datasources-api-performance.md b/docs/technical/zh/backend-datasources-api-performance.md new file mode 100644 index 00000000..30ed2087 --- /dev/null +++ b/docs/technical/zh/backend-datasources-api-performance.md @@ -0,0 +1,100 @@ +# DataSources 列表接口性能优化 + +## 背景 + +`GET /api/v1/datasources` 是数据源管理页面的核心接口，响应慢会直接阻塞页面渲染。 + +## 优化前的查询链路 + +`_load_datasource_list_context` 按顺序执行以下查询： + +| 序号 | 函数 | 查询内容 | 瓶颈 | +|------|------|---------|------| +| 1 | `_load_latest_running_tasks` | collection_tasks 窗口函数，stale check 依赖此结果 | 必须串行 | +| 2 | `_load_latest_completed_tasks` | collection_tasks 窗口函数（最近完成任务） | 串行等待 | +| 3 | `_load_datasource_data_counts` | `COUNT(*) GROUP BY source` on collected_data | **最慢，全表扫描** | +| 4 | `_load_datasource_endpoint_overrides` | datasource_configs 简单 SELECT | 串行等待 | + +## 第一阶段：并行化 + +将 2/3/4 三个互不依赖的查询改为 `asyncio.gather` + 独立 session 并行执行： + +```python +async def _fetch_completed(): + async with async_session_factory() as s: + return await _load_latest_completed_tasks(s, datasource_ids) + +async def _fetch_counts(): + async with async_session_factory() as s: + return await _load_datasource_data_counts(s, sources) + +async def _fetch_overrides(): + async with async_session_factory() as s: + return await _load_datasource_endpoint_overrides(s, sources) + +completed_tasks, data_counts, endpoint_overrides = await asyncio.gather( + _fetch_completed(), _fetch_counts(), _fetch_overrides(), +) +``` + +> **注意**：SQLAlchemy `AsyncSession` 不支持在同一 session 上并发，每个协程必须独立开 session。 + +## 第二阶段：删除重量级查询 + +### 删除 `_load_datasource_data_counts` + +`data_count` 字段仅用于前端在"最近采集"列显示 `(0条)` 的边缘提示，不值得为此维持一次 `COUNT(*) GROUP BY` 全表扫描。 + +- 前端同步移除 `(0条)` 显示逻辑 +- 移除 `BuiltInDataSource` 接口中的 `data_count` 字段 + +### 删除 `_load_latest_completed_tasks` + +`last_status` 和 `last_run_at` 已由 collector 在任务完成时直接更新到 `DataSource` 模型字段，不需要再 JOIN collection_tasks 获取： + +```python +# 优化前：需要查 completed_tasks +last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None) +last_status = datasource.last_status or (last_task.status if last_task else None) + +# 优化后：直接读模型字段 +last_run_at = datasource.last_run_at +last_status = datasource.last_status +``` + +同步移除 `last_records_processed` 字段（来源是 completed_tasks，列表不显示此字段）。 + +## 优化后的查询链路 + +``` +datasources SELECT → 主数据，必须 +_load_latest_running_tasks → 必须（进行中状态 + stale check） +_load_datasource_endpoint_overrides → 必须（endpoint 覆盖，编辑内置 collector 时需要默认值） +``` + +3 个查询（原来 5 个），后两个顺序执行（running tasks 先完成用于 stale check，endpoint overrides 轻量）。 + +## 前端 triggerDatasource 双调修复 + +`triggerDatasource` 中存在双重 `fetchData()` 调用： + +```typescript +// 修复前 +} else { + window.setTimeout(() => { fetchData() }, 800) // 无 task_id 时延迟刷 +} +fetchData() // 总是立即刷 → 与上面的延迟刷重叠 + +// 修复后（二者互斥） +if (res.data.task_id) { + setTaskProgress(...) + fetchData() // 有 task_id：立即刷一次 +} else { + window.setTimeout(fetchData, 800) // 无 task_id：等 800ms 再刷一次 +} +``` + +## 相关文件 + +- `backend/app/api/v1/datasources.py` — `_load_datasource_list_context`、`list_datasources` +- `frontend/src/pages/DataSources/DataSources.tsx` — `BuiltInDataSource` interface、`triggerDatasource` diff --git a/docs/technical/zh/backend-system-service-control.md b/docs/technical/zh/backend-system-service-control.md index 12bf6e56..866c077d 100644 --- a/docs/technical/zh/backend-system-service-control.md +++ b/docs/technical/zh/backend-system-service-control.md @@ -1,61 +1,57 @@ -# System Service Control +# 系统服务控制 -This document defines the fixed mapping between admin control-plane actions and -the existing `planet.sh` service-management commands. +本文定义后台控制面动作与现有 `planet.sh` 服务管理命令之间的固定映射。 -The goal is to reuse the current operational script semantics without exposing -arbitrary shell execution to the frontend or API callers. +目标是在复用当前运维脚本语义的同时，不向前端或 API 调用方暴露任意 shell 执行能力。 -## Scope +## 范围 -- This mapping is for admin-side operational controls only. -- The control plane must submit a fixed action name, not a raw shell command. -- The backend is responsible for translating an allowed action into a fixed - `planet.sh` invocation. +- 这套映射只用于管理端运维控制。 +- 控制面必须提交固定 action 名称，而不是原始 shell 命令。 +- 后端负责把允许的 action 翻译成固定的 `planet.sh` 调用。 -## Design Rules +## 设计规则 -- Only whitelist actions may be executed. -- The frontend must never send arbitrary shell strings. -- The backend must build command arguments from a fixed mapping table. -- High-risk actions should be restricted to `super_admin`. -- Prefer partial restarts over full-stack restarts when UI continuity matters. +- 只允许执行白名单 action。 +- 前端绝不能发送任意 shell 字符串。 +- 后端必须从固定映射表构造命令参数。 +- 高风险 action 应限制为 `super_admin`。 +- 在 UI 连续性重要时，优先局部重启，而不是全栈重启。 -## Action Mapping +## Action 映射 -| Action name | Intended use | `planet.sh` command | Notes | +| Action 名称 | 用途 | `planet.sh` 命令 | 备注 | | --- | --- | --- | --- | -| `restart-backend` | Restart backend API only | `./planet.sh restart -b` | Recommended first implementation for UI-triggered restart flows. | -| `restart-database` | Restart PostgreSQL and Redis containers | `./planet.sh restart -d` | Useful when database/cache services need a controlled bounce without restarting the UI. | -| `restart-system` | Restart the whole application stack | `./planet.sh restart` | Frontend continuity breaks briefly; UI should switch to guided recovery mode. | -| `restart-frontend` | Restart frontend dev server only | `./planet.sh restart -f` | Use with caution; UI continuity is weaker than backend-only restart. | -| `restart-backend-port` | Restart backend on a specific port | `./planet.sh restart -b ` | Port must be backend-validated before execution. | -| `restart-frontend-port` | Restart frontend on a specific port | `./planet.sh restart -f ` | Port must be backend-validated before execution. | -| `health-check` | Read current service health | `./planet.sh health` | Safe read-only operational action. | -| `show-logs-backend` | Inspect backend logs | `./planet.sh log -b` | Best used for CLI/operator tooling, not normal Web UI streaming. | -| `show-logs-frontend` | Inspect frontend logs | `./planet.sh log -f` | Best used for CLI/operator tooling, not normal Web UI streaming. | +| `restart-backend` | 只重启后端 API | `./planet.sh restart -b` | 推荐作为 UI 触发重启流程的第一阶段实现。 | +| `restart-database` | 重启 PostgreSQL 和 Redis 容器 | `./planet.sh restart -d` | 适合数据库/缓存需要受控重启但不希望重启 UI 的场景。 | +| `restart-system` | 重启整个应用栈 | `./planet.sh restart` | 前端会短暂中断；UI 应进入引导恢复模式。 | +| `restart-frontend` | 只重启前端开发服务器 | `./planet.sh restart -f` | 谨慎使用；UI 连续性弱于只重启后端。 | +| `restart-backend-port` | 在指定端口重启后端 | `./planet.sh restart -b ` | 执行前必须由后端校验端口。 | +| `restart-frontend-port` | 在指定端口重启前端 | `./planet.sh restart -f ` | 执行前必须由后端校验端口。 | +| `health-check` | 读取当前服务健康状态 | `./planet.sh health` | 安全的只读运维动作。 | +| `show-logs-backend` | 查看后端日志 | `./planet.sh log -b` | 更适合 CLI/运维工具，不建议作为普通 Web UI 日志流。 | +| `show-logs-frontend` | 查看前端日志 | `./planet.sh log -f` | 更适合 CLI/运维工具，不建议作为普通 Web UI 日志流。 | -## Not Exposed In UI By Default +## 默认不暴露到 UI 的能力 -The following existing script capabilities should not be exposed directly in the -Web UI unless there is an explicit product need and an additional safety review: +除非有明确产品需求并经过额外安全评审，否则以下脚本能力不应直接暴露到 Web UI： - `./planet.sh restart` - `./planet.sh start` - `./planet.sh stop` - `./planet.sh createuser` -- any future raw shell passthrough +- 任何未来的原始 shell 透传能力 -Reason: +原因： -- full restart can break the current control session; -- stop/start have larger blast radius; -- user creation is not a service-control operation; -- raw shell passthrough creates unnecessary privilege risk. +- 全量重启可能打断当前控制会话； +- stop/start 影响面更大； +- 用户创建不是服务控制操作； +- 原始 shell 透传会引入不必要的权限风险。 -## Recommended First-Phase UI Contract +## 第一阶段推荐 UI 契约 -### Frontend action payload +### 前端 action payload ```json { @@ -63,7 +59,7 @@ Reason: } ``` -### Backend command resolution +### 后端命令解析 ```text restart-backend -> ["./planet.sh", "restart", "-b"] @@ -73,19 +69,19 @@ restart-frontend -> ["./planet.sh", "restart", "-f"] health-check -> ["./planet.sh", "health"] ``` -## API Draft +## API 草案 -### Primary Endpoint +### 主接口 - `POST /api/v1/system/restart-tasks` -Purpose: +用途： -- create a controlled restart task; -- resolve a whitelist action into a fixed `planet.sh` command; -- hand execution off to an external runner or detached subprocess. +- 创建受控重启任务； +- 将白名单 action 解析成固定 `planet.sh` 命令； +- 把执行交给外部 runner 或 detached subprocess。 -### Request Body +### 请求体 ```json { @@ -93,7 +89,7 @@ Purpose: } ``` -Optional future shape: +未来可选形态： ```json { @@ -102,7 +98,7 @@ Optional future shape: } ``` -### Response +### 响应 ```json { @@ -114,11 +110,11 @@ Optional future shape: } ``` -### Task Query Endpoint +### 任务查询接口 - `GET /api/v1/system/restart-tasks/{task_id}` -Response shape: +响应结构： ```json { @@ -136,11 +132,11 @@ Response shape: } ``` -### Optional Log Endpoint +### 可选日志接口 - `GET /api/v1/system/restart-tasks/{task_id}/logs` -Suggested response: +建议响应： ```json { @@ -154,10 +150,9 @@ Suggested response: } ``` -This log endpoint is optional for phase one. The first version can work with -task state plus `/health` polling alone. +日志接口在第一阶段不是必需项。首版可以只依赖任务状态加 `/health` 轮询。 -## Task State Model +## 任务状态模型 ### Status @@ -177,35 +172,32 @@ task state plus `/health` polling alone. - `healthy` - `failed` -### Interpretation +### 含义 -- `status` is the high-level terminal or non-terminal state. -- `stage` is the operator-facing execution phase for the UI. -- `message` is the short human-readable line shown in the modal or full-screen - overlay. +- `status` 是高层终态/非终态状态。 +- `stage` 是面向运维人员和 UI 的执行阶段。 +- `message` 是 modal 或全屏遮罩中展示的短文本。 -## Permission Model +## 权限模型 -- `restart-backend` should require `super_admin`. -- Permission checks should follow the same role pattern already used in - [users.py](/home/ray/dev/linkong/planet/backend/app/api/v1/users.py). -- Frontend visibility may hide controls for non-`super_admin`, but backend must - still enforce authorization. +- `restart-backend` 应要求 `super_admin`。 +- 权限检查应沿用 [users.py](/home/ray/dev/linkong/planet/backend/app/api/v1/users.py) 中已有的角色模式。 +- 前端可以对非 `super_admin` 隐藏控件，但后端必须继续强制鉴权。 -## Storage Model +## 存储模型 -Recommended first implementation: +推荐第一阶段实现： -- store restart task state in Redis; -- keep task lifetime short; -- keep recent logs as a bounded list. +- 将重启任务状态存入 Redis； +- 任务生命周期保持较短； +- 最近日志用有界列表保存。 -Suggested keys: +建议 key： - `system:restart_task:{task_id}` - `system:restart_task:{task_id}:logs` -Suggested stored fields: +建议字段： - `task_id` - `action` @@ -217,22 +209,21 @@ Suggested stored fields: - `created_at` - `updated_at` -## Execution Model +## 执行模型 -The request-handling API process should not depend on itself surviving long -enough to stream the whole restart output. +处理请求的 API 进程不应依赖自身持续存活来流式输出完整重启日志。 -Recommended execution flow: +推荐执行流程： -1. validate caller and action -2. create task state in Redis -3. resolve action to fixed `planet.sh` argv -4. spawn detached executor -5. return `task_id` -6. executor updates task state while restart is in progress -7. frontend polls health and/or task state until recovery +1. 校验调用方和 action +2. 在 Redis 中创建任务状态 +3. 将 action 解析为固定 `planet.sh` argv +4. 启动 detached executor +5. 返回 `task_id` +6. executor 在重启过程中更新任务状态 +7. 前端轮询健康状态和/或任务状态，直到服务恢复 -Recommended command resolution examples: +推荐命令解析示例： ```text restart-backend -> ["./planet.sh", "restart", "-b"] @@ -241,25 +232,25 @@ restart-backend-port -> ["./planet.sh", "restart", "-b", ""] health-check -> ["./planet.sh", "health"] ``` -## Frontend Polling Flow +## 前端轮询流程 -Recommended first-phase UX: +推荐第一阶段 UX： -1. user clicks `重启后端` -2. confirmation modal explains temporary unavailability -3. frontend calls `POST /api/v1/system/restart-tasks` -4. UI enters blocking restart state -5. frontend polls `/health` every `1-2s` -6. temporary request failures are treated as expected -7. after `2-3` consecutive successful health checks, frontend reloads page +1. 用户点击 `重启后端` +2. 确认 modal 说明服务会短暂不可用 +3. 前端调用 `POST /api/v1/system/restart-tasks` +4. UI 进入阻塞式重启状态 +5. 前端每 `1-2s` 轮询 `/health` +6. 临时请求失败视为预期现象 +7. 连续 `2-3` 次健康检查成功后，前端刷新页面 -Optional richer polling: +可选增强轮询： -1. poll task status endpoint while backend is still reachable -2. switch to `/health` recovery polling after disconnect begins -3. refresh page after health recovery +1. 后端仍可达时轮询任务状态接口 +2. 断连开始后切换为 `/health` 恢复轮询 +3. 健康恢复后刷新页面 -## Frontend State Machine +## 前端状态机 - `idle` - `confirming` @@ -270,7 +261,7 @@ Optional richer polling: - `failed` - `timeout` -Suggested UI messages: +建议 UI 文案： - `已发送重启指令` - `正在停止后端服务` @@ -278,70 +269,65 @@ Suggested UI messages: - `服务已恢复，正在刷新页面` - `恢复超时，请手动检查服务状态` -## Phase-One Recommendation +## 第一阶段建议 -Implement only the following in phase one: +第一阶段只实现： - `restart-backend` -- `super_admin` permission gate -- task creation endpoint -- Redis-backed task state -- frontend confirmation modal -- frontend `/health` polling -- automatic page reload after recovery +- `super_admin` 权限门禁 +- 任务创建接口 +- Redis 任务状态 +- 前端确认 modal +- 前端 `/health` 轮询 +- 恢复后自动刷新页面 -Do not implement in phase one: +第一阶段不要实现： -- full `./planet.sh restart` -- raw shell command passthrough -- arbitrary service control -- full terminal stdout streaming -- multi-action concurrent restart queueing +- 完整 `./planet.sh restart` +- 原始 shell 命令透传 +- 任意服务控制 +- 完整终端 stdout 流式输出 +- 多 action 并发重启队列 -## Implementation Checklist +## 实现清单 -### Backend +### 后端 -1. add a dedicated system-control API module under `backend/app/api/v1/` -2. add a whitelist-based action resolver for `planet.sh` -3. store restart task state in Redis -4. add detached restart-runner script execution -5. expose: +1. 在 `backend/app/api/v1/` 下新增专用系统控制 API 模块 +2. 增加基于白名单的 `planet.sh` action 解析器 +3. 将重启任务状态存入 Redis +4. 增加 detached restart-runner 脚本执行 +5. 暴露： - `POST /api/v1/system/restart-tasks` - `GET /api/v1/system/restart-tasks/{task_id}` - - optional task log endpoint -6. enforce `super_admin` permission on all restart-task endpoints + - 可选任务日志接口 +6. 对所有 restart-task 接口强制 `super_admin` 权限 -### Frontend +### 前端 -1. add a `重启后端` control on the dashboard for `super_admin` -2. show a confirmation modal before dispatch -3. after submission, switch modal into blocking restart state -4. poll `/health` until backend recovery is confirmed -5. auto-refresh page after consecutive successful health checks -6. show short stage-oriented logs instead of raw terminal streaming +1. 在 dashboard 为 `super_admin` 增加 `重启后端` 控件 +2. 发送前展示确认 modal +3. 提交后将 modal 切换为阻塞式重启状态 +4. 轮询 `/health` 直到确认后端恢复 +5. 连续健康检查成功后自动刷新页面 +6. 展示简短阶段日志，而不是原始终端流 -### Operational Notes +### 运维说明 -1. phase one should target backend-only restart -2. frontend restart should remain out of scope initially -3. command execution must always originate from repository root -4. only fixed action names may cross the API boundary +1. 第一阶段目标应限定为只重启后端 +2. 前端重启初期保持在范围外 +3. 命令执行必须始终从仓库根目录发起 +4. API 边界只能传递固定 action 名称 -## Validation Requirements +## 校验要求 -- Reject any action not present in the whitelist. -- If a port-bearing action is added, validate the port as an integer in - `1..65535`. -- Resolve commands from the repository root so `planet.sh` runs with a stable - working directory. -- Record the requested action, operator identity, execution start time, and - result. +- 拒绝任何不在白名单中的 action。 +- 如果增加带端口 action，端口必须校验为 `1..65535` 的整数。 +- 从仓库根目录解析命令，确保 `planet.sh` 的工作目录稳定。 +- 记录请求 action、操作者身份、执行开始时间和结果。 -## Implementation Guidance +## 实现建议 -- For UI-triggered restart flows, prefer `restart-backend` first. -- Do not rely on the current API request process to stream full restart output - after it triggers its own restart. -- Use a task record plus polling/health-check recovery flow instead of raw - terminal streaming as the primary UX. +- UI 触发重启流程时，优先实现 `restart-backend`。 +- 不要依赖当前 API 请求进程在触发自身重启后继续输出完整日志。 +- 主 UX 使用任务记录加轮询/健康检查恢复流程，而不是原始终端流。 diff --git a/docs/technical/zh/earth-bgp-context.md b/docs/technical/zh/earth-bgp-context.md index bd0eb8ef..d97c8d07 100644 --- a/docs/technical/zh/earth-bgp-context.md +++ b/docs/technical/zh/earth-bgp-context.md @@ -1,31 +1,31 @@ -# BGP Context +# BGP 态势上下文 -## Current Goal +## 当前目标 -The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline: +BGP 模块正在从一个只展示异常的演示功能，演进为分层观测管线： `raw observations -> enrichment -> detectors -> incidents -> console/Earth visualization` -The practical product goal is no longer just to "show incidents on the globe". The current product objective is: +实际产品目标已经不只是“在地球上显示事件”。当前目标是： -1. keep BGP visually present on Earth even when incident density is low -2. make incidents clearly feel like a higher-confidence layer than anomalies -3. show that the observation network is still active even when there are no active incidents +1. 即使 incident 密度很低，也让 BGP 在 Earth 上保持可见存在感 +2. 让 incident 明显比 anomaly 更像高置信度事件层 +3. 即使没有活跃 incident，也能表达观测网络仍在运行 -In practice, that means Earth should behave like an observability surface, not only an incident map: +换句话说，Earth 应该表现为观测面，而不只是事件地图： -- `collectors` show that observation is happening -- `activity` shows where routing state is currently active or noisy -- `incidents` become the highest-confidence focus layer +- `collectors` 表达观测正在发生 +- `activity` 表达哪里的路由状态近期活跃或噪声较高 +- `incidents` 成为最高置信度的聚焦层 -## Current Backend Architecture +## 当前后端架构 -### Data Layers +### 数据层 1. `BGPObservation` - - File: `backend/app/models/bgp_observation.py` - - Purpose: store normalized raw routing observations from live/history sources. - - Typical fields: + - 文件：`backend/app/models/bgp_observation.py` + - 用途：存储从实时/历史来源归一化后的原始路由观测。 + - 典型字段： - `source` - `collector` - `peer_asn` @@ -42,80 +42,80 @@ In practice, that means Earth should behave like an observability surface, not o - `ingest_batch_id` 2. `BGPAnomaly` - - File: `backend/app/models/bgp_anomaly.py` - - Purpose: hold atomic detector outputs. - - Current detector output types include: + - 文件：`backend/app/models/bgp_anomaly.py` + - 用途：保存原子级 detector 输出。 + - 当前 detector 输出类型包括： - `origin_change` - `more_specific_burst` - `mass_withdrawal` 3. `BGPIncident` - - File: `backend/app/models/bgp_incident.py` - - Purpose: aggregate atomic anomalies into incident-level objects for humans and the UI. + - 文件：`backend/app/models/bgp_incident.py` + - 用途：把原子 anomaly 聚合成人类和 UI 可消费的 incident 对象。 -### Pipeline +### 管线 -Main flow is currently anchored in: +主流程目前集中在： - `backend/app/services/collectors/bgp_common.py` - `backend/app/services/bgp_enrichment.py` - `backend/app/services/bgp_detectors.py` - `backend/app/services/bgp_incidents.py` -Operational flow: +运行流程： -1. collectors fetch raw BGP data -2. `normalize_bgp_event()` standardizes payloads -3. observations are persisted to `bgp_observations` -4. enrichment augments events with analysis context -5. detectors create `bgp_anomalies` -6. incident aggregation rolls anomalies up into `bgp_incidents` +1. 采集器抓取原始 BGP 数据 +2. `normalize_bgp_event()` 规范化 payload +3. observation 写入 `bgp_observations` +4. enrichment 为事件补充分析上下文 +5. detector 创建 `bgp_anomalies` +6. incident 聚合把 anomaly 汇总为 `bgp_incidents` -### Current Ingest Sources +### 当前接入来源 1. `RIPE RIS Live` - - Collector file: `backend/app/services/collectors/ris_live.py` - - Used for realtime observation flow. + - 采集器文件：`backend/app/services/collectors/ris_live.py` + - 用于实时观测流。 2. `CAIDA BGPStream Backfill` - - Collector file: `backend/app/services/collectors/bgpstream.py` - - Used as history/backfill entry point. + - 采集器文件：`backend/app/services/collectors/bgpstream.py` + - 用作历史/回填入口。 -## Current Enrichment Status +## 当前 enrichment 状态 -Implemented enrichment skeleton in: +已在以下文件实现 enrichment 骨架： - `backend/app/services/bgp_enrichment.py` -Current enrichments: +当前 enrichment 内容： - prefix family / prefix length -- supernet / more-specific derivation -- deduplicated AS path -- path prepending hints -- collector region info -- prefix baseline hints -- new-origin detection -- ASN organization profile from PeeringDB where available -- prefix scope / impacted region hints -- prefix geography source priority: - - `OpenGeoFeed` (override/high confidence) - - `IPtoASN` (country-range baseline) - - `NRO delegated stats` (registry-allocation fallback) +- supernet / more-specific 推导 +- 去重 AS path +- path prepending 提示 +- collector 区域信息 +- prefix baseline 提示 +- new-origin 检测 +- 可用时从 PeeringDB 获取 ASN 组织画像 +- prefix scope / 受影响区域提示 +- prefix 地理来源优先级： + - `OpenGeoFeed`（override，高置信） + - `IPtoASN`（国家范围 baseline） + - `NRO delegated stats`（registry allocation fallback） -Current limitation: +当前限制： -- `RPKI` is still placeholder-only and returns `unknown` -- no real ROA validation source is integrated yet -- `inetnum` / `inet6num` whois fallback is still pending +- `RPKI` 仍只是占位，返回 `unknown` +- 尚未集成真实 ROA 校验来源 +- `inetnum` / `inet6num` whois fallback 仍待实现 -## Current API Surface +## 当前 API 面 -Primary API file: +主 API 文件： - `backend/app/api/v1/bgp.py` -Available endpoints: +可用接口： - `/api/v1/bgp/events` - `/api/v1/bgp/events/summary` @@ -127,16 +127,16 @@ Available endpoints: - `/api/v1/bgp/incidents/summary` - `/api/v1/bgp/incidents/{id}` -Visualization GeoJSON endpoints: +可视化 GeoJSON 接口： - `backend/app/api/v1/visualization.py` - `/api/v1/visualization/geo/bgp-collectors` - `/api/v1/visualization/geo/bgp-anomalies` - `/api/v1/visualization/geo/bgp-incidents` -## Current Earth Behavior +## 当前 Earth 行为 -Relevant files: +相关文件： - `frontend/public/earth/js/bgp.js` - `frontend/public/earth/js/main.js` @@ -144,56 +144,56 @@ Relevant files: - `frontend/public/earth/js/constants.js` - `frontend/public/earth/index.html` -Current design: +当前设计： -1. Collectors are always shown when BGP is enabled. -2. Incident markers are now the primary Earth BGP markers. -3. If there are no incidents, Earth falls back to anomaly markers. -4. If there are no anomalies either, collectors still provide presence. -5. A dedicated `activity layer` now adds: - - per-collector recent 15-minute activity halos - - clustered regional activity hints derived from active collectors -6. Incident markers now use: - - symbol-driven event cores - - outward ring pulses - - reduced diffuse glow compared with older Earth builds -5. The right-side stats now show: +1. BGP 启用时始终显示 collectors。 +2. Incident marker 现在是 Earth BGP 的主 marker。 +3. 如果没有 incident，Earth 回退显示 anomaly marker。 +4. 如果也没有 anomaly，collector 仍然提供存在感。 +5. 专用 `activity layer` 现在增加： + - 每个 collector 最近 15 分钟活动 halo + - 基于活跃 collector 推导的区域聚合活动提示 +6. Incident marker 现在使用： + - 由符号驱动的事件核心 + - 向外扩散的环形脉冲 + - 相比旧版 Earth 更少的弥散 glow +7. 右侧统计现在显示： - BGP events - collector count - BGP status summary -This is directionally correct, but still incomplete for low-event-density periods. Right now Earth can still feel too quiet when incidents are sparse because the system lacks a dedicated `activity layer` between raw observation and incident focus. +这个方向是对的，但在低事件密度时期仍不完整。当前 Earth 在 incident 稀疏时仍可能显得过于安静，因为系统还缺少位于原始观测和 incident 聚焦之间的专用 `activity layer`。 -Current BGP status strategy: +当前 BGP 状态策略： -- incidents present: show active incident count -- no incidents but anomalies present: show active anomaly count, plus active observation regions when available -- no incidents/anomalies but activity present: show `观测网络运行中` -- no incidents/anomalies but collectors present: show `观测网络运行中 · 当前未发现聚合级事件` -- no BGP data at all: show `暂无观测数据` +- 有 incident：显示活跃 incident 数量 +- 无 incident 但有 anomaly：显示活跃 anomaly 数量，并在可用时显示活跃观测区域 +- 无 incident/anomaly 但有 activity：显示 `观测网络运行中` +- 无 incident/anomaly 但有 collectors：显示 `观测网络运行中 · 当前未发现聚合级事件` +- 完全无 BGP 数据：显示 `暂无观测数据` -Earth info-card strategy: +Earth info-card 策略： -- `bgp` card is now incident-centric in wording -- `bgp_collector` card shows collector location and current event count +- `bgp` 卡片文案以 incident 为中心 +- `bgp_collector` 卡片显示 collector 位置和当前事件数 -## Current Product Gap +## 当前产品缺口 -The main product gap is not architecture correctness. It is low-density visualization strategy. +主要缺口不是架构正确性，而是低密度可视化策略。 -Current reality: +当前事实： -- incident count is naturally much lower than anomaly count -- that is expected, because incidents are aggregated and de-noised -- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer +- incident 数量天然远低于 anomaly 数量 +- 这是预期行为，因为 incident 是聚合和去噪后的结果 +- 但 incident-first 渲染会让 Earth 显得过于安静，除非有另一层始终可用的 activity layer -Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md). +推荐 `activity layer` 的实现细节在 [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md) 中展开。 -So the immediate next milestone is: +因此最近的里程碑是： `event map -> observability map` -That means Earth needs three simultaneously readable layers: +这意味着 Earth 需要三层同时可读： 1. `observation layer` - collectors @@ -205,108 +205,108 @@ That means Earth needs three simultaneously readable layers: - regional activity scoring - incident presence bonus 3. `incident layer` - - sparse but highly legible, high-confidence event objects - - symbol-driven markers - - outward ring pulse instead of broad diffuse glow + - 稀疏但高度清晰的高置信事件对象 + - 符号化 marker + - 向外环形脉冲，而不是大面积弥散 glow -## Incident Visual Direction +## Incident 视觉方向 -The Earth `incident` layer should not read like a large glowing patch. It should read like a compact, high-confidence event focus. +Earth 的 `incident` 层不应该像一大片发光区域，而应该像紧凑、高置信度的事件焦点。 -Design principles: +设计原则： -1. `incident` markers should use a strong primary symbol - - the symbol shape should carry type meaning where possible - - examples: - - `origin_change`: triangle-like warning marker - - `mass_withdrawal`: alert/exclamation-style marker - - `more_specific_burst`: split/radiating marker +1. `incident` marker 应使用强主符号 + - 符号形状尽量承载类型含义 + - 示例： + - `origin_change`：类似三角警告 marker + - `mass_withdrawal`：告警/感叹号风格 marker + - `more_specific_burst`：分裂/放射 marker -2. emphasis should come from outward ring pulses, not area flooding - - use a compact hot core - - use one or more expanding ring pulses - - avoid broad luminous blobs that make the event center feel vague +2. 强调应来自向外扩散的环形脉冲，而不是区域泛光 + - 使用紧凑高亮核心 + - 使用一个或多个扩张环形脉冲 + - 避免让事件中心变得模糊的大面积亮斑 -3. `collector` and `incident` must stay visually distinct - - collectors are observation infrastructure - - incidents are extracted event focus - - collector activity should stay quieter than incident pulse language +3. `collector` 和 `incident` 必须保持视觉区别 + - collector 是观测基础设施 + - incident 是抽取后的事件焦点 + - collector activity 应比 incident pulse 更安静 -4. calm periods still need observability presence - - collectors and activity layers should keep the map alive - - once incidents appear, they should clearly dominate nearby BGP visuals +4. 平静期仍需要观测存在感 + - collectors 和 activity layer 应让地图保持活跃 + - 一旦出现 incident，它们应明确压过附近 BGP 视觉元素 -5. incident geography should become `prefix-centric` - - collectors should remain evidence sources, not the primary event location - - preferred geography priority: +5. incident 地理位置应转向 `prefix-centric` + - collector 应保持证据来源身份，而不是主要事件位置 + - 推荐地理优先级： - `prefix_geography` - `prefix_scope` - `ASN organization region` - - `collector centroid` as final fallback - - `prefix_scope` should remain an observation-derived scope hint - - a new `prefix_geography` layer should be introduced for actual prefix-centric placement + - `collector centroid` 作为最终 fallback + - `prefix_scope` 应保持为由观测推导出的范围提示 + - 应新增真正面向 prefix 位置的 `prefix_geography` 层 -Reference inspiration: +参考灵感： - `World Monitor` - - sparse event symbols - - compact centers - - ring-like outward pulses - - stronger incident legibility than diffuse glow + - 稀疏事件符号 + - 紧凑中心 + - 类似环形的向外脉冲 + - 比弥散 glow 更强的 incident 可读性 -## Current Console Behavior +## 当前控制台行为 -Relevant page: +相关页面： - `frontend/src/pages/BGP/BGP.tsx` -Current BGP console page has three levels: +当前 BGP 控制台页面有三层： -1. observation summary - - total events - - collector count - - prefix count +1. 观测摘要 + - 总事件数 + - collector 数量 + - prefix 数量 -2. incident summary and incident table +2. incident 摘要和 incident 表格 -3. anomaly detail table plus recent observation events +3. anomaly 详情表和最近 observation events -This means the BGP page still has useful signal even when there are zero anomalies. +这意味着即使 anomaly 为零，BGP 页面仍有可用信号。 -## Known Product/Engineering Boundaries +## 已知产品/工程边界 -1. The current system is still closer to an event board than a full BGP sensing platform. -2. RIS coverage still needs to expand beyond narrow subscription scope. -3. BGPStream history is still not full MRT-to-prefix decoded analytics. -4. Collector geography still depends heavily on static RIPE RIS mappings. -5. Incident-to-cable/IXP/region association is still weak and early-stage. -6. Earth currently visualizes logical observation/impact structure, not true physical traffic paths. +1. 当前系统仍更接近事件看板，而不是完整 BGP sensing platform。 +2. RIS 覆盖范围仍需从较窄订阅范围继续扩展。 +3. BGPStream 历史数据仍不是完整 MRT-to-prefix 解码分析。 +4. Collector 地理位置仍高度依赖静态 RIPE RIS 映射。 +5. Incident 与海缆、IXP、区域之间的关联仍较弱，且处于早期阶段。 +6. Earth 当前可视化的是逻辑观测/影响结构，而不是真实物理流量路径。 -## Test Status +## 测试状态 -BGP-specific tests live in: +BGP 专项测试位于： - `backend/tests/test_bgp.py` -Verified status at this point: +当前已验证状态： -- `25 passed` for `backend/tests/test_bgp.py` -- `62 passed` for `backend/tests` +- `backend/tests/test_bgp.py` 为 `25 passed` +- `backend/tests` 为 `62 passed` -Covered areas include: +覆盖范围包括： - normalization - observation serialization - enrichment -- detectors, including route leak candidate and path flap +- detectors，包括 route leak candidate 和 path flap - incident aggregation - batch anomaly creation - BGP events/incidents API - summary endpoints -## Most Relevant Files +## 最相关文件 -Backend: +后端： - `backend/app/models/bgp_observation.py` - `backend/app/models/bgp_anomaly.py` @@ -318,7 +318,7 @@ Backend: - `backend/app/api/v1/bgp.py` - `backend/app/api/v1/visualization.py` -Frontend: +前端： - `frontend/src/pages/BGP/BGP.tsx` - `frontend/public/earth/js/bgp.js` @@ -327,29 +327,29 @@ Frontend: - `frontend/public/earth/js/constants.js` - `frontend/public/earth/index.html` -## Recommended Next Steps +## 推荐下一步 -### Next Backend / Detection Priority +### 后端 / 检测优先级 -1. Integrate real RPKI validation data. -2. Expand realtime collector coverage and include withdrawals more broadly. -3. Continue refining route leak and path instability detectors with stronger heuristics. +1. 集成真实 RPKI 校验数据。 +2. 扩展实时 collector 覆盖范围，并更广泛纳入 withdrawals。 +3. 用更强启发式继续完善 route leak 和 path instability detector。 -### Next Correlation / Storytelling Priority +### 关联 / 叙事优先级 -4. Strengthen incident aggregation semantics and titles. -5. Add weak correlation from incidents to: - - cable corridors - - landing points +4. 强化 incident 聚合语义和标题。 +5. 增加 incident 与以下对象的弱关联： + - 海缆走廊 + - 登陆点 - IXPs - - other traffic anomaly sources -6. Refine Earth hover/click handoff between collectors and incidents. + - 其它流量异常来源 +6. 优化 Earth 中 collector 和 incident 之间的 hover/click 交接。 -### Next Visualization Priority +### 可视化优先级 -7. Refine regional activity scoring so the activity layer is informative without becoming noisy. -8. Add more incident symbol types as new detectors land. -9. Add a real prefix geography source: - - `IPtoASN / IPtoCountry` as the first practical dataset - - `OpenGeoFeed` as a higher-quality override layer - - registry/whois only as fallback +7. 调整区域 activity scoring，让 activity layer 有信息量但不嘈杂。 +8. 随着新 detector 落地，增加更多 incident 符号类型。 +9. 增加真实 prefix geography 来源： + - `IPtoASN / IPtoCountry` 作为第一阶段可用数据集 + - `OpenGeoFeed` 作为更高质量 override 层 + - registry/whois 只作为 fallback diff --git a/docs/technical/zh/earth-frontend-context.md b/docs/technical/zh/earth-frontend-context.md index b2d6c5e9..944a0e88 100644 --- a/docs/technical/zh/earth-frontend-context.md +++ b/docs/technical/zh/earth-frontend-context.md @@ -1,11 +1,11 @@ -# Earth Frontend Context +# Earth 前端结构 本文件描述当前 Earth 大屏前端的真实结构，重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时，不再重复踩结构和状态同步上的坑。 相关规则建议一起参考： - [rules.md](/home/ray/dev/linkong/planet/rules.md) -- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) +- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md) ## 当前目标 @@ -378,4 +378,4 @@ Earth 前端和控制台前端不是同一套 UI 系统： 控制台相关结构见： -- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md) +- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md) diff --git a/docs/technical/zh/earth-layer-style-reference.md b/docs/technical/zh/earth-layer-style-reference.md index 993b18b8..8ffa1468 100644 --- a/docs/technical/zh/earth-layer-style-reference.md +++ b/docs/technical/zh/earth-layer-style-reference.md @@ -2,7 +2,7 @@ 本文记录当前 Earth 前端各图层的材质、颜色、透明度、线宽、半径偏移和 `renderOrder` 等样式属性。层级关系请配合 -[earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/earth-render-layer-order.md) +[earth-render-layer-order.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-render-layer-order.md) 查看。 ## 命名约定 diff --git a/docs/technical/zh/earth-news-live-streams-collector-format.md b/docs/technical/zh/earth-news-live-streams-collector-format.md index 2883e4d7..0def9db6 100644 --- a/docs/technical/zh/earth-news-live-streams-collector-format.md +++ b/docs/technical/zh/earth-news-live-streams-collector-format.md @@ -1,4 +1,4 @@ -# News Live Streams Collector Format +# 新闻直播采集格式 `news_live_streams` 采集器面向“频道目录 JSON”输入，而不是直接抓网页。 diff --git a/docs/technical/zh/earth-satellite-footprint-policy.md b/docs/technical/zh/earth-satellite-footprint-policy.md index 87891caa..129389db 100644 --- a/docs/technical/zh/earth-satellite-footprint-policy.md +++ b/docs/technical/zh/earth-satellite-footprint-policy.md @@ -1,11 +1,11 @@ -# Earth Satellite Footprint Policy +# Earth 卫星覆盖策略 本文件记录 Earth 卫星图层当前关于 `footprint` 的产品边界、资料依据和已落地实现，目标是避免把 Starlink 这套专用地表覆盖模型误用到其它星座上。 相关上下文： -- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) -- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md) +- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md) +- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md) - [backend/app/services/collectors/celestrak.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/celestrak.py) - [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js) diff --git a/docs/technical/zh/frontend-admin-frontend-context.md b/docs/technical/zh/frontend-admin-frontend-context.md index 3a92a847..a73656c6 100644 --- a/docs/technical/zh/frontend-admin-frontend-context.md +++ b/docs/technical/zh/frontend-admin-frontend-context.md @@ -1,11 +1,11 @@ -# Admin Frontend Context +# 控制台前端结构 本文件描述当前控制台前端的真实结构，目标是帮助后续页面开发、表格改造、布局治理和状态收口时快速找到正确入口。 相关规则建议一起参考： - [rules.md](/home/ray/dev/linkong/planet/rules.md) -- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) +- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md) ## 当前目标 @@ -263,7 +263,7 @@ 详细经验见： -- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) +- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-layout-guidelines.md) ## 当前推荐改动方式 @@ -290,4 +290,4 @@ Earth 相关结构见： -- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) +- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md) diff --git a/docs/technical/zh/frontend-layout-guidelines.md b/docs/technical/zh/frontend-layout-guidelines.md index 81b5c96d..a158bec8 100644 --- a/docs/technical/zh/frontend-layout-guidelines.md +++ b/docs/technical/zh/frontend-layout-guidelines.md @@ -1,4 +1,4 @@ -# Frontend Layout Guidelines +# 前端布局指南 本项目后台页面默认遵循“单屏工作区”布局规范。目标不是让页面永远不溢出，而是确保在常见桌面视口下： diff --git a/docs/technical/zh/manual.md b/docs/technical/zh/manual.md index 51eebae7..a38319ef 100644 --- a/docs/technical/zh/manual.md +++ b/docs/technical/zh/manual.md @@ -7,7 +7,7 @@ - 控制台：登录后的管理后台 - Docs：公开开发文档与使用手册 -快速启动路径见 [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md)。 +快速启动路径见 [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md)。 ## 入口总览 @@ -480,9 +480,9 @@ source ~/.zshrc && bun run build ## 相关文档 -- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/quickstart.md) -- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md) -- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) -- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/earth-layer-style-reference.md) -- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md) -- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md) +- [quickstart.md](/home/ray/dev/linkong/planet/docs/technical/zh/quickstart.md) +- [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md) +- [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md) +- [earth-layer-style-reference.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-layer-style-reference.md) +- [backend-system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-system-service-control.md) +- [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md) diff --git a/docs/technical/zh/ops-planet-sh-startup.md b/docs/technical/zh/ops-planet-sh-startup.md new file mode 100644 index 00000000..3da819bf --- /dev/null +++ b/docs/technical/zh/ops-planet-sh-startup.md @@ -0,0 +1,154 @@ +# planet.sh 启动性能优化 + +## 背景 + +`planet.sh` 管理所有服务的启动/停止/重启。原有实现存在以下问题： + +1. AI Provider 每次都重新构建（即使代码未变） +2. 杀端口速度极慢（最长等 45 秒） +3. 端口绑定检测用 Python 子进程（每次 ~300ms） +4. 无参 `restart` 与 `restart -b` 行为不一致 + +## 问题一：AI Provider 每次重建 + +### 根因 + +构建戳文件存放在 `/tmp/`，WSL/Linux 重启后 `/tmp` 被清空，导致三个条件中的"戳文件非空"这一条始终不满足，进而判定需要重建： + +```bash +# 三个条件必须同时成立才跳过重建 +image_exists AND stamp_non_empty AND fingerprint_match +``` + +### 修复 + +将戳文件路径从 `/tmp/` 改到持久路径： + +```bash +AI_PROVIDER_BUILD_STAMP_FILE="$HOME/.cache/planet/aiprovider_build.sha256" +``` + +写入时确保目录存在： + +```bash +write_ai_provider_build_stamp() { + mkdir -p "$(dirname "$AI_PROVIDER_BUILD_STAMP_FILE")" + compute_ai_provider_build_fingerprint > "$AI_PROVIDER_BUILD_STAMP_FILE" +} +``` + +### fingerprint 计算提速 + +原实现对整个 `aiprovider/` 打 tar 包再算 SHA，大目录下耗时可达数秒。改为 `find + stat`（只读文件元信息，不读内容）： + +```bash +compute_ai_provider_build_fingerprint() { + find aiprovider \ + -type f \ + ! -path '*/__pycache__/*' \ + ! -name '*.pyc' \ + ! -name '*.pyo' \ + | LC_ALL=C sort \ + | xargs -r stat --format="%Y %s %n" 2>/dev/null + sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null + python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null +} +``` + +速度提升约 10 倍（大量小文件场景），误报率相同（mtime+size 变化 ≡ 文件被修改）。 + +### 跳过重建的原理 + +fingerprint 一致时不执行 `docker compose build`，而是： + +```bash +docker start planet_aiprovider # 启动已存在的容器，几秒内完成 +``` + +`docker stop` 停容器，不删镜像；`cleanup_exit_containers` 删已退出容器，不删镜像。下次 `docker start` 会从现有镜像直接创建并启动容器。 + +## 问题二：杀端口速度慢 + +### 原因 + +`wait_for_port_release` 默认最多等 45 秒（15 次 × 3 秒）。 + +### 修复 + +将后台进程清理场景的超时缩短至 3 秒（TERM→1.5s→KILL→1.5s）： + +```bash +PORT_RELEASE_ATTEMPTS=15 +PORT_RELEASE_INTERVAL=0.2 # 每次等 0.2s，总计 3s + +# cleanup_backend_processes / kill_port_if_requested +wait_for_port_release "$port" 15 0.2 +``` + +`wait_for_port_release` 增加可选参数，允许不同场景使用不同超时： + +```bash +wait_for_port_release() { + local port="$1" + local max_attempts="${2:-$PORT_RELEASE_ATTEMPTS}" + local interval="${3:-$PORT_RELEASE_INTERVAL}" + ... +} +``` + +## 问题三：端口检测用 Python + +### 原因 + +`can_bind_port` 用 `python3 -c "import socket..."` 检测端口，每次调用约 300ms。 + +### 修复 + +优先使用系统工具（~10ms），Python 作为兜底： + +```bash +can_bind_port() { + local port="$1" + if command -v ss >/dev/null 2>&1; then + ! ss -tlnH 2>/dev/null | awk '{print $4}' | grep -qE ":${port}$" + return + fi + if command -v lsof >/dev/null 2>&1; then + [ -z "$(lsof -tiTCP:"${port}" -sTCP:LISTEN 2>/dev/null)" ] + return + fi + python3 - "$port" <<'PY' +import sys, socket +p = int(sys.argv[1]) +s = socket.socket() +s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) +try: + s.bind(("", p)); s.close(); sys.exit(0) +except OSError: + sys.exit(1) +PY +} +``` + +## 问题四：restart 行为不一致 + +### 现象 + +- `restart -b`：停全部服务 → 检查 AI Provider fingerprint → 按需重建 → 启动 +- `restart`（无参）：停全部服务 → AI Provider 总是判定需要重建（因戳文件在 /tmp） + +### 修复 + +修复戳文件路径后，无参 `restart` 同样使用 `stop + start`，fingerprint 检查正常生效，行为与 `restart -b` 完全一致。无需额外代码变更。 + +## 其他：移除不必要的 sleep + +启动链路中两处 `sleep 3` 在实际已有健康检查覆盖的情况下多余，已移除： + +- `start_backend_service`：数据库健康检查通过后的 `sleep 3` +- `restart_database_service`：重启后的等待 `sleep 3` + +## 相关文件 + +- `planet.sh` — 全量修改 +- `scripts/compute_aiprovider_dependency_fingerprint.py` — 依赖 fingerprint（未改动） diff --git a/docs/technical/zh/quickstart.md b/docs/technical/zh/quickstart.md index f69a2123..5f5a8e8a 100644 --- a/docs/technical/zh/quickstart.md +++ b/docs/technical/zh/quickstart.md @@ -1,6 +1,6 @@ -# Quickstart +# 快速开始 -这份 Quickstart 面向第一次启动 Planet 的开发者或演示操作者。目标是用最短路径把服务跑起来，并知道应该打开哪些入口。 +这份快速开始面向第一次启动 Planet 的开发者或演示操作者。目标是用最短路径把服务跑起来，并知道应该打开哪些入口。 ## 前置条件 @@ -187,7 +187,7 @@ ss -ltnp | grep -E ':3000|:8000' ## 下一步 -- 完整操作说明见 [manual.md](/home/ray/dev/linkong/planet/docs/technical/manual.md) -- 控制台结构见 [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md) -- Earth 结构见 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) -- 后端采集器见 [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/backend-collectors.md) +- 完整操作说明见 [manual.md](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md) +- 控制台结构见 [frontend-admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/frontend-admin-frontend-context.md) +- Earth 结构见 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-frontend-context.md) +- 后端采集器见 [backend-collectors.md](/home/ray/dev/linkong/planet/docs/technical/zh/backend-collectors.md) diff --git a/docs/version-history.md b/docs/version-history.md index 471db99e..40a45956 100644 --- a/docs/version-history.md +++ b/docs/version-history.md @@ -16,12 +16,13 @@ ## Current Version - `main` 当前主线历史推导到：`0.16.5` -- `dev` 当前开发分支历史推导到：`0.42.2` +- `dev` 当前开发分支历史推导到：`0.43.0` ## Timeline | Version | Type | Branch | Commit | Summary | | --- | --- | --- | --- | --- | +| `0.43.0` | feature | `dev` | `pending` | 新增 Earth 船舶追踪、自定义数据源映射、外部集成配置中心、Markdown 渲染器增强，并整理规则/技能文档加载约束 | | `0.42.2` | bugfix | `dev` | `pending` | Docs 中文模式补齐分组和文档标题翻译，并更新文档站品牌文案 | | `0.42.1` | bugfix | `dev` | `pending` | 修正 release skill 的 feature 版本计算规则，minor 进位时重置 patch 为 0 | | `0.42.0` | feature | `dev` | `pending` | 新增公开 `/docs` 文档站、中英文技术/使用文档、搜索与主题切换，并补充公共组件复用和 Earth 无高清材质边缘提示 | diff --git a/frontend/package.json b/frontend/package.json index 11971e3f..5b9629fd 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "planet-frontend", - "version": "0.42.2", + "version": "0.43.0", "private": true, "packageManager": "bun@1", "dependencies": { diff --git a/frontend/public/earth/index.html b/frontend/public/earth/index.html index 1eea3d96..3c800a8f 100644 --- a/frontend/public/earth/index.html +++ b/frontend/public/earth/index.html @@ -148,6 +148,16 @@ +
+ directions_boat +
+ 船只 + AIS Vessels +
+ +
hub
@@ -363,6 +373,10 @@ — 算力中心
+
+ — + AIS 船只 +
— BGP 事件 diff --git a/frontend/public/earth/js/constants.js b/frontend/public/earth/js/constants.js index ecf7f66d..2ea81481 100644 --- a/frontend/public/earth/js/constants.js +++ b/frontend/public/earth/js/constants.js @@ -198,6 +198,8 @@ export const PATHS = { cablesApi: '/api/v1/visualization/geo/cables', landingPointsApi: '/api/v1/visualization/geo/landing-points', computeCentersApi: '/api/v1/visualization/geo/compute-centers', + vesselsApi: '/api/v1/visualization/geo/vessels', + vesselTrackApi: (mmsi) => `/api/v1/visualization/vessels/${encodeURIComponent(mmsi)}/track`, bgpApi: '/api/v1/visualization/geo/bgp-anomalies', bgpIncidentsApi: '/api/v1/visualization/geo/bgp-incidents', bgpCollectorsApi: '/api/v1/visualization/geo/bgp-collectors', @@ -205,6 +207,36 @@ export const PATHS = { earthClientLogsApi: '/api/v1/system/logs/earth-client', }; +export const VESSEL_CONFIG = { + altitudeOffset: 0.56, + maxRenderedMarkers: 5000, + marker: { + baseScale: 7.5, + baseOpacity: 0.88, + hoverScale: 1.28, + lockedScale: 1.48, + dimmedScale: 0.78, + dimmedOpacity: 0.26, + }, + colors: { + cargo: "#4A90D9", + tanker: "#E85D04", + passenger: "#06D6A0", + fishing: "#FFD166", + military: "#73797E", + other: "#9B9B9B", + }, + sizeStabilization: { + min: 0.1, + max: 2.4, + }, + track: { + altitudeOffset: 0.7, + color: 0x7dd3fc, + opacity: 0.82, + }, +}; + export const COMPUTE_CENTER_CONFIG = { altitudeOffset: 0.48, maxRenderedMarkers: 300, diff --git a/frontend/public/earth/js/controls.js b/frontend/public/earth/js/controls.js index 83913a12..d734b3ad 100644 --- a/frontend/public/earth/js/controls.js +++ b/frontend/public/earth/js/controls.js @@ -36,6 +36,8 @@ import { getAtmosphereCloudsEnabled, setSatellitesEnabled, getSatellitesEnabled, + setVesselsEnabled, + getVesselsEnabled, } from "./main.js"; import { toggleTrails, @@ -52,6 +54,10 @@ import { getShowComputeCenters, getComputeCenterCount, } from "./compute-centers.js"; +import { + getShowVessels, + getVesselCount, +} from "./vessels.js"; import { ensureTVPanelReady, isTVPanelVisible, setTVPanelVisible } from "./tv.js"; import { createHUDPanel } from "./hud-panels.js"; import { @@ -1353,6 +1359,39 @@ function setComputeCentersLayerEnabled(button, enabled, { persist = true, silent return enabled; } +async function setVesselsLayerEnabled(button, enabled, { persist = true, silent = false } = {}) { + clearSelectionIfHiding(!enabled); + try { + if (enabled) { + setLayerButtonState(button, { + active: false, + loading: true, + tooltip: "船只加载中...", + }); + } + await setVesselsEnabled(enabled, { suppressStatus: silent, suppressLoadingUi: silent }); + setLayerButtonState(button, { + active: enabled, + loading: false, + tooltip: enabled ? "隐藏船只" : "显示船只", + }); + setEarthStatValue("vessel-count", `${getVesselCount()} 艘`); + syncMobileLayerCards(); + if (persist) persistEarthSettings(); + return enabled; + } catch (error) { + console.error("切换船只显示失败:", error); + setLayerButtonState(button, { + active: false, + loading: false, + tooltip: "显示船只", + }); + syncMobileLayerCards(); + if (persist) persistEarthSettings(); + return false; + } +} + function setTrailsLayerEnabled(button, enabled, { persist = true, silent = false } = {}) { toggleTrails(enabled); const disabledState = getLayerDisabledState("trails"); @@ -1527,6 +1566,23 @@ function getBuiltinLayerDefinitions() { setVisible: (visible, options = {}) => setBGPLayerEnabled(getLayerButton("bgp"), visible, options), }, + { + id: "vessels", + buttonId: "toggle-vessels", + icon: "directions_boat", + label: "船只", + meta: "AIS Vessels", + keywords: "船只 船舶 ais vessels ships maritime", + defaultActive: false, + displayOrder: 45, + startupPriority: 65, + startupMode: "visible", + startupLabel: "船只", + startupMessage: "正在加载船只...", + getVisible: () => getVesselsEnabled(), + setVisible: (visible, options = {}) => + setVesselsLayerEnabled(getLayerButton("vessels"), visible, options), + }, { id: "satellites", buttonId: "toggle-satellites", diff --git a/frontend/public/earth/js/info-card.js b/frontend/public/earth/js/info-card.js index 6ff81e6c..2c4d8987 100644 --- a/frontend/public/earth/js/info-card.js +++ b/frontend/public/earth/js/info-card.js @@ -201,6 +201,7 @@ function getMobilePopupTitle(type, data) { case 'bgp_collector': return data.collector || 'BGP观测站'; case 'supercomputer': return data.name || '超算'; case 'gpu_cluster': return data.name || 'GPU集群'; + case 'vessel': return data.name || '船只'; default: return '详情'; } } @@ -215,6 +216,7 @@ function getMobilePopupSubtitle(type, data) { case 'bgp_collector': return data.location || 'BGP观测站'; case 'supercomputer': return data.country || '超级计算机'; case 'gpu_cluster': return data.country || 'GPU集群'; + case 'vessel': return data.vessel_type || 'AIS 船只'; default: return ''; } } @@ -545,6 +547,23 @@ const CARD_CONFIG = { { key: 'source', label: '来源' }, { key: 'updated_at', label: '更新时间' } ] + }, + vessel: { + icon: '🚢', + title: '船只详情', + className: 'vessel', + fields: [ + { key: 'name', label: '名称' }, + { key: 'mmsi', label: 'MMSI' }, + { key: 'imo', label: 'IMO' }, + { key: 'flag', label: '旗帜' }, + { key: 'vessel_type', label: '船型' }, + { key: 'speed', label: '当前航速', unit: 'kn' }, + { key: 'course', label: '航向', unit: '°' }, + { key: 'status', label: '状态' }, + { key: 'length', label: '船长', unit: 'm' }, + { key: 'received_at', label: '更新时间' } + ] } }; diff --git a/frontend/public/earth/js/layer-startup-tasks.js b/frontend/public/earth/js/layer-startup-tasks.js index 86272dcf..c67a7a9a 100644 --- a/frontend/public/earth/js/layer-startup-tasks.js +++ b/frontend/public/earth/js/layer-startup-tasks.js @@ -18,6 +18,11 @@ import { loadComputeCenters, toggleComputeCenters, } from "./compute-centers.js"; +import { + getVesselLegendItems, + loadVessels, + toggleVessels, +} from "./vessels.js"; import { getCountryBoundaryLegendItems, loadCountryBoundaries, @@ -80,10 +85,33 @@ function registerBuiltinLayerStartupTasks() { registerCountryBoundaryStartupTask(); registerCableStartupTask(); registerComputeCenterStartupTask(); + registerVesselStartupTask(); registerBGPStartupTask(); registerSatelliteStartupTask(); } +function registerVesselStartupTask() { + registerLayerStartupTask("vessels", (context) => async (layer) => { + context.setLoadingMessage( + resolveStartupMessage(layer, "load", "正在加载船只..."), + ); + await context.yieldFrame(12); + try { + const vesselResult = await loadVessels(context.scene, context.earth); + if (!context.isCancelled()) { + toggleVessels(context.getShowVessels()); + context.updateVesselHud(vesselResult); + context.setLegendItems("vessels", getVesselLegendItems()); + context.refreshLegend(); + } + } catch (error) { + context.reportError(layer?.startupLabel || layer?.label || "船只", error); + } + if (context.isCancelled()) return; + await context.yieldFrame(16); + }); +} + function registerCableStartupTask() { registerLayerStartupTask("cables", (context) => async (layer) => { if (!context.isCablesEnabled()) return; diff --git a/frontend/public/earth/js/main.js b/frontend/public/earth/js/main.js index ae5c6584..60fa4f9a 100644 --- a/frontend/public/earth/js/main.js +++ b/frontend/public/earth/js/main.js @@ -168,6 +168,19 @@ import { toggleComputeCenters, updateComputeCenterVisualState, } from "./compute-centers.js"; +import { + clearVesselData, + clearVesselSelection, + getShowVessels, + getVesselCount, + getVesselLegendItems, + getVesselMarkers, + loadVessels, + setVesselMarkerState, + showVesselTrack, + toggleVessels, + updateVesselVisualState, +} from "./vessels.js"; import { setupControls, getAutoRotate, @@ -228,6 +241,7 @@ let inertialVelocity = { x: 0, y: 0 }; let hoveredCable = null; let hoveredBGP = null; let hoveredComputeCenter = null; +let hoveredVessel = null; let hoveredSatellite = null; let hoveredSatelliteIndex = null; let lockedSatellite = null; @@ -251,6 +265,7 @@ let isDataLoading = false; let currentLoadToken = 0; let cablesEnabled = true; let satellitesEnabled = false; +let vesselsEnabled = false; let cableToggleToken = 0; let satelliteToggleToken = 0; let satelliteHydrationToken = 0; @@ -278,6 +293,8 @@ const scratchBGPDirection = new THREE.Vector3(); const scratchBGPWorldPosition = new THREE.Vector3(); const scratchComputeCenterDirection = new THREE.Vector3(); const scratchComputeCenterWorldPosition = new THREE.Vector3(); +const scratchVesselDirection = new THREE.Vector3(); +const scratchVesselWorldPosition = new THREE.Vector3(); const scratchSatelliteWorldPosition = new THREE.Vector3(); const scratchSatelliteScreenPosition = new THREE.Vector3(); const scratchViewCenterWorld = new THREE.Vector3(); @@ -425,6 +442,7 @@ export function clearLockedObject() { clearCableSelection(); clearBGPSelection(); clearComputeCenterSelection(); + clearVesselSelection(); clearRelatedSatelliteHighlights(); setSatelliteRingState(null, "none", null); clearRuntimeSelection(); @@ -498,9 +516,11 @@ function resetTransientComputeCenterStates() { function clearTransientHoverState() { resetTransientBGPStates(); resetTransientComputeCenterStates(); + resetTransientVesselStates(); clearCountryBoundaryHover(); hoveredBGP = null; hoveredComputeCenter = null; + hoveredVessel = null; if (hoveredCable && !isSameCable(hoveredCable, lockedObject)) { setCableState(hoveredCable.userData.cableId, CABLE_STATE.NORMAL); @@ -515,6 +535,25 @@ function clearTransientHoverState() { setHoveredSatelliteIndex(null); } +function getFrontFacingVesselMarkers(markers) { + const earth = getEarth(); + if (!earth) return markers; + + scratchCameraToEarth.subVectors(camera.position, earth.position).normalize(); + + return markers.filter((marker) => { + scratchVesselWorldPosition.copy(marker.position); + marker.parent?.localToWorld(scratchVesselWorldPosition); + scratchVesselDirection + .subVectors(scratchVesselWorldPosition, earth.position) + .normalize(); + return ( + scratchCameraToEarth.dot(scratchVesselDirection) > + SATELLITE_CONFIG.frontFacingDotThreshold + ); + }); +} + function applyBGPHoverState(marker) { resetTransientBGPStates(); if (!marker) { @@ -549,6 +588,30 @@ function applyComputeCenterHoverState(marker) { } } +function resetTransientVesselStates() { + getVesselMarkers().forEach((marker) => { + if (marker !== lockedObject) { + setVesselMarkerState(marker, "normal"); + } + }); +} + +function applyVesselHoverState(marker) { + resetTransientVesselStates(); + if (!marker) { + hoveredVessel = null; + return; + } + hoveredVessel = marker; + if (marker !== lockedObject) { + setVesselMarkerState(marker, "hover"); + } +} + +function isSameVessel(marker1, marker2) { + return Boolean(marker1 && marker2 && marker1.userData?.mmsi === marker2.userData?.mmsi); +} + function getPrimaryBGPHoverTarget(bgpAnomalyIntersects, bgpCollectorIntersects) { if (bgpAnomalyIntersects.length > 0) { return bgpAnomalyIntersects[0].object; @@ -665,6 +728,37 @@ function showComputeCenterInfo(marker, coords) { }, coords); } +function formatVesselStatus(navStatus) { + if (navStatus === 1) return "锚泊"; + if (navStatus === 5) return "停靠"; + if (navStatus === 0) return "航行中"; + return navStatus ?? "-"; +} + +function showVesselInfo(marker, coords) { + setLegendMode("vessels"); + showInfoCard("vessel", { + name: marker.userData?.name || `MMSI ${marker.userData?.mmsi}`, + mmsi: marker.userData?.mmsi, + imo: marker.userData?.imo || "-", + flag: marker.userData?.flag || "-", + vessel_type: marker.userData?.vessel_type_name || "-", + speed: marker.userData?.sog ?? "-", + course: marker.userData?.cog ?? marker.userData?.heading ?? "-", + status: formatVesselStatus(marker.userData?.nav_status), + length: marker.userData?.length ?? "-", + received_at: marker.userData?.received_at + ? new Date(marker.userData.received_at).toLocaleString("zh-CN", { hour12: false }) + : "-", + }, coords); +} + +function getVesselBriefHtml(marker) { + const name = marker.userData?.name || `MMSI ${marker.userData?.mmsi}`; + const speed = marker.userData?.sog ?? "-"; + return `${name}
${marker.userData?.vessel_type_name || "Vessel"} · ${speed} kn`; +} + function getComputeCenterBriefHtml(marker) { const name = marker.userData?.name || "算力中心"; const type = formatComputeCenterTypeLabel(marker.userData?.site_type); @@ -864,6 +958,13 @@ function getComputeCenterFocusCoords(marker) { return { lat, lon }; } +function getVesselFocusCoords(marker) { + const lat = marker?.userData?.latitude; + const lon = marker?.userData?.longitude; + if (typeof lat !== "number" || typeof lon !== "number") return null; + return { lat, lon }; +} + async function focusSearchTarget(coords, zoom = Math.max(getZoomLevel(), 1.12)) { if (!coords || !camera) return; await focusEarthView(camera, { @@ -1037,6 +1138,33 @@ async function focusSearchComputeCenter(marker) { ); } +async function focusSearchVessel(marker) { + await setVesselsEnabled(true, { + suppressStatus: true, + suppressLoadingUi: true, + }); + interruptCruisePresentation({ resetLoop: true }); + clearLockedObject(); + setAutoRotate(false); + + const coords = getVesselFocusCoords(marker); + if (coords) { + await focusSearchTarget(coords, Math.max(getZoomLevel(), 1.2)); + } + + setVesselMarkerState(marker, "locked"); + lockedObject = marker; + lockedObjectType = "vessel"; + showVesselInfo(marker, getSearchCardCoords()); + showVesselTrack(marker, getEarth()).catch((error) => { + console.warn("船只轨迹加载失败:", error); + }); + showStatusMessage( + `已定位船只：${marker.userData?.name || marker.userData?.mmsi || "未知船只"}`, + "info", + ); +} + function resolveEarthSearchResults(query) { const results = []; const normalizedQuery = query.trim().toLowerCase(); @@ -1205,6 +1333,33 @@ function resolveEarthSearchResults(query) { }); }); + getVesselMarkers().forEach((marker) => { + const score = computeSearchScore( + normalizedQuery, + marker.userData?.name, + marker.userData?.mmsi, + marker.userData?.imo, + marker.userData?.flag, + marker.userData?.vessel_type_name, + "船只 船舶 ais vessel ship maritime", + ); + if (score < 0) return; + results.push({ + id: `vessel:${marker.userData?.mmsi || marker.uuid}`, + kind: "vessel", + icon: "directions_boat", + typeLabel: "船只", + title: marker.userData?.name || `MMSI ${marker.userData?.mmsi}`, + subtitle: [ + marker.userData?.vessel_type_name, + marker.userData?.flag, + marker.userData?.sog !== undefined ? `${marker.userData.sog} kn` : null, + ].filter(Boolean).join(" · ") || "AIS 船只", + score, + entity: marker, + }); + }); + return results .sort((left, right) => { if (right.score !== left.score) return right.score - left.score; @@ -1234,6 +1389,10 @@ async function handleSearchSelection(result) { } if (result.kind === "compute_center") { await focusSearchComputeCenter(result.entity); + return; + } + if (result.kind === "vessel") { + await focusSearchVessel(result.entity); } } @@ -1270,6 +1429,7 @@ function applyEarthStatsSummary(summary) { landingPointCount: `${summary.landingPointCount}个`, satelliteCount: `${summary.satelliteCount} 颗`, computeCenterCount: `${summary.computeCenterCount} 个`, + vesselCount: `${summary.vesselCount} 艘`, bgpAnomalyCount: `${summary.bgpEventCount} 起`, bgpCollectorCount: `${summary.bgpCollectorCount} 个`, bgpStatusSummary: formatBGPStatusFromSummary(summary), @@ -1291,6 +1451,7 @@ async function loadEarthStatsSummary() { landingPointCount: toCount(stats.landing_point_count), satelliteCount: toCount(stats.satellite_count), computeCenterCount: toCount(stats.compute_center_count), + vesselCount: toCount(stats.vessel_count), bgpEventCount: toCount(stats.bgp_event_count), bgpIncidentCount: toCount(stats.bgp_incident_count), bgpAnomalyCount: toCount(stats.bgp_anomaly_count), @@ -1945,6 +2106,23 @@ function updateSatelliteToggleUi(enabled, satelliteCount = getSatelliteCount()) setEarthStatValue("satellite-count", `${resolvedCount} 颗`); } +function updateVesselHud(result = {}) { + const count = Number(result.totalCount ?? getVesselCount() ?? 0); + setEarthStatValue("vessel-count", `${count} 艘`); +} + +function updateVesselToggleUi(enabled, vesselCount = getVesselCount()) { + const vesselBtn = document.getElementById("toggle-vessels"); + if (vesselBtn) { + setLayerButtonState(vesselBtn, { + active: enabled, + loading: false, + tooltip: enabled ? "隐藏船只" : "显示船只", + }); + } + setEarthStatValue("vessel-count", `${vesselCount || 0} 艘`); +} + function updateCableToggleUi(enabled) { const cableBtn = document.getElementById("toggle-cables"); if (cableBtn) { @@ -2063,6 +2241,29 @@ async function ensureSatellitesEnabled() { return loadResult.count; } +async function ensureVesselsEnabled() { + if (!scene || !camera || !renderer || destroyed) return 0; + const earth = getEarth(); + if (!earth) return 0; + + vesselsEnabled = true; + const result = await loadVessels(scene, earth); + toggleVessels(true); + updateVesselToggleUi(true, result.totalCount); + setLegendItems("vessels", getVesselLegendItems()); + refreshLegend(); + return result.totalCount; +} + +function disableVessels() { + vesselsEnabled = false; + toggleVessels(false); + clearVesselSelection(); + updateVesselToggleUi(false, 0); + setLegendItems("vessels", getVesselLegendItems()); + refreshLegend(); +} + function disableSatellites() { satellitesEnabled = false; satelliteToggleToken += 1; @@ -2079,6 +2280,7 @@ function updateStatsSummary() { const landingPointCount = getLandingPoints().length || earthStatsSummary?.landingPointCount || 0; const satelliteCount = getSatelliteCount() || earthStatsSummary?.satelliteCount || 0; + const vesselCount = getVesselCount() || earthStatsSummary?.vesselCount || 0; const computeCenterCount = getComputeCenterCount() || earthStatsSummary?.computeCenterCount || 0; const bgpEventCount = getBGPCount() || earthStatsSummary?.bgpEventCount || 0; @@ -2088,6 +2290,7 @@ function updateStatsSummary() { cableCount: `${cableCount}个`, landingPointCount: `${landingPointCount}个`, satelliteCount: `${satelliteCount} 颗`, + vesselCount: `${vesselCount} 艘`, computeCenterCount: `${computeCenterCount} 个`, bgpAnomalyCount: `${bgpEventCount} 起`, bgpCollectorCount: `${bgpCollectorCount} 个`, @@ -2333,6 +2536,7 @@ async function loadData() { clearBGPData(earth); clearCableData(earth); clearComputeCenterData(earth); + clearVesselData(earth); clearSatelliteData(); clearCountryBoundaryHover(); @@ -2370,8 +2574,10 @@ async function loadData() { updateCableToggleUi, updateSatelliteToggleUi, updateComputeCenterHud, + updateVesselHud, updateBGPHud, getShowComputeCenters, + getShowVessels, getShowCountryBoundaries, getShowBGP, isEarthTextureVisible: () => getEarthTextureVisible(), @@ -2429,6 +2635,7 @@ async function loadData() { setLegendItems("satellites", getSatelliteLegendItems()); setLegendItems("countryBoundaries", getCountryBoundaryLegendItems()); setLegendItems("computeCenters", getComputeCenterLegendItems()); + setLegendItems("vessels", getVesselLegendItems()); setLegendItems("bgp", getBGPLegendItems()); refreshLegend(); setLoading(false); @@ -2463,6 +2670,10 @@ export function getSatellitesEnabled() { return satellitesEnabled; } +export function getVesselsEnabled() { + return vesselsEnabled; +} + export async function setCablesEnabled( enabled, { suppressStatus = false, suppressLoadingUi = false } = {}, @@ -2668,6 +2879,62 @@ export async function setSatellitesEnabled( } } +export async function setVesselsEnabled( + enabled, + { suppressStatus = false, suppressLoadingUi = false } = {}, +) { + if (enabled === vesselsEnabled) { + updateVesselToggleUi(enabled); + return getVesselCount(); + } + + if (!enabled) { + clearSelectionAndInfo(); + disableVessels(); + if (!suppressStatus) { + showStatusMessage("船只已隐藏", "info"); + } + return 0; + } + + if (!suppressLoadingUi) { + setLoadingMessage("正在加载船只数据..."); + setLoading(true); + hideError(); + } + + try { + const vesselCount = await ensureVesselsEnabled(); + if (!suppressStatus) { + showStatusMessage("船只已显示", "info"); + } + return vesselCount; + } catch (error) { + vesselsEnabled = false; + clearVesselData(getEarth()); + updateVesselToggleUi(false, 0); + const message = `船只加载失败: ${error?.message || String(error)}`; + void reportEarthClientLog({ + level: "error", + category: "layer-toggle", + module: "vessels", + message, + detail: error, + }); + if (!suppressLoadingUi) { + showError(message); + } + if (!suppressStatus) { + showStatusMessage(message, "error"); + } + throw error; + } finally { + if (!suppressLoadingUi) { + setLoading(false); + } + } +} + function setupEventListeners() { const handleResize = () => onWindowResize(); const handleVisibilityChange = () => onVisibilityChange(); @@ -2865,6 +3132,11 @@ function onMouseMove(event) { const computeCenterIntersects = getShowComputeCenters() ? interactionRaycaster.intersectObjects(frontFacingComputeCenterMarkers) : []; + const vesselIntersects = getShowVessels() + ? interactionRaycaster.intersectObjects( + getFrontFacingVesselMarkers(getVesselMarkers()), + ) + : []; let hoveredSat = null; let hoveredSatIndexFromIntersect = null; @@ -2886,6 +3158,8 @@ function onMouseMove(event) { const hoveredComputeCenterMarker = computeCenterIntersects.length > 0 ? computeCenterIntersects[0].object : null; + const hoveredVesselMarker = + vesselIntersects.length > 0 ? vesselIntersects[0].object : null; if ( hoveredComputeCenter && @@ -2893,6 +3167,9 @@ function onMouseMove(event) { ) { clearTransientHoverState(); } + if (hoveredVessel && !isSameVessel(hoveredVessel, hoveredVesselMarker)) { + clearTransientHoverState(); + } if ( hoveredCable && @@ -2936,6 +3213,18 @@ function onMouseMove(event) { getComputeCenterBriefHtml(hoveredComputeCenterMarker), ); objectTooltipShown = true; + } else if ( + hoveredVesselMarker && + getShowVessels() && + lockedObjectType !== "vessel" + ) { + applyVesselHoverState(hoveredVesselMarker); + showTooltip( + event.clientX + TOOLTIP_CURSOR_OFFSET, + event.clientY + TOOLTIP_CURSOR_OFFSET, + getVesselBriefHtml(hoveredVesselMarker), + ); + objectTooltipShown = true; } else if (cableIntersects.length > 0 && getShowCables()) { const cable = cableIntersects[0].object; hoveredCable = cable; @@ -2966,9 +3255,12 @@ function onMouseMove(event) { applyBGPHoverState(lockedObject); } else if (lockedObjectType === "compute_center" && lockedObject) { applyComputeCenterHoverState(lockedObject); + } else if (lockedObjectType === "vessel" && lockedObject) { + applyVesselHoverState(lockedObject); } else if (!lockedObjectType && !isCruisePresentationPinned()) { resetTransientBGPStates(); resetTransientComputeCenterStates(); + resetTransientVesselStates(); hideInfoCard(); } @@ -3185,6 +3477,11 @@ function onClick(event) { getFrontFacingComputeCenterMarkers(getComputeCenterMarkers()), ) : []; + const vesselIntersects = getShowVessels() + ? interactionRaycaster.intersectObjects( + getFrontFacingVesselMarkers(getVesselMarkers()), + ) + : []; const satIntersects = getSatellitePointerIntersections(event); const clickedBGPMarker = getShowBGP() @@ -3193,6 +3490,9 @@ function onClick(event) { const clickedComputeCenterMarker = computeCenterIntersects.length > 0 ? computeCenterIntersects[0].object : null; + const clickedVesselMarker = vesselIntersects.length > 0 + ? vesselIntersects[0].object + : null; if (clickedBGPMarker?.userData?.type === "bgp") { interruptCruisePresentation(); @@ -3260,6 +3560,26 @@ function onClick(event) { return; } + if (clickedVesselMarker?.userData?.type === "vessel") { + interruptCruisePresentation(); + clearLockedObject(); + + const clickedMarker = clickedVesselMarker; + setVesselMarkerState(clickedMarker, "locked"); + lockedObject = clickedMarker; + lockedObjectType = "vessel"; + setAutoRotate(false); + showVesselInfo(clickedMarker, { x: event.clientX, y: event.clientY }); + showVesselTrack(clickedMarker, earth).catch((error) => { + console.warn("船只轨迹加载失败:", error); + }); + showStatusMessage( + `已选择船只: ${clickedMarker.userData?.name || clickedMarker.userData?.mmsi}`, + "info", + ); + return; + } + if (cableIntersects.length > 0 && getShowCables()) { interruptCruisePresentation(); clearLockedObject(); @@ -3412,6 +3732,7 @@ function animate() { : null; updateBGPVisualState(lockedObjectType, lockedObject, camera, activeCruiseMarker); updateComputeCenterVisualState(lockedObjectType, lockedObject, camera); + updateVesselVisualState(lockedObjectType, lockedObject, camera); if (lockedObjectType === "cable" && lockedObject) { applyLandingPointVisualState(lockedObject.userData.name, false, camera); diff --git a/frontend/public/earth/js/ui.js b/frontend/public/earth/js/ui.js index 467d763f..e5575294 100644 --- a/frontend/public/earth/js/ui.js +++ b/frontend/public/earth/js/ui.js @@ -194,6 +194,7 @@ export function updateEarthStats(stats) { if (has("computeCenterCount")) { setEarthStatValue("compute-center-count", String(stats.computeCenterCount || 0)); } + if (has("vesselCount")) setEarthStatValue("vessel-count", String(stats.vesselCount || 0)); if (has("bgpAnomalyCount")) setEarthStatValue("bgp-anomaly-count", String(stats.bgpAnomalyCount || 0)); if (has("bgpCollectorCount")) { setEarthStatValue("bgp-collector-count", String(stats.bgpCollectorCount || 0)); diff --git a/frontend/public/earth/js/vessels.js b/frontend/public/earth/js/vessels.js new file mode 100644 index 00000000..d6bde68b --- /dev/null +++ b/frontend/public/earth/js/vessels.js @@ -0,0 +1,280 @@ +import * as THREE from "three"; + +import { CONFIG, PATHS, VESSEL_CONFIG } from "./constants.js"; +import { getSurfaceMarkerCameraScale, latLonToVector3 } from "./utils.js"; + +const vesselGroup = new THREE.Group(); +const vesselMarkers = []; +const textureCache = new Map(); +let showVessels = false; +let activeTrackLine = null; + +const VESSEL_RENDER_ORDER = 4.4; + +function normalizeVesselType(value, code) { + const type = String(value || "").trim().toLowerCase(); + const numericCode = Number(code); + if (type.includes("cargo") || (numericCode >= 70 && numericCode <= 79)) return "cargo"; + if (type.includes("tanker") || (numericCode >= 80 && numericCode <= 89)) return "tanker"; + if (type.includes("passenger") || (numericCode >= 60 && numericCode <= 69)) return "passenger"; + if (type.includes("fishing") || numericCode === 30) return "fishing"; + if (type.includes("military") || numericCode === 35) return "military"; + return "other"; +} + +function createVesselTexture(type, anchored) { + const textureKey = `${type}:${anchored ? "anchored" : "moving"}`; + if (textureCache.has(textureKey)) return textureCache.get(textureKey); + + const color = VESSEL_CONFIG.colors[type] || VESSEL_CONFIG.colors.other; + const canvas = document.createElement("canvas"); + canvas.width = 96; + canvas.height = 96; + const context = canvas.getContext("2d"); + context.clearRect(0, 0, 96, 96); + context.save(); + context.translate(48, 48); + context.fillStyle = color; + context.globalAlpha = anchored ? 0.55 : 0.96; + context.shadowColor = color; + context.shadowBlur = anchored ? 8 : 14; + context.beginPath(); + if (anchored) { + context.arc(0, 0, 18, 0, Math.PI * 2); + } else { + context.moveTo(0, -28); + context.lineTo(21, 24); + context.lineTo(0, 13); + context.lineTo(-21, 24); + context.closePath(); + } + context.fill(); + context.restore(); + + const texture = new THREE.CanvasTexture(canvas); + texture.needsUpdate = true; + textureCache.set(textureKey, texture); + return texture; +} + +function buildVesselMarkerData(feature) { + const props = feature?.properties || {}; + const coordinates = feature?.geometry?.coordinates || []; + const longitude = Number(coordinates[0]); + const latitude = Number(coordinates[1]); + if (!Number.isFinite(latitude) || !Number.isFinite(longitude)) return null; + + const type = normalizeVesselType(props.vessel_type_name, props.vessel_type); + const navStatus = Number(props.nav_status); + const speed = Number(props.sog); + const anchored = navStatus === 1 || navStatus === 5 || (Number.isFinite(speed) && speed < 0.5); + + return { + ...props, + latitude, + longitude, + type, + anchored, + course: Number(props.cog ?? props.heading ?? 0), + }; +} + +function createVesselMarker(markerData) { + const material = new THREE.SpriteMaterial({ + map: createVesselTexture(markerData.type, markerData.anchored), + transparent: true, + depthWrite: false, + opacity: VESSEL_CONFIG.marker.baseOpacity, + rotation: markerData.anchored + ? 0 + : THREE.MathUtils.degToRad(-markerData.course), + }); + const marker = new THREE.Sprite(material); + marker.position.copy( + latLonToVector3( + markerData.latitude, + markerData.longitude, + CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset, + ), + ); + marker.scale.setScalar(VESSEL_CONFIG.marker.baseScale); + marker.renderOrder = VESSEL_RENDER_ORDER; + marker.visible = showVessels; + marker.userData = { + ...markerData, + type: "vessel", + vessel_kind: markerData.type, + baseScale: VESSEL_CONFIG.marker.baseScale, + state: "normal", + }; + vesselGroup.add(marker); + vesselMarkers.push(marker); +} + +function clearGroup(group) { + for (let index = group.children.length - 1; index >= 0; index -= 1) { + const child = group.children[index]; + child.material?.dispose?.(); + child.geometry?.dispose?.(); + group.remove(child); + } +} + +function getDistanceScale(camera) { + return getSurfaceMarkerCameraScale(camera, { + altitudeOffset: VESSEL_CONFIG.altitudeOffset, + referenceFov: 75, + min: VESSEL_CONFIG.sizeStabilization.min, + max: VESSEL_CONFIG.sizeStabilization.max, + }); +} + +export function getVesselMarkers() { + return vesselMarkers; +} + +export function getVesselCount() { + return vesselMarkers.length; +} + +export function getShowVessels() { + return showVessels; +} + +export function toggleVessels(show) { + showVessels = Boolean(show); + vesselGroup.visible = showVessels; + vesselMarkers.forEach((marker) => { + marker.visible = showVessels; + }); + if (activeTrackLine) { + activeTrackLine.visible = showVessels; + } +} + +export function clearVesselSelection() { + vesselMarkers.forEach((marker) => setVesselMarkerState(marker, "normal")); + clearVesselTrack(); +} + +function clearVesselTrack() { + if (activeTrackLine?.parent) { + activeTrackLine.parent.remove(activeTrackLine); + } + activeTrackLine?.geometry?.dispose?.(); + activeTrackLine?.material?.dispose?.(); + activeTrackLine = null; +} + +export function setVesselMarkerState(marker, state = "normal") { + if (!marker || marker.userData?.type !== "vessel") return; + marker.userData.state = state; +} + +export function clearVesselData(earth) { + vesselMarkers.length = 0; + clearVesselSelection(); + clearGroup(vesselGroup); + if (earth && vesselGroup.parent === earth) { + earth.remove(vesselGroup); + } +} + +export async function loadVessels(_scene, earth, options = {}) { + const params = new URLSearchParams(); + params.set("limit", String(options.limit || VESSEL_CONFIG.maxRenderedMarkers)); + const response = await fetch(`${PATHS.vesselsApi}?${params.toString()}`); + if (!response.ok) { + throw new Error(`Vessels HTTP ${response.status}`); + } + const payload = await response.json(); + const features = Array.isArray(payload?.features) ? payload.features : []; + + clearVesselData(earth); + features + .map((feature) => buildVesselMarkerData(feature)) + .filter(Boolean) + .slice(0, VESSEL_CONFIG.maxRenderedMarkers) + .forEach((markerData) => createVesselMarker(markerData)); + + if (earth && !vesselGroup.parent) { + earth.add(vesselGroup); + } + vesselGroup.visible = showVessels; + + return { + totalCount: vesselMarkers.length, + stats: payload?.stats || {}, + }; +} + +export async function showVesselTrack(marker, earth) { + clearVesselTrack(); + if (!marker?.userData?.mmsi || !earth) return null; + + const response = await fetch(PATHS.vesselTrackApi(marker.userData.mmsi)); + if (!response.ok) { + throw new Error(`Vessel track HTTP ${response.status}`); + } + const payload = await response.json(); + const coordinates = payload?.features?.[0]?.geometry?.coordinates || []; + if (coordinates.length < 2) return null; + + const points = coordinates + .map(([lon, lat]) => + latLonToVector3( + Number(lat), + Number(lon), + CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset, + ), + ) + .filter((point) => Number.isFinite(point.x) && Number.isFinite(point.y) && Number.isFinite(point.z)); + if (points.length < 2) return null; + + const geometry = new THREE.BufferGeometry().setFromPoints(points); + const material = new THREE.LineBasicMaterial({ + color: VESSEL_CONFIG.track.color, + transparent: true, + opacity: VESSEL_CONFIG.track.opacity, + depthWrite: false, + }); + activeTrackLine = new THREE.Line(geometry, material); + activeTrackLine.renderOrder = VESSEL_RENDER_ORDER - 0.1; + earth.add(activeTrackLine); + return activeTrackLine; +} + +export function getVesselLegendItems() { + return [ + { label: "货轮", color: VESSEL_CONFIG.colors.cargo }, + { label: "油轮", color: VESSEL_CONFIG.colors.tanker }, + { label: "客船", color: VESSEL_CONFIG.colors.passenger }, + { label: "渔船", color: VESSEL_CONFIG.colors.fishing }, + { label: "军舰", color: VESSEL_CONFIG.colors.military }, + { label: "其他", color: VESSEL_CONFIG.colors.other }, + ]; +} + +export function updateVesselVisualState(lockedObjectType, lockedObject, camera) { + const hasFocus = lockedObjectType === "vessel" && lockedObject; + const distanceScale = getDistanceScale(camera); + vesselMarkers.forEach((marker) => { + const isLocked = lockedObjectType === "vessel" && lockedObject === marker; + const state = marker.userData?.state || "normal"; + let opacity = VESSEL_CONFIG.marker.baseOpacity; + let scaleMultiplier = 1; + if (isLocked) { + opacity = 1; + scaleMultiplier = VESSEL_CONFIG.marker.lockedScale; + } else if (state === "hover") { + opacity = 0.98; + scaleMultiplier = VESSEL_CONFIG.marker.hoverScale; + } else if (hasFocus) { + opacity = VESSEL_CONFIG.marker.dimmedOpacity; + scaleMultiplier = VESSEL_CONFIG.marker.dimmedScale; + } + marker.material.opacity = showVessels ? opacity : 0; + marker.scale.setScalar(marker.userData.baseScale * scaleMultiplier * distanceScale); + marker.visible = showVessels; + }); +} diff --git a/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx b/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx index 983f0233..9eb28c3d 100644 --- a/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx +++ b/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx @@ -1,4 +1,5 @@ -import { memo } from 'react' +import { CheckOutlined, CopyOutlined } from '@ant-design/icons' +import { memo, useState } from 'react' import type { ReactNode } from 'react' import Scrollbar from '../Scrollbar/Scrollbar' @@ -15,13 +16,40 @@ interface MarkdownLink { external?: boolean } +interface ListLine { + indent: number + ordered: boolean + content: string +} + +interface ListItemNode { + content: string + checked?: boolean + children: ReactNode[] +} + +interface ParsedList { + node: ReactNode + nextIndex: number +} + +const INLINE_PATTERN = /(!\[[^\]]*]\([^)]+\)|\[[^\]]+\]\([^)]+\)|`[^`]+`|\*\*[^*]+\*\*|~~[^~]+~~|\*[^*]+\*|https?:\/\/[^\s<)]+)/g +const COPY_FEEDBACK_MS = 1400 + +function resolveMarkdownLink(href: string, transformLink?: MarkdownRendererProps['transformLink']): MarkdownLink { + const resolvedLink = transformLink?.(href) + return { + href: resolvedLink?.href || href, + external: resolvedLink?.external ?? /^https?:\/\//.test(href), + } +} + function renderInlineMarkdown(text: string, transformLink?: MarkdownRendererProps['transformLink']): ReactNode[] { const result: ReactNode[] = [] - const pattern = /(\[[^\]]+\]\([^)]+\)|`[^`]+`|\*\*[^*]+\*\*|\*[^*]+\*)/g let lastIndex = 0 let key = 0 - for (const match of text.matchAll(pattern)) { + for (const match of text.matchAll(INLINE_PATTERN)) { const matchedText = match[0] const start = match.index ?? 0 @@ -29,27 +57,55 @@ function renderInlineMarkdown(text: string, transformLink?: MarkdownRendererProp result.push(text.slice(lastIndex, start)) } - if (matchedText.startsWith('[')) { - const linkMatch = matchedText.match(/^\[([^\]]+)\]\(([^)]+)\)$/) - if (linkMatch) { - const resolvedLink = transformLink?.(linkMatch[2]) - const href = resolvedLink?.href || linkMatch[2] - const isExternal = resolvedLink?.external ?? true + const imageMatch = matchedText.match(/^!\[([^\]]*)]\(([^)]+)\)$/) + if (imageMatch) { + result.push( + {imageMatch[1]}, + ) + key += 1 + lastIndex = start + matchedText.length + continue + } - result.push( - - {linkMatch[1]} - , - ) - key += 1 - lastIndex = start + matchedText.length - continue - } + const linkMatch = matchedText.match(/^\[([^\]]+)]\(([^)]+)\)$/) + if (linkMatch) { + const link = resolveMarkdownLink(linkMatch[2], transformLink) + result.push( + + {linkMatch[1]} + , + ) + key += 1 + lastIndex = start + matchedText.length + continue + } + + if (/^https?:\/\//.test(matchedText)) { + const link = resolveMarkdownLink(matchedText, transformLink) + result.push( + + {matchedText} + , + ) + key += 1 + lastIndex = start + matchedText.length + continue } if (matchedText.startsWith('`')) { @@ -66,6 +122,13 @@ function renderInlineMarkdown(text: string, transformLink?: MarkdownRendererProp continue } + if (matchedText.startsWith('~~')) { + result.push({matchedText.slice(2, -2)}) + key += 1 + lastIndex = start + matchedText.length + continue + } + if (matchedText.startsWith('*')) { result.push({matchedText.slice(1, -1)}) key += 1 @@ -81,6 +144,161 @@ function renderInlineMarkdown(text: string, transformLink?: MarkdownRendererProp return result } +async function copyToClipboard(text: string): Promise { + if (navigator.clipboard?.writeText) { + await navigator.clipboard.writeText(text) + return + } + + const textarea = document.createElement('textarea') + textarea.value = text + textarea.setAttribute('readonly', '') + textarea.style.position = 'fixed' + textarea.style.top = '-9999px' + document.body.appendChild(textarea) + textarea.select() + document.execCommand('copy') + document.body.removeChild(textarea) +} + +function MarkdownCodeBlock({ code, language }: { code: string; language?: string }) { + const [copied, setCopied] = useState(false) + const label = language?.trim() || 'text' + + const handleCopy = async () => { + await copyToClipboard(code) + setCopied(true) + window.setTimeout(() => setCopied(false), COPY_FEEDBACK_MS) + } + + return ( +
+
+ {label} + +
+ + 
+          {code}
+
+ +
+ ) +} + +function parseListLine(line: string): ListLine | null { + const match = line.match(/^(\s*)([-*+]|\d+[.)])\s+(.+)$/) + if (!match) return null + + return { + indent: match[1].replace(/\t/g, ' ').length, + ordered: /^\d/.test(match[2]), + content: match[3], + } +} + +function parseTaskContent(content: string): { content: string; checked?: boolean } { + const taskMatch = content.match(/^\[( |x|X)]\s+(.+)$/) + if (!taskMatch) return { content } + + return { + content: taskMatch[2], + checked: taskMatch[1].toLowerCase() === 'x', + } +} + +function renderListItemContent( + item: ListItemNode, + transformLink?: MarkdownRendererProps['transformLink'], +): ReactNode { + if (typeof item.checked === 'boolean') { + return ( + <> + + {renderInlineMarkdown(item.content, transformLink)} + + ) + } + + return renderInlineMarkdown(item.content, transformLink) +} + +function parseList( + lines: string[], + startIndex: number, + baseIndent: number, + ordered: boolean, + transformLink?: MarkdownRendererProps['transformLink'], +): ParsedList { + const items: ListItemNode[] = [] + let index = startIndex + + while (index < lines.length) { + const listLine = parseListLine(lines[index]) + if (!listLine) break + if (listLine.indent < baseIndent || listLine.ordered !== ordered) break + + if (listLine.indent > baseIndent) { + if (items.length === 0) break + const nested = parseList(lines, index, listLine.indent, listLine.ordered, transformLink) + items[items.length - 1].children.push(nested.node) + index = nested.nextIndex + continue + } + + const taskContent = parseTaskContent(listLine.content) + items.push({ + content: taskContent.content, + checked: taskContent.checked, + children: [], + }) + index += 1 + } + + const Tag = ordered ? 'ol' : 'ul' + return { + node: ( + typeof item.checked === 'boolean') ? 'markdown-renderer__task-list' : undefined}> + {items.map((item, itemIndex) => ( +
  • + {renderListItemContent(item, transformLink)} + {item.children} +
    • + ))} +     + ), + nextIndex: index, + } +} + +function renderHeading( + level: number, + id: string | undefined, + content: ReactNode[], + key: string, +): ReactNode { + if (level === 1) return

    {content}

    + if (level === 2) return

    {content}

    + if (level === 3) return

    {content}

    + if (level === 4) return

    {content}

    + if (level === 5) return
    {content}
    + return
    {content}
    +} + function MarkdownRenderer({ markdown, className, @@ -106,8 +324,10 @@ function MarkdownRenderer({ continue } - if (trimmed.startsWith('```')) { + const fenceMatch = trimmed.match(/^```([^`]*)$/) + if (fenceMatch) { const codeLines: string[] = [] + const language = fenceMatch[1].trim().split(/\s+/)[0] index += 1 while (index < lines.length && !lines[index].trim().startsWith('```')) { codeLines.push(lines[index]) @@ -117,11 +337,11 @@ function MarkdownRenderer({ index += 1 } nodes.push( - - 
    -            {codeLines.join('\n')}
-
    -     , + , ) continue } @@ -135,63 +355,33 @@ function MarkdownRenderer({ const headingMatch = trimmed.match(/^(#{1,6})\s+(.+)$/) if (headingMatch) { const level = headingMatch[1].length - const headingText = headingMatch[2] + const headingText = headingMatch[2].replace(/\s+#+\s*$/, '') const content = renderInlineMarkdown(headingText, transformLink) const headingId = resolveHeadingId?.(headingText, level) - if (level === 1) nodes.push(

    {content}

    ) - else if (level === 2) nodes.push(

    {content}

    ) - else if (level === 3) nodes.push(

    {content}

    ) - else if (level === 4) nodes.push(

    {content}

    ) - else nodes.push(

    {content}

    ) + nodes.push(renderHeading(level, headingId, content, `block-${index}`)) index += 1 continue } - if (trimmed.startsWith('> ')) { + if (trimmed.startsWith('>')) { const quoteLines: string[] = [] - while (index < lines.length && lines[index].trim().startsWith('> ')) { - quoteLines.push(lines[index].trim().slice(2)) - index += 1 - } - nodes.push(
    {quoteLines.join(' ')}
    ) - continue - } - - const unorderedMatch = trimmed.match(/^[-*]\s+(.+)$/) - if (unorderedMatch) { - const items: string[] = [] - while (index < lines.length) { - const itemMatch = lines[index].trim().match(/^[-*]\s+(.+)$/) - if (!itemMatch) break - items.push(itemMatch[1]) + while (index < lines.length && lines[index].trim().startsWith('>')) { + quoteLines.push(lines[index].trim().replace(/^>\s?/, '')) index += 1 } nodes.push( -
      - {items.map((item, itemIndex) => ( -
    • {renderInlineMarkdown(item, transformLink)}
      • - ))} -
    , +
    + {renderInlineMarkdown(quoteLines.join(' '), transformLink)} +
    , ) continue } - const orderedMatch = trimmed.match(/^\d+\.\s+(.+)$/) - if (orderedMatch) { - const items: string[] = [] - while (index < lines.length) { - const itemMatch = lines[index].trim().match(/^\d+\.\s+(.+)$/) - if (!itemMatch) break - items.push(itemMatch[1]) - index += 1 - } - nodes.push( -
      - {items.map((item, itemIndex) => ( -
    1. {renderInlineMarkdown(item, transformLink)}
      2. - ))} -
    , - ) + const listLine = parseListLine(line) + if (listLine) { + const parsedList = parseList(lines, index, listLine.indent, listLine.ordered, transformLink) + nodes.push(parsedList.node) + index = parsedList.nextIndex continue } @@ -241,9 +431,24 @@ function MarkdownRenderer({ const paragraphLines: string[] = [] while (index < lines.length && lines[index].trim()) { + if ( + lines[index].trim().startsWith('```') || + lines[index].trim().startsWith('>') || + parseListLine(lines[index]) || + /^#{1,6}\s+/.test(lines[index].trim()) || + /^(-{3,}|\*{3,}|_{3,})$/.test(lines[index].trim()) + ) { + break + } paragraphLines.push(lines[index].trim()) index += 1 } + + if (paragraphLines.length === 0) { + index += 1 + continue + } + nodes.push(

    {renderInlineMarkdown(paragraphLines.join(' '), transformLink)}

    ) } diff --git a/frontend/src/index.css b/frontend/src/index.css index 1c45730f..a4a07e4d 100644 --- a/frontend/src/index.css +++ b/frontend/src/index.css @@ -2080,7 +2080,9 @@ body { .markdown-renderer h1, .markdown-renderer h2, .markdown-renderer h3, -.markdown-renderer h4 { +.markdown-renderer h4, +.markdown-renderer h5, +.markdown-renderer h6 { margin: 1.2em 0 0.5em; color: #111827; font-weight: 600; @@ -2099,12 +2101,20 @@ body { font-size: 16px; } +.markdown-renderer h4 { + font-size: 15px; +} + +.markdown-renderer h5, +.markdown-renderer h6 { + font-size: 14px; +} + .markdown-renderer p, .markdown-renderer ul, .markdown-renderer ol, .markdown-renderer blockquote, -.markdown-renderer pre, -.markdown-renderer__code-scroll, +.markdown-renderer__code-block, .markdown-renderer hr, .markdown-renderer__table-wrap { margin: 0 0 0.9em; @@ -2119,6 +2129,34 @@ body { margin-top: 0.25em; } +.markdown-renderer li > ul, +.markdown-renderer li > ol { + margin: 0.3em 0 0; +} + +.markdown-renderer__task-list { + list-style: none; + padding-left: 0; +} + +.markdown-renderer__task-item { + display: flex; + flex-wrap: wrap; + align-items: flex-start; + gap: 8px; +} + +.markdown-renderer__task-item > ul, +.markdown-renderer__task-item > ol { + flex-basis: 100%; +} + +.markdown-renderer__task-checkbox { + flex: 0 0 auto; + margin-top: 0.42em; + accent-color: #1677ff; +} + .markdown-renderer blockquote { padding: 10px 14px; border-left: 3px solid #91caff; @@ -2127,17 +2165,67 @@ body { color: #1f2937; } +.markdown-renderer__code-block { + overflow: hidden; + border-radius: 10px; + background: #0f172a; + box-shadow: inset 0 0 0 1px rgba(148, 163, 184, 0.18); +} + +.markdown-renderer__code-toolbar { + display: flex; + align-items: center; + justify-content: space-between; + min-height: 34px; + padding: 6px 8px 6px 12px; + border-bottom: 1px solid rgba(148, 163, 184, 0.18); + background: rgba(15, 23, 42, 0.94); + color: #cbd5e1; + font-size: 12px; +} + +.markdown-renderer__code-language { + overflow: hidden; + font-family: "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace; + text-overflow: ellipsis; + white-space: nowrap; +} + +.markdown-renderer__code-copy { + display: inline-flex; + align-items: center; + justify-content: center; + gap: 6px; + width: 26px; + height: 26px; + padding: 0; + border: 1px solid rgba(148, 163, 184, 0.28); + border-radius: 6px; + background: rgba(30, 41, 59, 0.88); + color: #e2e8f0; + cursor: pointer; + font-size: 12px; + line-height: 1; + transition: background 0.15s ease, border-color 0.15s ease, color 0.15s ease; +} + +.markdown-renderer__code-copy:hover { + border-color: rgba(191, 219, 254, 0.55); + background: rgba(51, 65, 85, 0.96); + color: #ffffff; +} + .markdown-renderer pre { overflow: visible; padding: 12px 14px; - border-radius: 10px; + border-radius: 0; background: #0f172a; color: #e2e8f0; } .markdown-renderer__code-scroll { max-width: 100%; - border-radius: 10px; + border-radius: 0; } .markdown-renderer__code-scroll > .scrollbar__viewport, @@ -2150,6 +2238,14 @@ body { margin: 0; } +.markdown-renderer__image { + display: block; + max-width: 100%; + height: auto; + margin: 0.6em 0; + border-radius: 8px; +} + .markdown-renderer hr { border: 0; border-top: 1px solid rgba(15, 23, 42, 0.12); diff --git a/frontend/src/pages/DataSources/DataSources.tsx b/frontend/src/pages/DataSources/DataSources.tsx index 4b1812c1..b4172519 100644 --- a/frontend/src/pages/DataSources/DataSources.tsx +++ b/frontend/src/pages/DataSources/DataSources.tsx @@ -3,24 +3,29 @@ import { useCollapsedActions } from '../../hooks' import { TableActions, actionCellProps } from '../../components/TableActions/TableActions' import { Table, Tag, Space, Button, Form, Input, Select, Progress, Checkbox, message, Modal, - Drawer, Tabs, Empty, Tooltip, Popconfirm, Collapse, InputNumber, Row, Col, Card + Drawer, Tabs, Empty, Tooltip, Popconfirm, Collapse, InputNumber, Row, Col, Card, Alert, Typography } from 'antd' import { PlayCircleOutlined, PauseCircleOutlined, PlusOutlined, EditOutlined, DeleteOutlined, ApiOutlined, CheckCircleOutlined, CloseCircleOutlined, ExperimentOutlined, - SyncOutlined, ClearOutlined, CopyOutlined + SyncOutlined, ClearOutlined, CopyOutlined, InfoCircleOutlined } from '@ant-design/icons' import axios, { type AxiosResponse } from 'axios' import AppLayout from '../../components/AppLayout/AppLayout' import ScrollbarOverlay from '../../components/Scrollbar/ScrollbarOverlay' import { formatDateTimeZhCN } from '../../utils/datetime' import { useWebSocket } from '../../hooks/useWebSocket' +import { Link } from 'react-router-dom' + +const { Text } = Typography +const COLLECTION_REFRESH_DELAY_MS = 800 interface BuiltInDataSource { id: number source: string name: string + display_name?: string module: string priority: string frequency: string @@ -30,14 +35,16 @@ interface BuiltInDataSource { last_run: string | null last_run_at?: string | null last_status?: string | null - last_records_processed?: number | null - data_count?: number is_running: boolean task_id: number | null progress: number | null phase?: string | null records_processed: number | null total_records: number | null + is_free?: boolean + requires_credentials?: boolean + credential_provider?: string | null + credential_status?: string } interface TaskTrackerState { @@ -207,6 +214,26 @@ interface ViewDataSource { module: string priority: string frequency: string + display_name?: string + is_free?: boolean + requires_credentials?: boolean + credential_provider?: string | null + credential_status?: string +} + +interface TargetSchemaField { + name: string + type: string + required: boolean + description: string +} + +interface TargetSchema { + key: string + label: string + description: string + destination: string + fields: TargetSchemaField[] } function DataSources() { @@ -221,6 +248,15 @@ function DataSources() { const [editingConfig, setEditingConfig] = useState(null) const [builtinEditingSource, setBuiltinEditingSource] = useState(null) const [viewingSource, setViewingSource] = useState(null) + const [mappingSource, setMappingSource] = useState(null) + const [mappingDrawerVisible, setMappingDrawerVisible] = useState(false) + const [targetSchemas, setTargetSchemas] = useState([]) + const [selectedTargetSchema, setSelectedTargetSchema] = useState('generic_records') + const [samplePayload, setSamplePayload] = useState(null) + const [sampleText, setSampleText] = useState('') + const [mappingText, setMappingText] = useState('') + const [mappingPreview, setMappingPreview] = useState(null) + const [mappingLoading, setMappingLoading] = useState>({}) const [recordCount, setRecordCount] = useState(0) const [testing, setTesting] = useState(false) const [triggerAllLoading, setTriggerAllLoading] = useState(false) @@ -281,7 +317,7 @@ function DataSources() { const getBuiltinOverrideDescription = useCallback( (source?: Pick | null) => - source ? `Built-in datasource override for ${source.name}` : undefined, + source ? `内置采集器覆盖配置：${source.name}` : undefined, [], ) @@ -292,7 +328,9 @@ function DataSources() { values.description || getBuiltinOverrideDescription(builtinEditingSource), source_type: builtinEditingSource ? 'http' : values.source_type, - headers: headersListToMap(values.headers), + auth_type: builtinEditingSource ? 'none' : values.auth_type, + auth_config: builtinEditingSource ? {} : values.auth_config, + headers: builtinEditingSource ? {} : headersListToMap(values.headers), }), [builtinEditingSource, getBuiltinOverrideDescription, headersListToMap]) const closeDrawerAfterLoadError = useCallback(( @@ -612,13 +650,11 @@ function DataSources() { status: 'running', }, })) + fetchData() } else { - window.setTimeout(() => { - fetchData() - }, 800) + window.setTimeout(fetchData, COLLECTION_REFRESH_DELAY_MS) } - fetchData() return { ok: true, response: res, @@ -811,6 +847,7 @@ function DataSources() { setViewingSource({ id: data.id, name: data.name, + display_name: data.display_name, description: null, source_type: data.collector_class, endpoint: overrideDetail?.endpoint || data.endpoint || '', @@ -821,6 +858,10 @@ function DataSources() { module: data.module, priority: data.priority, frequency: data.frequency, + is_free: data.is_free, + requires_credentials: data.requires_credentials, + credential_provider: data.credential_provider, + credential_status: data.credential_status, }) setRecordCount(statsRes.data.total_records || 0) setViewDrawerVisible(true) @@ -923,6 +964,150 @@ function DataSources() { } } + const setMappingStepLoading = (key: string, value: boolean) => { + setMappingLoading((prev) => ({ ...prev, [key]: value })) + } + + const parseJsonText = (value: string, label: string) => { + try { + return JSON.parse(value) + } catch { + throw new Error(`${label} 不是合法 JSON`) + } + } + + const openMappingDrawer = async (source: CustomDataSource) => { + setMappingSource(source) + setMappingDrawerVisible(true) + setSamplePayload(null) + setSampleText('') + setMappingText('') + setMappingPreview(null) + try { + const [schemasRes, mappingsRes] = await Promise.all([ + axios.get('/api/v1/datasources/target-schemas'), + axios.get('/api/v1/datasources/mappings', { + params: { datasource_config_id: source.id, active_only: true }, + }), + ]) + const schemas = schemasRes.data.data || [] + setTargetSchemas(schemas) + const activeMapping = mappingsRes.data.data?.[0] + const nextSchema = activeMapping?.target_schema || schemas[0]?.key || 'generic_records' + setSelectedTargetSchema(nextSchema) + if (activeMapping?.mapping_json) { + setMappingText(JSON.stringify(activeMapping.mapping_json, null, 2)) + } + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || '加载映射配置失败') + } + } + + const handleFetchSample = async () => { + if (!mappingSource) return + setMappingStepLoading('sample', true) + try { + const res = await axios.post('/api/v1/datasources/custom/sample', { + datasource_config_id: mappingSource.id, + }) + setSamplePayload(res.data.sample_payload) + setSampleText(JSON.stringify(res.data.sample_payload, null, 2)) + setMappingPreview(null) + messageApi.success('样本已抓取') + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || '抓取样本失败') + } finally { + setMappingStepLoading('sample', false) + } + } + + const handleProposeMapping = async () => { + const payload = samplePayload || parseJsonText(sampleText, '样本') + setMappingStepLoading('propose', true) + try { + const res = await axios.post('/api/v1/datasources/mappings/propose', { + sample_payload: payload, + target_schema: selectedTargetSchema, + use_ai: true, + }) + setMappingText(JSON.stringify(res.data.mapping_json, null, 2)) + setMappingPreview(null) + messageApi.success('映射草案已生成') + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || (error instanceof Error ? error.message : '生成映射失败')) + } finally { + setMappingStepLoading('propose', false) + } + } + + const handlePreviewMapping = async () => { + setMappingStepLoading('preview', true) + try { + const payload = samplePayload || parseJsonText(sampleText, '样本') + const mappingJson = parseJsonText(mappingText, '映射配置') + const res = await axios.post('/api/v1/datasources/mappings/preview', { + sample_payload: payload, + target_schema: selectedTargetSchema, + mapping_json: mappingJson, + limit: 20, + }) + setMappingPreview(res.data.preview) + messageApi[res.data.success ? 'success' : 'warning']( + res.data.success ? '预览校验通过' : '预览完成，但存在校验错误', + ) + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || (error instanceof Error ? error.message : '预览失败')) + } finally { + setMappingStepLoading('preview', false) + } + } + + const handleSaveMapping = async () => { + if (!mappingSource) return + setMappingStepLoading('save', true) + try { + const payload = samplePayload || parseJsonText(sampleText, '样本') + const mappingJson = parseJsonText(mappingText, '映射配置') + await axios.post('/api/v1/datasources/mappings', { + datasource_config_id: mappingSource.id, + target_schema: selectedTargetSchema, + mapping_json: mappingJson, + sample_payload: payload, + validation_status: mappingPreview?.failed_count === 0 ? 'valid' : 'draft', + is_active: true, + }) + messageApi.success('映射已保存并启用') + setMappingDrawerVisible(false) + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || (error instanceof Error ? error.message : '保存映射失败')) + } finally { + setMappingStepLoading('save', false) + } + } + + const handleRunMapped = async () => { + if (!mappingSource) return + setMappingStepLoading('run', true) + try { + const res = await axios.post(`/api/v1/datasources/${mappingSource.id}/run-mapped`) + if (res.data.status === 'success') { + messageApi.success(`已写入 ${res.data.written_count || 0} 条`) + } else { + messageApi.error(`采集失败：${res.data.failed_count || 0} 条未通过映射`) + } + } catch (error: unknown) { + const err = error as { response?: { data?: { detail?: string } } } + messageApi.error(err.response?.data?.detail || '运行映射采集失败') + } finally { + setMappingStepLoading('run', false) + } + } + const openDrawer = async (config?: CustomDataSource) => { setBuiltinEditingSource(null) setEditingConfig(config || null) @@ -997,14 +1182,17 @@ function DataSources() { { title: 'ID', dataIndex: 'id', key: 'id', width: 60, fixed: 'left' as const }, { title: '名称', - dataIndex: 'name', + dataIndex: 'display_name', key: 'name', - width: 180, + width: 220, ellipsis: true, render: (name: string, record: BuiltInDataSource) => ( - + + + {record.source} + ), }, { title: '模块', dataIndex: 'module', key: 'module', width: 80 }, @@ -1022,12 +1210,7 @@ function DataSources() { key: 'last_run', width: 180, render: (_: string | null, record: BuiltInDataSource) => { - const label = formatDateTimeZhCN(record.last_run_at || record.last_run) - if (!label || label === '-') return '-' - if ((record.data_count || 0) === 0 && record.last_status === 'success') { - return `${label} (0条)` - } - return label + return formatDateTimeZhCN(record.last_run_at || record.last_run) || '-' }, }, { @@ -1192,6 +1375,12 @@ function DataSources() { icon: , onClick: () => { void openDrawer(record) }, }, + { + key: 'mapping', + label: '映射', + icon: , + onClick: () => { void openMappingDrawer(record) }, + }, { key: 'toggle', label: record.is_active ? '禁用' : '启用', @@ -1215,6 +1404,7 @@ function DataSources() { ]} > +
    + + + setForceTriggerAll(event.target.checked)} @@ -1307,7 +1500,7 @@ function DataSources() { rowKey="id" loading={loading} pagination={false} - scroll={{ x: 800, y: builtinTableHeight }} + scroll={{ x: 1200, y: builtinTableHeight }} tableLayout="fixed" size="small" /> @@ -1320,19 +1513,22 @@ function DataSources() { key: 'custom', label: ( - 自定义数据源 + 自定义 API 源 ), children: (
    + + +
    {customSources.length === 0 ? (
    - +
    ) : (
    @@ -1360,7 +1556,7 @@ function DataSources() { {modalContextHolder}
    -

    数据源管理

    +

    数据源与 API 连接器

    @@ -1370,7 +1566,7 @@ function DataSources() {
    { @@ -1386,7 +1582,7 @@ function DataSources() { {builtinEditingSource && editingConfig ? ( ) : null} - + {!builtinEditingSource ? ( + + ) : null} @@ -1416,13 +1614,18 @@ function DataSources() {
    {builtinEditingSource ? ( +
    + + + +
    -
    内置数据源
    +
    内置采集器
    -
    Collector Key
    +
    采集器标识
     @@ -1438,7 +1641,7 @@ function DataSources() { )} - + {builtinEditingSource ? null : ( @@ -1448,7 +1651,7 @@ function DataSources() { rules={[{ required: true, message: '请选择类型' }]} > @@ -1463,113 +1666,117 @@ function DataSources() { - - - - -
    - auth_type === 'bearer'}> - {({ getFieldValue }) => { - if (getFieldValue('auth_type') === 'bearer') { - return ( - - - - ) - } - return null - }} + {!builtinEditingSource ? ( + + + - auth_type === 'api_key'}> - {({ getFieldValue }) => { - if (getFieldValue('auth_type') === 'api_key') { - return ( - <> - - +
    + auth_type === 'bearer'}> + {({ getFieldValue }) => { + if (getFieldValue('auth_type') === 'bearer') { + return ( + + - - - - - - - - ) - } - return null - }} - - auth_type === 'basic'}> - {({ getFieldValue }) => { - if (getFieldValue('auth_type') === 'basic') { - return ( - <> - - - - - - - - ) - } - return null - }} - -
    - - ), - }, - ]} - /> + ) + } + return null + }} +     + auth_type === 'api_key'}> + {({ getFieldValue }) => { + if (getFieldValue('auth_type') === 'api_key') { + return ( + <> + + + + + + + + + + + ) + } + return null + }} + + auth_type === 'basic'}> + {({ getFieldValue }) => { + if (getFieldValue('auth_type') === 'basic') { + return ( + <> + + + + + + + + ) + } + return null + }} + +
    + + ), + }, + ]} + /> + ) : null} - - {(fields, { add, remove }) => ( - <> - {fields.map(({ key, name, ...restField }) => ( - - - - - - - - - - ))} - - - )} - - ), - }, - ]} - /> + {!builtinEditingSource ? ( + + {(fields, { add, remove }) => ( + <> + {fields.map(({ key, name, ...restField }) => ( + + + + + + + + + + ))} + + + )} + + ), + }, + ]} + /> + ) : null} + { + setMappingDrawerVisible(false) + setMappingSource(null) + setMappingPreview(null) + }} + footer={ +
    + + + + + +
    + } + > + + + + + + + { + setSampleText(event.target.value) + setSamplePayload(null) + setMappingPreview(null) + }} + rows={8} + placeholder='{"data":[...]}' + /> + + + + + + {viewingSource.requires_credentials ? ( + +
    + + 需要采集器凭证 + + {viewingSource.credential_status === 'supported' + ? '请在设置中心维护该采集器的外部服务凭证。' + : '该采集器需要凭证，配置入口待接入。'} + + + {viewingSource.credential_status === 'supported' ? ( + + + + ) : null} +
    + + ) : null}     @@ -1709,29 +2073,9 @@ function DataSources() { - - - ), - }, - { - key: 'headers', - label: '请求头', - children: viewingSource.headers && Object.keys(viewingSource.headers).length > 0 ? ( - 
    -                        {JSON.stringify(viewingSource.headers, null, 2)}
-
    - ) : ( -
    无
    - ), - }, { key: 'config', - label: '高级配置', + label: '运行参数', children: viewingSource.config && Object.keys(viewingSource.config).length > 0 ? ( 
                             {JSON.stringify(viewingSource.config, null, 2)}
diff --git a/frontend/src/pages/Docs/Docs.css b/frontend/src/pages/Docs/Docs.css
index bf9f995a..d694ab8c 100644
--- a/frontend/src/pages/Docs/Docs.css
+++ b/frontend/src/pages/Docs/Docs.css
@@ -534,9 +534,19 @@
   color: var(--d-heading);
 }
 
+.docs-markdown.markdown-renderer h4,
+.docs-markdown.markdown-renderer h5,
+.docs-markdown.markdown-renderer h6 {
+  margin-top: 24px;
+  color: var(--d-heading);
+}
+
 .docs-markdown.markdown-renderer h1,
 .docs-markdown.markdown-renderer h2,
-.docs-markdown.markdown-renderer h3 {
+.docs-markdown.markdown-renderer h3,
+.docs-markdown.markdown-renderer h4,
+.docs-markdown.markdown-renderer h5,
+.docs-markdown.markdown-renderer h6 {
   scroll-margin-top: 24px;
 }
 
@@ -553,16 +563,39 @@
   font-size: 0.88em;
 }
 
-.docs-markdown.markdown-renderer pre {
+.docs-markdown .markdown-renderer__code-block {
   border: 1px solid var(--d-code-border);
   border-radius: 8px;
   background: var(--d-code-bg);
-  overflow: visible;
+  box-shadow: none;
+}
+
+.docs-markdown.markdown-renderer pre {
+  background: var(--d-code-bg);
+  color: var(--d-code-text);
+}
+
+.docs-markdown .markdown-renderer__code-toolbar {
+  border-bottom: 1px solid var(--d-code-border);
+  background: var(--d-state-bg);
+  color: var(--d-toc-text);
+}
+
+.docs-markdown .markdown-renderer__code-copy {
+  border-color: var(--d-code-border);
+  background: var(--d-bg);
+  color: var(--d-text);
+}
+
+.docs-markdown .markdown-renderer__code-copy:hover {
+  border-color: var(--d-link);
+  background: var(--d-code-bg);
+  color: var(--d-heading);
 }
 
 .docs-markdown .markdown-renderer__code-scroll {
   max-width: 100%;
-  margin: 0 0 0.9em;
+  margin: 0;
   border-radius: 8px;
 }
 
diff --git a/frontend/src/pages/Docs/docs-content.ts b/frontend/src/pages/Docs/docs-content.ts
index 91f9557f..a0b296f1 100644
--- a/frontend/src/pages/Docs/docs-content.ts
+++ b/frontend/src/pages/Docs/docs-content.ts
@@ -45,7 +45,6 @@ const DOCS_GROUP_LABELS: Record> = {
 }
 
 const DOCS_README_FILENAME = 'README.md'
-const FALLBACK_DOCS_ORDER = 999
 const MAX_HEADING_ID_LENGTH = 80
 export const defaultDocsSlug = 'overview'
 
@@ -138,27 +137,20 @@ export function slugFromFilename(filename: string): string {
   return filename === DOCS_README_FILENAME ? defaultDocsSlug : filename.replace(/\.md$/, '')
 }
 
-function fallbackTitleFromFilename(filename: string): string {
-  return filename
-    .replace(/\.md$/, '')
-    .split('-')
-    .map((part) => part.charAt(0).toUpperCase() + part.slice(1))
-    .join(' ')
-}
-
 export function getDocsEntries(lang: DocsLang): DocsEntry[] {
   const modules = lang === 'zh' ? zhModules : enModules
   return Object.entries(modules)
+    .filter(([path]) => DOCS_METADATA[filenameFromPath(path)])
     .map(([path, loader]) => {
       const filename = filenameFromPath(path)
       const meta = DOCS_METADATA[filename]
-      const langMeta = meta?.[lang]
+      const langMeta = meta[lang]
       return {
         slug: slugFromFilename(filename),
         filename,
-        title: langMeta?.title || fallbackTitleFromFilename(filename),
-        group: (langMeta?.group || 'Other') as DocsGroup,
-        order: langMeta?.order ?? FALLBACK_DOCS_ORDER,
+        title: langMeta.title,
+        group: langMeta.group,
+        order: langMeta.order,
         loader: loader as () => Promise,
       }
     })
diff --git a/frontend/src/pages/Playground/Playground.tsx b/frontend/src/pages/Playground/Playground.tsx
index e659e6ad..2fc79f99 100644
--- a/frontend/src/pages/Playground/Playground.tsx
+++ b/frontend/src/pages/Playground/Playground.tsx
@@ -25,6 +25,7 @@ import MarkdownRenderer from '../../components/MarkdownRenderer/MarkdownRenderer
 import Scrollbar from '../../components/Scrollbar/Scrollbar'
 import ScrollbarOverlay from '../../components/Scrollbar/ScrollbarOverlay'
 import { useAuthStore } from '../../stores/auth'
+import { Link } from 'react-router-dom'
 
 const { Title, Text, Paragraph } = Typography
 const API_BASE_URL = (import.meta as any).env?.VITE_API_URL || '/api/v1'
@@ -581,7 +582,7 @@ function Playground() {
     >
       
         
-          {providerStatus ? (
+              {providerStatus ? (
             
    
               
    
                 Provider
@@ -616,9 +617,22 @@ function Playground() {
                 最后同步
                 {providerStatusUpdatedAt || '-'}
               
    
+              {!providerStatus.configured ? (
+                前往 AI 配置}
+                />
+              ) : null}
             
    
           ) : (
-            
+            前往 AI 配置}
+            />
           )}
             
           
diff --git a/frontend/src/pages/Settings/Settings.tsx b/frontend/src/pages/Settings/Settings.tsx
index b27b907a..99863563 100644
--- a/frontend/src/pages/Settings/Settings.tsx
+++ b/frontend/src/pages/Settings/Settings.tsx
@@ -1,16 +1,18 @@
 import { useEffect, useRef, useState, type ReactNode } from 'react'
 import { useCollapsedActions } from '../../hooks'
 import { TableActions, actionCellProps } from '../../components/TableActions/TableActions'
-import { CheckCircleOutlined, DeleteOutlined, EditOutlined, PlusOutlined } from '@ant-design/icons'
+import { ApiOutlined, CheckCircleOutlined, DeleteOutlined, EditOutlined, PlusOutlined, SyncOutlined } from '@ant-design/icons'
 import {
   Button,
   Card,
+  Checkbox,
   Form,
   Input,
   InputNumber,
   message,
   Modal,
   Select,
+  Space,
   Switch,
   Table,
   Tabs,
@@ -23,8 +25,11 @@ import AppLayout from '../../components/AppLayout/AppLayout'
 import Scrollbar from '../../components/Scrollbar/Scrollbar'
 import TableScrollRegion from '../../components/Scrollbar/TableScrollRegion'
 import { formatDateTimeZhCN } from '../../utils/datetime'
+import { useSearchParams } from 'react-router-dom'
 
 const { Title, Text } = Typography
+const ANTHROPIC_MESSAGES_MAX_TOKENS = 1200
+const DEFAULT_PROVIDER_MAX_TOKENS = 4096
 
 interface SystemSettings {
   system_name: string
@@ -51,6 +56,7 @@ interface SecuritySettings {
 interface CollectorSettings {
   id: number
   name: string
+  display_name?: string
   source: string
   module: string
   priority: string
@@ -60,6 +66,10 @@ interface CollectorSettings {
   last_run_at: string | null
   last_status: string | null
   next_run_at: string | null
+  is_free?: boolean
+  requires_credentials?: boolean
+  credential_provider?: string | null
+  credential_status?: string
 }
 
 interface TVStreamSource {
@@ -88,6 +98,46 @@ interface TVSettings {
   sources: TVStreamSource[]
 }
 
+interface SecretStatus {
+  configured: boolean
+  preview: string
+}
+
+interface ExternalIntegrations {
+  ai_provider: {
+    service_url: string
+    service_token: SecretStatus
+    provider: string
+    provider_api: string
+    base_url: string
+    model: string
+    api_key: SecretStatus
+    max_tokens: number
+    anthropic_version: string
+    timeout_seconds: number
+    retry_attempts: number
+    source: string
+  }
+  barentswatch: {
+    endpoint: string
+    client_id: string
+    client_secret: SecretStatus
+    source: string
+  }
+}
+
+interface AIProviderPreset {
+  provider: string
+  label: string
+  provider_api: string
+  base_url: string
+  model: string
+  models: string[]
+  api_key_env: string
+  source: string
+  refresh_error?: string
+}
+
 function SettingsPanel({
   loading,
   children,
@@ -105,6 +155,8 @@ function SettingsPanel({
 }
 
 function Settings() {
+  const [searchParams, setSearchParams] = useSearchParams()
+  const requestedTab = searchParams.get('tab') || 'display'
   const [loading, setLoading] = useState(true)
   const [savingCollectorId, setSavingCollectorId] = useState(null)
   const [collectors, setCollectors] = useState([])
@@ -112,7 +164,11 @@ function Settings() {
   const [notificationSettings, setNotificationSettings] = useState(null)
   const [securitySettings, setSecuritySettings] = useState(null)
   const [tvSettings, setTvSettings] = useState(null)
+  const [integrations, setIntegrations] = useState(null)
+  const [aiProviderPresets, setAiProviderPresets] = useState([])
+  const [refreshingAiPreset, setRefreshingAiPreset] = useState(false)
   const [savingTvSettings, setSavingTvSettings] = useState(false)
+  const [savingIntegrations, setSavingIntegrations] = useState(false)
   const [editingSource, setEditingSource] = useState(null)
   const [tvActionsCollapsed, tvTableRef] = useCollapsedActions(780)
   const collectorTableRegionRef = useRef(null)
@@ -120,17 +176,49 @@ function Settings() {
   const [systemForm] = Form.useForm()
   const [notificationForm] = Form.useForm()
   const [securityForm] = Form.useForm()
+  const [integrationForm] = Form.useForm()
   const [tvEditForm] = Form.useForm()
+  const selectedAiProvider = Form.useWatch(['ai_provider', 'provider'], integrationForm)
+  const credentialCollectors = collectors.filter((collector) => collector.requires_credentials)
+  const settingsTabKeys = new Set([
+    'display',
+    'notifications',
+    'security',
+    'tv',
+    'ai',
+    'collector_credentials',
+    'collectors',
+  ])
+  const activeSettingsTab = requestedTab === 'system'
+    ? 'display'
+    : settingsTabKeys.has(requestedTab)
+      ? requestedTab
+      : 'display'
+
+  const updateSettingsTab = (tabKey: string) => {
+    const nextParams = new URLSearchParams(searchParams)
+    if (tabKey === 'display') {
+      nextParams.delete('tab')
+    } else {
+      nextParams.set('tab', tabKey)
+    }
+    setSearchParams(nextParams, { replace: true })
+  }
 
   const fetchSettings = async () => {
     try {
       setLoading(true)
-      const response = await axios.get('/api/v1/settings')
+      const [response, presetsResponse] = await Promise.all([
+        axios.get('/api/v1/settings'),
+        axios.get('/api/v1/settings/integrations/ai-provider/presets'),
+      ])
       setSystemSettings(response.data.system)
       setNotificationSettings(response.data.notifications)
       setSecuritySettings(response.data.security)
       setTvSettings(response.data.tv || null)
+      setIntegrations(response.data.integrations || null)
       setCollectors(response.data.collectors || [])
+      setAiProviderPresets(presetsResponse.data.data || [])
     } catch (error) {
       message.error('获取系统配置失败')
       console.error(error)
@@ -161,6 +249,33 @@ function Settings() {
     }
   }, [loading, securityForm, securitySettings])
 
+  useEffect(() => {
+    if (loading || !integrations) return
+    integrationForm.setFieldsValue({
+      ai_provider: {
+        service_url: integrations.ai_provider.service_url,
+        service_token: '',
+        provider: integrations.ai_provider.provider,
+        provider_api: integrations.ai_provider.provider_api,
+        base_url: integrations.ai_provider.base_url,
+        model: integrations.ai_provider.model,
+        api_key: '',
+        max_tokens: integrations.ai_provider.max_tokens,
+        anthropic_version: integrations.ai_provider.anthropic_version,
+        timeout_seconds: integrations.ai_provider.timeout_seconds,
+        retry_attempts: integrations.ai_provider.retry_attempts,
+        clear_service_token: false,
+        clear_api_key: false,
+      },
+      barentswatch: {
+        endpoint: integrations.barentswatch.endpoint,
+        client_id: integrations.barentswatch.client_id,
+        client_secret: '',
+        clear_client_secret: false,
+      },
+    })
+  }, [integrationForm, integrations, loading])
+
   useEffect(() => {
     const updateTableHeight = () => {
       const regionHeight = collectorTableRegionRef.current?.offsetHeight || 0
@@ -214,6 +329,59 @@ function Settings() {
     }
   }
 
+  const saveIntegrations = async (values: any) => {
+    try {
+      setSavingIntegrations(true)
+      const response = await axios.put('/api/v1/settings/integrations', values)
+      setIntegrations(response.data.integrations)
+      message.success('外部集成配置已保存')
+      await fetchSettings()
+    } catch {
+      message.error('外部集成配置保存失败')
+    } finally {
+      setSavingIntegrations(false)
+    }
+  }
+
+  const applyAiProviderPreset = (preset: AIProviderPreset) => {
+    integrationForm.setFieldsValue({
+      ai_provider: {
+        provider: preset.provider,
+        provider_api: preset.provider_api,
+        base_url: preset.base_url,
+        model: preset.model,
+        max_tokens: preset.provider_api === 'anthropic-messages'
+          ? ANTHROPIC_MESSAGES_MAX_TOKENS
+          : DEFAULT_PROVIDER_MAX_TOKENS,
+        anthropic_version: '2023-06-01',
+      },
+    })
+  }
+
+  const refreshSelectedAiProviderPreset = async () => {
+    const provider = integrationForm.getFieldValue(['ai_provider', 'provider'])
+    if (!provider) return
+    try {
+      setRefreshingAiPreset(true)
+      const response = await axios.post(`/api/v1/settings/integrations/ai-provider/presets/${provider}/refresh`)
+      const preset = response.data.data as AIProviderPreset
+      setAiProviderPresets((prev) => {
+        const next = prev.filter((item) => item.provider !== preset.provider)
+        return [...next, preset].sort((a, b) => a.label.localeCompare(b.label))
+      })
+      applyAiProviderPreset(preset)
+      if (preset.refresh_error) {
+        message.warning('刷新失败，已使用本地 fallback 配置')
+      } else {
+        message.success('已刷新选中 Provider 的最新模型配置')
+      }
+    } catch {
+      message.error('刷新 Provider 配置失败')
+    } finally {
+      setRefreshingAiPreset(false)
+    }
+  }
+
   const setDefaultSource = (sourceId: string) => {
     if (!tvSettings) return
     const next = { ...tvSettings, default_source_id: sourceId }
@@ -537,7 +705,7 @@ function Settings() {
 
   const tabItems = [
     {
-      key: 'system',
+      key: 'display',
       label: '系统显示',
       forceRender: true,
       children: (
@@ -729,6 +897,196 @@ function Settings() {
    ), }, + { + key: 'ai', + label: 'AI', + forceRender: true, + children: ( + + + LLM Provider}> + + + + + + +
    + + + + + + + + + + + + + + +
    + + + + + + + + + {integrations?.ai_provider.service_token.configured + ? `已配置 ${integrations.ai_provider.service_token.preview}` + : '未配置'} + + 通常不需要改；用于 backend 调本地 aiprovider。 + + + + + + + + 清除当前代理 token + + + + + + + + ), + }, + { + key: 'collector_credentials', + label: '采集器凭证', + forceRender: true, + children: ( + +
    + + + {credentialCollectors.length ? credentialCollectors.map((collector) => ( + + {collector.display_name || collector.name} + {collector.credential_status === 'supported' ? ' · 已支持配置' : ' · 待接入'} + + )) : ( + 当前没有需要凭证的内置采集器。 + )} + + + BarentsWatch AIS} + > + + + + + + + + + + + {integrations?.barentswatch.client_secret.configured + ? `已配置 ${integrations.barentswatch.client_secret.preview}` + : '未配置'} + + 留空表示保留现有 secret。 + + + + + + + + 清除当前 BarentsWatch client secret + + + + +
    +     + ), + }, { key: 'collectors', label: '采集调度', @@ -767,7 +1125,12 @@ function Settings() {
    - +
    diff --git a/planet.sh b/planet.sh index 2e9f5d10..f3852bda 100755 --- a/planet.sh +++ b/planet.sh @@ -55,8 +55,8 @@ DATABASE_RETRY_INTERVAL="${DATABASE_RETRY_INTERVAL:-5}" FRONTEND_MAX_RETRIES="${FRONTEND_MAX_RETRIES:-3}" FRONTEND_HEALTH_CHECK_ATTEMPTS="${FRONTEND_HEALTH_CHECK_ATTEMPTS:-10}" FRONTEND_HEALTH_CHECK_INTERVAL="${FRONTEND_HEALTH_CHECK_INTERVAL:-2}" -PORT_RELEASE_ATTEMPTS="${PORT_RELEASE_ATTEMPTS:-45}" -PORT_RELEASE_INTERVAL="${PORT_RELEASE_INTERVAL:-1}" +PORT_RELEASE_ATTEMPTS="${PORT_RELEASE_ATTEMPTS:-15}" +PORT_RELEASE_INTERVAL="${PORT_RELEASE_INTERVAL:-0.2}" DEFAULT_BACKEND_PORT="${DEFAULT_BACKEND_PORT:-8000}" DEFAULT_FRONTEND_PORT="${DEFAULT_FRONTEND_PORT:-3000}" DEFAULT_AI_PROVIDER_PORT="${DEFAULT_AI_PROVIDER_PORT:-8010}" @@ -64,7 +64,7 @@ FRONTEND_RUNTIME_BIN="${FRONTEND_RUNTIME_BIN:-}" FRONTEND_RUNTIME_SOURCE="${FRONTEND_RUNTIME_SOURCE:-}" FRONTEND_PID_FILE="/tmp/planet_frontend.pid" FRONTEND_VITE_ENTRY="$SCRIPT_DIR/frontend/node_modules/vite/bin/vite.js" -AI_PROVIDER_BUILD_STAMP_FILE="/tmp/planet_aiprovider_build.sha256" +AI_PROVIDER_BUILD_STAMP_FILE="$HOME/.cache/planet/aiprovider_build.sha256" AI_PROVIDER_BUILD_LOG_FILE="/tmp/planet_aiprovider_build.log" AI_PROVIDER_IMAGE_NAME="${AI_PROVIDER_IMAGE_NAME:-planet_aiprovider:latest}" AI_PROVIDER_CONTAINER_NAME="${AI_PROVIDER_CONTAINER_NAME:-planet_aiprovider}" @@ -587,11 +587,15 @@ compute_ai_provider_build_fingerprint() { ( cd "$SCRIPT_DIR" || exit 1 { - tar -cf - \ - aiprovider \ - docker-compose.yml \ - docker-compose.simple.yml 2>/dev/null - python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" + find aiprovider \ + -type f \ + ! -path '*/__pycache__/*' \ + ! -name '*.pyc' \ + ! -name '*.pyo' \ + | LC_ALL=C sort \ + | xargs -r stat --format="%Y %s %n" 2>/dev/null + sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null + python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null } ) | sha256sum | awk '{print $1}' } @@ -603,6 +607,7 @@ read_ai_provider_build_stamp() { write_ai_provider_build_stamp() { local fingerprint="$1" + mkdir -p "$(dirname "$AI_PROVIDER_BUILD_STAMP_FILE")" printf "%s\n" "$fingerprint" > "$AI_PROVIDER_BUILD_STAMP_FILE" } @@ -1108,10 +1113,9 @@ cleanup_backend_processes() { local backend_port="${1:-$DEFAULT_BACKEND_PORT}" terminate_backend_processes TERM "$backend_port" - if ! wait_for_port_release "$backend_port"; then + if ! wait_for_port_release "$backend_port" 15 0.2; then terminate_backend_processes KILL "$backend_port" - - wait_for_port_release "$backend_port" || true + wait_for_port_release "$backend_port" 15 0.2 || true fi } @@ -1246,7 +1250,6 @@ restart_database_service() { while [ "$retry" -le "$DATABASE_START_MAX_RETRIES" ]; do if restart_database_services && wait_for_database_health; then - sleep 3 return 0 fi @@ -1269,7 +1272,6 @@ start_backend_service() { set_wait_detail "启动数据库" ensure_database_services_healthy log_success "启动数据库已就绪" - sleep 3 # Backend depends on AI Provider reachability, but a backend-only restart # should reuse the existing healthy provider instead of rebuilding or @@ -1391,6 +1393,14 @@ terminate_process_tree() { can_bind_port() { local port="$1" + if command -v ss >/dev/null 2>&1; then + ! ss -tlnH 2>/dev/null | awk '{print $4}' | grep -qE ":${port}$" + return + fi + if command -v lsof >/dev/null 2>&1; then + [ -z "$(lsof -tiTCP:"${port}" -sTCP:LISTEN 2>/dev/null)" ] + return + fi python3 - "$port" <<'PY' >/dev/null 2>&1 import socket import sys @@ -1416,13 +1426,15 @@ PY wait_for_port_release() { local port="$1" + local max_attempts="${2:-$PORT_RELEASE_ATTEMPTS}" + local interval="${3:-$PORT_RELEASE_INTERVAL}" local attempt=1 - while [ "$attempt" -le "$PORT_RELEASE_ATTEMPTS" ]; do + while [ "$attempt" -le "$max_attempts" ]; do if can_bind_port "$port"; then return 0 fi - sleep "$PORT_RELEASE_INTERVAL" + sleep "$interval" attempt=$((attempt + 1)) done @@ -1449,7 +1461,7 @@ kill_port_if_requested() { terminate_process_tree TERM "$pid" done - if wait_for_port_release "$port"; then + if wait_for_port_release "$port" 15 0.2; then log_success "端口 ${port} 已释放" return 0 fi @@ -1460,7 +1472,7 @@ kill_port_if_requested() { terminate_process_tree KILL "$pid" done - if wait_for_port_release "$port"; then + if wait_for_port_release "$port" 15 0.2; then log_success "端口 ${port} 已释放" return 0 fi diff --git a/pyproject.toml b/pyproject.toml index 47f0e6ad..6c1fa06e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "planet" -version = "0.42.2" +version = "0.43.0" description = "智能星球计划 - 态势感知系统" requires-python = ">=3.14" dependencies = [ diff --git a/rules.md b/rules.md index abaa3285..2b10ae72 100644 --- a/rules.md +++ b/rules.md @@ -1,381 +1,503 @@ -# rules.md +--- +name: planet-rules +description: Planet repository rules split into LLM-loadable modules. +--- -**必须强制执行的约束。违反时立即终止并报错。** +# Planet Rules + +**These rules are mandatory. If a requested action conflicts with this file, stop and report the conflict.** + +## Loading Protocol + +Read this top section first, then load only the modules relevant to the task. + +Always load: + +- `core` +- `security` +- `workflow` + +Load selectively: + +| Module | Load when | +|--------|-----------| +| `docs` | Writing, translating, linking, or publishing documentation | +| `uiux` | Visual design, layout, interaction, accessibility, responsive behavior | +| `frontend` | React, TypeScript, CSS, Vite, Bun, admin console, docs UI | +| `backend` | FastAPI, SQLAlchemy, data collectors, database, API performance | +| `earth` | 3D Earth, canvas/Three.js, BGP/vessel/satellite/cable layers, map icons | +| `ai` | AI Provider, LLM gateway, prompts, model config, AI Playground | +| `release` | Version bumps, changelog, version history, commit/tag/push release work | + +Do not load the entire file by default for small tasks. Use `rg -n "^## Module:" rules.md` to find module boundaries, then read only the needed block. --- -## Code Style - Imports +## Module: core + +### Load When + +Always. + +### Must + +- Keep functions small and focused; one concern per file/module. +- Write self-documenting code; comments explain why, not what. +- Prefer dependency injection for testability. +- Use feature flags for incomplete features. +- Use config files or environment variables for environment-specific settings. +- Maintain one source of truth for business state. Temporary UI state, cached state, and persisted backend state must not become parallel truths. +- Transitional paths are temporary. Once a new implementation is stable, remove old branches, old interfaces, old mocks, and compatibility layers. +- Extract repeated request flow, response handling, auth/header assembly, validation, and state reconciliation into helpers or shared layers. +- Centralize default values, system prompts, placeholder structures, and fixed constants. +- Public interfaces, persisted fields, and state structures must have a current owner and caller. Delete unused ones. +- After large feature work, run an explicit cleanup pass for dead code, duplicated helpers, stale interfaces, and naming drift. + +### Code Style + +- Python: 4-space indentation, Black style, max line length 100. +- TypeScript: 2-space indentation, Prettier style, max line length 100. +- No trailing whitespace. +- Empty line at end of file. +- Sort imports alphabetically inside groups. +- Never use wildcard imports. +- Avoid unclear abbreviations except common ones such as `id`, `ok`, `err`. +- Prefer descriptive names. +- Keep functions around 50 lines or less where practical. +- Split files before they become mixed-responsibility modules. + +### Import Order + +Python: -### Python ```python -# Group order: stdlib → third-party → local +# stdlib -> third-party -> local import json from datetime import datetime -from typing import List, Optional -import redis -from fastapi import APIRouter, Depends +from fastapi import APIRouter from sqlalchemy.orm import Session from app.core.config import settings -from app.models.user import User -from app.schemas.user import UserCreate ``` -### TypeScript +TypeScript: + ```typescript -// Group order: React → Third-party → Local -import React, { useState, useEffect } from 'react'; -import { useQuery } from '@tanstack/react-query'; -import axios from 'axios'; +// React -> third-party -> local +import { useEffect, useState } from 'react' -import { useAuthStore } from '@/stores/auth'; -import { api } from '@/services/api'; +import axios from 'axios' + +import { api } from '@/services/api' ``` -**Rules:** -- Sort alphabetically within groups -- Use absolute imports for external packages, relative for local modules -- **NEVER** use wildcard imports (`from module import *`) +### Type Rules ---- +- Use type hints throughout Python. +- Define TypeScript interfaces/types for all structured data. +- Avoid `Any`; use specific unions, generics, or `unknown` where appropriate. +- Prefer typed helpers over repeated type casting. -## Code Style - Formatting +### Verify -- **Python:** 4-space indentation, Black formatter, max line 100 -- **TypeScript:** 2-space indentation, Prettier, max line 100 -- Run formatter **before committing** -- No trailing whitespace -- Empty line at end of file - ---- - -## Code Style - Type Hints - -```python -# Use strict typing - NO Any -from typing import List, Dict, Optional, Union -from datetime import datetime - -def get_gpu_clusters( - country: Optional[str] = None, - min_gpu_count: int = 0, -) -> List[Dict[str, Union[str, int, float]]]: - ... -``` - -**Rules:** -- Use type hints throughout -- **NEVER** use `Any` - use `unknown` or specific unions -- Define interfaces/types for all data structures -- Generic types preferred over type casting - ---- - -## Code Style - Naming Conventions - -| Pattern | Usage | Example | -|---------|-------|---------| -| `camelCase` | Variables, functions, methods | `gpuCluster`, `getData()` | -| `PascalCase` | Classes, components, types | `GPUCluster`, `DataSourceConfig` | -| `SCREAMING_SNAKE_CASE` | Constants, env vars | `API_KEY`, `DATABASE_URL` | -| `kebab-case` | File names, CSS | `data-source-config.css` | - -**Rules:** -- Descriptive names - avoid abbreviations except well-known ones (id, ok, err) -- Max function length: 50 lines -- Max file length: 500 lines - ---- - -## Code Style - Error Handling - -```python -# Use custom exceptions -class DataSourceError(Exception): - """Raised when data source fetch fails""" - pass - -# Proper error handling with logging -try: - data = await fetch_data(source) -except requests.RequestException as e: - logger.error(f"Failed to fetch from {source}: {e}") - raise DataSourceError(f"Source {source} unavailable") from e -``` - -**Rules:** -- **NEVER swallow errors silently** -- Use custom exceptions for domain errors --区分可恢复错误和不可恢复错误 -- Log errors with appropriate level (warn/error) -- Include context in all error messages -- Propagate errors to caller unless explicitly handled - ---- - -## Security - NON-NEGOTIABLE - -- **NEVER** commit `.env`, secrets, keys, or credentials -- Use environment variables for all credentials -- Validate and sanitize all user inputs -- Use parameterized queries for database operations (SQL injection prevention) -- JWT tokens with short expiration (15 min) -- Redis for token blacklist (logout support) -- Hash passwords with bcrypt/argon2 - **NEVER** store plain text - ---- - -## Git Workflow +- Use deterministic checks before broad manual inspection: ```bash -# Create feature branch -git checkout -b feature/data-collector-huggingface - -# Commit message format -git commit -m "feat: add Hugging Face data collector" -git commit -m "fix: resolve WebSocket heartbeat timeout" -git commit -m "docs: update API documentation" - -# Before opening PR -git fetch origin && git rebase origin/main -./.venv/bin/python -m pytest -s backend/tests && bun run build +git diff --check +rg -n "TODO|FIXME|console\.log|debugger|print\(" ``` -**Rules:** -- Feature branches from main -- Clear commit messages - "Add user authentication", not "fix" -- **NEVER** force push to main -- Run tests and lint **before** committing - --- -## Dependencies +## Module: security -**Rules:** -- Verify package legitimacy before adding -- Prefer well-maintained, widely-used libraries -- Pin dependency versions in `pyproject.toml`, `uv.lock`, and `package.json` -- Review security advisories with `pip-audit` and `bun audit` -- Frontend package management and script execution use **Bun only** -- Frontend commands must use `bun install` / `bun run ...` -- **NEVER** use `npm` / `pnpm` / `yarn` for the frontend project -- **NEVER** add unknown packages +### Load When ---- +Always. -## Data Collector Pattern - MANDATORY +### Must -```python -class BaseCollector: - async def fetch(self) -> List[Dict]: - """Fetch data from source""" - ... +- Never commit `.env`, secrets, keys, tokens, or credentials. +- Use environment variables or the configured settings store for credentials. +- Validate and sanitize user input. +- Use parameterized database queries. +- Never store plain-text passwords. +- Hash passwords with bcrypt/argon2. +- Use short-lived JWT tokens when auth tokens are involved. +- Use token blacklist or equivalent revocation support for logout. +- Do not expose full tokens in UI. Show only a short prefix and mask the rest. - def transform(self, raw_data: Dict) -> NormalizedData: - """Transform to internal format""" - ... +### Verify - async def run(self): - """Full pipeline: fetch -> transform -> save""" - raw = await self.fetch() - data = self.transform(raw) - await self.save(data) +```bash +git diff --name-only HEAD +rg -n "api[_-]?key|client_secret|BEGIN .*PRIVATE KEY|AKIA[0-9A-Z]" . ``` -**Rules:** -- Each data source has its own collector class -- Collectors **MUST** inherit from `BaseCollector` -- Implement `fetch()` and `transform()` methods -- Support incremental and full sync modes - --- -## WebSocket Communication - MANDATORY +## Module: workflow -```python -# Data frame format -{ - "timestamp": "2024-01-15T10:30:00Z", - "type": "update", # or "full" - "payload": { - "gpu_clusters": [...], - "submarine_cables": [...], - "ixp_nodes": [...] - } -} +### Load When -# Heartbeat every 30 seconds +Always. + +### Git + +- Do not revert user changes unless explicitly requested. +- Do not force push to protected branches. +- Use clear commit messages. +- Run relevant tests and builds before committing. +- Frontend package management must use Bun only. +- Never use `npm`, `pnpm`, or `yarn` in the frontend project. +- Verify package legitimacy before adding dependencies. +- Prefer maintained, widely used libraries. +- Pin dependency versions in `pyproject.toml`, `uv.lock`, and `package.json`. + +### Deterministic Context + +- Prefer compact CLI evidence over reading large files or full diffs: + +```bash +git status --short +git diff --stat HEAD +git diff --name-only HEAD +git diff --unified=0 HEAD -- +rg -n "" ``` -**Rules:** -- UE5 communicates via WebSocket (not REST) -- Send data frames at configured intervals (default: 5 minutes) -- Include camera position in control frames -- Support auto-cruise and manual interaction modes +--- + +## Module: docs + +### Load When + +Writing, translating, linking, restructuring, or publishing docs. + +### Must + +- Chinese docs under `docs/technical/zh/` must be Chinese prose, not copied English placeholders. +- Keep technical identifiers, API paths, config keys, code symbols, and product names in English where appropriate. +- Explain why a change exists, not only what files changed. +- Prefer updating an existing relevant doc over creating a duplicate. +- Use `##` and `###` headings; avoid going deeper than three levels. +- Use fenced code blocks with language tags. +- Use tables when comparing options or listing parameters. +- Do not reference PR numbers, issue numbers, or the current conversation. +- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` unless intentionally linking to English-only docs. +- Public Docs UI must only expose documents explicitly registered in `frontend/src/pages/Docs/docs-content.ts`. +- Development plans and task notes under `docs/plans/` are not automatically public documentation. + +### Required Content + +- Background/problem. +- Core design decisions and rationale. +- Key snippets or focused examples. +- Related files and each file's role. +- Operational caveats or verification steps when relevant. + +### Verify + +```bash +git diff --stat HEAD +git diff --name-only HEAD +ls docs/technical/zh/ +rg -n "\]\(([^)]+)\)" docs/technical/zh/.md +rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2 +``` + +Check zh/en duplicates: + +```bash +python - <<'PY' +from pathlib import Path +same = [] +for en in sorted(Path("docs/technical/en").glob("*.md")): + zh = Path("docs/technical/zh") / en.name + if zh.exists() and en.read_text() == zh.read_text(): + same.append(en.name) +if same: + raise SystemExit("identical en/zh docs: " + ", ".join(same)) +print("no identical en/zh docs") +PY +``` --- -## General Guidelines +## Module: uiux -- Keep functions small and focused (single responsibility) -- Write self-documenting code; comment **why**, not what -- One concern per file/module -- Dependency injection for testability -- Feature flags for incomplete features -- Use config files for environment-specific settings +### Load When -## Code Hygiene - MANDATORY +Changing layout, visual hierarchy, controls, interaction states, responsive behavior, or accessibility. -- Maintain a single source of truth for business state. Frontend temporary state, cached state, and persisted backend state must not evolve into parallel truths. -- Transitional paths are temporary. Once a new implementation is stable, remove old branches, old interfaces, old mocks, and compatibility layers instead of letting them linger. -- Repeated logic must be extracted. If request flow, response handling, auth/header assembly, validation, or state reconciliation appears more than once or twice, promote it into a helper or shared layer. -- Repeated backend resource lookup and response assembly must be centralized. Avoid scattering the same `load -> validate -> transform -> respond` pattern across multiple handlers or services. -- Default values, system prompts, placeholder structures, and other fixed constants must be centralized rather than re-declared in multiple states or code paths. -- When debugging layout, scrolling, or overflow issues, first inspect structural ownership of height, width, and overflow before applying isolated style patches. -- Distinct interaction modes must have explicit structure and state semantics. View, edit, loading, error, stopped, and retry states should not be forced through the exact same markup or logic path. -- Presentation state must not pretend to be business state. UI animation, phase labels, and optimistic display layers must defer to real persisted or backend task state when it exists. -- Responsive adaptations must preserve the primary action path. Reflow is fine; losing or displacing the main user action is not. -- After large feature commits, perform an explicit cleanup pass for dead code, temporary branches, duplicated helpers, stale interfaces, and naming drift before considering the work complete. -- If a file or module starts accumulating repeated patterns or mixed responsibilities, stop and refactor before continuing to add more features on top. -- Public interfaces, persisted fields, and state structures must have a current owner and caller. If something is no longer used, delete it instead of keeping it “just in case”. +### Must + +- Backend/admin pages are single-screen workspaces first, not long landing pages. +- Common desktop viewports should show the page header, summary/controls, and main work area. +- The main work area gets most available height. +- If text, controls, or tables become unreadable, give that region an internal scrollbar instead of crushing it. +- Overflow ownership must be explicit: + - parent height chain is valid + - height-constrained flex parents use `min-height: 0` + - only the intended scroll node owns `overflow: auto` +- Do not use `overflow: hidden` as a final fix unless another child owns scrolling. +- Tabs define their own scroll strategy; hidden panes must stay hidden. +- Long-form content such as AI briefs, logs, Markdown, raw JSON, and help text should stay readable. +- Prefer stable readable minimum heights plus scrolling for constrained content. +- Avoid brittle `100vh/100vw` in embedded/admin shells; prefer `height: 100%` chains. +- Verify layouts under browser zoom 125% and 150% when changing height-critical screens. +- Avoid wrapper components with implicit layout behavior, such as `Space`, in height-critical scroll regions unless the generated DOM is accounted for. +- Any UI state that hides data or a layer must also reconcile hover, lock, tooltip, and selection state. + +### Visual Controls + +- Use icons in buttons for common tools/actions when an established icon exists. +- Keep icon-only buttons accessible with `aria-label` and `title`. +- Use segmented controls for modes, switches/checkboxes for binary settings, sliders/inputs for numeric values, menus/selects for option sets, and tabs for views. +- Do not put cards inside cards. +- Do not use visible in-app text to explain obvious UI features or styling. +- Text must fit within its parent on mobile and desktop. +- Do not scale font size with viewport width. +- Letter spacing should usually be `0`. + +### Verify + +```bash +changed=$(git diff --name-only HEAD -- frontend/src) +[ -z "$changed" ] || rg -n "overflow|min-height|Space|Tabs|aria-label|title=" $changed +``` --- -## Query Performance - MANDATORY +## Module: frontend -- **NEVER** load whole tables into Python just to do filtering, pagination, counting, dedupe, or summary aggregation -- Filters, sorting, pagination, `count`, `distinct`, and grouped statistics **MUST** be pushed down to the database whenever the ORM/query builder can express them -- Summary/dashboard endpoints should prefer dedicated aggregate queries or aggregate endpoints, not multiple full-table scans -- For hot paths, avoid selecting large JSON/text payload columns unless the response really needs them -- If an endpoint returns a list, default to database-side pagination instead of `scalars().all()` followed by Python slicing -- When you suspect a query is slow, first check for: +### Load When + +Editing React, TypeScript, CSS, Vite, Bun, admin console, public Docs UI, or client-side services. + +### Must + +- Use Bun for frontend commands: + +```bash +bun install +bun run --cwd frontend build +``` + +- Never use `npm`, `pnpm`, or `yarn`. +- Keep shared behavior in reusable components/services, not page-local copies. +- Prefer existing project components and patterns. +- Keep page state, backend state, and persisted state clearly separated. +- Presentation state must not pretend to be business state. +- Loading, error, stopped, retry, edit, and view states need explicit semantics. +- Responsive adaptations must preserve the primary action path. +- Markdown rendering behavior belongs in the shared Markdown renderer, not individual docs. +- Public Docs navigation must be whitelist-driven through metadata, not file-system fallback. + +### TypeScript/CSS + +- Define interfaces for API payloads and component props. +- Avoid broad casts. +- Prefer CSS classes over inline styles except for truly dynamic values. +- For fixed-format UI elements, define stable dimensions with `aspect-ratio`, grid tracks, min/max constraints, or container-relative sizing. + +### Verify + +```bash +bun run --cwd frontend build +git diff --check -- frontend +rg -n "npm|pnpm|yarn" frontend package.json +``` + +--- + +## Module: backend + +### Load When + +Editing FastAPI, SQLAlchemy, collectors, database models, migrations, services, API routes, or performance-sensitive code. + +### Must + +- Use custom exceptions for domain errors. +- Never swallow errors silently. +- Distinguish recoverable and unrecoverable errors. +- Log errors with useful context and appropriate level. +- Propagate errors unless explicitly handled. +- Repeated `load -> validate -> transform -> respond` flow should be centralized. +- Each data source has its own collector class. +- Collectors must inherit the repository's base collector abstraction when available. +- Collectors implement `fetch()` and `transform()` or the current project-equivalent pipeline hooks. +- Support incremental and full sync modes where the source allows it. + +### Query Performance + +- Never load whole tables into Python for filtering, pagination, counting, dedupe, or summary aggregation. +- Push filters, sorting, pagination, `count`, `distinct`, and grouped statistics down to the database. +- Summary/dashboard endpoints should prefer aggregate queries or aggregate endpoints. +- Avoid selecting large JSON/text payload columns on hot paths unless needed. +- List endpoints default to database-side pagination. +- When a query is slow, first check: - full-table ORM loads - Python-side post-filtering - repeated summary queries that can be merged - - repeated per-request recomputation that should be cached or aggregated once + - repeated per-request recomputation that should be cached or aggregated ---- +### Country Data -## Release Workflow - MANDATORY +- Data sources carrying country, region, or territory fields must validate against `backend/app/core/countries.py`. +- Use `normalize_country(value)` as the single gate. +- If normalization returns `None`, log and reject or flag the value. +- Do not override canonical political labels with raw source labels. +- Add aliases to `COUNTRY_ENTRIES`; do not scatter aliases across collectors or API handlers. +- Frontend country labels should come from the canonical dictionary after normalization. -- When the user asks to `发版`, `bump version`, `release`, or `推送发布类改动`, treat it as a release workflow, not a plain commit -- Apply repository versioning rules consistently: - - `feature` -> `+0.1.0` - - `bugfix` -> `+0.0.1` - - `docs / maintenance / refactor` do **NOT** bump version unless the user explicitly wants a release anyway -- A release bump **MUST** update all version-bearing files together: - - `VERSION` - - `frontend/package.json` - - `pyproject.toml` - - `uv.lock` -- A release bump **MUST** update release records together: - - `docs/CHANGELOG.md` - - `docs/version-history.md` -- Before committing a release, verify the target version appears consistently in all required files -- Before pushing a release, run the smallest relevant validation available for the changed scope and report what was or was not validated -- If runtime output directories are part of the feature flow, confirm they are ignored appropriately so release commits do not accidentally include generated artifacts -- If asked to commit/push release work, do **NOT** skip changelog or version-history updates just because the code changes are small -- Use the repo skill at `/home/ray/dev/linkong/planet/.codex/skills/release-workflow/SKILL.md` whenever performing a release workflow for this repository - ---- - -## Country Data Validation - MANDATORY - -- **ALL** data sources that carry a country, region, or territory field (API responses, GeoJSON, CSVs, scraped data, third-party enrichment) **MUST** have their country values validated against the project's canonical country dictionary at `backend/app/core/countries.py` before being stored or displayed -- Use `normalize_country(value)` from `countries.py` as the single gate. If it returns `None`, the value is unrecognized and must be logged and rejected or flagged — **NEVER** silently pass it through -- The dictionary encodes official political positions (e.g., Taiwan → 中国（台湾）, Kosovo → 塞尔维亚, Gaza → 巴勒斯坦). Do **NOT** override these with raw source data labels -- When integrating a new data source, run a pre-flight check: extract all distinct country values from the source and verify each one resolves via `normalize_country`. Fix unresolved values before wiring up the collector -- Geographic boundary data (GeoJSON, shapefiles, tilesets) must be post-processed to align feature names and hover labels with the dictionary. The Natural Earth `ne_110m_admin_0_countries` dataset downloaded from GitHub was used as the base for the frontend boundary layer; political corrections were applied manually -- If a new country alias needs to be added to the dictionary, add it to `COUNTRY_ENTRIES` in `countries.py` — **NEVER** scatter aliases across individual collectors or API handlers -- Frontend hover tooltips and info cards that display country names must source the name from the canonical dictionary (via `NAME_ZH` after normalization), not raw source strings - ---- - -## Frontend Layout - MANDATORY - -- Backend/admin pages must be designed as a `single-screen workspace` first, not as a long vertically stacked document -- In common desktop viewports, users should be able to see: - - page header - - summary/controls - - the main work area -- The main work area must get the majority of the available height; secondary cards must not crowd it out -- If a card or panel would be compressed until text, controls, or tables become unreadable, stop shrinking it and give that region an internal scrollbar instead -- On small screens, high browser zoom, or reduced viewport height, switch to a compact mode or horizontal summary scrolling before allowing important content to be crushed -- Overflow ownership must be explicit: - - parent height chain must be valid - - height-constrained flex parents need `min-height: 0` - - only the intended scroll node should own `overflow: auto` -- Do **NOT** rely on `overflow: hidden` as the final fix for a crowded layout unless another child container is explicitly responsible for scrolling -- For tabs: - - hidden tab panes must stay hidden - - do not override library hidden-pane selectors in a way that makes inactive content visible - - each tab must define its own scroll strategy instead of inheriting a one-size-fits-all table layout -- For long-form content such as AI briefs, logs, markdown, raw JSON, or help text: - - prefer normal document flow inside the content block - - if height is constrained, use a stable minimum readable height plus scrolling - - do not let flex compression collapse the readable area into a thin strip -- Avoid brittle viewport sizing: - - prefer `height: 100%` chains over naive `100vh/100vw` usage in embedded/admin shells - - verify layouts under browser zoom `125%` and `150%` -- Avoid using wrapper components with implicit layout behavior, such as `Space`, for height-critical scroll regions unless their generated DOM is fully accounted for -- Any UI state that hides data or a layer must also reconcile related hover/lock/tooltip/selection state so hidden content is not still “active” in the UI - ---- - -## Icon System - MANDATORY - -All canvas-drawn marker icons for the 3D earth visualization **MUST** have a canonical SVG in: +### Verify +```bash +git diff --name-only HEAD -- backend +python3 -m py_compile +rg -n "scalars\\(\\)\\.all\\(\\)|\\.all\\(\\).*\\[:|len\\(.*\\.all\\(" backend/app +rg -n "text\\(\"SELECT \\*|execute.*SELECT \\*" backend/app +rg -n "normalize_country|COUNTRY_ENTRIES" backend/app ``` + +--- + +## Module: earth + +### Load When + +Editing `frontend/public/earth`, 3D Earth, canvas/Three.js rendering, BGP/vessel/satellite/cable layers, geographic boundaries, or Earth marker icons. + +### Must + +- Keep rendering state and UI state explicitly synchronized. +- If a layer is hidden, clear or reconcile related hover, lock, tooltip, and selection state. +- Avoid one-off visual patches before checking render order, coordinate ownership, and data lifecycle. +- Use Three.js for 3D elements. +- Verify 3D/canvas work with real rendering, not only TypeScript build. +- Do not let loading messages, phase labels, or optimistic UI override real backend task state. +- WebSocket data frames should include timestamps, type, and payload when streaming Earth state. +- Default heartbeat for real-time streams is 30 seconds unless a protocol states otherwise. + +### Icon System + +Canvas-drawn marker icons for Earth must have canonical SVG sources in: + +```text frontend/public/earth/assets/icons/ ``` -This directory is the **single source of truth** for icon shapes. The canvas/Three.js drawing code may use inline `Path2D` strings or `` draw calls derived from these SVGs, but the geometry must originate here. +This directory is the single source of truth for icon geometry. Canvas or Three.js drawing code may use `Path2D` strings or draw calls derived from these SVGs, but the shape must originate here. -### Naming convention +Naming: -`{module}-{description}.svg` in kebab-case. +| Prefix | Context | +|--------|---------| +| `marker-` | Surface map markers | +| `bgp-` | BGP/routing layer icons and event symbols | +| `compute-` | Compute center markers | -| Module prefix | Context | -|---------------|---------| -| `marker-` | Surface map markers (landing points, etc.) | -| `bgp-` | BGP/routing layer icons and event symbols | -| `compute-` | Compute center markers | +Rules: -Examples: `marker-landing-point.svg`, `bgp-event-triangle.svg`, `compute-gpu-cluster.svg` +- Use `fill="currentColor"` for single-color icons. +- Hardcode brand colors only when color is part of icon identity. +- State variants are handled by calling code via color/opacity; do not create separate SVGs per state. +- Use the native canvas coordinate space as `viewBox`, typically `0 0 128 128`. +- When adding an icon, create the SVG, document it in this module, and reference its geometry from rendering code. -### Existing icons +Current icons: | File | Used in | Description | |------|---------|-------------| -| `marker-landing-point.svg` | `cables.js` | Cable landing point pin (with circular cutout) | -| `bgp-collector.svg` | `bgp.js` | BGP collector marker (access_point icon + outer ring) | -| `bgp-glow-dot.svg` | `bgp.js` | Base radial glow dot under BGP collector | -| `bgp-event-ring.svg` | `bgp.js` | Ring overlay on event markers | +| `marker-landing-point.svg` | `cables.js` | Cable landing point pin | +| `bgp-collector.svg` | `bgp.js` | BGP collector marker | +| `bgp-glow-dot.svg` | `bgp.js` | Base radial glow dot | +| `bgp-event-ring.svg` | `bgp.js` | Event ring overlay | | `bgp-event-triangle.svg` | `bgp.js` | Origin anomaly | | `bgp-event-exclamation.svg` | `bgp.js` | Withdraw event | | `bgp-event-wave.svg` | `bgp.js` | Flap event | | `bgp-event-burst.svg` | `bgp.js` | Specific/burst anomaly | | `bgp-event-leak.svg` | `bgp.js` | Route leak | | `bgp-event-dot.svg` | `bgp.js` | Generic event | -| `compute-supercomputer.svg` | `compute-centers.js` | Supercomputer (#38bdf8) | -| `compute-gpu-cluster.svg` | `compute-centers.js` | GPU cluster (#2dd4bf) | +| `compute-supercomputer.svg` | `compute-centers.js` | Supercomputer | +| `compute-gpu-cluster.svg` | `compute-centers.js` | GPU cluster | -### Color rules +### Verify -- Use `fill=”currentColor”` for single-color icons so the caller controls the color (event symbols, landing point) -- Hardcode brand colors only when the color is part of the icon identity (compute center types) -- State variants (hover, locked, dimmed) are handled by the calling canvas code via color/opacity — **do not create separate SVG files per state** +```bash +bun run --cwd frontend build +rg -n "hover|locked|selected|tooltip|visible|Path2D|drawImage" frontend/public/earth +ls frontend/public/earth/assets/icons/ +``` -### Coordinate system +--- -- Use the native canvas coordinate space as the `viewBox` (typically `0 0 128 128`) -- Exception: `marker-landing-point.svg` uses a `viewBox` cropped from 1000-unit path space -- SVG must visually match the canvas output at the same scale +## Module: ai -### When adding a new icon +### Load When -1. Create the SVG in `assets/icons/` following naming rules above -2. Add a row to the table in this section -3. Reference the SVG path/geometry in the canvas drawing code — do not invent new shapes directly in JS +Editing AI Provider, LLM gateway, AI Playground, prompt templates, model selection, custom collector mapping generation, or LLM-assisted data transformation. + +### Must + +- AI provider endpoint/base URL/model/token configuration belongs in settings/integration config, not hardcoded page state. +- The local API route used by the console is not the same as the external LLM provider base URL. +- Common LLM provider presets should be selectable and refreshable from provider docs or catalog logic. +- Store fallback/default provider config centrally. +- Credential previews must reuse the existing product masking convention instead of inventing page-local display logic. +- Never send secrets to logs or docs. +- LLMs may assist with mapping generation or unknown API exploration, but runtime collection should use saved deterministic mapping rules. +- If custom collectors transform into existing domain data, require an explicit target schema. +- If custom collectors introduce entirely new data, do not pretend Earth can use it until a corresponding feature exists. +- Prompts, mapping schemas, default examples, and provider constants must be centralized. + +### Verify + +```bash +rg -n "AI_PROVIDER|provider_api|base_url|api_key|service_token|prompt|mapping" backend aiprovider frontend/src +git diff --check -- backend aiprovider frontend/src +``` + +--- + +## Module: release + +### Load When + +The user asks to `发版`, bump version, release, commit/push release work, or update changelog/version history as part of a release. + +### Must + +- Treat release work as release workflow, not a plain commit. +- Use `.codex/skills/release/SKILL.md` when Codex performs a release. +- Version bump rules: + - `feature` -> `+0.1.0` + - `improvement` -> `+0.0.1` + - `bugfix` -> `+0.0.1` + - `docs`, `maintenance`, `refactor` do not bump unless explicitly requested +- Mixed bugfix and small feature/UI work defaults to `improvement` unless the user explicitly chooses another release type. +- A release bump updates all version-bearing files together: + - `VERSION` + - `frontend/package.json` + - `pyproject.toml` + - `uv.lock` +- A release bump updates release records together: + - `docs/CHANGELOG.md` + - `docs/version-history.md` +- `uv.lock` must be regenerated by `uv lock`, never edited manually. +- Before committing a release, verify target version consistency. +- Before pushing a release, run the smallest relevant validation for the changed scope and report what was or was not validated. +- Do not include generated runtime output directories in release commits. + +### Verify + +```bash +git branch --show-current +git status --short +cat VERSION +rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock +git diff --stat HEAD +``` diff --git a/uv.lock b/uv.lock index 5f836d53..8f50f425 100644 --- a/uv.lock +++ b/uv.lock @@ -475,7 +475,7 @@ wheels = [ [[package]] name = "planet" -version = "0.42.2" +version = "0.43.0" source = { virtual = "." } dependencies = [ { name = "aiofiles" },