release: bump version to 0.48.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.claude/commands/cleanup.md

@@ -12,6 +12,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
 
 若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。
 
+## 节省上下文规则
+
+优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文：
+
+```bash
+git diff --name-only HEAD
+git diff --unified=0 HEAD -- <path>
+git diff --check
+rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
+```
+
+只有 focused diff 不足以安全判断或修改时，才读取完整文件。
+
 ## 审查清单
 
 按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）：
@@ -59,8 +72,13 @@ git diff HEAD --name-only
 
 ### Step 2 — 逐文件阅读并分析
 
-- 用 Read 工具读取完整文件（不只读 diff）
-- 对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式
+先从 focused diff 开始：
+
+```bash
+git diff --unified=0 HEAD -- <file>
+```
+
+用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。
 
 ### Step 3 — 报告问题清单
 
@@ -95,6 +113,7 @@ git diff HEAD --name-only
 - 只改在审查清单中发现的问题，不做额外优化
 - 每次 Edit 只修改确实有问题的行，保持 diff 最小
 - 改完后用 `grep` 验证旧的坏代码已消失
+- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具
 
 ### Step 5 — 输出总结
 

.claude/commands/docs.md

@@ -0,0 +1,93 @@
+---
+description: Create or update repository documentation from current code changes
+argument-hint: Optional: topic to document, or leave empty to infer from git diff
+allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
+---
+
+# /docs — Documentation Workflow
+
+## Goal
+
+Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
+
+## Repository Rules
+
+Before deciding scope, check whether the repository has a documentation rules file:
+
+```bash
+test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
+```
+
+If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
+
+## Workflow
+
+### Step 1 — Understand The Change
+
+```bash
+git diff HEAD --stat
+git diff HEAD --name-only
+git log --oneline -10
+rg --files docs
+```
+
+If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
+
+```bash
+git diff HEAD -- <path>
+rg -n "class |def |function |export |router|@router|interface |type " <path>
+```
+
+### Step 2 — Decide Scope
+
+- Prefer updating an existing relevant document over creating a duplicate.
+- Use one document for one coherent topic.
+- Split documents only when the change crosses meaningful domains.
+- Keep filenames lowercase and hyphenated.
+- Apply the repository-specific rules file before writing.
+
+For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
+
+### Step 3 — Write
+
+Explain:
+
+- Background/problem: what was wrong or missing before.
+- Core design decisions and rationale.
+- Operational or user-facing impact.
+- Relevant code paths, only when useful for future maintainers.
+
+Style:
+
+- Follow the repository’s existing language and heading conventions.
+- Use fenced code blocks with language tags.
+- Prefer tables for comparisons or parameter lists.
+- Keep snippets concise and relevant.
+
+### Step 4 — Verify
+
+- Read the completed docs once for clarity and stale statements.
+- Verify referenced paths exist with `test -e` or `rg --files`.
+- Run applicable checks from `docs/documentation-coverage-rules.md`.
+- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
+
+### Step 5 — Report
+
+Summarize changed docs and verification:
+
+```md
+Updated:
+- path/to/doc.md — what changed
+
+Verified:
+- checks that passed
+- checks that could not be run, if any
+```
+
+## Hard Constraints
+
+- Do not leave placeholder docs.
+- Do not duplicate bilingual files byte-for-byte.
+- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
+- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
+- Keep docs maintainable and concise.

.claude/commands/goal-driven.md

@@ -72,6 +72,8 @@ Verification
 ## 执行风格
 
 - 重证据，轻口头判断
+- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
+- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据
 - 重验收，轻自我感觉
 - 优先用测试、日志、产物、对比结果来证明完成
 - 对长期任务保持“未达标就继续”的节奏

.claude/commands/release.md

@@ -28,6 +28,19 @@ allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
 - `docs/CHANGELOG.md`
 - `docs/version-history.md`
 
+## 节省上下文规则
+
+发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取：
+
+```bash
+git status --short
+git diff --stat HEAD
+git diff --name-only HEAD
+rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
+```
+
+除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。
+
 ## 执行步骤
 
 ### Step 1 — 环境检查
@@ -45,7 +58,7 @@ cat VERSION                        # 读取当前版本
 ### Step 2 — 确定发版类型与新版本号
 
 - 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用
-- 否则根据当前 `git diff HEAD` 和 `git log` 推断
+- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
 - 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`）
 - **先输出发版计划供用户确认**：
 
@@ -91,12 +104,13 @@ cat VERSION                        # 读取当前版本
 
 针对本次变更范围做最小验证：
 
-- Python 文件有修改：`python3 -m py_compile <changed_files>`
-- Frontend 文件有修改：运行项目标准检查（若无则跳过并说明）
+- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile <changed_files>`
+- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明）
 - 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
 
 ```bash
-grep -h "version" VERSION frontend/package.json pyproject.toml
+cat VERSION
+rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
 ```
 
 ### Step 7 — 提交前预览

.codex/skills/cleanup/SKILL.md

@@ -21,6 +21,19 @@ If the user specifies a file or directory, check only that. Otherwise check all
 
 Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
 
+## Token-Saving Rule
+
+Prefer deterministic CLI checks before reading files into model context:
+
+```bash
+git diff --name-only HEAD
+git diff --unified=0 HEAD -- <path>
+git diff --check
+rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
+```
+
+Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
+
 ## Checklist
 
 ### 1. Duplicate Logic
@@ -64,7 +77,13 @@ Filter to the user-specified path if one was provided.
 
 ### Step 2 — Read and analyze each file
 
-Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
+Start with focused diffs:
+
+```bash
+git diff --unified=0 HEAD -- <file>
+```
+
+Use `rg`, `git diff --check`, and compiler/linter output for deterministic findings. Read the full file only for files that need surrounding context. For each issue found, record filename, line number, category, and suggested fix.
 
 ### Step 3 — Report findings before touching anything
 
@@ -99,6 +118,7 @@ Principles:
 - Only fix issues identified in the checklist — no extra improvements
 - Keep each Edit as small as possible
 - After fixing, verify the old bad pattern is gone with grep
+- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
 
 ### Step 5 — Summary
 

.codex/skills/docs/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: docs
+description: Create or update repository documentation from current code changes. Use when the user asks to write docs, update docs, summarize implementation changes into docs, or check documentation coverage. Load repository-specific coverage rules from docs/documentation-coverage-rules.md when present.
+---
+
+# Docs
+
+Use this skill when the task is documentation work: creating, updating, checking, or summarizing docs for code or behavior changes.
+
+## Goal
+
+Write documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep the skill generic; repository-specific rules belong in the repository, not in this skill.
+
+## Repository Rules
+
+Before deciding scope, check whether the repository has a documentation rules file:
+
+```bash
+test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
+```
+
+If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
+
+## Workflow
+
+1. Gather focused context:
+
+```bash
+git diff HEAD --stat
+git diff HEAD --name-only
+git log --oneline -10
+rg --files docs
+```
+
+If the user gives a topic, focus on that topic. Otherwise infer the doc topic from changed files. Avoid reading large full diffs by default; inspect focused files and symbols:
+
+```bash
+git diff HEAD -- <path>
+rg -n "class |def |function |export |router|@router|interface |type " <path>
+```
+
+2. Decide scope:
+
+- Prefer updating an existing relevant doc over creating a duplicate.
+- Use one document for one coherent topic.
+- Split documents only when changes cross meaningful domains.
+- Keep filenames lowercase and hyphenated.
+
+3. Write the doc:
+
+- Explain background/problem, design decisions, constraints, and operational impact.
+- Keep code snippets short and directly relevant.
+- List related files only when they help future maintainers navigate.
+- Use the repository’s existing language, heading style, and naming conventions.
+
+4. Verify:
+
+- Read the completed doc once for clarity and stale statements.
+- Verify important referenced paths exist with `test -e` or `rg --files`.
+- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
+- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
+
+## Hard Constraints
+
+- Do not leave placeholder docs or copied source text pretending to be documentation.
+- Do not duplicate bilingual files byte-for-byte.
+- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
+- Do not write changelog-style lists without the reasoning, constraints, and tradeoffs behind the change.
+- Keep docs concise enough to maintain.
+
+## Recommended Output
+
+After editing, summarize:
+
+```md
+Updated:
+- path/to/doc.md — what changed
+
+Verified:
+- checks that passed
+- checks that could not be run, if any
+```

.codex/skills/goal-driven/SKILL.md

@@ -72,6 +72,8 @@ In Codex, only use actual subagents when the user explicitly asks for delegation
 ## Operating Rules
 
 - Prefer objective checks over self-reported completion.
+- Prefer deterministic tool evidence over long model summaries: use `rg`, `git diff --stat`, targeted `git diff -- <path>`, tests, builds, linters, `curl`, or database queries when they can prove a criterion.
+- Do not paste large command output into the conversation; summarize the evidence and keep raw output in tool calls.
 - Do not confuse progress with completion.
 - If the worker says "done", verify it.
 - If verification fails, continue from the gap instead of restarting blindly.

.codex/skills/release/SKILL.md

@@ -18,7 +18,7 @@ Do not use this skill for ordinary commits that are not being released.
 
 ## Versioning Rules
 
-- `feature` -> bump `+0.1.0`
+- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
 - `bugfix` -> bump `+0.0.1`
 - `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
 
@@ -35,6 +35,19 @@ Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative
 - `docs/CHANGELOG.md`
 - `docs/version-history.md`
 
+## Token-Saving Rule
+
+Release work should be driven by deterministic CLI evidence. Prefer compact commands and targeted file reads:
+
+```bash
+git status --short
+git diff --stat HEAD
+git diff --name-only HEAD
+rg -n "version|^## |^Released:|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
+```
+
+Do not inspect full diffs unless deciding whether changed code belongs in the release.
+
 ## Workflow
 
 ### Step 1 — Environment check
@@ -52,8 +65,10 @@ If unrelated uncommitted changes exist, list them and ask the user whether to in
 ### Step 2 — Determine release type and next version
 
 - If the user provided an explicit type (`feature` / `bugfix`), use it
-- Otherwise infer from `git diff HEAD` and recent `git log`
-- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
+- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
+- Compute the next version:
+  - `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
+  - `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
 - **Show the release plan before making any changes:**
 
 ```
@@ -104,12 +119,13 @@ Get today's date with `date +%Y-%m-%d`.
 
 Run the smallest relevant validation for the changes in scope:
 
-- Python files changed: `python3 -m py_compile <changed_files>`
-- Frontend files changed: run the project-standard check if available; otherwise skip and say so
+- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
+- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
 - Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
 
 ```bash
-grep -h "version" VERSION frontend/package.json pyproject.toml
+cat VERSION
+rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
 ```
 
 ### Step 7 — Pre-commit preview

.dockerignore

@@ -0,0 +1,13 @@
+**
+
+!pyproject.toml
+!uv.lock
+!aiprovider/
+!aiprovider/**
+
+aiprovider/.env
+aiprovider/.env.*
+!aiprovider/.env.example
+**/__pycache__/
+**/*.pyc
+**/*.pyo

TODO.md

@@ -22,8 +22,19 @@
 - [ ] 可选优化（非必做）：将 BGP incident/collector 标点改为 HTML marker（参考 worldmonitor 的 `htmlElementsData` 思路），实现近乎固定屏幕尺寸与更高密度可点击性
 - [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化：`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置，避免把设备级偏好过早做成全局配置
 - [ ] 如果后续明确需要“账号级同步 Earth 偏好”，再单独设计 `Earth user preferences`：优先按用户维度而不是全局系统设置保存，并规划 `localStorage -> backend` 的平滑迁移策略
+- [ ] 为 Planet / Earth 补一个可用的日志查看系统：先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力，再决定是先做脚本级统一入口还是控制台内置日志面板
+- [ ] 重写控制台 UI，逐步抛弃 Ant Design，建立自有组件体系，并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
 - [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录，优先保持当前“实时聚合”链路不变，只先解决新闻源不可配置的问题
 - [ ] 为 Earth 态势新闻设计后续采集器化方案：明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式，再决定何时把新闻从实时抓取升级成正式 collector
+- [ ] AIS v3.1：修复船只聚合完整性，`/geo/vessels` 合并 raw observation 聚合结果与 legacy `vessel_position + vessel_static` 最新结果，确保 BarentsWatch-only 船只不会因为 AISStream 子集存在而消失，并增加 raw/legacy/final unique MMSI 诊断统计
+- [ ] AIS v3.2：把 AISStream 从收满 `max_messages` 后结束的批采集改成长连接 streaming service，持续写入 raw observations，通过内部 `/ws` 的 `vessels` channel 推送新船、位置和航向增量，Earth 前端按 MMSI upsert marker
+- [ ] AIS v3.3：修正 AISStream 采集页面状态语义，使用 connecting/streaming/reconnecting/stopped 与 indeterminate 状态，展示运行时长、消息数、unique MMSI、message rate、最近消息和错误，不再用一次性 REST 进度条表示长连接
+- [ ] AIS v3.4：修复船只身份字段和名称聚合，MMSI/IMO/callsign 按字符串显示且不带千分位符；查询并列出所有仍以 MMSI 号码或 `MMSI <number>` 作为船名的记录，标注来源、最近观测、message types 和缺失原因，并把这批 fallback-name 船只纳入名称聚合修复集合
+- [ ] Earth Live Sync：建立统一态势实时同步链路，新增 `earth_summary` WS channel，任意采集器成功后广播轻量 summary invalidation，前端收到后重新拉 `/api/v1/visualization/geo/summary` 并更新 HUD；同时为 BGP 增加 `bgp` WS channel，使 BGP incidents/anomalies/collectors 在不刷新页面时也能 upsert 图层；卫星采集完成后触发 summary 刷新，必要时按 TLE 版本重新 hydrate 卫星数据
+- [ ] AIS v4：开放船只多源聚合策略配置，支持 source priority、字段级规则、freshness 窗口和高级保护开关；保存时校验未知字段、非法模式和危险动态字段锁定，并在聚合接口返回命中的配置版本
+- [ ] AIS v5：实现船舶资料 enrichment 与冲突治理，按 `mmsi + imo + name + callsign` 异步补充船型细分、AIS 大类、旗国、尺寸、建造年份、运营方和图片缓存；详情面板展示缓存资料和字段来源，不在实时 AIS 请求链路现场抓第三方页面
+- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay，并在同层叠加国界轮廓参考线；要求国界线与底图稳定对齐，且 hover 到国家轮廓时能高亮当前国家，便于校准地表和增强交互
+- [ ] 把 Earth 新闻接入通用巡航队列：按新闻发生地和时间排序生成巡航目标，巡航聚焦到新闻事件时显示对应新闻卡片，并保持实现边界为“通用巡航层 + 新闻业务适配层”，不要再把新闻逻辑直接耦合回 `main.js` 状态机
 - [ ] 为未知位置的算力中心建立分层坐标补全链路：优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`，并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
 - [ ] 为算力中心补一份可维护的本地位置注册表，例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`，避免把地点知识长期硬编码在 `visualization.py`
 - [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析：即使公开 API 不给坐标，也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索

VERSION

@@ -1 +1 @@
-0.37.0
+0.48.0

aiprovider/Dockerfile

@@ -1,6 +1,12 @@
-FROM python:3.14-slim
+# syntax=docker/dockerfile:1.7
 
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+ARG PYTHON_IMAGE=python:3.14-slim
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
+
+FROM ${UV_IMAGE} AS uv
+FROM ${PYTHON_IMAGE}
+
+COPY --from=uv /uv /uvx /bin/
 
 WORKDIR /app
 
@@ -14,9 +20,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && rm -rf /var/lib/apt/lists/*
 
 COPY pyproject.toml uv.lock /app/
-RUN uv sync --frozen --no-dev
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-dev
 
-COPY . /app
+COPY aiprovider /app/aiprovider
 
 EXPOSE 8010
 

aiprovider/main.py

@@ -37,8 +37,26 @@ def verify_service_token(x_provider_token: str | None = Header(default=None)) ->
         )
 
 
-def get_provider_service() -> ProviderService:
-    return ProviderService()
+def get_provider_service(
+    x_ai_provider: str | None = Header(default=None),
+    x_ai_provider_api: str | None = Header(default=None),
+    x_ai_base_url: str | None = Header(default=None),
+    x_ai_api_key: str | None = Header(default=None),
+    x_ai_model: str | None = Header(default=None),
+    x_ai_max_tokens: str | None = Header(default=None),
+    x_ai_anthropic_version: str | None = Header(default=None),
+) -> ProviderService:
+    overrides = {
+        "provider": x_ai_provider,
+        "provider_api": x_ai_provider_api,
+        "base_url": x_ai_base_url,
+        "api_key": x_ai_api_key,
+        "model": x_ai_model,
+        "anthropic_version": x_ai_anthropic_version,
+    }
+    if x_ai_max_tokens:
+        overrides["max_tokens"] = x_ai_max_tokens
+    return ProviderService({key: value for key, value in overrides.items() if value not in (None, "")})
 
 
 @app.get("/health")

aiprovider/provider_service.py

@@ -46,19 +46,22 @@ def _resolve_provider_api(provider: str, configured_api: str) -> str:
 
 
 class ProviderService:
-    def __init__(self) -> None:
-        self.provider = _normalize_provider(settings.AI_PROVIDER)
+    def __init__(self, overrides: dict[str, Any] | None = None) -> None:
+        overrides = overrides or {}
+        self.provider = _normalize_provider(overrides.get("provider") or settings.AI_PROVIDER)
         self.provider_api = _resolve_provider_api(
             self.provider,
-            _normalize_provider_api(settings.AI_PROVIDER_API),
+            _normalize_provider_api(overrides.get("provider_api") or settings.AI_PROVIDER_API),
         )
-        self.base_url = settings.AI_BASE_URL.rstrip("/")
-        self.api_key = settings.AI_API_KEY
-        self.default_model = settings.AI_MODEL
+        self.base_url = str(overrides.get("base_url") or settings.AI_BASE_URL).rstrip("/")
+        self.api_key = str(overrides.get("api_key") or settings.AI_API_KEY)
+        self.default_model = str(overrides.get("model") or settings.AI_MODEL)
         self.timeout = settings.AI_TIMEOUT_SECONDS
         self.http_retry_attempts = max(settings.AI_HTTP_RETRY_ATTEMPTS, 1)
-        self.max_tokens = settings.AI_MAX_TOKENS
-        self.anthropic_version = settings.AI_ANTHROPIC_VERSION
+        self.max_tokens = int(overrides.get("max_tokens") or settings.AI_MAX_TOKENS)
+        self.anthropic_version = str(
+            overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
+        )
         self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
 
     def get_status(self) -> AIProviderStatusResponse:

backend/Dockerfile

@@ -1,6 +1,10 @@
-FROM python:3.14-slim
+ARG PYTHON_IMAGE=python:3.14-slim
+ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
 
-COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+FROM ${UV_IMAGE} AS uv
+FROM ${PYTHON_IMAGE}
+
+COPY --from=uv /uv /uvx /bin/
 
 WORKDIR /app
 

backend/app/api/main.py

@@ -12,6 +12,7 @@ from app.api.v1 import (
     settings,
     collected_data,
     visualization,
+    vessel_aggregation,
     bgp,
     news,
     system_control,
@@ -34,6 +35,11 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
 api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
 api_router.include_router(system_control.router, prefix="/system", tags=["system"])
 api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
+api_router.include_router(
+    vessel_aggregation.router,
+    prefix="/vessel-aggregation",
+    tags=["vessel-aggregation"],
+)
 api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
 api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
 api_router.include_router(news.router, prefix="/news", tags=["news"])

backend/app/api/v1/datasource_config.py

@@ -1,20 +1,52 @@
 """DataSourceConfig API for user-defined data sources"""
 
-from typing import Optional
+from typing import Any, Optional
 from datetime import datetime
 import base64
-from fastapi import APIRouter, Depends, HTTPException, status
-from sqlalchemy import select, func
+import json
+import re
+from fastapi import APIRouter, Depends, HTTPException, Query
+from sqlalchemy import delete, select, func
 from sqlalchemy.ext.asyncio import AsyncSession
 from pydantic import BaseModel, Field
 import httpx
 
+from app.core.target_schema_registry import get_target_schema, list_target_schemas
+from app.core.datasource_defaults import DEFAULT_DATASOURCES
 from app.db.session import get_db
 from app.models.user import User
 from app.models.datasource_config import DataSourceConfig
+from app.models.datasource_mapping import DataSourceMappingTemplate
+from app.models.collected_data import CollectedData
+from app.models.vessel import AISRawObservation, AISSourceHealth
 from app.core.security import get_current_user
 from app.core.cache import cache
 from app.core.time import to_iso8601_utc
+from app.schemas.ai import SituationalAnalysisRequest
+from app.services.ai_client import AIProviderClient, get_ai_provider_client
+from app.services.datasource_mapping import (
+    MappingError,
+    build_heuristic_mapping,
+    execute_mapping,
+    redact_for_llm,
+    stable_payload_hash,
+)
+from app.services.custom_datasource_runtime import (
+    CustomDatasourceRuntimeError,
+    fetch_rest_payload,
+    get_custom_stream_status,
+    run_mapped_rest_config,
+    run_mapped_websocket_config,
+    start_custom_stream,
+    stop_custom_stream,
+    test_websocket_config,
+)
+from app.services.datasource_connectivity import (
+    get_builtin_connection_status,
+    save_connectivity_success,
+    strip_connectivity_validation,
+    test_builtin_connectivity,
+)
 
 router = APIRouter()
 
@@ -22,7 +54,7 @@ router = APIRouter()
 class DataSourceConfigCreate(BaseModel):
     name: str = Field(..., min_length=1, max_length=100)
     description: Optional[str] = None
-    source_type: str = Field(..., description="http, api, database")
+    source_type: str = Field(..., description="rest, websocket, http, api, database")
     endpoint: str = Field(..., max_length=500)
     auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
     auth_config: dict = Field(default={})
@@ -59,6 +91,70 @@ class DataSourceConfigResponse(BaseModel):
         from_attributes = True
 
 
+def _is_builtin_config_name(name: str | None) -> bool:
+    return bool(name and name in DEFAULT_DATASOURCES)
+
+
+async def _ensure_builtin_connection_verified(
+    db: AsyncSession,
+    config_data: DataSourceConfigCreate,
+) -> None:
+    if not _is_builtin_config_name(config_data.name):
+        return
+
+    status_result = await get_builtin_connection_status(
+        db,
+        config_data.name,
+        config_data.endpoint,
+        config_data.auth_type,
+        config_data.headers,
+        config_data.config,
+    )
+    if not status_result.get("connected"):
+        raise HTTPException(
+            status_code=400,
+            detail=status_result.get("message") or "请先完成连接验证，再保存内置采集器配置。",
+        )
+
+
+class CustomSampleRequest(BaseModel):
+    datasource_config_id: Optional[int] = None
+    config: Optional[DataSourceConfigCreate] = None
+    limit_bytes: int = Field(default=200000, ge=1000, le=1000000)
+
+
+class MappingProposeRequest(BaseModel):
+    sample_payload: Any
+    target_schema: str
+    use_ai: bool = True
+
+
+class MappingPreviewRequest(BaseModel):
+    sample_payload: Any
+    target_schema: str
+    mapping_json: dict
+    limit: int = Field(default=20, ge=1, le=100)
+
+
+class MappingTemplateCreate(BaseModel):
+    datasource_config_id: int
+    target_schema: str
+    mapping_json: dict
+    sample_payload: Any | None = None
+    sample_payload_hash: Optional[str] = None
+    validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
+    is_active: bool = False
+
+
+class MappingTemplateUpdate(BaseModel):
+    target_schema: Optional[str] = None
+    mapping_json: Optional[dict] = None
+    sample_payload: Any | None = None
+    sample_payload_hash: Optional[str] = None
+    validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
+    is_active: Optional[bool] = None
+
+
 async def test_endpoint(
     endpoint: str,
     auth_type: str,
@@ -96,6 +192,136 @@ async def test_endpoint(
         }
 
 
+def _build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
+    request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
+    auth_type = str(auth_type or "none").lower()
+    auth_config = auth_config or {}
+
+    if auth_type == "bearer" and auth_config.get("token"):
+        request_headers["Authorization"] = f"Bearer {auth_config['token']}"
+    elif auth_type == "api_key" and auth_config.get("api_key"):
+        location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
+        if location != "query":
+            key_name = auth_config.get("key_name", "X-API-Key")
+            request_headers[str(key_name)] = str(auth_config["api_key"])
+    elif auth_type == "basic":
+        username = auth_config.get("username", "")
+        password = auth_config.get("password", "")
+        credentials = f"{username}:{password}"
+        encoded = base64.b64encode(credentials.encode()).decode()
+        request_headers["Authorization"] = f"Basic {encoded}"
+    return request_headers
+
+
+def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
+    params = {}
+    candidate = (config or {}).get("params") or (config or {}).get("query_params")
+    if isinstance(candidate, dict):
+        params.update(candidate)
+
+    auth_type = str(auth_type or "none").lower()
+    auth_config = auth_config or {}
+    if auth_type == "api_key" and auth_config.get("api_key"):
+        location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
+        if location == "query":
+            key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
+            params[str(key_name)] = auth_config["api_key"]
+    return params
+
+
+async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
+    if str(config.source_type or "").lower() in {"websocket", "ws"}:
+        raise HTTPException(status_code=400, detail="WebSocket sources must use connection test or run-mapped stream.")
+    request_config = config.config or {}
+    method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
+    if method not in {"GET", "POST"}:
+        raise HTTPException(status_code=400, detail="Only GET and POST sample requests are supported.")
+
+    headers = _build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
+    params = _build_query_params(config.auth_type, config.auth_config or {}, request_config)
+    timeout = float(request_config.get("timeout", 30))
+    json_body = request_config.get("json_body")
+    if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
+        candidate = request_config.get("body")
+        if isinstance(candidate, (dict, list)):
+            json_body = candidate
+
+    async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
+        response = await client.request(
+            method,
+            config.endpoint,
+            headers=headers,
+            params=params or None,
+            json=json_body,
+        )
+        response.raise_for_status()
+        content = response.content[:limit_bytes]
+        if "application/json" in response.headers.get("content-type", ""):
+            return json.loads(content.decode(response.encoding or "utf-8"))
+        return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
+
+
+def _parse_mapping_from_ai_text(content: str) -> dict[str, Any] | None:
+    if not content:
+        return None
+
+    candidates = [content]
+    fenced = re.findall(r"```(?:json)?\s*(\{.*?\})\s*```", content, flags=re.DOTALL)
+    candidates = fenced + candidates
+    for candidate in candidates:
+        try:
+            parsed = json.loads(candidate)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(parsed, dict) and isinstance(parsed.get("fields"), dict):
+            return parsed
+    return None
+
+
+async def _get_config_for_sample(
+    payload: CustomSampleRequest,
+    db: AsyncSession,
+) -> DataSourceConfig:
+    if payload.datasource_config_id is not None:
+        result = await db.execute(
+            select(DataSourceConfig).where(DataSourceConfig.id == payload.datasource_config_id)
+        )
+        config = result.scalar_one_or_none()
+        if not config:
+            raise HTTPException(status_code=404, detail="Configuration not found")
+        return config
+
+    if payload.config is None:
+        raise HTTPException(status_code=400, detail="datasource_config_id or config is required")
+
+    config_data = payload.config
+    return DataSourceConfig(
+        name=config_data.name,
+        description=config_data.description,
+        source_type=config_data.source_type,
+        endpoint=config_data.endpoint,
+        auth_type=config_data.auth_type,
+        auth_config=config_data.auth_config,
+        headers=config_data.headers,
+        config=config_data.config,
+    )
+
+
+def serialize_mapping_template(template: DataSourceMappingTemplate) -> dict[str, Any]:
+    return {
+        "id": template.id,
+        "datasource_config_id": template.datasource_config_id,
+        "target_schema": template.target_schema,
+        "mapping_json": template.mapping_json,
+        "sample_payload_hash": template.sample_payload_hash,
+        "validation_status": template.validation_status,
+        "version": template.version,
+        "is_active": template.is_active,
+        "created_at": to_iso8601_utc(template.created_at),
+        "updated_at": to_iso8601_utc(template.updated_at),
+    }
+
+
 @router.get("/configs")
 async def list_configs(
     active_only: bool = False,
@@ -105,7 +331,7 @@ async def list_configs(
     """List all user-defined data source configurations"""
     query = select(DataSourceConfig)
     if active_only:
-        query = query.where(DataSourceConfig.is_active == True)
+        query = query.where(DataSourceConfig.is_active)
     query = query.order_by(DataSourceConfig.created_at.desc())
 
     result = await db.execute(query)
@@ -132,6 +358,52 @@ async def list_configs(
     }
 
 
+@router.get("/configs/all")
+async def list_all_datasources(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """List all data sources: YAML defaults + DB overrides"""
+    from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
+
+    config = get_data_sources_config()
+
+    db_query = await db.execute(select(DataSourceConfig))
+    db_configs = {c.name: c for c in db_query.scalars().all()}
+
+    result = []
+    for name, yaml_key in COLLECTOR_URL_KEYS.items():
+        yaml_url = config.get_yaml_url(name)
+        db_config = db_configs.get(name)
+
+        result.append(
+            {
+                "name": name,
+                "default_url": yaml_url,
+                "endpoint": db_config.endpoint if db_config else yaml_url,
+                "is_overridden": db_config is not None and db_config.endpoint != yaml_url
+                if yaml_url
+                else db_config is not None,
+                "is_active": db_config.is_active if db_config else True,
+                "source_type": db_config.source_type if db_config else "http",
+                "auth_type": db_config.auth_type if db_config else "none",
+                "auth_configured": {
+                    "api_key": bool((db_config.auth_config or {}).get("api_key"))
+                    if db_config
+                    else False,
+                },
+                "headers": db_config.headers if db_config else {},
+                "config": strip_connectivity_validation(db_config.config if db_config else {}),
+                "config_id": db_config.id if db_config else None,
+                "description": db_config.description
+                if db_config
+                else f"Data source from YAML: {yaml_key}",
+            }
+        )
+
+    return {"total": len(result), "data": result}
+
+
 @router.get("/configs/{config_id}")
 async def get_config(
     config_id: int,
@@ -176,7 +448,7 @@ async def create_config(
         auth_type=config_data.auth_type,
         auth_config=config_data.auth_config,
         headers=config_data.headers,
-        config=config_data.config,
+        config=strip_connectivity_validation(config_data.config),
     )
 
     db.add(config)
@@ -208,6 +480,10 @@ async def update_config(
 
     update_data = config_data.model_dump(exclude_unset=True)
     for field, value in update_data.items():
+        if field == "config":
+            value = strip_connectivity_validation(value)
+        if field == "auth_config" and value == {} and (config.auth_config or {}):
+            continue
         setattr(config, field, value)
 
     await db.commit()
@@ -225,6 +501,8 @@ async def update_config(
 @router.delete("/configs/{config_id}")
 async def delete_config(
     config_id: int,
+    delete_mappings: bool = Query(False),
+    delete_source_data: bool = Query(False),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -235,12 +513,59 @@ async def delete_config(
     if not config:
         raise HTTPException(status_code=404, detail="Configuration not found")
 
+    deleted_mappings = 0
+    deleted_records = {
+        "collected_data": 0,
+        "ais_raw_observations": 0,
+        "ais_source_health": 0,
+    }
+
+    if delete_source_data:
+        collected_result = await db.execute(
+            delete(CollectedData).where(CollectedData.source == config.name)
+        )
+        raw_result = await db.execute(
+            delete(AISRawObservation).where(AISRawObservation.source == config.name)
+        )
+        health_result = await db.execute(
+            delete(AISSourceHealth).where(AISSourceHealth.source == config.name)
+        )
+        deleted_records = {
+            "collected_data": collected_result.rowcount or 0,
+            "ais_raw_observations": raw_result.rowcount or 0,
+            "ais_source_health": health_result.rowcount or 0,
+        }
+
+    if delete_mappings or delete_source_data:
+        mapping_result = await db.execute(
+            delete(DataSourceMappingTemplate).where(
+                DataSourceMappingTemplate.datasource_config_id == config_id
+            )
+        )
+        deleted_mappings = mapping_result.rowcount or 0
+
     await db.delete(config)
     await db.commit()
 
     cache.delete_pattern("datasource_configs:*")
 
-    return {"message": "Configuration deleted successfully"}
+    if delete_source_data and (config.config or {}).get("target_schema") == "vessel_ais":
+        from app.core.websocket.broadcaster import broadcaster
+
+        await broadcaster.broadcast_custom(
+            "vessels",
+            {
+                "action": "reload",
+                "source": config.name,
+                "reason": "custom_source_deleted",
+            },
+        )
+
+    return {
+        "message": "Configuration deleted successfully",
+        "deleted_mappings": deleted_mappings,
+        "deleted_records": deleted_records,
+    }
 
 
 @router.post("/configs/{config_id}/test")
@@ -257,6 +582,8 @@ async def test_config(
         raise HTTPException(status_code=404, detail="Configuration not found")
 
     try:
+        if str(config.source_type or "").lower() in {"websocket", "ws"}:
+            return await test_websocket_config(config)
         result = await test_endpoint(
             endpoint=config.endpoint,
             auth_type=config.auth_type,
@@ -287,6 +614,18 @@ async def test_new_config(
 ):
     """Test a new data source configuration without saving"""
     try:
+        if str(config_data.source_type or "").lower() in {"websocket", "ws"}:
+            config = DataSourceConfig(
+                name=config_data.name,
+                description=config_data.description,
+                source_type=config_data.source_type,
+                endpoint=config_data.endpoint,
+                auth_type=config_data.auth_type,
+                auth_config=config_data.auth_config,
+                headers=config_data.headers,
+                config=config_data.config,
+            )
+            return await test_websocket_config(config)
         result = await test_endpoint(
             endpoint=config_data.endpoint,
             auth_type=config_data.auth_type,
@@ -310,38 +649,363 @@ async def test_new_config(
         }
 
 
-@router.get("/configs/all")
-async def list_all_datasources(
+@router.post("/configs/builtin/connection-status")
+async def get_builtin_config_connection_status(
+    config_data: DataSourceConfigCreate,
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    """List all data sources: YAML defaults + DB overrides"""
-    from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
+    if not _is_builtin_config_name(config_data.name):
+        raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
 
-    config = get_data_sources_config()
+    return await get_builtin_connection_status(
+        db,
+        config_data.name,
+        config_data.endpoint,
+        config_data.auth_type,
+        config_data.headers,
+        config_data.config,
+    )
 
-    db_query = await db.execute(select(DataSourceConfig))
-    db_configs = {c.name: c for c in db_query.scalars().all()}
 
-    result = []
-    for name, yaml_key in COLLECTOR_URL_KEYS.items():
-        yaml_url = config.get_yaml_url(name)
-        db_config = db_configs.get(name)
+@router.post("/configs/builtin/connect")
+async def connect_builtin_config(
+    config_data: DataSourceConfigCreate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    if not _is_builtin_config_name(config_data.name):
+        raise HTTPException(status_code=400, detail="Only built-in datasource configs are supported.")
 
-        result.append(
+    result = await test_builtin_connectivity(
+        config_data.name,
+        config_data.endpoint,
+        config_data.auth_type,
+        config_data.headers,
+        config_data.config,
+        db,
+        config_data.auth_config,
+    )
+    if result.get("success") and result.get("checksum"):
+        validation = await save_connectivity_success(
+            db,
+            config_data.name,
+            result["checksum"],
+            result,
+            connected_by="connection_button",
+        )
+        await db.commit()
+        return {
+            **result,
+            "connected": True,
+            "validation": validation,
+        }
+
+    return {
+        **result,
+        "connected": False,
+    }
+
+
+@router.post("/custom/sample")
+async def fetch_custom_sample(
+    payload: CustomSampleRequest,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Fetch a sample payload for a saved or draft custom data source."""
+    config = await _get_config_for_sample(payload, db)
+    try:
+        sample = await fetch_custom_sample_from_config(config, payload.limit_bytes)
+    except httpx.HTTPStatusError as exc:
+        raise HTTPException(
+            status_code=exc.response.status_code,
+            detail=f"Sample request failed: HTTP {exc.response.status_code}",
+        ) from exc
+    except httpx.HTTPError as exc:
+        raise HTTPException(status_code=502, detail=f"Sample request failed: {exc}") from exc
+
+    return {
+        "success": True,
+        "sample_payload": sample,
+        "sample_payload_hash": stable_payload_hash(sample),
+        "redacted_preview": redact_for_llm(sample),
+    }
+
+
+@router.get("/target-schemas")
+async def get_datasource_target_schemas(
+    current_user: User = Depends(get_current_user),
+):
+    """List target schemas available for custom datasource mapping."""
+    return {"data": list_target_schemas()}
+
+
+@router.post("/mappings/propose")
+async def propose_datasource_mapping(
+    payload: MappingProposeRequest,
+    current_user: User = Depends(get_current_user),
+    ai_client: AIProviderClient = Depends(get_ai_provider_client),
+):
+    """Generate a mapping draft for a sample payload and target schema."""
+    schema = get_target_schema(payload.target_schema)
+    redacted_sample = redact_for_llm(payload.sample_payload)
+    fallback_mapping = build_heuristic_mapping(redacted_sample, payload.target_schema)
+
+    ai_error: str | None = None
+    mapping = fallback_mapping
+    generated_by = "heuristic"
+    if payload.use_ai:
+        try:
+            response = await ai_client.analyze(
+                SituationalAnalysisRequest(
+                    title=f"Generate datasource mapping for {schema.key}",
+                    objective=(
+                        "Return only JSON for a deterministic mapping DSL. "
+                        "The JSON must contain source.items_path and fields. "
+                        "Do not include prose or code."
+                    ),
+                    context={
+                        "target_schema": schema.to_dict(),
+                        "sample_payload": redacted_sample,
+                        "mapping_dsl_example": fallback_mapping,
+                    },
+                    observations=[
+                        "Use JSONPath-like paths beginning with $.",
+                        "Never generate executable code.",
+                        "Use field types from the target schema.",
+                    ],
+                    constraints=[
+                        "Return a single JSON object.",
+                        "Do not include credentials or secrets.",
+                        "Mark uncertain optional fields with default null.",
+                    ],
+                )
+            )
+            parsed = _parse_mapping_from_ai_text(response.content)
+            if parsed:
+                mapping = parsed
+                generated_by = "ai_provider"
+            else:
+                ai_error = "AI provider did not return a valid mapping JSON object."
+        except HTTPException as exc:
+            ai_error = str(exc.detail)
+
+    mapping.setdefault("meta", {})
+    if isinstance(mapping["meta"], dict):
+        mapping["meta"].update(
             {
-                "name": name,
-                "default_url": yaml_url,
-                "endpoint": db_config.endpoint if db_config else yaml_url,
-                "is_overridden": db_config is not None and db_config.endpoint != yaml_url
-                if yaml_url
-                else db_config is not None,
-                "is_active": db_config.is_active if db_config else True,
-                "source_type": db_config.source_type if db_config else "http",
-                "description": db_config.description
-                if db_config
-                else f"Data source from YAML: {yaml_key}",
+                "generated_by": generated_by,
+                "requires_review": True,
+                "ai_error": ai_error,
             }
         )
 
-    return {"total": len(result), "data": result}
+    return {
+        "target_schema": schema.to_dict(),
+        "mapping_json": mapping,
+        "sample_payload_hash": stable_payload_hash(payload.sample_payload),
+        "redacted_sample_payload": redacted_sample,
+    }
+
+
+@router.post("/mappings/preview")
+async def preview_datasource_mapping(
+    payload: MappingPreviewRequest,
+    current_user: User = Depends(get_current_user),
+):
+    """Preview deterministic mapping output for a sample payload."""
+    try:
+        preview = execute_mapping(
+            payload.sample_payload,
+            payload.mapping_json,
+            payload.target_schema,
+            limit=payload.limit,
+        )
+    except (MappingError, ValueError) as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+    return {
+        "success": preview["failed_count"] == 0,
+        "preview": preview,
+        "sample_payload_hash": stable_payload_hash(payload.sample_payload),
+    }
+
+
+@router.get("/mappings")
+async def list_datasource_mappings(
+    datasource_config_id: Optional[int] = None,
+    active_only: bool = False,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """List saved mapping templates."""
+    query = select(DataSourceMappingTemplate).order_by(
+        DataSourceMappingTemplate.datasource_config_id,
+        DataSourceMappingTemplate.version.desc(),
+    )
+    if datasource_config_id is not None:
+        query = query.where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
+    if active_only:
+        query = query.where(DataSourceMappingTemplate.is_active.is_(True))
+
+    result = await db.execute(query)
+    mappings = result.scalars().all()
+    return {"total": len(mappings), "data": [serialize_mapping_template(item) for item in mappings]}
+
+
+@router.post("/mappings")
+async def create_datasource_mapping(
+    payload: MappingTemplateCreate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Save a mapping template for a datasource config."""
+    get_target_schema(payload.target_schema)
+    datasource = await db.get(DataSourceConfig, payload.datasource_config_id)
+    if not datasource:
+        raise HTTPException(status_code=404, detail="Configuration not found")
+
+    if payload.sample_payload is not None:
+        try:
+            execute_mapping(payload.sample_payload, payload.mapping_json, payload.target_schema, limit=100)
+        except (MappingError, ValueError) as exc:
+            raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
+
+    result = await db.execute(
+        select(func.max(DataSourceMappingTemplate.version)).where(
+            DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id,
+            DataSourceMappingTemplate.target_schema == payload.target_schema,
+        )
+    )
+    next_version = int(result.scalar() or 0) + 1
+
+    if payload.is_active:
+        await db.execute(
+            DataSourceMappingTemplate.__table__.update()
+            .where(DataSourceMappingTemplate.datasource_config_id == payload.datasource_config_id)
+            .values(is_active=False)
+        )
+
+    template = DataSourceMappingTemplate(
+        datasource_config_id=payload.datasource_config_id,
+        target_schema=payload.target_schema,
+        mapping_json=payload.mapping_json,
+        sample_payload_hash=payload.sample_payload_hash
+        or (stable_payload_hash(payload.sample_payload) if payload.sample_payload is not None else None),
+        validation_status=payload.validation_status,
+        version=next_version,
+        is_active=payload.is_active,
+    )
+    db.add(template)
+    await db.commit()
+    await db.refresh(template)
+    return {"message": "Mapping template saved successfully", "data": serialize_mapping_template(template)}
+
+
+@router.put("/mappings/{mapping_id}")
+async def update_datasource_mapping(
+    mapping_id: int,
+    payload: MappingTemplateUpdate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Update a mapping template in place."""
+    template = await db.get(DataSourceMappingTemplate, mapping_id)
+    if not template:
+        raise HTTPException(status_code=404, detail="Mapping template not found")
+
+    target_schema = payload.target_schema or template.target_schema
+    mapping_json = payload.mapping_json or template.mapping_json
+    get_target_schema(target_schema)
+    if payload.sample_payload is not None:
+        try:
+            execute_mapping(payload.sample_payload, mapping_json, target_schema, limit=100)
+        except (MappingError, ValueError) as exc:
+            raise HTTPException(status_code=400, detail=f"Mapping validation failed: {exc}") from exc
+
+    if payload.is_active is True:
+        await db.execute(
+            DataSourceMappingTemplate.__table__.update()
+            .where(DataSourceMappingTemplate.datasource_config_id == template.datasource_config_id)
+            .where(DataSourceMappingTemplate.id != template.id)
+            .values(is_active=False)
+        )
+
+    template.target_schema = target_schema
+    template.mapping_json = mapping_json
+    if payload.sample_payload_hash is not None:
+        template.sample_payload_hash = payload.sample_payload_hash
+    elif payload.sample_payload is not None:
+        template.sample_payload_hash = stable_payload_hash(payload.sample_payload)
+    if payload.validation_status is not None:
+        template.validation_status = payload.validation_status
+    if payload.is_active is not None:
+        template.is_active = payload.is_active
+
+    await db.commit()
+    await db.refresh(template)
+    return {"message": "Mapping template updated successfully", "data": serialize_mapping_template(template)}
+
+
+@router.post("/{config_id}/run-mapped")
+async def run_mapped_datasource(
+    config_id: int,
+    background: bool = Query(False, description="For WebSocket sources, start a background stream task."),
+    debug_max_messages: int | None = Query(None, ge=1),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Run a saved custom datasource through its active deterministic mapping."""
+    datasource = await db.get(DataSourceConfig, config_id)
+    if not datasource:
+        raise HTTPException(status_code=404, detail="Configuration not found")
+
+    try:
+        if str(datasource.source_type or "").lower() in {"websocket", "ws"}:
+            if background and debug_max_messages is None:
+                started = start_custom_stream(config_id)
+                if not started:
+                    raise HTTPException(status_code=409, detail="Custom WebSocket source is already running")
+                return {
+                    "status": "started",
+                    "datasource_config_id": config_id,
+                    "stream": get_custom_stream_status(config_id),
+                }
+            return await run_mapped_websocket_config(
+                db,
+                datasource,
+                debug_max_messages=debug_max_messages,
+            )
+
+        return await run_mapped_rest_config(db, datasource)
+    except httpx.HTTPStatusError as exc:
+        raise HTTPException(
+            status_code=exc.response.status_code,
+            detail=f"Datasource request failed: HTTP {exc.response.status_code}",
+        ) from exc
+    except httpx.HTTPError as exc:
+        raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
+    except (CustomDatasourceRuntimeError, MappingError, ValueError) as exc:
+        raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
+
+
+@router.post("/{config_id}/stop-mapped")
+async def stop_mapped_datasource(
+    config_id: int,
+    current_user: User = Depends(get_current_user),
+):
+    stopped = await stop_custom_stream(config_id)
+    return {
+        "status": "stopped" if stopped else "not_running",
+        "datasource_config_id": config_id,
+        "stream": get_custom_stream_status(config_id),
+    }
+
+
+@router.get("/{config_id}/stream-status")
+async def get_mapped_stream_status(
+    config_id: int,
+    current_user: User = Depends(get_current_user),
+):
+    return get_custom_stream_status(config_id)

backend/app/api/v1/datasources.py

@@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from app.core.time import to_iso8601_utc
 from app.core.security import get_current_user
 from app.core.data_sources import get_data_sources_config
+from app.core.datasource_defaults import DEFAULT_DATASOURCES
 from app.db.session import get_db
 from app.models.collected_data import CollectedData
 from app.models.data_snapshot import DataSnapshot
@@ -35,6 +36,17 @@ def format_frequency_label(minutes: int) -> str:
     return f"{minutes}m"
 
 
+def datasource_metadata(source: str) -> dict:
+    info = DEFAULT_DATASOURCES.get(source, {})
+    return {
+        "display_name": info.get("display_name") or info.get("name") or source,
+        "is_free": bool(info.get("is_free", True)),
+        "requires_credentials": bool(info.get("requires_credentials", False)),
+        "credential_provider": info.get("credential_provider"),
+        "credential_status": info.get("credential_status", "none"),
+    }
+
+
 def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
     if datasource.last_run_at is None:
         return True
@@ -72,31 +84,6 @@ async def _load_latest_running_tasks(
     return {task.datasource_id: task for task in result.scalars().all()}
 
 
-async def _load_latest_completed_tasks(
-    db: AsyncSession,
-    datasource_ids: list[int],
-) -> dict[int, CollectionTask]:
-    if not datasource_ids:
-        return {}
-
-    ranked_tasks = (
-        select(
-            CollectionTask.id.label("task_id"),
-            _task_rank_column(CollectionTask.completed_at),
-        )
-        .where(CollectionTask.datasource_id.in_(datasource_ids))
-        .where(CollectionTask.completed_at.isnot(None))
-        .where(CollectionTask.status.in_(("success", "failed", "cancelled")))
-        .subquery()
-    )
-    result = await db.execute(
-        select(CollectionTask)
-        .join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
-        .where(ranked_tasks.c.row_num == 1)
-    )
-    return {task.datasource_id: task for task in result.scalars().all()}
-
-
 async def _load_latest_task_ids(
     db: AsyncSession,
     datasource_ids: list[int],
@@ -123,21 +110,6 @@ async def _load_latest_task_ids(
     return {datasource_id: task_id for datasource_id, task_id in result.all()}
 
 
-async def _load_datasource_data_counts(
-    db: AsyncSession,
-    sources: list[str],
-) -> dict[str, int]:
-    if not sources:
-        return {}
-
-    result = await db.execute(
-        select(CollectedData.source, func.count(CollectedData.id))
-        .where(CollectedData.source.in_(sources))
-        .group_by(CollectedData.source)
-    )
-    return {source: count for source, count in result.all()}
-
-
 async def _load_datasource_endpoint_overrides(
     db: AsyncSession,
     sources: list[str],
@@ -161,7 +133,7 @@ async def _load_datasource_endpoint_overrides(
 async def _load_datasource_list_context(
     db: AsyncSession,
     datasources: list[DataSource],
-) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
+) -> tuple[dict[int, CollectionTask], dict[str, str]]:
     datasource_ids = [datasource.id for datasource in datasources]
     sources = [datasource.source for datasource in datasources]
 
@@ -185,10 +157,8 @@ async def _load_datasource_list_context(
     if stale_datasource_ids:
         running_tasks = await _load_latest_running_tasks(db, datasource_ids)
 
-    completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
-    data_counts = await _load_datasource_data_counts(db, sources)
     endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
-    return running_tasks, completed_tasks, data_counts, endpoint_overrides
+    return running_tasks, endpoint_overrides
 
 
 async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -401,27 +371,19 @@ async def list_datasources(
 
     collector_list = []
     config = get_data_sources_config()
-    running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
-        db,
-        datasources,
-    )
+    running_tasks, endpoint_overrides = await _load_datasource_list_context(db, datasources)
     for datasource in datasources:
         running_task = running_tasks.get(datasource.id)
-        last_task = completed_tasks.get(datasource.id)
-        endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
-            datasource.source,
-        )
-        data_count = data_counts.get(datasource.source, 0)
-
-        last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
-        last_run = to_iso8601_utc(last_run_at)
-        last_status = datasource.last_status or (last_task.status if last_task else None)
+        endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(datasource.source)
+        last_run_at = datasource.last_run_at
+        last_status = datasource.last_status
 
         collector_list.append(
             {
                 "id": datasource.id,
                 "source": datasource.source,
                 "name": datasource.name,
+                **datasource_metadata(datasource.source),
                 "module": datasource.module,
                 "priority": datasource.priority,
                 "frequency": format_frequency_label(datasource.frequency_minutes),
@@ -429,15 +391,18 @@ async def list_datasources(
                 "is_active": datasource.is_active,
                 "collector_class": datasource.collector_class,
                 "endpoint": endpoint,
-                "last_run": last_run,
+                "last_run": to_iso8601_utc(last_run_at),
                 "last_run_at": to_iso8601_utc(last_run_at),
                 "last_status": last_status,
-                "last_records_processed": last_task.records_processed if last_task else None,
-                "data_count": data_count,
                 "is_running": running_task is not None,
                 "task_id": running_task.id if running_task else None,
                 "progress": running_task.progress if running_task else None,
                 "phase": running_task.phase if running_task else None,
+                "phase_progress": running_task.phase_progress if running_task else None,
+                "phase_message": running_task.phase_message if running_task else None,
+                "phase_current": running_task.phase_current if running_task else None,
+                "phase_total": running_task.phase_total if running_task else None,
+                "phase_unit": running_task.phase_unit if running_task else None,
                 "records_processed": running_task.records_processed if running_task else None,
                 "total_records": running_task.total_records if running_task else None,
             }
@@ -576,6 +541,7 @@ async def get_datasource(
     return {
         "id": datasource.id,
         "name": datasource.name,
+        **datasource_metadata(datasource.source),
         "module": datasource.module,
         "priority": datasource.priority,
         "frequency": format_frequency_label(datasource.frequency_minutes),
@@ -665,6 +631,11 @@ async def trigger_datasource(
                 "message": "当前采集任务尚未完成，重新触发会丢失本次未完成进度。是否强制重新采集？",
                 "task_id": running_task.id,
                 "phase": running_task.phase,
+                "phase_progress": running_task.phase_progress,
+                "phase_message": running_task.phase_message,
+                "phase_current": running_task.phase_current,
+                "phase_total": running_task.phase_total,
+                "phase_unit": running_task.phase_unit,
                 "progress": running_task.progress,
                 "records_processed": running_task.records_processed,
                 "total_records": running_task.total_records,
@@ -748,13 +719,29 @@ async def get_task_status(
         task = await get_running_task(db, datasource.id)
 
     if not task:
-        return {"is_running": False, "task_id": None, "progress": None, "phase": None, "status": "idle"}
+        return {
+            "is_running": False,
+            "task_id": None,
+            "progress": None,
+            "phase": None,
+            "phase_progress": None,
+            "phase_message": None,
+            "phase_current": None,
+            "phase_total": None,
+            "phase_unit": None,
+            "status": "idle",
+        }
 
     return {
         "is_running": task.status == "running",
         "task_id": task.id,
         "progress": task.progress,
         "phase": task.phase,
+        "phase_progress": task.phase_progress,
+        "phase_message": task.phase_message,
+        "phase_current": task.phase_current,
+        "phase_total": task.phase_total,
+        "phase_unit": task.phase_unit,
         "records_processed": task.records_processed,
         "total_records": task.total_records,
         "status": task.status,

backend/app/api/v1/settings.py

@@ -9,10 +9,37 @@ from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.security import get_current_user
 from app.core.time import to_iso8601_utc
+from app.core.config import settings as app_settings
+from app.core.data_sources import get_data_sources_config
+from app.core.datasource_defaults import DEFAULT_DATASOURCES
 from app.db.session import get_db
 from app.models.datasource import DataSource
+from app.models.datasource_config import DataSourceConfig
 from app.models.system_setting import SystemSetting
 from app.models.user import User
+from app.models.vessel import AISSourceHealth
+from app.services.barentswatch import (
+    BarentsWatchConfig,
+    check_barentswatch_config,
+    check_barentswatch_connectivity,
+    get_barentswatch_datasource_record,
+    resolve_barentswatch_config,
+)
+from app.services.credential_guides import (
+    generate_credential_guide,
+    get_credential_guide,
+    reset_credential_guide,
+)
+from app.services.datasource_connectivity import (
+    build_builtin_connectivity_checksum,
+    save_connectivity_success,
+)
+from app.services.ai_client import AIProviderClient, get_ai_provider_client
+from app.services.llm_provider_catalog import (
+    get_fallback_llm_provider_preset,
+    list_fallback_llm_provider_presets,
+    refresh_llm_provider_preset,
+)
 from app.services.scheduler import sync_datasource_job
 from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
 
@@ -39,6 +66,21 @@ DEFAULT_SETTINGS = {
         "password_policy": "medium",
     },
     "tv": DEFAULT_TV_SETTINGS,
+    "external_integrations": {
+        "ai_provider": {
+            "service_url": "",
+            "service_token": "",
+            "provider": "minimax",
+            "provider_api": "anthropic-messages",
+            "base_url": "https://api.minimaxi.com/anthropic",
+            "model": "MiniMax-M2.7",
+            "api_key": "",
+            "max_tokens": 1200,
+            "anthropic_version": "2023-06-01",
+            "timeout_seconds": 60,
+            "retry_attempts": 2,
+        }
+    },
 }
 
 
@@ -96,6 +138,34 @@ class TVSettingsUpdate(BaseModel):
     sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
 
 
+class AIProviderIntegrationUpdate(BaseModel):
+    service_url: str = ""
+    service_token: Optional[str] = None
+    provider: str = Field(default="minimax", max_length=80)
+    provider_api: str = Field(default="anthropic-messages", max_length=80)
+    base_url: str = Field(default="", max_length=500)
+    model: str = Field(default="", max_length=200)
+    api_key: Optional[str] = None
+    max_tokens: int = Field(default=1200, ge=1, le=200000)
+    anthropic_version: str = Field(default="2023-06-01", max_length=40)
+    timeout_seconds: int = Field(default=60, ge=5, le=600)
+    retry_attempts: int = Field(default=2, ge=1, le=10)
+    clear_service_token: bool = False
+    clear_api_key: bool = False
+
+
+class BarentsWatchIntegrationUpdate(BaseModel):
+    endpoint: str = ""
+    client_id: str = ""
+    client_secret: Optional[str] = None
+    clear_client_secret: bool = False
+
+
+class ExternalIntegrationsUpdate(BaseModel):
+    ai_provider: AIProviderIntegrationUpdate
+    barentswatch: BarentsWatchIntegrationUpdate
+
+
 def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
     merged = deepcopy(DEFAULT_SETTINGS[category])
     if payload:
@@ -146,6 +216,151 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
     return merge_with_defaults(category, record.payload)
 
 
+def _mask_secret(value: Optional[str]) -> dict:
+    if not value:
+        return {"configured": False, "preview": ""}
+    text = str(value)
+    if "-" in text:
+        prefix = text.split("-", 1)[0] + "-"
+        preview = prefix + ("*" * max(len(text) - len(prefix), 1))
+    else:
+        prefix_len = min(4, len(text))
+        preview = text[:prefix_len] + ("*" * max(len(text) - prefix_len, 1))
+    return {"configured": True, "preview": preview}
+
+
+async def get_runtime_ai_provider_config(db: AsyncSession) -> dict:
+    runtime_record = await get_setting_record(db, "external_integrations")
+    payload = merge_with_defaults(
+        "external_integrations",
+        runtime_record.payload if runtime_record else None,
+    )
+    ai_payload = payload.get("ai_provider") or {}
+    has_runtime_llm_config = bool(
+        runtime_record
+        and isinstance(runtime_record.payload, dict)
+        and isinstance(runtime_record.payload.get("ai_provider"), dict)
+    )
+    return {
+        "service_url": ai_payload.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
+        "service_token": ai_payload.get("service_token") or app_settings.AI_PROVIDER_SERVICE_TOKEN,
+        "timeout_seconds": int(
+            ai_payload.get("timeout_seconds") or app_settings.AI_PROVIDER_TIMEOUT_SECONDS
+        ),
+        "retry_attempts": int(
+            ai_payload.get("retry_attempts") or app_settings.AI_PROVIDER_RETRY_ATTEMPTS
+        ),
+        "llm_config": {
+            "provider": ai_payload.get("provider") or "minimax",
+            "provider_api": ai_payload.get("provider_api") or "anthropic-messages",
+            "base_url": ai_payload.get("base_url") or "https://api.minimaxi.com/anthropic",
+            "model": ai_payload.get("model") or "MiniMax-M2.7",
+            "api_key": ai_payload.get("api_key") or "",
+            "max_tokens": int(ai_payload.get("max_tokens") or 1200),
+            "anthropic_version": ai_payload.get("anthropic_version") or "2023-06-01",
+        } if has_runtime_llm_config else {},
+    }
+
+
+async def get_barentswatch_config_record(db: AsyncSession) -> Optional[DataSourceConfig]:
+    return await get_barentswatch_datasource_record(db)
+
+
+async def serialize_external_integrations(db: AsyncSession) -> dict:
+    ai_config = await get_runtime_ai_provider_config(db)
+    runtime_setting = await get_setting_record(db, "external_integrations")
+    display_llm_config = ai_config["llm_config"] or DEFAULT_SETTINGS["external_integrations"]["ai_provider"]
+    barentswatch_record = await get_barentswatch_config_record(db)
+    barentswatch_auth = barentswatch_record.auth_config if barentswatch_record else {}
+    barentswatch_auth = barentswatch_auth or {}
+    resolved_barentswatch = await resolve_barentswatch_config(db)
+    return {
+        "ai_provider": {
+            "service_url": ai_config["service_url"],
+            "service_token": _mask_secret(ai_config["service_token"]),
+            "provider": display_llm_config.get("provider") or "minimax",
+            "provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
+            "base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
+            "model": display_llm_config.get("model") or "MiniMax-M2.7",
+            "api_key": _mask_secret(display_llm_config.get("api_key")),
+            "max_tokens": int(display_llm_config.get("max_tokens") or 1200),
+            "anthropic_version": display_llm_config.get("anthropic_version") or "2023-06-01",
+            "timeout_seconds": ai_config["timeout_seconds"],
+            "retry_attempts": ai_config["retry_attempts"],
+            "source": "runtime" if runtime_setting else "env",
+        },
+        "barentswatch": {
+            "endpoint": resolved_barentswatch.endpoint,
+            "client_id": barentswatch_auth.get("client_id") or resolved_barentswatch.client_id,
+            "client_secret": _mask_secret(
+                barentswatch_auth.get("client_secret") or resolved_barentswatch.client_secret
+            ),
+            "source": resolved_barentswatch.credential_source,
+        },
+    }
+
+
+async def save_external_integrations_payload(
+    db: AsyncSession,
+    update: ExternalIntegrationsUpdate,
+) -> dict:
+    current_payload = await get_setting_payload(db, "external_integrations")
+    current_ai = current_payload.get("ai_provider") or {}
+    ai_payload = {
+        "service_url": update.ai_provider.service_url.strip()
+        or app_settings.AI_PROVIDER_SERVICE_URL,
+        "service_token": current_ai.get("service_token") or "",
+        "provider": update.ai_provider.provider.strip() or "minimax",
+        "provider_api": update.ai_provider.provider_api.strip() or "anthropic-messages",
+        "base_url": update.ai_provider.base_url.strip(),
+        "model": update.ai_provider.model.strip(),
+        "api_key": current_ai.get("api_key") or "",
+        "max_tokens": update.ai_provider.max_tokens,
+        "anthropic_version": update.ai_provider.anthropic_version.strip() or "2023-06-01",
+        "timeout_seconds": update.ai_provider.timeout_seconds,
+        "retry_attempts": update.ai_provider.retry_attempts,
+    }
+    if update.ai_provider.clear_service_token:
+        ai_payload["service_token"] = ""
+    elif update.ai_provider.service_token not in (None, ""):
+        ai_payload["service_token"] = update.ai_provider.service_token
+    if update.ai_provider.clear_api_key:
+        ai_payload["api_key"] = ""
+    elif update.ai_provider.api_key not in (None, ""):
+        ai_payload["api_key"] = update.ai_provider.api_key
+
+    await save_setting_payload(db, "external_integrations", {"ai_provider": ai_payload})
+
+    default_endpoint = get_data_sources_config().get_yaml_url("barentswatch_vessels")
+    barentswatch_record = await get_barentswatch_config_record(db)
+    if barentswatch_record is None:
+        barentswatch_record = DataSourceConfig(
+            name="barentswatch_vessels",
+            description="BarentsWatch Live AIS credentials",
+            source_type="api",
+            endpoint=update.barentswatch.endpoint.strip() or default_endpoint,
+            auth_type="oauth_client",
+            auth_config={},
+            headers={},
+            config={},
+            is_active=True,
+        )
+        db.add(barentswatch_record)
+
+    current_auth = dict(barentswatch_record.auth_config or {})
+    if update.barentswatch.clear_client_secret:
+        current_auth.pop("client_secret", None)
+    elif update.barentswatch.client_secret not in (None, ""):
+        current_auth["client_secret"] = update.barentswatch.client_secret
+    current_auth["client_id"] = update.barentswatch.client_id.strip()
+    barentswatch_record.endpoint = update.barentswatch.endpoint.strip() or default_endpoint
+    barentswatch_record.auth_type = "oauth_client"
+    barentswatch_record.auth_config = current_auth
+    await db.commit()
+
+    return await serialize_external_integrations(db)
+
+
 def format_frequency_label(minutes: int) -> str:
     if minutes % 1440 == 0:
         return f"{minutes // 1440}d"
@@ -154,10 +369,17 @@ def format_frequency_label(minutes: int) -> str:
     return f"{minutes}m"
 
 
-def serialize_collector(datasource: DataSource) -> dict:
+async def get_ais_source_health_by_source(db: AsyncSession) -> dict[str, dict]:
+    result = await db.execute(select(AISSourceHealth))
+    return {item.source: item.to_dict() for item in result.scalars().all()}
+
+
+def serialize_collector(datasource: DataSource, ais_health_by_source: dict[str, dict] | None = None) -> dict:
+    defaults = DEFAULT_DATASOURCES.get(datasource.source, {})
     return {
         "id": datasource.id,
         "name": datasource.name,
+        "display_name": defaults.get("display_name") or datasource.name,
         "source": datasource.source,
         "module": datasource.module,
         "priority": datasource.priority,
@@ -167,6 +389,11 @@ def serialize_collector(datasource: DataSource) -> dict:
         "last_run_at": to_iso8601_utc(datasource.last_run_at),
         "last_status": datasource.last_status,
         "next_run_at": to_iso8601_utc(datasource.next_run_at),
+        "is_free": bool(defaults.get("is_free", True)),
+        "requires_credentials": bool(defaults.get("requires_credentials", False)),
+        "credential_provider": defaults.get("credential_provider"),
+        "credential_status": defaults.get("credential_status", "none"),
+        "ais_health": (ais_health_by_source or {}).get(datasource.source),
     }
 
 
@@ -243,6 +470,135 @@ async def update_tv_settings(
     return {"status": "updated", "tv": normalize_tv_settings(saved)}
 
 
+@router.get("/integrations")
+async def get_external_integrations(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"integrations": await serialize_external_integrations(db)}
+
+
+@router.get("/integrations/barentswatch/connectivity")
+async def get_barentswatch_connectivity(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await check_barentswatch_connectivity(db)
+
+
+@router.post("/integrations/barentswatch/connect")
+async def connect_barentswatch_integration(
+    payload: BarentsWatchIntegrationUpdate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    current = await resolve_barentswatch_config(db)
+    config = BarentsWatchConfig(
+        endpoint=payload.endpoint.strip() or current.endpoint,
+        client_id=payload.client_id.strip() or current.client_id,
+        client_secret=(
+            ""
+            if payload.clear_client_secret
+            else payload.client_secret or current.client_secret
+        ),
+        credential_source="draft",
+        endpoint_source="draft",
+    )
+    result = await check_barentswatch_config(config)
+    if result.get("success"):
+        checksum, _context = await build_builtin_connectivity_checksum(
+            "barentswatch_vessels",
+            config.endpoint,
+            "none",
+            {},
+            {},
+            db,
+            credential_override={
+                "client_id": config.client_id,
+                "client_secret": config.client_secret,
+            },
+        )
+        validation = await save_connectivity_success(
+            db,
+            "barentswatch_vessels",
+            checksum,
+            result,
+            connected_by="connection_button",
+        )
+        await db.commit()
+        return {**result, "connected": True, "validation": validation}
+    return {**result, "connected": False}
+
+
+@router.get("/credential-guides/{provider}")
+async def read_credential_guide(
+    provider: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        return {"guide": await get_credential_guide(db, provider)}
+    except ValueError as exc:
+        raise HTTPException(status_code=404, detail=str(exc)) from exc
+
+
+@router.post("/credential-guides/{provider}/generate")
+async def generate_provider_credential_guide(
+    provider: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+    ai_client: AIProviderClient = Depends(get_ai_provider_client),
+):
+    try:
+        return {"guide": await generate_credential_guide(db, provider, ai_client)}
+    except ValueError as exc:
+        raise HTTPException(status_code=404, detail=str(exc)) from exc
+
+
+@router.post("/credential-guides/{provider}/reset")
+async def reset_provider_credential_guide(
+    provider: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        return {"guide": await reset_credential_guide(db, provider)}
+    except ValueError as exc:
+        raise HTTPException(status_code=404, detail=str(exc)) from exc
+
+
+@router.get("/integrations/ai-provider/presets")
+async def get_ai_provider_presets(
+    current_user: User = Depends(get_current_user),
+):
+    return {"data": list_fallback_llm_provider_presets()}
+
+
+@router.post("/integrations/ai-provider/presets/{provider}/refresh")
+async def refresh_ai_provider_preset(
+    provider: str,
+    current_user: User = Depends(get_current_user),
+):
+    try:
+        return {"data": await refresh_llm_provider_preset(provider)}
+    except ValueError as exc:
+        raise HTTPException(status_code=404, detail=str(exc)) from exc
+    except Exception as exc:
+        fallback = get_fallback_llm_provider_preset(provider)
+        fallback["refresh_error"] = str(exc)
+        return {"data": fallback}
+
+
+@router.put("/integrations")
+async def update_external_integrations(
+    payload: ExternalIntegrationsUpdate,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    saved = await save_external_integrations_payload(db, payload)
+    return {"status": "updated", "integrations": saved}
+
+
 @router.get("/collectors")
 async def get_collector_settings(
     current_user: User = Depends(get_current_user),
@@ -250,7 +606,8 @@ async def get_collector_settings(
 ):
     result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
     datasources = result.scalars().all()
-    return {"collectors": [serialize_collector(datasource) for datasource in datasources]}
+    ais_health_by_source = await get_ais_source_health_by_source(db)
+    return {"collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources]}
 
 
 @router.put("/collectors/{datasource_id}")
@@ -270,7 +627,8 @@ async def update_collector_settings(
     await db.commit()
     await db.refresh(datasource)
     await sync_datasource_job(datasource.id)
-    return {"status": "updated", "collector": serialize_collector(datasource)}
+    ais_health_by_source = await get_ais_source_health_by_source(db)
+    return {"status": "updated", "collector": serialize_collector(datasource, ais_health_by_source)}
 
 
 @router.get("")
@@ -284,11 +642,13 @@ async def get_all_settings(
         db,
         ["system", "notifications", "security"],
     )
+    ais_health_by_source = await get_ais_source_health_by_source(db)
     return {
         "system": setting_payloads["system"],
         "notifications": setting_payloads["notifications"],
         "security": setting_payloads["security"],
         "tv": await get_tv_settings_payload(db),
-        "collectors": [serialize_collector(datasource) for datasource in datasources],
+        "integrations": await serialize_external_integrations(db),
+        "collectors": [serialize_collector(datasource, ais_health_by_source) for datasource in datasources],
         "generated_at": to_iso8601_utc(datetime.now(UTC)),
     }

backend/app/api/v1/system_control.py

@@ -4,12 +4,15 @@ import os
 import subprocess
 import sys
 
-from fastapi import APIRouter, Depends, HTTPException, status
+from datetime import datetime
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
 from pydantic import BaseModel
 
 from app.core.config import ROOT_DIR
 from app.core.security import get_current_user
 from app.models.user import User
+from app.services.persistent_logs import record_audit_log, record_system_log
 from app.services.system_control import (
     build_task_id,
     clear_active_task_id,
@@ -23,6 +26,15 @@ from app.services.system_control import (
     set_active_task_id,
     upsert_task_state,
 )
+from app.services.system_logs import (
+    DEFAULT_LOG_LINE_LIMIT,
+    MAX_LOG_LINE_LIMIT,
+    SUPPORTED_LOG_LEVELS,
+    append_buffer_log,
+    list_log_sources,
+    normalize_log_level,
+    read_log_snapshot,
+)
 
 router = APIRouter()
 
@@ -47,6 +59,59 @@ class RestartTaskLogsResponse(BaseModel):
     lines: list[str]
 
 
+class SystemLogSourceSummary(BaseModel):
+    source_id: str
+    name: str
+    kind: str
+    location: str
+    description: str
+    category: str
+    status: str
+
+
+class SystemLogSourcesResponse(BaseModel):
+    items: list[SystemLogSourceSummary]
+
+
+class SystemLogDailyMarker(BaseModel):
+    date_token: str
+    total: int
+    dominant_level: str
+
+
+class SystemLogSnapshotResponse(BaseModel):
+    source_id: str
+    name: str
+    kind: str
+    location: str
+    description: str
+    category: str
+    status: str
+    level: str
+    selected_levels: list[str] = []
+    search_query: str = ""
+    available_levels: list[str]
+    daily_markers: list[SystemLogDailyMarker] = []
+    line_limit: int
+    line_count: int
+    lines: list[str]
+
+
+class EarthClientLogEventCreate(BaseModel):
+    level: str = "error"
+    message: str
+    category: str | None = None
+    url: str | None = None
+    module: str | None = None
+    detail: str | None = None
+
+
+class EarthClientLogEventResponse(BaseModel):
+    accepted: bool
+    source_id: str
+    level: str
+
+
 def ensure_super_admin(current_user: User) -> None:
     if not require_super_admin(current_user.role):
         raise HTTPException(
@@ -55,9 +120,22 @@ def ensure_super_admin(current_user: User) -> None:
         )
 
 
+def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
+    if raw_value in {None, ""}:
+        return None
+    try:
+        return datetime.strptime(raw_value, "%Y-%m-%d").date().isoformat()
+    except ValueError as exc:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"{field_name} must be in YYYY-MM-DD format",
+        ) from exc
+
+
 @router.post("/restart-tasks", response_model=RestartTaskResponse)
 async def create_restart_task(
     payload: RestartTaskCreate,
+    request: Request,
     current_user: User = Depends(get_current_user),
 ):
     ensure_super_admin(current_user)
@@ -133,11 +211,31 @@ async def create_restart_task(
             requested_by=requested_by,
         )
         clear_active_task_id(task_id)
+        await record_audit_log(
+            action="system.restart_task.requested",
+            actor_id=current_user.id,
+            actor_name=current_user.username,
+            target_type="restart_task",
+            target_id=task_id,
+            result="failed",
+            ip=request.client.host if request.client else None,
+            details={"action": payload.action, "message": task_state["message"]},
+        )
         raise HTTPException(
             status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
             detail=task_state["message"],
         ) from exc
 
+    await record_audit_log(
+        action="system.restart_task.requested",
+        actor_id=current_user.id,
+        actor_name=current_user.username,
+        target_type="restart_task",
+        target_id=task_id,
+        result="accepted",
+        ip=request.client.host if request.client else None,
+        details={"action": payload.action},
+    )
     return task_state
 
 
@@ -165,3 +263,92 @@ async def get_restart_task_logs(
     if task is None:
         raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Restart task not found")
     return {"task_id": task_id, "lines": get_task_logs(task_id)}
+
+
+@router.get("/logs/sources", response_model=SystemLogSourcesResponse)
+async def get_system_log_sources(
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+    return {"items": list_log_sources()}
+
+
+@router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
+async def get_system_log_snapshot(
+    source_id: str,
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}",
+        )
+    if str(level).strip().lower() not in SUPPORTED_LOG_LEVELS and normalize_log_level(level) == "all" and str(level).strip().lower() not in {"", "all"}:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
+    if levels:
+        for raw_level in str(levels).split(","):
+            normalized_level = str(raw_level).strip().lower()
+            if not normalized_level:
+                continue
+            if normalized_level not in SUPPORTED_LOG_LEVELS and normalize_log_level(normalized_level) == "all":
+                raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Unsupported log level")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
+
+    snapshot = read_log_snapshot(
+        source_id,
+        limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+    )
+    if snapshot is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
+    return snapshot
+
+
+@router.post("/logs/earth-client", response_model=EarthClientLogEventResponse)
+async def ingest_earth_client_log(
+    payload: EarthClientLogEventCreate,
+    request: Request,
+):
+    normalized_level = normalize_log_level(payload.level)
+    append_buffer_log(
+        "earth-client",
+        level=normalized_level,
+        message=payload.message,
+        context={
+            "category": payload.category or "",
+            "url": payload.url or "",
+            "module": payload.module or "",
+            "detail": payload.detail or "",
+        },
+    )
+    await record_system_log(
+        source="earth-client",
+        service="earth",
+        module=payload.module or "earth-client",
+        event="earth.client.runtime_log",
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or "client-runtime",
+        context={
+            "url": payload.url or "",
+            "detail": payload.detail or "",
+            "module": payload.module or "",
+            "client_ip": request.client.host if request.client else "",
+        },
+    )
+    return {"accepted": True, "source_id": "earth-client", "level": normalized_level}

backend/app/api/v1/tasks.py

@@ -27,7 +27,9 @@ async def list_tasks(
     offset = (page - 1) * page_size
     query = """
         SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
-               ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
+               ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
+               ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
+               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
         FROM collection_tasks ct
         JOIN data_sources ds ON ct.datasource_id = ds.id
         WHERE 1=1
@@ -66,6 +68,14 @@ async def list_tasks(
                 "completed_at": to_iso8601_utc(t[5]),
                 "records_processed": t[6],
                 "error_message": t[7],
+                "phase": t[8],
+                "phase_progress": t[9],
+                "phase_message": t[10],
+                "phase_current": t[11],
+                "phase_total": t[12],
+                "phase_unit": t[13],
+                "total_records": t[14],
+                "progress": t[15],
             }
             for t in tasks
         ],
@@ -81,7 +91,9 @@ async def get_task(
     result = await db.execute(
         text("""
             SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
-                   ct.started_at, ct.completed_at, ct.records_processed, ct.error_message
+                   ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
+                   ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
+                   ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
             FROM collection_tasks ct
             JOIN data_sources ds ON ct.datasource_id = ds.id
             WHERE ct.id = :id
@@ -105,6 +117,14 @@ async def get_task(
         "completed_at": to_iso8601_utc(task[5]),
         "records_processed": task[6],
         "error_message": task[7],
+        "phase": task[8],
+        "phase_progress": task[9],
+        "phase_message": task[10],
+        "phase_current": task[11],
+        "phase_total": task[12],
+        "phase_unit": task[13],
+        "total_records": task[14],
+        "progress": task[15],
     }
 
 

backend/app/api/v1/vessel_aggregation.py

@@ -0,0 +1,132 @@
+"""v4 strategy + v5 conflict-promotion + enrichment APIs for vessel_ais."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.user import User
+from app.models.vessel import AISConflictRecord
+from app.services.vessel_aggregation_strategy import (
+    StrategyValidationError,
+    load_strategy,
+    reset_strategy,
+    save_strategy,
+)
+from app.services.vessel_enrichment import (
+    get_vessel_enrichment_bundle,
+    upsert_vessel_media_enrichment,
+    upsert_vessel_profile_enrichment,
+)
+
+router = APIRouter()
+
+
+@router.get("/strategy")
+async def get_aggregation_strategy(db: AsyncSession = Depends(get_db)):
+    return await load_strategy(db)
+
+
+@router.put("/strategy")
+async def put_aggregation_strategy(
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        return await save_strategy(db, payload)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.delete("/strategy")
+async def reset_aggregation_strategy(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await reset_strategy(db)
+
+
+@router.post("/conflicts/{mmsi}/{field}/promote-to-rule")
+async def promote_conflict_to_rule(
+    mmsi: int,
+    field: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Lift the current conflict resolution into a persistent strategy rule."""
+
+    result = await db.execute(
+        select(AISConflictRecord)
+        .where(AISConflictRecord.target_schema == "vessel_ais")
+        .where(AISConflictRecord.entity_key == str(mmsi))
+        .where(AISConflictRecord.field == field)
+        .order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
+        .limit(1)
+    )
+    record = result.scalar_one_or_none()
+    if record is None or not record.selected_source:
+        raise HTTPException(status_code=404, detail="Conflict record with selected_source not found")
+
+    strategy = await load_strategy(db)
+    vessel_ais = dict(strategy.get("vessel_ais") or {})
+    field_rules = dict(vessel_ais.get("field_rules") or {})
+    field_rules[field] = {"mode": "source_priority", "source_priority": [record.selected_source]}
+    vessel_ais["field_rules"] = field_rules
+
+    incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
+    try:
+        return await save_strategy(db, incoming)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.delete("/conflicts/{mmsi}/{field}/promote-to-rule")
+async def revert_conflict_rule(
+    mmsi: int,
+    field: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    strategy = await load_strategy(db)
+    vessel_ais = dict(strategy.get("vessel_ais") or {})
+    field_rules = dict(vessel_ais.get("field_rules") or {})
+    if field in field_rules:
+        del field_rules[field]
+    vessel_ais["field_rules"] = field_rules
+
+    incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
+    try:
+        return await save_strategy(db, incoming)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.get("/enrichment/{mmsi}")
+async def get_vessel_enrichment(mmsi: int, db: AsyncSession = Depends(get_db)):
+    return await get_vessel_enrichment_bundle(db, mmsi)
+
+
+@router.put("/enrichment/{mmsi}/profile")
+async def put_vessel_profile_enrichment(
+    mmsi: int,
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await upsert_vessel_profile_enrichment(db, mmsi=mmsi, payload=payload)
+
+
+@router.put("/enrichment/{mmsi}/media")
+async def put_vessel_media_enrichment(
+    mmsi: int,
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await upsert_vessel_media_enrichment(db, mmsi=mmsi, payload=payload)

backend/app/api/v1/visualization.py

@@ -4,8 +4,9 @@ Unified API for all visualization data sources.
 Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
 """
 
-from datetime import UTC, datetime
+from datetime import UTC, datetime, timedelta
 import math
+import re
 import httpx
 from fastapi import APIRouter, HTTPException, Depends, Query, Response
 from sqlalchemy.ext.asyncio import AsyncSession
@@ -19,15 +20,30 @@ from app.core.time import to_iso8601_utc
 from app.db.session import get_db
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
+from app.models.bgp_observation import BGPObservation
 from app.models.collected_data import CollectedData
+from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
 from app.services.bgp_collectors import build_bgp_collector_coverage
 from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
 from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
+from app.services.persistent_logs import record_system_log
+from app.services.vessel_ais_aggregation import (
+    build_field_conflict_candidates,
+    count_unique_raw_vessel_mmsi,
+    get_aggregated_vessel,
+    get_aggregated_vessel_track,
+    get_aggregated_vessels,
+    get_vessel_conflict_records,
+    get_vessel_raw_observations,
+)
+from app.core.logging import get_logger
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 TERRAIN_TILE_URL_TEMPLATE = (
     "https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
 )
+VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
 
 
 # ============== Converter Functions ==============
@@ -181,6 +197,12 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
                 mean_motion=metadata.get("mean_motion"),
             )
 
+        constellation_group = _normalize_satellite_constellation_group(
+            metadata.get("constellation_group"),
+            record.name,
+        )
+        footprint_policy = _get_satellite_footprint_policy(constellation_group)
+
         features.append(
             {
                 "type": "Feature",
@@ -190,6 +212,8 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
                     "id": record.id,
                     "norad_cat_id": norad_id,
                     "name": record.name,
+                    "constellation_group": constellation_group,
+                    "footprint_policy": footprint_policy,
                     "international_designator": metadata.get("international_designator"),
                     "epoch": metadata.get("epoch"),
                     "inclination": metadata.get("inclination"),
@@ -210,6 +234,31 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
     return {"type": "FeatureCollection", "features": features}
 
 
+def _normalize_satellite_constellation_group(
+    raw_group: Any,
+    name: Optional[str],
+) -> Optional[str]:
+    normalized_group = str(raw_group or "").strip().lower()
+    if normalized_group:
+        return normalized_group
+
+    normalized_name = str(name or "").strip().upper()
+    if normalized_name.startswith("STARLINK"):
+        return "starlink"
+    if normalized_name.startswith("IRIDIUM"):
+        return "iridium-next"
+
+    return None
+
+
+def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
+    if constellation_group == "starlink":
+        return "starlink_ground_footprint"
+    if constellation_group == "iridium-next":
+        return "iridium_coverage_ring"
+    return "none"
+
+
 def _current_collected_data_stmt(source: str):
     return (
         select(CollectedData)
@@ -236,6 +285,120 @@ async def _load_current_collected_data(
     return list(result.scalars().all())
 
 
+async def _latest_task_id_for_source(
+    db: AsyncSession,
+    source: str,
+    *,
+    exclude_unknown_name: bool = False,
+) -> int | None:
+    stmt = (
+        select(
+            CollectedData.task_id,
+            func.max(CollectedData.collected_at).label("latest_collected_at"),
+            func.max(CollectedData.id).label("latest_id"),
+        )
+        .where(CollectedData.source == source)
+        .where(CollectedData.task_id.isnot(None))
+        .group_by(CollectedData.task_id)
+        .order_by(func.max(CollectedData.collected_at).desc(), func.max(CollectedData.id).desc())
+        .limit(1)
+    )
+    if exclude_unknown_name:
+        stmt = stmt.where(CollectedData.name != "Unknown")
+
+    result = await db.execute(stmt)
+    row = result.first()
+    return int(row.task_id) if row and row.task_id is not None else None
+
+
+async def _load_current_or_latest_task_data(
+    db: AsyncSession,
+    source: str,
+    *,
+    exclude_unknown_name: bool = False,
+    limit: Optional[int] = None,
+) -> List[CollectedData]:
+    records = await _load_current_collected_data(
+        db,
+        source,
+        exclude_unknown_name=exclude_unknown_name,
+        limit=limit,
+    )
+    if records:
+        return records
+
+    latest_task_id = await _latest_task_id_for_source(
+        db,
+        source,
+        exclude_unknown_name=exclude_unknown_name,
+    )
+    if latest_task_id is None:
+        return []
+
+    stmt = (
+        select(CollectedData)
+        .where(CollectedData.source == source)
+        .where(CollectedData.task_id == latest_task_id)
+        .order_by(CollectedData.id.desc())
+    )
+    if exclude_unknown_name:
+        stmt = stmt.where(CollectedData.name != "Unknown")
+    if limit is not None:
+        stmt = stmt.limit(limit)
+
+    result = await db.execute(stmt)
+    return list(result.scalars().all())
+
+
+async def _count_current_or_latest_task_data(
+    db: AsyncSession,
+    source: str,
+    *,
+    exclude_unknown_name: bool = False,
+) -> int:
+    current_stmt = (
+        select(func.count(CollectedData.id))
+        .where(CollectedData.source == source)
+        .where(CollectedData.is_current.is_(True))
+    )
+    if exclude_unknown_name:
+        current_stmt = current_stmt.where(CollectedData.name != "Unknown")
+
+    current_result = await db.execute(current_stmt)
+    current_scalar = current_result.scalar()
+    if current_scalar is None and hasattr(current_result, "scalars"):
+        current_rows = current_result.scalars().all()
+        current_count = sum(
+            1
+            for row in current_rows
+            if getattr(row, "source", None) == source
+            and (not exclude_unknown_name or getattr(row, "name", None) != "Unknown")
+        )
+    else:
+        current_count = int(current_scalar or 0)
+    if current_count > 0:
+        return current_count
+
+    latest_task_id = await _latest_task_id_for_source(
+        db,
+        source,
+        exclude_unknown_name=exclude_unknown_name,
+    )
+    if latest_task_id is None:
+        return 0
+
+    latest_stmt = (
+        select(func.count(CollectedData.id))
+        .where(CollectedData.source == source)
+        .where(CollectedData.task_id == latest_task_id)
+    )
+    if exclude_unknown_name:
+        latest_stmt = latest_stmt.where(CollectedData.name != "Unknown")
+
+    latest_result = await db.execute(latest_stmt)
+    return int(latest_result.scalar() or 0)
+
+
 async def _load_current_collected_data_by_sources(
     db: AsyncSession,
     sources: List[str],
@@ -475,7 +638,7 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
     if unit in {"pflop/s", "pflops", "pflop"}:
         normalized_tflops = capacity_value * 1000
     elif unit in {"gflop/s", "gflops", "gflop"}:
-        normalized_tflops = capacity_value / 1000
+        normalized_tflops = capacity_value
     else:
         normalized_tflops = capacity_value
 
@@ -573,6 +736,267 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
     return {"type": "FeatureCollection", "features": features}
 
 
+VESSEL_TYPE_FILTERS = {
+    "cargo": lambda props: str(props.get("vessel_type_name", "")).lower() == "cargo"
+    or 70 <= int(props.get("vessel_type") or -1) <= 79,
+    "tanker": lambda props: str(props.get("vessel_type_name", "")).lower() == "tanker"
+    or 80 <= int(props.get("vessel_type") or -1) <= 89,
+    "passenger": lambda props: str(props.get("vessel_type_name", "")).lower() == "passenger"
+    or 60 <= int(props.get("vessel_type") or -1) <= 69,
+    "fishing": lambda props: str(props.get("vessel_type_name", "")).lower() == "fishing"
+    or int(props.get("vessel_type") or -1) == 30,
+    "military": lambda props: str(props.get("vessel_type_name", "")).lower() == "military"
+    or int(props.get("vessel_type") or -1) == 35,
+    "other": lambda props: str(props.get("vessel_type_name", "")).lower()
+    not in {"cargo", "tanker", "passenger", "fishing", "military"},
+}
+
+
+def convert_vessels_to_geojson(rows: List[Any]) -> Dict[str, Any]:
+    features = []
+    seen_mmsi: set[int] = set()
+    for position, static in rows:
+        if position.lat is None or position.lon is None:
+            continue
+        if position.mmsi in seen_mmsi:
+            continue
+        seen_mmsi.add(position.mmsi)
+        props = {
+            "mmsi": position.mmsi,
+            "mmsi_display": str(position.mmsi),
+            "name": getattr(static, "name", None) or f"MMSI {position.mmsi}",
+            "name_is_fallback": _is_vessel_name_fallback(getattr(static, "name", None), position.mmsi),
+            "callsign": getattr(static, "callsign", None),
+            "imo": getattr(static, "imo", None),
+            "imo_display": str(getattr(static, "imo")) if getattr(static, "imo", None) else None,
+            "vessel_type": getattr(static, "vessel_type", None),
+            "vessel_type_name": getattr(static, "vessel_type_name", None) or "Other",
+            "flag": getattr(static, "flag", None),
+            "length": getattr(static, "length", None),
+            "width": getattr(static, "width", None),
+            "draught": getattr(static, "draught", None),
+            "sog": position.sog,
+            "cog": position.cog,
+            "heading": position.heading,
+            "nav_status": position.nav_status,
+            "received_at": to_iso8601_utc(position.received_at),
+            "data_type": "vessel",
+        }
+        features.append(
+            {
+                "type": "Feature",
+                "id": position.mmsi,
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [position.lon, position.lat],
+                },
+                "properties": props,
+            }
+        )
+
+    return {"type": "FeatureCollection", "features": features}
+
+
+def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict[str, Any]:
+    features = []
+    for vessel in vessels:
+        if vessel.get("lat") is None or vessel.get("lon") is None:
+            continue
+        source_summary = {}
+        for source, summary in (vessel.get("source_summary") or {}).items():
+            source_summary[source] = {
+                **summary,
+                "latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
+            }
+        props = {
+            "mmsi": vessel["mmsi"],
+            "mmsi_display": str(vessel["mmsi"]),
+            "name": vessel.get("name") or f"MMSI {vessel['mmsi']}",
+            "name_is_fallback": _is_vessel_name_fallback(vessel.get("name"), vessel["mmsi"]),
+            "callsign": vessel.get("callsign"),
+            "imo": vessel.get("imo"),
+            "imo_display": str(vessel.get("imo")) if vessel.get("imo") else None,
+            "vessel_type": vessel.get("vessel_type"),
+            "vessel_type_name": vessel.get("vessel_type_name") or "Other",
+            "flag": vessel.get("flag"),
+            "length": vessel.get("length"),
+            "width": vessel.get("width"),
+            "draught": vessel.get("draught"),
+            "sog": vessel.get("sog"),
+            "cog": vessel.get("cog"),
+            "heading": vessel.get("heading"),
+            "nav_status": vessel.get("nav_status"),
+            "received_at": to_iso8601_utc(vessel.get("received_at")),
+            "field_sources": vessel.get("field_sources") or {},
+            "selected_reasons": vessel.get("selected_reasons") or {},
+            "source_summary": source_summary,
+            "quality_flags": vessel.get("quality_flags") or [],
+            "conflict_count": vessel.get("conflict_count", 0),
+            "aggregation_strategy_version": vessel.get("aggregation_strategy_version", 0),
+            "data_type": "vessel",
+        }
+        features.append(
+            {
+                "type": "Feature",
+                "id": vessel["mmsi"],
+                "geometry": {
+                    "type": "Point",
+                    "coordinates": [vessel["lon"], vessel["lat"]],
+                },
+                "properties": props,
+            }
+        )
+    return {"type": "FeatureCollection", "features": features}
+
+
+def _parse_bbox(value: Optional[str]) -> tuple[float, float, float, float] | None:
+    if not value:
+        return None
+    parts = [part.strip() for part in value.split(",")]
+    if len(parts) != 4:
+        raise HTTPException(status_code=400, detail="bbox must be lon_min,lat_min,lon_max,lat_max")
+    try:
+        lon_min, lat_min, lon_max, lat_max = [float(part) for part in parts]
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail="bbox values must be numbers") from exc
+    if lat_min > lat_max:
+        lat_min, lat_max = lat_max, lat_min
+    if lon_min > lon_max:
+        lon_min, lon_max = lon_max, lon_min
+    return lon_min, lat_min, lon_max, lat_max
+
+
+def _is_vessel_name_fallback(name: Any, mmsi: Any) -> bool:
+    text = str(name or "").strip()
+    mmsi_text = str(mmsi or "").strip()
+    if not text:
+        return True
+    if mmsi_text and text == mmsi_text:
+        return True
+    return bool(VESSEL_NAME_FALLBACK_PATTERN.match(text))
+
+
+def _requested_vessel_types(value: Optional[str]) -> set[str]:
+    return {
+        item.strip().lower()
+        for item in (value or "").split(",")
+        if item.strip()
+    }
+
+
+def _matches_vessel_type(props: dict[str, Any], requested_types: set[str]) -> bool:
+    if not requested_types:
+        return True
+    for requested_type in requested_types:
+        predicate = VESSEL_TYPE_FILTERS.get(requested_type)
+        if predicate and predicate(props):
+            return True
+    return False
+
+
+def _feature_mmsi_key(feature: dict[str, Any]) -> str | None:
+    props = feature.get("properties", {})
+    mmsi = props.get("mmsi") or feature.get("id")
+    if mmsi in (None, ""):
+        return None
+    return str(mmsi)
+
+
+def _feature_in_bbox(feature: dict[str, Any], bbox: tuple[float, float, float, float] | None) -> bool:
+    if bbox is None:
+        return True
+    coordinates = feature.get("geometry", {}).get("coordinates") or []
+    if len(coordinates) < 2:
+        return False
+    try:
+        lon = float(coordinates[0])
+        lat = float(coordinates[1])
+    except (TypeError, ValueError):
+        return False
+    lon_min, lat_min, lon_max, lat_max = bbox
+    return lon_min <= lon <= lon_max and lat_min <= lat <= lat_max
+
+
+def _filter_vessel_features(
+    features: list[dict[str, Any]],
+    *,
+    bbox: tuple[float, float, float, float] | None,
+    requested_types: set[str],
+) -> list[dict[str, Any]]:
+    return [
+        feature
+        for feature in features
+        if _feature_in_bbox(feature, bbox)
+        and _matches_vessel_type(feature.get("properties", {}), requested_types)
+    ]
+
+
+def _merge_vessel_features(
+    raw_features: list[dict[str, Any]],
+    legacy_features: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+    """Prefer aggregated raw observations as the canonical source of truth.
+
+    Legacy `vessel_position` rows only fill MMSIs that the unified pipeline does
+    not yet know about, so a vessel never appears twice when both BarentsWatch
+    and AISStream observe it. Once the legacy table drains, this branch becomes
+    a no-op.
+    """
+
+    merged: list[dict[str, Any]] = []
+    seen: set[str] = set()
+    raw_keys: set[str] = set()
+    legacy_keys: set[str] = set()
+
+    for feature in raw_features:
+        key = _feature_mmsi_key(feature)
+        if key is None or key in seen:
+            continue
+        seen.add(key)
+        raw_keys.add(key)
+        merged.append(feature)
+
+    legacy_added = 0
+    for feature in legacy_features:
+        key = _feature_mmsi_key(feature)
+        if key is None:
+            continue
+        legacy_keys.add(key)
+        if key in seen:
+            continue
+        seen.add(key)
+        legacy_added += 1
+        merged.append(feature)
+
+    return merged, {
+        "raw_unique_mmsi": len(raw_keys),
+        "legacy_unique_mmsi": len(legacy_keys),
+        "legacy_backfilled_mmsi": legacy_added,
+        "final_unique_mmsi": len(seen),
+    }
+
+
+def _build_vessel_stats(features: List[dict[str, Any]]) -> dict[str, Any]:
+    by_type: dict[str, int] = {}
+    underway = 0
+    anchored_or_moored = 0
+    for feature in features:
+        props = feature.get("properties", {})
+        vessel_type = str(props.get("vessel_type_name") or "Other")
+        by_type[vessel_type] = by_type.get(vessel_type, 0) + 1
+        nav_status = props.get("nav_status")
+        if nav_status in (1, 5):
+            anchored_or_moored += 1
+        else:
+            underway += 1
+    return {
+        "total": len(features),
+        "by_type": by_type,
+        "underway": underway,
+        "anchored_or_moored": anchored_or_moored,
+    }
+
+
 def convert_bgp_anomalies_to_geojson(
     records: List[BGPAnomaly],
     geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
@@ -990,6 +1414,21 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
     except HTTPException:
         raise
     except Exception as e:
+        logger.exception_event(
+            "Failed to build cables GeoJSON response",
+            event="visualization.cables.load_failed",
+            context={"error": str(e)},
+        )
+        await record_system_log(
+            source="backend",
+            service="api",
+            module=__name__,
+            event="visualization.cables.load_failed",
+            level="error",
+            message="Failed to build cables GeoJSON response",
+            category="visualization",
+            context={"error": str(e)},
+        )
         raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
 
 
@@ -1026,6 +1465,21 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
     except HTTPException:
         raise
     except Exception as e:
+        logger.exception_event(
+            "Failed to build landing points GeoJSON response",
+            event="visualization.landing_points.load_failed",
+            context={"error": str(e)},
+        )
+        await record_system_log(
+            source="backend",
+            service="api",
+            module=__name__,
+            event="visualization.landing_points.load_failed",
+            level="error",
+            message="Failed to build landing points GeoJSON response",
+            category="visualization",
+            context={"error": str(e)},
+        )
         raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
 
 
@@ -1122,7 +1576,7 @@ async def get_satellites_geojson(
     db: AsyncSession = Depends(get_db),
 ):
     """获取卫星 TLE GeoJSON 数据"""
-    records = await _load_current_collected_data(
+    records = await _load_current_or_latest_task_data(
         db,
         "celestrak_tle",
         exclude_unknown_name=True,
@@ -1232,6 +1686,304 @@ async def get_compute_centers_geojson(
     }
 
 
+@router.get("/geo/vessels")
+async def get_vessels_geojson(
+    bbox: Optional[str] = Query(
+        None,
+        description="Viewport bbox as lon_min,lat_min,lon_max,lat_max",
+    ),
+    type: Optional[str] = Query(
+        None,
+        description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
+    ),
+    limit: Optional[int] = Query(
+        None,
+        ge=0,
+        description="Maximum vessel features to return. Omit or pass 0 for no limit.",
+    ),
+    db: AsyncSession = Depends(get_db),
+):
+    """Return latest vessel positions as GeoJSON points."""
+    parsed_bbox = _parse_bbox(bbox)
+    requested_types = _requested_vessel_types(type)
+    merged_features, diagnostics = await _load_merged_vessel_features(db)
+    features = _filter_vessel_features(
+        merged_features,
+        bbox=parsed_bbox,
+        requested_types=requested_types,
+    )
+    if limit and limit > 0:
+        features = features[:limit]
+    return {
+        "type": "FeatureCollection",
+        "features": features,
+        "count": len(features),
+        "stats": _build_vessel_stats(features),
+        "diagnostics": {
+            **diagnostics,
+            "filtered_count": len(features),
+        },
+    }
+
+
+async def _load_merged_vessel_features(db: AsyncSession) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+    aggregated_vessels = await get_aggregated_vessels(db)
+    raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
+
+    latest_times = (
+        select(
+            VesselPosition.mmsi.label("mmsi"),
+            func.max(VesselPosition.received_at).label("received_at"),
+        )
+        .group_by(VesselPosition.mmsi)
+        .subquery()
+    )
+    stmt = (
+        select(VesselPosition, VesselStatic)
+        .join(
+            latest_times,
+            (VesselPosition.mmsi == latest_times.c.mmsi)
+            & (VesselPosition.received_at == latest_times.c.received_at),
+        )
+        .outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
+        .order_by(VesselPosition.received_at.desc())
+    )
+
+    result = await db.execute(stmt)
+    rows = list(result.all())
+    legacy_geojson = convert_vessels_to_geojson(rows)
+    merged_features, diagnostics = _merge_vessel_features(
+        raw_geojson.get("features", []),
+        legacy_geojson.get("features", []),
+    )
+    return merged_features, {
+        **diagnostics,
+        "raw_feature_count": len(raw_geojson.get("features", [])),
+        "legacy_feature_count": len(legacy_geojson.get("features", [])),
+    }
+
+
+@router.get("/vessels/custom-supplements")
+async def get_vessel_custom_supplements(db: AsyncSession = Depends(get_db)):
+    """Group custom vessel_ais sources by their declared merge target for diagnostics."""
+
+    from app.models.datasource_config import DataSourceConfig
+
+    result = await db.execute(
+        select(DataSourceConfig.name, DataSourceConfig.config, DataSourceConfig.is_active)
+        .where(DataSourceConfig.config["target_schema"].as_string() == "vessel_ais")
+    )
+    grouped: dict[str, dict[str, Any]] = {}
+    for name, config, is_active in result.all():
+        config = config or {}
+        merge_target = str(config.get("merge_target_source") or "barentswatch_vessels")
+        bucket = grouped.setdefault(merge_target, {"merge_target": merge_target, "sources": []})
+        bucket["sources"].append({"name": name, "is_active": bool(is_active)})
+    return {"groups": list(grouped.values())}
+
+
+@router.get("/vessels/name-fallbacks")
+async def get_vessel_name_fallbacks(
+    limit: int = Query(500, ge=0, description="Maximum fallback-name vessels to return. 0 means no limit."),
+    db: AsyncSession = Depends(get_db),
+):
+    """Return vessels whose display name still falls back to MMSI."""
+    aggregated_vessels = await get_aggregated_vessels(db)
+    raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
+
+    latest_times = (
+        select(
+            VesselPosition.mmsi.label("mmsi"),
+            func.max(VesselPosition.received_at).label("received_at"),
+        )
+        .group_by(VesselPosition.mmsi)
+        .subquery()
+    )
+    result = await db.execute(
+        select(VesselPosition, VesselStatic)
+        .join(
+            latest_times,
+            (VesselPosition.mmsi == latest_times.c.mmsi)
+            & (VesselPosition.received_at == latest_times.c.received_at),
+        )
+        .outerjoin(VesselStatic, VesselStatic.mmsi == VesselPosition.mmsi)
+        .order_by(VesselPosition.received_at.desc())
+    )
+    legacy_geojson = convert_vessels_to_geojson(list(result.all()))
+    features, diagnostics = _merge_vessel_features(
+        raw_geojson.get("features", []),
+        legacy_geojson.get("features", []),
+    )
+
+    fallback_items = []
+    for feature in features:
+        props = feature.get("properties", {})
+        mmsi = props.get("mmsi")
+        name = props.get("name")
+        if not _is_vessel_name_fallback(name, mmsi):
+            continue
+        source_summary = props.get("source_summary") or {}
+        fallback_items.append(
+            {
+                "mmsi": str(mmsi),
+                "display_name": name or f"MMSI {mmsi}",
+                "reason": "missing_real_name",
+                "received_at": props.get("received_at"),
+                "sources": sorted(source_summary.keys()),
+                "source_summary": source_summary,
+                "message_types": sorted(
+                    {
+                        message_type
+                        for summary in source_summary.values()
+                        for message_type in (summary.get("message_types") or [])
+                    }
+                ),
+                "field_sources": props.get("field_sources") or {},
+            }
+        )
+
+    if limit and limit > 0:
+        fallback_items = fallback_items[:limit]
+    return {
+        "count": len(fallback_items),
+        "items": fallback_items,
+        "diagnostics": diagnostics,
+    }
+
+
+@router.get("/vessels/{mmsi}")
+async def get_vessel_detail(mmsi: int, db: AsyncSession = Depends(get_db)):
+    from app.services.vessel_enrichment import get_vessel_enrichment_bundle
+
+    aggregated = await get_aggregated_vessel(db, mmsi)
+    enrichment = await get_vessel_enrichment_bundle(db, mmsi)
+    if aggregated is not None:
+        return {
+            **aggregated,
+            "received_at": to_iso8601_utc(aggregated.get("received_at")),
+            "latitude": aggregated["lat"],
+            "longitude": aggregated["lon"],
+            "enrichment": enrichment,
+        }
+
+    latest_position_stmt = (
+        select(VesselPosition)
+        .where(VesselPosition.mmsi == mmsi)
+        .order_by(VesselPosition.received_at.desc())
+        .limit(1)
+    )
+    static = await db.get(VesselStatic, mmsi)
+    result = await db.execute(latest_position_stmt)
+    position = result.scalar_one_or_none()
+    if position is None:
+        raise HTTPException(status_code=404, detail="Vessel not found")
+    geojson = convert_vessels_to_geojson([(position, static)])
+    return {
+        **(geojson["features"][0]["properties"]),
+        "latitude": position.lat,
+        "longitude": position.lon,
+        "enrichment": enrichment,
+    }
+
+
+@router.get("/vessels/{mmsi}/track")
+async def get_vessel_track(
+    mmsi: int,
+    hours: int = Query(6, ge=1, le=24),
+    db: AsyncSession = Depends(get_db),
+):
+    cutoff = datetime.now(UTC) - timedelta(hours=hours)
+    aggregated_points = await get_aggregated_vessel_track(db, mmsi, cutoff=cutoff)
+    if aggregated_points:
+        return {
+            "type": "FeatureCollection",
+            "features": [
+                {
+                    "type": "Feature",
+                    "geometry": {
+                        "type": "LineString",
+                        "coordinates": [[point["lon"], point["lat"]] for point in aggregated_points],
+                    },
+                    "properties": {
+                        "mmsi": mmsi,
+                        "hours": hours,
+                        "point_count": len(aggregated_points),
+                        "start_at": to_iso8601_utc(aggregated_points[0]["observed_at"]),
+                        "end_at": to_iso8601_utc(aggregated_points[-1]["observed_at"]),
+                        "point_sources": [point["source"] for point in aggregated_points],
+                    },
+                }
+            ],
+            "count": 1,
+        }
+
+    result = await db.execute(
+        select(VesselPosition)
+        .where(VesselPosition.mmsi == mmsi)
+        .where(VesselPosition.received_at >= cutoff)
+        .order_by(VesselPosition.received_at.asc())
+    )
+    positions = list(result.scalars().all())
+    if not positions:
+        return {
+            "type": "FeatureCollection",
+            "features": [],
+            "count": 0,
+        }
+
+    return {
+        "type": "FeatureCollection",
+        "features": [
+            {
+                "type": "Feature",
+                "geometry": {
+                    "type": "LineString",
+                    "coordinates": [[position.lon, position.lat] for position in positions],
+                },
+                "properties": {
+                    "mmsi": mmsi,
+                    "hours": hours,
+                    "point_count": len(positions),
+                    "start_at": to_iso8601_utc(positions[0].received_at),
+                    "end_at": to_iso8601_utc(positions[-1].received_at),
+                },
+            }
+        ],
+        "count": 1,
+    }
+
+
+@router.get("/vessels/{mmsi}/observations")
+async def get_vessel_observations(
+    mmsi: int,
+    limit: int = Query(100, ge=1, le=500),
+    db: AsyncSession = Depends(get_db),
+):
+    """Return raw AIS observations for debugging source-level collector facts."""
+
+    observations = await get_vessel_raw_observations(db, mmsi, limit=limit)
+    return {
+        "mmsi": mmsi,
+        "count": len(observations),
+        "observations": [item.to_dict() for item in observations],
+        "conflict_candidates": build_field_conflict_candidates(observations),
+    }
+
+
+@router.get("/vessels/{mmsi}/conflicts")
+async def get_vessel_conflicts(mmsi: int, db: AsyncSession = Depends(get_db)):
+    """Return recorded AIS conflicts plus current raw-observation candidates."""
+
+    records = await get_vessel_conflict_records(db, mmsi)
+    observations = await get_vessel_raw_observations(db, mmsi, limit=500)
+    return {
+        "mmsi": mmsi,
+        "count": len(records),
+        "conflicts": [item.to_dict() for item in records],
+        "candidates": build_field_conflict_candidates(observations),
+    }
+
+
 @router.get("/geo/bgp-anomalies")
 async def get_bgp_anomalies_geojson(
     severity: Optional[str] = Query(None),
@@ -1287,6 +2039,82 @@ async def get_bgp_collectors_geojson(db: AsyncSession = Depends(get_db)):
     return {**geojson, "count": len(geojson.get("features", []))}
 
 
+@router.get("/geo/summary")
+async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db)):
+    """Return lightweight Earth HUD counts without loading layer GeoJSON payloads."""
+    cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
+    landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
+    satellite_count = await _count_current_or_latest_task_data(
+        db,
+        "celestrak_tle",
+        exclude_unknown_name=True,
+    )
+    supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
+    gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
+    compute_center_count = supercomputer_count + gpu_cluster_count
+
+    active_incident_result = await db.execute(
+        select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
+    )
+    active_anomaly_result = await db.execute(
+        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
+    )
+    active_incident_count = int(active_incident_result.scalar() or 0)
+    active_anomaly_count = int(active_anomaly_result.scalar() or 0)
+    bgp_collector_result = await db.execute(
+        select(func.count(func.distinct(BGPObservation.collector)))
+        .where(BGPObservation.collector.isnot(None))
+        .where(func.length(func.btrim(BGPObservation.collector)) > 0)
+        .where(BGPObservation.source.in_(("ris_live_bgp", "bgpstream_bgp")))
+    )
+    bgp_collector_scalar = bgp_collector_result.scalar()
+    if bgp_collector_scalar is None:
+        bgp_collectors = await build_bgp_collector_coverage(
+            db,
+            source_filter=("ris_live_bgp", "bgpstream_bgp"),
+        )
+        bgp_collector_count = len(
+            [item for item in bgp_collectors if item.get("collector")]
+        )
+    else:
+        bgp_collector_count = int(bgp_collector_scalar or 0)
+    raw_unique_window_hours = 24
+    raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
+        db,
+        observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
+    )
+    legacy_unique_result = await db.execute(
+        select(func.count(func.distinct(VesselPosition.mmsi)))
+    )
+    legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
+    vessel_count = max(raw_unique_mmsi, legacy_unique_mmsi)
+    aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
+
+    return {
+        "generated_at": to_iso8601_utc(datetime.now(UTC)),
+        "stats": {
+            "cable_count": cable_count,
+            "landing_point_count": landing_point_count,
+            "satellite_count": satellite_count,
+            "compute_center_count": compute_center_count,
+            "vessel_count": vessel_count,
+            "vessel_raw_unique_mmsi": raw_unique_mmsi,
+            "vessel_raw_unique_window_hours": raw_unique_window_hours,
+            "vessel_legacy_unique_mmsi": legacy_unique_mmsi,
+            "aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
+            "aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
+            "aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,
+            "aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
+            "supercomputer_count": supercomputer_count,
+            "gpu_cluster_count": gpu_cluster_count,
+            "bgp_event_count": active_incident_count or active_anomaly_count,
+            "bgp_incident_count": active_incident_count,
+            "bgp_anomaly_count": active_anomaly_count,
+            "bgp_collector_count": bgp_collector_count,
+        },
+    }
+
+
 @router.get("/all")
 async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
     """获取所有可视化数据的统一端点

backend/app/api/v1/websocket.py

@@ -2,7 +2,6 @@
 
 import asyncio
 import json
-import logging
 from datetime import UTC, datetime
 from typing import Optional
 
@@ -10,10 +9,11 @@ from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
 from jose import jwt, JWTError
 
 from app.core.config import settings
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
-logger = logging.getLogger(__name__)
+logger = get_logger(__name__, service="api")
 router = APIRouter()
 
 
@@ -22,28 +22,52 @@ async def authenticate_token(token: str) -> Optional[dict]:
     try:
         payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
         if payload.get("type") != "access":
-            logger.warning(f"WebSocket auth failed: wrong token type")
+            logger.warning_event(
+                "WebSocket auth failed: wrong token type",
+                event="auth.websocket.invalid_token_type",
+            )
             return None
         return payload
     except JWTError as e:
-        logger.warning(f"WebSocket auth failed: {e}")
+        logger.warning_event(
+            "WebSocket auth failed",
+            event="auth.websocket.decode_failed",
+            context={"error": str(e)},
+        )
         return None
 
 
 @router.websocket("/ws")
 async def websocket_endpoint(
     websocket: WebSocket,
-    token: str = Query(...),
+    token: str | None = Query(None),
 ):
     """WebSocket endpoint for real-time data"""
-    logger.info(f"WebSocket connection attempt with token: {token[:20]}...")
-    payload = await authenticate_token(token)
-    if payload is None:
-        logger.warning("WebSocket authentication failed, closing connection")
+    logger.info_event(
+        "WebSocket connection attempt",
+        event="auth.websocket.connection_attempt",
+        context={"token_preview": f"{token[:8]}..." if token else "anonymous"},
+    )
+    payload = await authenticate_token(token) if token else None
+    if token and payload is None:
+        logger.warning_event(
+            "WebSocket authentication failed, closing connection",
+            event="auth.websocket.connection_rejected",
+        )
         await websocket.close(code=4001)
         return
 
-    user_id = str(payload.get("sub"))
+    is_anonymous = payload is None
+    user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
+    supported_channels = ["vessels"] if is_anonymous else [
+        "gpu_clusters",
+        "submarine_cables",
+        "ixp_nodes",
+        "alerts",
+        "dashboard",
+        "datasource_tasks",
+        "vessels",
+    ]
     await manager.connect(websocket, user_id)
 
     try:
@@ -54,14 +78,7 @@ async def websocket_endpoint(
                     "connection_id": f"conn_{user_id}",
                     "server_version": settings.VERSION,
                     "heartbeat_interval": 30,
-                    "supported_channels": [
-                        "gpu_clusters",
-                        "submarine_cables",
-                        "ixp_nodes",
-                        "alerts",
-                        "dashboard",
-                        "datasource_tasks",
-                    ],
+                    "supported_channels": supported_channels,
                 },
             }
         )
@@ -79,12 +96,24 @@ async def websocket_endpoint(
                     )
                 elif data.get("type") == "subscribe":
                     channels = data.get("data", {}).get("channels", [])
+                    if is_anonymous:
+                        channels = [channel for channel in channels if channel in supported_channels]
+                    manager.subscribe(websocket, channels)
                     await websocket.send_json(
                         {
                             "type": "subscription_confirmed",
                             "data": {"action": "subscribe", "channels": channels},
                         }
                     )
+                elif data.get("type") == "unsubscribe":
+                    channels = data.get("data", {}).get("channels", [])
+                    manager.unsubscribe(websocket, channels)
+                    await websocket.send_json(
+                        {
+                            "type": "subscription_confirmed",
+                            "data": {"action": "unsubscribe", "channels": channels},
+                        }
+                    )
                 elif data.get("type") == "control_frame":
                     await websocket.send_json(
                         {"type": "control_acknowledged", "data": {"received": True}}

backend/app/core/cache.py

@@ -1,15 +1,15 @@
 """Redis caching service"""
 
 import json
-import logging
 from datetime import timedelta
 from typing import Optional, Any
 
 import redis
 
 from app.core.config import settings
+from app.core.logging import get_logger
 
-logger = logging.getLogger(__name__)
+logger = get_logger(__name__)
 
 
 # Lazy Redis client initialization
@@ -47,7 +47,7 @@ class CacheService:
                 return json.loads(value)
             return None
         except Exception as e:
-            logger.warning(f"Cache get error: {e}")
+            logger.warning_event("Cache get error", event="cache.get.failed", context={"error": str(e)})
             return None
 
     def set(
@@ -61,7 +61,7 @@ class CacheService:
             serialized = json.dumps(value, default=str)
             return self.client.setex(key, expire_seconds, serialized)
         except Exception as e:
-            logger.warning(f"Cache set error: {e}")
+            logger.warning_event("Cache set error", event="cache.set.failed", context={"error": str(e)})
             return False
 
     def delete(self, key: str) -> bool:
@@ -69,7 +69,7 @@ class CacheService:
         try:
             return self.client.delete(key) > 0
         except Exception as e:
-            logger.warning(f"Cache delete error: {e}")
+            logger.warning_event("Cache delete error", event="cache.delete.failed", context={"error": str(e)})
             return False
 
     def delete_pattern(self, pattern: str) -> int:
@@ -80,7 +80,7 @@ class CacheService:
                 return self.client.delete(*keys)
             return 0
         except Exception as e:
-            logger.warning(f"Cache delete_pattern error: {e}")
+            logger.warning_event("Cache delete_pattern error", event="cache.delete_pattern.failed", context={"error": str(e)})
             return 0
 
     def get_or_set(

backend/app/core/data_sources.py

@@ -1,7 +1,6 @@
 import os
 import yaml
 from functools import lru_cache
-from typing import Optional
 
 
 COLLECTOR_URL_KEYS = {
@@ -31,6 +30,8 @@ COLLECTOR_URL_KEYS = {
     "opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
     "nro_delegated_prefix_geo": "nro.delegated_stats_url",
     "news_live_streams": "news_live_streams.channels_url",
+    "barentswatch_vessels": "barentswatch_vessels.url",
+    "aisstream_vessels": "aisstream_vessels.url",
 }
 
 
@@ -73,7 +74,7 @@ class DataSourcesConfig:
             from app.models.datasource_config import DataSourceConfig
 
             query = select(DataSourceConfig).where(
-                DataSourceConfig.name == collector_name, DataSourceConfig.is_active == True
+                DataSourceConfig.name == collector_name, DataSourceConfig.is_active
             )
             result = await db.execute(query)
             db_config = result.scalar_one_or_none()

backend/app/core/data_sources.yaml

@@ -94,3 +94,11 @@ news_live_streams:
   streams_url: "https://iptv-org.github.io/api/streams.json"
   # IPTV-org 台标 JSON
   logos_url: "https://iptv-org.github.io/api/logos.json"
+
+barentswatch_vessels:
+  # BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
+  url: "https://live.ais.barentswatch.no/v1/latest/combined"
+
+aisstream_vessels:
+  # AISStream realtime WebSocket endpoint. Requires an AISStream API key.
+  url: "wss://stream.aisstream.io/v0/stream"

backend/app/core/datasource_defaults.py

@@ -4,163 +4,258 @@ DEFAULT_DATASOURCES = {
     "top500": {
         "id": 1,
         "name": "TOP500 Supercomputers",
+        "display_name": "TOP500 超算榜单",
         "module": "L1",
         "priority": "P0",
         "frequency_minutes": 240,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "epoch_ai_gpu": {
         "id": 2,
         "name": "Epoch AI GPU Clusters",
+        "display_name": "Epoch AI GPU 集群",
         "module": "L1",
         "priority": "P0",
         "frequency_minutes": 360,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "huggingface_models": {
         "id": 3,
         "name": "HuggingFace Models",
+        "display_name": "Hugging Face 模型",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 720,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "huggingface_datasets": {
         "id": 4,
         "name": "HuggingFace Datasets",
+        "display_name": "Hugging Face 数据集",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 720,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "huggingface_spaces": {
         "id": 5,
         "name": "HuggingFace Spaces",
+        "display_name": "Hugging Face Spaces",
         "module": "L2",
         "priority": "P2",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "peeringdb_ixp": {
         "id": 6,
         "name": "PeeringDB IXP",
+        "display_name": "PeeringDB 交换中心",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "peeringdb_network": {
         "id": 7,
         "name": "PeeringDB Networks",
+        "display_name": "PeeringDB 网络",
         "module": "L2",
         "priority": "P2",
         "frequency_minutes": 2880,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "peeringdb_facility": {
         "id": 8,
         "name": "PeeringDB Facilities",
+        "display_name": "PeeringDB 设施",
         "module": "L2",
         "priority": "P2",
         "frequency_minutes": 2880,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "telegeography_cables": {
         "id": 9,
         "name": "Submarine Cables",
+        "display_name": "海底光缆",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "telegeography_landing": {
         "id": 10,
         "name": "Cable Landing Points",
+        "display_name": "光缆登陆点",
         "module": "L2",
         "priority": "P2",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "telegeography_systems": {
         "id": 11,
         "name": "Cable Systems",
+        "display_name": "光缆系统",
         "module": "L2",
         "priority": "P2",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "arcgis_cables": {
         "id": 15,
         "name": "ArcGIS Submarine Cables",
+        "display_name": "ArcGIS 海底光缆",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "arcgis_landing_points": {
         "id": 16,
         "name": "ArcGIS Landing Points",
+        "display_name": "ArcGIS 登陆点",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "arcgis_cable_landing_relation": {
         "id": 17,
         "name": "ArcGIS Cable-Landing Relations",
+        "display_name": "ArcGIS 光缆登陆关系",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "fao_landing_points": {
         "id": 18,
         "name": "FAO Landing Points",
+        "display_name": "FAO 登陆点",
         "module": "L2",
         "priority": "P1",
         "frequency_minutes": 10080,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "spacetrack_tle": {
         "id": 19,
         "name": "Space-Track TLE",
+        "display_name": "Space-Track 轨道根数",
         "module": "L3",
         "priority": "P2",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": True,
+        "credential_provider": "spacetrack",
+        "credential_status": "planned",
     },
     "celestrak_tle": {
         "id": 20,
         "name": "CelesTrak TLE",
+        "display_name": "CelesTrak 轨道根数",
         "module": "L3",
         "priority": "P2",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "ris_live_bgp": {
         "id": 21,
         "name": "RIPE RIS Live BGP",
+        "display_name": "RIPE RIS 实时 BGP",
         "module": "L3",
         "priority": "P1",
         "frequency_minutes": 15,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "bgpstream_bgp": {
         "id": 22,
         "name": "CAIDA BGPStream Backfill",
+        "display_name": "CAIDA BGPStream 回填",
         "module": "L3",
         "priority": "P1",
         "frequency_minutes": 360,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "iptoasn_prefix_geo": {
         "id": 23,
         "name": "IPtoASN Prefix Geography",
+        "display_name": "IPtoASN 前缀地理",
         "module": "L3",
         "priority": "P1",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "opengeofeed_prefix_geo": {
         "id": 24,
         "name": "OpenGeoFeed Prefix Geography",
+        "display_name": "OpenGeoFeed 前缀地理",
         "module": "L3",
         "priority": "P1",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "nro_delegated_prefix_geo": {
         "id": 25,
         "name": "NRO Delegated Prefix Geography",
+        "display_name": "NRO 分配前缀地理",
         "module": "L3",
         "priority": "P1",
         "frequency_minutes": 1440,
+        "is_free": True,
+        "requires_credentials": False,
     },
     "news_live_streams": {
         "id": 26,
         "name": "News Live Streams",
+        "display_name": "新闻直播源",
         "module": "L4",
         "priority": "P2",
         "frequency_minutes": 720,
+        "is_free": True,
+        "requires_credentials": False,
+    },
+    "barentswatch_vessels": {
+        "id": 27,
+        "name": "BarentsWatch AIS Vessels",
+        "display_name": "BarentsWatch AIS 船舶",
+        "module": "L4",
+        "priority": "P1",
+        "frequency_minutes": 1,
+        "is_free": True,
+        "requires_credentials": True,
+        "credential_provider": "barentswatch",
+        "credential_status": "supported",
+    },
+    "aisstream_vessels": {
+        "id": 28,
+        "name": "AISStream Vessels",
+        "display_name": "AISStream 实时船舶",
+        "module": "L4",
+        "priority": "P1",
+        "frequency_minutes": 1,
+        "is_free": True,
+        "requires_credentials": True,
+        "credential_provider": "aisstream",
+        "credential_status": "supported",
     },
 }
 

backend/app/core/logging.py

@@ -0,0 +1,161 @@
+from __future__ import annotations
+
+import json
+import logging
+import os
+import re
+
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from app.core.request_context import get_request_id
+
+DEFAULT_SERVICE = "backend"
+DEFAULT_EVENT = "app.log"
+DEFAULT_LOG_LEVEL = os.getenv("PLANET_LOG_LEVEL", "INFO").upper()
+REDACTED = "[REDACTED]"
+SENSITIVE_FIELD_NAMES = {
+    "access_token",
+    "api_key",
+    "authorization",
+    "cookie",
+    "password",
+    "refresh_token",
+    "secret",
+    "token",
+}
+SENSITIVE_TEXT_PATTERNS = (
+    re.compile(r"(?i)(authorization\s*[:=]\s*)(.+)"),
+    re.compile(r"(?i)(bearer\s+)([A-Za-z0-9._\-]+)"),
+    re.compile(r"(?i)(token\s*[:=]\s*)(.+)"),
+    re.compile(r"(?i)(password\s*[:=]\s*)(.+)"),
+    re.compile(r"(?i)(cookie\s*[:=]\s*)(.+)"),
+)
+
+
+def sanitize_log_value(value: Any) -> Any:
+    if isinstance(value, Mapping):
+        return {
+            str(key): (REDACTED if str(key).lower() in SENSITIVE_FIELD_NAMES else sanitize_log_value(item))
+            for key, item in value.items()
+        }
+    if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
+        return [sanitize_log_value(item) for item in value]
+    if isinstance(value, str):
+        sanitized = value
+        for pattern in SENSITIVE_TEXT_PATTERNS:
+            sanitized = pattern.sub(lambda match: f"{match.group(1)}{REDACTED}", sanitized)
+        return sanitized
+    return value
+
+
+def _normalize_context(context: Any) -> dict[str, Any]:
+    if context is None:
+        return {}
+    if isinstance(context, Mapping):
+        sanitized = sanitize_log_value(context)
+        return {str(key): value for key, value in sanitized.items()}
+    return {"value": sanitize_log_value(context)}
+
+
+class PlanetContextFilter(logging.Filter):
+    def filter(self, record: logging.LogRecord) -> bool:
+        record.request_id = getattr(record, "request_id", None) or get_request_id() or "-"
+        record.service = getattr(record, "service", None) or DEFAULT_SERVICE
+        record.event = getattr(record, "event", None) or DEFAULT_EVENT
+        record.context = _normalize_context(getattr(record, "context", None))
+        record.message = sanitize_log_value(record.getMessage())
+        return True
+
+
+class PlanetFormatter(logging.Formatter):
+    def format(self, record: logging.LogRecord) -> str:
+        timestamp = self.formatTime(record, self.datefmt)
+        level = record.levelname
+        service = getattr(record, "service", DEFAULT_SERVICE)
+        module_name = record.name
+        event = getattr(record, "event", DEFAULT_EVENT)
+        request_id = getattr(record, "request_id", "-")
+        message = sanitize_log_value(record.getMessage())
+        context = _normalize_context(getattr(record, "context", None))
+        context_suffix = ""
+        if context:
+            context_suffix = f" context={json.dumps(context, ensure_ascii=False, sort_keys=True)}"
+        rendered = (
+            f"{timestamp} {level} service={service} module={module_name} "
+            f"event={event} request_id={request_id} message={message}{context_suffix}"
+        )
+        if record.exc_info:
+            rendered = f"{rendered}\n{self.formatException(record.exc_info)}"
+        return rendered
+
+
+class PlanetLoggerAdapter(logging.LoggerAdapter):
+    def process(self, msg: Any, kwargs: dict[str, Any]) -> tuple[Any, dict[str, Any]]:
+        extra = dict(self.extra)
+        extra.update(kwargs.get("extra", {}))
+        if "context" in extra:
+            extra["context"] = _normalize_context(extra.get("context"))
+        kwargs["extra"] = extra
+        return sanitize_log_value(msg), kwargs
+
+    def log_event(
+        self,
+        level: int,
+        message: str,
+        *,
+        event: str,
+        context: Mapping[str, Any] | None = None,
+        **extra: Any,
+    ) -> None:
+        self.log(level, message, extra={"event": event, "context": context or {}, **extra})
+
+    def debug_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
+        self.log_event(logging.DEBUG, message, event=event, context=context, **extra)
+
+    def info_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
+        self.log_event(logging.INFO, message, event=event, context=context, **extra)
+
+    def warning_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
+        self.log_event(logging.WARNING, message, event=event, context=context, **extra)
+
+    def error_event(self, message: str, *, event: str, context: Mapping[str, Any] | None = None, **extra: Any) -> None:
+        self.log_event(logging.ERROR, message, event=event, context=context, **extra)
+
+    def exception_event(
+        self,
+        message: str,
+        *,
+        event: str,
+        context: Mapping[str, Any] | None = None,
+        **extra: Any,
+    ) -> None:
+        self.error(message, exc_info=True, extra={"event": event, "context": context or {}, **extra})
+
+
+def get_logger(name: str, *, service: str = DEFAULT_SERVICE) -> PlanetLoggerAdapter:
+    return PlanetLoggerAdapter(logging.getLogger(name), {"service": service})
+
+
+def configure_logging(level: str | None = None) -> None:
+    root_logger = logging.getLogger()
+    if getattr(configure_logging, "_configured", False):
+        if level:
+            root_logger.setLevel(level.upper())
+        return
+
+    handler = logging.StreamHandler()
+    handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
+    handler.addFilter(PlanetContextFilter())
+
+    root_logger.handlers.clear()
+    root_logger.addHandler(handler)
+    root_logger.setLevel((level or DEFAULT_LOG_LEVEL).upper())
+
+    for logger_name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
+        target_logger = logging.getLogger(logger_name)
+        target_logger.handlers.clear()
+        target_logger.propagate = True
+
+    logging.captureWarnings(True)
+    configure_logging._configured = True

backend/app/core/request_context.py

@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+from contextvars import ContextVar
+
+
+request_id_context: ContextVar[str | None] = ContextVar("request_id", default=None)
+
+
+def set_request_id(request_id: str | None) -> None:
+    request_id_context.set(request_id)
+
+
+def get_request_id() -> str | None:
+    return request_id_context.get()

backend/app/core/target_schema_registry.py

@@ -0,0 +1,157 @@
+"""Registry of target schemas supported by mapped custom data sources."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any
+
+from pydantic import BaseModel, Field, ValidationError, field_validator
+
+
+class VesselAISRecord(BaseModel):
+    mmsi: int = Field(ge=100000000, le=999999999)
+    lat: float = Field(ge=-90, le=90)
+    lon: float = Field(ge=-180, le=180)
+    sog: float | None = None
+    cog: float | None = Field(default=None, ge=0, le=360)
+    heading: int | None = Field(default=None, ge=0, le=511)
+    nav_status: int | None = None
+    name: str | None = None
+    callsign: str | None = None
+    vessel_type: str | int | None = None
+    vessel_type_name: str | None = None
+    received_at: datetime | None = None
+
+
+class GeoPointRecord(BaseModel):
+    lat: float = Field(ge=-90, le=90)
+    lon: float = Field(ge=-180, le=180)
+    name: str | None = None
+    type: str | None = None
+    source_id: str | None = None
+    observed_at: datetime | None = None
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class GenericRecord(BaseModel):
+    data: dict[str, Any] = Field(default_factory=dict)
+    source_id: str | None = None
+    observed_at: datetime | None = None
+
+    @field_validator("data")
+    @classmethod
+    def require_payload(cls, value: dict[str, Any]) -> dict[str, Any]:
+        if not value:
+            raise ValueError("generic_records requires a non-empty data object")
+        return value
+
+
+@dataclass(frozen=True)
+class TargetField:
+    name: str
+    type: str
+    required: bool = False
+    description: str = ""
+    example: Any = None
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "name": self.name,
+            "type": self.type,
+            "required": self.required,
+            "description": self.description,
+            "example": self.example,
+        }
+
+
+@dataclass(frozen=True)
+class TargetSchema:
+    key: str
+    label: str
+    description: str
+    fields: tuple[TargetField, ...]
+    model: type[BaseModel]
+    destination: str
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "key": self.key,
+            "label": self.label,
+            "description": self.description,
+            "destination": self.destination,
+            "fields": [field.to_dict() for field in self.fields],
+        }
+
+    def validate_record(self, record: dict[str, Any]) -> tuple[dict[str, Any] | None, list[str]]:
+        try:
+            return self.model.model_validate(record).model_dump(mode="json"), []
+        except ValidationError as exc:
+            return None, [
+                ".".join(str(part) for part in error["loc"]) + f": {error['msg']}"
+                for error in exc.errors()
+            ]
+
+
+TARGET_SCHEMAS: dict[str, TargetSchema] = {
+    "vessel_ais": TargetSchema(
+        key="vessel_ais",
+        label="船舶 AIS",
+        description="船只位置、航速、航向、MMSI 等 AIS 数据。",
+        destination="vessel_position",
+        model=VesselAISRecord,
+        fields=(
+            TargetField("mmsi", "integer", True, "MMSI 九位船舶标识", 257123000),
+            TargetField("lat", "float", True, "纬度", 59.91),
+            TargetField("lon", "float", True, "经度", 10.75),
+            TargetField("sog", "float", False, "对地航速，单位节", 12.4),
+            TargetField("cog", "float", False, "对地航向，0-360 度", 184.5),
+            TargetField("heading", "integer", False, "船首向，0-511", 186),
+            TargetField("nav_status", "integer", False, "导航状态码", 0),
+            TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
+            TargetField("callsign", "string", False, "呼号", "LAAB"),
+            TargetField("vessel_type", "string", False, "船型代码", 70),
+            TargetField("vessel_type_name", "string", False, "船型名称", "Cargo"),
+            TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
+        ),
+    ),
+    "geo_points": TargetSchema(
+        key="geo_points",
+        label="通用地理点",
+        description="带经纬度的通用实体或事件点位。",
+        destination="generic_geo_points",
+        model=GeoPointRecord,
+        fields=(
+            TargetField("lat", "float", True, "纬度", 1.3),
+            TargetField("lon", "float", True, "经度", 103.8),
+            TargetField("name", "string", False, "点位名称", "Singapore"),
+            TargetField("type", "string", False, "点位类型", "datacenter"),
+            TargetField("source_id", "string", False, "来源侧 ID", "sg-1"),
+            TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
+            TargetField("metadata", "object", False, "扩展字段", {"provider": "example"}),
+        ),
+    ),
+    "generic_records": TargetSchema(
+        key="generic_records",
+        label="通用结构化记录",
+        description="未知结构数据沉淀，不直接进入 Earth 图层。",
+        destination="collected_data",
+        model=GenericRecord,
+        fields=(
+            TargetField("data", "object", True, "结构化记录主体", {"raw": "value"}),
+            TargetField("source_id", "string", False, "来源侧 ID", "record-1"),
+            TargetField("observed_at", "datetime", False, "观测时间", "2026-04-28T00:00:00Z"),
+        ),
+    ),
+}
+
+
+def list_target_schemas() -> list[dict[str, Any]]:
+    return [schema.to_dict() for schema in TARGET_SCHEMAS.values()]
+
+
+def get_target_schema(key: str) -> TargetSchema:
+    try:
+        return TARGET_SCHEMAS[key]
+    except KeyError as exc:
+        raise ValueError(f"Unsupported target schema: {key}") from exc

backend/app/core/websocket/broadcaster.py

@@ -75,7 +75,7 @@ class DataBroadcaster:
                 "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
-            channel=channel if channel in manager.active_connections else "all",
+            channel=channel,
         )
 
     async def broadcast_datasource_task_update(self, data: Dict[str, Any]):

backend/app/core/websocket/manager.py

@@ -1,9 +1,6 @@
 """WebSocket Connection Manager"""
 
-import json
-import asyncio
 from typing import Dict, Set, Optional
-from datetime import datetime
 from fastapi import WebSocket
 import redis.asyncio as redis
 
@@ -15,6 +12,8 @@ class ConnectionManager:
 
     def __init__(self):
         self.active_connections: Dict[str, Set[WebSocket]] = {}  # user_id -> connections
+        self.channel_subscriptions: Dict[str, Set[WebSocket]] = {}
+        self.websocket_channels: Dict[WebSocket, Set[str]] = {}
         self.redis_client: Optional[redis.Redis] = None
 
     async def connect(self, websocket: WebSocket, user_id: str):
@@ -40,6 +39,39 @@ class ConnectionManager:
             self.active_connections[user_id].discard(websocket)
             if not self.active_connections[user_id]:
                 del self.active_connections[user_id]
+        self.unsubscribe_all(websocket)
+
+    def subscribe(self, websocket: WebSocket, channels: list[str]):
+        normalized_channels = {
+            str(channel).strip()
+            for channel in channels
+            if str(channel).strip()
+        }
+        if not normalized_channels:
+            return
+
+        socket_channels = self.websocket_channels.setdefault(websocket, set())
+        for channel in normalized_channels:
+            self.channel_subscriptions.setdefault(channel, set()).add(websocket)
+            socket_channels.add(channel)
+
+    def unsubscribe(self, websocket: WebSocket, channels: list[str]):
+        for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
+            subscribers = self.channel_subscriptions.get(channel)
+            if subscribers is not None:
+                subscribers.discard(websocket)
+                if not subscribers:
+                    del self.channel_subscriptions[channel]
+            socket_channels = self.websocket_channels.get(websocket)
+            if socket_channels is not None:
+                socket_channels.discard(channel)
+                if not socket_channels:
+                    del self.websocket_channels[websocket]
+
+    def unsubscribe_all(self, websocket: WebSocket):
+        channels = list(self.websocket_channels.get(websocket, set()))
+        if channels:
+            self.unsubscribe(websocket, channels)
 
     async def send_personal_message(self, message: dict, user_id: str):
         if user_id in self.active_connections:
@@ -54,13 +86,19 @@ class ConnectionManager:
             for user_id in self.active_connections:
                 await self.send_personal_message(message, user_id)
         else:
-            await self.send_personal_message(message, channel)
+            for connection in list(self.channel_subscriptions.get(channel, set())):
+                try:
+                    await connection.send_json(message)
+                except Exception:
+                    self.unsubscribe_all(connection)
 
     async def close_all(self):
         for user_id in self.active_connections:
             for connection in self.active_connections[user_id]:
                 await connection.close()
         self.active_connections.clear()
+        self.channel_subscriptions.clear()
+        self.websocket_channels.clear()
 
 
 manager = ConnectionManager()

backend/app/db/session.py

@@ -5,10 +5,22 @@ from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sess
 from sqlalchemy.orm import declarative_base
 
 from app.core.config import settings
+from app.core.logging import get_logger
+
+logger = get_logger(__name__)
+
+DB_POOL_CONFIG = {
+    "pool_pre_ping": True,
+    "pool_recycle": 1800,
+    "pool_size": 10,
+    "max_overflow": 20,
+    "pool_timeout": 30,
+}
 
 engine = create_async_engine(
     settings.DATABASE_URL,
     echo=settings.DEBUG if hasattr(settings, "DEBUG") else False,
+    **DB_POOL_CONFIG,
 )
 
 async_session_factory = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
@@ -97,6 +109,22 @@ async def init_db():
     import app.models.system_setting  # noqa: F401
     import app.models.playground_session  # noqa: F401
     import app.models.playground_message  # noqa: F401
+    import app.models.system_log  # noqa: F401
+    import app.models.vessel  # noqa: F401
+    import app.models.vessel_enrichment  # noqa: F401
+    import app.models.datasource_mapping  # noqa: F401
+
+    logger.warning_event(
+        "Database pool settings active",
+        event="database.pool.initialized",
+        context={
+            "pool_pre_ping": DB_POOL_CONFIG["pool_pre_ping"],
+            "pool_recycle": DB_POOL_CONFIG["pool_recycle"],
+            "pool_size": DB_POOL_CONFIG["pool_size"],
+            "max_overflow": DB_POOL_CONFIG["max_overflow"],
+            "pool_timeout": DB_POOL_CONFIG["pool_timeout"],
+        },
+    )
 
     async with engine.begin() as conn:
         await conn.run_sync(Base.metadata.create_all)
@@ -119,7 +147,12 @@ async def init_db():
             text(
                 """
                 ALTER TABLE collection_tasks
-                ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued'
+                ADD COLUMN IF NOT EXISTS phase VARCHAR(30) DEFAULT 'queued',
+                ADD COLUMN IF NOT EXISTS phase_progress DOUBLE PRECISION,
+                ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
+                ADD COLUMN IF NOT EXISTS phase_current BIGINT,
+                ADD COLUMN IF NOT EXISTS phase_total BIGINT,
+                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
                 """
             )
         )
@@ -131,6 +164,30 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collected_data_source_current_id
+                ON collected_data (source, is_current, id)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collected_data_source_task_id
+                ON collected_data (source, task_id, id)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_ais_raw_schema_observed_entity
+                ON ais_raw_observations (target_schema, observed_at, entity_key)
+                """
+            )
+        )
         await conn.execute(
             text(
                 """

backend/app/main.py

@@ -1,4 +1,5 @@
 from contextlib import asynccontextmanager
+from uuid import uuid4
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
@@ -7,6 +8,8 @@ from starlette.middleware.base import BaseHTTPMiddleware
 from app.api.main import api_router
 from app.api.v1 import websocket
 from app.core.config import settings
+from app.core.logging import configure_logging
+from app.core.request_context import set_request_id
 from app.core.websocket.broadcaster import broadcaster
 from app.db.session import init_db
 from app.services.scheduler import (
@@ -17,6 +20,9 @@ from app.services.scheduler import (
 )
 
 
+configure_logging()
+
+
 class WebSocketCORSMiddleware(BaseHTTPMiddleware):
     async def dispatch(self, request, call_next):
         if request.url.path.startswith("/ws") and request.method == "GET":
@@ -28,6 +34,18 @@ class WebSocketCORSMiddleware(BaseHTTPMiddleware):
         return await call_next(request)
 
 
+class RequestContextMiddleware(BaseHTTPMiddleware):
+    async def dispatch(self, request, call_next):
+        request_id = request.headers.get("X-Request-ID") or uuid4().hex
+        set_request_id(request_id)
+        try:
+            response = await call_next(request)
+        finally:
+            set_request_id(None)
+        response.headers["X-Request-ID"] = request_id
+        return response
+
+
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     await init_db()
@@ -58,6 +76,7 @@ app.add_middleware(
     allow_headers=["*"],
 )
 
+app.add_middleware(RequestContextMiddleware)
 app.add_middleware(WebSocketCORSMiddleware)
 
 app.include_router(api_router, prefix="/api/v1")

backend/app/models/__init__.py

@@ -11,6 +11,9 @@ from app.models.bgp_observation import BGPObservation
 from app.models.system_setting import SystemSetting
 from app.models.playground_session import PlaygroundSession
 from app.models.playground_message import PlaygroundMessage
+from app.models.system_log import SystemLog, AuditLog
+from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
+from app.models.datasource_mapping import DataSourceMappingTemplate
 
 __all__ = [
     "User",
@@ -26,4 +29,14 @@ __all__ = [
     "BGPAnomaly",
     "BGPIncident",
     "BGPObservation",
+    "SystemLog",
+    "AuditLog",
+    "PlaygroundSession",
+    "PlaygroundMessage",
+    "VesselPosition",
+    "VesselStatic",
+    "AISRawObservation",
+    "AISConflictRecord",
+    "AISSourceHealth",
+    "DataSourceMappingTemplate",
 ]

backend/app/models/collected_data.py

@@ -48,6 +48,8 @@ class CollectedData(Base):
     # Indexes for common queries
     __table_args__ = (
         Index("idx_collected_data_source_collected", "source", "collected_at"),
+        Index("idx_collected_data_source_current_id", "source", "is_current", "id"),
+        Index("idx_collected_data_source_task_id", "source", "task_id", "id"),
         Index("idx_collected_data_source_type", "source", "data_type"),
         Index("idx_collected_data_source_source_id", "source", "source_id"),
     )

backend/app/models/datasource_mapping.py

@@ -0,0 +1,32 @@
+"""Mapping templates for user-defined data source payloads."""
+
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class DataSourceMappingTemplate(Base):
+    __tablename__ = "datasource_mapping_templates"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    datasource_config_id = Column(
+        Integer,
+        ForeignKey("datasource_configs.id"),
+        nullable=False,
+        index=True,
+    )
+    target_schema = Column(String(80), nullable=False, index=True)
+    mapping_json = Column(JSON, nullable=False, default={})
+    sample_payload_hash = Column(String(64), nullable=True)
+    validation_status = Column(String(30), nullable=False, default="draft")
+    version = Column(Integer, nullable=False, default=1)
+    is_active = Column(Boolean, nullable=False, default=False, index=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+    def __repr__(self):
+        return (
+            f"<DataSourceMappingTemplate {self.id}: "
+            f"{self.datasource_config_id}/{self.target_schema}/v{self.version}>"
+        )

backend/app/models/system_log.py

@@ -0,0 +1,40 @@
+from sqlalchemy import JSON, Column, DateTime, Integer, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class SystemLog(Base):
+    __tablename__ = "system_logs"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True)
+    module = Column(String(120), nullable=True)
+    event = Column(String(160), nullable=True, index=True)
+    level = Column(String(20), nullable=False, index=True)
+    message = Column(Text, nullable=False)
+    request_id = Column(String(64), nullable=True, index=True)
+    trace_id = Column(String(64), nullable=True)
+    user_id = Column(Integer, nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    context = Column(JSON, nullable=False, default=dict)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class AuditLog(Base):
+    __tablename__ = "audit_logs"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
+    actor_id = Column(Integer, nullable=True, index=True)
+    actor_name = Column(String(255), nullable=True)
+    action = Column(String(120), nullable=False, index=True)
+    target_type = Column(String(80), nullable=True)
+    target_id = Column(String(120), nullable=True)
+    result = Column(String(40), nullable=True, index=True)
+    request_id = Column(String(64), nullable=True, index=True)
+    ip = Column(String(64), nullable=True)
+    details = Column(JSON, nullable=False, default=dict)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())

backend/app/models/task.py

@@ -1,6 +1,6 @@
 """Collection Task model"""
 
-from sqlalchemy import Column, DateTime, Integer, String, Text, Float
+from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
 from sqlalchemy.sql import func
 
 from app.db.session import Base
@@ -13,6 +13,11 @@ class CollectionTask(Base):
     datasource_id = Column(Integer, nullable=False, index=True)
     status = Column(String(20), nullable=False)  # pending, running, success, failed, cancelled
     phase = Column(String(30), default="queued")
+    phase_progress = Column(Float)
+    phase_message = Column(String(255))
+    phase_current = Column(BigInteger)
+    phase_total = Column(BigInteger)
+    phase_unit = Column(String(30))
     started_at = Column(DateTime(timezone=True))
     completed_at = Column(DateTime(timezone=True))
     records_processed = Column(Integer, default=0)

backend/app/models/vessel.py

@@ -0,0 +1,186 @@
+"""Vessel AIS models for live maritime tracking."""
+
+from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class VesselStatic(Base):
+    """Slow-changing vessel identity and dimensions."""
+
+    __tablename__ = "vessel_static"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    name = Column(String(128), nullable=True)
+    callsign = Column(String(16), nullable=True)
+    vessel_type = Column(SmallInteger, nullable=True, index=True)
+    vessel_type_name = Column(String(64), nullable=True, index=True)
+    flag = Column(String(4), nullable=True, index=True)
+    length = Column(Float, nullable=True)
+    width = Column(Float, nullable=True)
+    draught = Column(Float, nullable=True)
+    imo = Column(BigInteger, nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "name": self.name,
+            "callsign": self.callsign,
+            "vessel_type": self.vessel_type,
+            "vessel_type_name": self.vessel_type_name,
+            "flag": self.flag,
+            "length": self.length,
+            "width": self.width,
+            "draught": self.draught,
+            "imo": self.imo,
+            "updated_at": to_iso8601_utc(self.updated_at),
+        }
+
+
+class VesselPosition(Base):
+    """Append-only AIS positions retained for short history windows."""
+
+    __tablename__ = "vessel_position"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    mmsi = Column(BigInteger, nullable=False, index=True)
+    lat = Column(Float, nullable=False)
+    lon = Column(Float, nullable=False)
+    sog = Column(Float, nullable=True)
+    cog = Column(Float, nullable=True)
+    heading = Column(SmallInteger, nullable=True)
+    nav_status = Column(SmallInteger, nullable=True, index=True)
+    received_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+
+    __table_args__ = (
+        Index("idx_vessel_pos_mmsi_time", "mmsi", "received_at"),
+        Index("idx_vessel_pos_time", "received_at"),
+        Index("idx_vessel_pos_lat_lon", "lat", "lon"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "mmsi": self.mmsi,
+            "lat": self.lat,
+            "lon": self.lon,
+            "sog": self.sog,
+            "cog": self.cog,
+            "heading": self.heading,
+            "nav_status": self.nav_status,
+            "received_at": to_iso8601_utc(self.received_at),
+        }
+
+
+class AISRawObservation(Base):
+    """Source-level AIS fact before aggregation and conflict resolution."""
+
+    __tablename__ = "ais_raw_observations"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
+    source = Column(String(100), nullable=False, index=True)
+    entity_key = Column(String(64), nullable=False, index=True)
+    delivery_mode = Column(String(32), nullable=False, index=True)
+    transport = Column(String(32), nullable=False, index=True)
+    message_type = Column(String(64), nullable=True, index=True)
+    source_message_id = Column(String(128), nullable=True, index=True)
+    observation_hash = Column(String(64), nullable=False, unique=True, index=True)
+    observed_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    collected_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+    normalized_payload = Column(JSON, default=dict)
+    raw_payload = Column(JSON, default=dict)
+    quality_flags = Column(JSON, default=list)
+
+    __table_args__ = (
+        Index("idx_ais_raw_entity_observed", "target_schema", "entity_key", "observed_at"),
+        Index("idx_ais_raw_schema_observed_entity", "target_schema", "observed_at", "entity_key"),
+        Index("idx_ais_raw_source_entity", "source", "entity_key"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "target_schema": self.target_schema,
+            "source": self.source,
+            "entity_key": self.entity_key,
+            "delivery_mode": self.delivery_mode,
+            "transport": self.transport,
+            "message_type": self.message_type,
+            "source_message_id": self.source_message_id,
+            "observation_hash": self.observation_hash,
+            "observed_at": to_iso8601_utc(self.observed_at),
+            "collected_at": to_iso8601_utc(self.collected_at),
+            "normalized_payload": self.normalized_payload or {},
+            "raw_payload": self.raw_payload or {},
+            "quality_flags": self.quality_flags or [],
+        }
+
+
+class AISConflictRecord(Base):
+    """Recorded field-level disagreement between AIS sources."""
+
+    __tablename__ = "ais_conflict_records"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
+    entity_key = Column(String(64), nullable=False, index=True)
+    field = Column(String(64), nullable=False, index=True)
+    candidates = Column(JSON, default=dict)
+    selected_source = Column(String(100), nullable=True, index=True)
+    selected_value = Column(JSON, nullable=True)
+    selected_reason = Column(String(64), nullable=True, index=True)
+    resolved_by = Column(String(32), nullable=False, default="system", index=True)
+    status = Column(String(32), nullable=False, default="open", index=True)
+    created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    __table_args__ = (
+        Index("idx_ais_conflict_entity_field", "target_schema", "entity_key", "field"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "target_schema": self.target_schema,
+            "entity_key": self.entity_key,
+            "field": self.field,
+            "candidates": self.candidates or {},
+            "selected_source": self.selected_source,
+            "selected_value": self.selected_value,
+            "selected_reason": self.selected_reason,
+            "resolved_by": self.resolved_by,
+            "status": self.status,
+            "created_at": to_iso8601_utc(self.created_at),
+            "updated_at": to_iso8601_utc(self.updated_at),
+        }
+
+
+class AISSourceHealth(Base):
+    """Runtime health signal for an AIS collector source."""
+
+    __tablename__ = "ais_source_health"
+
+    source = Column(String(100), primary_key=True)
+    connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
+    last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    last_error = Column(String(500), nullable=True)
+    message_rate = Column(Float, nullable=True)
+    lag_seconds = Column(Float, nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+
+    def to_dict(self) -> dict:
+        return {
+            "source": self.source,
+            "connection_state": self.connection_state,
+            "last_seen_at": to_iso8601_utc(self.last_seen_at),
+            "last_success_at": to_iso8601_utc(self.last_success_at),
+            "last_error": self.last_error,
+            "message_rate": self.message_rate,
+            "lag_seconds": self.lag_seconds,
+            "updated_at": to_iso8601_utc(self.updated_at),
+        }

backend/app/models/vessel_enrichment.py

@@ -0,0 +1,63 @@
+"""Vessel enrichment cache tables (v5).
+
+Profile and media enrichment are stored separately so cache TTLs can differ
+and so the conflict-resolution + display layers can read either independently.
+"""
+
+from sqlalchemy import BigInteger, Column, DateTime, Float, JSON, String
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class VesselProfileEnrichment(Base):
+    """Cached static vessel profile (type, flag, dimensions, operator, etc.)."""
+
+    __tablename__ = "vessel_profile_enrichment"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    source = Column(String(100), nullable=False, default="system")
+    payload = Column(JSON, nullable=False, default=dict)
+    fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+    expires_at = Column(DateTime(timezone=True), nullable=True)
+    confidence = Column(Float, nullable=True)
+    reference_url = Column(String(500), nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "source": self.source,
+            "payload": self.payload or {},
+            "fetched_at": to_iso8601_utc(self.fetched_at),
+            "expires_at": to_iso8601_utc(self.expires_at),
+            "confidence": self.confidence,
+            "reference_url": self.reference_url,
+        }
+
+
+class VesselMediaEnrichment(Base):
+    """Cached vessel imagery / external detail references."""
+
+    __tablename__ = "vessel_media_enrichment"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    source = Column(String(100), nullable=False, default="system")
+    payload = Column(JSON, nullable=False, default=dict)
+    fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+    expires_at = Column(DateTime(timezone=True), nullable=True)
+    confidence = Column(Float, nullable=True)
+    reference_url = Column(String(500), nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "source": self.source,
+            "payload": self.payload or {},
+            "fetched_at": to_iso8601_utc(self.fetched_at),
+            "expires_at": to_iso8601_utc(self.expires_at),
+            "confidence": self.confidence,
+            "reference_url": self.reference_url,
+        }

backend/app/services/ai_client.py

@@ -3,9 +3,11 @@ from __future__ import annotations
 import asyncio
 
 import httpx
-from fastapi import HTTPException, status
+from fastapi import Depends, HTTPException, status
+from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
+from app.db.session import get_db
 from app.schemas.ai import (
     AIProviderStatusResponse,
     SituationalAnalysisRequest,
@@ -14,11 +16,27 @@ from app.schemas.ai import (
 
 
 class AIProviderClient:
-    def __init__(self) -> None:
-        self.service_url = settings.AI_PROVIDER_SERVICE_URL.rstrip("/")
-        self.service_token = settings.AI_PROVIDER_SERVICE_TOKEN
-        self.timeout = settings.AI_PROVIDER_TIMEOUT_SECONDS
-        self.retry_attempts = max(settings.AI_PROVIDER_RETRY_ATTEMPTS, 1)
+    def __init__(
+        self,
+        *,
+        service_url: str | None = None,
+        service_token: str | None = None,
+        timeout: int | None = None,
+        retry_attempts: int | None = None,
+        llm_config: dict | None = None,
+    ) -> None:
+        self.service_url = (
+            service_url if service_url is not None else settings.AI_PROVIDER_SERVICE_URL
+        ).rstrip("/")
+        self.service_token = (
+            service_token if service_token is not None else settings.AI_PROVIDER_SERVICE_TOKEN
+        )
+        self.timeout = timeout if timeout is not None else settings.AI_PROVIDER_TIMEOUT_SECONDS
+        self.retry_attempts = max(
+            retry_attempts if retry_attempts is not None else settings.AI_PROVIDER_RETRY_ATTEMPTS,
+            1,
+        )
+        self.llm_config = llm_config or {}
 
     def _headers(self, request_id: str | None = None) -> dict[str, str]:
         headers = {"Content-Type": "application/json"}
@@ -26,6 +44,19 @@ class AIProviderClient:
             headers["X-Provider-Token"] = self.service_token
         if request_id:
             headers["X-Request-ID"] = request_id
+        llm_header_map = {
+            "provider": "X-AI-Provider",
+            "provider_api": "X-AI-Provider-API",
+            "base_url": "X-AI-Base-URL",
+            "api_key": "X-AI-API-Key",
+            "model": "X-AI-Model",
+            "max_tokens": "X-AI-Max-Tokens",
+            "anthropic_version": "X-AI-Anthropic-Version",
+        }
+        for key, header_name in llm_header_map.items():
+            value = self.llm_config.get(key)
+            if value not in (None, ""):
+                headers[header_name] = str(value)
         return headers
 
     async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
@@ -105,5 +136,14 @@ class AIProviderClient:
         )
 
 
-def get_ai_provider_client() -> AIProviderClient:
-    return AIProviderClient()
+async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
+    from app.api.v1.settings import get_runtime_ai_provider_config
+
+    runtime_config = await get_runtime_ai_provider_config(db)
+    return AIProviderClient(
+        service_url=runtime_config["service_url"],
+        service_token=runtime_config["service_token"],
+        timeout=runtime_config["timeout_seconds"],
+        retry_attempts=runtime_config["retry_attempts"],
+        llm_config=runtime_config.get("llm_config") or {},
+    )

backend/app/services/barentswatch.py

@@ -0,0 +1,209 @@
+"""BarentsWatch AIS credential resolution and connectivity checks."""
+
+from __future__ import annotations
+
+import os
+import shlex
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import httpx
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.data_sources import get_data_sources_config
+from app.models.datasource_config import DataSourceConfig
+
+
+BARENTSWATCH_LATEST_URL = "https://live.ais.barentswatch.no/v1/latest/combined"
+BARENTSWATCH_TOKEN_URL = "https://id.barentswatch.no/connect/token"
+BARENTSWATCH_DATASOURCE_NAME = "barentswatch_vessels"
+
+
+@dataclass(frozen=True)
+class BarentsWatchConfig:
+    endpoint: str
+    client_id: str
+    client_secret: str
+    credential_source: str
+    endpoint_source: str
+
+
+def _read_zshrc_env(path: Path | None = None) -> dict[str, str]:
+    zshrc_path = path or Path.home() / ".zshrc"
+    if not zshrc_path.exists():
+        return {}
+
+    values: dict[str, str] = {}
+    for raw_line in zshrc_path.read_text(encoding="utf-8", errors="ignore").splitlines():
+        line = raw_line.strip()
+        if not line or line.startswith("#"):
+            continue
+        if line.startswith("export "):
+            line = line[len("export ") :].strip()
+        if "=" not in line:
+            continue
+
+        key, value = line.split("=", 1)
+        key = key.strip()
+        if not key or not key.replace("_", "").isalnum() or not key[0].isalpha():
+            continue
+
+        try:
+            parsed = shlex.split(value, comments=True, posix=True)
+        except ValueError:
+            parsed = [value.strip().strip("'\"")]
+        if parsed:
+            values[key] = parsed[0]
+    return values
+
+
+def _first_env_value(zshrc_env: dict[str, str], *keys: str) -> tuple[str, str]:
+    for key in keys:
+        value = os.getenv(key)
+        if value:
+            return value, "environment"
+    for key in keys:
+        value = zshrc_env.get(key)
+        if value:
+            return value, "~/.zshrc"
+    return "", ""
+
+
+async def get_barentswatch_datasource_record(db: AsyncSession) -> DataSourceConfig | None:
+    result = await db.execute(
+        select(DataSourceConfig)
+        .where(DataSourceConfig.name == BARENTSWATCH_DATASOURCE_NAME)
+        .where(DataSourceConfig.is_active.is_(True))
+    )
+    return result.scalar_one_or_none()
+
+
+async def resolve_barentswatch_config(db: AsyncSession | None = None) -> BarentsWatchConfig:
+    record = await get_barentswatch_datasource_record(db) if db else None
+    auth_config = dict(record.auth_config or {}) if record else {}
+    config = dict(record.config or {}) if record else {}
+    zshrc_env = _read_zshrc_env()
+
+    env_client_id, env_source = _first_env_value(
+        zshrc_env,
+        "BARENTSWATCH_CLIENT_ID",
+        "BARRENTSWATCH_CLIENT_ID",
+    )
+    env_client_secret, secret_env_source = _first_env_value(
+        zshrc_env,
+        "BARENTSWATCH_CLIENT_SECRET",
+        "BARRENTSWATCH_CLIENT_SECRET",
+    )
+    client_id = auth_config.get("client_id") or config.get("client_id") or env_client_id
+    client_secret = (
+        auth_config.get("client_secret") or config.get("client_secret") or env_client_secret
+    )
+
+    credential_source = ""
+    if auth_config.get("client_id") or auth_config.get("client_secret"):
+        credential_source = "datasource_config"
+    elif config.get("client_id") or config.get("client_secret"):
+        credential_source = "datasource_runtime_config"
+    elif env_source or secret_env_source:
+        credential_source = env_source or secret_env_source
+
+    yaml_endpoint = get_data_sources_config().get_yaml_url(BARENTSWATCH_DATASOURCE_NAME)
+    endpoint = record.endpoint if record and record.endpoint else yaml_endpoint
+    return BarentsWatchConfig(
+        endpoint=endpoint or BARENTSWATCH_LATEST_URL,
+        client_id=str(client_id or ""),
+        client_secret=str(client_secret or ""),
+        credential_source=credential_source or "missing",
+        endpoint_source="datasource_config" if record and record.endpoint else "default",
+    )
+
+
+async def fetch_barentswatch_access_token(
+    client: httpx.AsyncClient,
+    config: BarentsWatchConfig,
+) -> str | None:
+    if not config.client_id or not config.client_secret:
+        return None
+
+    response = await client.post(
+        BARENTSWATCH_TOKEN_URL,
+        data={
+            "client_id": config.client_id,
+            "client_secret": config.client_secret,
+            "scope": "ais",
+            "grant_type": "client_credentials",
+        },
+        headers={"Content-Type": "application/x-www-form-urlencoded"},
+    )
+    response.raise_for_status()
+    payload = response.json()
+    token = payload.get("access_token")
+    return str(token) if token else None
+
+
+async def check_barentswatch_connectivity(db: AsyncSession) -> dict[str, Any]:
+    config = await resolve_barentswatch_config(db)
+    return await check_barentswatch_config(config)
+
+
+async def check_barentswatch_config(config: BarentsWatchConfig) -> dict[str, Any]:
+    if not config.client_id or not config.client_secret:
+        return {
+            "success": False,
+            "stage": "credentials",
+            "message": "未找到 BarentsWatch client id/client secret，请先配置采集器凭证。",
+            "endpoint": config.endpoint,
+            "credential_source": config.credential_source,
+            "settings_tab": "collector_credentials",
+        }
+
+    try:
+        async with httpx.AsyncClient(timeout=20.0) as client:
+            token = await fetch_barentswatch_access_token(client, config)
+            if not token:
+                return {
+                    "success": False,
+                    "stage": "token",
+                    "message": "BarentsWatch token 响应中没有 access_token，请检查凭证。",
+                    "endpoint": config.endpoint,
+                    "credential_source": config.credential_source,
+                    "settings_tab": "collector_credentials",
+                }
+
+            async with client.stream(
+                "GET",
+                config.endpoint,
+                headers={"Authorization": f"Bearer {token}"},
+            ) as response:
+                response.raise_for_status()
+
+        return {
+            "success": True,
+            "stage": "endpoint",
+            "message": "BarentsWatch AIS token 和数据接口均可连通。",
+            "endpoint": config.endpoint,
+            "credential_source": config.credential_source,
+            "endpoint_source": config.endpoint_source,
+        }
+    except httpx.HTTPStatusError as exc:
+        status_code = exc.response.status_code
+        stage = "token" if str(exc.request.url) == BARENTSWATCH_TOKEN_URL else "endpoint"
+        return {
+            "success": False,
+            "stage": stage,
+            "message": f"BarentsWatch {stage} 请求返回 HTTP {status_code}，请检查凭证或接口地址。",
+            "endpoint": config.endpoint,
+            "credential_source": config.credential_source,
+            "settings_tab": "collector_credentials",
+        }
+    except httpx.HTTPError as exc:
+        return {
+            "success": False,
+            "stage": "network",
+            "message": f"BarentsWatch 链路检查失败：{exc.__class__.__name__}",
+            "endpoint": config.endpoint,
+            "credential_source": config.credential_source,
+            "settings_tab": "collector_credentials",
+        }

backend/app/services/collectors/__init__.py

@@ -36,6 +36,8 @@ from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
 from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
 from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
 from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
+from app.services.collectors.aisstream import AISStreamCollector
+from app.services.collectors.vessel_ais import VesselAISCollector
 
 collector_registry.register(TOP500Collector())
 collector_registry.register(EpochAIGPUCollector())
@@ -63,3 +65,41 @@ collector_registry.register(IPtoASNPrefixGeoCollector())
 collector_registry.register(OpenGeoFeedPrefixGeoCollector())
 collector_registry.register(NRODelegatedPrefixGeoCollector())
 collector_registry.register(NewsLiveStreamsCollector())
+collector_registry.register(VesselAISCollector())
+collector_registry.register(AISStreamCollector())
+
+__all__ = [
+    "BaseCollector",
+    "HTTPCollector",
+    "IntervalCollector",
+    "collector_registry",
+    "CollectorRegistry",
+    "TOP500Collector",
+    "EpochAIGPUCollector",
+    "HuggingFaceModelCollector",
+    "HuggingFaceDatasetCollector",
+    "HuggingFaceSpacesCollector",
+    "PeeringDBIXPCollector",
+    "PeeringDBNetworkCollector",
+    "PeeringDBFacilityCollector",
+    "TeleGeographyCableCollector",
+    "TeleGeographyLandingPointCollector",
+    "TeleGeographyCableSystemCollector",
+    "CloudflareRadarDeviceCollector",
+    "CloudflareRadarTrafficCollector",
+    "CloudflareRadarTopASCollector",
+    "ArcGISCableCollector",
+    "FAOLandingPointCollector",
+    "ArcGISLandingPointCollector",
+    "ArcGISCableLandingRelationCollector",
+    "SpaceTrackTLECollector",
+    "CelesTrakTLECollector",
+    "RISLiveCollector",
+    "BGPStreamBackfillCollector",
+    "IPtoASNPrefixGeoCollector",
+    "OpenGeoFeedPrefixGeoCollector",
+    "NRODelegatedPrefixGeoCollector",
+    "NewsLiveStreamsCollector",
+    "VesselAISCollector",
+    "AISStreamCollector",
+]

backend/app/services/collectors/aisstream.py

@@ -0,0 +1,491 @@
+"""AISStream WebSocket collector for realtime vessel AIS observations."""
+
+from datetime import UTC, datetime
+import asyncio
+import json
+import os
+from typing import Any
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.data_sources import get_data_sources_config
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
+from app.models.datasource_config import DataSourceConfig
+from app.models.task import CollectionTask
+from app.services.collectors.base import BaseCollector
+from app.services.vessel_ais_aggregation import (
+    AISSTREAM_DELIVERY_MODE,
+    AISSTREAM_TRANSPORT,
+    record_vessel_ais_observation,
+    update_ais_source_health,
+)
+from app.services.vessel_types import normalize_vessel_type_name
+
+DEFAULT_AISSTREAM_URL = "wss://stream.aisstream.io/v0/stream"
+DEFAULT_BOUNDING_BOXES = [[[-90, -180], [90, 180]]]
+DEFAULT_MESSAGE_TYPES = ["PositionReport", "ShipStaticData"]
+
+
+class AISStreamCollector(BaseCollector):
+    """Collect AISStream WebSocket messages into the raw AIS observation layer."""
+
+    name = "aisstream_vessels"
+    priority = "P1"
+    module = "L4"
+    frequency_hours = 1
+    data_type = "vessel_ais"
+    fail_on_empty = False
+
+    async def _load_datasource_config(self) -> DataSourceConfig | None:
+        if self._db_session is None:
+            return None
+        result = await self._db_session.execute(
+            select(DataSourceConfig)
+            .where(DataSourceConfig.name == self.name)
+            .where(DataSourceConfig.is_active.is_(True))
+        )
+        return result.scalar_one_or_none()
+
+    async def _get_effective_config(self) -> dict[str, Any]:
+        datasource_config = await self._load_datasource_config()
+        config = dict(datasource_config.config or {}) if datasource_config else {}
+        auth_config = dict(datasource_config.auth_config or {}) if datasource_config else {}
+        endpoint = (
+            (datasource_config.endpoint if datasource_config else None)
+            or self._resolved_url
+            or get_data_sources_config().get_yaml_url(self.name)
+            or DEFAULT_AISSTREAM_URL
+        )
+        api_key = (
+            auth_config.get("api_key")
+            or config.get("api_key")
+            or os.getenv("AISSTREAM_API_KEY")
+        )
+        return {
+            "endpoint": endpoint,
+            "api_key": api_key,
+            "bounding_boxes": config.get("bounding_boxes") or DEFAULT_BOUNDING_BOXES,
+            "message_types": config.get("message_types") or DEFAULT_MESSAGE_TYPES,
+            "max_messages": int(config.get("max_messages") or 500),
+            "streaming_enabled": config.get("streaming_enabled", True) is not False,
+            "streaming_commit_interval": int(config.get("streaming_commit_interval") or 1),
+            "streaming_max_messages": int(config.get("streaming_max_messages") or 0),
+            "reconnect_delay_seconds": float(config.get("reconnect_delay_seconds") or 5),
+            "receive_timeout_seconds": float(config.get("receive_timeout_seconds") or 30),
+        }
+
+    def _build_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
+        return {
+            "APIKey": config["api_key"],
+            "BoundingBoxes": config["bounding_boxes"],
+            "FilterMessageTypes": config["message_types"],
+        }
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        config = await self._get_effective_config()
+        if not config["api_key"]:
+            raise RuntimeError("AISStream API key is not configured")
+
+        try:
+            import websockets
+        except ImportError as exc:
+            raise RuntimeError("Python package 'websockets' is required for AISStream") from exc
+
+        subscription = self._build_subscription(config)
+
+        messages: list[dict[str, Any]] = []
+        try:
+            async with websockets.connect(config["endpoint"]) as websocket:
+                await websocket.send(json.dumps(subscription))
+                while len(messages) < config["max_messages"]:
+                    try:
+                        raw_message = await asyncio.wait_for(
+                            websocket.recv(),
+                            timeout=config["receive_timeout_seconds"],
+                        )
+                    except TimeoutError:
+                        break
+                    payload = json.loads(raw_message)
+                    if isinstance(payload, dict):
+                        messages.append(payload)
+        except Exception as exc:
+            if self._db_session is not None:
+                await update_ais_source_health(
+                    self._db_session,
+                    source=self.name,
+                    connection_state="disconnected",
+                    last_error=f"{exc.__class__.__name__}: {exc}",
+                )
+                await self._db_session.commit()
+            raise
+
+        return messages
+
+    async def run(self, db: AsyncSession) -> dict[str, Any]:
+        """Run AISStream as a long-lived streaming collector by default."""
+        config = await self._get_effective_config()
+        if not config.get("streaming_enabled", True):
+            return await super().run(db)
+        if not config["api_key"]:
+            return {"status": "failed", "error": "AISStream API key is not configured"}
+
+        from app.services.collectors.registry import collector_registry
+
+        if not collector_registry.is_active(self.name):
+            return {"status": "skipped", "reason": "Collector is disabled"}
+
+        try:
+            import websockets
+        except ImportError as exc:
+            return {"status": "failed", "error": "Python package 'websockets' is required for AISStream"}
+
+        start_time = datetime.now(UTC)
+        task = CollectionTask(
+            datasource_id=getattr(self, "_datasource_id", 1),
+            status="running",
+            phase="connecting",
+            phase_message="正在连接 AISStream 实时流",
+            phase_unit="messages",
+            started_at=start_time,
+        )
+        db.add(task)
+        await db.commit()
+        self._current_task = task
+        self._db_session = db
+        self._last_broadcast_progress = None
+        await self.resolve_url(db)
+        await self._publish_task_update(force=True)
+
+        records_added = 0
+        messages_seen = 0
+        unique_mmsi: set[str] = set()
+        reconnect_delay = config["reconnect_delay_seconds"]
+
+        try:
+            while True:
+                config = await self._get_effective_config()
+                subscription = self._build_subscription(config)
+                try:
+                    await update_ais_source_health(
+                        db,
+                        source=self.name,
+                        connection_state="connecting",
+                    )
+                    await self.set_phase("connecting", message="正在连接 AISStream 实时流")
+                    await db.commit()
+
+                    async with websockets.connect(config["endpoint"]) as websocket:
+                        await websocket.send(json.dumps(subscription))
+                        await update_ais_source_health(
+                            db,
+                            source=self.name,
+                            connection_state="connected",
+                            last_success_at=datetime.now(UTC),
+                        )
+                        await self.set_phase(
+                            "streaming",
+                            message="正在接收 AISStream 实时消息",
+                            reset_progress=False,
+                        )
+                        await db.commit()
+
+                        while True:
+                            try:
+                                raw_message = await asyncio.wait_for(
+                                    websocket.recv(),
+                                    timeout=config["receive_timeout_seconds"],
+                                )
+                            except TimeoutError:
+                                await update_ais_source_health(
+                                    db,
+                                    source=self.name,
+                                    connection_state="connected",
+                                    last_success_at=datetime.now(UTC),
+                                )
+                                await db.commit()
+                                continue
+
+                            payload = json.loads(raw_message)
+                            if not isinstance(payload, dict):
+                                continue
+                            messages_seen += 1
+                            record = self._normalize_message(payload)
+                            if not record:
+                                continue
+                            unique_mmsi.add(str(record["mmsi"]))
+                            created = await self._save_stream_record(db, record)
+                            if created:
+                                records_added += 1
+
+                            task.records_processed = messages_seen
+                            task.total_records = None
+                            task.progress = None
+                            task.phase = "streaming"
+                            task.phase_message = "正在接收 AISStream 实时消息"
+                            task.phase_current = messages_seen
+                            task.phase_total = None
+                            task.phase_unit = "messages"
+                            await self._publish_task_update(force=True)
+
+                            if config["streaming_max_messages"] and messages_seen >= config["streaming_max_messages"]:
+                                task.status = "success"
+                                task.phase = "stopped"
+                                task.phase_message = "AISStream 测试流已停止"
+                                task.completed_at = datetime.now(UTC)
+                                await db.commit()
+                                await self._publish_task_update(force=True)
+                                return {
+                                    "status": "success",
+                                    "task_id": task.id,
+                                    "records_processed": records_added,
+                                    "messages_seen": messages_seen,
+                                    "unique_mmsi": len(unique_mmsi),
+                                    "execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
+                                }
+                except asyncio.CancelledError:
+                    raise
+                except Exception as exc:
+                    await update_ais_source_health(
+                        db,
+                        source=self.name,
+                        connection_state="reconnecting",
+                        last_error=f"{exc.__class__.__name__}: {exc}",
+                    )
+                    task.phase = "reconnecting"
+                    task.phase_message = "AISStream 连接中断，正在重连"
+                    task.error_message = f"{exc.__class__.__name__}: {exc}"
+                    await db.commit()
+                    await self._publish_task_update(force=True)
+                    await asyncio.sleep(reconnect_delay)
+        except asyncio.CancelledError:
+            task.status = "cancelled"
+            task.phase = "stopped"
+            task.phase_message = "AISStream 实时流已停止"
+            task.completed_at = datetime.now(UTC)
+            await update_ais_source_health(
+                db,
+                source=self.name,
+                connection_state="disconnected",
+                last_error=None,
+            )
+            await db.commit()
+            await self._publish_task_update(force=True)
+            raise
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        records = []
+        for item in raw_data:
+            record = self._normalize_message(item)
+            if record:
+                records.append(record)
+        return records
+
+    async def _save_data(
+        self,
+        db: AsyncSession,
+        data: list[dict[str, Any]],
+        task_id: int | None = None,
+        snapshot_id: int | None = None,
+    ) -> int:
+        now = datetime.now(UTC)
+        records_added = 0
+        latest_observed_at = now
+        for index, item in enumerate(data):
+            observed_at = item.get("received_at") or now
+            observation = await record_vessel_ais_observation(
+                db,
+                source=self.name,
+                normalized_payload=item,
+                raw_payload=item.get("_raw_payload") or item,
+                delivery_mode=AISSTREAM_DELIVERY_MODE,
+                transport=AISSTREAM_TRANSPORT,
+                message_type=item.get("_message_type") or "PositionReport",
+                source_message_id=item.get("_source_message_id"),
+                observed_at=observed_at,
+                collected_at=now,
+            )
+            if observation is not None:
+                records_added += 1
+            if isinstance(observed_at, datetime) and observed_at > latest_observed_at:
+                latest_observed_at = observed_at
+            if (index + 1) % 1000 == 0:
+                await self.update_progress(index + 1, commit=True)
+
+        await update_ais_source_health(
+            db,
+            source=self.name,
+            connection_state="connected",
+            observed_count=len(data),
+            last_seen_at=latest_observed_at,
+            last_success_at=now if data else None,
+            lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
+        )
+        await db.commit()
+        await self.update_progress(records_added, force=True)
+        return records_added
+
+    async def _save_stream_record(self, db: AsyncSession, item: dict[str, Any]) -> bool:
+        now = datetime.now(UTC)
+        observed_at = item.get("received_at") or now
+        observation = await record_vessel_ais_observation(
+            db,
+            source=self.name,
+            normalized_payload=item,
+            raw_payload=item.get("_raw_payload") or item,
+            delivery_mode=AISSTREAM_DELIVERY_MODE,
+            transport=AISSTREAM_TRANSPORT,
+            message_type=item.get("_message_type") or "PositionReport",
+            source_message_id=item.get("_source_message_id"),
+            observed_at=observed_at,
+            collected_at=now,
+        )
+        await update_ais_source_health(
+            db,
+            source=self.name,
+            connection_state="connected",
+            observed_count=1,
+            last_seen_at=observed_at if isinstance(observed_at, datetime) else now,
+            last_success_at=now,
+            lag_seconds=max((now - observed_at).total_seconds(), 0) if isinstance(observed_at, datetime) else None,
+        )
+        await db.commit()
+        await self._broadcast_vessel_delta(item, created=observation is not None)
+        return observation is not None
+
+    async def _broadcast_vessel_delta(self, item: dict[str, Any], *, created: bool) -> None:
+        await broadcaster.broadcast_custom(
+            "vessels",
+            {
+                "action": "upsert",
+                "source": self.name,
+                "created": created,
+                "vessels": [
+                    {
+                        "mmsi": item.get("mmsi"),
+                        "mmsi_display": str(item.get("mmsi")) if item.get("mmsi") is not None else None,
+                        "name": item.get("name"),
+                        "lat": item.get("lat"),
+                        "lon": item.get("lon"),
+                        "sog": item.get("sog"),
+                        "cog": item.get("cog"),
+                        "heading": item.get("heading"),
+                        "nav_status": item.get("nav_status"),
+                        "vessel_type": item.get("vessel_type"),
+                        "vessel_type_name": item.get("vessel_type_name"),
+                        "received_at": to_iso8601_utc(item.get("received_at")),
+                    }
+                ],
+            },
+        )
+
+    def _normalize_message(self, item: dict[str, Any]) -> dict[str, Any] | None:
+        message_type = str(item.get("MessageType") or item.get("message_type") or "")
+        metadata = item.get("MetaData") if isinstance(item.get("MetaData"), dict) else {}
+        message = item.get("Message") if isinstance(item.get("Message"), dict) else {}
+        body = message.get(message_type) if isinstance(message.get(message_type), dict) else message
+        if not isinstance(body, dict):
+            body = {}
+
+        mmsi = _as_int(_pick(metadata, "MMSI", "mmsi") or _pick(body, "MMSI", "mmsi"))
+        if mmsi is None:
+            return None
+
+        received_at = _parse_datetime(
+            _pick(metadata, "time_utc", "Time_UTC", "timestamp")
+            or _pick(body, "Timestamp", "timestamp", "time")
+        )
+        ship_name = _clean_text(
+            _pick(body, "Name", "ShipName", "name")
+            or _pick(metadata, "ShipName", "ship_name", "name")
+        )
+        record: dict[str, Any] = {
+            "mmsi": mmsi,
+            "received_at": received_at,
+            "_message_type": message_type or None,
+            "_source_message_id": item.get("MessageID") or item.get("message_id"),
+            "_raw_payload": item,
+        }
+
+        lat = _as_float(_pick(body, "Latitude", "lat", "latitude"))
+        lon = _as_float(_pick(body, "Longitude", "lon", "lng", "longitude"))
+        if lat is not None and lon is not None:
+            if not (-90 <= lat <= 90 and -180 <= lon <= 180):
+                return None
+            record.update(
+                {
+                    "lat": lat,
+                    "lon": lon,
+                    "sog": _as_float(_pick(body, "Sog", "SOG", "speedOverGround")),
+                    "cog": _as_float(_pick(body, "Cog", "COG", "courseOverGround")),
+                    "heading": _as_int(_pick(body, "TrueHeading", "Heading", "heading")),
+                    "nav_status": _as_int(_pick(body, "NavigationalStatus", "nav_status")),
+                }
+            )
+
+        vessel_type = _as_int(_pick(body, "Type", "ShipType", "vessel_type"))
+        record.update(
+            {
+                "name": ship_name,
+                "callsign": _pick(body, "CallSign", "callsign"),
+                "imo": _as_int(_pick(body, "ImoNumber", "IMO", "imo")),
+                "vessel_type": vessel_type,
+                "vessel_type_name": _pick(body, "TypeName", "ShipTypeName", "vessel_type_name")
+                or normalize_vessel_type_name(vessel_type),
+                "length": _as_float(_pick(body, "DimensionToBow", "Length", "length")),
+                "width": _as_float(_pick(body, "DimensionToPort", "Width", "width")),
+            }
+        )
+        return record
+
+
+def _pick(item: dict[str, Any], *keys: str) -> Any:
+    for key in keys:
+        if key in item and item[key] not in (None, ""):
+            return item[key]
+    return None
+
+
+def _clean_text(value: Any) -> str | None:
+    if value in (None, ""):
+        return None
+    text = str(value).strip()
+    return text or None
+
+
+def _as_float(value: Any) -> float | None:
+    try:
+        if value in (None, ""):
+            return None
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _as_int(value: Any) -> int | None:
+    try:
+        if value in (None, ""):
+            return None
+        return int(float(value))
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_datetime(value: Any) -> datetime | None:
+    if isinstance(value, datetime):
+        return value if value.tzinfo else value.replace(tzinfo=UTC)
+    if not value:
+        return None
+    if isinstance(value, (int, float)):
+        timestamp = float(value)
+        if timestamp > 10_000_000_000:
+            timestamp /= 1000
+        return datetime.fromtimestamp(timestamp, UTC)
+    if isinstance(value, str):
+        try:
+            parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+            return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+        except ValueError:
+            return None
+    return None

backend/app/services/collectors/base.py

@@ -54,6 +54,11 @@ class BaseCollector(ABC):
                 "task_id": self._current_task.id,
                 "status": self._current_task.status,
                 "phase": self._current_task.phase,
+                "phase_progress": self._current_task.phase_progress,
+                "phase_message": self._current_task.phase_message,
+                "phase_current": self._current_task.phase_current,
+                "phase_total": self._current_task.phase_total,
+                "phase_unit": self._current_task.phase_unit,
                 "progress": progress,
                 "records_processed": self._current_task.records_processed,
                 "total_records": self._current_task.total_records,
@@ -80,12 +85,52 @@ class BaseCollector(ABC):
 
             await self._publish_task_update(force=force)
 
-    async def set_phase(self, phase: str):
+    async def set_phase(self, phase: str, *, message: str | None = None, reset_progress: bool = True):
         if self._current_task and self._db_session:
             self._current_task.phase = phase
+            self._current_task.phase_message = message
+            if reset_progress:
+                self._current_task.phase_progress = None
+                self._current_task.phase_current = None
+                self._current_task.phase_total = None
+                self._current_task.phase_unit = None
             await self._db_session.commit()
             await self._publish_task_update(force=True)
 
+    async def update_phase_progress(
+        self,
+        *,
+        current: int | None = None,
+        total: int | None = None,
+        unit: str | None = None,
+        message: str | None = None,
+        progress: float | None = None,
+        commit: bool = False,
+        force: bool = False,
+    ):
+        """Update progress for the current phase without changing task totals."""
+        if not self._current_task or not self._db_session:
+            return
+
+        if progress is None and current is not None and total and total > 0:
+            progress = (current / total) * 100
+
+        if progress is not None:
+            self._current_task.phase_progress = max(0.0, min(float(progress), 100.0))
+        if current is not None:
+            self._current_task.phase_current = max(0, int(current))
+        if total is not None:
+            self._current_task.phase_total = max(0, int(total))
+        if unit is not None:
+            self._current_task.phase_unit = unit
+        if message is not None:
+            self._current_task.phase_message = message
+
+        if commit:
+            await self._db_session.commit()
+
+        await self._publish_task_update(force=force)
+
     @abstractmethod
     async def fetch(self) -> List[Dict[str, Any]]:
         """Fetch raw data from source"""
@@ -251,7 +296,7 @@ class BaseCollector(ABC):
         await self._publish_task_update(force=True)
 
         try:
-            await self.set_phase("fetching")
+            await self.set_phase("fetching", message="正在拉取原始数据")
             raw_data = await self.fetch()
             task.total_records = len(raw_data)
             await db.commit()
@@ -260,15 +305,20 @@ class BaseCollector(ABC):
             if self.fail_on_empty and not raw_data:
                 raise RuntimeError(f"Collector {self.name} returned no data")
 
-            await self.set_phase("transforming")
+            await self.set_phase("transforming", message="正在转换采集数据")
             data = self.transform(raw_data)
             snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
 
-            await self.set_phase("saving")
+            await self.set_phase("saving", message="正在保存采集数据")
             records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
 
             task.status = "success"
             task.phase = "completed"
+            task.phase_progress = 100.0
+            task.phase_message = "采集完成"
+            task.phase_current = records_count
+            task.phase_total = records_count
+            task.phase_unit = "records"
             task.records_processed = records_count
             task.progress = 100.0
             task.completed_at = datetime.now(UTC)
@@ -285,6 +335,7 @@ class BaseCollector(ABC):
             await db.rollback()
             task.status = "cancelled"
             task.phase = "cancelled"
+            task.phase_message = "采集已取消"
             task.error_message = "Collection cancelled by operator and rolled back"
             task.completed_at = datetime.now(UTC)
             if snapshot_id is not None:
@@ -301,6 +352,7 @@ class BaseCollector(ABC):
             await db.rollback()
             task.status = "failed"
             task.phase = "failed"
+            task.phase_message = str(e)
             task.error_message = str(e)
             task.completed_at = datetime.now(UTC)
             if snapshot_id is not None:

backend/app/services/collectors/celestrak.py

@@ -46,6 +46,9 @@ class CelesTrakTLECollector(BaseCollector):
                     if response.status_code == 200:
                         data = response.json()
                         if isinstance(data, list):
+                            for item in data:
+                                if isinstance(item, dict):
+                                    item["_celestrak_group"] = group
                             all_satellites.extend(data)
                             print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
                 except Exception as e:
@@ -78,6 +81,7 @@ class CelesTrakTLECollector(BaseCollector):
                     "name": item.get("OBJECT_NAME", "Unknown"),
                     "reference_date": item.get("EPOCH", ""),
                     "metadata": {
+                        "constellation_group": item.get("_celestrak_group"),
                         "norad_cat_id": item.get("NORAD_CAT_ID"),
                         "international_designator": item.get("OBJECT_ID"),
                         "epoch": item.get("EPOCH"),

backend/app/services/collectors/iptoasn.py

@@ -108,6 +108,11 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
                 self._current_task.total_records = total_expected
                 self._current_task.records_processed = 0
                 self._current_task.progress = 0.0
+                self._current_task.phase_progress = 0.0
+                self._current_task.phase_message = "正在下载 IPtoASN 数据"
+                self._current_task.phase_current = 0
+                self._current_task.phase_total = total_expected
+                self._current_task.phase_unit = "bytes"
                 await self._db_session.commit()
                 await self._publish_task_update(force=True)
 
@@ -135,7 +140,14 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
                         return
                     last_emit["value"] = aggregated
                     last_emit["t"] = now
-                await self.update_progress(min(aggregated, total_expected), commit=True)
+                current = min(aggregated, total_expected)
+                await self.update_phase_progress(
+                    current=current,
+                    total=total_expected,
+                    unit="bytes",
+                    message="正在下载 IPtoASN 数据",
+                )
+                await self.update_progress(current, commit=True)
 
             batches = await asyncio.gather(
                 *(
@@ -148,6 +160,12 @@ class IPtoASNPrefixGeoCollector(BaseCollector):
                 )
             )
             if total_expected > 0:
+                await self.update_phase_progress(
+                    current=total_expected,
+                    total=total_expected,
+                    unit="bytes",
+                    message="IPtoASN 数据下载完成",
+                )
                 await self.update_progress(total_expected, commit=True, force=True)
 
         rows: list[dict[str, Any]] = []

backend/app/services/collectors/nro_delegated.py

@@ -39,12 +39,23 @@ class NRODelegatedPrefixGeoCollector(BaseCollector):
                 self._current_task.total_records = total_expected
                 self._current_task.records_processed = 0
                 self._current_task.progress = 0.0
+                self._current_task.phase_progress = 0.0
+                self._current_task.phase_message = "正在下载 NRO delegated 数据"
+                self._current_task.phase_current = 0
+                self._current_task.phase_total = total_expected
+                self._current_task.phase_unit = "bytes"
                 await self._db_session.commit()
                 await self._publish_task_update(force=True)
 
             async def on_progress(downloaded: int, total: int | None) -> None:
                 if not total or total <= 0:
                     return
+                await self.update_phase_progress(
+                    current=min(downloaded, total),
+                    total=total,
+                    unit="bytes",
+                    message="正在下载 NRO delegated 数据",
+                )
                 await self.update_progress(min(downloaded, total), commit=True)
 
             body_path = await self._downloader.download_file(

backend/app/services/collectors/opengeofeed.py

@@ -40,12 +40,23 @@ class OpenGeoFeedPrefixGeoCollector(BaseCollector):
                 self._current_task.total_records = total_expected
                 self._current_task.records_processed = 0
                 self._current_task.progress = 0.0
+                self._current_task.phase_progress = 0.0
+                self._current_task.phase_message = "正在下载 OpenGeoFeed 数据"
+                self._current_task.phase_current = 0
+                self._current_task.phase_total = total_expected
+                self._current_task.phase_unit = "bytes"
                 await self._db_session.commit()
                 await self._publish_task_update(force=True)
 
             async def on_progress(downloaded: int, total: int | None) -> None:
                 if not total or total <= 0:
                     return
+                await self.update_phase_progress(
+                    current=min(downloaded, total),
+                    total=total,
+                    unit="bytes",
+                    message="正在下载 OpenGeoFeed 数据",
+                )
                 await self.update_progress(min(downloaded, total), commit=True)
 
             body_path = await self._downloader.download_file(

backend/app/services/collectors/vessel_ais.py

@@ -0,0 +1,279 @@
+"""BarentsWatch AIS collector for vessel tracking."""
+
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.time import to_iso8601_utc
+from app.core.websocket.broadcaster import broadcaster
+from app.services.barentswatch import (
+    BARENTSWATCH_LATEST_URL,
+    fetch_barentswatch_access_token,
+    resolve_barentswatch_config,
+)
+from app.services.collectors.base import BaseCollector
+from app.services.vessel_ais_aggregation import (
+    BARENTSWATCH_DELIVERY_MODE,
+    BARENTSWATCH_TRANSPORT,
+    record_vessel_ais_observation,
+    update_ais_source_health,
+)
+from app.services.vessel_types import normalize_vessel_type_name
+
+
+class VesselAISCollector(BaseCollector):
+    """Collect latest AIS positions and append them to vessel tables."""
+
+    name = "barentswatch_vessels"
+    priority = "P1"
+    module = "L4"
+    frequency_hours = 1
+    data_type = "vessel_ais"
+
+    @property
+    def base_url(self) -> str:
+        return self._resolved_url or BARENTSWATCH_LATEST_URL
+
+    async def _get_access_token(self, client: httpx.AsyncClient) -> str | None:
+        config = await resolve_barentswatch_config(self._db_session)
+        return await fetch_barentswatch_access_token(client, config)
+
+    async def fetch(self) -> list[dict[str, Any]]:
+        async with httpx.AsyncClient(timeout=60.0) as client:
+            headers: dict[str, str] = {}
+            token = await self._get_access_token(client)
+            if token:
+                headers["Authorization"] = f"Bearer {token}"
+
+            response = await client.get(self.base_url, headers=headers)
+            if response.status_code == 401 and not token:
+                return self._get_sample_data()
+            response.raise_for_status()
+            payload = response.json()
+
+        if isinstance(payload, list):
+            return [item for item in payload if isinstance(item, dict)]
+        if isinstance(payload, dict):
+            for key in ("features", "data", "items", "vessels"):
+                value = payload.get(key)
+                if isinstance(value, list):
+                    if key == "features":
+                        return [
+                            {
+                                **(item.get("properties") or {}),
+                                "geometry": item.get("geometry"),
+                            }
+                            for item in value
+                            if isinstance(item, dict)
+                        ]
+                    return [item for item in value if isinstance(item, dict)]
+        return self._get_sample_data()
+
+    def transform(self, raw_data: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        transformed = []
+        for item in raw_data:
+            record = self._normalize_record(item)
+            if record:
+                transformed.append(record)
+        return transformed
+
+    async def _save_data(
+        self,
+        db: AsyncSession,
+        data: list[dict[str, Any]],
+        task_id: int | None = None,
+        snapshot_id: int | None = None,
+    ) -> int:
+        now = datetime.now(UTC)
+        records_added = 0
+
+        for index, item in enumerate(data):
+            observed_at = item.get("received_at") or now
+            await record_vessel_ais_observation(
+                db,
+                source=self.name,
+                normalized_payload=item,
+                raw_payload=item,
+                delivery_mode=BARENTSWATCH_DELIVERY_MODE,
+                transport=BARENTSWATCH_TRANSPORT,
+                observed_at=observed_at,
+                collected_at=now,
+            )
+            records_added += 1
+
+            if (index + 1) % 1000 == 0:
+                await self.update_progress(index + 1, commit=True)
+
+        latest_observed_at = max(
+            (item.get("received_at") for item in data if item.get("received_at")),
+            default=now,
+        )
+        await update_ais_source_health(
+            db,
+            source=self.name,
+            connection_state="connected",
+            observed_count=len(data),
+            last_seen_at=latest_observed_at,
+            last_success_at=now if data else None,
+            lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
+        )
+        await db.commit()
+        await self._broadcast_vessel_snapshot(data)
+        await self.update_progress(records_added, force=True)
+        return records_added
+
+    async def _broadcast_vessel_snapshot(self, data: list[dict[str, Any]]) -> None:
+        """Push REST collector updates through the same realtime vessel channel."""
+        if not data:
+            return
+
+        batch_size = 500
+        for offset in range(0, len(data), batch_size):
+            batch = data[offset : offset + batch_size]
+            await broadcaster.broadcast_custom(
+                "vessels",
+                {
+                    "action": "upsert",
+                    "source": self.name,
+                    "created": True,
+                    "vessels": [
+                        {
+                            "mmsi": item.get("mmsi"),
+                            "mmsi_display": str(item.get("mmsi")) if item.get("mmsi") is not None else None,
+                            "name": item.get("name"),
+                            "callsign": item.get("callsign"),
+                            "lat": item.get("lat"),
+                            "lon": item.get("lon"),
+                            "sog": item.get("sog"),
+                            "cog": item.get("cog"),
+                            "heading": item.get("heading"),
+                            "nav_status": item.get("nav_status"),
+                            "vessel_type": item.get("vessel_type"),
+                            "vessel_type_name": item.get("vessel_type_name"),
+                            "received_at": to_iso8601_utc(item.get("received_at")),
+                        }
+                        for item in batch
+                    ],
+                },
+            )
+
+    def _normalize_record(self, item: dict[str, Any]) -> dict[str, Any] | None:
+        mmsi = _as_int(_pick(item, "mmsi", "MMSI", "Mmsi"))
+        lat = _as_float(_pick(item, "lat", "latitude", "Latitude"))
+        lon = _as_float(_pick(item, "lon", "lng", "longitude", "Longitude"))
+
+        geometry = item.get("geometry")
+        coordinates = geometry.get("coordinates") if isinstance(geometry, dict) else None
+        if (lat is None or lon is None) and isinstance(coordinates, list) and len(coordinates) >= 2:
+            lon = _as_float(coordinates[0])
+            lat = _as_float(coordinates[1])
+
+        if mmsi is None or lat is None or lon is None:
+            return None
+        if not (-90 <= lat <= 90 and -180 <= lon <= 180):
+            return None
+
+        vessel_type = _as_int(_pick(item, "vessel_type", "shipType", "ship_type", "ShipType"))
+        vessel_type_name = (
+            _pick(item, "vessel_type_name", "shipTypeName", "ship_type_name", "VesselTypeName")
+            or normalize_vessel_type_name(vessel_type)
+        )
+        received_at = _parse_datetime(_pick(item, "received_at", "timestamp", "time", "msgtime"))
+
+        return {
+            "mmsi": mmsi,
+            "name": _pick(item, "name", "shipName", "ship_name", "Name"),
+            "callsign": _pick(item, "callsign", "callSign", "CallSign"),
+            "vessel_type": vessel_type,
+            "vessel_type_name": vessel_type_name,
+            "flag": _pick(item, "flag", "country", "Flag"),
+            "length": _as_float(_pick(item, "length", "shipLength", "Length")),
+            "width": _as_float(_pick(item, "width", "shipWidth", "Width")),
+            "draught": _as_float(_pick(item, "draught", "draft", "Draught")),
+            "imo": _as_int(_pick(item, "imo", "IMO", "imoNumber")),
+            "lat": lat,
+            "lon": lon,
+            "sog": _as_float(_pick(item, "sog", "speedOverGround", "SOG")),
+            "cog": _as_float(_pick(item, "cog", "courseOverGround", "COG")),
+            "heading": _as_int(_pick(item, "heading", "trueHeading", "Heading")),
+            "nav_status": _as_int(_pick(item, "nav_status", "navStatus", "NavigationalStatus")),
+            "received_at": received_at,
+        }
+
+    def _get_sample_data(self) -> list[dict[str, Any]]:
+        return [
+            {
+                "mmsi": 257123000,
+                "name": "OSLO TRADER",
+                "lat": 59.91,
+                "lon": 10.73,
+                "sog": 12.4,
+                "cog": 214,
+                "heading": 215,
+                "nav_status": 0,
+                "vessel_type": 70,
+                "vessel_type_name": "Cargo",
+                "flag": "NO",
+                "length": 185,
+            },
+            {
+                "mmsi": 257456000,
+                "name": "NORDIC FJORD",
+                "lat": 60.39,
+                "lon": 5.32,
+                "sog": 0.2,
+                "cog": 82,
+                "heading": 80,
+                "nav_status": 1,
+                "vessel_type": 60,
+                "vessel_type_name": "Passenger",
+                "flag": "NO",
+                "length": 126,
+            },
+        ]
+
+
+def _pick(item: dict[str, Any], *keys: str) -> Any:
+    for key in keys:
+        if key in item and item[key] not in (None, ""):
+            return item[key]
+    return None
+
+
+def _as_float(value: Any) -> float | None:
+    try:
+        if value in (None, ""):
+            return None
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _as_int(value: Any) -> int | None:
+    try:
+        if value in (None, ""):
+            return None
+        return int(float(value))
+    except (TypeError, ValueError):
+        return None
+
+
+def _parse_datetime(value: Any) -> datetime | None:
+    if isinstance(value, datetime):
+        return value if value.tzinfo else value.replace(tzinfo=UTC)
+    if not value:
+        return None
+    if isinstance(value, (int, float)):
+        timestamp = float(value)
+        if timestamp > 10_000_000_000:
+            timestamp /= 1000
+        return datetime.fromtimestamp(timestamp, UTC)
+    if isinstance(value, str):
+        try:
+            parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+            return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+        except ValueError:
+            return None
+    return None

backend/app/services/credential_guides.py

@@ -0,0 +1,224 @@
+"""Credential setup guides for collector integrations."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from sqlalchemy import select
+
+from app.models.system_setting import SystemSetting
+from app.schemas.ai import SituationalAnalysisRequest
+from app.services.ai_client import AIProviderClient
+
+
+CREDENTIAL_GUIDES_CATEGORY = "collector_credential_guides"
+
+
+@dataclass(frozen=True)
+class CredentialGuideDefault:
+    provider: str
+    title: str
+    prompt: str
+    markdown: str
+
+
+BARENTSWATCH_DEFAULT_GUIDE = CredentialGuideDefault(
+    provider="barentswatch",
+    title="BarentsWatch AIS 凭证获取教程",
+    prompt=(
+        "请生成一份中文教程，指导开发者获取 BarentsWatch Live AIS API 的 "
+        "OAuth client credentials。教程要面向已经有本地开发环境的人，包含注册/登录、"
+        "创建 client、申请或确认 ais scope、复制 client id 和 client secret、"
+        "在系统设置中填写并验证连接、常见失败排查。不要编造具体页面按钮文案，"
+        "必须参考官方 tutorial：https://developer.barentswatch.no/docs/tutorial 。"
+        "必须强调 Live AIS 要选择 AIS-client / AIS - API，而不是普通 API-client。"
+        "如果步骤可能变化，要提醒以 BarentsWatch developer portal 当前页面为准。"
+    ),
+    markdown="""## BarentsWatch AIS 凭证获取
+
+官方教程：https://developer.barentswatch.no/docs/tutorial
+
+1. 先打开上面的 BarentsWatch 官方 tutorial，按官方流程登录或注册开发者账号。
+2. 在 Developer access 页面选择 `AIS - API`，不要选择普通的 `BarentsWatch - API`。
+3. 在 `AIS - API` 下创建用于 Planet 的 AIS client。
+4. 创建时记下你设置的 password / client secret。
+5. 回到 My Page 复制完整 `Client ID`。它通常长得像 `your.email@example.com:client-name`。
+6. 回到 Planet 的 `设置 -> 采集器设置 -> BarentsWatch AIS`，填入 `Client ID` 和 `Client Secret`。
+7. 点击 `连接` 验证 token 和 AIS endpoint 是否可访问。
+8. 连接成功后保存凭证。
+
+### 请求规则
+
+- Token 地址：`https://id.barentswatch.no/connect/token`
+- 请求方式：`POST`
+- Content-Type：`application/x-www-form-urlencoded`
+- Body 必须包含：`grant_type=client_credentials`、`client_id`、`client_secret`、`scope=ais`
+- `client_id`、`client_secret`、`scope`、`grant_type` 都要放在 body，不要放在 header。
+- AIS 数据请求使用 header：`Authorization: Bearer <access_token>`
+
+### 常见排查
+
+- `未找到凭证`：确认 `Client ID` 和 `Client Secret` 已填写，或已经写入 `~/.zshrc`。
+- `HTTP 401/403`：通常是选成了普通 `BarentsWatch - API` client、client secret 错误，或 token 请求没有使用 `scope=ais`。
+- `network` 错误：检查本机是否能访问 `id.barentswatch.no` 和 `live.ais.barentswatch.no`。
+- Endpoint 建议保持默认：`https://live.ais.barentswatch.no/v1/latest/combined`。
+""",
+)
+
+AISSTREAM_DEFAULT_GUIDE = CredentialGuideDefault(
+    provider="aisstream",
+    title="AISStream API Key 获取教程",
+    prompt=(
+        "请生成一份中文教程，指导开发者获取 AISStream 的 API Key 并配置到 Planet。"
+        "教程要面向已经有本地开发环境的人，包含注册/登录 AISStream、获取 API Key、"
+        "理解免费额度和订阅范围、在 Planet 设置中心填写 API Key、配置 bounding boxes "
+        "和 message types、验证连接、常见失败排查。必须提醒用户以 AISStream 当前官网和"
+        "服务条款为准，不要编造具体页面按钮文案。"
+    ),
+    markdown="""## AISStream API Key 获取
+
+官方入口：https://aisstream.io/
+
+1. 打开 AISStream 官网，按当前页面指引注册或登录账号。
+2. 在账号/API 管理页面创建或复制你的 API Key。
+3. 先确认当前账号额度、使用条款和可订阅区域。实时 AIS 流量可能很大，不建议一开始订阅全球范围。
+4. 回到 Planet 的 `设置 -> 采集器设置 -> AISStream 实时船舶`。
+5. 在 `AISStream 凭证` 中填入 API Key。
+6. Endpoint 通常保持默认：`wss://stream.aisstream.io/v0/stream`。
+7. 按需配置 `Bounding Boxes JSON` 和 `消息类型`。
+8. 点击连接测试，确认系统能读取凭证且 WebSocket endpoint 格式有效。
+9. 保存采集器设置后再运行 `aisstream_vessels` collector。
+
+### 推荐配置
+
+默认消息类型：
+
+```json
+["PositionReport", "ShipStaticData"]
+```
+
+默认 Bounding Boxes 示例：
+
+```json
+[[[-90, -180], [90, 180]]]
+```
+
+这个示例表示全球范围。实际使用时建议先改成较小区域，降低消息量和处理压力。
+
+### 请求规则
+
+- Endpoint：`wss://stream.aisstream.io/v0/stream`
+- 传输方式：WebSocket
+- API Key 放在订阅 payload 中，不放在 HTTP header。
+- Planet 会把 AISStream 标记为 `delivery_mode = realtime_stream`、`transport = websocket`。
+- AISStream collector 只写入 AIS raw observations，不直接覆盖最终船只展示表。
+
+### 常见排查
+
+- `未找到凭证`：确认 API Key 已保存到采集器设置，或设置了 `AISSTREAM_API_KEY` 环境变量 / `~/.zshrc`。
+- `endpoint 必须是 ws:// 或 wss://`：AISStream 是 WebSocket 流接口，不要填普通 `https://` API 地址。
+- 采集量过大：缩小 `Bounding Boxes JSON`，减少 `message_types`，或降低单次最大消息数。
+- 没有船只数据：确认订阅区域内确实有 AIS 活动，并检查 API Key 当前额度和权限。
+- 连接中断：实时流可能受网络和上游限流影响，collector 会记录源健康状态供聚合服务回退。
+""",
+)
+
+
+DEFAULT_CREDENTIAL_GUIDES = {
+    BARENTSWATCH_DEFAULT_GUIDE.provider: BARENTSWATCH_DEFAULT_GUIDE,
+    AISSTREAM_DEFAULT_GUIDE.provider: AISSTREAM_DEFAULT_GUIDE,
+}
+
+
+async def _get_guide_store(db) -> tuple[SystemSetting | None, dict[str, Any]]:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == CREDENTIAL_GUIDES_CATEGORY)
+    )
+    record = result.scalar_one_or_none()
+    payload = dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
+    return record, payload
+
+
+async def get_credential_guide(db, provider: str) -> dict[str, Any]:
+    default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
+    if default is None:
+        raise ValueError(f"Unsupported credential guide provider: {provider}")
+
+    _record, store = await _get_guide_store(db)
+    custom = store.get(provider) if isinstance(store.get(provider), dict) else None
+    return {
+        "provider": provider,
+        "title": custom.get("title") if custom else default.title,
+        "markdown": custom.get("markdown") if custom else default.markdown,
+        "prompt": default.prompt,
+        "source": "ai" if custom else "default",
+    }
+
+
+async def save_credential_guide(db, provider: str, title: str, markdown: str) -> dict[str, Any]:
+    default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
+    if default is None:
+        raise ValueError(f"Unsupported credential guide provider: {provider}")
+
+    record, store = await _get_guide_store(db)
+    store[provider] = {
+        "title": title or default.title,
+        "markdown": markdown,
+    }
+    if record is None:
+        db.add(SystemSetting(category=CREDENTIAL_GUIDES_CATEGORY, payload=store))
+    else:
+        record.payload = store
+    await db.commit()
+    return await get_credential_guide(db, provider)
+
+
+async def reset_credential_guide(db, provider: str) -> dict[str, Any]:
+    default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
+    if default is None:
+        raise ValueError(f"Unsupported credential guide provider: {provider}")
+
+    record, store = await _get_guide_store(db)
+    if provider in store:
+        store.pop(provider, None)
+        if record is not None:
+            record.payload = store
+            await db.commit()
+    return await get_credential_guide(db, provider)
+
+
+async def generate_credential_guide(
+    db,
+    provider: str,
+    ai_client: AIProviderClient,
+) -> dict[str, Any]:
+    default = DEFAULT_CREDENTIAL_GUIDES.get(provider)
+    if default is None:
+        raise ValueError(f"Unsupported credential guide provider: {provider}")
+
+    response = await ai_client.analyze(
+        SituationalAnalysisRequest(
+            title=f"Generate credential guide for {provider}",
+            objective=default.prompt,
+            context={
+                "provider": provider,
+                "current_default_guide": default.markdown,
+                "product_context": "Planet collector credential settings",
+            },
+            observations=[
+                "Use concise Chinese markdown.",
+                "Prefer stable concepts over brittle UI labels.",
+                "Include verification and troubleshooting steps.",
+            ],
+            constraints=[
+                "Do not ask the user for secrets.",
+                "Do not include fabricated screenshots.",
+                "Return markdown only.",
+            ],
+        )
+    )
+    markdown = response.content.strip()
+    if not markdown:
+        markdown = default.markdown
+    return await save_credential_guide(db, provider, default.title, markdown)

backend/app/services/custom_datasource_runtime.py

@@ -0,0 +1,391 @@
+"""Runtime helpers for mapped custom data sources."""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import json
+from datetime import UTC, datetime
+from typing import Any
+
+import httpx
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.target_schema_registry import TARGET_SCHEMAS
+from app.db.session import async_session_factory
+from app.models.datasource_config import DataSourceConfig
+from app.models.datasource_mapping import DataSourceMappingTemplate
+from app.services.datasource_mapping import (
+    MappingError,
+    execute_mapping,
+    extract_path,
+    persist_mapped_records,
+)
+
+DEFAULT_MAPPING_TEMPLATES: dict[str, dict[str, Any]] = {
+    "vessel_ais": {
+        "source": {"items_path": "$"},
+        "fields": {
+            "mmsi": {"path": "$.mmsi", "type": "integer"},
+            "name": {"path": "$.name", "type": "string", "default": None},
+            "lat": {"path": "$.lat", "type": "float"},
+            "lon": {"path": "$.lon", "type": "float"},
+            "sog": {"path": "$.sog", "type": "float", "default": None},
+            "cog": {"path": "$.cog", "type": "float", "default": None},
+            "heading": {"path": "$.heading", "type": "integer", "default": None},
+            "nav_status": {"path": "$.nav_status", "type": "integer", "default": None},
+            "callsign": {"path": "$.callsign", "type": "string", "default": None},
+            "vessel_type": {"path": "$.vessel_type", "type": "string", "default": None},
+            "vessel_type_name": {"path": "$.vessel_type_name", "type": "string", "default": None},
+            "received_at": {"path": "$.received_at", "type": "datetime", "default": None},
+        },
+        "meta": {"generated_by": "default_template", "requires_review": False},
+    },
+}
+
+RUNNING_CUSTOM_STREAM_TASKS: dict[int, asyncio.Task[Any]] = {}
+
+
+class CustomDatasourceRuntimeError(RuntimeError):
+    """Raised when a custom datasource cannot run."""
+
+
+def build_request_headers(auth_type: str, auth_config: dict, headers: dict) -> dict[str, str]:
+    request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
+    auth_type = str(auth_type or "none").lower()
+    auth_config = auth_config or {}
+
+    if auth_type == "bearer" and auth_config.get("token"):
+        request_headers["Authorization"] = f"Bearer {auth_config['token']}"
+    elif auth_type == "api_key" and auth_config.get("api_key"):
+        location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
+        if location != "query":
+            key_name = auth_config.get("key_name", "X-API-Key")
+            request_headers[str(key_name)] = str(auth_config["api_key"])
+    elif auth_type == "basic":
+        username = auth_config.get("username", "")
+        password = auth_config.get("password", "")
+        credentials = f"{username}:{password}"
+        encoded = base64.b64encode(credentials.encode()).decode()
+        request_headers["Authorization"] = f"Basic {encoded}"
+    return request_headers
+
+
+def build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict[str, Any]:
+    params: dict[str, Any] = {}
+    candidate = (config or {}).get("params") or (config or {}).get("query_params")
+    if isinstance(candidate, dict):
+        params.update(candidate)
+
+    auth_type = str(auth_type or "none").lower()
+    auth_config = auth_config or {}
+    if auth_type == "api_key" and auth_config.get("api_key"):
+        location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
+        if location == "query":
+            key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
+            params[str(key_name)] = auth_config["api_key"]
+    return params
+
+
+async def load_active_mapping(
+    db: AsyncSession,
+    datasource_config_id: int,
+) -> DataSourceMappingTemplate:
+    result = await db.execute(
+        select(DataSourceMappingTemplate)
+        .where(DataSourceMappingTemplate.datasource_config_id == datasource_config_id)
+        .where(DataSourceMappingTemplate.is_active.is_(True))
+        .order_by(DataSourceMappingTemplate.version.desc())
+        .limit(1)
+    )
+    mapping = result.scalar_one_or_none()
+    if mapping is not None:
+        return mapping
+
+    datasource = await db.get(DataSourceConfig, datasource_config_id)
+    if datasource is None:
+        raise CustomDatasourceRuntimeError("Configuration not found")
+    target_schema = (datasource.config or {}).get("target_schema")
+    template_body = DEFAULT_MAPPING_TEMPLATES.get(str(target_schema or "")) if target_schema else None
+    if not template_body or target_schema not in TARGET_SCHEMAS:
+        raise CustomDatasourceRuntimeError(
+            "No active mapping template found and no default template available for this target schema"
+        )
+
+    mapping = DataSourceMappingTemplate(
+        datasource_config_id=datasource_config_id,
+        target_schema=str(target_schema),
+        mapping_json=template_body,
+        sample_payload_hash=None,
+        validation_status="valid",
+        version=1,
+        is_active=True,
+    )
+    db.add(mapping)
+    await db.commit()
+    await db.refresh(mapping)
+    return mapping
+
+
+async def fetch_rest_payload(config: DataSourceConfig, limit_bytes: int) -> Any:
+    request_config = config.config or {}
+    method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
+    if method not in {"GET", "POST"}:
+        raise CustomDatasourceRuntimeError("Only GET and POST sample requests are supported.")
+
+    headers = build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
+    params = build_query_params(config.auth_type, config.auth_config or {}, request_config)
+    timeout = float(request_config.get("timeout", 30))
+    json_body = request_config.get("json_body")
+    if json_body is None and str(request_config.get("body_type") or "").lower() in {"json", ""}:
+        candidate = request_config.get("body")
+        if isinstance(candidate, (dict, list)):
+            json_body = candidate
+
+    async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
+        response = await client.request(
+            method,
+            config.endpoint,
+            headers=headers,
+            params=params or None,
+            json=json_body,
+        )
+        response.raise_for_status()
+        content = response.content[:limit_bytes]
+        if "application/json" in response.headers.get("content-type", ""):
+            return json.loads(content.decode(response.encoding or "utf-8"))
+        return {"text": content.decode(response.encoding or "utf-8", errors="replace")}
+
+
+async def run_mapped_rest_config(
+    db: AsyncSession,
+    datasource: DataSourceConfig,
+) -> dict[str, Any]:
+    mapping = await load_active_mapping(db, datasource.id)
+    sample = await fetch_rest_payload(datasource, 5_000_000)
+    mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
+    if mapped["failed_count"] > 0:
+        return {
+            "status": "failed",
+            "datasource_config_id": datasource.id,
+            "mapping_id": mapping.id,
+            "mapping_version": mapping.version,
+            "target_schema": mapping.target_schema,
+            "mapped_count": mapped["mapped_count"],
+            "failed_count": mapped["failed_count"],
+            "errors": mapped["errors"][:20],
+        }
+
+    request_config = datasource.config or {}
+    written_count = await persist_mapped_records(
+        db,
+        datasource_name=datasource.name,
+        datasource_config_id=datasource.id,
+        target_schema=mapping.target_schema,
+        records=mapped["records"],
+        mapping_version=mapping.version,
+        delivery_mode=request_config.get("delivery_mode") or "polling",
+        transport="http",
+    )
+    return {
+        "status": "success",
+        "datasource_config_id": datasource.id,
+        "mapping_id": mapping.id,
+        "mapping_version": mapping.version,
+        "target_schema": mapping.target_schema,
+        "fetched_count": mapped["total_items"],
+        "mapped_count": mapped["mapped_count"],
+        "written_count": written_count,
+    }
+
+
+def _items_from_ws_message(payload: Any, config: dict) -> Any:
+    message_path = config.get("ws_message_path")
+    items_path = config.get("ws_items_path")
+    value = extract_path(payload, message_path) if message_path else payload
+    return extract_path(value, items_path) if items_path else value
+
+
+async def _connect_websocket(endpoint: str, headers: dict[str, str]):
+    import websockets
+
+    try:
+        return await websockets.connect(endpoint, additional_headers=headers or None)
+    except TypeError:
+        return await websockets.connect(endpoint, extra_headers=headers or None)
+
+
+async def test_websocket_config(config: DataSourceConfig) -> dict[str, Any]:
+    if not str(config.endpoint or "").startswith(("ws://", "wss://")):
+        raise CustomDatasourceRuntimeError("WebSocket datasource endpoint must start with ws:// or wss://")
+
+    runtime_config = config.config or {}
+    headers = build_request_headers(config.auth_type, config.auth_config or {}, config.headers or {})
+    receive_timeout = float(runtime_config.get("receive_timeout_seconds") or runtime_config.get("timeout") or 10)
+    async with await _connect_websocket(config.endpoint, headers) as websocket:
+        subscribe_message = runtime_config.get("ws_subscribe_message")
+        if isinstance(subscribe_message, (dict, list)):
+            await websocket.send(json.dumps(subscribe_message))
+        elif isinstance(subscribe_message, str) and subscribe_message.strip():
+            await websocket.send(subscribe_message)
+        raw_message = await asyncio.wait_for(websocket.recv(), timeout=receive_timeout)
+        return {
+            "success": True,
+            "message_preview": raw_message[:1000] if isinstance(raw_message, str) else str(raw_message)[:1000],
+        }
+
+
+async def run_mapped_websocket_config(
+    db: AsyncSession,
+    datasource: DataSourceConfig,
+    *,
+    debug_max_messages: int | None = None,
+    use_config_debug_max_messages: bool = True,
+) -> dict[str, Any]:
+    if not str(datasource.endpoint or "").startswith(("ws://", "wss://")):
+        raise CustomDatasourceRuntimeError("WebSocket datasource endpoint must start with ws:// or wss://")
+
+    mapping = await load_active_mapping(db, datasource.id)
+    runtime_config = datasource.config or {}
+    max_messages = debug_max_messages
+    if max_messages is None and use_config_debug_max_messages:
+        max_messages = runtime_config.get("debug_max_messages")
+    max_messages = int(max_messages) if max_messages else None
+    receive_timeout = float(runtime_config.get("receive_timeout_seconds") or runtime_config.get("timeout") or 30)
+    reconnect = bool(runtime_config.get("ws_reconnect", True))
+    reconnect_delay = float(runtime_config.get("reconnect_delay_seconds") or 3)
+    headers = build_request_headers(datasource.auth_type, datasource.auth_config or {}, datasource.headers or {})
+
+    messages_seen = 0
+    mapped_count = 0
+    failed_count = 0
+    written_count = 0
+    errors: list[dict[str, Any]] = []
+    started_at = datetime.now(UTC)
+
+    while True:
+        try:
+            async with await _connect_websocket(datasource.endpoint, headers) as websocket:
+                subscribe_message = runtime_config.get("ws_subscribe_message")
+                if isinstance(subscribe_message, (dict, list)):
+                    await websocket.send(json.dumps(subscribe_message))
+                elif isinstance(subscribe_message, str) and subscribe_message.strip():
+                    await websocket.send(subscribe_message)
+
+                while True:
+                    raw_message = await asyncio.wait_for(websocket.recv(), timeout=receive_timeout)
+                    messages_seen += 1
+                    try:
+                        payload = json.loads(raw_message)
+                    except json.JSONDecodeError as exc:
+                        failed_count += 1
+                        errors.append({"message": "invalid_json", "error": str(exc)})
+                        continue
+
+                    extracted = _items_from_ws_message(payload, runtime_config)
+                    try:
+                        mapped = execute_mapping(extracted, mapping.mapping_json, mapping.target_schema)
+                    except (MappingError, ValueError) as exc:
+                        failed_count += 1
+                        errors.append({"message": "mapping_failed", "error": str(exc)})
+                        continue
+
+                    mapped_count += mapped["mapped_count"]
+                    failed_count += mapped["failed_count"]
+                    if mapped["errors"]:
+                        errors.extend(mapped["errors"][:5])
+                    if mapped["records"]:
+                        written_count += await persist_mapped_records(
+                            db,
+                            datasource_name=datasource.name,
+                            datasource_config_id=datasource.id,
+                            target_schema=mapping.target_schema,
+                            records=mapped["records"],
+                            mapping_version=mapping.version,
+                            delivery_mode=runtime_config.get("delivery_mode") or "realtime_stream",
+                            transport="websocket",
+                        )
+
+                    if max_messages and messages_seen >= max_messages:
+                        return {
+                            "status": "success",
+                            "datasource_config_id": datasource.id,
+                            "mapping_id": mapping.id,
+                            "mapping_version": mapping.version,
+                            "target_schema": mapping.target_schema,
+                            "messages_seen": messages_seen,
+                            "mapped_count": mapped_count,
+                            "failed_count": failed_count,
+                            "written_count": written_count,
+                            "errors": errors[:20],
+                            "execution_time_seconds": (datetime.now(UTC) - started_at).total_seconds(),
+                        }
+        except asyncio.CancelledError:
+            raise
+        except Exception as exc:
+            failed_count += 1
+            errors.append({"message": "websocket_error", "error": f"{exc.__class__.__name__}: {exc}"})
+            if not reconnect or max_messages:
+                return {
+                    "status": "failed" if written_count == 0 else "partial",
+                    "datasource_config_id": datasource.id,
+                    "mapping_id": mapping.id,
+                    "mapping_version": mapping.version,
+                    "target_schema": mapping.target_schema,
+                    "messages_seen": messages_seen,
+                    "mapped_count": mapped_count,
+                    "failed_count": failed_count,
+                    "written_count": written_count,
+                    "errors": errors[:20],
+                }
+            await asyncio.sleep(reconnect_delay)
+
+
+async def run_custom_stream_by_id(config_id: int) -> dict[str, Any]:
+    async with async_session_factory() as db:
+        datasource = await db.get(DataSourceConfig, config_id)
+        if not datasource:
+            raise CustomDatasourceRuntimeError("Configuration not found")
+        return await run_mapped_websocket_config(
+            db,
+            datasource,
+            use_config_debug_max_messages=False,
+        )
+
+
+def start_custom_stream(config_id: int) -> bool:
+    existing = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
+    if existing is not None and not existing.done():
+        return False
+    task = asyncio.create_task(run_custom_stream_by_id(config_id), name=f"custom-stream:{config_id}")
+    RUNNING_CUSTOM_STREAM_TASKS[config_id] = task
+
+    def _cleanup(done_task: asyncio.Task[Any]) -> None:
+        if RUNNING_CUSTOM_STREAM_TASKS.get(config_id) is done_task:
+            RUNNING_CUSTOM_STREAM_TASKS.pop(config_id, None)
+
+    task.add_done_callback(_cleanup)
+    return True
+
+
+async def stop_custom_stream(config_id: int) -> bool:
+    task = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
+    if task is None or task.done():
+        RUNNING_CUSTOM_STREAM_TASKS.pop(config_id, None)
+        return False
+    task.cancel()
+    try:
+        await task
+    except asyncio.CancelledError:
+        return True
+    return task.cancelled()
+
+
+def get_custom_stream_status(config_id: int) -> dict[str, Any]:
+    task = RUNNING_CUSTOM_STREAM_TASKS.get(config_id)
+    return {
+        "config_id": config_id,
+        "running": bool(task and not task.done()),
+        "done": bool(task and task.done()),
+    }

backend/app/services/datasource_connectivity.py

@@ -0,0 +1,437 @@
+"""Connectivity validation helpers for built-in datasource overrides."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+import hashlib
+import json
+import os
+from typing import Any
+
+import httpx
+from sqlalchemy import func, select
+
+from app.core.data_sources import get_data_sources_config
+from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.models.collected_data import CollectedData
+from app.models.datasource import DataSource
+from app.models.datasource_config import DataSourceConfig
+from app.models.system_setting import SystemSetting
+from app.services.barentswatch import (
+    _read_zshrc_env,
+    fetch_barentswatch_access_token,
+    resolve_barentswatch_config,
+)
+
+
+CONNECTIVITY_VALIDATION_KEY = "connectivity_validation"
+CONNECTIVITY_STORE_CATEGORY = "datasource_connectivity_validations"
+SUPPORTED_CREDENTIAL_PROVIDERS = {"barentswatch", "spacetrack", "aisstream"}
+
+
+def _sha256_json(payload: Any) -> str:
+    encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
+    return hashlib.sha256(encoded).hexdigest()
+
+
+def _resolve_spacetrack_credentials() -> tuple[str, str, str]:
+    zshrc_env = _read_zshrc_env()
+    username = os.getenv("SPACETRACK_USERNAME") or zshrc_env.get("SPACETRACK_USERNAME") or ""
+    password = os.getenv("SPACETRACK_PASSWORD") or zshrc_env.get("SPACETRACK_PASSWORD") or ""
+    source = "environment" if os.getenv("SPACETRACK_USERNAME") or os.getenv("SPACETRACK_PASSWORD") else ""
+    if not source and (username or password):
+        source = "~/.zshrc"
+    return username, password, source or "missing"
+
+
+async def _resolve_aisstream_api_key(
+    db=None,
+    credential_override: dict[str, str] | None = None,
+) -> tuple[str, str]:
+    if credential_override and credential_override.get("api_key"):
+        return str(credential_override["api_key"]), "draft"
+
+    env_key = os.getenv("AISSTREAM_API_KEY")
+    zshrc_key = _read_zshrc_env().get("AISSTREAM_API_KEY")
+    if db is not None:
+        result = await db.execute(
+            select(DataSourceConfig)
+            .where(DataSourceConfig.name == "aisstream_vessels")
+            .where(DataSourceConfig.is_active.is_(True))
+        )
+        record = result.scalar_one_or_none()
+        if record:
+            auth_config = record.auth_config or {}
+            runtime_config = record.config or {}
+            api_key = auth_config.get("api_key") or runtime_config.get("api_key")
+            if api_key:
+                return str(api_key), "datasource_config"
+
+    if env_key:
+        return env_key, "environment"
+    if zshrc_key:
+        return zshrc_key, "~/.zshrc"
+    return "", "missing"
+
+
+def strip_connectivity_validation(config: dict | None) -> dict:
+    cleaned = dict(config or {})
+    cleaned.pop(CONNECTIVITY_VALIDATION_KEY, None)
+    return cleaned
+
+
+def merge_connectivity_validation(existing_config: dict | None, next_config: dict | None) -> dict:
+    merged = strip_connectivity_validation(next_config)
+    validation = (existing_config or {}).get(CONNECTIVITY_VALIDATION_KEY)
+    if validation:
+        merged[CONNECTIVITY_VALIDATION_KEY] = validation
+    return merged
+
+
+def get_connectivity_validation(config: DataSourceConfig | None) -> dict | None:
+    validation = (config.config or {}).get(CONNECTIVITY_VALIDATION_KEY) if config else None
+    return validation if isinstance(validation, dict) else None
+
+
+async def build_builtin_connectivity_checksum(
+    source: str,
+    endpoint: str,
+    auth_type: str,
+    headers: dict | None,
+    config: dict | None,
+    db=None,
+    credential_override: dict[str, str] | None = None,
+) -> tuple[str, dict[str, Any]]:
+    defaults = DEFAULT_DATASOURCES.get(source, {})
+    credential_provider = defaults.get("credential_provider")
+    credential_fingerprint = ""
+    credential_source = "none"
+    has_credentials = not defaults.get("requires_credentials", False)
+
+    if credential_provider == "barentswatch":
+        if credential_override:
+            client_id = credential_override.get("client_id", "")
+            client_secret = credential_override.get("client_secret", "")
+            credential_source = "draft"
+        else:
+            barentswatch_config = await resolve_barentswatch_config(db)
+            client_id = barentswatch_config.client_id
+            client_secret = barentswatch_config.client_secret
+            credential_source = barentswatch_config.credential_source
+        has_credentials = bool(client_id and client_secret)
+        credential_fingerprint = _sha256_json(
+            {
+                "client_id": client_id,
+                "client_secret": client_secret,
+            }
+        )
+    elif credential_provider == "spacetrack":
+        username, password, credential_source = _resolve_spacetrack_credentials()
+        has_credentials = bool(username and password)
+        credential_fingerprint = _sha256_json(
+            {
+                "username": username,
+                "password": password,
+            }
+        )
+    elif credential_provider == "aisstream":
+        api_key, credential_source = await _resolve_aisstream_api_key(db, credential_override)
+        has_credentials = bool(api_key)
+        credential_fingerprint = _sha256_json({"api_key": api_key})
+    elif defaults.get("requires_credentials"):
+        credential_source = str(credential_provider or "unsupported")
+
+    checksum_payload = {
+        "source": source,
+        "endpoint": endpoint,
+        "auth_type": "none",
+        "headers": headers or {},
+        "config": strip_connectivity_validation(config),
+        "credential_provider": credential_provider or "none",
+        "credential_fingerprint": credential_fingerprint,
+    }
+    return _sha256_json(checksum_payload), {
+        "requires_credentials": bool(defaults.get("requires_credentials", False)),
+        "credential_provider": credential_provider,
+        "credential_source": credential_source,
+        "has_credentials": has_credentials,
+    }
+
+
+async def test_builtin_connectivity(
+    source: str,
+    endpoint: str,
+    auth_type: str,
+    headers: dict | None,
+    config: dict | None,
+    db=None,
+    credential_override: dict[str, str] | None = None,
+) -> dict[str, Any]:
+    defaults = DEFAULT_DATASOURCES.get(source)
+    if not defaults:
+        return {
+            "success": False,
+            "message": "未知内置采集器，无法执行连接校验。",
+        }
+
+    checksum, credential_context = await build_builtin_connectivity_checksum(
+        source,
+        endpoint,
+        auth_type,
+        headers,
+        config,
+        db,
+        credential_override,
+    )
+    if credential_context["requires_credentials"] and not credential_context["has_credentials"]:
+        return {
+            "success": False,
+            "checksum": checksum,
+            "stage": "credentials",
+            "message": "该采集器需要凭证，请先到采集器凭证设置中配置。",
+            "settings_tab": "collector_credentials",
+            **credential_context,
+        }
+    if (
+        credential_context["requires_credentials"]
+        and credential_context["credential_provider"] not in SUPPORTED_CREDENTIAL_PROVIDERS
+    ):
+        return {
+            "success": False,
+            "checksum": checksum,
+            "stage": "credentials",
+            "message": "该采集器的凭证链路尚未接入，暂时无法完成连接校验。",
+            "settings_tab": "collector_credentials",
+            **credential_context,
+        }
+
+    request_headers = {str(key): str(value) for key, value in (headers or {}).items()}
+    request_config = strip_connectivity_validation(config)
+    timeout = float(request_config.get("timeout") or 30)
+    request_endpoint = endpoint
+
+    if credential_context["credential_provider"] == "aisstream":
+        if not str(request_endpoint).startswith(("ws://", "wss://")):
+            return {
+                "success": False,
+                "checksum": checksum,
+                "stage": "endpoint",
+                "message": "AISStream endpoint 必须是 ws:// 或 wss:// WebSocket 地址。",
+                **credential_context,
+            }
+        return {
+            "success": True,
+            "checksum": checksum,
+            "stage": "credentials",
+            "message": "AISStream 凭证已配置，WebSocket endpoint 格式有效。",
+            **credential_context,
+        }
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
+            if credential_context["credential_provider"] == "barentswatch":
+                barentswatch_config = await resolve_barentswatch_config(db)
+                token = await fetch_barentswatch_access_token(client, barentswatch_config)
+                if not token:
+                    return {
+                        "success": False,
+                        "checksum": checksum,
+                        "stage": "token",
+                        "message": "凭证可读取，但 token 响应中没有 access_token。",
+                        "settings_tab": "collector_credentials",
+                        **credential_context,
+                    }
+                request_headers["Authorization"] = f"Bearer {token}"
+            elif credential_context["credential_provider"] == "spacetrack":
+                username, password, _source = _resolve_spacetrack_credentials()
+                login_url = "https://www.space-track.org/ajaxauth/login"
+                login_response = await client.post(
+                    login_url,
+                    data={
+                        "identity": username,
+                        "password": password,
+                    },
+                )
+                login_response.raise_for_status()
+
+            started = datetime.now(UTC)
+            async with client.stream("GET", request_endpoint, headers=request_headers) as response:
+                response.raise_for_status()
+                status_code = response.status_code
+            elapsed_ms = (datetime.now(UTC) - started).total_seconds() * 1000
+
+        return {
+            "success": True,
+            "checksum": checksum,
+            "stage": "endpoint",
+            "message": "连接验证成功。",
+            "status_code": status_code,
+            "response_time_ms": elapsed_ms,
+            **credential_context,
+        }
+    except httpx.HTTPStatusError as exc:
+        return {
+            "success": False,
+            "checksum": checksum,
+            "stage": "endpoint",
+            "message": f"连接验证失败：HTTP {exc.response.status_code}",
+            "error": f"HTTP Error: {exc.response.status_code}",
+            **credential_context,
+        }
+    except httpx.HTTPError as exc:
+        return {
+            "success": False,
+            "checksum": checksum,
+            "stage": "network",
+            "message": f"连接验证失败：{exc.__class__.__name__}",
+            "error": str(exc),
+            **credential_context,
+        }
+
+
+def make_success_validation(checksum: str, result: dict[str, Any]) -> dict[str, Any]:
+    return {
+        "checksum": checksum,
+        "status": "success",
+        "validated_at": datetime.now(UTC).isoformat(),
+        "status_code": result.get("status_code"),
+        "credential_source": result.get("credential_source"),
+    }
+
+
+def is_builtin_validation_current(config: DataSourceConfig | None, checksum: str) -> bool:
+    validation = get_connectivity_validation(config)
+    return bool(
+        validation
+        and validation.get("status") == "success"
+        and validation.get("checksum") == checksum
+    )
+
+
+async def get_connectivity_store(db) -> dict[str, Any]:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
+    )
+    record = result.scalar_one_or_none()
+    return dict(record.payload or {}) if record and isinstance(record.payload, dict) else {}
+
+
+async def save_connectivity_success(
+    db,
+    source: str,
+    checksum: str,
+    result: dict[str, Any],
+    *,
+    connected_by: str,
+) -> dict[str, Any]:
+    store = await get_connectivity_store(db)
+    validation = {
+        **make_success_validation(checksum, result),
+        "connected_by": connected_by,
+    }
+    store[source] = validation
+
+    existing = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == CONNECTIVITY_STORE_CATEGORY)
+    )
+    record = existing.scalar_one_or_none()
+    if record is None:
+        db.add(SystemSetting(category=CONNECTIVITY_STORE_CATEGORY, payload=store))
+    else:
+        record.payload = store
+    return validation
+
+
+async def load_builtin_override_config(db, source: str) -> DataSourceConfig | None:
+    result = await db.execute(
+        select(DataSourceConfig)
+        .where(DataSourceConfig.name == source)
+        .where(DataSourceConfig.is_active.is_(True))
+    )
+    return result.scalar_one_or_none()
+
+
+async def get_builtin_effective_candidate(db, source: str) -> dict[str, Any]:
+    override = await load_builtin_override_config(db, source)
+    default_endpoint = get_data_sources_config().get_yaml_url(source)
+    return {
+        "name": source,
+        "endpoint": (override.endpoint if override and override.endpoint else default_endpoint) or "",
+        "auth_type": override.auth_type if override else "none",
+        "headers": override.headers if override else {},
+        "config": strip_connectivity_validation(override.config if override else {}),
+    }
+
+
+async def has_collected_data(db, source: str) -> bool:
+    result = await db.execute(select(func.count(CollectedData.id)).where(CollectedData.source == source))
+    if (result.scalar() or 0) > 0:
+        return True
+
+    datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
+    datasource = datasource_result.scalar_one_or_none()
+    return bool(datasource and datasource.last_status == "success")
+
+
+async def get_builtin_connection_status(
+    db,
+    source: str,
+    endpoint: str,
+    auth_type: str,
+    headers: dict | None,
+    config: dict | None,
+) -> dict[str, Any]:
+    checksum, credential_context = await build_builtin_connectivity_checksum(
+        source,
+        endpoint,
+        auth_type,
+        headers,
+        config,
+        db,
+    )
+    store = await get_connectivity_store(db)
+    validation = store.get(source)
+    if isinstance(validation, dict) and validation.get("status") == "success":
+        if validation.get("checksum") == checksum:
+            return {
+                "connected": True,
+                "checksum": checksum,
+                "connected_by": validation.get("connected_by") or "connection_button",
+                "message": "当前配置已完成连接验证。",
+                **credential_context,
+            }
+
+    effective = await get_builtin_effective_candidate(db, source)
+    effective_checksum, _ = await build_builtin_connectivity_checksum(
+        source,
+        effective["endpoint"],
+        effective["auth_type"],
+        effective["headers"],
+        effective["config"],
+        db,
+    )
+    if checksum == effective_checksum and await has_collected_data(db, source):
+        return {
+            "connected": True,
+            "checksum": checksum,
+            "connected_by": "collection",
+            "message": "当前配置已有成功采集数据，视为已连接。",
+            **credential_context,
+        }
+
+    if isinstance(validation, dict) and validation.get("status") == "success":
+        return {
+            "connected": False,
+            "checksum": checksum,
+            "connected_by": None,
+            "message": "接口地址或凭证指纹已变化，请重新点击连接验证。",
+            **credential_context,
+        }
+
+    return {
+        "connected": False,
+        "checksum": checksum,
+        "connected_by": None,
+        "message": "当前配置尚未连接，请点击连接验证。",
+        **credential_context,
+    }

backend/app/services/datasource_mapping.py

@@ -0,0 +1,410 @@
+"""Deterministic mapping support for custom data sources."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.target_schema_registry import TargetSchema, get_target_schema
+
+SECRET_KEY_PATTERN = re.compile(
+    r"(token|secret|password|passwd|authorization|api[_-]?key|client[_-]?secret)",
+    re.IGNORECASE,
+)
+
+
+class MappingError(ValueError):
+    """Raised when a mapping definition cannot be executed."""
+
+
+def stable_payload_hash(payload: Any) -> str:
+    encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, default=str).encode()
+    return hashlib.sha256(encoded).hexdigest()
+
+
+def redact_for_llm(value: Any) -> Any:
+    if isinstance(value, dict):
+        redacted = {}
+        for key, item in value.items():
+            if SECRET_KEY_PATTERN.search(str(key)):
+                redacted[key] = "[REDACTED]"
+            else:
+                redacted[key] = redact_for_llm(item)
+        return redacted
+    if isinstance(value, list):
+        return [redact_for_llm(item) for item in value[:20]]
+    return value
+
+
+def extract_path(payload: Any, path: str | None) -> Any:
+    if not path or path == "$":
+        return payload
+
+    normalized = path.strip()
+    if normalized.startswith("$."):
+        normalized = normalized[2:]
+    elif normalized.startswith("$"):
+        normalized = normalized[1:]
+    normalized = normalized.strip(".")
+    if not normalized:
+        return payload
+
+    current = payload
+    for raw_segment in normalized.split("."):
+        segment = raw_segment.strip()
+        if not segment:
+            continue
+
+        list_all = segment.endswith("[*]")
+        if list_all:
+            segment = segment[:-3]
+
+        index = None
+        match = re.fullmatch(r"(.+)\[(\d+)\]", segment)
+        if match:
+            segment = match.group(1)
+            index = int(match.group(2))
+
+        if segment:
+            if isinstance(current, dict):
+                current = current.get(segment)
+            else:
+                return None
+
+        if list_all:
+            return current if isinstance(current, list) else []
+
+        if index is not None:
+            if not isinstance(current, list) or index >= len(current):
+                return None
+            current = current[index]
+
+    return current
+
+
+def _convert_value(value: Any, target_type: str | None) -> Any:
+    if value is None or target_type in (None, "", "any"):
+        return value
+
+    if target_type == "string":
+        return str(value)
+    if target_type == "integer":
+        return int(value)
+    if target_type == "float":
+        return float(value)
+    if target_type == "boolean":
+        if isinstance(value, bool):
+            return value
+        if isinstance(value, str):
+            return value.strip().lower() in {"1", "true", "yes", "y", "on"}
+        return bool(value)
+    if target_type == "datetime":
+        if isinstance(value, datetime):
+            return value
+        if isinstance(value, (int, float)):
+            return datetime.fromtimestamp(value)
+        if isinstance(value, str):
+            return datetime.fromisoformat(value.replace("Z", "+00:00"))
+        return value
+    if target_type == "object":
+        if isinstance(value, dict):
+            return value
+        raise ValueError("expected object")
+    if target_type == "array":
+        if isinstance(value, list):
+            return value
+        raise ValueError("expected array")
+
+    return value
+
+
+def _apply_enum(value: Any, enum_map: Any) -> Any:
+    if not isinstance(enum_map, dict):
+        return value
+    key = str(value)
+    return enum_map.get(key, enum_map.get(value, value))
+
+
+def _map_one(item: Any, field_mapping: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
+    output: dict[str, Any] = {}
+    errors: list[str] = []
+
+    for field_name, rule in field_mapping.items():
+        if isinstance(rule, str):
+            rule = {"path": rule}
+        if not isinstance(rule, dict):
+            errors.append(f"{field_name}: mapping rule must be an object or path string")
+            continue
+
+        value = extract_path(item, rule.get("path"))
+        if value is None and "default" in rule:
+            value = rule.get("default")
+        value = _apply_enum(value, rule.get("enum"))
+
+        try:
+            value = _convert_value(value, rule.get("type"))
+        except (TypeError, ValueError) as exc:
+            errors.append(f"{field_name}: failed to convert value {value!r}: {exc}")
+            continue
+
+        if value is not None or rule.get("include_null", False):
+            output[field_name] = value
+
+    return output, errors
+
+
+def execute_mapping(
+    payload: Any,
+    mapping_json: dict[str, Any],
+    target_schema: str | TargetSchema,
+    *,
+    limit: int | None = None,
+) -> dict[str, Any]:
+    schema = get_target_schema(target_schema) if isinstance(target_schema, str) else target_schema
+    source = mapping_json.get("source") or {}
+    fields = mapping_json.get("fields")
+    if not isinstance(fields, dict) or not fields:
+        raise MappingError("mapping_json.fields must be a non-empty object")
+
+    items_path = source.get("items_path") or mapping_json.get("items_path") or "$"
+    items = extract_path(payload, items_path)
+    if isinstance(items, dict):
+        items = [items]
+    elif not isinstance(items, list):
+        items = []
+
+    if limit is not None:
+        items = items[:limit]
+
+    mapped_records: list[dict[str, Any]] = []
+    errors: list[dict[str, Any]] = []
+    for index, item in enumerate(items):
+        mapped, mapping_errors = _map_one(item, fields)
+        validated, validation_errors = schema.validate_record(mapped)
+        all_errors = mapping_errors + validation_errors
+        if all_errors:
+            errors.append({"index": index, "errors": all_errors, "record": mapped})
+            continue
+        if validated is not None:
+            mapped_records.append(validated)
+
+    return {
+        "target_schema": schema.key,
+        "total_items": len(items),
+        "mapped_count": len(mapped_records),
+        "failed_count": len(errors),
+        "records": mapped_records,
+        "errors": errors,
+    }
+
+
+def build_heuristic_mapping(sample_payload: Any, target_schema_key: str) -> dict[str, Any]:
+    schema = get_target_schema(target_schema_key)
+    items_path = "$"
+    sample_item = sample_payload
+    if isinstance(sample_payload, dict):
+        for key in ("data", "items", "results", "features", "vessels"):
+            candidate = sample_payload.get(key)
+            if isinstance(candidate, list) and candidate:
+                items_path = f"$.{key}[*]"
+                sample_item = candidate[0]
+                break
+    elif isinstance(sample_payload, list) and sample_payload:
+        items_path = "$"
+        sample_item = sample_payload[0]
+
+    available = _flatten_keys(sample_item if isinstance(sample_item, dict) else {})
+    fields: dict[str, Any] = {}
+    for field in schema.fields:
+        candidate = _best_field_match(field.name, available)
+        if candidate:
+            fields[field.name] = {"path": f"$.{candidate}", "type": field.type}
+        elif field.name == "data" and target_schema_key == "generic_records":
+            fields[field.name] = {"path": "$", "type": "object"}
+        elif not field.required:
+            fields[field.name] = {"path": f"$.{field.name}", "type": field.type, "default": None}
+
+    return {
+        "source": {"items_path": items_path},
+        "fields": fields,
+        "meta": {
+            "generated_by": "heuristic",
+            "requires_review": True,
+        },
+    }
+
+
+def _flatten_keys(payload: dict[str, Any], prefix: str = "") -> list[str]:
+    keys: list[str] = []
+    for key, value in payload.items():
+        dotted = f"{prefix}.{key}" if prefix else str(key)
+        keys.append(dotted)
+        if isinstance(value, dict):
+            keys.extend(_flatten_keys(value, dotted))
+    return keys
+
+
+def _best_field_match(field_name: str, candidates: list[str]) -> str | None:
+    aliases = {
+        "lat": ("lat", "latitude", "y"),
+        "lon": ("lon", "lng", "longitude", "x"),
+        "mmsi": ("mmsi",),
+        "sog": ("sog", "speed", "speedOverGround"),
+        "cog": ("cog", "course", "courseOverGround"),
+        "received_at": ("received_at", "timestamp", "time", "updated_at"),
+        "observed_at": ("observed_at", "timestamp", "time", "updated_at"),
+        "source_id": ("id", "source_id", "uuid"),
+    }.get(field_name, (field_name,))
+
+    lowered = {candidate.lower(): candidate for candidate in candidates}
+    for alias in aliases:
+        if alias.lower() in lowered:
+            return lowered[alias.lower()]
+    for candidate in candidates:
+        tail = candidate.split(".")[-1].lower()
+        if tail in {alias.lower() for alias in aliases}:
+            return candidate
+    return None
+
+
+def _parse_datetime(value: Any) -> datetime | None:
+    if value is None:
+        return None
+    if isinstance(value, datetime):
+        return value
+    if isinstance(value, str):
+        return datetime.fromisoformat(value.replace("Z", "+00:00"))
+    return None
+
+
+async def persist_mapped_records(
+    db: AsyncSession,
+    *,
+    datasource_name: str,
+    datasource_config_id: int,
+    target_schema: str,
+    records: list[dict[str, Any]],
+    mapping_version: int,
+    delivery_mode: str | None = None,
+    transport: str | None = None,
+) -> int:
+    """Persist validated mapped records to the destination for a target schema."""
+    if target_schema == "vessel_ais":
+        from app.core.time import to_iso8601_utc
+        from app.core.websocket.broadcaster import broadcaster
+        from app.services.vessel_ais_aggregation import (
+            record_vessel_ais_observation,
+            update_ais_source_health,
+        )
+
+        now = datetime.now(UTC)
+        latest_observed_at = now
+        written_count = 0
+        for record in records:
+            observed_at = _parse_datetime(record.get("received_at")) or now
+            observation = await record_vessel_ais_observation(
+                db,
+                source=datasource_name,
+                normalized_payload=record,
+                raw_payload=record,
+                delivery_mode=delivery_mode or "polling",
+                transport=transport or "http",
+                message_type="PositionReport",
+                observed_at=observed_at,
+                collected_at=now,
+            )
+            if observation is not None:
+                written_count += 1
+            if observed_at > latest_observed_at:
+                latest_observed_at = observed_at
+
+        await update_ais_source_health(
+            db,
+            source=datasource_name,
+            connection_state="connected",
+            observed_count=len(records),
+            last_seen_at=latest_observed_at,
+            last_success_at=now if records else None,
+            lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
+        )
+        await db.commit()
+        if records:
+            await broadcaster.broadcast_custom(
+                "vessels",
+                {
+                    "action": "upsert",
+                    "source": datasource_name,
+                    "created": True,
+                    "vessels": [
+                        {
+                            "mmsi": record.get("mmsi"),
+                            "mmsi_display": str(record.get("mmsi")) if record.get("mmsi") is not None else None,
+                            "name": record.get("name"),
+                            "callsign": record.get("callsign"),
+                            "lat": record.get("lat"),
+                            "lon": record.get("lon"),
+                            "sog": record.get("sog"),
+                            "cog": record.get("cog"),
+                            "heading": record.get("heading"),
+                            "nav_status": record.get("nav_status"),
+                            "vessel_type": record.get("vessel_type"),
+                            "vessel_type_name": record.get("vessel_type_name"),
+                            "received_at": to_iso8601_utc(_parse_datetime(record.get("received_at"))),
+                        }
+                        for record in records
+                    ],
+                },
+            )
+        return written_count
+
+    from app.models.collected_data import CollectedData
+
+    collected_at = datetime.now(UTC)
+    for index, record in enumerate(records):
+        if target_schema == "geo_points":
+            source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
+            name = record.get("name")
+            metadata = {
+                "latitude": record.get("lat"),
+                "longitude": record.get("lon"),
+                "type": record.get("type"),
+                "mapping_version": mapping_version,
+                "target_schema": target_schema,
+                **(record.get("metadata") or {}),
+            }
+            reference_date = _parse_datetime(record.get("observed_at"))
+        else:
+            source_id = record.get("source_id") or f"{datasource_config_id}:{index}"
+            name = None
+            metadata = {
+                "data": record.get("data") or {},
+                "mapping_version": mapping_version,
+                "target_schema": target_schema,
+            }
+            reference_date = _parse_datetime(record.get("observed_at"))
+
+        db.add(
+            CollectedData(
+                source=datasource_name,
+                source_id=str(source_id),
+                entity_key=f"{datasource_name}:{source_id}",
+                data_type=target_schema,
+                name=name,
+                title=name,
+                extra_data=metadata,
+                collected_at=collected_at,
+                reference_date=reference_date,
+                is_valid=1,
+                is_current=True,
+                change_type="created",
+                change_summary={},
+            )
+        )
+
+    await db.commit()
+    return len(records)

backend/app/services/earth_news.py

@@ -30,6 +30,14 @@ class RegionProfile:
     accent: str
 
 
+@dataclass(frozen=True)
+class RegionAnchor:
+    region: str
+    label: str
+    latitude: float
+    longitude: float
+
+
 @dataclass(frozen=True)
 class NewsFeedSource:
     id: str
@@ -95,6 +103,39 @@ REGION_PROFILES: dict[str, RegionProfile] = {
     ),
 }
 
+REGION_ANCHORS: dict[str, RegionAnchor] = {
+    "americas": RegionAnchor(
+        region="americas",
+        label="美洲",
+        latitude=37.0902,
+        longitude=-95.7129,
+    ),
+    "europe": RegionAnchor(
+        region="europe",
+        label="欧洲",
+        latitude=50.1109,
+        longitude=8.6821,
+    ),
+    "middle-east-africa": RegionAnchor(
+        region="middle-east-africa",
+        label="中东与非洲",
+        latitude=25.2048,
+        longitude=55.2708,
+    ),
+    "asia-pacific": RegionAnchor(
+        region="asia-pacific",
+        label="亚太",
+        latitude=1.3521,
+        longitude=103.8198,
+    ),
+    "global": RegionAnchor(
+        region="global",
+        label="全球",
+        latitude=20.0,
+        longitude=0.0,
+    ),
+}
+
 
 def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
     return (
@@ -213,6 +254,10 @@ def get_region_profile(region: str) -> RegionProfile:
     return REGION_PROFILES.get(region, REGION_PROFILES["global"])
 
 
+def get_region_anchor(region: str) -> RegionAnchor:
+    return REGION_ANCHORS.get(region, REGION_ANCHORS["global"])
+
+
 def get_sources_for_region(region: str) -> list[NewsFeedSource]:
     return sorted(
         [source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
@@ -342,6 +387,7 @@ def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
 
 def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
     published_at = item.published_at
+    anchor = get_region_anchor(item.feed_region)
     return {
         "id": item.id,
         "title": item.title,
@@ -352,6 +398,10 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, An
         "region": item.feed_region,
         "homepage_url": item.homepage_url,
         "published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
+        "latitude": anchor.latitude,
+        "longitude": anchor.longitude,
+        "location_label": anchor.label,
+        "location_inferred": True,
         "is_focus_match": item.feed_region == active_region,
     }
 

backend/app/services/llm_provider_catalog.py

@@ -0,0 +1,150 @@
+"""LLM provider presets used by Settings and the runtime AI provider bridge."""
+
+from __future__ import annotations
+
+from typing import Any
+
+import httpx
+
+MODELS_DEV_URL = "https://models.dev/api.json"
+
+
+FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
+    "minimax": {
+        "provider": "minimax",
+        "label": "MiniMax",
+        "provider_api": "anthropic-messages",
+        "base_url": "https://api.minimaxi.com/anthropic",
+        "model": "MiniMax-M2.7",
+        "models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
+        "api_key_env": "MINIMAX_API_KEY",
+        "source": "fallback",
+    },
+    "openai": {
+        "provider": "openai",
+        "label": "OpenAI",
+        "provider_api": "openai-completions",
+        "base_url": "https://api.openai.com/v1",
+        "model": "gpt-5.1",
+        "models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
+        "api_key_env": "OPENAI_API_KEY",
+        "source": "fallback",
+    },
+    "anthropic": {
+        "provider": "anthropic",
+        "label": "Anthropic",
+        "provider_api": "anthropic-messages",
+        "base_url": "https://api.anthropic.com/v1",
+        "model": "claude-sonnet-4-6",
+        "models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
+        "api_key_env": "ANTHROPIC_API_KEY",
+        "source": "fallback",
+    },
+    "deepseek": {
+        "provider": "deepseek",
+        "label": "DeepSeek",
+        "provider_api": "openai-completions",
+        "base_url": "https://api.deepseek.com/v1",
+        "model": "deepseek-chat",
+        "models": ["deepseek-chat", "deepseek-reasoner"],
+        "api_key_env": "DEEPSEEK_API_KEY",
+        "source": "fallback",
+    },
+    "alibaba": {
+        "provider": "alibaba",
+        "label": "Alibaba Qwen / DashScope",
+        "provider_api": "openai-completions",
+        "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+        "model": "qwen3-max",
+        "models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
+        "api_key_env": "DASHSCOPE_API_KEY",
+        "source": "fallback",
+    },
+    "moonshotai": {
+        "provider": "moonshotai",
+        "label": "Moonshot AI / Kimi",
+        "provider_api": "openai-completions",
+        "base_url": "https://api.moonshot.ai/v1",
+        "model": "kimi-k2.5",
+        "models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
+        "api_key_env": "MOONSHOT_API_KEY",
+        "source": "fallback",
+    },
+    "openrouter": {
+        "provider": "openrouter",
+        "label": "OpenRouter",
+        "provider_api": "openai-completions",
+        "base_url": "https://openrouter.ai/api/v1",
+        "model": "openai/gpt-5.1",
+        "models": ["openai/gpt-5.1", "anthropic/claude-sonnet-4.5", "qwen/qwen3-max"],
+        "api_key_env": "OPENROUTER_API_KEY",
+        "source": "fallback",
+    },
+    "ollama": {
+        "provider": "ollama",
+        "label": "Ollama Local",
+        "provider_api": "ollama-generate",
+        "base_url": "http://127.0.0.1:11434",
+        "model": "qwen2.5:7b",
+        "models": ["qwen2.5:7b", "llama3.1:8b", "mistral:7b"],
+        "api_key_env": "",
+        "source": "fallback",
+    },
+}
+
+MODELS_DEV_PROVIDER_KEYS = {
+    "minimax": "minimax",
+    "openai": "openai",
+    "anthropic": "anthropic",
+    "deepseek": "deepseek",
+    "alibaba": "alibaba",
+    "moonshotai": "moonshotai",
+    "openrouter": "openrouter",
+}
+
+
+def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
+    return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
+
+
+def get_fallback_llm_provider_preset(provider: str) -> dict[str, Any]:
+    key = provider.strip().lower()
+    if key not in FALLBACK_LLM_PROVIDER_PRESETS:
+        raise ValueError(f"Unsupported LLM provider preset: {provider}")
+    return dict(FALLBACK_LLM_PROVIDER_PRESETS[key])
+
+
+async def refresh_llm_provider_preset(provider: str) -> dict[str, Any]:
+    fallback = get_fallback_llm_provider_preset(provider)
+    models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
+    if not models_dev_key:
+        return fallback
+
+    async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
+        response = await client.get(
+            MODELS_DEV_URL,
+            headers={"User-Agent": "Planet/1.0"},
+        )
+        response.raise_for_status()
+        catalog = response.json()
+
+    upstream = catalog.get(models_dev_key)
+    if not isinstance(upstream, dict):
+        return fallback
+
+    upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
+    model_ids = list(upstream_models.keys())[:80]
+    base_url = upstream.get("api") or fallback["base_url"]
+    if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
+        base_url = "https://api.deepseek.com/v1"
+
+    refreshed = {
+        **fallback,
+        "label": upstream.get("name") or fallback["label"],
+        "base_url": base_url,
+        "model": model_ids[0] if model_ids else fallback["model"],
+        "models": model_ids or fallback["models"],
+        "api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
+        "source": MODELS_DEV_URL,
+    }
+    return refreshed

backend/app/services/persistent_logs.py

@@ -0,0 +1,86 @@
+from __future__ import annotations
+
+from typing import Any
+
+from app.core.logging import get_logger, sanitize_log_value
+from app.core.request_context import get_request_id
+from app.db.session import async_session_factory
+from app.models.system_log import AuditLog, SystemLog
+
+logger = get_logger(__name__)
+
+
+async def record_system_log(
+    *,
+    source: str,
+    level: str,
+    message: str,
+    service: str | None = None,
+    module: str | None = None,
+    event: str | None = None,
+    request_id: str | None = None,
+    trace_id: str | None = None,
+    user_id: int | None = None,
+    category: str | None = None,
+    context: dict[str, Any] | None = None,
+) -> None:
+    try:
+        async with async_session_factory() as session:
+            session.add(
+                SystemLog(
+                    source=source,
+                    service=service,
+                    module=module,
+                    event=event,
+                    level=level.lower(),
+                    message=str(sanitize_log_value(message)),
+                    request_id=request_id or get_request_id(),
+                    trace_id=trace_id,
+                    user_id=user_id,
+                    category=category,
+                    context=sanitize_log_value(context or {}),
+                )
+            )
+            await session.commit()
+    except Exception:
+        logger.exception_event(
+            "Failed to persist system log",
+            event="system_log.persist.failed",
+            context={"event_name": event, "source": source},
+        )
+
+
+async def record_audit_log(
+    *,
+    action: str,
+    actor_id: int | None = None,
+    actor_name: str | None = None,
+    target_type: str | None = None,
+    target_id: str | None = None,
+    result: str | None = None,
+    request_id: str | None = None,
+    ip: str | None = None,
+    details: dict[str, Any] | None = None,
+) -> None:
+    try:
+        async with async_session_factory() as session:
+            session.add(
+                AuditLog(
+                    actor_id=actor_id,
+                    actor_name=actor_name,
+                    action=action,
+                    target_type=target_type,
+                    target_id=target_id,
+                    result=result,
+                    request_id=request_id or get_request_id(),
+                    ip=ip,
+                    details=sanitize_log_value(details or {}),
+                )
+            )
+            await session.commit()
+    except Exception:
+        logger.exception_event(
+            "Failed to persist audit log",
+            event="audit_log.persist.failed",
+            context={"action": action},
+        )

backend/app/services/scheduler.py

@@ -1,7 +1,6 @@
 """Task Scheduler for running collection jobs."""
 
 import asyncio
-import logging
 from datetime import UTC, datetime, timedelta
 from typing import Any, Dict, Optional
 
@@ -9,13 +8,19 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
 from apscheduler.triggers.interval import IntervalTrigger
 from sqlalchemy import select
 
+from app.core.logging import get_logger
 from app.db.session import async_session_factory
 from app.core.time import to_iso8601_utc
 from app.models.datasource import DataSource
 from app.models.task import CollectionTask
 from app.services.collectors.registry import collector_registry
+from app.services.datasource_connectivity import (
+    build_builtin_connectivity_checksum,
+    get_builtin_effective_candidate,
+    save_connectivity_success,
+)
 
-logger = logging.getLogger(__name__)
+logger = get_logger(__name__)
 
 scheduler = AsyncIOScheduler()
 RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
@@ -54,7 +59,11 @@ async def _update_next_run_at(datasource: DataSource, session) -> None:
 async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
     collector = collector_registry.get(datasource.source)
     if not collector:
-        logger.warning("Collector not found for datasource %s", datasource.source)
+        logger.warning_event(
+            "Collector not found for datasource",
+            event="collector.schedule.collector_missing",
+            context={"collector_name": datasource.source},
+        )
         return
 
     collector_registry.set_active(datasource.source, datasource.is_active)
@@ -72,13 +81,17 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
             replace_existing=True,
             kwargs={"collector_name": datasource.source},
         )
-        logger.info(
-            "Scheduled collector: %s (every %sm)",
-            datasource.source,
-            datasource.frequency_minutes,
+        logger.info_event(
+            "Scheduled collector",
+            event="collector.schedule.updated",
+            context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
         )
     else:
-        logger.info("Collector disabled: %s", datasource.source)
+        logger.info_event(
+            "Collector disabled",
+            event="collector.schedule.disabled",
+            context={"collector_name": datasource.source},
+        )
 
     await _update_next_run_at(datasource, session)
 
@@ -87,18 +100,30 @@ async def run_collector_task(collector_name: str):
     """Run a single collector task."""
     collector = collector_registry.get(collector_name)
     if not collector:
-        logger.error("Collector not found: %s", collector_name)
+        logger.error_event(
+            "Collector not found",
+            event="collector.run.collector_missing",
+            context={"collector_name": collector_name},
+        )
         return
 
     async with async_session_factory() as db:
         result = await db.execute(select(DataSource).where(DataSource.source == collector_name))
         datasource = result.scalar_one_or_none()
         if not datasource:
-            logger.error("Datasource not found for collector: %s", collector_name)
+            logger.error_event(
+                "Datasource not found for collector",
+                event="collector.run.datasource_missing",
+                context={"collector_name": collector_name},
+            )
             return
 
         if not datasource.is_active:
-            logger.info("Skipping disabled collector: %s", collector_name)
+            logger.info_event(
+                "Skipping disabled collector",
+                event="collector.run.skipped_disabled",
+                context={"collector_name": collector_name},
+            )
             return
 
         running_result = await db.execute(
@@ -122,10 +147,10 @@ async def run_collector_task(collector_name: str):
                 and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
             )
             if not is_stale:
-                logger.warning(
-                    "Skipping collector %s trigger because task %s is already running",
-                    collector_name,
-                    existing_running.id,
+                logger.warning_event(
+                    "Skipping collector trigger because task is already running",
+                    event="collector.run.skipped_already_running",
+                    context={"collector_name": collector_name, "task_id": existing_running.id},
                 )
                 return
 
@@ -143,31 +168,64 @@ async def run_collector_task(collector_name: str):
                 else stale_reason
             )
             await db.commit()
-            logger.warning(
-                "Marked stale running task %s as failed before rerun of %s",
-                existing_running.id,
-                collector_name,
+            logger.warning_event(
+                "Marked stale running task as failed before rerun",
+                event="collector.run.stale_task_failed",
+                context={"collector_name": collector_name, "task_id": existing_running.id},
             )
 
         try:
             collector._datasource_id = datasource.id
-            logger.info("Running collector: %s (datasource_id=%s)", collector_name, datasource.id)
+            logger.info_event(
+                "Running collector",
+                event="collector.run.started",
+                context={"collector_name": collector_name, "datasource_id": datasource.id},
+            )
             task_result = await collector.run(db)
             datasource.last_run_at = datetime.now(UTC)
             datasource.last_status = task_result.get("status")
+            if datasource.last_status == "success":
+                effective_candidate = await get_builtin_effective_candidate(db, datasource.source)
+                checksum, _credential_context = await build_builtin_connectivity_checksum(
+                    datasource.source,
+                    effective_candidate["endpoint"],
+                    effective_candidate["auth_type"],
+                    effective_candidate["headers"],
+                    effective_candidate["config"],
+                    db,
+                )
+                await save_connectivity_success(
+                    db,
+                    datasource.source,
+                    checksum,
+                    {"status_code": None},
+                    connected_by="collection",
+                )
             await _update_next_run_at(datasource, db)
-            logger.info("Collector %s completed: %s", collector_name, task_result)
+            logger.info_event(
+                "Collector completed",
+                event="collector.run.completed",
+                context={"collector_name": collector_name, "datasource_id": datasource.id, "result": task_result},
+            )
         except asyncio.CancelledError:
             datasource.last_run_at = datetime.now(UTC)
             datasource.last_status = "cancelled"
             await db.commit()
-            logger.warning("Collector %s cancelled by operator", collector_name)
+            logger.warning_event(
+                "Collector cancelled by operator",
+                event="collector.run.cancelled",
+                context={"collector_name": collector_name, "datasource_id": datasource.id},
+            )
             raise
         except Exception as exc:
             datasource.last_run_at = datetime.now(UTC)
             datasource.last_status = "failed"
             await db.commit()
-            logger.exception("Collector %s failed: %s", collector_name, exc)
+            logger.exception_event(
+                "Collector failed",
+                event="collector.run.failed",
+                context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
+            )
 
 
 async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
@@ -194,7 +252,11 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
 
         if stale_tasks:
             await db.commit()
-            logger.warning("Cleaned up %s stale running collection task(s)", len(stale_tasks))
+            logger.warning_event(
+                "Cleaned up stale running collection tasks",
+                event="collector.cleanup.stale_tasks_cleaned",
+                context={"count": len(stale_tasks)},
+            )
 
         return len(stale_tasks)
 
@@ -203,14 +265,14 @@ def start_scheduler() -> None:
     """Start the scheduler."""
     if not scheduler.running:
         scheduler.start()
-        logger.info("Scheduler started")
+        logger.info_event("Scheduler started", event="scheduler.started")
 
 
 def stop_scheduler() -> None:
     """Stop the scheduler."""
     if scheduler.running:
         scheduler.shutdown(wait=False)
-        logger.info("Scheduler stopped")
+        logger.info_event("Scheduler stopped", event="scheduler.stopped")
 
 
 async def sync_scheduler_with_datasources() -> None:
@@ -271,12 +333,20 @@ def run_collector_now(collector_name: str) -> bool:
     """Run a collector immediately (not scheduled)."""
     collector = collector_registry.get(collector_name)
     if not collector:
-        logger.error("Collector not found: %s", collector_name)
+        logger.error_event(
+            "Collector not found",
+            event="collector.trigger.collector_missing",
+            context={"collector_name": collector_name},
+        )
         return False
 
     existing_task = get_running_collector_task(collector_name)
     if existing_task is not None and not existing_task.done():
-        logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
+        logger.warning_event(
+            "Collector is already running in-memory; skipping duplicate trigger",
+            event="collector.trigger.skipped_already_running",
+            context={"collector_name": collector_name},
+        )
         return False
 
     try:
@@ -289,10 +359,18 @@ def run_collector_now(collector_name: str) -> bool:
                 RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
 
         task.add_done_callback(_cleanup_task)
-        logger.info("Triggered collector: %s", collector_name)
+        logger.info_event(
+            "Triggered collector",
+            event="collector.trigger.started",
+            context={"collector_name": collector_name},
+        )
         return True
     except Exception as exc:
-        logger.error("Failed to trigger collector %s: %s", collector_name, exc)
+        logger.error_event(
+            "Failed to trigger collector",
+            event="collector.trigger.failed",
+            context={"collector_name": collector_name, "error": str(exc)},
+        )
         return False
 
 

backend/app/services/system_control.py

@@ -19,6 +19,10 @@ ALLOWED_ACTIONS: dict[str, dict[str, Any]] = {
         "command": ["./planet.sh", "restart", "-b"],
         "recovery_mode": "backend",
     },
+    "restart-frontend": {
+        "command": ["./planet.sh", "restart", "-f"],
+        "recovery_mode": "frontend",
+    },
     "restart-ai-provider": {
         "command": ["./planet.sh", "restart", "-a"],
         "recovery_mode": "ai-provider",

backend/app/services/system_logs.py

@@ -0,0 +1,532 @@
+from __future__ import annotations
+
+import json
+import re
+import shutil
+import subprocess
+
+from collections import Counter, deque
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+from app.core.security import redis_client
+
+DEFAULT_LOG_LINE_LIMIT = 200
+MAX_LOG_LINE_LIMIT = 1000
+BUFFER_LOG_LIMIT = 1000
+BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
+LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
+
+LOG_LEVEL_ERROR = "error"
+LOG_LEVEL_WARNING = "warning"
+LOG_LEVEL_INFO = "info"
+LOG_LEVEL_DEBUG = "debug"
+LOG_LEVEL_ALL = "all"
+
+SUPPORTED_LOG_LEVELS = {
+    LOG_LEVEL_ALL,
+    LOG_LEVEL_ERROR,
+    LOG_LEVEL_WARNING,
+    LOG_LEVEL_INFO,
+    LOG_LEVEL_DEBUG,
+}
+
+LOG_LEVEL_ALIASES = {
+    "warn": LOG_LEVEL_WARNING,
+    "warning": LOG_LEVEL_WARNING,
+    "err": LOG_LEVEL_ERROR,
+    "error": LOG_LEVEL_ERROR,
+    "info": LOG_LEVEL_INFO,
+    "information": LOG_LEVEL_INFO,
+    "debug": LOG_LEVEL_DEBUG,
+    "trace": LOG_LEVEL_DEBUG,
+    "critical": LOG_LEVEL_ERROR,
+    "fatal": LOG_LEVEL_ERROR,
+}
+
+TIMESTAMP_FORMATS = (
+    "%Y-%m-%d %H:%M:%S.%f",
+    "%Y-%m-%d %H:%M:%S",
+    "%Y-%m-%dT%H:%M:%S.%f",
+    "%Y-%m-%dT%H:%M:%S",
+)
+
+LEVEL_PATTERNS = (
+    ("CRITICAL", LOG_LEVEL_ERROR),
+    ("FATAL", LOG_LEVEL_ERROR),
+    ("ERROR", LOG_LEVEL_ERROR),
+    ("WARNING", LOG_LEVEL_WARNING),
+    ("WARN", LOG_LEVEL_WARNING),
+    ("INFO", LOG_LEVEL_INFO),
+    ("DEBUG", LOG_LEVEL_DEBUG),
+    ("TRACE", LOG_LEVEL_DEBUG),
+)
+
+LEADING_LEVEL_PATTERN = re.compile(
+    r"^\s*(?:\[[^\]]+\]\s*)?(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b[:\s-]*",
+    re.IGNORECASE,
+)
+EMBEDDED_LEVEL_PATTERN = re.compile(
+    r"\b(CRITICAL|FATAL|ERROR|WARNING|WARN|INFO|DEBUG|TRACE)\b",
+    re.IGNORECASE,
+)
+CONTROL_CHAR_PATTERN = re.compile(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]")
+
+
+@dataclass(frozen=True)
+class LogSource:
+    source_id: str
+    name: str
+    kind: str
+    location: str
+    description: str
+    category: str
+    status: str = "ok"
+    buffer_key: str | None = None
+    container_name: str | None = None
+
+
+@dataclass
+class StructuredLogEntry:
+    timestamp: datetime | None
+    level: str | None
+    display_line: str
+    raw_line: str
+    search_text: str
+
+
+@dataclass
+class DailyLogMarker:
+    date_token: str
+    total: int
+    dominant_level: str
+
+
+LOG_SOURCES: dict[str, LogSource] = {
+    "backend": LogSource(
+        source_id="backend",
+        name="后端服务",
+        kind="file",
+        location="/tmp/planet_backend.log",
+        description="FastAPI 后端、调度器和采集任务共享日志。",
+        category="service",
+    ),
+    "frontend": LogSource(
+        source_id="frontend",
+        name="前端开发服务",
+        kind="file",
+        location="/tmp/planet_frontend.log",
+        description="控制台与 Earth 前端开发服务输出。",
+        category="service",
+    ),
+    "ai-provider": LogSource(
+        source_id="ai-provider",
+        name="AI Provider",
+        kind="docker",
+        location="docker://planet_aiprovider",
+        description="AI Provider 容器实时输出日志。",
+        category="service",
+        container_name="planet_aiprovider",
+    ),
+    "earth-client": LogSource(
+        source_id="earth-client",
+        name="Earth 浏览器端",
+        kind="buffer",
+        location="redis://planet:system_logs:earth-client",
+        description="Earth 浏览器端上报的运行时错误与关键业务日志。",
+        category="client",
+        buffer_key=f"{LOG_BUFFER_KEY_PREFIX}:earth-client",
+    ),
+}
+
+
+def normalize_log_level(level: str | None) -> str:
+    if level is None:
+        return LOG_LEVEL_ALL
+    normalized = str(level).strip().lower()
+    if normalized in {"", LOG_LEVEL_ALL}:
+        return LOG_LEVEL_ALL
+    return LOG_LEVEL_ALIASES.get(normalized, LOG_LEVEL_ALL)
+
+
+def normalize_log_levels(level: str | None = None, levels: str | None = None) -> tuple[str, ...]:
+    normalized_levels: list[str] = []
+    if levels:
+        for item in str(levels).split(","):
+            normalized = normalize_log_level(item)
+            if normalized != LOG_LEVEL_ALL and normalized not in normalized_levels:
+                normalized_levels.append(normalized)
+    normalized_level = normalize_log_level(level)
+    if normalized_level != LOG_LEVEL_ALL and normalized_level not in normalized_levels:
+        normalized_levels.append(normalized_level)
+    return tuple(normalized_levels)
+
+
+def get_source_status(source: LogSource) -> str:
+    if source.kind == "file":
+        path = Path(source.location)
+        if not path.exists():
+            return "missing"
+        return "ok" if path.stat().st_size > 0 else "empty"
+    if source.kind == "docker":
+        return "ok" if shutil.which("docker") else "docker_unavailable"
+    if source.kind == "buffer":
+        if not source.buffer_key:
+            return "source_unavailable"
+        try:
+            return "ok" if redis_client.llen(source.buffer_key) > 0 else "empty"
+        except Exception:
+            return "source_unavailable"
+    return "source_unavailable"
+
+
+def list_log_sources() -> list[dict[str, str]]:
+    items: list[dict[str, str]] = []
+    for source in LOG_SOURCES.values():
+        items.append(
+            {
+                "source_id": source.source_id,
+                "name": source.name,
+                "kind": source.kind,
+                "location": source.location,
+                "description": source.description,
+                "category": source.category,
+                "status": get_source_status(source),
+            }
+        )
+    return items
+
+
+def get_buffer_log_key(source_id: str) -> str:
+    return f"{LOG_BUFFER_KEY_PREFIX}:{source_id}"
+
+
+def append_buffer_log(
+    source_id: str,
+    *,
+    level: str,
+    message: str,
+    context: dict[str, Any] | None = None,
+) -> None:
+    payload = {
+        "timestamp": datetime.now(tz=UTC).isoformat(),
+        "level": normalize_log_level(level),
+        "message": message,
+        "context": context or {},
+    }
+    buffer_key = get_buffer_log_key(source_id)
+    redis_client.rpush(buffer_key, json.dumps(payload, ensure_ascii=False))
+    redis_client.ltrim(buffer_key, -BUFFER_LOG_LIMIT, -1)
+    redis_client.expire(buffer_key, BUFFER_LOG_TTL_SECONDS)
+
+
+def parse_timestamp(raw_value: str | None) -> datetime | None:
+    if not raw_value:
+        return None
+    candidate = str(raw_value).strip()
+    if not candidate:
+        return None
+    candidate = candidate.replace("Z", "+00:00")
+    try:
+        parsed = datetime.fromisoformat(candidate)
+        return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+    except ValueError:
+        pass
+    for fmt in TIMESTAMP_FORMATS:
+        try:
+            return datetime.strptime(candidate, fmt).replace(tzinfo=UTC)
+        except ValueError:
+            continue
+    return None
+
+
+def parse_prefixed_timestamp(line: str) -> tuple[datetime | None, str]:
+    stripped = line.strip()
+    if not stripped:
+        return None, ""
+    for prefix_length in (35, 32, 29, 26, 23, 19):
+        if len(stripped) < prefix_length:
+            continue
+        prefix = stripped[:prefix_length]
+        timestamp = parse_timestamp(prefix)
+        if timestamp is not None:
+            return timestamp, stripped[prefix_length:].lstrip()
+    first_token = stripped.split(maxsplit=1)[0]
+    timestamp = parse_timestamp(first_token)
+    if timestamp is not None:
+        remainder = stripped[len(first_token):].lstrip()
+        return timestamp, remainder
+    return None, stripped
+
+
+def infer_log_level_from_text(text: str, *, allow_embedded: bool = True) -> str | None:
+    leading_match = LEADING_LEVEL_PATTERN.match(text)
+    if leading_match:
+        return normalize_log_level(leading_match.group(1))
+
+    if allow_embedded:
+        embedded_match = EMBEDDED_LEVEL_PATTERN.search(text)
+        if embedded_match:
+            return normalize_log_level(embedded_match.group(1))
+        upper_text = text.upper()
+        for pattern, normalized in LEVEL_PATTERNS:
+            if f"{pattern}:" in upper_text or f"{pattern} " in upper_text:
+                return normalized
+    return None
+
+
+def build_display_line(timestamp: datetime | None, level: str | None, message: str) -> str:
+    message_part = message.strip() if message else ""
+    parts = []
+    if timestamp is not None:
+        parts.append(timestamp.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S"))
+    if level:
+        parts.append(level.upper())
+    if message_part:
+        parts.append(message_part)
+    return " ".join(parts).strip()
+
+
+def sanitize_text_log_line(line: str) -> str:
+    return CONTROL_CHAR_PATTERN.sub("", line)
+
+
+def parse_text_log_entry(line: str) -> StructuredLogEntry:
+    sanitized_line = sanitize_text_log_line(line).rstrip("\n")
+    timestamp, remainder = parse_prefixed_timestamp(sanitized_line)
+    level = infer_log_level_from_text(remainder or sanitized_line, allow_embedded=False)
+    display_line = sanitized_line
+    return StructuredLogEntry(
+        timestamp=timestamp,
+        level=level,
+        display_line=display_line,
+        raw_line=display_line,
+        search_text=display_line.lower(),
+    )
+
+
+def build_buffer_entry(payload: dict[str, Any]) -> StructuredLogEntry:
+    timestamp = parse_timestamp(str(payload.get("timestamp", "")).strip())
+    level = normalize_log_level(payload.get("level"))
+    if level == LOG_LEVEL_ALL:
+        level = None
+    message = str(payload.get("message", "")).strip()
+    context = payload.get("context")
+    context_map = context if isinstance(context, dict) else {}
+    context_fragments = []
+    for key in ("category", "module", "url", "detail"):
+        value = str(context_map.get(key, "")).strip()
+        if value:
+            context_fragments.append(f"{key}={value}")
+    message_with_context = " | ".join([message, *context_fragments]) if context_fragments else message
+    display_line = build_display_line(timestamp, level, message_with_context)
+    search_text = " ".join(
+        [
+            message,
+            json.dumps(context_map, ensure_ascii=False, sort_keys=True),
+            display_line,
+        ]
+    ).lower()
+    return StructuredLogEntry(
+        timestamp=timestamp,
+        level=level,
+        display_line=display_line,
+        raw_line=json.dumps(payload, ensure_ascii=False, sort_keys=True),
+        search_text=search_text,
+    )
+
+
+def read_file_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
+    path = Path(source.location)
+    if not path.exists():
+        return []
+    with path.open("r", encoding="utf-8", errors="replace") as handle:
+        recent_lines = deque(handle, maxlen=scan_limit)
+    return [
+        parse_text_log_entry(line)
+        for line in recent_lines
+        if sanitize_text_log_line(line).strip()
+    ]
+
+
+def read_docker_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
+    if not shutil.which("docker") or not source.container_name:
+        return []
+    try:
+        completed = subprocess.run(
+            [
+                "docker",
+                "logs",
+                "--timestamps",
+                "--tail",
+                str(scan_limit),
+                source.container_name,
+            ],
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+    except OSError:
+        return []
+    if completed.returncode != 0:
+        return []
+    return [
+        parse_text_log_entry(line)
+        for line in completed.stdout.splitlines()
+        if line.strip()
+    ]
+
+
+def read_buffer_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
+    if not source.buffer_key:
+        return []
+    try:
+        raw_items = redis_client.lrange(source.buffer_key, -scan_limit, -1)
+    except Exception:
+        return []
+    entries: list[StructuredLogEntry] = []
+    for raw_item in raw_items:
+        try:
+            payload = json.loads(raw_item)
+        except json.JSONDecodeError:
+            entries.append(parse_text_log_entry(str(raw_item)))
+            continue
+        if isinstance(payload, dict):
+            entries.append(build_buffer_entry(payload))
+        else:
+            entries.append(parse_text_log_entry(str(raw_item)))
+    return entries
+
+
+def read_source_entries(source: LogSource, scan_limit: int) -> list[StructuredLogEntry]:
+    if source.kind == "file":
+        return read_file_entries(source, scan_limit)
+    if source.kind == "docker":
+        return read_docker_entries(source, scan_limit)
+    if source.kind == "buffer":
+        return read_buffer_entries(source, scan_limit)
+    return []
+
+
+def matches_levels(entry: StructuredLogEntry, selected_levels: tuple[str, ...]) -> bool:
+    if not selected_levels:
+        return True
+    return entry.level in selected_levels
+
+
+def matches_date_range(
+    entry: StructuredLogEntry,
+    start_date: str | None,
+    end_date: str | None,
+) -> bool:
+    if not start_date and not end_date:
+        return True
+    if entry.timestamp is None:
+        return False
+    date_token = entry.timestamp.astimezone(UTC).date().isoformat()
+    if start_date and date_token < start_date:
+        return False
+    if end_date and date_token > end_date:
+        return False
+    return True
+
+
+def matches_search(entry: StructuredLogEntry, search: str | None) -> bool:
+    if search is None:
+        return True
+    query = search.strip().lower()
+    if not query:
+        return True
+    return query in entry.search_text
+
+
+def build_daily_log_markers(entries: list[StructuredLogEntry]) -> list[dict[str, Any]]:
+    grouped: dict[str, list[StructuredLogEntry]] = {}
+    for entry in entries:
+        if entry.timestamp is None:
+            continue
+        date_token = entry.timestamp.astimezone(UTC).date().isoformat()
+        grouped.setdefault(date_token, []).append(entry)
+
+    markers: list[DailyLogMarker] = []
+    for date_token, group in sorted(grouped.items()):
+        level_counts = Counter(
+            entry.level
+            for entry in group
+            if entry.level in SUPPORTED_LOG_LEVELS and entry.level != LOG_LEVEL_ALL
+        )
+        dominant_level = LOG_LEVEL_INFO
+        if level_counts:
+            dominant_level = sorted(
+                level_counts.items(),
+                key=lambda item: (
+                    -item[1],
+                    ("error", "warning", "info", "debug").index(item[0]),
+                ),
+            )[0][0]
+        markers.append(
+            DailyLogMarker(
+                date_token=date_token,
+                total=len(group),
+                dominant_level=dominant_level,
+            )
+        )
+    return [marker.__dict__ for marker in markers]
+
+
+def read_log_snapshot(
+    source_id: str,
+    limit: int,
+    *,
+    level: str = LOG_LEVEL_ALL,
+    levels: str | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    search: str | None = None,
+) -> dict[str, Any] | None:
+    source = LOG_SOURCES.get(source_id)
+    if source is None:
+        return None
+
+    selected_levels = normalize_log_levels(level, levels)
+    search_query = (search or "").strip()
+    scan_limit = max(min(MAX_LOG_LINE_LIMIT * 5, 5000), limit * 5, BUFFER_LOG_LIMIT if source.kind == "buffer" else 1000)
+    all_entries = read_source_entries(source, scan_limit)
+    marker_entries = [
+        entry
+        for entry in all_entries
+        if matches_levels(entry, selected_levels) and matches_search(entry, search_query)
+    ]
+    filtered_entries = [
+        entry
+        for entry in marker_entries
+        if matches_date_range(entry, start_date, end_date)
+    ]
+    visible_entries = filtered_entries[-limit:]
+
+    compatibility_level = selected_levels[0] if len(selected_levels) == 1 else LOG_LEVEL_ALL
+    return {
+        "source_id": source.source_id,
+        "name": source.name,
+        "kind": source.kind,
+        "location": source.location,
+        "description": source.description,
+        "category": source.category,
+        "status": get_source_status(source),
+        "level": compatibility_level,
+        "selected_levels": list(selected_levels),
+        "search_query": search_query,
+        "available_levels": [
+            LOG_LEVEL_ALL,
+            LOG_LEVEL_ERROR,
+            LOG_LEVEL_WARNING,
+            LOG_LEVEL_INFO,
+            LOG_LEVEL_DEBUG,
+        ],
+        "daily_markers": build_daily_log_markers(marker_entries),
+        "line_limit": limit,
+        "line_count": len(visible_entries),
+        "lines": [entry.display_line for entry in visible_entries],
+    }

backend/app/services/vessel_aggregation_strategy.py

@@ -0,0 +1,198 @@
+"""Persistence + validation for the v4 vessel_ais aggregation strategy."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.system_setting import SystemSetting
+
+VESSEL_AGGREGATION_STRATEGY_CATEGORY = "vessel_aggregation_strategy"
+
+DYNAMIC_FIELDS: tuple[str, ...] = ("lat", "lon", "sog", "cog", "heading", "nav_status")
+STATIC_FIELDS: tuple[str, ...] = (
+    "name",
+    "callsign",
+    "imo",
+    "flag",
+    "vessel_type",
+    "vessel_type_name",
+    "length",
+    "width",
+    "draught",
+)
+ALLOWED_FIELDS: frozenset[str] = frozenset(DYNAMIC_FIELDS + STATIC_FIELDS)
+ALLOWED_DYNAMIC_MODES: frozenset[str] = frozenset({"newest"})
+ALLOWED_STATIC_MODES: frozenset[str] = frozenset({"source_priority", "non_empty", "newest", "locked"})
+ALLOWED_LOCKED_DYNAMIC_MODES: frozenset[str] = frozenset({"newest", "source_priority", "locked"})
+
+
+DEFAULT_STRATEGY: dict[str, Any] = {
+    "version": 1,
+    "vessel_ais": {
+        "source_priority": ["aisstream_vessels", "barentswatch_vessels"],
+        "field_rules": {},
+        "freshness": {
+            "realtime_stream_seconds": 900,
+            "polling_seconds": 3600,
+        },
+        "allow_dynamic_lock": False,
+    },
+}
+
+
+class StrategyValidationError(ValueError):
+    """Raised when a saved strategy payload is malformed."""
+
+
+def _coerce_str_list(value: Any, *, label: str) -> list[str]:
+    if value is None:
+        return []
+    if not isinstance(value, list):
+        raise StrategyValidationError(f"{label} must be a list of source names")
+    out: list[str] = []
+    for item in value:
+        if not isinstance(item, str) or not item.strip():
+            raise StrategyValidationError(f"{label} entries must be non-empty strings")
+        out.append(item.strip())
+    return out
+
+
+def validate_strategy(payload: dict[str, Any]) -> dict[str, Any]:
+    """Validate and normalize a strategy payload. Raise StrategyValidationError on issues."""
+
+    if not isinstance(payload, dict):
+        raise StrategyValidationError("strategy payload must be an object")
+
+    vessel_ais = payload.get("vessel_ais")
+    if not isinstance(vessel_ais, dict):
+        raise StrategyValidationError("strategy.vessel_ais is required and must be an object")
+
+    allow_dynamic_lock = bool(vessel_ais.get("allow_dynamic_lock", False))
+    source_priority = _coerce_str_list(
+        vessel_ais.get("source_priority"),
+        label="vessel_ais.source_priority",
+    )
+
+    raw_rules = vessel_ais.get("field_rules") or {}
+    if not isinstance(raw_rules, dict):
+        raise StrategyValidationError("vessel_ais.field_rules must be an object")
+    field_rules: dict[str, dict[str, Any]] = {}
+    for field, rule in raw_rules.items():
+        if field not in ALLOWED_FIELDS:
+            raise StrategyValidationError(f"unknown vessel_ais field: {field}")
+        if not isinstance(rule, dict):
+            raise StrategyValidationError(f"field_rules.{field} must be an object")
+        mode = str(rule.get("mode") or "").strip()
+        if not mode:
+            raise StrategyValidationError(f"field_rules.{field}.mode is required")
+        is_dynamic = field in DYNAMIC_FIELDS
+        if is_dynamic:
+            allowed_modes = ALLOWED_LOCKED_DYNAMIC_MODES if allow_dynamic_lock else ALLOWED_DYNAMIC_MODES
+            if mode not in allowed_modes:
+                if not allow_dynamic_lock:
+                    raise StrategyValidationError(
+                        f"field_rules.{field}.mode='{mode}' requires allow_dynamic_lock=true"
+                    )
+                raise StrategyValidationError(
+                    f"field_rules.{field}.mode must be one of {sorted(allowed_modes)}"
+                )
+        else:
+            if mode not in ALLOWED_STATIC_MODES:
+                raise StrategyValidationError(
+                    f"field_rules.{field}.mode must be one of {sorted(ALLOWED_STATIC_MODES)}"
+                )
+        normalized_rule: dict[str, Any] = {"mode": mode}
+        rule_priority = rule.get("source_priority")
+        if rule_priority is not None:
+            normalized_rule["source_priority"] = _coerce_str_list(
+                rule_priority,
+                label=f"field_rules.{field}.source_priority",
+            )
+        if mode == "locked":
+            locked_source = rule.get("locked_source")
+            if not isinstance(locked_source, str) or not locked_source.strip():
+                raise StrategyValidationError(
+                    f"field_rules.{field}.locked_source must be a non-empty string when mode=locked"
+                )
+            normalized_rule["locked_source"] = locked_source.strip()
+        field_rules[field] = normalized_rule
+
+    raw_freshness = vessel_ais.get("freshness") or {}
+    if not isinstance(raw_freshness, dict):
+        raise StrategyValidationError("vessel_ais.freshness must be an object")
+    freshness: dict[str, int] = {}
+    for key in ("realtime_stream_seconds", "polling_seconds"):
+        value = raw_freshness.get(key, DEFAULT_STRATEGY["vessel_ais"]["freshness"][key])
+        try:
+            seconds = int(value)
+        except (TypeError, ValueError) as exc:
+            raise StrategyValidationError(f"freshness.{key} must be an integer") from exc
+        if seconds < 0:
+            raise StrategyValidationError(f"freshness.{key} must be non-negative")
+        freshness[key] = seconds
+
+    return {
+        "version": int(payload.get("version") or 0) + 1,
+        "vessel_ais": {
+            "source_priority": source_priority,
+            "field_rules": field_rules,
+            "freshness": freshness,
+            "allow_dynamic_lock": allow_dynamic_lock,
+        },
+    }
+
+
+async def _select_setting(db: AsyncSession) -> SystemSetting | None:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == VESSEL_AGGREGATION_STRATEGY_CATEGORY)
+    )
+    return result.scalar_one_or_none()
+
+
+def _current_version(setting: SystemSetting | None) -> int:
+    if setting is None:
+        return 0
+    payload = setting.payload or {}
+    return int(payload.get("version") or 0)
+
+
+async def load_strategy(db: AsyncSession) -> dict[str, Any]:
+    setting = await _select_setting(db)
+    if setting is None or not isinstance(setting.payload, dict):
+        return DEFAULT_STRATEGY
+    payload = setting.payload
+    if "vessel_ais" not in payload:
+        return DEFAULT_STRATEGY
+    return payload
+
+
+async def save_strategy(db: AsyncSession, payload: dict[str, Any]) -> dict[str, Any]:
+    """Validate + persist; bumps version automatically."""
+
+    existing = await _select_setting(db)
+    incoming = dict(payload)
+    incoming.setdefault("version", _current_version(existing))
+    validated = validate_strategy(incoming)
+
+    if existing is None:
+        existing = SystemSetting(category=VESSEL_AGGREGATION_STRATEGY_CATEGORY, payload=validated)
+        db.add(existing)
+    else:
+        existing.payload = validated
+    await db.commit()
+    return validated
+
+
+async def reset_strategy(db: AsyncSession) -> dict[str, Any]:
+    existing = await _select_setting(db)
+    payload = {**DEFAULT_STRATEGY, "version": _current_version(existing) + 1}
+    if existing is None:
+        existing = SystemSetting(category=VESSEL_AGGREGATION_STRATEGY_CATEGORY, payload=payload)
+        db.add(existing)
+    else:
+        existing.payload = payload
+    await db.commit()
+    return payload

backend/app/services/vessel_ais_aggregation.py

@@ -0,0 +1,698 @@
+"""AIS raw observation and aggregation support for vessel collectors."""
+
+from datetime import UTC, datetime, timedelta
+from hashlib import sha256
+import json
+from typing import Any, Iterable
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
+from app.services.vessel_aggregation_strategy import (
+    DEFAULT_STRATEGY,
+    load_strategy,
+)
+from app.services.vessel_types import normalize_vessel_type_name
+
+VESSEL_AIS_SCHEMA = "vessel_ais"
+DEFAULT_AGGREGATION_WINDOW_HOURS = 24
+BARENTSWATCH_DELIVERY_MODE = "polling"
+BARENTSWATCH_TRANSPORT = "http"
+AISSTREAM_DELIVERY_MODE = "realtime_stream"
+AISSTREAM_TRANSPORT = "websocket"
+DELIVERY_MODE_PRIORITY = {
+    "realtime_stream": 40,
+    "batch_stream": 30,
+    "polling": 20,
+    "snapshot": 10,
+}
+DYNAMIC_FIELDS = ("lat", "lon", "sog", "cog", "heading", "nav_status")
+CONFLICT_FIELDS = (
+    "name",
+    "callsign",
+    "imo",
+    "flag",
+    "vessel_type",
+    "vessel_type_name",
+    "length",
+    "width",
+    "draught",
+)
+
+
+def _json_default(value: Any) -> Any:
+    if isinstance(value, datetime):
+        return value.astimezone(UTC).isoformat()
+    return str(value)
+
+
+def _stable_payload(value: Any) -> str:
+    return json.dumps(value, sort_keys=True, separators=(",", ":"), default=_json_default)
+
+
+def _jsonable(value: Any) -> Any:
+    if isinstance(value, datetime):
+        return value.astimezone(UTC).isoformat()
+    if isinstance(value, dict):
+        return {str(key): _jsonable(item) for key, item in value.items()}
+    if isinstance(value, list):
+        return [_jsonable(item) for item in value]
+    return value
+
+
+def _coerce_datetime(value: Any) -> datetime | None:
+    if isinstance(value, datetime):
+        return value if value.tzinfo else value.replace(tzinfo=UTC)
+    if isinstance(value, (int, float)):
+        timestamp = float(value)
+        if timestamp > 10_000_000_000:
+            timestamp /= 1000
+        return datetime.fromtimestamp(timestamp, UTC)
+    if isinstance(value, str) and value:
+        try:
+            parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+            return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+        except ValueError:
+            return None
+    return None
+
+
+def build_observation_hash(
+    *,
+    source: str,
+    entity_key: str,
+    message_type: str | None,
+    observed_at: datetime,
+    normalized_payload: dict[str, Any],
+    source_message_id: str | None = None,
+) -> str:
+    """Build a deterministic idempotency key for one source-level AIS observation."""
+
+    if source_message_id:
+        basis = {
+            "source": source,
+            "entity_key": entity_key,
+            "source_message_id": source_message_id,
+        }
+    else:
+        basis = {
+            "source": source,
+            "entity_key": entity_key,
+            "message_type": message_type,
+            "observed_at": observed_at.astimezone(UTC).isoformat(),
+            "payload": normalized_payload,
+        }
+    return sha256(_stable_payload(basis).encode("utf-8")).hexdigest()
+
+
+def build_field_conflict_candidates(
+    observations: Iterable[AISRawObservation],
+    fields: Iterable[str] = CONFLICT_FIELDS,
+) -> list[dict[str, Any]]:
+    """Return current field disagreements from raw observations without mutating state."""
+
+    candidates_by_field: dict[str, dict[str, Any]] = {}
+    for observation in observations:
+        payload = observation.normalized_payload or {}
+        for field in fields:
+            value = payload.get(field)
+            if value in (None, ""):
+                continue
+            field_candidates = candidates_by_field.setdefault(field, {})
+            field_candidates[observation.source] = value
+
+    conflicts = []
+    for field, candidates in sorted(candidates_by_field.items()):
+        unique_values = {_stable_payload(value) for value in candidates.values()}
+        if len(unique_values) <= 1:
+            continue
+        conflicts.append(
+            {
+                "field": field,
+                "candidates": candidates,
+                "status": "candidate",
+            }
+        )
+    return conflicts
+
+
+def _payload_value(payload: dict[str, Any], field: str) -> Any:
+    value = payload.get(field)
+    return None if value in (None, "") else value
+
+
+def _clean_text(value: Any) -> str | None:
+    if value in (None, ""):
+        return None
+    text = str(value).strip()
+    return text or None
+
+
+def _raw_metadata_value(observation: AISRawObservation, field: str) -> Any:
+    raw_payload = observation.raw_payload or {}
+    metadata = raw_payload.get("MetaData") if isinstance(raw_payload, dict) else None
+    if not isinstance(metadata, dict):
+        return None
+    if field == "name":
+        return _clean_text(metadata.get("ShipName") or metadata.get("ship_name") or metadata.get("name"))
+    return None
+
+
+def _delivery_priority(observation: AISRawObservation) -> int:
+    return DELIVERY_MODE_PRIORITY.get(str(observation.delivery_mode or ""), 0)
+
+
+def _has_valid_position(payload: dict[str, Any]) -> bool:
+    try:
+        lat = float(payload.get("lat"))
+        lon = float(payload.get("lon"))
+    except (TypeError, ValueError):
+        return False
+    return -90 <= lat <= 90 and -180 <= lon <= 180
+
+
+def _is_future_observation(observation: AISRawObservation, now: datetime) -> bool:
+    return observation.observed_at > now
+
+
+def _strategy_source_rank(
+    source: str,
+    strategy: dict[str, Any],
+) -> int:
+    priority = (strategy.get("vessel_ais") or {}).get("source_priority") or []
+    if source in priority:
+        return len(priority) - priority.index(source)
+    return 0
+
+
+def _is_stream_stale(
+    observation: AISRawObservation,
+    *,
+    now: datetime,
+    strategy: dict[str, Any],
+) -> bool:
+    delivery_mode = str(observation.delivery_mode or "")
+    freshness = (strategy.get("vessel_ais") or {}).get("freshness") or {}
+    if delivery_mode == "realtime_stream":
+        window = int(freshness.get("realtime_stream_seconds", 0) or 0)
+    else:
+        window = int(freshness.get("polling_seconds", 0) or 0)
+    if window <= 0:
+        return False
+    return (now - observation.observed_at).total_seconds() > window
+
+
+def _select_position_observation(
+    observations: list[AISRawObservation],
+    *,
+    now: datetime,
+    strategy: dict[str, Any] | None = None,
+) -> tuple[AISRawObservation | None, list[str]]:
+    strategy = strategy or DEFAULT_STRATEGY
+    rejected_flags: list[str] = []
+    fresh_candidates: list[AISRawObservation] = []
+    stale_candidates: list[AISRawObservation] = []
+    for observation in observations:
+        payload = observation.normalized_payload or {}
+        if not _has_valid_position(payload):
+            rejected_flags.append("invalid_position")
+            continue
+        if _is_future_observation(observation, now):
+            rejected_flags.append("future_timestamp")
+            continue
+        if _is_stream_stale(observation, now=now, strategy=strategy):
+            stale_candidates.append(observation)
+            rejected_flags.append("freshness_fallback")
+            continue
+        fresh_candidates.append(observation)
+
+    candidates = fresh_candidates or stale_candidates
+    if not candidates:
+        return None, sorted(set(rejected_flags))
+
+    candidates.sort(
+        key=lambda item: (
+            item.observed_at,
+            _delivery_priority(item),
+            _strategy_source_rank(item.source, strategy),
+            item.collected_at,
+            item.id or 0,
+        ),
+        reverse=True,
+    )
+    return candidates[0], sorted(set(rejected_flags))
+
+
+def _select_static_field(
+    observations: list[AISRawObservation],
+    field: str,
+    strategy: dict[str, Any] | None = None,
+) -> tuple[Any, str | None, str | None]:
+    strategy = strategy or DEFAULT_STRATEGY
+    candidates = []
+    for observation in observations:
+        value = _payload_value(observation.normalized_payload or {}, field)
+        if value is None:
+            value = _raw_metadata_value(observation, field)
+        if value is None:
+            continue
+        candidates.append((observation, value))
+
+    if not candidates:
+        return None, None, None
+
+    field_rules = (strategy.get("vessel_ais") or {}).get("field_rules") or {}
+    rule = field_rules.get(field) or {"mode": "source_priority"}
+    mode = rule.get("mode")
+
+    if mode == "locked":
+        locked_source = rule.get("locked_source")
+        for observation, value in candidates:
+            if observation.source == locked_source:
+                return value, observation.source, "locked"
+
+    if mode in ("source_priority", "locked"):
+        priority = rule.get("source_priority") or (strategy.get("vessel_ais") or {}).get("source_priority") or []
+        ranked = sorted(
+            candidates,
+            key=lambda item: (
+                priority.index(item[0].source) if item[0].source in priority else len(priority) + 1,
+                -_delivery_priority(item[0]),
+                -(item[0].observed_at.timestamp() if item[0].observed_at else 0),
+            ),
+        )
+        observation, value = ranked[0]
+        return value, observation.source, "source_priority"
+
+    if mode == "newest":
+        ranked = sorted(
+            candidates,
+            key=lambda item: (item[0].observed_at, _delivery_priority(item[0]), item[0].id or 0),
+            reverse=True,
+        )
+        observation, value = ranked[0]
+        return value, observation.source, "newest_observation"
+
+    # default / non_empty: prefer delivery mode priority, then newest
+    candidates.sort(
+        key=lambda item: (
+            _delivery_priority(item[0]),
+            item[0].observed_at,
+            item[0].collected_at,
+            item[0].id or 0,
+        ),
+        reverse=True,
+    )
+    selected_observation, selected_value = candidates[0]
+    unique_values = {_stable_payload(value) for _, value in candidates}
+    reason = "delivery_mode_priority" if len(unique_values) > 1 else "non_empty_priority"
+    return selected_value, selected_observation.source, reason
+
+
+def _build_source_summary(observations: list[AISRawObservation]) -> dict[str, dict[str, Any]]:
+    summary: dict[str, dict[str, Any]] = {}
+    for observation in observations:
+        source_summary = summary.setdefault(
+            observation.source,
+            {
+                "observation_count": 0,
+                "latest_observed_at": None,
+                "delivery_mode": observation.delivery_mode,
+                "transport": observation.transport,
+                "message_types": [],
+            },
+        )
+        source_summary["observation_count"] += 1
+        latest_observed_at = source_summary["latest_observed_at"]
+        if latest_observed_at is None or observation.observed_at > latest_observed_at:
+            source_summary["latest_observed_at"] = observation.observed_at
+        if observation.message_type and observation.message_type not in source_summary["message_types"]:
+            source_summary["message_types"].append(observation.message_type)
+    return summary
+
+
+def _build_aggregated_vessel(
+    entity_key: str,
+    observations: list[AISRawObservation],
+    *,
+    now: datetime,
+    strategy: dict[str, Any] | None = None,
+) -> dict[str, Any] | None:
+    strategy = strategy or DEFAULT_STRATEGY
+    position_observation, rejected_flags = _select_position_observation(
+        observations, now=now, strategy=strategy
+    )
+    if position_observation is None:
+        return None
+
+    payload = position_observation.normalized_payload or {}
+    mmsi = int(entity_key)
+    result: dict[str, Any] = {
+        "mmsi": mmsi,
+        "lat": float(payload["lat"]),
+        "lon": float(payload["lon"]),
+        "received_at": position_observation.observed_at,
+        "field_sources": {},
+        "selected_reasons": {},
+        "source_summary": _build_source_summary(observations),
+        "quality_flags": sorted(
+            set((position_observation.quality_flags or []) + rejected_flags)
+        ),
+        "aggregation_strategy_version": int(strategy.get("version") or 0),
+    }
+
+    for field in DYNAMIC_FIELDS:
+        value = _payload_value(payload, field)
+        if field in ("lat", "lon") or value is not None:
+            result[field] = value
+            result["field_sources"][field] = position_observation.source
+            result["selected_reasons"][field] = "newest_observation"
+
+    for field in CONFLICT_FIELDS:
+        selected_value, selected_source, reason = _select_static_field(
+            observations, field, strategy=strategy
+        )
+        if selected_value is None:
+            continue
+        result[field] = selected_value
+        result["field_sources"][field] = selected_source
+        result["selected_reasons"][field] = reason
+
+    result["name"] = result.get("name") or f"MMSI {mmsi}"
+    result["vessel_type_name"] = result.get("vessel_type_name") or normalize_vessel_type_name(
+        result.get("vessel_type")
+    )
+    return result
+
+
+async def _upsert_conflict_records(
+    db: AsyncSession,
+    entity_key: str,
+    observations: list[AISRawObservation],
+    aggregated: dict[str, Any],
+) -> int:
+    conflicts = build_field_conflict_candidates(observations)
+    now = datetime.now(UTC)
+    for conflict in conflicts:
+        field = conflict["field"]
+        result = await db.execute(
+            select(AISConflictRecord)
+            .where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
+            .where(AISConflictRecord.entity_key == entity_key)
+            .where(AISConflictRecord.field == field)
+            .limit(1)
+        )
+        record = result.scalar_one_or_none()
+        if record is None:
+            record = AISConflictRecord(
+                target_schema=VESSEL_AIS_SCHEMA,
+                entity_key=entity_key,
+                field=field,
+            )
+            db.add(record)
+        record.candidates = conflict["candidates"]
+        record.selected_source = (aggregated.get("field_sources") or {}).get(field)
+        record.selected_value = aggregated.get(field)
+        record.selected_reason = (aggregated.get("selected_reasons") or {}).get(field)
+        record.resolved_by = "system"
+        record.status = "open"
+        record.updated_at = now
+    return len(conflicts)
+
+
+def _group_observations(observations: Iterable[AISRawObservation]) -> dict[str, list[AISRawObservation]]:
+    grouped: dict[str, list[AISRawObservation]] = {}
+    for observation in observations:
+        grouped.setdefault(str(observation.entity_key), []).append(observation)
+    return grouped
+
+
+async def record_vessel_ais_observation(
+    db: AsyncSession,
+    *,
+    source: str,
+    normalized_payload: dict[str, Any],
+    raw_payload: dict[str, Any] | None = None,
+    delivery_mode: str,
+    transport: str,
+    message_type: str | None = "PositionReport",
+    source_message_id: str | None = None,
+    observed_at: datetime | None = None,
+    collected_at: datetime | None = None,
+    quality_flags: list[str] | None = None,
+) -> AISRawObservation | None:
+    """Insert one raw observation if the source-level fact has not already been stored."""
+
+    entity_key = str(normalized_payload["mmsi"])
+    collected_at = collected_at or datetime.now(UTC)
+    observed_at = (
+        _coerce_datetime(observed_at)
+        or _coerce_datetime(normalized_payload.get("received_at"))
+        or collected_at
+    )
+    normalized_json = _jsonable(normalized_payload)
+    raw_json = _jsonable(raw_payload or {})
+
+    observation_hash = build_observation_hash(
+        source=source,
+        entity_key=entity_key,
+        message_type=message_type,
+        observed_at=observed_at,
+        normalized_payload=normalized_json,
+        source_message_id=source_message_id,
+    )
+    existing_result = await db.execute(
+        select(AISRawObservation.id).where(AISRawObservation.observation_hash == observation_hash)
+    )
+    if existing_result.scalar_one_or_none() is not None:
+        return None
+
+    observation = AISRawObservation(
+        target_schema=VESSEL_AIS_SCHEMA,
+        source=source,
+        entity_key=entity_key,
+        delivery_mode=delivery_mode,
+        transport=transport,
+        message_type=message_type,
+        source_message_id=source_message_id,
+        observation_hash=observation_hash,
+        observed_at=observed_at,
+        collected_at=collected_at,
+        normalized_payload=normalized_json,
+        raw_payload=raw_json,
+        quality_flags=quality_flags or [],
+    )
+    db.add(observation)
+    return observation
+
+
+async def aggregate_vessel_observations(
+    db: AsyncSession,
+    observations: Iterable[AISRawObservation],
+    *,
+    write_conflicts: bool = False,
+    strategy: dict[str, Any] | None = None,
+) -> list[dict[str, Any]]:
+    strategy = strategy if strategy is not None else await _safe_load_strategy(db)
+    now = datetime.now(UTC)
+    vessels = []
+    for entity_key, entity_observations in _group_observations(observations).items():
+        aggregated = _build_aggregated_vessel(
+            entity_key, entity_observations, now=now, strategy=strategy
+        )
+        if aggregated is None:
+            continue
+        if write_conflicts:
+            aggregated["conflict_count"] = await _upsert_conflict_records(
+                db,
+                entity_key,
+                entity_observations,
+                aggregated,
+            )
+        else:
+            aggregated["conflict_count"] = len(build_field_conflict_candidates(entity_observations))
+        vessels.append(aggregated)
+
+    vessels.sort(key=lambda item: item.get("received_at") or datetime.min.replace(tzinfo=UTC), reverse=True)
+    return vessels
+
+
+async def _safe_load_strategy(db: AsyncSession) -> dict[str, Any]:
+    """Tolerate fake test sessions where load_strategy may misbehave."""
+    try:
+        return await load_strategy(db)
+    except Exception:
+        return DEFAULT_STRATEGY
+
+
+async def get_aggregated_vessels(
+    db: AsyncSession,
+    *,
+    bbox: tuple[float, float, float, float] | None = None,
+    limit: int | None = None,
+    observed_since: datetime | None = None,
+) -> list[dict[str, Any]]:
+    observed_since = observed_since or (
+        datetime.now(UTC) - timedelta(hours=DEFAULT_AGGREGATION_WINDOW_HOURS)
+    )
+    stmt = (
+        select(AISRawObservation)
+        .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
+        .where(AISRawObservation.observed_at >= observed_since)
+        .order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
+    )
+    if limit and limit > 0:
+        stmt = stmt.limit(max(limit * 20, limit))
+
+    result = await db.execute(stmt)
+    if not hasattr(result, "scalars"):
+        return []
+    vessels = await aggregate_vessel_observations(db, result.scalars().all())
+
+    if bbox is not None:
+        lon_min, lat_min, lon_max, lat_max = bbox
+        vessels = [
+            vessel
+            for vessel in vessels
+            if lon_min <= float(vessel["lon"]) <= lon_max
+            and lat_min <= float(vessel["lat"]) <= lat_max
+        ]
+
+    if limit and limit > 0:
+        return vessels[:limit]
+    return vessels
+
+
+async def get_aggregated_vessel(db: AsyncSession, mmsi: int) -> dict[str, Any] | None:
+    observations = await get_vessel_raw_observations(db, mmsi, limit=1000)
+    vessels = await aggregate_vessel_observations(db, observations)
+    return vessels[0] if vessels else None
+
+
+async def get_aggregated_vessel_track(
+    db: AsyncSession,
+    mmsi: int,
+    *,
+    cutoff: datetime,
+) -> list[dict[str, Any]]:
+    result = await db.execute(
+        select(AISRawObservation)
+        .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
+        .where(AISRawObservation.entity_key == str(mmsi))
+        .where(AISRawObservation.observed_at >= cutoff)
+        .order_by(AISRawObservation.observed_at.asc(), AISRawObservation.id.asc())
+    )
+    if not hasattr(result, "scalars"):
+        return []
+
+    points: list[dict[str, Any]] = []
+    seen: set[tuple[str, float, float, str]] = set()
+    for observation in result.scalars().all():
+        payload = observation.normalized_payload or {}
+        if not _has_valid_position(payload):
+            continue
+        lat = float(payload["lat"])
+        lon = float(payload["lon"])
+        key = (
+            observation.observed_at.isoformat(),
+            round(lat, 5),
+            round(lon, 5),
+            observation.source,
+        )
+        if key in seen:
+            continue
+        seen.add(key)
+        points.append(
+            {
+                "lat": lat,
+                "lon": lon,
+                "observed_at": observation.observed_at,
+                "source": observation.source,
+                "selected_reason": "track_timeline",
+                "quality_flags": observation.quality_flags or [],
+            }
+        )
+    return points
+
+
+async def update_ais_source_health(
+    db: AsyncSession,
+    *,
+    source: str,
+    connection_state: str,
+    observed_count: int = 0,
+    last_seen_at: datetime | None = None,
+    last_success_at: datetime | None = None,
+    last_error: str | None = None,
+    lag_seconds: float | None = None,
+) -> AISSourceHealth:
+    """Upsert the health row for an AIS source."""
+
+    now = datetime.now(UTC)
+    health = await db.get(AISSourceHealth, source)
+    if health is None:
+        health = AISSourceHealth(source=source)
+        db.add(health)
+
+    health.connection_state = connection_state
+    health.last_seen_at = last_seen_at or health.last_seen_at
+    health.last_success_at = last_success_at or health.last_success_at
+    health.last_error = last_error
+    health.message_rate = float(observed_count)
+    health.lag_seconds = lag_seconds
+    health.updated_at = now
+    return health
+
+
+async def count_unique_raw_vessel_mmsi(
+    db: AsyncSession,
+    *,
+    observed_since: datetime | None = None,
+) -> int:
+    """Count unique raw vessel MMSI values for HUD counts; never aggregates."""
+    from sqlalchemy import func as sa_func
+
+    unique_mmsi_stmt = (
+        select(AISRawObservation.entity_key)
+        .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
+        .distinct()
+    )
+    if observed_since is not None:
+        unique_mmsi_stmt = unique_mmsi_stmt.where(
+            AISRawObservation.observed_at >= observed_since,
+        )
+
+    result = await db.execute(
+        select(sa_func.count()).select_from(unique_mmsi_stmt.subquery()),
+    )
+    return int(result.scalar() or 0)
+
+
+async def get_vessel_raw_observations(
+    db: AsyncSession,
+    mmsi: int,
+    *,
+    limit: int = 100,
+) -> list[AISRawObservation]:
+    result = await db.execute(
+        select(AISRawObservation)
+        .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
+        .where(AISRawObservation.entity_key == str(mmsi))
+        .order_by(AISRawObservation.observed_at.desc(), AISRawObservation.id.desc())
+        .limit(limit)
+    )
+    return list(result.scalars().all())
+
+
+async def get_vessel_conflict_records(
+    db: AsyncSession,
+    mmsi: int,
+) -> list[AISConflictRecord]:
+    result = await db.execute(
+        select(AISConflictRecord)
+        .where(AISConflictRecord.target_schema == VESSEL_AIS_SCHEMA)
+        .where(AISConflictRecord.entity_key == str(mmsi))
+        .order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
+    )
+    return list(result.scalars().all())

backend/app/services/vessel_enrichment.py

@@ -0,0 +1,109 @@
+"""v5 vessel enrichment service.
+
+Read-only side: `get_vessel_enrichment_bundle` is the only path the
+aggregation/detail endpoints use. It never reaches out to third parties; it
+just returns whatever the upsert side has already cached. Expired rows are
+filtered out so old data never leaks back into the live UI.
+"""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.vessel_enrichment import VesselMediaEnrichment, VesselProfileEnrichment
+
+
+def _coerce_datetime(value: Any) -> datetime | None:
+    if value in (None, ""):
+        return None
+    if isinstance(value, datetime):
+        return value if value.tzinfo else value.replace(tzinfo=UTC)
+    if isinstance(value, (int, float)):
+        ts = float(value)
+        if ts > 10_000_000_000:
+            ts /= 1000
+        return datetime.fromtimestamp(ts, UTC)
+    if isinstance(value, str):
+        try:
+            parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+            return parsed if parsed.tzinfo else parsed.replace(tzinfo=UTC)
+        except ValueError:
+            return None
+    return None
+
+
+def _build_payload(record, *, now: datetime) -> dict[str, Any] | None:
+    if record is None:
+        return None
+    expires_at = record.expires_at
+    if isinstance(expires_at, datetime):
+        if expires_at.tzinfo is None:
+            expires_at = expires_at.replace(tzinfo=UTC)
+        if expires_at < now:
+            return None
+    return record.to_dict()
+
+
+async def get_vessel_enrichment_bundle(db: AsyncSession, mmsi: int) -> dict[str, Any]:
+    now = datetime.now(UTC)
+    profile = await db.get(VesselProfileEnrichment, mmsi)
+    media = await db.get(VesselMediaEnrichment, mmsi)
+    return {
+        "mmsi": mmsi,
+        "profile": _build_payload(profile, now=now),
+        "media": _build_payload(media, now=now),
+    }
+
+
+async def upsert_vessel_profile_enrichment(
+    db: AsyncSession,
+    *,
+    mmsi: int,
+    payload: dict[str, Any],
+) -> dict[str, Any]:
+    record = await db.get(VesselProfileEnrichment, mmsi)
+    if record is None:
+        record = VesselProfileEnrichment(mmsi=mmsi)
+        db.add(record)
+    return _apply_upsert(record, payload)
+
+
+async def upsert_vessel_media_enrichment(
+    db: AsyncSession,
+    *,
+    mmsi: int,
+    payload: dict[str, Any],
+) -> dict[str, Any]:
+    record = await db.get(VesselMediaEnrichment, mmsi)
+    if record is None:
+        record = VesselMediaEnrichment(mmsi=mmsi)
+        db.add(record)
+    return _apply_upsert(record, payload)
+
+
+def _apply_upsert(record, payload: dict[str, Any]) -> dict[str, Any]:
+    if not isinstance(payload, dict):
+        raise ValueError("enrichment payload must be an object")
+    body = payload.get("payload")
+    if body is not None and not isinstance(body, dict):
+        raise ValueError("payload.payload must be an object")
+    if body is not None:
+        record.payload = body
+    if "source" in payload and isinstance(payload["source"], str) and payload["source"].strip():
+        record.source = payload["source"].strip()
+    fetched_at = _coerce_datetime(payload.get("fetched_at"))
+    record.fetched_at = fetched_at or datetime.now(UTC)
+    record.expires_at = _coerce_datetime(payload.get("expires_at"))
+    confidence = payload.get("confidence")
+    if confidence is not None:
+        try:
+            record.confidence = float(confidence)
+        except (TypeError, ValueError):
+            record.confidence = None
+    if "reference_url" in payload:
+        ref = payload.get("reference_url")
+        record.reference_url = str(ref) if ref else None
+    return record.to_dict()

backend/app/services/vessel_types.py

@@ -0,0 +1,31 @@
+"""Shared AIS vessel type helpers."""
+
+from typing import Any
+
+VESSEL_TYPE_NAMES = {
+    30: "Fishing",
+    35: "Military",
+    60: "Passenger",
+    70: "Cargo",
+    80: "Tanker",
+}
+
+
+def normalize_vessel_type_name(vessel_type: Any) -> str:
+    """Map AIS numeric vessel type codes to display buckets."""
+
+    try:
+        type_code = int(float(vessel_type))
+    except (TypeError, ValueError):
+        return "Other"
+    if 70 <= type_code <= 79:
+        return "Cargo"
+    if 80 <= type_code <= 89:
+        return "Tanker"
+    if 60 <= type_code <= 69:
+        return "Passenger"
+    if type_code == 30:
+        return "Fishing"
+    if type_code == 35:
+        return "Military"
+    return VESSEL_TYPE_NAMES.get(type_code, "Other")

backend/scripts/system_restart_runner.py

@@ -2,6 +2,7 @@ from __future__ import annotations
 
 import argparse
 import os
+import shlex
 import subprocess
 import sys
 import time
@@ -59,6 +60,8 @@ def wait_for_recovery(action: str) -> tuple[bool, str]:
     recovery_mode = get_action_recovery_mode(action)
     if recovery_mode == "backend":
         return wait_for_http("http://localhost:8000/health"), "backend health recovery"
+    if recovery_mode == "frontend":
+        return wait_for_http("http://localhost:3000"), "frontend entrypoint recovery"
     if recovery_mode == "ai-provider":
         return wait_for_http("http://localhost:8010/health"), "ai provider health recovery"
     if recovery_mode == "database":
@@ -108,8 +111,9 @@ def main() -> int:
         )
         append_task_log(args.task_id, "restart command started")
 
+        shell_command = shlex.join(command)
         completed = subprocess.run(
-            command,
+            ["zsh", "-ic", shell_command],
             cwd=str(ROOT_DIR),
             env=env,
             capture_output=True,

backend/tests/test_api.py

@@ -35,6 +35,7 @@ async def test_health_check():
     data = response.json()
     assert data["status"] == "healthy"
     assert "version" in data
+    assert response.headers["x-request-id"]
 
 
 @pytest.mark.asyncio
@@ -161,6 +162,345 @@ async def test_alerts_endpoint_with_auth(auth_headers):
         app.dependency_overrides.clear()
 
 
+@pytest.mark.asyncio
+async def test_system_log_sources_requires_super_admin(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
+        assert response.status_code == 403
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_sources_with_super_admin(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.system_control.list_log_sources",
+            return_value=[
+                {
+                    "source_id": "backend",
+                    "name": "后端服务",
+                    "kind": "file",
+                    "location": "/tmp/planet_backend.log",
+                    "description": "FastAPI 后端、调度器和采集任务共享日志。",
+                    "category": "service",
+                    "status": "ok",
+                }
+            ],
+        ):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get("/api/v1/system/logs/sources", headers=auth_headers)
+        assert response.status_code == 200
+        data = response.json()
+        assert data["items"][0]["source_id"] == "backend"
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_snapshot_with_super_admin(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.system_control.read_log_snapshot",
+            return_value={
+                "source_id": "backend",
+                "name": "后端服务",
+                "kind": "file",
+                "location": "/tmp/planet_backend.log",
+                "description": "FastAPI 后端、调度器和采集任务共享日志。",
+                "category": "service",
+                "status": "ok",
+                "level": "all",
+                "selected_levels": [],
+                "search_query": "",
+                "available_levels": ["all", "error", "warning", "info", "debug"],
+                "daily_markers": [],
+                "line_limit": 50,
+                "line_count": 2,
+                "lines": ["line 1", "line 2"],
+            },
+        ):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get("/api/v1/system/logs/backend?limit=50", headers=auth_headers)
+        assert response.status_code == 200
+        data = response.json()
+        assert data["source_id"] == "backend"
+        assert data["line_count"] == 2
+        assert data["lines"] == ["line 1", "line 2"]
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_snapshot_supports_level_filter(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.system_control.read_log_snapshot",
+            return_value={
+                "source_id": "backend",
+                "name": "后端服务",
+                "kind": "file",
+                "location": "/tmp/planet_backend.log",
+                "description": "FastAPI 后端、调度器和采集任务共享日志。",
+                "category": "service",
+                "status": "ok",
+                "level": "error",
+                "selected_levels": ["error"],
+                "search_query": "",
+                "available_levels": ["all", "error", "warning", "info", "debug"],
+                "daily_markers": [],
+                "line_limit": 50,
+                "line_count": 1,
+                "lines": ["ERROR: failed"],
+            },
+        ):
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get("/api/v1/system/logs/backend?limit=50&level=error", headers=auth_headers)
+        assert response.status_code == 200
+        data = response.json()
+        assert data["level"] == "error"
+        assert data["lines"] == ["ERROR: failed"]
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_snapshot_supports_date_range_filter(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.system_control.read_log_snapshot",
+            return_value={
+                "source_id": "backend",
+                "name": "后端服务",
+                "kind": "file",
+                "location": "/tmp/planet_backend.log",
+                "description": "FastAPI 后端、调度器和采集任务共享日志。",
+                "category": "service",
+                "status": "ok",
+                "level": "all",
+                "selected_levels": [],
+                "search_query": "",
+                "available_levels": ["all", "error", "warning", "info", "debug"],
+                "daily_markers": [],
+                "line_limit": 50,
+                "line_count": 1,
+                "lines": ["2026-04-23 INFO: service started"],
+            },
+        ) as mock_read_log_snapshot:
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get(
+                    "/api/v1/system/logs/backend?limit=50&start_date=2026-04-20&end_date=2026-04-23",
+                    headers=auth_headers,
+                )
+        assert response.status_code == 200
+        mock_read_log_snapshot.assert_called_once_with(
+            "backend",
+            50,
+            level="all",
+            levels=None,
+            start_date="2026-04-20",
+            end_date="2026-04-23",
+            search=None,
+        )
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_snapshot_supports_levels_and_search_filter(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        with patch(
+            "app.api.v1.system_control.read_log_snapshot",
+            return_value={
+                "source_id": "backend",
+                "name": "后端服务",
+                "kind": "file",
+                "location": "/tmp/planet_backend.log",
+                "description": "FastAPI 后端、调度器和采集任务共享日志。",
+                "category": "service",
+                "status": "ok",
+                "level": "all",
+                "selected_levels": ["error", "warning"],
+                "search_query": "timeout",
+                "available_levels": ["all", "error", "warning", "info", "debug"],
+                "daily_markers": [],
+                "line_limit": 50,
+                "line_count": 1,
+                "lines": ["2026-04-23 10:00:00 ERROR timeout"],
+            },
+        ) as mock_read_log_snapshot:
+            async with AsyncClient(transport=transport, base_url="http://test") as client:
+                response = await client.get(
+                    "/api/v1/system/logs/backend?limit=50&levels=error,warning&search=timeout",
+                    headers=auth_headers,
+                )
+        assert response.status_code == 200
+        mock_read_log_snapshot.assert_called_once_with(
+            "backend",
+            50,
+            level="all",
+            levels="error,warning",
+            start_date=None,
+            end_date=None,
+            search="timeout",
+        )
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_system_log_snapshot_rejects_invalid_date_range(auth_headers):
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="root",
+            email="root@example.com",
+            password_hash="hashed",
+            role="super_admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {
+        __import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get(
+                "/api/v1/system/logs/backend?start_date=2026-04-31",
+                headers=auth_headers,
+            )
+        assert response.status_code == 400
+        assert "start_date must be in YYYY-MM-DD format" in response.json()["detail"]
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_ingest_earth_client_log_accepts_public_events():
+    transport = ASGITransport(app=app)
+    try:
+        with patch("app.api.v1.system_control.append_buffer_log") as mock_append_buffer_log:
+            with patch("app.api.v1.system_control.record_system_log", new_callable=AsyncMock) as mock_record_system_log:
+                async with AsyncClient(transport=transport, base_url="http://test") as client:
+                    response = await client.post(
+                        "/api/v1/system/logs/earth-client",
+                        json={
+                            "level": "error",
+                            "message": "登陆点加载失败: 登陆点接口返回 HTTP 500",
+                            "category": "startup-load",
+                            "module": "layer-startup",
+                        },
+                    )
+        assert response.status_code == 200
+        data = response.json()
+        assert data["accepted"] is True
+        assert data["source_id"] == "earth-client"
+        mock_append_buffer_log.assert_called_once()
+        mock_record_system_log.assert_awaited_once()
+        persisted_kwargs = mock_record_system_log.await_args.kwargs
+        assert persisted_kwargs["source"] == "earth-client"
+        assert persisted_kwargs["event"] == "earth.client.runtime_log"
+        assert persisted_kwargs["category"] == "startup-load"
+        assert persisted_kwargs["level"] == "error"
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_request_id_header_is_echoed_when_provided():
+    transport = ASGITransport(app=app)
+    async with AsyncClient(transport=transport, base_url="http://test") as client:
+        response = await client.get("/health", headers={"X-Request-ID": "planet-test-request"})
+    assert response.status_code == 200
+    assert response.headers["x-request-id"] == "planet-test-request"
+
+
 @pytest.mark.asyncio
 async def test_invalid_token():
     """Test that invalid token is rejected"""
@@ -263,6 +603,8 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
         assert "content_blocks" in data
         assert "text_blocks" in data
         assert "thinking_blocks" in data
+    finally:
+        app.dependency_overrides.clear()
 
 
 @pytest.mark.asyncio
@@ -382,8 +724,6 @@ async def test_save_playground_session_with_auth(auth_headers):
         assert data["state"]["objective"] == "测试目标"
     finally:
         app.dependency_overrides.clear()
-    finally:
-        app.dependency_overrides.clear()
 
 
 @pytest.mark.asyncio

backend/tests/test_collectors.py

@@ -1,11 +1,14 @@
 """Unit tests for data collectors"""
 
 import pytest
-from datetime import datetime
-from unittest.mock import AsyncMock, MagicMock, patch
+from unittest.mock import AsyncMock, patch
 
+from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.services.credential_guides import DEFAULT_CREDENTIAL_GUIDES
 from app.services.collectors.top500 import TOP500Collector
-from app.services.collectors.base import BaseCollector, HTTPCollector
+from app.services.collectors.registry import collector_registry
+from app.services.datasource_connectivity import SUPPORTED_CREDENTIAL_PROVIDERS
+from app.models.task import CollectionTask
 
 
 class TestBaseCollector:
@@ -19,6 +22,31 @@ class TestBaseCollector:
         assert collector.module == "L1"
         assert collector.frequency_hours == 4
 
+    @pytest.mark.asyncio
+    async def test_update_phase_progress_tracks_phase_fields(self, mock_db_session):
+        """Test phase-level progress updates independently from record totals"""
+        collector = TOP500Collector()
+        task = CollectionTask(datasource_id=1, status="running", phase="fetching")
+        collector._current_task = task
+        collector._db_session = mock_db_session
+
+        with patch.object(collector, "_publish_task_update", new=AsyncMock()) as publish:
+            await collector.update_phase_progress(
+                current=512,
+                total=1024,
+                unit="bytes",
+                message="Downloading dataset",
+                commit=True,
+            )
+
+        assert task.phase_progress == 50.0
+        assert task.phase_current == 512
+        assert task.phase_total == 1024
+        assert task.phase_unit == "bytes"
+        assert task.phase_message == "Downloading dataset"
+        mock_db_session.commit.assert_awaited_once()
+        publish.assert_awaited_once()
+
 
 class TestTOP500Collector:
     """Tests for TOP500Collector"""
@@ -119,3 +147,30 @@ class TestHTTPCollector:
         assert hasattr(collector, "parse_response")
         assert callable(collector.fetch)
         assert callable(collector.parse_response)
+
+
+def test_aisstream_collector_is_registered():
+    collector = collector_registry.get("aisstream_vessels")
+
+    assert collector is not None
+    assert collector.data_type == "vessel_ais"
+
+
+def test_supported_credential_collectors_have_guides_and_connectivity_provider():
+    missing: list[str] = []
+    for source, info in DEFAULT_DATASOURCES.items():
+        if not info.get("requires_credentials"):
+            continue
+        if info.get("credential_status") != "supported":
+            continue
+
+        provider = info.get("credential_provider")
+        if not provider:
+            missing.append(f"{source}: missing credential_provider")
+            continue
+        if provider not in DEFAULT_CREDENTIAL_GUIDES:
+            missing.append(f"{source}: missing credential guide for {provider}")
+        if provider not in SUPPORTED_CREDENTIAL_PROVIDERS:
+            missing.append(f"{source}: missing connectivity provider for {provider}")
+
+    assert missing == []

backend/tests/test_custom_datasource_runtime_live.py

@@ -0,0 +1,149 @@
+"""End-to-end integration test for the custom WebSocket datasource runner.
+
+Boots an in-process WebSocket server that mimics the bun mock AIS server
+(`scripts/mock-ais-ws-server.ts`) and runs the real
+`run_mapped_websocket_config` against it. Catches regressions where the
+runner stops connecting, fails to extract the configured message path,
+or quietly drops mapped records before broadcasting.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from contextlib import asynccontextmanager
+from datetime import UTC, datetime
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import pytest
+import websockets
+
+from app.models.datasource_config import DataSourceConfig
+from app.services import custom_datasource_runtime
+from app.services.custom_datasource_runtime import run_mapped_websocket_config
+
+
+def _make_payload(seq: int) -> str:
+    return json.dumps(
+        {
+            "type": "vessel",
+            "sequence": seq,
+            "data": {
+                "mmsi": str(999_000_000 + seq),
+                "name": f"MOCK VESSEL {seq:03d}",
+                "lat": 36.20 + seq * 0.001,
+                "lon": 14.20 + seq * 0.001,
+                "sog": 12.0,
+                "cog": 90.0,
+                "heading": 90,
+                "vessel_type": 70,
+                "vessel_type_name": "Cargo",
+                "received_at": datetime.now(UTC).isoformat(),
+            },
+        }
+    )
+
+
+@asynccontextmanager
+async def _mock_ais_server(emit_count: int):
+    received_subscribe: list[str] = []
+
+    async def handler(ws):
+        try:
+            try:
+                msg = await asyncio.wait_for(ws.recv(), timeout=0.5)
+                received_subscribe.append(msg)
+            except (asyncio.TimeoutError, websockets.ConnectionClosed):
+                pass
+            for seq in range(1, emit_count + 1):
+                await ws.send(_make_payload(seq))
+                await asyncio.sleep(0.01)
+            # keep the socket open briefly so the runner observes the messages
+            await asyncio.sleep(0.05)
+        except websockets.ConnectionClosed:
+            return
+
+    async with websockets.serve(handler, "127.0.0.1", 0) as server:
+        port = next(iter(server.sockets)).getsockname()[1]
+        yield port, received_subscribe
+
+
+@pytest.mark.asyncio
+async def test_websocket_runner_streams_from_live_mock(monkeypatch):
+    mapping = SimpleNamespace(
+        id=11,
+        version=3,
+        target_schema="vessel_ais",
+        mapping_json={
+            "source": {"items_path": "$"},
+            "fields": {
+                "mmsi": {"path": "$.mmsi", "type": "integer"},
+                "lat": {"path": "$.lat", "type": "float"},
+                "lon": {"path": "$.lon", "type": "float"},
+                "name": {"path": "$.name", "type": "string"},
+                "vessel_type": {"path": "$.vessel_type", "type": "integer", "default": None},
+                "vessel_type_name": {"path": "$.vessel_type_name", "type": "string", "default": None},
+                "sog": {"path": "$.sog", "type": "float", "default": None},
+                "cog": {"path": "$.cog", "type": "float", "default": None},
+                "heading": {"path": "$.heading", "type": "integer", "default": None},
+                "received_at": {"path": "$.received_at", "type": "datetime"},
+            },
+        },
+    )
+
+    class FakeResult:
+        def scalar_one_or_none(self):
+            return mapping
+
+    class FakeDB:
+        async def execute(self, _stmt):
+            return FakeResult()
+
+    persist = AsyncMock(return_value=1)
+    monkeypatch.setattr(custom_datasource_runtime, "persist_mapped_records", persist)
+
+    async with _mock_ais_server(emit_count=3) as (port, received_subscribe):
+        result = await run_mapped_websocket_config(
+            FakeDB(),
+            DataSourceConfig(
+                id=99,
+                name="mock_ais_ws",
+                source_type="websocket",
+                endpoint=f"ws://127.0.0.1:{port}",
+                auth_type="none",
+                headers={},
+                config={
+                    "ws_message_path": "$.data",
+                    "ws_subscribe_message": {
+                        "type": "subscribe",
+                        "anchor": {"lat": 36.2, "lon": 14.2},
+                        "spread_km": 50,
+                        "rate_hz": 1,
+                    },
+                    "debug_max_messages": 2,
+                    "delivery_mode": "realtime_stream",
+                    "ws_reconnect": False,
+                },
+            ),
+            use_config_debug_max_messages=True,
+        )
+
+    assert result["status"] == "success"
+    assert result["messages_seen"] == 2
+    assert result["written_count"] == 2
+    assert result["mapped_count"] == 2
+    assert result["target_schema"] == "vessel_ais"
+    # subscribe message must reach the server unchanged
+    assert received_subscribe, "runner did not forward ws_subscribe_message"
+    parsed = json.loads(received_subscribe[0])
+    assert parsed["type"] == "subscribe"
+    assert parsed["anchor"] == {"lat": 36.2, "lon": 14.2}
+    assert parsed["rate_hz"] == 1
+    # mapped records carry the real MMSIs from the mock stream
+    persisted_records = []
+    for call in persist.await_args_list:
+        persisted_records.extend(call.kwargs["records"])
+    assert {record["mmsi"] for record in persisted_records} == {999_000_001, 999_000_002}
+    assert all(record["vessel_type"] == 70 for record in persisted_records)
+    assert all(record["vessel_type_name"] == "Cargo" for record in persisted_records)

backend/tests/test_datasource_mapping.py

@@ -0,0 +1,328 @@
+from types import SimpleNamespace
+
+import pytest
+from unittest.mock import AsyncMock
+from httpx import ASGITransport, AsyncClient
+
+from app.api.v1.datasource_config import get_ai_provider_client
+from app.core.websocket import broadcaster as broadcaster_module
+from app.core.security import get_current_user
+from app.core.target_schema_registry import get_target_schema, list_target_schemas
+from app.main import app
+from app.models.user import User
+from app.models.datasource_config import DataSourceConfig
+from app.services import custom_datasource_runtime
+from app.services.custom_datasource_runtime import run_mapped_websocket_config
+from app.services.datasource_mapping import execute_mapping, persist_mapped_records, redact_for_llm
+
+
+SAMPLE_AIS = {
+    "data": [
+        {
+            "mmsi": "257123000",
+            "latitude": "59.91",
+            "longitude": "10.75",
+            "speedOverGround": "12.4",
+            "timestamp": "2026-04-28T00:00:00Z",
+            "api_token": "secret-value",
+        }
+    ]
+}
+
+
+def test_registry_exposes_v1_target_schemas():
+    keys = {schema["key"] for schema in list_target_schemas()}
+
+    assert {"vessel_ais", "geo_points", "generic_records"}.issubset(keys)
+    assert get_target_schema("vessel_ais").destination == "vessel_position"
+
+
+def test_mapping_engine_maps_and_validates_vessel_ais():
+    mapping = {
+        "source": {"items_path": "$.data[*]"},
+        "fields": {
+            "mmsi": {"path": "$.mmsi", "type": "integer"},
+            "lat": {"path": "$.latitude", "type": "float"},
+            "lon": {"path": "$.longitude", "type": "float"},
+            "sog": {"path": "$.speedOverGround", "type": "float"},
+            "received_at": {"path": "$.timestamp", "type": "datetime"},
+        },
+    }
+
+    result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
+
+    assert result["mapped_count"] == 1
+    assert result["failed_count"] == 0
+    assert result["records"][0]["mmsi"] == 257123000
+    assert result["records"][0]["lat"] == 59.91
+
+
+def test_mapping_engine_reports_schema_errors():
+    mapping = {
+        "source": {"items_path": "$.data[*]"},
+        "fields": {
+            "mmsi": {"path": "$.mmsi", "type": "integer"},
+            "lat": {"path": "$.missing_lat", "type": "float"},
+            "lon": {"path": "$.longitude", "type": "float"},
+        },
+    }
+
+    result = execute_mapping(SAMPLE_AIS, mapping, "vessel_ais")
+
+    assert result["mapped_count"] == 0
+    assert result["failed_count"] == 1
+    assert any("lat" in error for error in result["errors"][0]["errors"])
+
+
+def test_redact_for_llm_masks_secret_like_fields():
+    redacted = redact_for_llm(SAMPLE_AIS)
+
+    assert redacted["data"][0]["api_token"] == "[REDACTED]"
+
+
+@pytest.mark.asyncio
+async def test_persist_mapped_records_writes_generic_records():
+    class FakeDB:
+        def __init__(self):
+            self.added = []
+            self.committed = False
+
+        def add(self, value):
+            self.added.append(value)
+
+        async def commit(self):
+            self.committed = True
+
+    db = FakeDB()
+
+    count = await persist_mapped_records(
+        db,
+        datasource_name="custom_weather",
+        datasource_config_id=42,
+        target_schema="generic_records",
+        records=[{"source_id": "row-1", "data": {"temp": 25}}],
+        mapping_version=3,
+    )
+
+    assert count == 1
+    assert db.committed is True
+    assert db.added[0].source == "custom_weather"
+    assert db.added[0].data_type == "generic_records"
+    assert db.added[0].extra_data["mapping_version"] == 3
+
+
+@pytest.mark.asyncio
+async def test_persist_mapped_vessel_records_writes_raw_and_broadcasts(monkeypatch):
+    record_observation = AsyncMock(return_value=object())
+    update_health = AsyncMock()
+    broadcast_custom = AsyncMock()
+    monkeypatch.setattr(
+        "app.services.vessel_ais_aggregation.record_vessel_ais_observation",
+        record_observation,
+    )
+    monkeypatch.setattr(
+        "app.services.vessel_ais_aggregation.update_ais_source_health",
+        update_health,
+    )
+    monkeypatch.setattr(broadcaster_module, "broadcast_custom", broadcast_custom)
+
+    class FakeDB:
+        def __init__(self):
+            self.committed = False
+
+        async def commit(self):
+            self.committed = True
+
+    db = FakeDB()
+
+    count = await persist_mapped_records(
+        db,
+        datasource_name="mock_ais_ws",
+        datasource_config_id=42,
+        target_schema="vessel_ais",
+        records=[
+            {
+                "mmsi": 999000001,
+                "lat": 31.2,
+                "lon": 121.4,
+                "name": "MOCK VESSEL 001",
+                "received_at": "2026-05-01T00:00:00Z",
+            }
+        ],
+        mapping_version=1,
+        delivery_mode="realtime_stream",
+        transport="websocket",
+    )
+
+    assert count == 1
+    assert db.committed is True
+    record_observation.assert_awaited_once()
+    assert record_observation.await_args.kwargs["source"] == "mock_ais_ws"
+    assert record_observation.await_args.kwargs["delivery_mode"] == "realtime_stream"
+    assert record_observation.await_args.kwargs["transport"] == "websocket"
+    update_health.assert_awaited_once()
+    broadcast_custom.assert_awaited_once()
+    assert broadcast_custom.await_args.args[0] == "vessels"
+    assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "999000001"
+
+
+@pytest.mark.asyncio
+async def test_custom_websocket_runner_maps_and_persists_vessel_records(monkeypatch):
+    mapping = SimpleNamespace(
+        id=7,
+        version=2,
+        target_schema="vessel_ais",
+        mapping_json={
+            "source": {"items_path": "$"},
+            "fields": {
+                "mmsi": {"path": "$.mmsi", "type": "integer"},
+                "lat": {"path": "$.lat", "type": "float"},
+                "lon": {"path": "$.lon", "type": "float"},
+                "name": {"path": "$.name", "type": "string"},
+                "received_at": {"path": "$.received_at", "type": "datetime"},
+            },
+        },
+    )
+
+    class FakeResult:
+        def scalar_one_or_none(self):
+            return mapping
+
+    class FakeDB:
+        async def execute(self, _stmt):
+            return FakeResult()
+
+    class FakeWebSocket:
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, *_args):
+            return None
+
+        async def send(self, _message):
+            return None
+
+        async def recv(self):
+            return (
+                '{"type":"vessel","data":{"mmsi":"999000001","name":"MOCK VESSEL 001",'
+                '"lat":31.2,"lon":121.4,"received_at":"2026-05-01T00:00:00Z"}}'
+            )
+
+    persist = AsyncMock(return_value=1)
+    monkeypatch.setattr(custom_datasource_runtime, "_connect_websocket", AsyncMock(return_value=FakeWebSocket()))
+    monkeypatch.setattr(custom_datasource_runtime, "persist_mapped_records", persist)
+
+    result = await run_mapped_websocket_config(
+        FakeDB(),
+        DataSourceConfig(
+            id=42,
+            name="mock_ais_ws",
+            source_type="websocket",
+            endpoint="ws://localhost:8787/ais",
+            auth_type="none",
+            headers={},
+            config={"ws_message_path": "$.data", "debug_max_messages": 1},
+        ),
+    )
+
+    assert result["status"] == "success"
+    assert result["messages_seen"] == 1
+    assert result["written_count"] == 1
+    persist.assert_awaited_once()
+    assert persist.await_args.kwargs["datasource_name"] == "mock_ais_ws"
+    assert persist.await_args.kwargs["records"][0]["mmsi"] == 999000001
+    assert persist.await_args.kwargs["delivery_mode"] == "realtime_stream"
+    assert persist.await_args.kwargs["transport"] == "websocket"
+
+
+@pytest.mark.asyncio
+async def test_mapping_preview_api_uses_deterministic_engine():
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    app.dependency_overrides = {get_current_user: override_get_current_user}
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.post(
+                "/api/v1/datasources/mappings/preview",
+                json={
+                    "sample_payload": SAMPLE_AIS,
+                    "target_schema": "vessel_ais",
+                    "mapping_json": {
+                        "source": {"items_path": "$.data[*]"},
+                        "fields": {
+                            "mmsi": {"path": "$.mmsi", "type": "integer"},
+                            "lat": {"path": "$.latitude", "type": "float"},
+                            "lon": {"path": "$.longitude", "type": "float"},
+                        },
+                    },
+                },
+            )
+    finally:
+        app.dependency_overrides.clear()
+
+    assert response.status_code == 200
+    payload = response.json()
+    assert payload["success"] is True
+    assert payload["preview"]["records"][0]["mmsi"] == 257123000
+
+
+@pytest.mark.asyncio
+async def test_mapping_propose_api_redacts_sample_before_ai():
+    seen_context = {}
+
+    class FakeAIClient:
+        async def analyze(self, request, request_id=None):
+            seen_context.update(request.context)
+            return SimpleNamespace(
+                content=(
+                    '{"source":{"items_path":"$.data[*]"},"fields":{'
+                    '"mmsi":{"path":"$.mmsi","type":"integer"},'
+                    '"lat":{"path":"$.latitude","type":"float"},'
+                    '"lon":{"path":"$.longitude","type":"float"}}}'
+                )
+            )
+
+    def override_get_current_user():
+        return User(
+            id=1,
+            username="testuser",
+            email="test@example.com",
+            password_hash="hashed",
+            role="admin",
+            is_active=True,
+        )
+
+    def override_ai_client():
+        return FakeAIClient()
+
+    app.dependency_overrides = {
+        get_current_user: override_get_current_user,
+        get_ai_provider_client: override_ai_client,
+    }
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.post(
+                "/api/v1/datasources/mappings/propose",
+                json={
+                    "sample_payload": SAMPLE_AIS,
+                    "target_schema": "vessel_ais",
+                    "use_ai": True,
+                },
+            )
+    finally:
+        app.dependency_overrides.clear()
+
+    assert response.status_code == 200
+    payload = response.json()
+    assert payload["mapping_json"]["meta"]["generated_by"] == "ai_provider"
+    assert seen_context["sample_payload"]["data"][0]["api_token"] == "[REDACTED]"

backend/tests/test_earth_news.py

@@ -0,0 +1,49 @@
+from datetime import UTC, datetime
+
+from app.services.earth_news import ParsedNewsItem, _serialize_item
+
+
+def test_serialize_item_includes_region_anchor_for_cruise():
+    item = ParsedNewsItem(
+        id="google-apac:test",
+        title="Example APAC story",
+        summary="Example summary",
+        url="https://example.com/story",
+        source="Example Source",
+        feed_name="Global Monitor / APAC",
+        feed_region="asia-pacific",
+        homepage_url="https://example.com",
+        published_at=datetime(2026, 4, 23, 2, 30, tzinfo=UTC),
+    )
+
+    payload = _serialize_item(item, active_region="asia-pacific")
+
+    assert payload["latitude"] == 1.3521
+    assert payload["longitude"] == 103.8198
+    assert payload["location_label"] == "亚太"
+    assert payload["location_inferred"] is True
+    assert payload["is_focus_match"] is True
+    assert payload["published_at"] == "2026-04-23T02:30:00Z"
+
+
+def test_serialize_item_falls_back_to_global_anchor():
+    item = ParsedNewsItem(
+        id="custom:test",
+        title="Fallback story",
+        summary="Fallback summary",
+        url="https://example.com/fallback",
+        source="Fallback Source",
+        feed_name="Fallback Feed",
+        feed_region="unknown-region",
+        homepage_url="https://example.com",
+        published_at=None,
+    )
+
+    payload = _serialize_item(item, active_region="americas")
+
+    assert payload["latitude"] == 20.0
+    assert payload["longitude"] == 0.0
+    assert payload["location_label"] == "全球"
+    assert payload["location_inferred"] is True
+    assert payload["is_focus_match"] is False
+    assert payload["published_at"] is None

backend/tests/test_logging.py

@@ -0,0 +1,78 @@
+from __future__ import annotations
+
+import logging
+
+from io import StringIO
+
+from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
+from app.core.request_context import set_request_id
+
+
+def _capture_output(callback):
+    stream = StringIO()
+    handler = logging.StreamHandler(stream)
+    handler.setFormatter(PlanetFormatter(datefmt="%Y-%m-%d %H:%M:%S"))
+    handler.addFilter(PlanetContextFilter())
+
+    adapter = get_logger("tests.logging")
+    target_logger = adapter.logger
+    original_handlers = list(target_logger.handlers)
+    original_level = target_logger.level
+    original_propagate = target_logger.propagate
+
+    target_logger.handlers = [handler]
+    target_logger.setLevel(logging.INFO)
+    target_logger.propagate = False
+
+    try:
+        callback(adapter)
+    finally:
+        handler.flush()
+        target_logger.handlers = original_handlers
+        target_logger.setLevel(original_level)
+        target_logger.propagate = original_propagate
+
+    return stream.getvalue()
+
+
+def test_structured_logger_injects_request_id_and_event():
+    set_request_id("req-test-123")
+    try:
+        output = _capture_output(
+            lambda logger: logger.info_event(
+                "collector started",
+                event="collector.run.started",
+                context={"collector_name": "bgp_news"},
+            )
+        )
+    finally:
+        set_request_id(None)
+
+    assert "request_id=req-test-123" in output
+    assert "event=collector.run.started" in output
+    assert "service=backend" in output
+    assert '"collector_name": "bgp_news"' in output
+
+
+def test_structured_logger_redacts_sensitive_text_and_context():
+    set_request_id("req-test-redact")
+    try:
+        output = _capture_output(
+            lambda logger: logger.error_event(
+                "Authorization: Bearer super-secret-token",
+                event="auth.token.failed",
+                context={
+                    "token": "plain-secret",
+                    "nested": {"password": "hunter2"},
+                    "safe": "visible",
+                },
+            )
+        )
+    finally:
+        set_request_id(None)
+
+    assert "super-secret-token" not in output
+    assert "plain-secret" not in output
+    assert "hunter2" not in output
+    assert "[REDACTED]" in output
+    assert '"safe": "visible"' in output

backend/tests/test_models.py

@@ -121,6 +121,25 @@ class TestCollectionTaskModel:
         )
         assert task.records_processed == 100
 
+    def test_task_with_phase_progress(self):
+        """Test collection task phase-level progress fields"""
+        task = CollectionTask(
+            datasource_id=1,
+            status="running",
+            phase="fetching",
+            phase_progress=42.5,
+            phase_message="Downloading dataset",
+            phase_current=1024,
+            phase_total=4096,
+            phase_unit="bytes",
+        )
+        assert task.phase == "fetching"
+        assert task.phase_progress == 42.5
+        assert task.phase_message == "Downloading dataset"
+        assert task.phase_current == 1024
+        assert task.phase_total == 4096
+        assert task.phase_unit == "bytes"
+
     def test_task_error_message(self):
         """Test collection task with error message"""
         task = CollectionTask(

backend/tests/test_system_logs.py

@@ -0,0 +1,217 @@
+from __future__ import annotations
+
+import json
+
+from pathlib import Path
+
+from app.services import system_logs
+
+
+class FakeRedis:
+    def __init__(self) -> None:
+        self.store: dict[str, list[str]] = {}
+
+    def rpush(self, key: str, value: str) -> None:
+        self.store.setdefault(key, []).append(value)
+
+    def ltrim(self, key: str, start: int, end: int) -> None:
+        items = self.store.get(key, [])
+        normalized_end = None if end == -1 else end + 1
+        self.store[key] = items[start:normalized_end]
+
+    def expire(self, key: str, seconds: int) -> None:
+        return None
+
+    def lrange(self, key: str, start: int, end: int) -> list[str]:
+        items = self.store.get(key, [])
+        normalized_end = None if end == -1 else end + 1
+        return items[start:normalized_end]
+
+    def llen(self, key: str) -> int:
+        return len(self.store.get(key, []))
+
+
+def test_read_log_snapshot_uses_structured_buffer_timestamp_level_and_search(monkeypatch):
+    fake_redis = FakeRedis()
+    monkeypatch.setattr(system_logs, "redis_client", fake_redis)
+    monkeypatch.setattr(
+        system_logs,
+        "LOG_SOURCES",
+        {
+            "earth-client": system_logs.LogSource(
+                source_id="earth-client",
+                name="Earth 浏览器端",
+                kind="buffer",
+                location="redis://planet:system_logs:earth-client",
+                description="Earth 浏览器端上报日志",
+                category="client",
+                buffer_key=system_logs.get_buffer_log_key("earth-client"),
+            )
+        },
+    )
+
+    fake_redis.rpush(
+        system_logs.get_buffer_log_key("earth-client"),
+        json.dumps(
+            {
+                "timestamp": "2026-04-22T10:15:30Z",
+                "level": "warning",
+                "message": "news feed degraded",
+                "context": {"module": "news", "detail": "timeout"},
+            },
+            ensure_ascii=False,
+        ),
+    )
+    fake_redis.rpush(
+        system_logs.get_buffer_log_key("earth-client"),
+        json.dumps(
+            {
+                "timestamp": "2026-04-23T06:01:00Z",
+                "level": "error",
+                "message": "landing points failed",
+                "context": {"module": "layer-startup", "detail": "http 500"},
+            },
+            ensure_ascii=False,
+        ),
+    )
+
+    snapshot = system_logs.read_log_snapshot(
+        "earth-client",
+        50,
+        levels="error,warning",
+        start_date="2026-04-23",
+        end_date="2026-04-23",
+        search="landing",
+    )
+
+    assert snapshot is not None
+    assert snapshot["selected_levels"] == ["error", "warning"]
+    assert snapshot["search_query"] == "landing"
+    assert snapshot["line_count"] == 1
+    assert snapshot["lines"][0].startswith("2026-04-23 06:01:00 ERROR landing points failed")
+    assert snapshot["daily_markers"] == [
+        {"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
+    ]
+
+
+def test_read_log_snapshot_parses_file_timestamp_and_builds_markers(tmp_path: Path, monkeypatch):
+    log_path = tmp_path / "backend.log"
+    log_path.write_text(
+        "\n".join(
+            [
+                "2026-04-22 08:00:00 INFO service booted",
+                "2026-04-23 09:15:00 WARNING disk pressure detected",
+                "2026-04-23 09:16:00 ERROR sync failed",
+                "2026-04-24 10:00:00 DEBUG collector trace",
+            ]
+        ),
+        encoding="utf-8",
+    )
+
+    monkeypatch.setattr(
+        system_logs,
+        "LOG_SOURCES",
+        {
+            "backend": system_logs.LogSource(
+                source_id="backend",
+                name="后端服务",
+                kind="file",
+                location=str(log_path),
+                description="测试文件日志",
+                category="service",
+            )
+        },
+    )
+
+    snapshot = system_logs.read_log_snapshot(
+        "backend",
+        50,
+        levels="warning,error",
+        search="failed",
+    )
+
+    assert snapshot is not None
+    assert snapshot["line_count"] == 1
+    assert snapshot["lines"] == ["2026-04-23 09:16:00 ERROR sync failed"]
+    assert snapshot["daily_markers"] == [
+        {"date_token": "2026-04-23", "total": 1, "dominant_level": "error"}
+    ]
+    assert snapshot["status"] == "ok"
+
+
+def test_append_buffer_log_persists_normalized_level(monkeypatch):
+    fake_redis = FakeRedis()
+    monkeypatch.setattr(system_logs, "redis_client", fake_redis)
+
+    system_logs.append_buffer_log(
+        "earth-client",
+        level="warn",
+        message="feed delayed",
+        context={"module": "news"},
+    )
+
+    stored_items = fake_redis.lrange(system_logs.get_buffer_log_key("earth-client"), 0, -1)
+    payload = json.loads(stored_items[0])
+    assert payload["level"] == "warning"
+    assert payload["message"] == "feed delayed"
+
+
+def test_infer_log_level_prefers_leading_prefix_over_query_string():
+    line = 'INFO: 127.0.0.1 - "GET /api/v1/system/logs/backend?limit=200&level=error&levels=error HTTP/1.1" 200 OK'
+
+    entry = system_logs.parse_text_log_entry(line)
+
+    assert entry.level == "info"
+
+
+def test_parse_text_log_entry_does_not_promote_exception_context_to_error():
+    line = "websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout"
+
+    entry = system_logs.parse_text_log_entry(line)
+
+    assert entry.level is None
+
+
+def test_parse_text_log_entry_still_detects_explicit_error_prefix():
+    line = "ERROR: [Errno 98] Address already in use"
+
+    entry = system_logs.parse_text_log_entry(line)
+
+    assert entry.level == "error"
+
+
+def test_read_log_snapshot_strips_nul_bytes_from_file_lines(tmp_path: Path, monkeypatch):
+    log_path = tmp_path / "backend.log"
+    log_path.write_bytes(
+        (
+            b"INFO: service booted\n"
+            b"ERROR: bind failed\n"
+            + b"\x00" * 32
+            + b"2026-04-23 23:41:32 INFO service=backend message=request served\n"
+        )
+    )
+
+    monkeypatch.setattr(
+        system_logs,
+        "LOG_SOURCES",
+        {
+            "backend": system_logs.LogSource(
+                source_id="backend",
+                name="后端服务",
+                kind="file",
+                location=str(log_path),
+                description="测试文件日志",
+                category="service",
+            )
+        },
+    )
+
+    snapshot = system_logs.read_log_snapshot("backend", 50)
+
+    assert snapshot is not None
+    assert snapshot["line_count"] == 3
+    assert snapshot["lines"] == [
+        "INFO: service booted",
+        "ERROR: bind failed",
+        "2026-04-23 23:41:32 INFO service=backend message=request served",
+    ]

backend/tests/test_vessel_aggregation_strategy.py

@@ -0,0 +1,161 @@
+"""Tests for the v4 vessel_ais aggregation strategy."""
+
+from datetime import datetime, timedelta, timezone
+from unittest.mock import AsyncMock
+
+import pytest
+
+from app.models.vessel import AISRawObservation
+from app.services.vessel_aggregation_strategy import (
+    DEFAULT_STRATEGY,
+    StrategyValidationError,
+    validate_strategy,
+)
+from app.services.vessel_ais_aggregation import aggregate_vessel_observations
+
+
+def _obs(*, source: str, mmsi: int, observed_at: datetime, **payload) -> AISRawObservation:
+    payload = {"mmsi": mmsi, "lat": 50.0, "lon": 10.0, **payload}
+    delivery_mode = "realtime_stream" if source == "aisstream_vessels" else "polling"
+    transport = "websocket" if source == "aisstream_vessels" else "http"
+    return AISRawObservation(
+        target_schema="vessel_ais",
+        source=source,
+        entity_key=str(mmsi),
+        delivery_mode=delivery_mode,
+        transport=transport,
+        message_type="PositionReport",
+        observation_hash=f"{source}:{mmsi}:{observed_at.isoformat()}",
+        observed_at=observed_at,
+        collected_at=observed_at,
+        normalized_payload=payload,
+        raw_payload=payload,
+        quality_flags=[],
+    )
+
+
+def test_validate_rejects_unknown_field():
+    with pytest.raises(StrategyValidationError, match="unknown vessel_ais field"):
+        validate_strategy({"vessel_ais": {"field_rules": {"definitely_not_a_field": {"mode": "newest"}}}})
+
+
+def test_validate_rejects_dynamic_lock_without_flag():
+    with pytest.raises(StrategyValidationError, match="allow_dynamic_lock"):
+        validate_strategy(
+            {
+                "vessel_ais": {
+                    "field_rules": {"lat": {"mode": "source_priority"}},
+                    "allow_dynamic_lock": False,
+                }
+            }
+        )
+
+
+def test_validate_allows_dynamic_lock_with_flag():
+    normalized = validate_strategy(
+        {
+            "version": 0,
+            "vessel_ais": {
+                "field_rules": {"lat": {"mode": "source_priority", "source_priority": ["barentswatch_vessels"]}},
+                "allow_dynamic_lock": True,
+            },
+        }
+    )
+    assert normalized["vessel_ais"]["field_rules"]["lat"]["mode"] == "source_priority"
+    assert normalized["version"] == 1
+
+
+def test_validate_increments_version():
+    first = validate_strategy({"version": 5, "vessel_ais": {}})
+    assert first["version"] == 6
+
+
+@pytest.mark.asyncio
+async def test_strategy_field_rule_promotes_specific_source(monkeypatch):
+    now = datetime(2026, 5, 4, 12, 0, tzinfo=timezone.utc)
+
+    obs_a = _obs(
+        source="aisstream_vessels",
+        mmsi=257123000,
+        observed_at=now,
+        name="AISSTREAM ONE",
+        vessel_type_name="Cargo",
+    )
+    obs_b = _obs(
+        source="barentswatch_vessels",
+        mmsi=257123000,
+        observed_at=now - timedelta(seconds=1),
+        name="BARENTSWATCH ONE",
+        vessel_type_name="Cargo",
+    )
+
+    strategy = {
+        "version": 7,
+        "vessel_ais": {
+            "source_priority": [],
+            "field_rules": {
+                "name": {"mode": "source_priority", "source_priority": ["barentswatch_vessels", "aisstream_vessels"]},
+            },
+            "freshness": {"realtime_stream_seconds": 0, "polling_seconds": 0},
+            "allow_dynamic_lock": False,
+        },
+    }
+
+    db = AsyncMock()
+    vessels = await aggregate_vessel_observations(
+        db,
+        [obs_a, obs_b],
+        write_conflicts=False,
+        strategy=strategy,
+    )
+    assert len(vessels) == 1
+    vessel = vessels[0]
+    assert vessel["name"] == "BARENTSWATCH ONE"
+    assert vessel["field_sources"]["name"] == "barentswatch_vessels"
+    assert vessel["selected_reasons"]["name"] == "source_priority"
+    assert vessel["aggregation_strategy_version"] == 7
+
+
+@pytest.mark.asyncio
+async def test_strategy_freshness_falls_back_to_polling_when_realtime_stale():
+    now = datetime(2026, 5, 4, 12, 0, tzinfo=timezone.utc)
+
+    stale_realtime = _obs(
+        source="aisstream_vessels",
+        mmsi=257123000,
+        observed_at=now - timedelta(hours=1),
+        lat=58.0,
+        lon=10.0,
+    )
+    fresh_polling = _obs(
+        source="barentswatch_vessels",
+        mmsi=257123000,
+        observed_at=now - timedelta(seconds=30),
+        lat=60.0,
+        lon=11.0,
+    )
+
+    strategy = {
+        "version": 1,
+        "vessel_ais": {
+            "source_priority": ["aisstream_vessels", "barentswatch_vessels"],
+            "field_rules": {},
+            "freshness": {"realtime_stream_seconds": 900, "polling_seconds": 7200},
+            "allow_dynamic_lock": False,
+        },
+    }
+
+    db = AsyncMock()
+    vessels = await aggregate_vessel_observations(
+        db,
+        [stale_realtime, fresh_polling],
+        write_conflicts=False,
+        strategy=strategy,
+    )
+    assert vessels[0]["field_sources"]["lat"] == "barentswatch_vessels"
+    assert vessels[0]["lat"] == 60.0
+
+
+def test_default_strategy_is_stable():
+    assert DEFAULT_STRATEGY["vessel_ais"]["allow_dynamic_lock"] is False
+    assert "freshness" in DEFAULT_STRATEGY["vessel_ais"]

backend/tests/test_vessel_enrichment.py

@@ -0,0 +1,155 @@
+"""Tests for v5 enrichment + conflict promote-to-rule."""
+
+from datetime import datetime, timedelta, timezone
+from unittest.mock import AsyncMock
+
+import pytest
+
+from app.models.vessel import AISConflictRecord, AISRawObservation
+from app.models.vessel_enrichment import VesselMediaEnrichment, VesselProfileEnrichment
+from app.services.vessel_ais_aggregation import aggregate_vessel_observations
+from app.services.vessel_enrichment import (
+    _apply_upsert,
+    get_vessel_enrichment_bundle,
+)
+
+
+class _StoreSession:
+    """Minimal AsyncSession stand-in that tracks mmsi-keyed enrichment + a strategy."""
+
+    def __init__(self, *, profile=None, media=None, conflicts=None):
+        self.profile = profile
+        self.media = media
+        self.conflicts = list(conflicts or [])
+        self.added: list = []
+        self.committed = False
+
+    async def get(self, model, key):
+        if model is VesselProfileEnrichment:
+            return self.profile if self.profile and self.profile.mmsi == key else None
+        if model is VesselMediaEnrichment:
+            return self.media if self.media and self.media.mmsi == key else None
+        return None
+
+
+@pytest.mark.asyncio
+async def test_enrichment_bundle_filters_expired_records():
+    now = datetime.now(timezone.utc)
+    fresh = VesselProfileEnrichment(
+        mmsi=257123000,
+        source="local_cache",
+        payload={"vessel_subtype": "Container"},
+        fetched_at=now - timedelta(hours=1),
+        expires_at=now + timedelta(days=7),
+        confidence=0.9,
+    )
+    expired_media = VesselMediaEnrichment(
+        mmsi=257123000,
+        source="vesselfinder",
+        payload={"images": ["https://example.com/a.jpg"]},
+        fetched_at=now - timedelta(days=30),
+        expires_at=now - timedelta(days=1),
+    )
+    db = _StoreSession(profile=fresh, media=expired_media)
+
+    bundle = await get_vessel_enrichment_bundle(db, 257123000)
+
+    assert bundle["profile"]["payload"]["vessel_subtype"] == "Container"
+    assert bundle["media"] is None
+
+
+def test_apply_upsert_preserves_payload_and_metadata():
+    record = VesselProfileEnrichment(mmsi=257123000)
+    out = _apply_upsert(
+        record,
+        {
+            "source": "vesselfinder",
+            "payload": {"vessel_subtype": "Container", "operator": "Maersk"},
+            "expires_at": "2026-12-31T00:00:00Z",
+            "confidence": 0.85,
+            "reference_url": "https://www.vesselfinder.com/vessels/257123000",
+        },
+    )
+    assert out["payload"]["operator"] == "Maersk"
+    assert out["confidence"] == 0.85
+    assert record.reference_url == "https://www.vesselfinder.com/vessels/257123000"
+    assert record.expires_at is not None
+    assert record.expires_at.year == 2026
+
+
+def _obs(*, source: str, mmsi: int, observed_at, **payload) -> AISRawObservation:
+    payload = {"mmsi": mmsi, "lat": 60.0, "lon": 5.0, **payload}
+    delivery_mode = "realtime_stream" if source == "aisstream_vessels" else "polling"
+    transport = "websocket" if source == "aisstream_vessels" else "http"
+    return AISRawObservation(
+        target_schema="vessel_ais",
+        source=source,
+        entity_key=str(mmsi),
+        delivery_mode=delivery_mode,
+        transport=transport,
+        message_type="PositionReport",
+        observation_hash=f"{source}:{mmsi}:{observed_at.isoformat()}",
+        observed_at=observed_at,
+        collected_at=observed_at,
+        normalized_payload=payload,
+        raw_payload=payload,
+        quality_flags=[],
+    )
+
+
+@pytest.mark.asyncio
+async def test_promoted_rule_wins_during_aggregation():
+    """Simulate the strategy that conflict-promote-to-rule writes."""
+    now = datetime.now(timezone.utc)
+    obs_a = _obs(
+        source="aisstream_vessels",
+        mmsi=257111000,
+        observed_at=now,
+        name="STREAM NAME",
+        vessel_type_name="Cargo",
+    )
+    obs_b = _obs(
+        source="barentswatch_vessels",
+        mmsi=257111000,
+        observed_at=now - timedelta(seconds=1),
+        name="REST NAME",
+        vessel_type_name="Cargo",
+    )
+    promoted_strategy = {
+        "version": 99,
+        "vessel_ais": {
+            "source_priority": [],
+            "field_rules": {
+                "name": {"mode": "source_priority", "source_priority": ["barentswatch_vessels"]}
+            },
+            "freshness": {"realtime_stream_seconds": 0, "polling_seconds": 0},
+            "allow_dynamic_lock": False,
+        },
+    }
+
+    db = AsyncMock()
+    vessels = await aggregate_vessel_observations(
+        db,
+        [obs_a, obs_b],
+        write_conflicts=False,
+        strategy=promoted_strategy,
+    )
+    assert vessels[0]["name"] == "REST NAME"
+    assert vessels[0]["selected_reasons"]["name"] == "source_priority"
+    assert vessels[0]["aggregation_strategy_version"] == 99
+
+
+def test_conflict_record_holds_selected_source():
+    """Sanity: the promote-to-rule API reads selected_source from this column."""
+    record = AISConflictRecord(
+        target_schema="vessel_ais",
+        entity_key="257111000",
+        field="name",
+        candidates={"a": "X", "b": "Y"},
+        selected_source="barentswatch_vessels",
+        selected_value="Y",
+        selected_reason="delivery_mode_priority",
+    )
+    serialized = record.to_dict()
+    assert serialized["selected_source"] == "barentswatch_vessels"
+    assert serialized["field"] == "name"

backend/tests/test_vessels.py

@@ -0,0 +1,675 @@
+from datetime import datetime, timedelta, timezone
+from unittest.mock import AsyncMock
+
+import pytest
+from httpx import ASGITransport, AsyncClient
+
+from app.api.v1 import visualization
+from app.api.v1.visualization import convert_vessels_to_geojson
+from app.db.session import get_db
+from app.main import app
+from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic
+from app.services import barentswatch
+from app.services.collectors.aisstream import AISStreamCollector
+from app.services.collectors.vessel_ais import VesselAISCollector
+from app.services.vessel_ais_aggregation import (
+    aggregate_vessel_observations,
+    build_field_conflict_candidates,
+    build_observation_hash,
+    record_vessel_ais_observation,
+)
+
+
+def test_vessel_collector_transforms_barentswatch_like_records():
+    collector = VesselAISCollector()
+    records = collector.transform(
+        [
+            {
+                "mmsi": "257123000",
+                "lat": "59.91",
+                "lon": "10.73",
+                "sog": 12.4,
+                "cog": 214,
+                "nav_status": 0,
+                "shipType": 70,
+                "name": "OSLO TRADER",
+            },
+            {"mmsi": "bad", "lat": 120, "lon": 10},
+        ]
+    )
+
+    assert len(records) == 1
+    assert records[0]["mmsi"] == 257123000
+    assert records[0]["vessel_type_name"] == "Cargo"
+    assert records[0]["lat"] == pytest.approx(59.91)
+
+
+def test_vessel_observation_hash_is_stable_for_same_payload():
+    observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
+    payload = {
+        "mmsi": 257123000,
+        "lat": 59.91,
+        "lon": 10.73,
+        "received_at": observed_at,
+    }
+
+    first = build_observation_hash(
+        source="barentswatch_vessels",
+        entity_key="257123000",
+        message_type="PositionReport",
+        observed_at=observed_at,
+        normalized_payload=payload,
+    )
+    second = build_observation_hash(
+        source="barentswatch_vessels",
+        entity_key="257123000",
+        message_type="PositionReport",
+        observed_at=observed_at,
+        normalized_payload=dict(reversed(payload.items())),
+    )
+
+    assert first == second
+    assert len(first) == 64
+
+
+@pytest.mark.asyncio
+async def test_record_vessel_ais_observation_skips_existing_hash():
+    observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
+
+    class _Result:
+        def scalar_one_or_none(self):
+            return 123
+
+    class _Session:
+        def __init__(self):
+            self.added = []
+
+        async def execute(self, _stmt):
+            return _Result()
+
+        def add(self, item):
+            self.added.append(item)
+
+    db = _Session()
+    observation = await record_vessel_ais_observation(
+        db,
+        source="barentswatch_vessels",
+        normalized_payload={
+            "mmsi": 257123000,
+            "lat": 59.91,
+            "lon": 10.73,
+            "received_at": observed_at,
+        },
+        delivery_mode="polling",
+        transport="http",
+        observed_at=observed_at.isoformat(),
+    )
+
+    assert observation is None
+    assert db.added == []
+
+
+def test_build_field_conflict_candidates_from_raw_observations():
+    observations = [
+        AISRawObservation(
+            source="barentswatch_vessels",
+            normalized_payload={"name": "OSLO TRADER", "flag": "NO"},
+        ),
+        AISRawObservation(
+            source="aisstream_vessels",
+            normalized_payload={"name": "OSLO TRADER II", "flag": "NO"},
+        ),
+    ]
+
+    conflicts = build_field_conflict_candidates(observations)
+
+    assert conflicts == [
+        {
+            "field": "name",
+            "candidates": {
+                "aisstream_vessels": "OSLO TRADER II",
+                "barentswatch_vessels": "OSLO TRADER",
+            },
+            "status": "candidate",
+        }
+    ]
+
+
+@pytest.mark.asyncio
+async def test_aggregate_vessel_observations_prefers_realtime_and_records_conflict():
+    observed_at = datetime.now(timezone.utc) - timedelta(minutes=5)
+
+    class _Result:
+        def scalar_one_or_none(self):
+            return None
+
+    class _Session:
+        def __init__(self):
+            self.added = []
+
+        async def execute(self, _stmt):
+            return _Result()
+
+        def add(self, item):
+            self.added.append(item)
+
+    db = _Session()
+    observations = [
+        AISRawObservation(
+            id=1,
+            source="barentswatch_vessels",
+            entity_key="257123000",
+            delivery_mode="polling",
+            transport="http",
+            observed_at=observed_at,
+            collected_at=observed_at,
+            normalized_payload={
+                "mmsi": 257123000,
+                "name": "OSLO TRADER",
+                "lat": 59.91,
+                "lon": 10.73,
+            },
+        ),
+        AISRawObservation(
+            id=2,
+            source="aisstream_vessels",
+            entity_key="257123000",
+            delivery_mode="realtime_stream",
+            transport="websocket",
+            observed_at=observed_at + timedelta(seconds=10),
+            collected_at=observed_at + timedelta(seconds=10),
+            normalized_payload={
+                "mmsi": 257123000,
+                "vessel_type": 79,
+                "lat": 59.92,
+                "lon": 10.74,
+            },
+            raw_payload={"MetaData": {"ShipName": "OSLO TRADER II         "}},
+        ),
+    ]
+
+    vessels = await aggregate_vessel_observations(db, observations)
+
+    assert vessels[0]["lat"] == pytest.approx(59.92)
+    assert vessels[0]["field_sources"]["lat"] == "aisstream_vessels"
+    assert vessels[0]["name"] == "OSLO TRADER II"
+    assert vessels[0]["vessel_type_name"] == "Cargo"
+    assert vessels[0]["source_summary"]["aisstream_vessels"]["observation_count"] == 1
+    assert vessels[0]["source_summary"]["barentswatch_vessels"]["delivery_mode"] == "polling"
+    assert vessels[0]["conflict_count"] == 0
+    assert db.added == []
+
+
+@pytest.mark.asyncio
+async def test_vessel_collector_writes_raw_observations_only(monkeypatch):
+    collector = VesselAISCollector()
+    collector.update_progress = AsyncMock()
+    record_observation = AsyncMock()
+    update_health = AsyncMock()
+    broadcast_custom = AsyncMock()
+    monkeypatch.setattr(
+        "app.services.collectors.vessel_ais.record_vessel_ais_observation",
+        record_observation,
+    )
+    monkeypatch.setattr(
+        "app.services.collectors.vessel_ais.update_ais_source_health",
+        update_health,
+    )
+    monkeypatch.setattr(
+        "app.services.collectors.vessel_ais.broadcaster.broadcast_custom",
+        broadcast_custom,
+    )
+
+    class _Session:
+        def __init__(self):
+            self.added = []
+            self.committed = False
+
+        async def get(self, *_args):
+            return None
+
+        def add(self, item):
+            self.added.append(item)
+
+        async def execute(self, _stmt):
+            return None
+
+        async def commit(self):
+            self.committed = True
+
+    db = _Session()
+    observed_at = datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc)
+
+    saved = await collector._save_data(
+        db,
+        [
+            {
+                "mmsi": 257123000,
+                "name": "OSLO TRADER",
+                "lat": 59.91,
+                "lon": 10.73,
+                "received_at": observed_at,
+            }
+        ],
+    )
+
+    assert saved == 1
+    assert db.committed is True
+    # BarentsWatch must funnel through the unified AIS pipeline only — no legacy writes.
+    assert not any(isinstance(item, VesselStatic) for item in db.added)
+    assert not any(isinstance(item, VesselPosition) for item in db.added)
+    record_observation.assert_awaited_once()
+    assert record_observation.await_args.kwargs["source"] == "barentswatch_vessels"
+    assert record_observation.await_args.kwargs["normalized_payload"]["mmsi"] == 257123000
+    update_health.assert_awaited_once()
+    broadcast_custom.assert_awaited_once()
+    assert broadcast_custom.await_args.args[0] == "vessels"
+    assert broadcast_custom.await_args.args[1]["action"] == "upsert"
+    assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "257123000"
+
+
+def test_aisstream_collector_normalizes_position_report():
+    collector = AISStreamCollector()
+
+    records = collector.transform(
+        [
+            {
+                "MessageType": "PositionReport",
+                "MetaData": {
+                    "MMSI": 257123000,
+                    "ShipName": "OSLO TRADER         ",
+                    "time_utc": "2026-04-30T12:00:00Z",
+                },
+                "Message": {
+                    "PositionReport": {
+                        "Latitude": 59.91,
+                        "Longitude": 10.73,
+                        "Sog": 12.4,
+                        "Cog": 214,
+                        "TrueHeading": 215,
+                        "NavigationalStatus": 0,
+                    }
+                },
+            }
+        ]
+    )
+
+    assert len(records) == 1
+    assert records[0]["mmsi"] == 257123000
+    assert records[0]["lat"] == pytest.approx(59.91)
+    assert records[0]["name"] == "OSLO TRADER"
+    assert records[0]["_message_type"] == "PositionReport"
+
+
+def test_aisstream_collector_maps_ship_static_type_name():
+    collector = AISStreamCollector()
+
+    records = collector.transform(
+        [
+            {
+                "MessageType": "ShipStaticData",
+                "MetaData": {
+                    "MMSI": 257123000,
+                    "time_utc": "2026-04-30T12:00:00Z",
+                },
+                "Message": {
+                    "ShipStaticData": {
+                        "Name": "OSLO TRADER",
+                        "Type": 79,
+                        "CallSign": "LAAB",
+                    }
+                },
+            }
+        ]
+    )
+
+    assert len(records) == 1
+    assert records[0]["vessel_type"] == 79
+    assert records[0]["vessel_type_name"] == "Cargo"
+
+
+@pytest.mark.asyncio
+async def test_aisstream_collector_writes_only_raw_observations(monkeypatch):
+    collector = AISStreamCollector()
+    collector.update_progress = AsyncMock()
+    record_observation = AsyncMock(return_value=object())
+    update_health = AsyncMock()
+    monkeypatch.setattr(
+        "app.services.collectors.aisstream.record_vessel_ais_observation",
+        record_observation,
+    )
+    monkeypatch.setattr(
+        "app.services.collectors.aisstream.update_ais_source_health",
+        update_health,
+    )
+
+    class _Session:
+        def __init__(self):
+            self.added = []
+            self.committed = False
+
+        def add(self, item):
+            self.added.append(item)
+
+        async def commit(self):
+            self.committed = True
+
+    db = _Session()
+    saved = await collector._save_data(
+        db,
+        [
+            {
+                "mmsi": 257123000,
+                "lat": 59.91,
+                "lon": 10.73,
+                "received_at": datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
+                "_message_type": "PositionReport",
+            }
+        ],
+    )
+
+    assert saved == 1
+    assert db.added == []
+    assert db.committed is True
+    record_observation.assert_awaited_once()
+    assert record_observation.await_args.kwargs["source"] == "aisstream_vessels"
+    update_health.assert_awaited_once()
+
+
+@pytest.mark.asyncio
+async def test_aisstream_stream_record_broadcasts_vessel_delta(monkeypatch):
+    collector = AISStreamCollector()
+    record_observation = AsyncMock(return_value=object())
+    update_health = AsyncMock()
+    broadcast_custom = AsyncMock()
+    monkeypatch.setattr(
+        "app.services.collectors.aisstream.record_vessel_ais_observation",
+        record_observation,
+    )
+    monkeypatch.setattr(
+        "app.services.collectors.aisstream.update_ais_source_health",
+        update_health,
+    )
+    monkeypatch.setattr(
+        "app.services.collectors.aisstream.broadcaster.broadcast_custom",
+        broadcast_custom,
+    )
+
+    class _Session:
+        async def commit(self):
+            pass
+
+    created = await collector._save_stream_record(
+        _Session(),
+        {
+            "mmsi": 257123000,
+            "lat": 59.91,
+            "lon": 10.73,
+            "cog": 214,
+            "received_at": datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
+        },
+    )
+
+    assert created is True
+    record_observation.assert_awaited_once()
+    broadcast_custom.assert_awaited_once()
+    assert broadcast_custom.await_args.args[0] == "vessels"
+    assert broadcast_custom.await_args.args[1]["action"] == "upsert"
+    assert broadcast_custom.await_args.args[1]["vessels"][0]["mmsi_display"] == "257123000"
+
+
+def test_barentswatch_reads_credentials_from_zshrc(tmp_path):
+    zshrc = tmp_path / ".zshrc"
+    zshrc.write_text(
+        "\n".join(
+            [
+                "export BARENTSWATCH_CLIENT_ID='client-from-zshrc'",
+                'export BARENTSWATCH_CLIENT_SECRET="secret-from-zshrc" # local dev credential',
+            ]
+        ),
+        encoding="utf-8",
+    )
+
+    values = barentswatch._read_zshrc_env(zshrc)
+
+    assert values["BARENTSWATCH_CLIENT_ID"] == "client-from-zshrc"
+    assert values["BARENTSWATCH_CLIENT_SECRET"] == "secret-from-zshrc"
+
+
+@pytest.mark.asyncio
+async def test_barentswatch_resolves_config_from_zshrc(tmp_path, monkeypatch):
+    zshrc = tmp_path / ".zshrc"
+    zshrc.write_text(
+        "\n".join(
+            [
+                "export BARENTSWATCH_CLIENT_ID=client-from-zshrc",
+                "export BARENTSWATCH_CLIENT_SECRET=secret-from-zshrc",
+            ]
+        ),
+        encoding="utf-8",
+    )
+    monkeypatch.delenv("BARENTSWATCH_CLIENT_ID", raising=False)
+    monkeypatch.delenv("BARENTSWATCH_CLIENT_SECRET", raising=False)
+    monkeypatch.delenv("BARRENTSWATCH_CLIENT_ID", raising=False)
+    monkeypatch.delenv("BARRENTSWATCH_CLIENT_SECRET", raising=False)
+    monkeypatch.setattr(barentswatch.Path, "home", lambda: tmp_path)
+
+    config = await barentswatch.resolve_barentswatch_config(None)
+
+    assert config.client_id == "client-from-zshrc"
+    assert config.client_secret == "secret-from-zshrc"
+    assert config.credential_source == "~/.zshrc"
+
+
+def test_convert_vessels_to_geojson():
+    position = VesselPosition(
+        mmsi=257123000,
+        lat=59.91,
+        lon=10.73,
+        sog=12.4,
+        cog=214,
+        heading=215,
+        nav_status=0,
+        received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
+    )
+    static = VesselStatic(
+        mmsi=257123000,
+        name="OSLO TRADER",
+        vessel_type=70,
+        vessel_type_name="Cargo",
+        flag="NO",
+        length=185,
+    )
+
+    payload = convert_vessels_to_geojson([(position, static)])
+
+    assert payload["type"] == "FeatureCollection"
+    assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
+    assert payload["features"][0]["properties"]["mmsi"] == 257123000
+    assert payload["features"][0]["properties"]["vessel_type_name"] == "Cargo"
+
+
+def test_convert_vessels_to_geojson_dedupes_mmsi_rows():
+    first = VesselPosition(
+        mmsi=257123000,
+        lat=59.91,
+        lon=10.73,
+        received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
+    )
+    duplicate = VesselPosition(
+        mmsi=257123000,
+        lat=60.01,
+        lon=10.83,
+        received_at=datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc),
+    )
+    other = VesselPosition(
+        mmsi=257456000,
+        lat=60.3,
+        lon=5.3,
+        received_at=datetime(2026, 4, 28, 0, 59, tzinfo=timezone.utc),
+    )
+
+    payload = convert_vessels_to_geojson(
+        [
+            (first, VesselStatic(mmsi=257123000, name="OSLO TRADER")),
+            (duplicate, VesselStatic(mmsi=257123000, name="OSLO TRADER DUP")),
+            (other, VesselStatic(mmsi=257456000, name="BERGEN FERRY")),
+        ]
+    )
+
+    mmsis = [feature["properties"]["mmsi"] for feature in payload["features"]]
+    assert mmsis == [257123000, 257456000]
+    assert payload["features"][0]["geometry"]["coordinates"] == [10.73, 59.91]
+
+
+@pytest.mark.asyncio
+async def test_vessels_geojson_endpoint_filters_type_and_bbox():
+    now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
+    rows = [
+        (
+            VesselPosition(mmsi=1, lat=59.9, lon=10.7, received_at=now),
+            VesselStatic(mmsi=1, name="Cargo Ship", vessel_type=70, vessel_type_name="Cargo"),
+        ),
+        (
+            VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now - timedelta(minutes=1)),
+            VesselStatic(mmsi=2, name="Passenger Ship", vessel_type=60, vessel_type_name="Passenger"),
+        ),
+    ]
+
+    class _Result:
+        def all(self):
+            return rows
+
+    class _FakeSession:
+        async def execute(self, _query):
+            return _Result()
+
+    async def override_get_db():
+        yield _FakeSession()
+
+    app.dependency_overrides[get_db] = override_get_db
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get(
+                "/api/v1/visualization/geo/vessels",
+                params={"bbox": "0,50,20,70", "type": "cargo", "limit": 0},
+            )
+
+        assert response.status_code == 200
+        data = response.json()
+        assert data["count"] == 1
+        assert data["features"][0]["properties"]["name"] == "Cargo Ship"
+        assert data["stats"]["by_type"]["Cargo"] == 1
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_vessels_geojson_merges_raw_and_legacy_sources(monkeypatch):
+    now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
+    monkeypatch.setattr(
+        visualization,
+        "get_aggregated_vessels",
+        AsyncMock(
+            return_value=[
+                {
+                    "mmsi": 1,
+                    "lat": 59.9,
+                    "lon": 10.7,
+                    "received_at": now,
+                    "name": "AISSTREAM SHIP",
+                    "vessel_type_name": "Cargo",
+                    "source_summary": {"aisstream_vessels": {"message_types": ["PositionReport"]}},
+                }
+            ]
+        ),
+    )
+    rows = [
+        (
+            VesselPosition(mmsi=1, lat=60.0, lon=10.8, received_at=now),
+            VesselStatic(mmsi=1, name="LEGACY DUP", vessel_type_name="Cargo"),
+        ),
+        (
+            VesselPosition(mmsi=2, lat=60.3, lon=5.3, received_at=now),
+            VesselStatic(mmsi=2, name="BARENTSWATCH ONLY", vessel_type_name="Passenger"),
+        ),
+    ]
+
+    class _Result:
+        def all(self):
+            return rows
+
+    class _FakeSession:
+        async def execute(self, _query):
+            return _Result()
+
+    async def override_get_db():
+        yield _FakeSession()
+
+    app.dependency_overrides[get_db] = override_get_db
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/visualization/geo/vessels")
+
+        assert response.status_code == 200
+        data = response.json()
+        names = {feature["properties"]["mmsi"]: feature["properties"]["name"] for feature in data["features"]}
+        assert data["count"] == 2
+        assert names == {1: "AISSTREAM SHIP", 2: "BARENTSWATCH ONLY"}
+        assert data["diagnostics"]["legacy_backfilled_mmsi"] == 1
+    finally:
+        app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_vessel_name_fallbacks_reports_mmsi_display_names(monkeypatch):
+    now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
+    monkeypatch.setattr(
+        visualization,
+        "get_aggregated_vessels",
+        AsyncMock(
+            return_value=[
+                {
+                    "mmsi": 257123000,
+                    "lat": 59.9,
+                    "lon": 10.7,
+                    "received_at": now,
+                    "name": "MMSI 257123000",
+                    "vessel_type_name": "Other",
+                    "source_summary": {
+                        "aisstream_vessels": {
+                            "latest_observed_at": now,
+                            "message_types": ["PositionReport"],
+                        }
+                    },
+                }
+            ]
+        ),
+    )
+
+    class _Result:
+        def all(self):
+            return []
+
+    class _FakeSession:
+        async def execute(self, _query):
+            return _Result()
+
+    async def override_get_db():
+        yield _FakeSession()
+
+    app.dependency_overrides[get_db] = override_get_db
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/visualization/vessels/name-fallbacks")
+
+        assert response.status_code == 200
+        data = response.json()
+        assert data["count"] == 1
+        assert data["items"][0]["mmsi"] == "257123000"
+        assert data["items"][0]["message_types"] == ["PositionReport"]
+    finally:
+        app.dependency_overrides.clear()

backend/tests/test_visualization_compute_centers.py

@@ -215,3 +215,141 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
         assert data["features"][0]["properties"]["data_type"] == "compute_center"
     finally:
         app.dependency_overrides.clear()
+
+
+@pytest.mark.asyncio
+async def test_visualization_geo_summary_returns_counts(monkeypatch):
+    records = [
+        _build_record(
+            record_id=1,
+            source="arcgis_cables",
+            data_type="submarine_cable",
+            name="Test Cable",
+            country="",
+            city="",
+            latitude=0,
+            longitude=0,
+            metadata={
+                "route_coordinates": [[[0, 0], [1, 1]]],
+                "status": "active",
+            },
+        ),
+        _build_record(
+            record_id=2,
+            source="arcgis_landing_points",
+            data_type="landing_point",
+            name="Test Landing",
+            country="United States",
+            city="New York",
+            latitude=40.7,
+            longitude=-74.0,
+            metadata={"city_id": 10},
+        ),
+        _build_record(
+            record_id=3,
+            source="celestrak_tle",
+            data_type="satellite_tle",
+            name="TESTSAT",
+            country="",
+            city="",
+            latitude=0,
+            longitude=0,
+            metadata={
+                "norad_cat_id": 12345,
+                "tle_line1": "1 12345U 98067A   24001.00000000  .00000000  00000-0  00000-0 0  9991",
+                "tle_line2": "2 12345  51.6000 100.0000 0001000  10.0000  20.0000 15.50000000    01",
+            },
+        ),
+        _build_record(
+            record_id=4,
+            source="top500",
+            data_type="supercomputer",
+            name="Frontier",
+            country="United States",
+            city="Oak Ridge",
+            latitude=35.93,
+            longitude=-84.31,
+            metadata={"rank": 1, "rmax": 1102000.0},
+        ),
+        _build_record(
+            record_id=5,
+            source="epoch_ai_gpu",
+            data_type="gpu_cluster",
+            name="Colossus",
+            country="United States",
+            city="Memphis",
+            latitude=35.15,
+            longitude=-90.05,
+            metadata={"value": "20000", "unit": "TFlop/s"},
+        ),
+    ]
+
+    class _ScalarResult:
+        def __init__(self, rows=None, scalar_value=None):
+            self._rows = rows or []
+            self._scalar_value = scalar_value
+
+        def scalar(self):
+            return self._scalar_value
+
+        def all(self):
+            return list(self._rows)
+
+        def scalars(self):
+            class _Scalars:
+                def __init__(self, rows):
+                    self._rows = rows
+
+                def all(self):
+                    return self._rows
+
+            return _Scalars(self._rows)
+
+    class _FakeSession:
+        async def execute(self, query):
+            query_text = str(query).lower()
+            if "bgp_incidents" in query_text:
+                return _ScalarResult(scalar_value=2)
+            if "bgp_anomalies" in query_text:
+                return _ScalarResult(scalar_value=3)
+            if "ais_raw_observations" in query_text or "vessel_position" in query_text:
+                return _ScalarResult(rows=[])
+            return _ScalarResult(rows=records)
+
+        async def get(self, *_args, **_kwargs):
+            return None
+
+    async def override_get_db():
+        yield _FakeSession()
+
+    async def _fake_build_bgp_collector_coverage(*_args, **_kwargs):
+        return [
+            {"collector": "rrc00"},
+            {"collector": "rrc01"},
+        ]
+
+    monkeypatch.setattr(
+        "app.api.v1.visualization.build_bgp_collector_coverage",
+        _fake_build_bgp_collector_coverage,
+    )
+
+    app.dependency_overrides[get_db] = override_get_db
+    transport = ASGITransport(app=app)
+    try:
+        async with AsyncClient(transport=transport, base_url="http://test") as client:
+            response = await client.get("/api/v1/visualization/geo/summary")
+
+        assert response.status_code == 200
+        stats = response.json()["stats"]
+        assert stats["cable_count"] == 1
+        assert stats["landing_point_count"] == 1
+        assert stats["satellite_count"] == 1
+        assert stats["compute_center_count"] == 2
+        assert stats["supercomputer_count"] == 1
+        assert stats["gpu_cluster_count"] == 1
+        assert stats["bgp_event_count"] == 2
+        assert stats["bgp_incident_count"] == 2
+        assert stats["bgp_anomaly_count"] == 3
+        assert stats["bgp_collector_count"] == 2
+    finally:
+        app.dependency_overrides.clear()

backend/tests/test_websocket_manager.py

@@ -0,0 +1,46 @@
+import pytest
+
+from app.core.websocket.manager import ConnectionManager
+
+
+class FakeWebSocket:
+    def __init__(self):
+        self.accepted = False
+        self.sent = []
+        self.closed = False
+
+    async def accept(self):
+        self.accepted = True
+
+    async def send_json(self, message):
+        self.sent.append(message)
+
+    async def close(self):
+        self.closed = True
+
+
+@pytest.mark.asyncio
+async def test_channel_subscribers_receive_channel_broadcasts():
+    manager = ConnectionManager()
+    socket = FakeWebSocket()
+
+    await manager.connect(socket, "user-1")
+    manager.subscribe(socket, ["dashboard"])
+    await manager.broadcast({"type": "data_frame", "channel": "dashboard"}, channel="dashboard")
+
+    assert socket.accepted is True
+    assert socket.sent == [{"type": "data_frame", "channel": "dashboard"}]
+
+
+@pytest.mark.asyncio
+async def test_disconnect_removes_channel_subscriptions():
+    manager = ConnectionManager()
+    socket = FakeWebSocket()
+
+    await manager.connect(socket, "user-1")
+    manager.subscribe(socket, ["dashboard"])
+    manager.disconnect(socket, "user-1")
+    await manager.broadcast({"type": "data_frame", "channel": "dashboard"}, channel="dashboard")
+
+    assert socket.sent == []
+    assert "dashboard" not in manager.channel_subscriptions

database_schema.sql

@@ -86,6 +86,12 @@ CREATE TABLE collection_tasks (
     id BIGSERIAL PRIMARY KEY,
     datasource_id INTEGER NOT NULL REFERENCES data_sources(id) ON DELETE CASCADE,
     status task_status NOT NULL DEFAULT 'pending',
+    phase VARCHAR(30) DEFAULT 'queued',
+    phase_progress FLOAT,
+    phase_message VARCHAR(255),
+    phase_current BIGINT,
+    phase_total BIGINT,
+    phase_unit VARCHAR(30),
     started_at TIMESTAMP WITH TIME ZONE,
     completed_at TIMESTAMP WITH TIME ZONE,
     records_processed INTEGER DEFAULT 0,

docker-compose.local-model.yml

@@ -18,6 +18,9 @@ services:
     build:
       context: .
       dockerfile: aiprovider/Dockerfile
+      args:
+        PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
+        UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
     container_name: planet_aiprovider
     ports:
       - "8010:8010"

docker-compose.simple.yml

@@ -5,6 +5,12 @@ services:
     build:
       context: .
       dockerfile: aiprovider/Dockerfile
+      args:
+        PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
+        UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
+    env_file:
+      - ./aiprovider/.env
+      - ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
     container_name: planet_aiprovider
     ports:
       - "8010:8010"

docker-compose.yml

@@ -5,8 +5,12 @@ services:
     build:
       context: .
       dockerfile: aiprovider/Dockerfile
+      args:
+        PYTHON_IMAGE: ${PYTHON_IMAGE:-python:3.14-slim}
+        UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:latest}
     env_file:
       - ./aiprovider/.env
+      - ${PLANET_AI_PROVIDER_RUNTIME_ENV_FILE:-./aiprovider/.env}
     container_name: planet_aiprovider
     ports:
       - "8010:8010"

docs/CHANGELOG.md

@@ -8,6 +8,367 @@ This project follows the repository versioning rule:
 - `improvement` -> `+0.0.1`（bugfix + 小功能混合）
 - `bugfix` -> `+0.0.1`
 
+## [0.48.0] — 2026-05-07
+
+Released: 2026-05-07
+
+### ✨ Highlights
+- 自定义数据源新增 REST / WebSocket 映射运行时，并提供本地 AIS mock WebSocket，用于实时船只 upsert 链路验证。
+- AIS 原始观测、聚合策略、字段来源、冲突记录与船舶 enrichment 继续完善，Earth 船只实时展示链路更接近生产数据形态。
+- Earth 全球态势 summary 改为轻量 SQL 聚合，并在卫星 current 异常时回退到最近有效 TLE 批次，避免统计接口被大规模明细读取拖慢。
+
+### 🔧 Improvements
+- 修复 `/geo/summary` 与 `/geo/satellites` 在大表下加载慢或超时的问题，并补充 `collected_data` 与 AIS raw 相关索引。
+- WebSocket 管理器支持匿名连接、频道订阅清理和更稳的连接生命周期测试，前端 WebSocket candidates / fallback 更可靠。
+- `planet.sh` 强化端口释放、端口诊断和前端启动流程，mock AIS server 提供 Bun 脚本入口。
+
+---
+
+## [0.47.0] — 2026-04-30
+
+Released: 2026-04-30
+
+### ✨ Highlights
+- 新增 AISStream WebSocket 船只采集器，并将 AIS 多源数据写入原始观测层，由聚合接口统一去重、合并和解释字段来源。
+- 设置页新增 AISStream API Key、采集范围 preset、运行状态、连接验证和凭证教程入口，让全球 AIS 采集链路可配置、可观察。
+- Earth 船只图层默认不再限制 5000 艘，并统一 marker 颜色、详情卡、hover 和搜索结果的船型归一化显示。
+
+### 🔧 Improvements
+- 聚合接口新增 `field_sources`、`selected_reasons`、`source_summary`、`quality_flags` 和冲突记录调试接口，动态字段默认优先采用更新的实时流观测。
+- AISStream 标准化支持 `MetaData.ShipName` 船名兜底，并将 AIS 数字船型映射为 Cargo / Tanker / Passenger / Fishing / Military。
+- 将仓库 docs 技能改为通用文档工作流，Planet 专属白名单、双语、裸文件标题和凭证教程规则迁移到 `docs/documentation-coverage-rules.md`。
+- 更新 AIS v4/v5 TODO 与计划文档，明确后续聚合策略配置、船舶资料 enrichment 和媒体缓存边界。
+
+---
+
+## [0.46.3] — 2026-04-30
+
+Released: 2026-04-30
+
+### 🐛 Fixes
+- 优化 Starlink footprint 显示后的地球拖拽性能，避免旋转地球时每帧重建 footprint 大网格，同时保持现有视觉效果不变。
+- 恢复点击线缆后的呼吸透明度动画，让 locked / hover 线缆重新使用既有 pulse 配置。
+
+---
+
+## [0.46.2] — 2026-04-30
+
+Released: 2026-04-30
+
+### 🐛 Fixes
+- 修复 Earth 启动时高清材质、云图和图层可见性绕过 `startupPriority` 的问题，统一由启动队列按文档顺序加载。
+- 修复保存为关闭的高清材质/图层仍会先加载再关闭的问题，并保持海陆基座作为国界线图层的常驻底图。
+- 修复搜索跳转会误关媒体面板、船只轨迹末端不贴合当前船只、Iridium footprint 被地表层遮挡等 Earth 交互问题。
+
+### 📝 Documentation
+- 更新 Earth 图层顺序、样式参考、使用手册和 AIS 聚合计划，补齐中英文说明与后续接入策略。
+
+---
+
+## [0.46.1] — 2026-04-30
+
+Released: 2026-04-30
+
+### 🐛 Fixes
+- 修复新增 technical docs 文件存在但未进入 Docs 前端白名单时，侧栏不显示且 Markdown 链接无法解析到 `/docs/<slug>` 的问题。
+- 补齐数据源/采集器连接验证与 Earth Interactable 使用说明的英文文档，保证公开 Docs 切换 EN 时同名页面可访问。
+- 清理中英文 technical docs 中裸 `.md` 文件名链接标题，改为面向读者的语义标题。
+
+### 📝 Documentation
+- 将 Docs 前端白名单、公开文档双语配对、裸文件名链接标题三项检查写入 Claude 与 Codex 的 docs 技能流程。
+
+---
+
+## [0.46.0] — 2026-04-30
+
+Released: 2026-04-30
+
+### ✨ Highlights
+- Earth 新增通用 Interactable 图标层，船只、算力中心、BGP 事件与观测站统一使用批量 Points、屏幕拾取、状态 glow 和状态缩放。
+- BGP 事件保留向外扩散圈，观测站保留雷达扫描层，并与 Interactable 主图标解耦到稳定的地表渲染层级。
+- 登陆点回归黄色球形 Sprite，贴近海缆层级并保持更稳定的地表显示和遮挡表现。
+
+### 🔧 Improvements
+- 新增 SVG asset 到 canvas texture 的 Interactable 资产加载路径，支持统一图标资源、缓存和可选染色。
+- 同坐标 Interactable 自动做地表切向避让，降低重叠物件无法选择的问题。
+- 优化 Earth toolbar 初始尺寸注入，避免首次显示原始尺寸后再跳到缩放尺寸。
+- 补充 Interactable 计划、使用说明、图层顺序和 Earth 前端上下文文档。
+- 修复船只 hover/locked 状态仅发光但放大反馈不明显的问题，将已有状态缩放接入通用图标层。
+
+---
+
+## [0.45.0] — 2026-04-29
+
+### ✨ Highlights
+- 采集任务新增阶段级量化进度，`fetching` 可展示百分比、阶段说明和字节下载量。
+- AI Provider 启动链路支持从 `aiprovider/.env` 与 `~/.zshrc` 注入运行期配置，并避免密钥/模型变化触发镜像重建。
+- AI Provider Docker build context 收敛到服务必需文件，`uv sync` 接入 BuildKit 缓存以减少重复下载。
+
+### 🔧 Improvements
+- IPtoASN、OpenGeoFeed、NRO delegated 下载型采集器接入真实字节进度上报。
+- 数据源页、采集中任务弹窗和任务历史页展示阶段摘要，并在 tooltip 中保留完整进度细节。
+- 调整 Earth 船只默认高度偏移，进一步贴近地表展示。
+
+---
+
+## [0.44.2] — 2026-04-29
+
+### 📝 Documentation
+- 补充 Earth 船只图层技术文档，记录分桶 `THREE.Points` 批量渲染、同尺寸交互 overlay 和屏幕空间 picking 的设计约束。
+- 同步 Earth 渲染图层顺序和样式参考，明确 AIS 船只 renderOrder、depthTest、图标尺寸、航向分桶与 hover 命中半径。
+- 更新船只渲染性能计划状态，标注 `0.44.1` 已落地的实现与后续全球 AIS / LOD 演进方向。
+
+---
+
+## [0.44.1] — 2026-04-29
+
+### 🐛 Fixes
+- 修复 Earth 船只图层拖动不跟手的问题，将普通船只从独立 Sprite 切换为按航向分桶的批量 Points 渲染。
+- 修正船只 hover/click 拾取错位，改为屏幕空间命中检测并在拖拽/惯性期间跳过 hover 拾取。
+- 统一 AIS 船只普通态与交互态方向，并让 hover/locked glow 与普通图标保持同尺寸覆盖。
+- 恢复船只深度测试并收敛默认图标尺寸，避免北部岛屿/冰面附近出现明显压盖陆地的视觉问题。
+
+---
+
+## [0.44.0] — 2026-04-29
+
+### ✨ Highlights
+- 重构数据源与采集器设置边界：数据源页回归目录和采集触发，采集器 endpoint、请求头、凭证与连接验证统一进入设置页
+- BarentsWatch AIS 完整接入凭证解析、连接检查、默认教程、AI 生成教程和船只采集/可视化链路
+- Earth 新增船只图例、缩放反馈胶囊、缩放感知拖拽灵敏度，并将船只渲染性能优化方案沉淀到 plans
+- 仪表盘重启服务新增前端重启 action，并让 runner 通过 `~/.zshrc` 继承本地环境变量
+
+### 🔧 Improvements
+- 采集器连接状态改为基于成功采集或手动连接校验 checksum 判断，避免只依赖前端样式状态
+- 数据源页新增采集中任务标签和任务进度弹窗，内置与自定义数据源统一展示
+- Earth 国界线进一步贴近地表，并补充船只图层渲染顺序、样式和用户手册说明
+- docs skill 与 Claude/Codex 文档流程补齐技术文档和 plans 的职责边界
+
+---
+
+## [0.43.1] — 2026-04-28
+
+### 🐛 Fixes
+- 修正 `planet.sh` 在全量 restart 后启动 AI Provider 时的提示语义，避免把预期内未就绪描述成异常不健康
+
+---
+
+## [0.43.0] — 2026-04-28
+
+### ✨ Highlights
+- 新增 Earth 船舶追踪链路，接入 BarentsWatch AIS 凭证配置、采集器、后端 vessel 模型/API 与前端 Earth 船舶图层
+- 新增自定义数据源映射流程，支持样本抓取、目标 schema、AI 辅助生成映射、预览校验和映射执行
+- Settings 拆分 AI Provider 与采集器凭证配置，DataSources 只保留采集状态、运行参数和必要引导
+
+### 🔧 Improvements
+- AI Provider 支持运行时 LLM 配置、provider preset 下拉与刷新，并在 Playground 中引导到 AI 配置页
+- Markdown 渲染器补齐代码块复制按钮、语言标签、任务列表、图片、自动链接、删除线和文档主题样式
+- Docs 公开导航改为显式元数据白名单，避免开发任务文档自动出现在“其他”分组
+- 将 Codex/Claude cleanup、docs、goal-driven、release 流程补充 CLI-first 约束，并把 `rules.md` 整理成可按模块加载的工程规则
+
+---
+
+## [0.42.2] — 2026-04-28
+
+### 🐛 Fixes
+- Docs 中文模式下补齐左侧分组、文档标题、页头分类与搜索结果分类翻译，并更新文档站品牌标题/副标题文案
+
+---
+
+## [0.42.1] — 2026-04-28
+
+### 🐛 Fixes
+- 修正 release skill 的 feature 版本计算规则：minor 进位时 patch 必须重置为 `0`，例如 `0.41.2` 应发布为 `0.42.0`
+
+---
+
+## [0.42.0] — 2026-04-28
+
+### ✨ Highlights
+- 新增公开 `/docs` 文档站，支持中英文技术文档、使用手册、Quickstart、搜索、目录锚点与浅色/深色/跟随系统主题
+- Earth 在无高清材质时新增轻量 Fresnel 边缘提示，并调整卫星覆盖默认显示与地表材质可读性
+
+### 🔧 Improvements
+- 将技术文档整理为 `docs/technical/zh` 与 `docs/technical/en`，并补充控制台、`planet.sh`、Earth 与公共组件使用说明
+- 新增 `SegmentedControl` 公共滑块组件，支持缩放参数，复用到 docs 语言与主题切换
+- Markdown 渲染器接入自定义滚动条，表格与代码块在深色模式和 overflow 场景下保持可读
+- Docs 搜索结果支持内部滚动、点击外部关闭、重新聚焦恢复上次搜索结果
+- Earth 工具栏展开状态与设置持久化版本迁移继续收口，改善默认面板和快捷关闭行为
+
+---
+
+## [0.41.2] — 2026-04-27
+
+### 🔧 Improvements
+- `planet.sh` 启动链路新增 verbose 滚动输出窗口，并在后端端口占用时打印目标地址和监听进程诊断
+- Docker 构建支持通过 build args 覆盖 Python 与 uv 镜像，方便 Docker Hub 不稳定时切换镜像源
+
+### 🐛 Fixes
+- Earth 海缆登陆点改为基于相机射线与地球遮挡判断可见性，修复旋转后 pin 可见性滞后一帧的问题
+
+### 🔧 Improvements
+- `docker-compose*.yml` 为 AI Provider 构建传入 `PYTHON_IMAGE` / `UV_IMAGE` 参数，默认仍使用官方镜像
+- 后端启动失败遇到 `Address already in use` 时输出 `lsof`、`ss` 与 PID 命令行信息
+- verbose 模式下 AI Provider build、后端与前端启动日志会在 spinner 下方保留最新 5 行滚动展示
+
+---
+
+## [0.41.1] — 2026-04-27
+
+### 🐛 Fixes
+- 修复新闻直播面板设置项持久化失效：`closeTransientMobileOverlays` 通过旁路路径隐藏面板导致下次 persist 快照到错误状态，改为不重新从 DOM 读取面板可见性
+- 修复登陆点 pin 在地球侧面被半截遮挡：改为在接近地平线前（dot < 0.05）主动隐藏，避免深度测试切片
+
+### 🔧 Improvements
+- 将所有画布绘制的图标抽取为 SVG，存入 `frontend/public/earth/assets/icons/`，新增图标规范到 `rules.md`
+
+---
+
+## [0.41.0] — 2026-04-27
+
+### ✨ Highlights
+- Earth 图层系统完成地表到天空的注册顺序与关注优先的面板顺序拆分，支持基座海陆色块、国界、高清材质、云图、地形、算力、BGP、卫星、轨迹与海缆的稳定层级
+- 国界层新增真实行政区轮廓交互与中国/台湾联动高亮，修复高清材质、地形、footprint、卫星与经纬线之间的遮挡和 hover 竞争
+
+### 🔧 Improvements
+- 新增无轮廓基座地图，所有图层关闭时仍保留 `#010609` 海洋与 `#080f1b` 陆地色块
+- 将大气云图抽象为独立图层并接入桌面/移动端图层开关、持久化状态与启动同步
+- 高清材质改为独立纹理覆盖层，地形显示在高清材质上方，并在高清材质关闭/恢复时保持原地形开关意图
+- 补充 Earth 渲染层级与图层样式文档，记录正式图层名、变量名、材质颜色、线宽与 renderOrder
+
+---
+
+## [0.40.5] — 2026-04-26
+
+### 🔧 Improvements
+- 卫星拖尾改用 Instanced screen-space ribbon，单 draw call 渲染所有轨迹段，支持像素级宽度控制
+- Iridium 地面覆盖重写为球面投影径向网格，修复填充光晕不可见问题；新增外圈 LineLoop
+- 搜索面板打开时改用双 rAF 延迟聚焦输入框，确保 CSS 过渡完成后焦点可靠触发
+- 代码清理：提取 `IRIDIUM_OVERLAY_COLOR`、`IRIDIUM_REFERENCE_ALTITUDE_KM` 常量，消除重复三角函数调用
+
+---
+
+## [0.39.0] — 2026-04-24
+
+## [0.40.4] — 2026-04-26
+
+### 🔧 Improvements
+- 新增页面可见性恢复处理，页面从后台切回前台时主动刷新卫星位置，避免累积后台时间在下一帧一次性回放
+- 抽出卫星轨迹状态与轨迹几何清理 helper，统一后台恢复与清空数据时的轨迹重置路径
+
+### 🐛 Fixes
+- 修复页面在后台停留较久后恢复前台时，卫星轨迹因超大 `deltaTime` 突然跳变、拖尾异常拉长的问题
+- 修复后台恢复后首帧仍沿用旧轨迹缓存，导致轨迹与当前卫星位置短时错位的问题
+
+---
+
+## [0.40.3] — 2026-04-25
+
+### 🔧 Improvements
+- 卫星点云升级为自定义 ShaderMaterial，支持 per-point alpha 控制，锁定/悬停卫星从点云中精确隐藏
+- 修复锁定环与自发光选中标记的 depthTest 错误（false → true），消除远端渲染穿透 artifact
+- 新增锁定环悬停态缩放与线宽（LOCKED_RING_HOVER_SCALE / LOCKED_RING_HOVER_LINE_WIDTH）
+- 修复 updateLockedDotWorldTransform / updateLockedHaloWorldTransform 未强制刷新 matrixWorld 导致的位置漂移
+
+---
+
+## [0.40.2] — 2026-04-24
+
+### 🔧 Improvements
+- 卫星点大小随镜头缩放动态调整，拉近变大、拉远变小，响应与相机距离线性对应
+- 调小卫星点默认基础尺寸（dotSize 2.8），缩放范围更合理
+
+---
+
+## [0.40.1] — 2026-04-24
+
+### 🔧 Improvements
+- 卫星选中标记（lockedring / lockeddot / 光晕）颜色统一跟随图例轨道倾角分类配色
+- 修复 Starlink footprint 在特定视角下遮蔽卫星点的渲染顺序问题（Group renderOrder 影响子 Mesh 排序）
+- footprint 材质改为 `depthTest: false` + 相机朝向 limbFade，替代 polygonOffset 深度竞争方案
+- 修复选中海缆时误触发附近卫星高亮（该行为属于 BGP 事件点逻辑，不应用于海缆）
+
+---
+
+## [0.40.0] — 2026-04-24
+
+### ✨ Highlights
+- Earth 卫星 footprint 正式按星座能力分层：Starlink 保留专用地表覆盖，Iridium 改为独立外圈覆盖表达，其它非 Starlink 星座不再误用同一套 footprint
+- Earth 卫星详情卡补齐覆盖能力与当前显示说明，用户现在可以直接看见每颗卫星为什么显示 footprint、为何回退为自身发光
+
+### 🔧 Improvements
+- 后端可视化接口新增并透传 `constellation_group` 与 `footprint_policy`，前端据此执行 capability-gated footprint renderer
+- 新增 Iridium 独立 coverage ring adapter，并继续保留 Starlink 专用 footprint 调校与昼夜可读性增强
+- 新增 Earth 卫星 footprint 策略技术文档，明确 GNSS、generic LEO、GEO 与 Iridium 的显示边界
+
+### 🐛 Fixes
+- 修复前后端对 Iridium footprint policy 命名不一致，导致策略分发语义含混的问题
+- 清理 Starlink footprint 渲染中的未使用常量与过时命名，减少后续继续调校时的歧义
+
+---
+
+## [0.39.0] — 2026-04-24
+
+### ✨ Highlights
+- 后端正式落下统一结构化日志地基：请求上下文、事件名、脱敏与持久化链路开始收口为可扩展的企业级日志体系
+- 系统日志页重构为真正的日志工作台：顶部筛选更紧凑，终端日志区成为主视觉，移动端 Earth 新闻/态势细节交互继续补稳
+
+### 🔧 Improvements
+- 新增 `backend/app/core/logging.py`，统一 `request_id`、`service`、`event` 注入与敏感字段脱敏，并接入后端主入口、调度器、缓存、数据库和可视化链路
+- 系统日志页筛选区重排为更紧凑的两层结构，信息摘要并入终端工具栏 tooltip，日志终端区留出更稳定的按钮避让空间
+- Earth 移动端态势抽屉补齐宽度约束与图例换行规则，新闻详情抽屉在巡航切换时可同步更新标题和摘要
+
+### 🐛 Fixes
+- 修复 `/tmp/planet_backend.log` 中混入空字节时，日志摘要条行数与实际可见日志不一致的问题
+- 修复移动端“态势”tab 在内容渲染后被图例文本撑宽、超出一屏的问题
+- 修复移动端新闻详情抽屉在巡航切换下一条新闻时标题更新但 summary 不同步的问题
+
+---
+
+## [0.38.0] — 2026-04-23
+
+### ✨ Highlights
+- Earth 新闻正式接入通用巡航层：新闻和 BGP 统一进入可配置巡航模块，桌面端与移动端都能在巡航聚焦时展示对应新闻卡片
+- 系统日志页升级为结构化过滤链路：按真实时间戳、结构化级别和字符串检索统一筛选，不再依赖前端或后端从日志文本里猜结果
+
+### 🔧 Improvements
+- 新闻巡航补齐业务适配层：按发生地与时间生成巡航目标，桌面端与移动端统一标题 + summary 卡片风格，并增加连线与打字机摘要展示
+- 日志页筛选体验重排，统一服务源、级别、行数、时间和检索布局，日历标记改为由后端返回的结构化每日聚合结果驱动
+- 后端补充 `system_logs` 结构化解析与多级别精确过滤能力，Earth 浏览器端日志缓冲与系统日志 API 现在走同一套筛选语义
+
+### 🐛 Fixes
+- 修复新闻巡航模块开启后难以关闭、桌面/移动端设置状态互相污染的问题
+- 修复新闻巡航卡片缺少摘要、移动端详情样式不统一、新闻巡航缺少连线的问题
+- 修复日志级别筛选会被访问日志 query string 中的 `level=error` 等参数污染，从而把 `INFO` 行误判为 `ERROR` 的问题
+
+---
+
+## [0.37.2] — 2026-04-23
+
+### ✨ Highlights
+- Earth 图层系统新增经纬线开关，桌面图层面板与移动端抽屉都可直接控制
+
+### 🔧 Improvements
+- 经纬线正式接入 Earth layer registry，复用现有图层切换、移动端图层卡片与设置持久化流
+
+### 🐛 Fixes
+- 修复经纬线只能默认常驻、无法作为独立图层开关控制的问题
+
+---
+
+## [0.37.1] — 2026-04-23
+
+### ✨ Highlights
+- `planet.sh` 后端重启链路修复 `uvicorn --reload` 残留 worker 场景，`restart` 现在能真正替换旧实例
+
+### 🔧 Improvements
+- 收口后端清理逻辑，统一按 `uvicorn` 进程、端口占用进程和进程组执行清理，减少 reload 场景漏杀分支
+
+### 🐛 Fixes
+- 修复部分机器执行 `./planet.sh restart --allow-lan` 后后端仍停留旧实例，导致 `/api/v1/visualization/geo/compute-centers` 返回 `404` 的问题
+
+---
+
 ## [0.35.1] — 2026-04-22
 ## [0.37.0] — 2026-04-23
 

docs/documentation-coverage-rules.md

@@ -0,0 +1,117 @@
+# Documentation Coverage Rules
+
+This file contains Planet-specific documentation coverage rules. Documentation skills and agents should read this file before deciding which docs to update. Keep tool-specific workflow in skills; keep product and repository rules here.
+
+## Scope Rules
+
+- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
+- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
+- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
+- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
+- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
+- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
+- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
+- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
+- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
+
+## Public Docs Rules
+
+- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
+- If a technical document should be visible in the public Docs page or linked from a technical README, register it in `frontend/src/pages/Docs/docs-content.ts` under `DOCS_METADATA`. Files under `docs/technical/{zh,en}/` are not automatically routable.
+- For every public technical doc, keep the bilingual file pair in sync by filename: `docs/technical/zh/<name>.md` and `docs/technical/en/<name>.md`. If content is intentionally Chinese-only or English-only, state that intentionally in the final note.
+- Public docs should use readable link text, not raw filenames such as `manual.md`.
+
+## Credential Collector Rules
+
+- Any built-in collector marked `requires_credentials: true` and `credential_status: supported` must have:
+  - a `credential_provider` in `backend/app/core/datasource_defaults.py`;
+  - a default credential guide in `backend/app/services/credential_guides.py`;
+  - a supported connectivity provider in `backend/app/services/datasource_connectivity.py`;
+  - settings UI guidance or a credential form in `frontend/src/pages/Settings/Settings.tsx`;
+  - a regression test that fails if the guide/provider is missing.
+
+## Recommended Checks
+
+Run the checks that match the affected docs.
+
+### Duplicate Bilingual Docs
+
+```bash
+python - <<'PY'
+from pathlib import Path
+same = []
+for en in sorted(Path("docs/technical/en").glob("*.md")):
+    zh = Path("docs/technical/zh") / en.name
+    if zh.exists() and en.read_text() == zh.read_text():
+        same.append(en.name)
+if same:
+    raise SystemExit("identical en/zh docs: " + ", ".join(same))
+print("no identical en/zh docs")
+PY
+```
+
+### Language-Less Technical Links
+
+```bash
+rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
+```
+
+This should return no matches.
+
+### Public Docs Registry
+
+```bash
+python - <<'PY'
+import re
+from pathlib import Path
+
+metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
+known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
+known.add("README.md")
+
+missing = []
+for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
+    if not readme.exists():
+        continue
+    for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
+        path = Path(href)
+        if "docs/technical/" not in href:
+            continue
+        filename = path.name
+        if filename not in known:
+            missing.append(f"{readme}: {filename}")
+
+if missing:
+    raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
+print("docs README links are whitelisted")
+PY
+```
+
+### Public Bilingual Pairs
+
+```bash
+python - <<'PY'
+import re
+from pathlib import Path
+
+metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
+filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
+missing = []
+for filename in filenames:
+    for lang in ("zh", "en"):
+        path = Path("docs/technical") / lang / filename
+        if not path.exists():
+            missing.append(str(path))
+if missing:
+    raise SystemExit("missing bilingual docs: " + ", ".join(missing))
+print("public docs have zh/en file pairs")
+PY
+```
+
+### Raw Filename Link Titles
+
+```bash
+rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
+```
+
+This should return no matches for polished public docs.

docs/plans/README.md

@@ -19,10 +19,16 @@
 - [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
 - [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
 - [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
+- [earth-country-boundary-overlay-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-country-boundary-overlay-plan.md)
 - [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
 - [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
 - [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
 - [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
+- [earth-news-cruise-summary-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
+- [earth-vessel-rendering-performance-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
+- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
+- [earth-interactable-layer-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-interactable-layer-plan.md)
+- [frontend-public-docs-site-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-public-docs-site-plan.md)
 - [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
 - [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
 

docs/plans/custom-source-live-mock-plan.md

@@ -0,0 +1,384 @@
+# Custom Source Live Mock 计划
+
+**状态**：实施中  
+**创建日期**：2026-05-01  
+**任务名**：`Custom Source Live Mock`  
+**核心目标**：把自定义源升级为同时支持 REST 与 WebSocket 的可映射采集入口，并提供本地 AIS mock WebSocket 服务，用于验证 Earth 船只实时新增与 upsert 链路。
+
+## 背景
+
+真实 AIS 接口变化频率不可控，无法稳定验证 Earth 页面“不刷新也能看到新船只”的实时链路。当前系统已经有自定义源基础设施：
+
+- `datasource_configs` 保存 endpoint、auth、headers、config。
+- `datasource_mapping_templates` 保存目标 schema 的确定性映射模板。
+- `run-mapped` 支持保存后的自定义 REST 源通过 active mapping 写入目标数据。
+
+但现有能力主要面向 REST sample 和批量 mapping，缺少以下能力：
+
+- 自定义源不能明确选择 `REST` 或 `WebSocket` 采集模式。
+- WebSocket 长连接、订阅消息、重连、消息路径提取还没有通用 runtime。
+- `vessel_ais` 自定义数据写入后需要进入 AIS raw observation 和 `vessels` WS channel，才能真实验证 Earth 实时 upsert。
+- 删除自定义源时没有清晰的数据清理选项。
+- 设置中心里“采集调度 / 凭证 / 自定义源”入口混杂，用户很难判断该在哪里配置。
+
+## 已确认决策
+
+| 项目 | 决策 |
+|-----|------|
+| 计划名称 | `Custom Source Live Mock` |
+| 自定义源传输类型 | 支持 `REST` 与 `WebSocket` |
+| 采集写入方式 | 先映射到目标 schema，再由 destination handler 写入 |
+| AIS mock 目标 | 优先打通 `vessel_ais`，验证 Earth 船只实时新增和同 MMSI upsert |
+| mock 服务 runtime | 使用 `bun` 启动本地 mock WS 服务 |
+| 凭证配置 | 支持 headers、bearer、api key、basic，并保留 query/header API key 位置配置 |
+| 删除策略 | 删除自定义源时允许选择是否删除该源写入的数据 |
+| 合并语义 | 自定义源必须选择“合并到哪个内置数据”，作为内置源的补充数据进入同一聚合链路 |
+| UI 方向 | 自定义源创建和维护放在“配置中心 > 采集器设置”的采集器下拉框内联入口；数据源页保留总览与运行控制 |
+
+## 范围
+
+### 本阶段要做
+
+- 自定义源可选择 `REST` 或 `WebSocket`。
+- 自定义源支持请求头、凭证、query params、body、WS subscribe message。
+- WebSocket 自定义源支持长连接、重连、消息解析、mapping、写入。
+- `vessel_ais` 自定义源写入 AIS raw observations，并广播 `vessels` channel。
+- 提供 mock AIS WS 服务，持续发送新增 MMSI 和位置变更。
+- 删除自定义源时提供“是否删除该源数据”的选项。
+- 梳理设置中心信息架构，明确后续 UI 重构方向。
+
+### 暂不做
+
+- 不新增任意动态数据库表。
+- 不允许用户提交可执行脚本作为 mapping。
+- 不让 LLM 进入正式采集链路。
+- 不把 mock 数据直接写 legacy `vessel_position`，优先写 AIS raw observations，保持可追踪和可删除。
+- 不在本阶段完成完整 `Earth Live Sync`，但要为后续 summary invalidation 留出 hook。
+
+## 现状入口
+
+| 能力 | 当前位置 |
+|-----|----------|
+| 自定义源配置模型 | `backend/app/models/datasource_config.py` |
+| 自定义源 mapping 模型 | `backend/app/models/datasource_mapping.py` |
+| 自定义源 API | `backend/app/api/v1/datasource_config.py` |
+| 目标 schema registry | `backend/app/core/target_schema_registry.py` |
+| mapping engine | `backend/app/services/datasource_mapping.py` |
+| 数据源总览 UI | `frontend/src/pages/DataSources/DataSources.tsx` |
+| 采集器设置 UI | `frontend/src/pages/Settings/Settings.tsx` |
+
+## 目标架构
+
+```mermaid
+flowchart LR
+    A[Custom Source Config] --> B{source_type}
+    B -->|rest| C[Mapped REST Runner]
+    B -->|websocket| D[Mapped WS Runner]
+    C --> E[Mapping Engine]
+    D --> E
+    E --> F[Target Schema Validator]
+    F --> G{Destination Handler}
+    G -->|vessel_ais| H[AIS Raw Observations]
+    H --> I[AIS Aggregation]
+    H --> J[vessels WS Channel]
+    J --> K[Earth Vessel Upsert]
+```
+
+## 数据配置设计
+
+短期可以继续复用 `DataSourceConfig`，避免大迁移。语义约定如下：
+
+| 字段 | 用途 |
+|-----|------|
+| `name` | 自定义源唯一名称，例如 `mock_ais_ws` |
+| `source_type` | `rest` 或 `websocket` |
+| `endpoint` | `http(s)://...` 或 `ws(s)://...` |
+| `auth_type` | `none`、`bearer`、`api_key`、`basic` |
+| `auth_config` | token、api_key、key name、basic username/password 等 |
+| `headers` | 静态请求头 |
+| `config` | method、params、body、timeout、retry、WS 订阅消息、重连策略、消息路径等 |
+
+建议 `config` 结构：
+
+```json
+{
+  "transport": "websocket",
+  "delivery_mode": "realtime_stream",
+  "merge_target_source": "barentswatch_vessels",
+  "target_schema": "vessel_ais",
+  "method": "GET",
+  "params": {},
+  "body": null,
+  "timeout": 30,
+  "retry": 3,
+  "ws_subscribe_message": {"type": "subscribe", "channel": "vessels"},
+  "ws_message_path": "$.data",
+  "ws_items_path": "$.vessels[*]",
+  "ws_reconnect": true,
+  "reconnect_delay_seconds": 3,
+  "debug_max_messages": null,
+  "delete_policy": "config_only"
+}
+```
+
+## 后端实施计划
+
+### Phase 1 — 自定义源类型与连接测试
+
+- 允许 `source_type` 为 `rest` 或 `websocket`。
+- REST 连接测试保留现有 HTTP 请求逻辑。
+- WebSocket 连接测试新增：
+  - 校验 endpoint 必须是 `ws://` 或 `wss://`。
+  - 注入 headers 和 auth。
+  - 连接后可选发送 `ws_subscribe_message`。
+  - 读取一条消息或超时返回诊断。
+
+### Phase 2 — Mapped REST Runner 补齐
+
+现有 `run-mapped` 继续作为 REST 一次性采集入口，补齐：
+
+- `GET/POST` method。
+- query params。
+- JSON body。
+- headers 和 auth 注入。
+- sample limit 与响应大小限制。
+- `vessel_ais` destination handler。
+
+### Phase 3 — Mapped WebSocket Runner
+
+新增通用 WebSocket runner，读取 `DataSourceConfig + active mapping`：
+
+- 建立长连接。
+- 发送可选订阅消息。
+- 循环接收消息。
+- JSON parse。
+- 按 `ws_message_path/ws_items_path` 提取 item 或 list。
+- 使用 mapping engine 转换。
+- 使用 target schema validator 校验。
+- 调用 destination handler 写入。
+- 更新采集任务状态：
+  - `connecting`
+  - `streaming`
+  - `reconnecting`
+  - `stopped`
+- 维护运行指标：
+  - `messages_seen`
+  - `records_written`
+  - `unique_entities`
+  - `last_message_at`
+  - `last_error`
+- 后台长连接不读取 `config.debug_max_messages`；该字段只用于显式的一次性调试运行，避免正式 WS 流被测试上限截断。
+
+### Phase 4 — Destination Handler
+
+为 target schema 建立明确写入处理器。
+
+`vessel_ais` handler：
+
+- 写入 `AISRawObservation`。
+- `source = datasource.name`。
+- `delivery_mode` 来自 config，默认 WS 为 `realtime_stream`、REST 为 `polling`。
+- `transport` 来自 `source_type`。
+- 生成幂等 observation hash。
+- 更新 AIS source health。
+- 广播 `vessels` channel，payload 使用当前 Earth 已支持的 upsert 格式。
+
+`generic_records` handler：
+
+- 写入通用 collected data 或后续 generic store。
+- 不直接进入 Earth。
+
+### Phase 5 — 删除与数据清理
+
+删除自定义源时新增清理策略：
+
+| 选项 | 行为 |
+|-----|------|
+| 只删除配置 | 删除 `datasource_configs`，保留 mapping 和历史数据需要另行处理 |
+| 删除配置和 mapping | 删除配置及对应 `datasource_mapping_templates` |
+| 删除配置、mapping 和该源数据 | 同时删除该源写入的数据 |
+
+数据删除范围：
+
+- `collected_data.source == datasource.name`
+- `ais_raw_observations.source == datasource.name`
+- `ais_source_health.source == datasource.name`
+
+不建议直接删除 legacy `vessel_position`，因为当前 legacy 表不带 source，无法安全归因。自定义 AIS 源应优先只写 raw observations。
+
+删除数据后应触发：
+
+- `vessels` channel 的 reload/invalidation 事件，提示 Earth 重新拉船只聚合。
+- 后续接入 `Earth Live Sync` 后，触发 `earth_summary` invalidation。
+
+### Phase 6 — Mock AIS WebSocket 服务
+
+新增脚本：
+
+`scripts/mock-ais-ws-server.ts`
+
+运行方式建议：
+
+```bash
+bun run mock:ais-ws
+```
+
+服务行为：
+
+- 监听 `ws://localhost:8787/ais`。
+- 接受任意客户端连接。
+- 可记录收到的 subscribe message。
+- 每 1-2 秒发送一条 AIS-like JSON。
+- 每隔 N 条生成新 MMSI，验证船只数量增长。
+- 已存在 MMSI 随时间改变 `lat/lon/cog/heading`，验证同 MMSI upsert。
+- 支持固定 seed，保证测试可复现。
+
+示例 payload：
+
+```json
+{
+  "type": "vessel",
+  "data": {
+    "mmsi": "999000001",
+    "name": "MOCK VESSEL 001",
+    "lat": 31.23,
+    "lon": 121.47,
+    "sog": 12.4,
+    "cog": 86,
+    "heading": 90,
+    "received_at": "2026-05-01T00:00:00Z"
+  }
+}
+```
+
+## 前端实施计划
+
+### 信息架构调整
+
+自定义源不作为割裂的新入口，而是作为内置采集器的补充源，直接纳入“配置中心 > 采集器设置”的采集器选择器：
+
+- 采集器下拉框同时展示内置采集器和自定义补充源。
+- 下拉框右侧提供加号按钮，用于添加自定义源。
+- 新建自定义源时必须选择“合并到内置数据”，例如合并到 `barentswatch_vessels`。
+- 选择自定义源后，右侧基础配置区域沿用正常采集器配置形态，支持连接测试、保存、endpoint、headers、auth、高级 JSON。
+- 自定义源比内置源多一个“删除自定义源”按钮。
+- 删除时弹出确认框，可勾选“同时删除该自定义源生成的所有数据”。
+
+数据源页保留：
+
+- 内置源总览。
+- 内置源最近状态。
+- 内置源手动触发。
+- 不展示自定义源管理入口；自定义源创建、维护、删除统一在采集器设置中完成。
+
+### 自定义源表单
+
+新增或重构自定义源表单：
+
+- 源名称。
+- 类型：`REST` / `WebSocket`。
+- 合并到内置数据：必选，用于声明该源补充哪个内置数据域。
+- endpoint。
+- method/body/params，仅 REST 显示。
+- subscribe message/message path/items path，仅 WS 显示。
+- auth type。
+- headers。
+- target schema。
+- sample/test 按钮。
+- mapping assistant/preview。
+- 保存并运行。
+
+### 删除确认
+
+删除自定义源时弹出确认：
+
+- 默认只删除配置。
+- 可勾选删除 mapping。
+- 可勾选删除该源写入的数据。
+- 显示将删除的数据范围和不可恢复提示。
+
+## 验证方案
+
+### Mock WS 验证路径
+
+1. 启动 mock 服务：
+
+```bash
+bun run mock:ais-ws
+```
+
+2. 新建自定义源：
+
+| 字段 | 值 |
+|-----|----|
+| name | `mock_ais_ws` |
+| source_type | `websocket` |
+| endpoint | `ws://localhost:8787/ais` |
+| merge_target_source | `barentswatch_vessels` |
+| target_schema | `vessel_ais` |
+| ws_message_path | `$.data` |
+
+3. 保存 active mapping：
+
+```json
+{
+  "source": {
+    "items_path": "$"
+  },
+  "fields": {
+    "mmsi": {"path": "$.mmsi", "type": "integer"},
+    "name": {"path": "$.name", "type": "string"},
+    "lat": {"path": "$.lat", "type": "float"},
+    "lon": {"path": "$.lon", "type": "float"},
+    "sog": {"path": "$.sog", "type": "float", "default": null},
+    "cog": {"path": "$.cog", "type": "float", "default": null},
+    "heading": {"path": "$.heading", "type": "integer", "default": null},
+    "received_at": {"path": "$.received_at", "type": "datetime", "default": null}
+  }
+}
+```
+
+4. 启动自定义源。
+
+5. 打开 Earth 船只图层，不刷新页面观察：
+
+- `vessels` WS channel 收到 `source = mock_ais_ws`。
+- HUD 船只数在新 MMSI 到达时增加。
+- 地球出现 `MOCK VESSEL`。
+- 同 MMSI 后续消息更新位置和航向，不重复叠加。
+
+### 自动化测试
+
+后端测试：
+
+- WebSocket 自定义源连接测试。
+- WS message path 和 items path 提取。
+- mapping 到 `vessel_ais`。
+- 写入 AIS raw observation。
+- 广播 `vessels` channel。
+- 删除自定义源时按策略删除 mapping 和源数据。
+
+前端测试：
+
+- REST/WS 表单条件显示。
+- 删除确认选项。
+- mock 源配置保存 payload。
+- mapping preview 展示错误和成功记录。
+
+## 风险与约束
+
+- WebSocket 自定义源是长连接，不能沿用一次性 REST 进度条。
+- 如果 mock 源写 legacy vessel 表，删除会变得不安全，因此先只写 raw observations。
+- 自定义 WS 可能消息量很大，必须有 backpressure、日志限流和任务取消能力。
+- 任意外部 WS 不能信任 payload，必须经过 mapping 和 schema validation。
+- headers/auth 不能进入 LLM mapping prompt。
+
+## 交付顺序
+
+1. Mock AIS WS 服务。
+2. 后端自定义 WS runner。
+3. `vessel_ais` destination handler 和 `vessels` broadcast。
+4. 删除自定义源及数据清理。
+5. 设置中心采集器下拉框内联自定义源 UI。
+6. 配置中心信息架构重整。
+7. 与 `Earth Live Sync` 对接 summary invalidation。

docs/plans/datasource-custom-api-mapping-plan.md

@@ -0,0 +1,426 @@
+# 自定义 API 数据源与 LLM 映射系统 — 实施计划
+
+**状态**：规划中
+**创建日期**：2026-04-28
+**核心原则**：LLM 辅助生成映射配置；生产采集使用确定性转换引擎
+
+## 已确认决策
+
+| 项目 | 决策 |
+|-----|------|
+| 自定义 API 的定位 | 作为内置数据源的补充入口，不直接等同于 Earth 新功能 |
+| LLM 的职责 | 探索未知 API、分析样本 JSON、生成 mapping 草案 |
+| 采集时是否调用 LLM | 不调用；采集链路必须确定性、可审计、可复现 |
+| 自定义数据如何进入 Earth | 必须映射到已支持的目标 schema，或先进入通用数据沉淀 |
+| 外部凭证放置位置 | Settings / 外部集成统一管理 provider token；DataSources 引用 provider profile |
+| TimescaleDB | 放入 TODO；高频时序数据稳定后再评估迁移 |
+
+---
+
+## 一、背景与问题
+
+当前系统已经有 `datasource_configs`，可以配置自定义数据源的 endpoint、auth、headers、config，也已经有部分 collector 会读取这些配置。但这只能解决“怎么请求数据”，还没有解决以下问题：
+
+- API 返回 JSON 后，如何转换成系统已有领域模型。
+- 自定义数据源是补充已有能力，还是全新数据沉淀。
+- 转换规则由谁生成、谁校验、谁执行。
+- 未知数据是否能自动在 Earth 上展示。
+- 外部 token 是放在全局配置中心，还是放在每个 datasource 下。
+
+专业做法是把“请求配置”“外部凭证”“目标 schema”“字段映射”“采集执行”拆开：
+
+- Settings 管外部集成凭证，例如 AI Provider、BarentsWatch、未来付费 AIS API。
+- DataSources 管具体数据源实例，例如 endpoint、调度频率、目标 schema、mapping 版本。
+- LLM 只在配置阶段辅助生成 mapping，不进入生产采集链路。
+- Earth 只消费明确 schema 的数据，不消费任意未知 JSON。
+
+---
+
+## 二、目标架构
+
+### 2.1 自定义 API 数据源生命周期
+
+```mermaid
+flowchart LR
+    A[配置 endpoint/auth/request] --> B[抓取 sample JSON]
+    B --> C[选择目标 schema]
+    C --> D[LLM 生成 mapping 草案]
+    D --> E[确定性 mapping engine 预览]
+    E --> F[schema validation]
+    F --> G[保存 mapping version]
+    G --> H[scheduler 执行 mapped collector]
+    H --> I[写入目标表或 generic_records]
+```
+
+### 2.2 目标 schema 分层
+
+| schema | 用途 | Earth 可视化 |
+|-------|------|-------------|
+| `vessel_ais` | 船只 AIS 位置、航速、航向、MMSI 等 | 进入船舶图层 |
+| `geo_points` | 通用点位数据，包含经纬度、名称、类型、时间 | 进入通用 geo layer（TODO） |
+| `news_events` | 新闻/事件类数据，带时间、地点、摘要、来源 | 复用新闻/事件链路 |
+| `compute_centers` | 算力中心、机房、数据中心数据 | 复用算力中心图层 |
+| `generic_records` | 未知结构化数据沉淀 | 不直接展示 |
+
+v1 建议优先实现：
+
+- `vessel_ais`
+- `geo_points`
+- `generic_records`
+
+其他 schema 可先在 registry 中预留名称，但不承诺完整落库与可视化。
+
+### 2.3 LLM 的边界
+
+LLM 可以做：
+
+- 根据 API 文档或 sample JSON 解释字段含义。
+- 推荐目标 schema。
+- 生成 mapping JSON 草案。
+- 给出字段置信度和需要人工确认的字段。
+- 帮用户发现分页、数组路径、时间字段、坐标字段。
+
+LLM 不应该做：
+
+- 在正式采集时参与每批数据转换。
+- 生成并执行 Python/JavaScript 代码。
+- 接触 API key、bearer token、basic auth password。
+- 自动创建新的 Earth 图层或数据库表。
+
+---
+
+## 三、后端实施计划
+
+### Phase 1 — Target Schema Registry
+
+新增代码级 registry，统一描述系统支持的目标数据类型。
+
+每个 target schema 至少包含：
+
+- `key`：例如 `vessel_ais`。
+- `label`：前端展示名称。
+- `description`：适用场景。
+- `fields`：字段名、类型、是否必填、说明、示例。
+- `validator`：Pydantic 或等价校验器。
+- `destination`：写入目标，例如 vessel 表、generic_records、future geo layer。
+
+示例概念：
+
+```json
+{
+  "key": "vessel_ais",
+  "fields": [
+    {"name": "mmsi", "type": "integer", "required": true},
+    {"name": "lat", "type": "float", "required": true},
+    {"name": "lon", "type": "float", "required": true},
+    {"name": "sog", "type": "float", "required": false},
+    {"name": "cog", "type": "float", "required": false},
+    {"name": "received_at", "type": "datetime", "required": false}
+  ]
+}
+```
+
+### Phase 2 — Mapping Template Model
+
+新增 mapping 配置持久化表，建议命名为 `datasource_mapping_templates`。
+
+关键字段：
+
+- `id`
+- `datasource_config_id`
+- `target_schema`
+- `mapping_json`
+- `sample_payload_hash`
+- `validation_status`
+- `version`
+- `is_active`
+- `created_at`
+- `updated_at`
+
+`mapping_json` 是声明式 DSL，不允许任意代码执行。
+
+示例：
+
+```json
+{
+  "source": {
+    "items_path": "$.data.vessels[*]"
+  },
+  "fields": {
+    "mmsi": {"path": "$.mmsi", "type": "integer"},
+    "lat": {"path": "$.latitude", "type": "float"},
+    "lon": {"path": "$.longitude", "type": "float"},
+    "sog": {"path": "$.speedOverGround", "type": "float", "default": null},
+    "received_at": {"path": "$.timestamp", "type": "datetime"}
+  }
+}
+```
+
+### Phase 3 — Deterministic Mapping Engine
+
+实现独立 mapping engine，输入 sample/raw payload 和 mapping JSON，输出目标 schema 记录。
+
+v1 支持能力：
+
+- JSONPath/JMESPath 风格路径提取。
+- 数组展开。
+- 默认值。
+- 基础类型转换：string、integer、float、boolean、datetime。
+- 坐标范围校验。
+- 简单枚举映射。
+- 错误收集：缺字段、类型转换失败、路径不存在。
+
+明确不支持：
+
+- 任意表达式执行。
+- 用户提交脚本。
+- LLM runtime 修复。
+
+### Phase 4 — LLM Mapping Assistant API
+
+新增配置阶段 API：
+
+- `POST /api/v1/datasources/custom/sample`
+  - 按 datasource 请求配置抓取 sample JSON。
+- `GET /api/v1/datasources/target-schemas`
+  - 返回可选目标 schema 和字段说明。
+- `POST /api/v1/datasources/mappings/propose`
+  - 输入 sample JSON + target schema，调用 AI provider 生成 mapping 草案。
+- `POST /api/v1/datasources/mappings/preview`
+  - 使用确定性 mapping engine 预览转换结果。
+- `POST /api/v1/datasources/mappings`
+  - 保存 mapping 版本。
+- `PUT /api/v1/datasources/mappings/{id}`
+  - 更新 mapping，生成新版本或覆盖草稿。
+- `POST /api/v1/datasources/{id}/run-mapped`
+  - 手动触发一次 mapped collector。
+
+安全要求：
+
+- `propose` 请求发送给 LLM 前必须脱敏 sample。
+- auth headers、token、password 不进入 prompt。
+- LLM 返回结果必须再经过 mapping schema 校验。
+
+### Phase 5 — Generic Mapped HTTP Collector
+
+新增通用 collector：
+
+- 读取 `DataSourceConfig` 请求配置。
+- 读取 active mapping template。
+- 拉取 API 数据。
+- 使用 mapping engine 转换。
+- 使用 target schema validator 校验。
+- 调用 destination handler 写入目标表或 generic storage。
+- 将失败记录写入错误日志或 dead-letter 结构。
+
+对于 `generic_records`：
+
+- 保存 datasource id。
+- 保存 target schema。
+- 保存 normalized JSON。
+- 保存 raw payload 摘要或 raw reference。
+- 保存采集时间、source timestamp、mapping version。
+
+---
+
+## 四、前端实施计划
+
+### Phase 1 — Settings 外部集成
+
+Settings 中保留统一外部集成配置：
+
+- AI Provider：base URL、model、API key。
+- BarentsWatch：client id/client secret 或 bearer token。
+- 未来付费接口：AISHub、MarineTraffic、VesselFinder 等 provider profile。
+
+DataSources 不直接管理全局 secret，只引用 provider profile。
+
+### Phase 2 — Settings 自定义源向导
+
+自定义数据源配置入口应放在 `/settings` 的“采集器设置”或后续专门的自定义采集器设置区。`/datasources` 保持数据源目录和采集触发职责，不再承载编辑入口。
+
+自定义数据源配置改成向导或右侧 drawer：
+
+1. Request
+   - endpoint
+   - method
+   - auth profile
+   - headers
+   - query/body config
+   - schedule
+2. Sample
+   - 点击抓取 sample
+   - 展示 JSON tree
+   - 支持选择数组根路径
+3. Target Schema
+   - 选择 `vessel_ais`、`geo_points`、`generic_records`
+   - 展示该 schema 必填字段
+4. Mapping Proposal
+   - 调用 LLM 生成 mapping 草案
+   - 显示字段匹配置信度
+   - 标出需要人工确认的字段
+5. Preview
+   - 用确定性 engine 预览前 N 条转换结果
+   - 展示校验错误
+6. Save & Enable
+   - 保存 mapping version
+   - 启用调度或仅保存草稿
+
+### Phase 3 — 运维视图
+
+为 mapped datasource 展示：
+
+- 上次运行时间。
+- 成功记录数。
+- 失败记录数。
+- 当前 mapping version。
+- 目标 schema。
+- 最近错误。
+- 手动运行按钮。
+
+---
+
+## 五、数据库与存储策略
+
+### v1：继续使用 PostgreSQL
+
+PostgreSQL 可以承载当前规模的采集、关系查询、JSONB 沉淀和基础时序查询。v1 不必因为“时序数据”立刻引入 TimescaleDB。
+
+适合继续用 PostgreSQL 的场景：
+
+- 数据量可控。
+- 最近状态查询为主。
+- 历史保留窗口较短。
+- 查询模式还没稳定。
+- 需要快速迭代 schema 与 mapping。
+
+### TODO：TimescaleDB
+
+以下条件满足后，再评估 TimescaleDB：
+
+- AIS、遥测、轨迹类数据达到高频持续写入。
+- 需要按时间窗口做聚合、降采样、retention policy。
+- 单表时间序列查询明显成为瓶颈。
+- 历史轨迹保留从 24h 扩展到数周或数月。
+
+候选迁移对象：
+
+- `vessel_position`
+- future telemetry tables
+- future generic time-series records
+
+备选方案：
+
+- PostgreSQL 原生按天/月分区。
+- TimescaleDB hypertable。
+- 热数据 PostgreSQL，冷数据对象存储。
+
+---
+
+## 六、安全与治理
+
+### Secret 管理
+
+- Settings 中保存 provider credentials。
+- API 返回配置时必须 mask secret。
+- LLM prompt 只能包含脱敏 sample 和 schema 说明。
+- 后续 TODO：引入字段级加密或 KMS。
+
+### Mapping 治理
+
+- 每次 mapping 变更保留版本。
+- active mapping 只能有一个。
+- 允许保存 draft mapping。
+- 运行记录关联 mapping version。
+- 校验失败不能自动启用。
+
+### 错误处理
+
+常见错误类型：
+
+- API 401/403：凭证错误或过期。
+- API 429：限流，需要调整 schedule。
+- JSON path 不存在：上游结构变化。
+- 类型转换失败：mapping 规则错误。
+- schema validation failed：转换结果不满足目标模型。
+
+每次运行需要记录：
+
+- datasource id。
+- mapping version。
+- started_at / finished_at。
+- fetched count。
+- mapped count。
+- written count。
+- failed count。
+- error summary。
+
+---
+
+## 七、测试计划
+
+### Backend Unit Tests
+
+- mapping engine：
+  - path 提取。
+  - 数组展开。
+  - 默认值。
+  - 类型转换。
+  - datetime parse。
+  - 枚举映射。
+  - 缺字段错误。
+- target schema registry：
+  - `vessel_ais` 必填字段校验。
+  - `geo_points` 经纬度范围校验。
+  - `generic_records` 接受未知结构。
+- LLM assistant：
+  - mock provider 返回 mapping。
+  - 验证 secret 不进入 prompt。
+  - 验证非法 mapping 被拒绝。
+
+### Backend Integration Tests
+
+- sample JSON -> propose mapping -> preview -> save mapping。
+- mapped collector 使用保存的 mapping 写入 `generic_records`。
+- `vessel_ais` sample 写入船舶相关目标结构。
+- 上游 JSON 结构变化时，运行失败并记录错误。
+
+### Frontend Tests
+
+- 自定义数据源向导完整流程。
+- 未配置 AI Provider 时，提示去 Settings 配置，但允许手写 mapping。
+- LLM 返回不完整 mapping 时，Preview 阶段显示校验错误。
+- 保存 mapping 后展示 active version 和运行状态。
+
+---
+
+## 八、分期工作量
+
+| 阶段 | 内容 | 估算 |
+|-----|------|------|
+| Phase 0 | 完成本规划、确认 schema registry 设计 | 0.5 天 |
+| Phase 1 | target schema registry + mapping template model | 1–2 天 |
+| Phase 2 | deterministic mapping engine | 2–3 天 |
+| Phase 3 | sample/propose/preview/save API | 2–3 天 |
+| Phase 4 | DataSources 自定义源向导 | 3–5 天 |
+| Phase 5 | generic mapped collector + run history | 2–4 天 |
+| Phase 6 | vessel_ais / geo_points destination handler | 2–4 天 |
+
+---
+
+## 九、当前差距与下一步
+
+当前差距：
+
+- `datasource_configs` 只描述请求配置，不描述目标 schema 和 mapping。
+- 自定义源没有 sample -> schema -> mapping -> preview -> save 的闭环。
+- 生产采集还没有通用 mapped collector。
+- Settings 与 DataSources 的职责边界需要在 UI 上进一步明确。
+- Earth 还没有通用 `geo_points` 图层。
+
+下一步建议：
+
+1. 先实现 target schema registry 和 mapping engine，不急着接 LLM。
+2. 用固定 sample JSON 做 `vessel_ais` 和 `generic_records` 的单元测试。
+3. 再接 LLM propose API，让 LLM 产出的只是 mapping 草案。
+4. 最后做前端向导，把人工确认和 preview 放到启用之前。

docs/plans/earth-interactable-layer-plan.md

@@ -0,0 +1,313 @@
+# Earth Interactable Layer Plan
+
+## 背景
+
+状态：Phase 1 已经开始落地，Phase 2 的 BGP 事件 / 观测站迁移和 Phase 3 的算力中心迁移也已完成。`frontend/public/earth/js/interactable.js` 已新增，AIS 船只、BGP 事件、BGP 观测站和算力中心图层已经改为通过 `createInteractableLayer()` 使用通用批量 `Points`、hover / locked overlay、默认 glow、状态更新、asset icon 预加载、屏幕空间 picking、固定 / 距离缩放和跨 Interactable 同坐标避让。登陆点因 `THREE.Points` 边缘深度裁切和贴地层级要求，已退回专用 `THREE.Sprite` 黄色球路径，并与海缆同高度同 renderOrder。后续阶段聚焦把可复用的扩圈 / 雷达扇形动画正式沉淀成 `animations` 扩展。
+
+当前实现说明和接入示例见：
+
+- [earth-interactable-usage.md](/home/ray/dev/linkong/planet/docs/technical/zh/earth-interactable-usage.md)
+
+当前 AIS 船只图层已经形成了一个适合作为基准的交互图标模式：
+
+- 普通态使用批量 `THREE.Points` 渲染，避免每个对象一个 `Sprite` 带来的 draw call 和透明排序压力。
+- hover / locked 态使用单点 overlay 叠加 glow，不改变普通批次，交互反馈清晰且成本低。
+- moving / anchored 船只通过 canvas 点纹理表达不同形状，moving 船只还按航向分桶。
+- 拾取走屏幕空间命中，拖动和惯性期间跳过高频 hover picking。
+- 图层高度贴近地表，仅保留很小的深度余量，避免“浮在表面层”的观感。
+
+这个模式不应该只服务船只。后续 BGP 事件、BGP 观测站、算力中心、新闻事件、告警、地面传感器等都可能需要“图标类可交互元素”。如果每个图层继续各写一套 icon、glow、hover、locked、动画、picking 和图例逻辑，视觉会漂移，性能策略也会重复分叉。登陆点已经验证为例外：需要完整贴地且不被球面边缘裁切时，专用 Sprite 路径比通用 `Points` 更合适。
+
+目标是把船只图层的成功做法抽象成一个通用接口：业务图层只描述“要画什么、在哪里、怎么交互”，底层统一负责批量渲染、默认 glow、状态 overlay、动画槽位、拾取和生命周期。
+
+## 目标
+
+1. 建立统一的 Earth 交互图标接口，作为未来地表图标类元素的默认入口。
+2. 以 AIS 船只 glow 为默认 glow 视觉，其它图标默认沿用同一套 glow 质感。
+3. 保留图标颜色、状态颜色、hover 放大、locked 强调、dimmed 聚焦、动画扩展等能力。
+4. 支持 canvas / SVG / image icon，不强行要求所有图标都可重着色。
+5. 保持船只当前性能路线：批量绘制普通态，少量 overlay 处理交互态。
+6. 给 BGP 事件扩圈、BGP 观测站雷达扇形等补充动画留出正式扩展点。
+
+## 非目标
+
+- 不在第一阶段重写所有 Earth 图层。
+- 不把卫星、海缆、国家边界、真实地形这类非图标图层纳入同一个接口。
+- 不为了抽象牺牲业务图标的差异表达，例如船只航向、BGP 事件严重级别、观测站雷达扫掠。
+- 不要求图片图标支持运行时重着色；图片图标只能通过预制多状态图片或 overlay tint 做有限表达。
+
+## 核心设计
+
+建议新增一个通用模块，例如：
+
+```text
+frontend/public/earth/js/interactable.js
+```
+
+它导出一个工厂或注册函数：
+
+```js
+createInteractableLayer({
+  id,
+  earth,
+  renderOrder,
+  altitudeOffset,
+  icon,
+  scale,
+  glow,
+  colors,
+  states,
+  animations,
+  picking,
+  data,
+  getPosition,
+  getKind,
+  getRotation,
+  getPayload,
+});
+```
+
+业务模块仍保留自己的数据加载、图例、详情卡字段和业务语义。例如 `vessels.js` 负责 AIS 数据和船型映射，但 icon 渲染、hover overlay、locked overlay、默认 glow 和屏幕空间 picking 可以逐步迁入 `interactable.js`。
+
+## 参数草案
+
+| 参数 | 类型 / 示例 | 默认值 | 说明 |
+| --- | --- | --- | --- |
+| `id` | `"vessels"` | 必填 | 图层唯一标识，用于 debug、picking、legend 和状态缓存。 |
+| `earth` | `THREE.Object3D` | 必填 | 图层挂载目标，通常是 Earth root。 |
+| `renderOrder` | `4.4` | `4` | 普通 icon 批次和 overlay 的基础渲染顺序。 |
+| `altitudeOffset` | `0.2` | `0.2` | 图层高度，语义为 `CONFIG.earthRadius + altitudeOffset`。地表图标默认贴近真实地形基础层。 |
+| `icon` | `{ type, source, draw, size, bins }` | 必填 | 图标来源。支持 canvas draw、SVG URL、image URL、内置 shape。 |
+| `icon.fitSize` | `60` 或 `{ width: 60, height: 60 }` | `atlasCellSize` | asset 图标在 atlas canvas 内的最大绘制尺寸，默认居中等比 contain。SVG / 图片文件只负责原始形状，不需要为了显示大小手写 transform。 |
+| `scale` | `{ base, min, max }` | `{ base: 1 }` | 基础缩放和距离稳定范围。当前船只可映射到 `VESSEL_POINT_SIZE` / `baseScale`。 |
+| `sizeMode` | `"fixed" / "distance"` | `"fixed"` | 是否固定屏幕像素尺寸；非 fixed 时按相机到地表距离做比例缩放。 |
+| `sizeScale` | `{ min, max, referenceFov }` | `{ min: 0.12, max: 3, referenceFov: 75 }` | `sizeMode !== "fixed"` 时的缩放限制和参考视角。 |
+| `glow.enabled` | `true / false` | `true` | 是否启用默认 glow。默认 glow 以船只 hover / locked overlay 为基准。 |
+| `glow.intensity` | `0.0 - 2.0` | `1` | glow 强度，内部映射到 canvas `shadowBlur`、opacity 或 shader uniform。 |
+| `glow.colorMode` | `"state" / "icon" / "fixed"` | `"state"` | glow 颜色来源，默认跟随状态颜色。 |
+| `hover.scale` | `1.0 - 2.0` | `1.18` | hover 放大倍率。当前船只保持同尺寸 glow overlay，接口仍保留放大能力供其它图层使用。 |
+| `hover.mode` | `"scale" / "glow-only" / "custom"` | `"scale"` | hover 反馈方式。船只可用 `"glow-only"`，其它图标默认放大。 |
+| `colors.normal` | `"#4A90D9"` | icon 原色 | 普通态颜色。只有可上色 icon 生效。 |
+| `colors.hover` | `"#7dd3fc"` | normal | hover 态颜色。 |
+| `colors.locked` | `"#ffffff"` | hover | locked 态颜色。 |
+| `colors.dimmed` | `"#9B9B9B"` | normal | 聚焦其它对象时的弱化颜色。 |
+| `colors.byKind` | `{ cargo: "#4A90D9" }` | `{}` | 按业务类型着色，如船型、BGP 严重级别。 |
+| `colorable` | `true / false` | 由 icon 类型推断 | canvas shape 和 SVG mask 通常可上色；图片默认不可上色。 |
+| `opacity` | `{ normal, hover, locked, dimmed }` | 船只当前值 | 各状态透明度。 |
+| `rotation` | `{ enabled, bins, getAngle }` | disabled | 是否按角度分桶，例如船只按 COG 分 32 桶。 |
+| `animations` | `IconAnimationSpec[]` | `[]` | 补充动画列表，例如扩圈、雷达扇形、脉冲、轨迹尾迹。 |
+| `picking.radiusPx` | `22` | `20` | 屏幕空间命中半径。 |
+| `picking.throttleMs` | `100` | `80` | hover picking 节流。 |
+| `picking.skipWhileDragging` | `true` | `true` | 拖动和惯性期间跳过 hover picking。 |
+| `zIndexPolicy` | `"surface-icon"` | `"surface-icon"` | 预设层级策略，避免每个业务图层手写高度和 renderOrder。 |
+| `avoidance.enabled` | `true / false` | `true` | 是否参与跨 Interactable 的同坐标避让。默认开启，同一经纬度下的图标会沿地表切平面小幅排开，方便辨认和选择。 |
+| `avoidance.radius` | `number` | `1.1` | 同坐标避让的第一圈半径，单位为地球本地坐标单位。 |
+| `avoidance.precision` | `number` | `4` | 经纬度归并精度，默认约等于只处理几乎完全重叠的图标。 |
+| `legend` | `{ label, color, shape }[]` | `[]` | 可选图例声明，业务层也可以继续自己导出。 |
+| `metadata` | object | `{}` | 业务扩展数据，不参与渲染但参与 tooltip / info-card / search。 |
+
+## Icon 规格
+
+图标输入建议分三类：
+
+```js
+{
+  type: "canvas-shape",
+  size: 128,
+  draw(ctx, state) {
+    // draw triangle / dot / custom shape
+  },
+}
+```
+
+```js
+{
+  type: "svg-mask",
+  source: "/earth/assets/icons/bgp-event-dot.svg",
+  colorable: true,
+}
+```
+
+```js
+{
+  type: "image",
+  source: "/earth/assets/icons/vendor-logo.png",
+  colorable: false,
+  stateSources: {
+    hover: "/earth/assets/icons/vendor-logo-hover.png",
+  },
+}
+```
+
+颜色策略：
+
+- `canvas-shape` 默认可上色，适合船只、事件点、雷达站这类符号。
+- `svg-mask` 如果能作为 mask 使用，则可上色；如果是完整多色 SVG，则按图片处理。
+- `image` 默认不可上色；需要状态变化时使用 `stateSources` 或额外 glow / ring。
+
+## 默认 Glow 规范
+
+默认 glow 以当前船只 overlay 为视觉基准：
+
+- 普通态尽量不启用 glow，保持地图干净。
+- hover / locked 态叠加同位置 overlay。
+- glow 颜色默认跟随状态颜色或业务类型颜色。
+- glow blur 应该稳定，不随 camera zoom 夸张膨胀。
+- 允许通过 `glow.intensity` 控制强度，但不要让业务图层各自发明完全不同的光晕语言。
+
+建议内部把 glow 拆成两个层次：
+
+1. `textureGlow`：canvas texture 里的 `shadowBlur`，适合小图标 hover / locked。
+2. `effectGlow`：额外 ring / halo / pulse，适合告警、BGP 事件和锁定强调。
+
+## 状态模型
+
+通用状态至少包含：
+
+| 状态 | 触发 | 默认表现 |
+| --- | --- | --- |
+| `normal` | 普通显示 | 批量 Points，使用 normal 颜色和 opacity。 |
+| `hover` | 指针悬停 | 默认放大并显示 glow；船只可配置为同尺寸 glow-only。 |
+| `locked` | 点击锁定 / 详情打开 | 强 glow、更高 opacity，可选 ring 或 pulse。 |
+| `dimmed` | 聚焦其它对象 | 降低 opacity，保留上下文。 |
+| `hidden` | 图层关闭或过滤 | 不参与绘制和 picking。 |
+| `alert` | 业务告警 | 可叠加动画，不替代 locked 状态。 |
+
+状态更新需要增量化：只在 hover 目标、locked 目标、过滤条件、数据版本或相机距离阈值变化时更新，不在每帧遍历全部 icon 写材质属性。
+
+## 动画扩展
+
+动画不直接塞进 icon 基础参数，而是作为 `animations` 列表注册。每个动画声明自己的 geometry / material / update 策略：
+
+```js
+{
+  type: "expanding-ring",
+  when: ["alert", "locked"],
+  color: "state",
+  radiusPx: [10, 42],
+  durationMs: 1400,
+  opacity: [0.8, 0],
+}
+```
+
+```js
+{
+  type: "radar-sweep",
+  when: ["normal", "hover", "locked"],
+  angleDeg: 72,
+  rotationMs: 2600,
+  opacity: 0.36,
+}
+```
+
+首批建议内置动画：
+
+| 动画 | 用例 | 说明 |
+| --- | --- | --- |
+| `pulse-ring` | locked、告警点 | 原地呼吸环，强调选中对象。 |
+| `expanding-ring` | BGP 事件 | 向外扩散的事件波纹。 |
+| `radar-sweep` | BGP 观测站 | 扇形扫描，可持续旋转。 |
+| `orbiting-dot` | 数据流 / collector 活跃态 | 小点绕 icon 环绕，表达活动状态。 |
+| `trail` | 移动目标 | 可选短尾迹，船只或飞机类目标使用。 |
+
+动画必须支持批量或分组绘制，避免为每个对象创建独立的高频更新对象。只有 locked / hover / 少量 alert 对象可以使用单对象 overlay。
+
+## 渲染策略
+
+### 普通态
+
+普通态优先使用分桶 `THREE.Points`：
+
+- 按 icon 类型、可上色策略、旋转分桶、纹理 key 分组。
+- 每组一个 `BufferGeometry`，存 `position`、`color`、必要的 `payloadIndex`。
+- `PointsMaterial.sizeAttenuation = false`，保持屏幕尺寸稳定。
+- `depthTest = true`，`depthWrite = false`，避免遮挡关系破坏地表。
+
+### 交互态
+
+hover / locked 使用少量 overlay：
+
+- overlay 复用 `THREE.Points` 单点对象或小型 ring mesh。
+- overlay texture 从统一 cache 获取。
+- overlay 更新只写当前 hover / locked 的 position、texture、opacity、size。
+
+### 高密度升级
+
+当某类图标超过分桶 Points 的舒适区，才考虑升级：
+
+- `InstancedBufferGeometry` billboard。
+- 自定义 shader 支持 per-instance rotation / scale / opacity。
+- 视口 bbox / LOD / cluster。
+
+这个升级不应该改变业务接口，只替换底层 renderer。
+
+## Picking 策略
+
+沿用船只当前方向：
+
+- 默认屏幕空间 picking，而不是 Three.js 对每个 Sprite / Points 做 raycast。
+- 每个 icon 保留世界坐标和业务 payload。
+- 每次 pointer move 将候选点投影到屏幕，按半径和深度判断命中。
+- 拖动、惯性旋转、相机剧烈变化期间跳过 hover picking。
+- click 时允许做一次更精确的 picking。
+
+后续可以按图层或经纬度网格增加空间索引，减少候选点数量。
+
+## 与现有图层的迁移路径
+
+### Phase 1：抽出船只基准能力
+
+- 从 `vessels.js` 提取 texture cache、canvas icon draw、overlay glow、分桶 Points 创建、状态增量更新。
+- 保持 `vessels.js` 的公开 API 不变：`loadVessels()`、`toggleVessels()`、`getVesselMarkers()` 等继续可用。
+- 新模块先只服务船只，确保视觉没有回退。
+
+### Phase 2：迁移 BGP 事件和观测站
+
+- BGP 事件使用 `canvas-shape`，已接入 `Interactable`。
+- 严重级别映射到 `colors.byKind`，并通过通用 `getPointSizeMultiplier` 保留严重级别尺寸倍率。
+- 当前扩圈效果保留在 BGP 业务动画中，并跟随 `Interactable` marker 位置更新。
+- BGP 观测站主图标已接入 `Interactable`，活跃度映射到颜色和 `getPointSizeMultiplier`。
+- BGP 观测站 halo / 覆盖扇形继续由 BGP 业务动画表达扫描，并跟随 `Interactable` marker 位置更新。
+
+### Phase 3：迁移算力中心并评估登陆点
+
+- 算力中心保留现有业务 icon，但接入统一 hover / locked / glow。（已完成）
+- 登陆点曾接入同一套 `Points` 渲染，但 pin 类 SVG 在地球边缘会被深度测试裁切；当前保留专用 `THREE.Sprite`，并使用 canvas 生成黄色扁平球，贴到海缆层级。
+- TODO：登陆点暂不迁移到完整 Interactable。后续若要统一交互接口，优先考虑 Sprite-backed adapter，只对齐 `getMarkers()`、`getPointerIntersections()`、`setMarkerState()`、`updateVisualState()` 等外观协议，不强行复用 `THREE.Points`、atlas 和跨图层避让。
+- 检查图例、搜索和 info-card 是否只依赖业务 payload，而不是依赖渲染对象类型。
+
+### Phase 4：形成 Earth 图标层规范
+
+- 在 `docs/technical/zh/earth-frontend-context.md` 记录当前实现入口。
+- 在 `docs/technical/zh/earth-layer-style-reference.md` 记录默认 glow、状态颜色、默认高度和动画参数。
+- 在 `docs/technical/zh/earth-render-layer-order.md` 记录 surface icon renderOrder 范围。
+
+## 风险与约束
+
+- 过早抽象可能让船只这种高质量基准被平均化，因此第一阶段必须以船只视觉不回退为验收标准。
+- 图片 icon 不可上色，接口需要明确 `colorable = false` 的行为，避免业务层误以为颜色一定生效。
+- 动画如果默认开启过多，会重新引入 overdraw 和每帧更新压力；默认只给 hover / locked 或少量 alert 使用。
+- 地形开启时，贴地 icon 需要在高度、`depthTest`、`polygonOffset` 和 renderOrder 之间保持平衡。
+- 统一 glow 不等于所有图标一模一样；业务可以调强度和颜色，但不应破坏整体视觉语言。
+
+## 验收标准
+
+1. 船只迁入通用接口后，普通态、hover、locked、航向、颜色、轨迹和 picking 行为保持一致。
+2. 新增一个 BGP 事件示例图层配置，不需要复制船只渲染代码即可得到 icon、glow、hover 和扩圈动画。
+3. 新增一个 BGP 观测站示例图层配置，不需要自写独立动画循环即可得到雷达扇形。
+4. 关闭图层后对应 icon、overlay、动画和 picking 全部停止。
+5. 高密度数据下普通态仍走批量绘制，hover / locked 只更新少量 overlay。
+6. 文档同步说明默认高度、默认 glow、状态模型和动画扩展点。
+
+## 相关文件
+
+| 文件 | 当前角色 | 未来关系 |
+| --- | --- | --- |
+| `frontend/public/earth/js/vessels.js` | 船只基准实现，包含分桶 Points、hover / locked overlay、默认 glow 形态 | Phase 1 的抽象来源 |
+| `frontend/public/earth/js/constants.js` | 保存船只高度、颜色、透明度、轨迹参数 | 后续可加入通用 surface icon 默认配置 |
+| `frontend/public/earth/js/bgp.js` | BGP 事件和观测站视觉逻辑 | BGP 事件和观测站主图标已接入 Interactable；扩圈、halo 和覆盖扇形仍保留业务动画 |
+| `frontend/public/earth/js/compute-centers.js` | 算力中心 icon 和交互 | 已通过 Interactable 接入统一 Points、overlay、glow 和 picking |
+| `frontend/public/earth/js/cables.js` | 登陆点 icon 和海缆线 | 登陆点当前使用专用 `THREE.Sprite` 黄色球，不再走 Interactable；海缆线仍独立渲染 |
+| `frontend/public/earth/js/main.js` | 当前集中处理 hover、click、locked 和 info-card 入口 | 后续需要接入通用 icon picking 结果 |
+| `docs/technical/zh/earth-layer-style-reference.md` | 当前视觉参数参考 | 实现后同步默认 glow 和通用参数 |
+| `docs/technical/zh/earth-render-layer-order.md` | 当前层级参考 | 实现后同步 surface icon 层级范围 |

docs/plans/earth-news-cruise-summary-plan.md

@@ -0,0 +1,366 @@
+# Earth 新闻巡航摘要增强计划
+
+## 背景
+
+`Earth` 的新闻巡航模式目前直接消费 `/api/v1/news/earth-feed` 返回的 `items[].summary`。这个字段主要来自 RSS/Atom 的 `description`、`summary` 或 `content`，再经过 HTML 清理与长度截断。
+
+这个实现足够轻量，但在巡航展示里有三个问题：
+
+- 不是所有新闻源都会提供摘要，部分源只返回标题和链接。
+- 聚合源的摘要质量不稳定，可能只是重复标题、来源署名或片段文本。
+- 巡航模式需要更稳定的“态势说明”，否则新闻卡片的 `SUMMARY` 区域会显得空或信息密度不足。
+
+目标不是把所有新闻都交给大模型，而是建立一个分层摘要管线：能用新闻源自带内容时零成本处理，需要增强时优先用本地模型，云端 LLM 只作为可控兜底。
+
+## 当前相关实现
+
+| 文件 | 作用 |
+| --- | --- |
+| `backend/app/services/earth_news.py` | 拉取 RSS/Atom 新闻源、解析标题/摘要、按区域聚合并返回 Earth 新闻 payload |
+| `backend/app/api/v1/news.py` | 暴露 `/api/v1/news/earth-feed` |
+| `frontend/public/earth/js/news.js` | 拉取新闻 payload 并渲染媒体面板新闻列表 |
+| `frontend/public/earth/js/news-cruise-adapter.js` | 将新闻条目映射为巡航事件，并把 `summary` 传给信息卡 |
+| `frontend/public/earth/js/info-card.js` | 展示新闻巡航卡片中的 `SUMMARY` |
+
+当前摘要生成逻辑集中在 `earth_news.py`：
+
+```python
+summary = _extract_item_text(node, "description", "content")
+clean_summary = _truncate(_strip_html(summary), 180)
+```
+
+这意味着后端还没有区分“摘要来自哪里”“质量是否足够”“是否需要异步增强”。
+
+## 总体方案
+
+采用四级摘要来源：
+
+| 优先级 | 来源 | 成本 | 适用情况 | 风险 |
+| --- | --- | --- | --- | --- |
+| 1 | 新闻源自带 summary / description | 低 | RSS/Atom 已提供可读摘要 | 字段可能为空或重复标题 |
+| 2 | 本地规则提取 | 低 | 有正文片段但没有可靠摘要 | 只能抽取，不能真正概括 |
+| 3 | 本地 Gemma/Ollama 摘要 | 中 | 巡航会展示且前两级质量不足 | 本地模型质量与机器性能相关 |
+| 4 | 云端 LLM 兜底 | 高 | 用户手动增强、重点新闻、失败补偿 | 成本与网络依赖 |
+
+推荐默认策略：
+
+```text
+provider summary -> extractive summary -> cached local model summary -> async local model summary -> optional cloud LLM
+```
+
+巡航 UI 永远先展示已有摘要，不等待模型调用。模型摘要在后台补齐，写入缓存后下一轮巡航或刷新时使用。
+
+## 数据结构
+
+后端应把原来的 `summary: str` 升级为可追踪的摘要元信息，同时为了兼容前端保留顶层 `summary` 字段。
+
+建议新增结构：
+
+```json
+{
+  "summary": "短摘要文本",
+  "summary_meta": {
+    "source": "provider",
+    "quality": "good",
+    "generated_at": "2026-04-29T00:00:00Z",
+    "content_hash": "sha256:...",
+    "model": null,
+    "language": "zh-CN"
+  }
+}
+```
+
+字段说明：
+
+| 字段 | 可选值 | 说明 |
+| --- | --- | --- |
+| `source` | `provider` / `extractive` / `local_llm` / `cloud_llm` / `fallback` | 摘要来源 |
+| `quality` | `good` / `partial` / `poor` | 后端对摘要可用性的判断 |
+| `generated_at` | ISO 时间 | 模型或规则生成时间 |
+| `content_hash` | SHA-256 | 用于缓存命中和判断内容变化 |
+| `model` | 字符串或 `null` | 例如 `gemma3:4b` |
+| `language` | 语言代码 | 默认 `zh-CN`，也可跟随新闻语言 |
+
+前端第一阶段不需要显示 `summary_meta`，但可以用于后续调试面板或质量标记。
+
+## 后端设计
+
+### NewsSummaryService
+
+新增 `backend/app/services/news_summary.py`，提供统一入口：
+
+```python
+async def resolve_news_summary(item: ParsedNewsItem, *, mode: str) -> NewsSummaryResult:
+    ...
+```
+
+核心职责：
+
+- 标准化新闻输入：标题、URL、来源、发布时间、摘要片段、正文片段。
+- 判断 provider summary 是否可用。
+- 生成本地规则摘要。
+- 查询模型摘要缓存。
+- 在允许时调用本地 Ollama/Gemma。
+- 在增强模式或手动触发时调用云端 LLM。
+- 返回摘要文本与 `summary_meta`。
+
+### 摘要质量判断
+
+第一版可以用轻量规则：
+
+- 少于 30 个字符：`poor`
+- 与标题高度重复：`partial`
+- 包含明显来源署名或聚合噪声：`partial`
+- 60-180 个字符且不重复标题：`good`
+
+伪代码：
+
+```python
+def score_summary(title: str, summary: str) -> SummaryQuality:
+    if len(summary.strip()) < 30:
+        return "poor"
+    if normalized_overlap(title, summary) > 0.75:
+        return "partial"
+    if looks_like_source_attribution(summary):
+        return "partial"
+    return "good"
+```
+
+### 本地规则摘要
+
+如果新闻源没有摘要，但有 `content`、`description`、`snippet` 或正文片段：
+
+- 清理 HTML。
+- 去掉标题重复内容。
+- 去掉来源署名、发布时间、图片说明。
+- 优先取前 1-2 个完整句子。
+- 控制在 80-140 个中文字符或 40-80 个英文词。
+
+### 本地 Gemma/Ollama Provider
+
+不要在业务里写死 Gemma，抽象为 `LocalLLMSummaryProvider`，默认可以指向 Ollama：
+
+```env
+NEWS_SUMMARY_PROVIDER=ollama
+OLLAMA_BASE_URL=http://localhost:11434
+NEWS_SUMMARY_MODEL=gemma3:4b
+NEWS_SUMMARY_TIMEOUT_SECONDS=20
+NEWS_SUMMARY_MAX_INPUT_CHARS=5000
+NEWS_SUMMARY_MAX_PER_HOUR=60
+```
+
+Ollama 请求示例：
+
+```http
+POST /api/generate
+Content-Type: application/json
+
+{
+  "model": "gemma3:4b",
+  "prompt": "...",
+  "stream": false,
+  "options": {
+    "temperature": 0.2,
+    "num_predict": 180
+  }
+}
+```
+
+摘要 prompt 要强调“只基于原文”，避免模型补事实：
+
+```text
+你是新闻摘要器。只根据输入新闻内容生成摘要，不要添加原文没有的信息。
+输出中文，1-2 句话，80-140 字。
+如果原文信息不足，只概括已知事实，不要推测。
+
+标题：{title}
+来源：{source}
+发布时间：{published_at}
+正文或片段：
+{content}
+```
+
+### 缓存
+
+需要缓存模型摘要，避免重复花时间和费用。
+
+缓存 key：
+
+```text
+sha256(url + title + published_at + normalized_content)
+```
+
+建议新增表或复用系统设置缓存。若要可查询与清理，推荐独立表：
+
+```text
+news_summary_cache
+- id
+- cache_key
+- url
+- title
+- content_hash
+- summary
+- source
+- quality
+- provider
+- model
+- generated_at
+- expires_at
+- failure_count
+- last_error
+```
+
+缓存策略：
+
+- 同一 `cache_key` 命中后直接返回。
+- `provider` / `extractive` 可以短期缓存。
+- `local_llm` / `cloud_llm` 可以长缓存，内容 hash 变化才重算。
+- LLM 失败后记录 `failure_count`，短时间内不重复调用。
+
+## 前端与巡航行为
+
+前端第一阶段只需要继续使用 `item.summary`，不阻塞现有逻辑。
+
+后续可选增强：
+
+- `news.js` 在渲染新闻列表时，如果 `summary_meta.quality === "poor"`，可以用更紧凑的标题卡样式。
+- `news-cruise-adapter.js` 选择巡航项时，可以优先选择 `summary_meta.quality !== "poor"` 的新闻。
+- `info-card.js` 不显示“AI 生成中”这类文案，避免把系统内部状态暴露给用户。
+
+如果后端异步生成完成，可以通过下一次 `/api/v1/news/earth-feed` 刷新自然更新。第一版不需要 WebSocket。
+
+## 调用策略
+
+默认使用“省钱模式”：
+
+- 只处理本次 payload 中即将进入巡航队列的前 N 条。
+- `provider` 或 `extractive` 达到 `good` 时不调用模型。
+- 本地模型失败时不影响新闻 payload。
+- 云端 LLM 默认关闭，只允许手动增强或后台配置开启。
+
+推荐限制：
+
+| 配置 | 默认值 | 说明 |
+| --- | --- | --- |
+| `NEWS_SUMMARY_MODE` | `economy` | `off` / `economy` / `enhanced` / `manual` |
+| `NEWS_SUMMARY_CRUISE_PREFETCH_LIMIT` | `10` | 每次新闻 payload 预热多少条巡航摘要 |
+| `NEWS_SUMMARY_MAX_PER_HOUR` | `60` | 本地模型每小时最多处理数量 |
+| `NEWS_SUMMARY_CLOUD_MAX_PER_DAY` | `20` | 云端 LLM 每天最多处理数量 |
+| `NEWS_SUMMARY_TIMEOUT_SECONDS` | `20` | 单条模型摘要超时 |
+| `NEWS_SUMMARY_MAX_INPUT_CHARS` | `5000` | 输入截断上限 |
+
+## Gemma 本地部署建议
+
+Gemma 适合作为“本地省钱层”，但不应成为强绑定依赖。建议通过 Ollama 接入，未来可切换 Qwen、Llama 或其他本地模型。
+
+开发环境：
+
+```bash
+ollama pull gemma3:4b
+ollama serve
+```
+
+集成原则：
+
+- 后端只依赖 Ollama HTTP API，不直接依赖 Gemma SDK。
+- 模型名称来自配置，不写死在代码里。
+- 健康检查访问 `/api/tags` 或执行一条极短测试 prompt。
+- 如果 Ollama 不可用，摘要管线自动退回 `provider` / `extractive`。
+
+中文新闻较多时，需要单独评估 Gemma 与 Qwen 系本地模型的中文摘要质量。不要只看单条效果，至少抽样 50 条新闻比较：
+
+- 事实准确性
+- 中文自然度
+- 长度稳定性
+- 延迟
+- 是否会补充原文没有的信息
+
+## 云端 LLM 兜底
+
+云端 LLM 不作为默认路径，只用于：
+
+- 用户点击“增强摘要”。
+- 管理员开启增强模式。
+- 本地模型连续失败且新闻进入重点巡航队列。
+
+云端结果同样写入 `news_summary_cache`，并受每日限额控制。
+
+## 分阶段实施
+
+### 第一阶段：零成本摘要质量增强
+
+- 在 `earth_news.py` 中引入 `summary_meta`。
+- 增加 provider summary 质量判断。
+- 增加本地规则摘要兜底。
+- `/api/v1/news/earth-feed` 保持兼容，继续返回顶层 `summary`。
+- 前端无需大改。
+
+验收标准：
+
+- 没有摘要的新闻也能尽量得到短摘要。
+- `summary_meta.source` 和 `summary_meta.quality` 可用于调试。
+- 现有新闻面板和巡航模式不破坏。
+
+### 第二阶段：本地 Gemma/Ollama 摘要
+
+- 新增 `LocalLLMSummaryProvider`。
+- 接入 Ollama `/api/generate`。
+- 添加超时、输入截断、错误退避。
+- 增加模型摘要缓存。
+- 巡航 payload 后台预热前 N 条摘要。
+
+验收标准：
+
+- Ollama 可用时，低质量摘要能被本地模型增强。
+- Ollama 不可用时，新闻接口仍然正常返回。
+- 同一新闻不会重复调用模型。
+
+### 第三阶段：设置与可观测性
+
+- 在设置中增加新闻摘要模式：
+  - `关闭`
+  - `省钱模式`
+  - `增强模式`
+  - `仅手动`
+- 增加本地模型连通性检查。
+- 暴露缓存命中率、模型调用次数、失败次数。
+- 日志记录摘要来源和失败原因。
+
+验收标准：
+
+- 用户可以不改环境变量就知道本地摘要服务是否可用。
+- 管理员能看出成本和失败情况。
+
+### 第四阶段：云端 LLM 兜底
+
+- 接入现有 AI Provider 或新增 cloud summary provider。
+- 增加每日限额与手动增强入口。
+- 对云端生成结果落缓存。
+
+验收标准：
+
+- 云端调用可控、可关闭、可限流。
+- 云端失败不影响巡航。
+
+## 风险与防护
+
+| 风险 | 防护 |
+| --- | --- |
+| 本地模型生成不存在的事实 | prompt 明确禁止扩写；摘要只作为原文概括；保留来源链接 |
+| 本地模型慢导致新闻接口卡住 | 模型摘要异步化；接口先返回已有摘要 |
+| 成本失控 | 默认不启用云端；按小时/天限流；缓存命中优先 |
+| 摘要语言不一致 | 配置目标语言，默认 `zh-CN` |
+| 新闻源正文不足 | 只概括标题和片段，不强行扩写 |
+| 模型服务不可用 | 自动回退，不影响巡航主流程 |
+
+## 推荐优先级
+
+先做第一阶段和第二阶段的最小闭环：
+
+1. `summary_meta` + 质量判断。
+2. 本地规则摘要。
+3. Ollama provider。
+4. 缓存。
+5. 巡航前 N 条异步预热。
+
+云端 LLM 和设置页可以后置。这样能先验证“摘要缺失比例、本地模型质量、实际延迟”三个关键问题，再决定是否投入更重的 UI 与云端增强。

docs/plans/earth-predicted-orbit-plan.md

@@ -96,3 +96,11 @@ GEO 轨道点数高，采样率需要按轨道类型分层。
 2. 解锁后轨道立即清除
 3. 不同轨道类型下点数可控
 4. 页面切换回来不会闪出旧轨道残留
+
+## Satellite Footprint Follow-Up Items
+
+从技术文档迁出的 footprint 后续项，作为卫星覆盖能力的计划 backlog：
+
+1. 为 `iridium-next` 新建独立 footprint adapter。
+2. 在 UI 上补一个只读提示，让用户知道当前卫星是否支持 footprint。
+3. 如果未来拿到 GEO beam contour / operator metadata，再为 GEO 开 operator-specific footprint。

docs/plans/earth-renderer-architecture-separation-plan.md

@@ -25,6 +25,49 @@
 - 状态和渲染更新散落在多个模块
 - 后续再加新图层时容易复制旧逻辑
 
+## Current High-Frequency Risks
+
+### 1. Visual state and business state drift apart
+
+Earth 里最常见的 bug 不是“没渲染”，而是状态没有一起收口：
+
+- 图层关了，tooltip 还在
+- 锁定对象隐藏了，info card 还在
+- legend 没跟图层切换
+- loading 已结束，但按钮还像没开
+
+后续架构治理需要把这类同步责任从临时 UI patch 转为统一状态流。
+
+### 2. HUD layout fixes skip structure analysis
+
+Earth HUD 历史上反复出现：
+
+- 面板只剩一条缝
+- markdown 被裁掉
+- tabs / iframe 被 `overflow: hidden` 吃掉
+
+这类问题应纳入布局治理计划，而不是散落在单个功能改动里临时修。
+
+### 3. Transitional paths keep accumulating
+
+Earth 已经经历过多轮 HUD、toolbar、media panel 重构，容易留下：
+
+- 旧 helper
+- 旧 class
+- 旧 fallback 逻辑
+- 已废弃变体
+
+架构分离阶段需要把 cleanup pass 作为计划项，而不是让技术上下文承担提醒职责。
+
+### 4. Cruise logic and business events couple too deeply
+
+巡航相关风险是通用巡航层继续混入业务事件细节，导致 BGP、新闻、卫星、海缆各自复制一套状态机。
+
+架构目标应保持：
+
+- 通用巡航层管理目标、队列、focus、停留、隐藏和切换
+- 业务模块只提供队列、坐标、卡片内容和高亮副作用
+
 ## Target Architecture
 
 Earth 对每类对象都尽量拆成三层：

docs/plans/earth-vessel-ais-aggregation-plan.md

@@ -0,0 +1,478 @@
+# AIS 多源采集、冲突记录与聚合接口计划
+
+**状态**：v0-v3 已实现，v3.1-v3.4 为 v4/v5 前置稳定化任务，v4 / v5 已落最小可用子集
+**创建日期**：2026-04-30  
+**核心原则**：采集器只写原始观测；去重、合并、冲突解释放在聚合接口中完成
+
+## 已确认决策
+
+| 项目 | 决策 |
+|-----|------|
+| AISStream 接入方式 | 单独实现 WebSocket 采集器，不塞进现有 BarentsWatch HTTP collector |
+| 采集器职责 | 连接上游、标准化字段、写入原始观测，不直接决定最终展示值 |
+| 去重合并位置 | 放在聚合服务和聚合 API 中，而不是散落在每个 collector 的保存逻辑里 |
+| 冲突处理 | 先记录冲突事实和当前选择原因，后续再开放用户规则配置 |
+| 默认可信度 | 同类 AIS 数据源优先按 `delivery_mode` 评估：`realtime_stream` 优于 `batch_stream`，再优于 `polling` 和 `snapshot` |
+| 过期保护 | 实时流源断流超过 freshness 窗口后，不能仅凭“实时源”身份压过更新的轮询数据 |
+| 源健康状态 | 聚合时必须参考采集器健康状态，不能只看配置中的理论优先级 |
+| 媒体富化 | 船只图片等媒体信息不进入 AIS 实时聚合主链路，后续单独做 enrichment |
+| v4/v5 顺序 | 在聚合完整性、AISStream 实时链路、采集状态语义和基础身份信息显示修好之前，不进入策略配置和 enrichment UI |
+
+## 背景
+
+当前 AIS 链路以 BarentsWatch 为主。它是 HTTP polling 模式，覆盖挪威附近海域，适合作为稳定的免费起点，但不适合承担全球实时船只数据的全部职责。后续接入 AISStream 后，会出现同一个 MMSI 被多个来源同时上报的情况：
+
+- 位置、航速、航向可能在多个来源之间存在秒级差异。
+- 船名、IMO、呼号、船型、尺寸等静态字段可能不完整，甚至互相冲突。
+- WebSocket 或其他实时流通常更接近实时，但也可能断流或批量延迟。
+- 如果每个 collector 自己做去重合并，规则会分散、不可审计，也很难让用户后续配置“某个字段信任哪个来源”。
+
+因此第一阶段不应让采集器直接覆盖最终船只表。更稳的方式是先保留观测事实，再由聚合接口统一给出当前展示视图。
+
+## 目标架构
+
+```mermaid
+flowchart LR
+    A[BarentsWatch HTTP collector] --> D[AIS raw observations]
+    B[AISStream WebSocket collector] --> D
+    C[Custom mapped vessel_ais sources] --> D
+    D --> E[AIS aggregation service]
+    E --> F[Conflict records]
+    E --> G[GeoJSON vessels API]
+    E --> H[Vessel detail API]
+    I[Aggregation strategy config] --> E
+```
+
+### 原始观测层
+
+原始观测层保存每个来源看到的事实。建议模型包含：
+
+| 字段 | 用途 |
+|-----|------|
+| `target_schema` | 例如 `vessel_ais` |
+| `source` | 例如 `barentswatch_vessels`、`aisstream_vessels` |
+| `entity_key` | AIS 使用 MMSI |
+| `delivery_mode` | `realtime_stream`、`batch_stream`、`polling`、`snapshot` |
+| `transport` | `websocket`、`sse`、`http`、`file` 等 |
+| `observed_at` | 上游数据时间，优先使用 AIS 消息时间 |
+| `collected_at` | 本系统接收或采集时间 |
+| `source_message_id` | 上游消息 ID 或可推导 ID，没有则为空 |
+| `observation_hash` | 幂等去重指纹，用于防止同一来源重复写入同一条观测 |
+| `normalized_payload` | 标准化后的 AIS JSON |
+| `raw_payload` | 可选，保存原始或裁剪后的上游记录 |
+| `quality_flags` | 观测级质量标记，例如 `stale`、`position_jump`、`future_timestamp` |
+
+`delivery_mode` 和 `transport` 不应混为一谈。WebSocket 是传输方式；streaming 是交付模式。聚合可信度主要看 `delivery_mode`，`transport` 只作为辅助信息。
+
+原始观测层需要做存储级幂等去重，但这里的去重不是业务合并。推荐使用 `source + entity_key + message_type + observed_at + payload_hash` 或上游稳定消息 ID 作为唯一约束，避免 WebSocket 重连、HTTP 重试或批量回放导致同一事实重复入库。
+
+### 源健康状态
+
+每个采集器应维护独立的健康状态，供聚合服务读取：
+
+| 字段 | 用途 |
+|-----|------|
+| `source` | 采集器标识 |
+| `connection_state` | `connected`、`reconnecting`、`disconnected`、`disabled` 等 |
+| `last_seen_at` | 最近收到上游消息或响应的时间 |
+| `last_success_at` | 最近成功写入观测的时间 |
+| `last_error` | 最近错误摘要 |
+| `message_rate` | 最近窗口内的消息速率 |
+| `lag_seconds` | 上游观测时间与本系统接收时间的延迟 |
+
+聚合优先级不能只看 `source_priority`。例如 `aisstream_vessels` 默认优先于 `barentswatch_vessels`，但如果它处于 `disconnected` 或 `lag_seconds` 超过 freshness 窗口，则动态字段应回退到更新的可用来源。
+
+### 身份键边界
+
+v1 可以继续用 MMSI 作为 `entity_key`，因为它是 AIS 动态消息里最稳定、最容易获得的主键。但文档和模型都要为后续扩展留出口：MMSI 可能复用、填错或缺少静态信息，后续身份解析应结合 `mmsi + imo + callsign + name + dimensions` 判断是否需要拆分或合并实体。
+
+### 冲突记录层
+
+聚合服务发现同一个实体、同一个字段存在多个非空不同值时，写入冲突记录。冲突记录不代表错误，只代表“有多个可用候选值”。
+
+```json
+{
+  "target_schema": "vessel_ais",
+  "entity_key": "257123000",
+  "field": "name",
+  "candidates": {
+    "barentswatch_vessels": "OSLO TRADER",
+    "aisstream_vessels": "OSLO TRADER II"
+  },
+  "selected_source": "aisstream_vessels",
+  "selected_value": "OSLO TRADER II",
+  "selected_reason": "delivery_mode_priority",
+  "resolved_by": "system",
+  "status": "open"
+}
+```
+
+第一阶段只需要记录冲突和当前选择原因，不需要做人工逐条确认。后续 UI 的目标也不是让用户处理每条冲突，而是把冲突沉淀成字段级规则。
+
+## 聚合规则
+
+### 字段分类
+
+| 类型 | 字段 | 默认策略 |
+|-----|------|----------|
+| 动态位置 | `lat`、`lon`、`sog`、`cog`、`heading`、`nav_status` | 优先最新 `observed_at`，同时间再按来源优先级 |
+| 静态身份 | `name`、`callsign`、`imo`、`flag` | 非空优先，再按字段策略或来源优先级 |
+| 静态规格 | `vessel_type`、`vessel_type_name`、`length`、`width`、`draught` | 非空优先；冲突时记录候选值 |
+| 轨迹点 | `track_points` | 按时间线合并；同一时间窗口内相近点去重；保留点级 `source` |
+| 元信息 | `field_sources`、`conflict_count`、`selected_reasons`、`quality_flags` | 聚合接口生成，便于调试和后续 UI 展示 |
+
+### 默认优先级
+
+默认优先级应使用两个维度：
+
+```yaml
+delivery_mode_priority:
+  - realtime_stream
+  - batch_stream
+  - polling
+  - snapshot
+
+transport_priority:
+  - websocket
+  - sse
+  - http
+  - file
+```
+
+`delivery_mode_priority` 是主判断。比如 AISStream 如果提供实时推送，应标记为 `realtime_stream + websocket`；BarentsWatch 当前是 `polling + http`。
+
+### 断流保护
+
+实时流不能永久凭身份占优。聚合时需要 freshness 窗口：
+
+```yaml
+freshness:
+  realtime_stream_seconds: 900
+  polling_seconds: 3600
+```
+
+如果 `aisstream_vessels` 最近 15 分钟没有该 MMSI 的新观测，而 BarentsWatch 轮询源有更新位置，则位置类字段应采用 BarentsWatch 的更新观测，并记录选择原因 `newest_observation` 或 `freshness_fallback`。
+
+### 异常位置保护
+
+多源 AIS 接入后，聚合服务必须过滤或降权明显异常的位置观测：
+
+- 经纬度必须在合法范围内。
+- `observed_at` 不能明显来自未来。
+- 同一 MMSI 短时间内跨越不合理距离时，标记 `position_jump`，默认不直接采用该点。
+- 当异常点来自当前优先源时，应记录 `selected_reason = anomaly_rejected`，再回退到其他可用来源。
+
+异常保护不应静默丢弃事实。原始观测仍应保留，聚合结果通过 `quality_flags` 和冲突记录解释为什么没有采用它。
+
+### 轨迹聚合
+
+轨迹接口不能简单拼接所有来源，否则前端会出现折返、抖动和重复点。默认规则：
+
+- 以 `observed_at` 排序，生成统一时间线。
+- 同一来源的完全重复点通过 `observation_hash` 去重。
+- 多来源在短时间窗口内上报的相近位置视为同一轨迹点，优先选择 freshness 和 source priority 更高的一条。
+- 每个轨迹点保留 `source`、`selected_reason` 和必要的 `quality_flags`。
+- 对被判定为 `position_jump` 的点，默认不进入展示轨迹，但可通过调试参数查看。
+
+## 聚合接口
+
+现有展示接口应逐步改为消费聚合服务，而不是自己直接拼 `VesselPosition + VesselStatic`。
+
+```text
+GET /api/v1/visualization/geo/vessels
+GET /api/v1/visualization/vessels/{mmsi}
+GET /api/v1/visualization/vessels/{mmsi}/track
+GET /api/v1/visualization/vessels/{mmsi}/conflicts
+```
+
+GeoJSON properties 建议增加：
+
+```json
+{
+  "mmsi": 257123000,
+  "name": "OSLO TRADER",
+  "lat": 59.91,
+  "lon": 10.73,
+  "received_at": "2026-04-30T10:00:00Z",
+  "field_sources": {
+    "name": "aisstream_vessels",
+    "lat": "aisstream_vessels",
+    "lon": "aisstream_vessels",
+    "vessel_type": "barentswatch_vessels"
+  },
+  "selected_reasons": {
+    "name": "delivery_mode_priority",
+    "lat": "newest_observation",
+    "vessel_type": "non_empty_priority"
+  },
+  "quality_flags": [],
+  "conflict_count": 2
+}
+```
+
+## 开放配置计划
+
+### Phase 1 — 内置默认策略和只读解释
+
+- 实现后端默认策略。
+- 聚合接口返回 `field_sources`、`selected_reasons`、`conflict_count`。
+- 冲突记录可查询，但不允许用户修改。
+- 保持现有前端船只图层接口形状基本兼容，新增字段只作为调试和后续 UI 输入。
+
+### Phase 2 — 系统设置中的 JSON/YAML 策略配置
+
+新增系统设置项，例如：
+
+```yaml
+collector_aggregation:
+  vessel_ais:
+    source_priority:
+      - aisstream_vessels
+      - barentswatch_vessels
+    field_rules:
+      name:
+        mode: source_priority
+      vessel_type:
+        mode: source_priority
+        source_priority:
+          - barentswatch_vessels
+          - aisstream_vessels
+      lat:
+        mode: newest
+      lon:
+        mode: newest
+```
+
+配置校验要求：
+
+- 未知 source 只警告，不阻断保存，便于先配置后启用。
+- 未知 field 必须拒绝，避免拼写错误悄悄失效。
+- 动态位置字段默认不允许被固定来源永久锁死，除非显式开启高级选项。
+- 空值不覆盖非空值是全局保护，不建议开放关闭。
+
+### Phase 3 — 冲突治理 UI
+
+基于冲突记录提供页面或 drawer：
+
+- 查看某个 MMSI 的冲突字段。
+- 查看每个字段的候选来源和值。
+- 查看当前选择原因。
+- 将一次人工选择保存成字段规则，而不是只处理单条冲突。
+- 支持恢复默认策略。
+
+## AISStream 采集器计划
+
+AISStream 采集器单独实现，建议命名为 `aisstream_vessels`。它的职责是：
+
+- 维护 WebSocket 连接、订阅范围和重连。
+- 将上游 AIS 消息标准化为 `vessel_ais` payload。
+- 标记 `delivery_mode = realtime_stream`，`transport = websocket`。
+- 写入原始观测层。
+- 不直接 upsert 最终展示数据。
+
+配置应放入采集器设置，而不是硬编码：
+
+```yaml
+aisstream_vessels:
+  api_key: "${AISSTREAM_API_KEY}"
+  bounding_boxes:
+    - [[-180, -90], [180, 90]]
+  message_types:
+    - PositionReport
+    - ShipStaticData
+```
+
+默认不建议直接订阅全球范围。AISStream 采集器应支持以下订阅策略：
+
+- 使用配置的固定 `bounding_boxes`。
+- 后续支持按 Earth 当前视口或关注区域动态调整订阅范围。
+- 支持限制 `message_types`，避免静态信息、位置报告和扩展消息全量涌入。
+- 断线后使用指数退避重连，并把连接状态写入源健康状态。
+- 重连后可能收到重复或回放消息，因此必须依赖原始观测层的幂等去重。
+
+### 媒体富化边界
+
+VesselFinder 等服务里的船只图片不属于 AIS 实时数据本身。图片、船籍详情、公司信息等后续应作为独立 enrichment 链路：
+
+- 通过 MMSI、IMO、船名等字段异步查询。
+- 使用独立缓存和授权配置。
+- 不阻塞 `vessel_ais` 实时观测入库。
+- 聚合接口只暴露已经缓存好的媒体引用，不在请求链路中现场抓取。
+
+## 版本拆分
+
+计划先按 v0-v3 建立基础能力，再用 v3.1-v3.4 修复当前稳定性缺口，最后进入 v4/v5：
+
+### v0 — 聚合基础设施（已实现）
+
+目标是不改变前端展示行为，先把数据底座铺好。
+
+1. 新增原始观测模型、冲突记录模型和源健康状态模型。
+2. 为现有 BarentsWatch collector 写入原始观测，同时保留现有 `vessel_position` / `vessel_static` 兼容写入。
+3. 实现存储级 `observation_hash` 幂等去重。
+4. 补基础管理命令或调试接口，用于查看某个 MMSI 的原始观测和冲突候选。
+
+### v1 — 聚合读接口（已实现）
+
+目标是让展示接口开始消费聚合结果，但前端形状保持兼容。
+
+1. 实现 AIS 聚合服务，先兼容读取现有表，再逐步切换到原始观测层。
+2. 将 `/geo/vessels` 和 `/vessels/{mmsi}` 改为走聚合服务。
+3. 将 `/vessels/{mmsi}/track` 改为走轨迹聚合逻辑。
+4. 返回 `field_sources`、`selected_reasons`、`quality_flags`、`conflict_count`。
+5. 加入 freshness fallback 和异常位置保护。
+
+### v2 — AISStream WebSocket collector（已实现）
+
+目标是接入第二个真实 AIS 来源，并验证多源冲突和回退逻辑。
+
+1. 实现 `aisstream_vessels` collector。
+2. 支持 API key、订阅范围、消息类型、重连和限流配置。
+3. 将 AISStream 写入原始观测层，不直接 upsert 最终展示表。
+4. 接入源健康状态和 message rate 统计。
+5. 提供 AISStream API Key 获取教程、设置页入口和连接验证支持。
+6. 为重复消息、断流回退、WS 优先级写集成测试。
+
+### v3 — AISStream 可用性与配置体验（已实现）
+
+目标是让 AISStream 从“能采集”变成日常可观察、可调试、可配置的数据源。
+
+1. 设置页展示 AISStream 运行状态：连接状态、最近收到、最近成功、本轮消息数、延迟和最近错误。
+2. AISStream 设置页提供常用采集范围 preset，并保留自定义 Bounding Boxes JSON。
+3. 聚合结果返回 `source_summary`，展示每艘船的来源、观测数量、最新观测时间、传输模式和消息类型。
+4. 保留 `field_sources` 和 `selected_reasons`，用于解释动态字段来自实时流、静态字段来自可用非空来源。
+5. 船名标准化会读取 AISStream `MetaData.ShipName`；船型展示会从 `vessel_type_name` 和 AIS 数字 `vessel_type` 共同归一化，保证 marker 颜色、详情卡、hover 和搜索结果一致。
+6. `/geo/vessels` 不再默认限制 5000 艘；不传 `limit` 或传 `limit=0` 表示全量返回，前端默认也不再二次裁剪到 5000。
+
+### v3.1 — 聚合完整性修复（v4 前置）
+
+目标是先保证“所有已采集到的船都能显示”，BarentsWatch 不因为接入 AISStream 而被 raw observation 聚合结果遮蔽。
+
+当前风险是 `/geo/vessels` 只要 raw observation 聚合返回非空，就直接使用 raw 聚合结果，不再补读兼容层 `vessel_position + vessel_static`。如果 raw observation 中只存在 AISStream 的几百艘船，或 BarentsWatch 历史数据没有完整回填到 raw 层，最终 Earth 就会只显示 AISStream 子集。
+
+1. `/geo/vessels` 必须合并 raw observation 聚合结果和 legacy latest position 结果。
+2. raw 与 legacy 同一 MMSI 同时存在时只显示一艘，优先使用 raw 聚合结果及其 `field_sources` / `selected_reasons`。
+3. raw 中不存在的 BarentsWatch-only MMSI 必须从 `vessel_position + vessel_static` 补齐。
+4. `bbox`、`type`、`limit` 过滤必须作用在合并后的最终集合上；不传 `limit` 或 `limit=0` 仍表示全量返回。
+5. 增加诊断统计，至少能看到 raw AISStream unique MMSI、raw BarentsWatch unique MMSI、legacy unique MMSI、final merged unique MMSI 和被 legacy 补齐的数量。
+6. 为 raw 只有 AISStream 子集、legacy 有更多 BarentsWatch 船只的场景补回归测试。
+
+### v3.2 — AISStream 真实时链路（v4 前置）
+
+目标是把 AISStream 从“一次 collector 收一批消息后结束”改成真正的 WebSocket 长连接实时数据源，并把实时变化推送到 Earth。
+
+当前 `aisstream_vessels` 只在 collector `fetch()` 中连接 `wss://stream.aisstream.io/v0/stream`，默认收 `max_messages = 500` 条后结束。这不符合 WebSocket 流式数据源的运行语义，也不能保证新船、位置变化和航向变化实时出现在前端。
+
+1. 为 AISStream 增加 streaming service / long-running runner，不再依赖单次 `fetch -> transform -> save -> completed` 表达实时采集。
+2. 外部 AISStream WebSocket 保持长连接，断线后指数退避重连，并持续更新 `AISSourceHealth`。
+3. 每条或小批量 AIS 消息标准化后写入 `ais_raw_observations`，按时间或数量短周期 commit，避免长事务堆积。
+4. 将新增船只、位置变化、航向变化和静态字段补充转换成 vessel delta。
+5. 通过应用内部 `/ws` 的 `vessels` channel 广播 delta，复用 `DataBroadcaster.broadcast_custom("vessels", payload)`。
+6. Earth 前端订阅 `vessels` channel，`vessels.js` 支持按 MMSI upsert marker，而不是每次全量 reload。
+7. 船只改变航向时，前端必须更新 course bin / marker bucket，避免 marker 方向滞后。
+8. freshness 超时或 AISStream 健康异常时，动态字段可回退到 BarentsWatch 最新可用观测。
+
+### v3.3 — Streaming 采集状态语义（v4 前置）
+
+目标是让采集页面正确表达 AISStream 这类长连接数据源，不再使用一次性 REST collector 的完成型进度条。
+
+REST collector 的自然状态是 `fetch -> transform -> save -> progress 0..100 -> completed`。AISStream 的自然状态应是 `connecting -> streaming -> reconnecting -> stopped/failed`，没有固定总量，也不应在收到一批消息后显示“采集完成”。
+
+1. AISStream 采集状态使用 indeterminate / streaming 状态，而不是百分比完成进度条。
+2. 设置页运行状态卡展示连接状态、已运行时长、本轮消息数、新增观测数、unique MMSI、message rate、最近消息时间、延迟和最近错误。
+3. `phase_message` 使用“正在接收 AISStream 实时消息”“重连中”“已停止”等长连接语义。
+4. 停止、重连和配置变更要有明确操作入口；配置变化后必须安全重订阅。
+5. 后端任务状态不能因为没有 `total_records` 就长期显示 `0%` 或误判失败。
+6. WebSocket 健康状态和 collector task 状态要分离：上游短暂断线是 `reconnecting`，不是普通采集任务完成或失败。
+
+### v3.4 — 船只身份字段和名称聚合修复（v4 前置）
+
+目标是把 MMSI、IMO、callsign 这类身份编号按字符串显示，并把仍然使用 MMSI 作为船名的记录视为信息聚合未完成，而不是正常船名。
+
+1. 前端详情卡、hover、搜索结果和日志中的 `mmsi`、`imo`、`callsign` 必须作为 identifier 字段展示，禁止走 `toLocaleString()` 或数字千分位格式。
+2. GeoJSON 可增加 `mmsi_display` / `imo_display` 等字符串字段，但前端仍必须对 identifier key 做兜底格式保护。
+3. 聚合服务生成船名时，不能把 `MMSI 257123000` 当成真实 `name` 的成功结果；它只能作为 display fallback。
+4. 增加诊断查询，列出所有当前仍以 MMSI 号码或 `MMSI <number>` 作为船只名称的记录，包括：
+   - `vessel_static.name` 为空或等于 MMSI fallback 的 MMSI；
+   - raw observation 中没有任何非空 `name` / `MetaData.ShipName` / `ShipStaticData.Name` 的 MMSI；
+   - 聚合结果最终 `name` 仍为 fallback 的 MMSI；
+   - 每个 MMSI 的可用来源、最近观测时间、message types 和缺失原因。
+5. 对这些 fallback-name 船只建立待修复集合，优先通过 AISStream `ShipStaticData`、BarentsWatch 静态字段和后续 enrichment 缓存补齐。
+6. 船只详情面板需要区分“真实船名”和“显示兜底”：真实船名缺失时展示 `MMSI <id>` 可以继续作为标题，但字段来源应标注为 `fallback`，避免误以为聚合成功。
+7. 为 MMSI 千分位格式、fallback-name 诊断和名称来源解释补回归测试。
+
+### v4 — 策略配置（v0 可用）
+
+目标是开放系统级配置，但仍以安全默认值兜底。
+
+已落地的最小子集：
+
+1. 策略持久化在 `system_settings.category = 'vessel_aggregation_strategy'`，保存时自动版本递增。
+2. `app/services/vessel_aggregation_strategy.py` 暴露 `load_strategy / save_strategy / reset_strategy / validate_strategy`，并维护 `DEFAULT_STRATEGY` 兜底。
+3. 校验规则：
+   - 未知 `field_rules.<name>` → `400 unknown vessel_ais field`；
+   - 未知 mode → `400 mode must be one of ...`；
+   - 动态字段（`lat/lon/sog/cog/heading/nav_status`）使用非 `newest` mode 时必须显式 `allow_dynamic_lock=true`，否则拒绝；
+   - `freshness.realtime_stream_seconds` / `polling_seconds` 必须为非负整数；
+   - `mode=locked` 必须带非空 `locked_source`。
+4. 聚合服务 `vessel_ais_aggregation.py` 在 `_select_position_observation` 中按 `freshness` 把过期实时流降级到 stale 候选；在 `_select_static_field` 中按 `field_rules.mode = source_priority / locked / newest / non_empty` 选源。
+5. 聚合输出每条 vessel 携带 `aggregation_strategy_version`，并在 `/geo/vessels` GeoJSON properties + `/vessels/{mmsi}` 详情中暴露。
+6. API：
+   - `GET /api/v1/vessel-aggregation/strategy`
+   - `PUT /api/v1/vessel-aggregation/strategy`（校验失败 400）
+   - `DELETE /api/v1/vessel-aggregation/strategy`（恢复默认并 bump version）
+
+未做项（留给 v4 后续）：
+
+- 系统设置 UI 中的策略编辑器尚未做，目前直接调 API；
+- `transport_priority`、`quality_flags` 级别的策略尚未引入；
+- `source_priority` 中的未知 source 不强校验，留给后续 warn-only 提示。
+
+### v5 — 船舶资料 enrichment 与冲突治理（v0 可用）
+
+目标是把 AIS 实时流里不稳定或低频出现的静态信息，补成可缓存、可审计的船舶资料层，同时把冲突解释变成可操作能力。
+
+已落地的最小子集：
+
+1. 新增模型 `app/models/vessel_enrichment.py::VesselProfileEnrichment` + `VesselMediaEnrichment`：以 `mmsi` 为主键，记录 `source / payload / fetched_at / expires_at / confidence / reference_url`；通过 `Base.metadata.create_all` 在 `init_db` 中建表。
+2. 服务 `app/services/vessel_enrichment.py` 提供 `upsert_vessel_profile_enrichment` / `upsert_vessel_media_enrichment` / `get_vessel_enrichment_bundle`；读路径只读缓存，过期记录（`expires_at < now`）直接过滤为 `None`，永不联网。
+3. 聚合接口在 `/api/v1/visualization/vessels/{mmsi}` 响应中追加 `enrichment.profile` 与 `enrichment.media` 字段（含 `source / fetched_at / expires_at / confidence / reference_url`）；命中失败时返回 `null`，不阻塞 AIS 实时链路。
+4. 冲突治理 API：
+   - `POST /api/v1/vessel-aggregation/conflicts/{mmsi}/{field}/promote-to-rule` 读取最近 `AISConflictRecord.selected_source`，写入 `field_rules[field] = {mode: source_priority, source_priority: [<source>]}` 并 bump version；
+   - `DELETE` 对应路径移除该 field 的覆盖，恢复默认。
+5. 前端 Earth `info-card.js` 渲染 `船舶资料` 区块：profile.payload 标量字段平铺、媒体 `images` 数组缩略图、来源 / 更新时间 / 置信度元数据；缓存命中失败回退到 `资料缓存中`；常规字段在 `field_sources` 命中时附带来源 tag。
+
+未做项（留给 v5 后续）：
+
+- 没有真正的异步 enrichment 抓取作业；当前依赖外部脚本/管理 API 写入缓存；
+- 冲突治理 UI 还没接入设置中心，目前只暴露 API；
+- enrichment 命中状态尚未广播到 `vessels` channel，详情面板首次打开时按需请求即可。
+
+## 测试计划
+
+- 同一来源同一 `mmsi + observed_at + lat + lon` 重复记录只聚合一次。
+- 多来源同一 MMSI 的位置字段优先选择最新观测。
+- 实时流和轮询源同时间冲突时，实时流优先。
+- 实时流过期后，更新的轮询源可以接管动态字段。
+- 实时流源健康状态异常时，动态字段可以回退到更新的可用来源。
+- 静态字段不会被空值覆盖。
+- 静态字段冲突会写入冲突记录。
+- 明显异常位置不会进入默认展示轨迹，并会留下 `quality_flags`。
+- 同一时间窗口内多来源相近轨迹点只展示一个点。
+- AISStream 重连或回放导致的重复消息不会重复进入聚合结果。
+- raw observation 聚合结果和 legacy latest position 结果会按 MMSI 合并，BarentsWatch-only 船只不会因为 AISStream 子集存在而消失。
+- 不传 `limit` 或传 `limit=0` 时，`/geo/vessels` 全量返回合并后的船只集合。
+- AISStream 长连接收到新船、位置变化和航向变化后，会通过内部 `/ws` 的 `vessels` channel 推送增量。
+- AISStream streaming 状态不会显示成固定百分比完成进度条，也不会在收到一批消息后误报采集完成。
+- `mmsi`、`imo`、`callsign` 等身份编号在前端不显示千分位符。
+- 聚合结果中仍以 MMSI fallback 作为船名的记录可以被诊断查询完整列出，并带来源和缺失原因。
+- 字段级配置可以覆盖默认来源优先级。
+- 聚合接口在没有冲突表时仍可返回兼容 GeoJSON。
+
+## 相关文件
+
+- [实时船只监控系统计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-tracking-plan.md)
+- [自定义 API 数据源与 LLM 映射系统计划](/home/ray/dev/linkong/planet/docs/plans/datasource-custom-api-mapping-plan.md)
+- [BarentsWatch AIS collector](/home/ray/dev/linkong/planet/backend/app/services/collectors/vessel_ais.py)
+- [船只模型](/home/ray/dev/linkong/planet/backend/app/models/vessel.py)
+- [可视化 API](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py)

docs/plans/earth-vessel-rendering-performance-plan.md

@@ -0,0 +1,249 @@
+# Earth Vessel Rendering Performance Plan
+
+## 当前状态
+
+该计划的前端核心部分已经在 `0.44.1` 落地，但最终实现不是原文设想的 `InstancedBufferGeometry` quad，而是更稳的分桶 `THREE.Points` 方案：
+
+- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶，使用 `PointsMaterial` 批量绘制。
+- 航行船只仍是带方向的三角形，停泊或低速船只仍是圆点。
+- hover / locked 不再放大成世界尺寸 Sprite，而是在原点位叠加同尺寸单点 glow overlay。
+- picking 改为屏幕空间命中，拖动和惯性期间跳过 hover picking。
+- 普通态关闭 glow，交互态才显示 glow，降低 overdraw 并让默认地图更干净。
+
+后续如果需要全球 AIS 或更高船只密度，再评估是否从分桶 `Points` 升级到真正 instanced quad 或视口 bbox / LOD。
+
+## 背景
+
+Earth 船只图层已经形成了一套较好的视觉语言：
+
+- 航行船只使用三角形标记
+- 标记按航向旋转
+- 停泊或低速船只使用圆点
+- 不同船型使用不同颜色
+- hover / locked 状态有 glow、透明度和聚焦反馈
+- 标记带有轻微 glow / soft edge，和 Earth HUD 的观感一致
+
+当前性能问题不应通过降级成无方向、无船型语义的普通小点来解决。目标是在保留现有观赏性的前提下，把底层从“每艘船一个 Sprite 对象”优化为批量绘制和轻量交互。
+
+## 当前问题判断
+
+卫星图层能承载几万个对象，是因为它主要走 `THREE.Points` / `BufferGeometry` / instanced trail 路径。船只图层目前每艘船创建一个 `THREE.Sprite` 和独立 `SpriteMaterial`，这会带来：
+
+- draw call 随船只数量增长
+- 透明 sprite 排序和 overdraw 成本上升
+- 每帧遍历所有船只更新 opacity / scale / visible
+- pointer move 时对船只 sprite 做对象级 raycast
+- hover reset 时全量遍历 marker
+
+因此，即使免费 BarentsWatch AIS 只开放挪威周边数据，前端仍可能因为对象级 sprite、raycast 和每帧全量更新出现地球拖动卡顿。
+
+## 目标
+
+1. 保留当前船只标记的视觉质量。
+2. 保留 hover tooltip、点击详情、lock、轨迹等交互。
+3. 显著降低 draw call、每帧 JS 遍历和 pointer picking 成本。
+4. 为后续全球 AIS 或更高船只数量预留扩展空间。
+
+## 非目标
+
+- 不把船只降级为普通无方向 `Points`。
+- 不取消船型颜色、航向三角和停泊圆点。
+- 不为了短期性能直接删除 hover / click 交互。
+
+## Phase 1：交互路径止血
+
+这一阶段不改视觉，只减少 pointer move 和 hover 状态开销。
+
+### 1. 拖动和惯性期间跳过船只 picking
+
+地球拖动时用户主要关注视角变化，不需要每个 pointer move 都命中船只。
+
+处理方式：
+
+- `isDragging === true` 时跳过船只 hover picking。
+- 惯性旋转期间也跳过船只 hover picking。
+- 拖动结束后再恢复 hover 检测。
+
+### 2. vessel hover picking 节流
+
+对船只 hover 命中增加节流，例如 `80ms ~ 120ms` 一次。鼠标高速移动时复用上一次 hover 状态，不在每个 pointer event 上都做 raycast。
+
+### 3. hover reset 从全量遍历改为增量更新
+
+当前 `resetTransientVesselStates()` 会遍历所有船只。改为记录：
+
+- `hoveredVessel`
+- `lockedObject`
+
+当 hover 目标变化时，只更新旧 hover 和新 hover。
+
+### 4. 点击路径只在 click 时做一次精确 picking
+
+点击仍保留精确命中，但只在 click 事件里执行，不参与拖动和高频 pointer move。
+
+## Phase 2：每帧更新减负
+
+这一阶段仍保留 `Sprite` 外观，但减少每帧对全部 marker 的写操作。
+
+### 1. `updateVesselVisualState()` 增量化
+
+当前每帧都会遍历船只并写：
+
+- `marker.material.opacity`
+- `marker.scale`
+- `marker.visible`
+
+优化方向：
+
+- 图层关闭时直接 return。
+- 没有船只时直接 return。
+- 只有以下状态变化时才更新 marker：
+  - show/hide 变化
+  - hover 变化
+  - locked 变化
+  - camera zoom / distance scale 变化超过阈值
+  - focus dim 状态变化
+
+### 2. 缓存 distance scale
+
+`getDistanceScale(camera)` 可以按 camera distance 或 zoom 阈值缓存。缩放没有明显变化时，不必每帧重设所有船只 scale。
+
+### 3. 降低透明 overdraw
+
+在不破坏视觉的前提下微调：
+
+- marker 基础尺寸
+- glow blur 半径
+- 最大 size stabilization
+
+目标是减少屏幕空间重叠面积，而不是改变符号设计。
+
+## Phase 3：保留视觉的批量渲染（已落地为分桶 Points）
+
+原设想是把每艘船的视觉从 `THREE.Sprite` 迁移为 instanced sprite batch。实际落地时选择了更稳的分桶 `THREE.Points`：
+
+- 不依赖自定义 shader。
+- 不依赖 `Points` 自带 raycaster。
+- 用 canvas 纹理保留三角、圆点、船型颜色和航向。
+- 用 hover / locked 单点 overlay 保留交互 glow。
+
+如果未来全球 AIS 导致分桶 `Points` 仍不够，再升级到 instanced quad。
+
+### 1. 原候选方案：instanced quad
+
+每艘船仍然显示为带贴图/软边的 billboard，但底层使用：
+
+- `THREE.InstancedBufferGeometry`
+- 每类船只一个或少量 material
+- per-instance attributes
+
+可按形状和船型拆 batch：
+
+- moving cargo
+- moving tanker
+- moving passenger
+- moving fishing
+- moving military
+- moving other
+- anchored / slow dot
+
+这样 draw call 从“每艘船一个”变为“每类船只一个”。
+
+### 2. 原候选方案：per-instance attributes
+
+每个 instance 存：
+
+- position
+- color
+- rotation
+- scale
+- opacity
+- state
+- mmsi / data index
+
+hover、locked、dimmed 可通过更新少量 instance attribute 实现，不再逐个修改 material。
+
+### 3. 当前落地方案：分桶 `THREE.Points`
+
+当前实现按以下方式复刻视觉：
+
+- moving 船只按 `VESSEL_COURSE_BINS` 做航向分桶。
+- anchored / slow 船只使用圆点分桶。
+- 每个分桶生成一组 `THREE.Points`，共享 `PointsMaterial` 和 canvas 点纹理。
+- `VESSEL_CONFIG.colors` 仍通过 vertex colors 表示船型颜色。
+- hover / locked 在原位置叠加同尺寸单点 overlay，普通态不带 glow，交互态才带 glow。
+
+这样 draw call 从“每艘船一个”变为“每个形状 / 航向分桶一组”，同时避免自定义 shader 的兼容风险。
+
+### 4. 复刻当前视觉
+
+视觉上继续使用当前 canvas texture 或等效 shader：
+
+- moving 使用三角形纹理
+- anchored 使用圆点纹理
+- 保留 soft glow
+- 保留航向 rotation
+- 保留 hover / locked 放大
+
+因此用户看到的效果应与当前船只图层基本一致。
+
+## Phase 4：picking 改造
+
+批量渲染后不再适合对所有 sprite object 做 `raycaster.intersectObjects()`。
+
+### 1. 屏幕空间 picking
+
+参考卫星 picking：
+
+1. 过滤背面船只。
+2. 将候选船只世界坐标投影到屏幕。
+3. 用鼠标位置计算距离。
+4. 取距离最近且小于半径阈值的船只。
+
+### 2. 可选空间索引
+
+如果后续船只数量明显上升，可增加轻量空间索引：
+
+- 经纬度网格 bucket
+- 屏幕空间 bucket
+- viewport bbox 过滤
+
+第一阶段不必引入复杂索引。
+
+## Phase 5：数据层和 LOD
+
+当接入全球 AIS 或船只数量显著增加时，再做数据层优化。
+
+### 1. 请求视口范围
+
+前端请求 `/api/v1/visualization/geo/vessels` 时带上当前视口 `bbox`，减少无关船只。
+
+### 2. 后端排序策略
+
+从单纯 `received_at desc` 改为综合排序：
+
+- 数据新鲜度
+- 船型优先级
+- 当前视口相关性
+- 是否正在航行
+
+### 3. 远景聚合
+
+远景可显示聚合或 top N，近景展开单船。
+
+## 验收指标
+
+1. 船只视觉效果保持当前质量：三角、圆点、颜色、航向、hover、lock 都保留。
+2. 开启船只图层后拖动地球不应明显掉帧。
+3. pointer move 不应因为船只 hover 导致卡顿。
+4. 船只数量达到 `1000` 级别时仍可顺畅旋转地球。
+5. `renderer.info.render.calls` 相比 Sprite 版本显著下降。
+6. hover / click 命中体验不低于当前版本。
+
+## 建议落地顺序
+
+1. 先做 Phase 1，快速恢复地球拖动手感。
+2. 再做 Phase 2，减少每帧 JS 写操作。
+3. Phase 3 和 Phase 4 已按分桶 `THREE.Points` + 屏幕空间 picking 落地。
+4. Phase 5 等全球船只数据或数量压力出现后再推进。
+5. 如果分桶 `THREE.Points` 达到瓶颈，再评估 instanced quad。

docs/plans/earth-vessel-tracking-plan.md

@@ -0,0 +1,281 @@
+# 实时船只监控系统 — 实施计划
+
+**状态**：规划中
+**创建日期**：2026-04-27
+**优先数据源**：BarentsWatch AIS（免费但需要 OAuth client credentials）→ AISHub / MarineTraffic（TODO，付费）
+
+## 已确认决策
+
+| 项目 | 决策 |
+|-----|------|
+| 数据源 | BarentsWatch 先行；AISHub / MarineTraffic TODO |
+| 船只规模 | BarentsWatch 阶段全部显示；全球数据接入后按需加船型过滤（默认 Cargo + Tanker + Passenger） |
+| 更新频率 | 准实时：前端 5 分钟轮询，后端 Collector 每分钟拉取写库 |
+| 历史轨迹 | 保留（`vessel_position` 表保留 24h，后期按需扩展） |
+| 推送方式 | 前端展示仍可先用 HTTP 拉取聚合结果；AISStream 等实时源应单独实现 WebSocket 采集器 |
+
+---
+
+## 一、技术背景
+
+船只通过 AIS（自动识别系统）每 2–10 秒广播位置、航速、航向、目的地等信息。全球约 50 万艘持证船只在线，实时数据通过以下方式获取：
+
+| 来源类型 | 典型服务 | 覆盖范围 | 成本 | 状态 |
+|---------|---------|---------|------|------|
+| **BarentsWatch AIS API** | live.ais.barentswatch.no | 挪威海域实时 | 免费，需要 AIS API client credentials | **当前使用** |
+| **AISHub** | aishub.net | 全球实时 | 免费/小额 | TODO：付费接入 |
+| **MarineTraffic API** | marinetraffic.com | 全球实时 | $50–$500/月 | TODO：评估 tier |
+| **VesselFinder API** | vesselfinder.com | 全球实时 | $50–$300/月 | TODO：备选 |
+| **自建 SDR 接收** | RTL-SDR + AIS-catcher | 仅本地 30–50km | 硬件 $30 | 不考虑 |
+| **NOAA 历史数据** | Marine Cadastre | 美国近海历史 | 免费 | 可用于冷启动 |
+
+### BarentsWatch AIS API
+
+- 端点：`https://live.ais.barentswatch.no/v1/latest/combined`
+- 需要在 BarentsWatch developer portal 创建 `AIS - API` client，通过 client credentials 获取 `scope=ais` 的 access token 后请求 AIS endpoint
+- 字段：mmsi, lat, lon, sog, cog, heading, nav_status, name, vessel_type, flag
+- 刷新频率：数据约 30–60s 更新一次，可随意轮询
+
+### TODO：多源 AIS 与实时流接入
+
+- [ ] 接入 AISStream WebSocket 采集器，作为 BarentsWatch 覆盖不足的实时补充
+- [ ] 将 BarentsWatch、AISStream、自定义 `vessel_ais` 映射源统一写入原始观测层
+- [ ] 通过聚合接口做去重、字段合并、冲突记录和默认来源选择
+- [ ] 开放字段级聚合策略配置，让用户决定不同字段优先信任哪个来源
+- [ ] 评估 AISHub 订阅（全球覆盖，约 $30/月），接入全球实时流
+- [ ] 评估 MarineTraffic API tier，对比 AISHub 数据质量与成本
+- [ ] 实现多数据源适配器，通过 `datasource_config` 切换
+- [ ] 真实高频 AIS 稳定接入后，评估将 `vessel_position` 迁移为 TimescaleDB hypertable（保留 Postgres 原生分区作为备选）
+
+多源 AIS 的详细设计见 [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)。
+
+---
+
+## 二、实施计划
+
+### Phase 0 — 数据源验证与链路打通（1–2 天）
+
+- 接入 BarentsWatch AIS API，验证 OAuth token、数据格式与字段
+- 构建全球 mock 数据生成器（用于前端渲染压测，补充 BarentsWatch 的地域限制）
+- 确认前端可渲染船只点，整条链路走通
+
+### Phase 1 — 后端基础设施（3–4 天）
+
+#### 1.1 数据库 Schema
+
+```sql
+-- 船只静态信息（每 6h 刷新一次）
+CREATE TABLE vessel_static (
+    mmsi          BIGINT PRIMARY KEY,
+    name          VARCHAR(128),
+    callsign      VARCHAR(16),
+    vessel_type   SMALLINT,
+    vessel_type_name VARCHAR(64),
+    flag          VARCHAR(4),         -- ISO 国家码
+    length        FLOAT,
+    width         FLOAT,
+    draught       FLOAT,
+    imo           BIGINT,
+    updated_at    TIMESTAMPTZ
+);
+
+-- 船只实时位置（高频写入，保留 24h 轨迹）
+CREATE TABLE vessel_position (
+    id            BIGSERIAL PRIMARY KEY,
+    mmsi          BIGINT NOT NULL,
+    lat           FLOAT NOT NULL,
+    lon           FLOAT NOT NULL,
+    sog           FLOAT,              -- Speed over ground（节）
+    cog           FLOAT,              -- Course over ground（度）
+    heading       SMALLINT,           -- 真北航向
+    nav_status    SMALLINT,           -- 0=航行 1=锚泊 5=停靠 ...
+    received_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX idx_vessel_pos_mmsi_time ON vessel_position(mmsi, received_at DESC);
+CREATE INDEX idx_vessel_pos_time      ON vessel_position(received_at DESC);
+
+-- 最新位置物化视图（地图渲染主数据源，避免全表扫描）
+CREATE MATERIALIZED VIEW vessel_latest AS
+SELECT DISTINCT ON (mmsi)
+    vp.*, vs.name, vs.vessel_type_name, vs.flag, vs.length
+FROM vessel_position vp
+LEFT JOIN vessel_static vs USING (mmsi)
+ORDER BY mmsi, received_at DESC;
+
+CREATE UNIQUE INDEX ON vessel_latest(mmsi);
+```
+
+> 后期如需完整历史轨迹查询，迁移 `vessel_position` 到 TimescaleDB 或按天分区。
+
+#### 1.2 Collector：VesselAISCollector
+
+文件：`backend/app/services/collectors/vessel_ais.py`
+
+- 继承 `BaseCollector`，注册到 `collector_registry`
+- 轮询间隔：30–60s（由数据源限速决定）
+- 支持多数据源切换，通过 `datasource_config` 配置 URL + API Key
+- 写入逻辑：upsert `vessel_latest`，append `vessel_position`
+- 接入现有调度系统（`scheduler.py`）
+
+#### 1.3 API 端点
+
+```
+GET /api/v1/visualization/geo/vessels
+    ?bbox=lon_min,lat_min,lon_max,lat_max   # 视口裁剪
+    ?type=cargo,tanker,passenger             # 船型过滤
+    ?limit=0                                 # 可选；不传或 0 表示不裁剪数量
+→ GeoJSON FeatureCollection（Point）
+
+GET /api/v1/visualization/vessels/{mmsi}           # 单船详情
+GET /api/v1/visualization/vessels/{mmsi}/track     # 历史轨迹（默认 6h）
+    ?hours=6
+→ GeoJSON LineString
+```
+
+GeoJSON Feature 格式：
+
+```json
+{
+  "type": "Feature",
+  "geometry": { "type": "Point", "coordinates": [lon, lat] },
+  "properties": {
+    "mmsi": 123456789,
+    "name": "EVER GIVEN",
+    "vessel_type": 70,
+    "vessel_type_name": "Cargo",
+    "flag": "PA",
+    "sog": 12.4,
+    "cog": 247.0,
+    "heading": 245,
+    "nav_status": 0,
+    "length": 400,
+    "received_at": "2026-04-27T10:00:00Z"
+  }
+}
+```
+
+#### 1.4 更新机制
+
+**前端聚合结果拉取 + 后端实时采集**：
+
+- 前端 `setInterval(fetchVessels, 5 * 60 * 1000)` 定期拉取最新快照
+- 后端 BarentsWatch collector 继续以 HTTP polling 方式采集
+- AISStream 等实时源以独立 WebSocket collector 写入原始观测层
+- 展示接口从聚合服务读取当前船只视图，而不是由单个 collector 决定最终展示值
+- 前端默认不再给 `/geo/vessels` 传 `limit=5000`，`VESSEL_CONFIG.maxRenderedMarkers = 0` 表示不做前端数量裁剪；后续如性能不足再引入显式 LOD 上限
+- marker 颜色、详情卡、hover 和搜索结果必须共享 `vessel_type_display` 船型归一化结果，避免 AIS 数字类型码已驱动颜色但卡片仍显示 `Other`
+- 前端是否升级为 WebSocket delta push 是独立优化，不影响后端采集器可以使用 WebSocket 接上游实时源
+
+---
+
+### Phase 2 — 前端渲染（3–4 天）
+
+文件：`frontend/public/earth/js/vessels.js`
+
+#### 2.1 渲染方案
+
+参考现有卫星系统（`satellites.js`）的 InstancedMesh 模式：
+
+- `THREE.InstancedMesh`：每个实例 = 一艘船，矩阵包含位置 + 旋转（朝向 COG）
+- 行进船：三角箭头图标，朝向 COG 方向
+- 静止/锚泊船：圆点图标
+- SVG 图标输出到 `frontend/public/earth/assets/icons/vessel-arrow.svg` 和 `vessel-dot.svg`
+
+#### 2.2 船型颜色规范
+
+| 船型 | 颜色 |
+|-----|------|
+| 货轮 Cargo | `#4A90D9` 蓝 |
+| 油轮 Tanker | `#E85D04` 橙红 |
+| 客船 Passenger | `#06D6A0` 绿 |
+| 渔船 Fishing | `#FFD166` 黄 |
+| 军舰 Military | `#73797E` 灰 |
+| 其他 | `#9B9B9B` 浅灰 |
+| 锚泊/停靠 | 降低饱和度 0.4x |
+
+#### 2.3 LOD（相机距离细节层次）
+
+| 相机距离 | 渲染策略 |
+|---------|---------|
+| > 400 | 默认渲染当前接口返回的全部船只；如性能不足，再引入可配置 LOD 上限 |
+| 200–400 | 默认渲染当前接口返回的全部船只；如性能不足，再引入可配置 LOD 上限 |
+| < 200 | 渲染当前视口 bbox 内全部船只 |
+
+前端根据相机位置动态计算 bbox，附加到 API 请求中。
+
+#### 2.4 图层集成
+
+接入现有图层系统，新增"船只"图层项，支持：
+- 图层开/关，状态持久化
+- 子过滤（按船型选择显示哪类，可在图例或设置面板中配置）
+- 与海缆、BGP、卫星层级共存（renderOrder 待定，参考现有层级文档）
+
+#### 2.5 Info Card
+
+复用 `showInfoCard` 机制，点击船只弹出：
+
+```
+EVER GIVEN                              🚢
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+MMSI        123456789
+IMO         9811000
+旗帜        巴拿马 🇵🇦
+船型        散货轮
+当前航速     12.4 kn
+航向        247°
+状态        航行中
+目的地      ROTTERDAM
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+[ 查看轨迹 ]   [ MarineTraffic ↗ ]
+```
+
+#### 2.6 轨迹可视化
+
+点击"查看轨迹" → 请求 `/vessels/{mmsi}/track` → 用 `THREE.CatmullRomCurve3` 渲染插值轨迹线，风格与海缆一致。
+
+---
+
+### Phase 3 — 功能完善（2–3 天）
+
+| 功能 | 说明 |
+|-----|------|
+| **船只搜索** | 接入现有搜索面板，按名称 / MMSI 搜索 |
+| **统计 HUD** | 显示当前在线船只数、各类型分布 |
+| **密度热图** | 超低 zoom 时切换为 hex-bin 热力图（避免点云爆炸） |
+| **港口标注** | 加载 WorldPorts 数据集，显示主要港口标记 |
+| **关键水道监控** | 马六甲、霍尔木兹、苏伊士等高亮 + 流量统计 |
+
+---
+
+### Phase 4 — 性能与生产化（2–3 天）
+
+- `vessel_position` 按天分区，7 天自动清理
+- TODO：真实数据量达到百万级/日后，将 `vessel_position` 升级为 TimescaleDB hypertable，配置 retention policy 与压缩策略
+- GeoJSON endpoint 用 Redis 缓存 15s
+- 若需 bbox 精确查询，引入 PostGIS `geography` + `ST_DWithin`
+- InstancedMesh + frustum culling，目标 5 万船只 60fps
+
+---
+
+## 三、工作量估算
+
+| Phase | 内容 | 估计时间 |
+|-------|-----|---------|
+| Phase 0 | 数据源验证、mock | 1–2 天 |
+| Phase 1 | 后端 Schema + Collector + API | 3–4 天 |
+| Phase 2 | 前端渲染（InstancedMesh + 图层 + Info Card） | 3–4 天 |
+| Phase 3 | 搜索 + 统计 + 轨迹 | 2–3 天 |
+| Phase 4 | 性能优化 + 生产数据源接入 | 2–3 天 |
+| **合计** | | **约 2–3 周** |
+
+---
+
+## 四、参考资料
+
+- BarentsWatch AIS API 文档：https://www.barentswatch.no/en/developer/ais-api/
+- MarineTraffic API：https://www.marinetraffic.com/en/ais-api-services
+- AISHub：https://www.aishub.net/api
+- AIS 导航状态码：ITU-R M.1371-5
+- 船型编码（vessel_type）：ITU/IMO AIS Message 5 Type and Cargo
+- WorldPorts 数据集：https://msi.nga.mil/Publications/WPI

docs/plans/enterprise-logging-system-plan.md

@@ -0,0 +1,793 @@
+# Planet 企业级日志系统实施计划
+
+## Goal
+
+把 Planet 当前“能看一点运行输出”的日志能力，升级为一套真正可用、可定位、可纠错、可追责、可演进的企业级日志系统。
+
+这里的“企业级”不是指一上来就接入很重的外部平台，而是指这套系统需要同时满足下面五件事：
+
+1. 排障可用
+2. 历史可查
+3. 业务可解释
+4. 权限操作可追责
+5. 出错后能够反向定位到请求、任务、模块和操作者
+
+最终目标不是“把更多 stdout 放到日志页里”，而是建立一套统一的日志契约与落地链路：
+
+- 统一日志字段
+- 统一事件命名
+- 统一采集入口
+- 统一查询视图
+- 清晰的实时日志、持久化事件、审计日志分层
+
+## Why
+
+当前仓库已经有一些日志基础，但离真正可用的日志系统还有明显距离。
+
+已有基础：
+
+- 后端运行日志可通过 `/tmp/planet_backend.log` 查看
+- 前端开发服务日志可通过 `/tmp/planet_frontend.log` 查看
+- AI Provider 可从 Docker 容器读取日志
+- Earth 浏览器端关键日志可上报到后端并进入 Redis 缓冲
+- 已有 `system_logs` / `audit_logs` 持久化能力
+- 管理台已有“系统日志”页面，支持来源、级别、日期、搜索
+
+当前缺口：
+
+- 后端日志仍以 `uvicorn` / 文本输出为主，不是统一结构化事件流
+- 不同模块的日志格式不一致，很多地方只有 message，没有 event 语义
+- 还没有统一的后端 logger 封装与字段注入机制
+- 前端虽然能上报错误，但还没有统一 logger API 和统一事件词汇
+- Earth 与管理台之间的错误事件还没有形成可串联的事件链路
+- 历史持久化还偏点状，很多高价值失败并没有系统性落库
+- 系统日志页当前更像“运行输出查看器”，不是“多层日志查询台”
+- 审计日志与运行日志尚未形成明确的产品级联动
+
+所以当前真正的问题不是“有没有日志页”，而是：
+
+**当前系统能看见输出，但还不能稳定回答“发生了什么、影响了谁、在哪条链路上坏了、是否已修复、是谁触发的”。**
+
+## Current State
+
+截至 2026-04-23，当前代码中的日志相关能力大致如下。
+
+### 1. 日志来源
+
+当前系统日志页主要读取以下来源：
+
+- `backend`
+  读取 `/tmp/planet_backend.log`
+- `frontend`
+  读取 `/tmp/planet_frontend.log`
+- `ai-provider`
+  读取 Docker 容器日志
+- `earth-client`
+  读取 Redis 缓冲的浏览器端日志
+
+这些来源定义在：
+
+- [backend/app/services/system_logs.py](/home/ray/dev/linkong/planet/backend/app/services/system_logs.py)
+
+### 2. 当前日志读取模型
+
+当前 `read_log_snapshot()` 的职责是：
+
+- 读取某个来源的最近若干行
+- 解析基础级别与时间
+- 按级别、日期、搜索进行过滤
+- 返回用于日志页展示的快照
+
+这个模型适合“运维查看器”，但不适合企业级日志系统，原因是：
+
+- 读取基于文本尾部扫描，不是基于事件模型
+- 不同来源的结构粒度完全不同
+- 过滤依赖文本解析，准确率有限
+- 没有请求、任务、用户、资源、动作等核心关联字段
+
+### 3. 已有持久化能力
+
+当前已经存在两个持久化入口：
+
+- `record_system_log(...)`
+- `record_audit_log(...)`
+
+位置：
+
+- [backend/app/services/persistent_logs.py](/home/ray/dev/linkong/planet/backend/app/services/persistent_logs.py)
+
+这说明系统并不是从 0 开始，但也说明当前最大的问题是：
+
+**持久化能力存在，但没有成为统一默认路径。**
+
+### 4. 已有 request_id 基础
+
+当前系统已具备 `request_id` 相关基础，部分持久化能力也会尝试写入 `request_id`。
+
+这为后续做：
+
+- 请求链路排障
+- 前后端关联查询
+- 任务执行追踪
+
+提供了很好的基础。
+
+### 5. 当前日志页定位
+
+当前日志页已经具备：
+
+- 来源切换
+- 级别筛选
+- 日期筛选
+- 搜索
+- 文本控制台视图
+
+但它仍然是“单层视图”：
+
+- 上面是筛选器
+- 下面是一块文本控制台
+
+它还不是：
+
+- 运行日志 + 事件日志 + 审计日志 的统一入口
+- 也没有事件详情、关联跳转、纠错建议、链路追踪能力
+
+## Core Principles
+
+这套日志系统后续必须遵循下面几个原则。
+
+### 1. 分层，而不是混存
+
+日志必须拆成三层：
+
+1. 运行日志
+2. 持久化事件日志
+3. 审计日志
+
+它们的用途不同，绝不能继续混成一个概念。
+
+#### 运行日志
+
+用于：
+
+- 实时排障
+- 观察服务运行状态
+- 看 stdout / stderr / exception / collector 输出
+
+特点：
+
+- 数据量大
+- 时效性强
+- 保留周期短
+- 不要求每条都落库
+
+#### 持久化事件日志
+
+用于：
+
+- 记录高价值错误
+- 记录关键业务失败
+- 支撑历史追溯
+- 支撑趋势分析
+
+特点：
+
+- 只持久化有价值事件
+- 必须结构化
+- 必须有统一 event 命名
+
+#### 审计日志
+
+用于：
+
+- 留痕
+- 追责
+- 还原高权限操作
+
+特点：
+
+- 必须单独建模
+- 不与普通运行日志混用
+
+### 2. 结构化优先
+
+正式日志必须可拆字段，不能长期依赖自由文本。
+
+最低要求至少能拿到：
+
+- `timestamp`
+- `level`
+- `service`
+- `module`
+- `event`
+- `message`
+- `request_id`
+- `trace_id`
+- `user_id` / `actor`
+- `context`
+
+### 3. 事件命名优先于 message 命名
+
+人看的 message 可以变化，但机器查询和跨模块关联必须依赖稳定事件名。
+
+例如：
+
+- `collector.run.started`
+- `collector.run.completed`
+- `collector.run.failed`
+- `earth.layer.load_failed`
+- `earth.cruise.route_build_failed`
+- `system.restart_task.failed`
+- `auth.websocket.invalid_token`
+
+### 4. 查询链路必须可串联
+
+企业级日志系统的核心不是“有很多日志”，而是“能串起来”。
+
+最终一条高价值事件，至少要能回链到下面任意几类对象：
+
+- 某个请求
+- 某个任务
+- 某个用户
+- 某个数据源
+- 某个 Earth 模块
+- 某个管理动作
+
+### 5. 默认脱敏
+
+日志体系必须明确禁止记录：
+
+- token
+- password
+- Authorization header
+- cookie
+- session
+- 明文敏感个人信息
+
+并且需要有统一脱敏器，而不是靠调用者自觉。
+
+### 6. “可纠错”不是一句口号
+
+这里的“可纠错”至少包含三层：
+
+1. 日志字段足够解释错误，方便人排查
+2. 系统能识别常见错误模式并给出纠偏建议
+3. 关键错误支持闭环动作，例如重试、重建索引、重新触发采集、跳转到对应对象
+
+也就是说，这套日志系统最终不只是“告诉你出错了”，而要尽量接近“告诉你为什么出错、怎么修、去哪修”。
+
+## Non-Goals
+
+第一阶段不追求：
+
+- 全量接入 ELK / Loki / Datadog / OpenTelemetry 全家桶
+- 做分布式 trace 全链路可视化大屏
+- 把所有历史日志都迁进数据库
+- 先做特别复杂的规则引擎
+
+第一阶段追求的是：
+
+- 在当前仓库和当前部署方式下，先把基础日志体系做正确
+- 再为后续平台化接入预留好接口
+
+## Target Architecture
+
+推荐目标架构如下。
+
+### Layer 1: Runtime Logs
+
+职责：
+
+- 承载后端、前端开发服务、容器输出、浏览器端缓冲事件
+- 提供最近窗口内的实时查看能力
+
+来源：
+
+- 文件
+- Docker
+- Redis 缓冲
+- 后续可扩展到 stdout collector
+
+接口：
+
+- `GET /api/v1/system/logs/sources`
+- `GET /api/v1/system/logs/{source_id}`
+
+这层继续保留，但需要做结构化增强和来源补强。
+
+### Layer 2: Persistent System Events
+
+职责：
+
+- 只存高价值事件
+- 供历史追溯、事件列表、趋势和纠错使用
+
+数据来源：
+
+- 后端关键异常
+- 浏览器端关键失败
+- 采集器/调度器关键失败
+- 业务关键告警与降级事件
+
+接口建议：
+
+- `GET /api/v1/system/events`
+- `GET /api/v1/system/events/{id}`
+- `POST /api/v1/system/events/{id}/actions/...`（后续）
+
+### Layer 3: Audit Logs
+
+职责：
+
+- 留痕高权限操作
+- 记录操作者、对象、结果、请求号
+
+接口建议：
+
+- `GET /api/v1/system/audit-logs`
+
+### Layer 4: Error Intelligence / Triage
+
+职责：
+
+- 对高频错误做归类
+- 对已知错误给出解释与建议动作
+- 对相同错误进行 fingerprint 聚合
+
+这是“可纠错”能力的关键层。
+
+建议字段：
+
+- `fingerprint`
+- `root_cause_type`
+- `known_fix_hint`
+- `runbook_url`
+- `related_resource_type`
+- `related_resource_id`
+
+## Canonical Event Model
+
+推荐统一事件字段模型如下。
+
+### Runtime Log Record
+
+```json
+{
+  "timestamp": "2026-04-23T10:15:30Z",
+  "level": "error",
+  "service": "backend",
+  "module": "app.services.scheduler",
+  "event": "collector.run.failed",
+  "message": "Collector bgp_news failed",
+  "request_id": "req_xxx",
+  "trace_id": "trace_xxx",
+  "user_id": null,
+  "actor": null,
+  "resource_type": "collector",
+  "resource_id": "bgp_news",
+  "context": {
+    "datasource_id": 12,
+    "exception_type": "TimeoutError"
+  }
+}
+```
+
+### Persistent System Event
+
+```json
+{
+  "id": 1024,
+  "event": "earth.layer.load_failed",
+  "level": "error",
+  "source": "earth-client",
+  "service": "earth",
+  "module": "cables",
+  "message": "Failed to load cable layer",
+  "fingerprint": "earth.layer.load_failed:cables:network_timeout",
+  "request_id": "req_xxx",
+  "trace_id": null,
+  "user_id": 1,
+  "resource_type": "earth_layer",
+  "resource_id": "cables",
+  "category": "visualization",
+  "status": "open",
+  "context": {
+    "url": "/api/v1/visualization/geo/cables"
+  },
+  "created_at": "2026-04-23T10:15:30Z"
+}
+```
+
+### Audit Log
+
+```json
+{
+  "id": 88,
+  "action": "system.restart_task.requested",
+  "actor_id": 1,
+  "actor_name": "root",
+  "target_type": "restart_task",
+  "target_id": "restart_20260423_xxx",
+  "result": "success",
+  "request_id": "req_xxx",
+  "ip": "127.0.0.1",
+  "details": {
+    "action": "restart_backend"
+  },
+  "created_at": "2026-04-23T10:15:30Z"
+}
+```
+
+## Implementation Plan
+
+## Phase 0: Logging Inventory And Naming Freeze
+
+目标：
+
+- 先统一“记录什么”和“怎么命名”，避免后面越做越乱
+
+工作项：
+
+- 盘点当前所有 `logging.getLogger` 使用点
+- 盘点裸 `print`
+- 盘点 `record_system_log` / `record_audit_log` 已落点位
+- 建立统一事件命名表
+- 定义 service / module / category / resource 字段枚举
+- 输出日志字段白名单和脱敏规范
+
+完成标准：
+
+- 有一份稳定的事件命名清单
+- 有一份字段规范清单
+- 后续新增日志不再“临时起名”
+
+## Phase 1: Backend Structured Logging Foundation
+
+目标：
+
+- 把后端从“散落 logging + 文本输出”升级成“统一结构化 logger”
+
+工作项：
+
+- 新增统一后端 logger helper，例如 `app/core/logging.py`
+- 自动注入：
+  - `service`
+  - `module`
+  - `request_id`
+  - `trace_id`
+- 增加统一脱敏 filter
+- 把关键模块先切到统一 logger：
+  - API 层
+  - scheduler
+  - collectors
+  - websocket
+  - visualization
+  - system control
+- 约束：
+  - 正式路径禁止裸 `print`
+  - 正式异常优先 `logger.exception(..., extra={...})`
+
+完成标准：
+
+- 后端关键模块都有稳定 `event`
+- request 日志和异常日志能挂上 `request_id`
+- 不再依赖只看 `uvicorn` 原生文本输出来定位问题
+
+## Phase 2: Persistent Event Layer
+
+目标：
+
+- 把“值得长期保留的错误和关键事件”系统性落库
+
+工作项：
+
+- 重新定义 `record_system_log()` 的使用边界
+- 明确哪些事件必须持久化：
+  - API 关键失败
+  - 调度器失败
+  - 采集器失败
+  - Earth 客户端关键错误
+  - 数据源不可用
+  - 业务降级与恢复
+- 补齐字段：
+  - `event`
+  - `resource_type`
+  - `resource_id`
+  - `category`
+  - `fingerprint`
+  - `status`
+- 增加高频错误去重/聚合策略
+
+完成标准：
+
+- 高价值错误不再只存在于运行日志里
+- 能查询最近一周/一月的关键失败事件
+- 相同错误具备聚合基础
+
+## Phase 3: Frontend And Earth Unified Logger
+
+目标：
+
+- 把前端从“点状 error 上报”升级成统一前端事件流
+
+工作项：
+
+- 在前端新增统一 logger API
+- 统一方法：
+  - `debug`
+  - `info`
+  - `warn`
+  - `error`
+- 统一字段：
+  - `page`
+  - `module`
+  - `event`
+  - `message`
+  - `url`
+  - `user_agent`
+  - `context`
+- Earth 模块优先接入：
+  - layer load failed
+  - cruise build failed
+  - popup render failed
+  - connector render failed
+  - websocket dropped
+- 管理台优先接入：
+  - settings save failed
+  - datasource toggle failed
+  - restart task submit failed
+
+完成标准：
+
+- 前端日志事件名与后端可对齐
+- Earth 和管理台关键失败不再只停留在 console
+- 浏览器端关键问题能进入统一系统日志/事件层
+
+## Phase 4: Audit Logging Completion
+
+目标：
+
+- 把管理员与高权限操作真正做成企业级审计
+
+工作项：
+
+- 扩大审计覆盖面：
+  - 系统重启
+  - 数据源启停
+  - 调度规则变更
+  - 配置变更
+  - 人工触发采集
+  - 删除/修改关键配置
+- 增加字段：
+  - actor
+  - target
+  - before / after
+  - request_id
+  - IP
+- 审计页支持：
+  - 动作筛选
+  - 操作者筛选
+  - 时间筛选
+  - 目标对象筛选
+
+完成标准：
+
+- 所有高权限操作都能追到人、时间、对象、结果
+
+## Phase 5: Log Console To Enterprise Observability UI
+
+目标：
+
+- 把当前“系统日志”页升级为真正的多层日志工作台
+
+工作项：
+
+- 将页面拆为三个主视图：
+  1. 运行日志
+  2. 关键事件
+  3. 审计日志
+- 运行日志视图：
+  - 保留大控制台
+  - 支持来源、级别、日期、搜索
+- 关键事件视图：
+  - 列表化展示高价值事件
+  - 支持聚合、状态、指纹、对象筛选
+- 审计视图：
+  - 列表化展示管理员动作
+- 增加详情抽屉：
+  - 原始 message
+  - context
+  - request_id
+  - related resource
+  - recommended action
+
+完成标准：
+
+- 日志页不再只是“终端文本窗口”
+- 运维排障、历史追溯、审计留痕三者分层清晰
+
+## Phase 6: Corrective Intelligence
+
+目标：
+
+- 让系统从“能看日志”进化到“能辅助修错”
+
+工作项：
+
+- 引入错误 fingerprint
+- 对已知错误配置：
+  - 根因类型
+  - 修复建议
+  - runbook 链接
+  - 推荐动作
+- 支持常见纠错动作：
+  - 重试采集任务
+  - 重载配置
+  - 跳转到对应模块/资源
+  - 打开相关日志过滤视图
+- 高频错误支持聚合与静默窗口
+
+完成标准：
+
+- 已知错误能给出明确建议
+- 运维不需要每次都从零猜
+
+## Recommended Module Changes
+
+### Backend
+
+建议新增/增强的模块：
+
+- `backend/app/core/logging.py`
+  - 统一 logger 封装
+  - formatter
+  - filter
+  - request/trace 注入
+- `backend/app/services/persistent_logs.py`
+  - 扩展字段
+  - 统一持久化策略
+- `backend/app/services/system_logs.py`
+  - 逐步从“文本尾部查看器”升级为“运行日志聚合器”
+- `backend/app/services/log_classification.py`
+  - 指纹
+  - 根因分类
+  - 纠错建议
+- `backend/app/api/v1/system_control.py`
+  - 补充事件 / 审计 / 日志多视图接口
+
+### Frontend
+
+建议新增/增强：
+
+- `frontend/src/lib/logger.ts`
+  - 统一前端 logger API
+- `frontend/src/pages/Logs/Logs.tsx`
+  - 升级为多层工作台
+- `frontend/public/earth/js/...`
+  - 各 Earth 模块接入统一事件 logger
+
+## Event Naming Convention
+
+建议采用：
+
+`<domain>.<resource>.<action>.<result>`
+
+示例：
+
+- `collector.datasource.run.started`
+- `collector.datasource.run.failed`
+- `earth.layer.cables.load.failed`
+- `earth.cruise.route.build.failed`
+- `system.restart_task.requested`
+- `system.restart_task.completed`
+- `auth.websocket.connect.failed`
+- `settings.datasource.priority.updated`
+
+规则：
+
+- 不用自然语言句子
+- 不把 ID 塞进 event 名里
+- 资源对象通过字段承载，不通过 event 名承载
+
+## Query Model
+
+最终推荐支持的查询维度：
+
+- 时间范围
+- level
+- source
+- service
+- module
+- event
+- request_id
+- trace_id
+- user_id / actor
+- resource_type / resource_id
+- category
+- fingerprint
+- status
+- full-text search
+
+## Retention Strategy
+
+推荐保留策略：
+
+- 运行日志：
+  - 文件 / 容器 / Redis 缓冲保留短周期
+- 持久化事件：
+  - 保留中长期
+- 审计日志：
+  - 长期保留
+
+初版可以先这样：
+
+- 运行日志：7 到 14 天
+- 关键事件：90 到 180 天
+- 审计日志：180 天以上
+
+后续再根据存储与合规要求调整。
+
+## Security And Compliance
+
+必须落实：
+
+- 敏感字段脱敏
+- 前端上报白名单
+- 防止日志注入
+- 审计日志不可被普通管理员随意篡改
+- 高敏感纠错动作必须再次鉴权
+
+## Success Criteria
+
+当下面这些条件成立时，才算这套日志系统真的“成了”：
+
+1. 一个后端请求失败时，能通过 `request_id` 在运行日志、持久化事件、审计日志之间串联查询
+2. 一个 Earth 前端错误能定位到页面、模块、事件名和上下文
+3. 一个采集器失败能同时看到运行日志、持久化事件和可执行纠错动作
+4. 一个管理员操作能查到操作者、目标对象、结果和 request_id
+5. 日志页不再只是文本控制台，而是完整的“运行日志 / 关键事件 / 审计日志”工作台
+6. 高频已知错误能聚合并给出修复建议
+
+## Delivery Order
+
+推荐严格按下面顺序做，不要乱跳：
+
+1. Phase 0 命名与字段规范冻结
+2. Phase 1 后端结构化 logging 基础
+3. Phase 2 高价值事件持久化
+4. Phase 3 前端 / Earth 统一 logger
+5. Phase 4 审计覆盖补齐
+6. Phase 5 日志工作台 UI 重构
+7. Phase 6 指纹 / 纠错 / runbook
+
+原因：
+
+- 如果不先统一字段和命名，后面 UI 和持久化会越来越乱
+- 如果不先做后端结构化基础，前端上报再多也串不起来
+- 如果不先补持久化层，就只有“实时可看”，没有“历史可查”
+
+## First Actionable Milestone
+
+如果要从明天就开始做，最合理的第一个里程碑是：
+
+### M1: 让后端关键路径全部拥有统一结构化事件
+
+范围：
+
+- API 请求入口/出口
+- scheduler
+- collectors
+- websocket
+- visualization
+- system control
+
+交付物：
+
+- 统一 logger helper
+- 统一 event naming 表
+- 统一 request_id 注入
+- 统一脱敏策略
+- 关键模块替换完成
+
+完成这个里程碑后，Planet 才算真正拥有了“企业级日志系统的地基”。
+

docs/plans/frontend-markdown-renderer-plan.md

@@ -0,0 +1,97 @@
+# Markdown 渲染器完善计划
+
+## 背景
+
+Planet 控制台当前有三类主要 Markdown 使用场景：
+
+- 文档中心：技术文档、计划文档、运行手册。
+- AI Playground：模型回复、分析结果、代码片段。
+- BGP 简报：由系统生成并保存的态势报告。
+
+这些场景都复用 `frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx`。因此 Markdown 能力应该集中在共享渲染器内完成，页面只负责传入内容、链接转换和布局约束，不能让每篇文档或每个页面手写复制按钮、表格样式、列表样式等交互细节。
+
+## 目标
+
+建设一个稳定、可复用、适合技术文档和 AI 输出的 Markdown 渲染器，优先覆盖常用语法、代码块操作和清晰的阅读样式，并为后续语法高亮、锚点导航、内容安全策略留出接口。
+
+## 成功标准
+
+- 代码块支持 fenced language、语言标签、复制按钮、复制成功状态和横向滚动。
+- 常用块语法稳定渲染：标题 1-6、段落、引用、分割线、表格、无序列表、有序列表、任务列表。
+- 常用行内语法稳定渲染：链接、自动链接、图片、行内代码、粗体、斜体、删除线。
+- 文档中心、AI Playground、BGP 简报继续复用同一个组件，不出现页面级重复实现。
+- 样式在普通业务面板和文档中心都有合理表现，文档中心可以通过 `.docs-markdown` 覆盖主题变量。
+- 前端 TypeScript build 通过，`git diff --check` 无空白错误。
+
+## 当前实施范围
+
+### 第一阶段：共享渲染器补齐
+
+- 在 `MarkdownRenderer` 内解析 fenced code block 的语言信息。
+- 引入 `MarkdownCodeBlock` 子组件，负责语言标签、复制按钮和复制状态。
+- 保留现有 `Scrollbar` 横向滚动能力，避免长代码撑破页面。
+- 扩展标题渲染到 h1-h6，并保留 `getHeadingId` 对文档目录的支持。
+- 扩展列表解析，支持 `-`、`*`、`+`、`1.`、`1)` 和 GitHub 风格任务列表。
+- 扩展行内解析，支持图片、自动链接、删除线。
+
+### 第二阶段：样式统一
+
+- 全局 Markdown 样式覆盖业务场景，保持紧凑、清晰、可扫描。
+- 文档中心用 `.docs-markdown` 适配主题变量，避免硬编码颜色破坏明暗主题。
+- 代码块 toolbar 和 copy button 不依赖具体页面。
+- 图片默认响应式展示，避免超出内容区域。
+
+### 第三阶段：验证
+
+- 使用前端 build 验证 TypeScript 和 Vite 构建。
+- 使用 `git diff --check` 验证补丁格式。
+- 手动检查至少一个文档页中代码块复制按钮、语言标签和表格滚动是否出现。
+
+## 后续增强项
+
+### 语法高亮
+
+当前不新增高亮依赖，避免一次性引入过重运行时代码。后续可以在以下方案中二选一：
+
+- `shiki`：适合文档中心，视觉质量高，但包体和初始化成本更高。
+- `highlight.js`：接入简单，覆盖语言广，但样式控制需要额外约束。
+
+建议当文档代码块数量稳定增加后再引入，并做按需加载或懒加载。
+
+### 更完整 CommonMark 支持
+
+当前渲染器覆盖 Planet 常见内容，不追求完整 CommonMark 兼容。后续如果需要完整规范，建议切换到成熟生态：
+
+- `react-markdown`
+- `remark-gfm`
+- `rehype-sanitize`
+- `rehype-slug`
+
+切换前需要评估：链接转换、目录 ID、现有样式、AI 输出安全策略和包体影响。
+
+### 安全策略
+
+目前渲染器不解析原始 HTML，这是正确默认值。后续如需支持 HTML，必须先明确：
+
+- 是否允许用户输入 Markdown。
+- 是否需要 HTML 白名单。
+- 是否需要 `rehype-sanitize`。
+- 图片和链接是否需要域名策略。
+
+### 文档页能力
+
+可继续补齐：
+
+- 标题锚点悬浮复制。
+- Mermaid 图表。
+- 代码块折叠。
+- 文档内搜索结果定位到代码块。
+- 复制按钮埋点，用于判断文档片段是否真正被使用。
+
+## 维护约束
+
+- Markdown 语法能力优先放在共享渲染器，不在具体文档页面散落实现。
+- 文档内容只表达内容，不承载 UI 行为。
+- 新增 Markdown 能力必须同时考虑文档中心、AI Playground、BGP 简报三个调用方。
+- 不解析原始 HTML，除非同步引入明确的 sanitize 策略。
+- 与主题相关的样式优先走页面容器变量覆盖，不在组件内写死文档中心颜色。

docs/plans/frontend-public-docs-site-plan.md

@@ -0,0 +1,486 @@
+# Frontend Public Docs Site Plan
+
+## 目标
+
+新增一个公开访问的 `/docs` 页面，作为 Planet 的开发设计文档与使用手册入口。
+
+这个页面应类似常见开源软件文档站：
+
+- 不需要登录即可访问
+- 与 `/earth` 和 admin 后台平级，但视觉和信息架构独立
+- 直接整理并展示仓库内 `docs/technical` 的 Markdown 文档
+- 支持搜索、分类导航、文档目录和内部跳转
+- 让 `docs/technical` 继续作为文档真源，避免页面内容和仓库文档漂移
+
+## 非目标
+
+本阶段不做：
+
+- 后端全文搜索服务
+- 数据库驱动的 CMS
+- 独立文档构建系统，例如 Docusaurus / VitePress
+- 每篇文档单独手写 React 页面
+- 用户权限、编辑器、在线保存或评论功能
+- 把 `docs/plans`、`docs/deprecated` 全量公开为正式手册
+
+后续可以再决定是否把 plans / deprecated 做成独立的“路线图 / 历史归档”分区。
+
+## 技术路线
+
+### 推荐方案：Markdown 直接渲染
+
+使用 Vite 在前端构建阶段直接加载 `docs/technical/**/*.md`：
+
+```ts
+const modules = import.meta.glob('../../../docs/technical/**/*.md', {
+  query: '?raw',
+  import: 'default',
+})
+```
+
+这样每篇 Markdown 文件仍然留在仓库文档目录中，`/docs` 页面只是读取、索引和渲染这些文档。
+
+当前项目已经满足主要前提：
+
+- 前端使用 Vite + React
+- `frontend/vite.config.ts` 已配置 `server.fs.allow: ['..']`
+- 已有 `MarkdownRenderer` 可作为基础
+- `docs/technical` 文档数量较少，前端本地搜索足够
+
+### 不推荐方案：每篇文档单独写 React
+
+不建议把每篇文档重写成 `.tsx` 页面，因为：
+
+- 文档会出现两份真源
+- 修改技术文档时还要同步 UI 页面
+- 计划文档、技术上下文、变量表这类内容天然适合 Markdown
+- 后续新增文档的成本会变高
+
+只有当某篇文档需要强交互演示、实时图表或复杂 UI 时，才考虑给该文档补充一个 React 组件扩展。
+
+## 信息架构
+
+### 公开路由
+
+新增：
+
+- `/docs`
+- `/docs/:slug`
+
+路由行为：
+
+- `/docs` 默认打开 `docs/technical/README.md`，或打开人工指定的首页文档
+- `/docs/:slug` 打开对应技术文档
+- 未找到文档时显示 docs 专属 404，而不是跳回 admin
+- `/docs` 加入 `App.tsx` 的公开路由白名单
+
+### 文档分类
+
+将 `docs/technical` 中的现有文档整理进以下分组：
+
+#### Overview
+
+- `README.md`
+
+#### Earth
+
+- `earth-frontend-context.md`
+- `earth-layer-style-reference.md`
+- `earth-render-layer-order.md`
+- `earth-satellite-footprint-policy.md`
+- `earth-bgp-context.md`
+- `earth-news-live-streams-collector-format.md`
+
+#### Frontend
+
+- `frontend-admin-frontend-context.md`
+- `frontend-layout-guidelines.md`
+
+#### Backend
+
+- `backend-collectors.md`
+- `backend-system-service-control.md`
+
+#### Agents
+
+- `agents-aiprovider.md`
+
+#### Ops
+
+- `ops-docker-compose-buildx-upgrade.md`
+
+### 页面布局
+
+桌面端：
+
+- 顶部：产品名、搜索框、当前文档标题
+- 左侧：文档分组导航
+- 中间：Markdown 正文
+- 右侧：当前文档目录，也就是 h2 / h3 anchors
+
+移动端：
+
+- 顶部固定搜索入口
+- 导航折叠为抽屉或下拉
+- 正文单列显示
+- 当前文档目录折叠为“本文目录”
+
+视觉风格：
+
+- 像开源软件 docs 页面，清晰、安静、可长时间阅读
+- 不复用 admin 后台的重操作感布局
+- 不做 Earth 的沉浸式深色 HUD 风格
+- 优先阅读性、扫描效率和代码/表格可读性
+
+## 前端实现设计
+
+### 文件结构
+
+建议新增：
+
+```text
+frontend/src/pages/Docs/
+  Docs.tsx
+  docs-content.ts
+  docs-search.ts
+  docs-slugs.ts
+  Docs.css
+```
+
+可选拆分：
+
+```text
+frontend/src/pages/Docs/components/
+  DocsSidebar.tsx
+  DocsSearch.tsx
+  DocsToc.tsx
+  DocsMarkdown.tsx
+```
+
+如果初版代码量不大，可以先保持在 `Docs.tsx` + 少量 helper 文件中，避免过度拆分。
+
+### 文档注册表
+
+创建一个 registry，负责将 Markdown 文件路径映射为文档元信息：
+
+```ts
+interface DocsEntry {
+  slug: string
+  path: string
+  title: string
+  group: string
+  order: number
+  loader: () => Promise<string>
+}
+```
+
+slug 规则：
+
+- `docs/technical/README.md` -> `overview`
+- `docs/technical/earth-layer-style-reference.md` -> `earth-layer-style-reference`
+- 只暴露稳定 slug，不暴露本机绝对路径
+
+标题规则：
+
+- 优先读取 Markdown 第一个 `# heading`
+- 没有 h1 时用人工 registry title
+- 再 fallback 到文件名转换标题
+
+### Markdown 渲染
+
+初版可以复用现有：
+
+- [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx)
+
+但建议增强或包装为 docs 专用渲染：
+
+- heading 生成稳定 `id`
+- 右侧 TOC 使用同一套 heading 解析结果
+- 内部 Markdown 链接转换为 `/docs/:slug`
+- 外部链接保留 `target="_blank" rel="noreferrer"`
+- 表格横向滚动
+- 代码块保留等宽字体和语言标记
+- 支持 GitHub 风格的相对文档链接
+
+内部链接转换示例：
+
+- `earth-render-layer-order.md` -> `/docs/earth-render-layer-order`
+- `./earth-layer-style-reference.md` -> `/docs/earth-layer-style-reference`
+- `/home/ray/dev/linkong/planet/docs/technical/foo.md` -> `/docs/foo`
+
+对非 `docs/technical` 的链接：
+
+- 初版可保留原始链接文本
+- 或显示为不可跳转的 repo path
+- 后续再扩展为跨文档区导航
+
+### 搜索
+
+初版使用纯前端本地搜索。
+
+索引字段：
+
+- title
+- slug
+- group
+- headings
+- markdown 正文纯文本
+
+搜索策略：
+
+- 页面首次加载后异步加载所有 `docs/technical` Markdown
+- 生成内存索引
+- 用户输入时本地过滤
+- 简单打分即可：
+  - 标题命中权重最高
+  - heading 命中其次
+  - 文件名 / slug 命中其次
+  - 正文命中最低
+
+搜索结果展示：
+
+- 文档标题
+- 分组
+- 命中的 heading 或正文摘要
+- 点击跳转到文档
+
+当前只有 13 篇文档，不需要 Lunr、Fuse 或后端搜索。后续文档数量显著增长时，再考虑引入轻量搜索库。
+
+### 路由接入
+
+修改：
+
+- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
+
+新增 lazy import：
+
+```ts
+const Docs = lazy(() => import('./pages/Docs/Docs'))
+```
+
+公开路由：
+
+```ts
+const publicPaths = new Set(['/', '/earth', '/docs'])
+```
+
+注意：`/docs/:slug` 不能只用精确匹配 `Set`。
+
+建议改为：
+
+```ts
+const isPublicRoute =
+  window.location.pathname === '/' ||
+  window.location.pathname === '/earth' ||
+  window.location.pathname === '/docs' ||
+  window.location.pathname.startsWith('/docs/')
+```
+
+新增 routes：
+
+```tsx
+<Route path="/docs" element={<Docs />} />
+<Route path="/docs/:slug" element={<Docs />} />
+```
+
+### 样式
+
+建议独立 `Docs.css`，不依赖 admin 页面布局。
+
+核心样式要求：
+
+- 文档正文最大宽度控制在适合阅读的范围
+- 表格横向滚动，不撑破布局
+- 代码块横向滚动
+- 左侧导航固定或 sticky
+- 右侧 TOC sticky
+- 移动端隐藏右侧 TOC，导航折叠
+- 搜索结果浮层或独立面板不遮挡正文阅读
+
+注意：
+
+- 不做营销 hero
+- 不做卡片堆叠式首页
+- 首页第一屏应直接是文档入口和内容，而不是宣传页
+
+## 实施阶段
+
+### Phase 1：基础文档站
+
+目标：
+
+- `/docs` 可公开访问
+- 能看到 `docs/technical` 文档列表
+- 能打开每篇 Markdown
+- 能基本渲染标题、段落、列表、代码块、表格
+
+任务：
+
+- 新增 `Docs` 页面
+- 新增 docs registry
+- 接入 Vite raw Markdown loading
+- 接入 `/docs` 和 `/docs/:slug`
+- 加入公开路由白名单
+- 初版 CSS 布局
+
+验收：
+
+- 未登录访问 `/docs` 不跳转登录
+- `/docs/earth-layer-style-reference` 可打开样式参考文档
+- `/docs/backend-collectors` 可打开后端采集器文档
+- 构建通过：`source ~/.zshrc && bun run build`
+
+### Phase 2：搜索与 TOC
+
+目标：
+
+- 支持本地搜索所有 technical 文档
+- 当前文档右侧显示目录
+- 搜索结果可跳转
+
+任务：
+
+- 实现 heading parser
+- 实现 TOC 组件
+- 实现 search index
+- 搜索结果显示文档标题、分组和摘要
+- 当前文档标题与 active nav 高亮
+
+验收：
+
+- 搜索 `Fresnel` 能找到 Earth 图层样式文档
+- 搜索 `collector` 能找到 backend collectors
+- 点击搜索结果进入对应文档
+- 右侧 TOC 点击后滚动到对应 heading
+
+### Phase 3：链接清理与文档体验
+
+目标：
+
+- Markdown 内部链接在 docs 站内自然跳转
+- 长表格、代码块、绝对路径链接的显示更友好
+
+任务：
+
+- 转换 `docs/technical/*.md` 相对链接
+- 转换 repo 内 technical 文档绝对路径
+- 外链新窗口打开
+- 文件路径链接以代码样式显示
+- 增强空状态和 404
+
+验收：
+
+- 从 `docs/technical/README.md` 点击 technical 文档链接进入 `/docs/:slug`
+- 不支持的 repo 内路径不会导致前端崩溃
+- 外部链接行为正常
+
+### Phase 4：文档内容整理
+
+目标：
+
+- `docs/technical` 的首页适合作为公开手册入口
+- 每篇文档标题、摘要和分类清晰
+
+任务：
+
+- 检查每篇文档是否有唯一 h1
+- 给 README 补公开手册导览
+- 必要时补文档摘要
+- 保持文档内容仍然服务开发维护，不改成营销语气
+
+验收：
+
+- `/docs` 首页能说明各技术文档用途
+- 左侧分类和 README 内容一致
+- 没有明显重复、过期或找不到的主入口
+
+## 需要改动的文件
+
+预计新增：
+
+- `frontend/src/pages/Docs/Docs.tsx`
+- `frontend/src/pages/Docs/Docs.css`
+- `frontend/src/pages/Docs/docs-content.ts`
+- `frontend/src/pages/Docs/docs-search.ts`
+
+预计修改：
+
+- `frontend/src/App.tsx`
+- `frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx` 或新增 docs 专用 wrapper
+- `docs/technical/README.md`
+
+可选修改：
+
+- `frontend/src/index.css`，只放全局极少量 docs shell reset 时才需要
+- `docs/CHANGELOG.md`，实施完成后记录
+- `docs/version-history.md`，若进入版本发布流程再更新
+
+## 风险与注意事项
+
+### 构建路径风险
+
+Vite 从 `frontend/src` 读取 `../../../docs/technical/**/*.md` 时，需要确认开发和生产构建都可解析。
+
+缓解：
+
+- 使用相对路径 glob
+- 构建验证必须跑 `source ~/.zshrc && bun run build`
+- 不使用运行时 `fetch('/docs/...')` 读取仓库文件，避免生产环境缺文件
+
+### Markdown 能力不足
+
+现有 `MarkdownRenderer` 是轻量实现，可能不完整支持所有 GitHub Markdown。
+
+缓解：
+
+- 初版优先覆盖当前 `docs/technical` 实际用到的语法
+- 若后续需要脚注、嵌套列表、复杂代码高亮，再考虑引入 `react-markdown` 等依赖
+
+### Bundle 体积
+
+把所有 Markdown 打进前端 bundle 会增加体积。
+
+当前文档数量少，风险可接受。
+
+缓解：
+
+- 使用 lazy page chunk
+- Markdown loader 保持异步
+- 搜索索引在 `/docs` 页面内初始化，不影响 `/earth` 和 admin 首屏
+
+### 公开内容边界
+
+`docs/technical` 会被公开展示，需要避免包含密钥、内部机器地址、临时方案或不应公开的操作细节。
+
+缓解：
+
+- 实施前快速审阅 `docs/technical`
+- 暂不公开 `docs/plans` 和 `docs/deprecated`
+- 以后如需公开更多文档，先建立 allowlist
+
+## 验收清单
+
+- `/docs` 未登录可访问
+- `/docs/:slug` 未登录可访问
+- `/docs` 不影响 `/earth`
+- 未登录访问 admin 仍然跳登录
+- 左侧导航包含所有 `docs/technical` 文档
+- 文档按 Overview / Earth / Frontend / Backend / Agents / Ops 分类
+- Markdown 表格正常显示并可横向滚动
+- 代码块正常显示并可横向滚动
+- 搜索可搜索标题、heading 和正文
+- 搜索结果点击可跳转
+- 当前文档 TOC 可跳转
+- 不存在的 slug 显示 docs 404
+- `source ~/.zshrc && bun run build` 通过
+
+## 后续增强
+
+- 给文档页面增加复制 heading 链接按钮
+- 给代码块增加复制按钮
+- 增加“上一页 / 下一页”导航
+- 增加最近更新信息
+- 从 git metadata 读取文档更新时间
+- 引入轻量全文搜索库
+- 支持 plans / deprecated 独立分区
+- 增加页面内反馈入口

docs/technical/README.md

@@ -1,26 +0,0 @@
-# Technical Docs
-
-这里放“当前实现和当前结构”的文档，重点回答：
-
-- 现在代码是怎么组织的
-- 当前入口在哪
-- 状态和组件如何工作
-- 后续改动应该沿着哪条实现边界继续走
-
-适合放入这里的内容：
-
-- 前端上下文
-- Earth 前端结构
-- 后端运行控制
-- collector 现状
-- 采集格式约定
-
-不适合放入这里的内容：
-
-- 尚未完成的 roadmap
-- 未来迭代方案
-- 大范围重构计划
-
-这些应放入：
-
-- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)