release: bump version to 0.74.4

Release 0.58.0 includes the Earth high-precision boundary PMTiles/MVT pipeline, standardized Earth boundary source collectors, China POV boundary configuration templates, and removal of the legacy low-precision GeoJSON fallback. It also adds Earth news target-location queueing/archive support, fixes datasource task status visibility, documents the Earth surface depth-spacing rules that prevent far-zoom z-fighting snow/black blocks, and updates bilingual operations/developer docs.

.claude/commands/cleanup.md

@@ -1,139 +0,0 @@
----
-description: 审查当前工作区未提交代码中的垃圾代码，并在不影响逻辑的前提下自动清理
-argument-hint: 可选：指定要检查的文件或目录（默认检查所有未提交修改）
-allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
----
-
-# /cleanup — 垃圾代码审查与清理
-
-分析当前工作区（git diff）中的未提交代码，找出并修复常见垃圾代码，**不得改变任何运行逻辑**。
-
-## 检查范围
-
-若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。
-
-## 节省上下文规则
-
-优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文：
-
-```bash
-git diff --name-only HEAD
-git diff --unified=0 HEAD -- <path>
-git diff --check
-rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
-```
-
-只有 focused diff 不足以安全判断或修改时，才读取完整文件。
-
-## 审查清单
-
-按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）：
-
-### 1. 重复逻辑 (Duplicate Logic)
-- 完全相同或高度相似的代码块在多处出现
-- 同一函数/方法被多个地方各自实现，已有公共版本未被复用
-- 相同的 DOM 查询、正则、模板字符串在同一文件重复
-
-### 2. Magic Numbers / Magic Strings
-- 裸数字直接参与计算（如偏移量、时间、尺寸、阈值），没有命名常量
-- 硬编码字符串（如 id 名、状态值、URL 片段）散落在逻辑中
-- 例外：`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
-
-### 3. 命名问题
-- 含义不明的缩写变量（如 `or_`, `tmp2`, `x2`）
-- 命名与实际用途不符
-- 同一概念在不同地方用不同名字表达
-
-### 4. 死代码 / 无效代码
-- 注释掉的旧代码块（3行以上）
-- 声明后从未使用的变量/参数/导入
-- 永远不会执行的条件分支
-
-### 5. 代码风格问题
-- 尾部空白字符（trailing whitespace）
-- 同一文件内风格不一致（如混用单双引号、缩进不统一）
-- 空行使用不一致（连续多个空行等）
-
-### 6. 其他常见问题
-- 私有辅助函数应被 export 但没有，导致调用方重复实现
-- 类型/接口重复定义
-- 过于冗长的条件表达式可以简化（不改逻辑）
-
-## 执行步骤
-
-### Step 1 — 获取待检查文件列表
-
-```bash
-# 无参数时：获取所有未提交修改
-git diff HEAD --name-only
-
-# 有参数时：用 $ARGUMENTS 过滤
-```
-
-### Step 2 — 逐文件阅读并分析
-
-先从 focused diff 开始：
-
-```bash
-git diff --unified=0 HEAD -- <file>
-```
-
-用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。
-
-### Step 3 — 报告问题清单
-
-在修改前，先以列表形式输出所有发现的问题：
-
-```
-发现 N 个问题：
-
-[文件] js/foo.js
-  · L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询，可提取到 getPanel()
-  · L91: Magic number — 硬编码 14 作为偏移量，应命名为 TOOLTIP_OFFSET
-
-[文件] js/bar.js
-  · L12: 命名问题 — 变量 `or_` 语义不明，应命名为 outerR/outerG/outerB
-  ...
-```
-
-如果没有发现问题，直接输出"未发现垃圾代码，当前代码质量良好。"并停止。
-
-### Step 4 — 执行修复
-
-对每个问题，使用 Edit 工具进行**最小化修改**：
-
-- **重复逻辑**：提取为共享常量/函数，更新所有调用点
-- **Magic number**：在文件顶部或逻辑附近声明 `const NAME = value`，替换所有引用
-- **命名问题**：重命名变量，更新所有使用处
-- **死代码**：直接删除
-- **尾部空白/风格**：修正
-- **未 export 的函数**：添加 `export`，在调用方改为导入（不重复实现）
-
-**修复原则：**
-- 只改在审查清单中发现的问题，不做额外优化
-- 每次 Edit 只修改确实有问题的行，保持 diff 最小
-- 改完后用 `grep` 验证旧的坏代码已消失
-- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具
-
-### Step 5 — 输出总结
-
-```
-清理完成：
-
-修复了 N 个问题：
-  ✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
-  ✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14（4处引用）
-  ✓ controls.js — export updateLayerButtonState，移除 main.js 中的重复实现
-  ...
-
-未修改的问题（需人工确认）：
-  ! foo.js L45 — 注释代码块较长，建议手动确认是否可删除
-```
-
-## 约束
-
-- **禁止**改变函数签名、接口定义、导出 API（除非问题正是私有函数应被 export）
-- **禁止**添加新功能、新抽象、新参数
-- **禁止**修改注释内容（只删除注释掉的死代码）
-- **禁止**修改测试文件逻辑
-- 如果一个 Magic number 的语义不完全确定，**跳过**，在总结中标记为"需人工确认"

.claude/commands/docs.md

@@ -1,170 +0,0 @@
----
-description: 分析本次 git 变更，在 docs/technical/zh/ 中新建或更新对应的技术文档
-argument-hint: 可选：指定要记录的主题，或留空自动从 git diff 推断
-allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
----
-
-# /docs — 技术文档写入工作流
-
-## 目标
-
-根据当前 git 变更（或用户指定主题）在 `docs/technical/zh/` 中写入或更新技术文档，记录**为什么**这样做，而不只是记录做了什么。
-
-## 执行步骤
-
-### Step 1 — 理解变更范围
-
-```bash
-git diff HEAD --stat          # 变更文件一览
-git diff HEAD --name-only     # 变更文件列表
-git log --oneline -10         # 近期 commit 上下文
-```
-
-若 `$ARGUMENTS` 指定了主题，优先聚焦该主题；否则从文件列表和 diff stat 推断变更主题。不要默认读取完整仓库 diff；只对决定文档主题所需的文件读取 focused diff：
-
-```bash
-git diff HEAD -- <path>
-rg -n "class |def |function |export |router|@router|interface |type " <path>
-```
-
-### Step 2 — 确认文档范围
-
-分析变更，判断：
-
-1. **应写几篇文档**：单一主题写一篇，跨领域变更可拆分（如后端性能优化 + 运维启动脚本分开写）
-2. **是新建还是更新**：检查 `docs/technical/zh/` 中是否已有相关文档
-3. **文档命名**：按 `领域-主题-副题.md` 格式，全小写，用连字符，如：
-   - `backend-datasources-api-performance.md`
-   - `ops-planet-sh-startup.md`
-   - `earth-bgp-context.md`
-
-```bash
-ls docs/technical/zh/          # 查看现有文档
-```
-
-**先输出写作计划供用户确认**（若变更明确且范围小，可直接执行）：
-
-```
-文档计划：
-  新建：docs/technical/zh/ops-planet-sh-startup.md — planet.sh 启动性能优化
-  更新：docs/technical/zh/backend-datasources-api-performance.md — 补充并行化细节
-```
-
-### Step 2.5 — 覆盖范围检查
-
-写文档前必须按变更类型检查配套文档，不要只更新一篇专题文档：
-
-- 用户可见流程变化：更新 `docs/technical/zh/manual.md`，通常也更新 `docs/technical/zh/quickstart.md`。
-- `manual.md`、`quickstart.md` 这类用户手册存在英文版时，同步更新 `docs/technical/en/...`，至少避免英文版与中文版互相矛盾。
-- 控制台页面职责、路由入口、表格/抽屉/设置页行为变化：更新 `docs/technical/zh/frontend-admin-frontend-context.md`。
-- Earth 前端行为、HUD、巡航、图层、图例、交互变化：更新 `docs/technical/zh/earth-frontend-context.md`。
-- 新增 Earth 图层、调整 `renderOrder`、半径/高度偏移、深度策略、拾取策略、legend mode、图层面板顺序或启动加载顺序：更新 `docs/technical/zh/earth-render-layer-order.md`。
-- Earth 图层视觉样式、颜色、图例符号语义变化：若影响样式索引，同步更新 `docs/technical/zh/earth-layer-style-reference.md`。
-- 采集器、数据源、凭证、设置页、连接检查、scheduler、后端 API 变化：更新相关后端文档，优先检查 `docs/technical/zh/backend-collectors.md` 和 datasource/settings 专题文档。
-- 如果某个旧 plan 的假设已经被当前实现推翻，在对应 `docs/plans/*.md` 增加现状修正或更新该段，不要让计划文档继续给出相反方向。
-- 新增 technical 文档后，如果需要被发现，更新 `docs/technical/zh/README.md`。
-- 对本次变更提取旧词做 stale search，例如旧 tab 名、旧路由职责、旧认证假设、改名前 UI 文案：
-
-```bash
-rg -n "旧文案|旧路由职责|旧认证假设" docs/technical docs/plans
-```
-
-### Step 3 — 写文档
-
-遵循以下原则：
-
-**记录 WHY，不只记录 WHAT**
-- 好：`将戳文件从 /tmp 移到 ~/.cache/planet/，因为 WSL 重启后 /tmp 被清空`
-- 差：`修改了 AI_PROVIDER_BUILD_STAMP_FILE 的值`
-
-**必须包含的内容**：
-- 背景/问题：改动之前存在什么问题，为什么要改
-- 核心设计决策及其理由
-- 关键代码片段（用 diff 或 before/after 展示）
-- 相关文件列表
-
-**格式要求**：
-- 使用 `##` 和 `###` 分级，不要超过三级
-- 代码块注明语言（python / bash / typescript / sql）
-- 表格用于对比多个选项或列出参数
-- 中文写作，技术术语保留英文原文
-- `docs/technical/zh/` 中的文档不得用英文原文占位；如果存在 `docs/technical/en/` 对应文件，禁止逐字复制成中文文件
-- 中文文档内部链接应指向 `docs/technical/zh/...`，除非明确引用英文专属文档
-
-**文档结构模板**：
-
-```markdown
-# 标题（说明做了什么）
-
-## 背景
-
-为什么要做这个改动，改动前存在什么问题。
-
-## 核心变更
-
-### 子主题一
-
-before/after 或决策说明 + 关键代码
-
-### 子主题二
-
-...
-
-## 相关文件
-
-- `path/to/file.py` — 简短说明
-```
-
-### Step 4 — 验证
-
-- 读一遍写好的文档，确认逻辑清晰、代码片段无明显错误
-- 用 `rg --files` 或 `test -e` 确认文档中的文件路径在项目中真实存在，避免凭记忆判断：
-- 检查中文文档没有误复制英文版：
-
-```bash
-python - <<'PY'
-from pathlib import Path
-same = []
-for en in sorted(Path("docs/technical/en").glob("*.md")):
-    zh = Path("docs/technical/zh") / en.name
-    if zh.exists() and en.read_text() == zh.read_text():
-        same.append(en.name)
-if same:
-    raise SystemExit("identical en/zh docs: " + ", ".join(same))
-print("no identical en/zh docs")
-PY
-```
-
-- 检查中文文档内部链接没有继续指向无语言目录：
-
-```bash
-rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
-```
-
-```bash
-# 对文档中提到的关键路径做快速验证
-ls <mentioned_paths>
-```
-
-如需检查大量链接，优先用确定性提取：
-
-```bash
-rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
-```
-
-### Step 5 — 完成确认
-
-输出摘要：
-
-```
-✓ 新建：docs/technical/zh/ops-planet-sh-startup.md（约 xxx 字）
-✓ 更新：docs/technical/zh/backend-datasources-api-performance.md
-```
-
-## 注意事项
-
-- 不要写流水账式的"改了 A、改了 B、改了 C"，要写改动背后的约束和权衡
-- 不要在文档中引用 PR 号、issue 号、或当前对话——这些会随时间失效
-- 代码片段保持简洁，只保留说明问题的关键部分，省略无关样板代码
-- 如果某个变更已有文档记录，优先在原文档中追加，而不是新建
-- 文档是给未来的开发者看的，假设读者熟悉项目但不了解这次改动的背景

.claude/commands/goal-driven.md

@@ -1,93 +0,0 @@
----
-description: 用 goal-driven 方法推动一个复杂任务持续执行，直到明确成功标准被满足
-argument-hint: 建议填写任务目标；若同时给出成功标准更好
-allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
----
-
-# /goal-driven — 目标驱动执行模式
-
-使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务：先固定目标与成功标准，再持续执行和反复验收，直到标准真正满足。
-
-适用场景：
-
-- 长周期实现任务
-- 高复杂度工程任务
-- 可被明确验收的研究、实现、迁移、验证类工作
-
-不适用场景：
-
-- 纯脑暴
-- 无法定义成功标准的模糊任务
-- 很小的一次性修改
-
-## 输入要求
-
-若 `$ARGUMENTS` 只包含目标，没有成功标准，先补全一版可执行的成功标准再开始。
-
-启动时先输出：
-
-```md
-Goal
-- ...
-
-Criteria for success
-- ...
-
-Plan
-1. ...
-2. ...
-3. ...
-
-Verification
-- ...
-```
-
-## 执行规则
-
-1. 先把任务固化为两个核心块：
-   - `Goal`
-   - `Criteria for success`
-
-2. 成功标准必须尽量客观，可验证，可落地。
-   优先写成：
-   - 需要交付什么
-   - 需要通过哪些测试或验证
-   - 如何判断结果真的完成
-
-3. 进入持续执行循环：
-   - 完成一个阶段
-   - 检查当前结果是否满足成功标准
-   - 若未满足，明确剩余差距并继续推进
-
-4. 任何“完成了”“差不多了”“已实现”之类的结论，都必须经过验证，不能直接接受。
-
-5. 如果验证失败：
-   - 明确指出哪条成功标准没满足
-   - 继续工作，不要把阶段性进展误判为完成
-
-6. 只有在以下情况之一才能停止：
-   - 成功标准已满足
-   - 用户明确要求停止
-
-## 执行风格
-
-- 重证据，轻口头判断
-- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
-- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据
-- 重验收，轻自我感觉
-- 优先用测试、日志、产物、对比结果来证明完成
-- 对长期任务保持“未达标就继续”的节奏
-
-## 简版模板
-
-```md
-Goal: [[[[[在此填写最终目标]]]]]
-
-Criteria for success: [[[[[在此填写成功标准]]]]]
-
-循环执行：
-1. 推进任务
-2. 检查是否满足成功标准
-3. 若未满足，继续工作
-4. 直到满足标准或用户明确停止
-```

.claude/commands/release.md

@@ -1,160 +0,0 @@
----
-description: 发版工作流：根据变更类型决定版本号，更新所有版本文件和 changelog，运行验证，commit 并 push
-argument-hint: 可选：feature | bugfix | 或直接描述本次发布内容
-allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
----
-
-# /release — Planet 发版工作流
-
-## 版本号规则
-
-| 变更类型 | 版本跳动 | 适用场景 |
-|---------|---------|---------|
-| `feature` | `+0.1.0` | 纯新功能，无 bugfix |
-| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合，或以 UI/体验改进为主的迭代 |
-| `bugfix` | `+0.0.1` | 纯 bug 修复，无新功能 |
-| `docs` / `maintenance` / `refactor` | 默认不发版，除非用户明确要求 |
-
-意图混合时以用户明确描述为准；bugfix + 小 feature 混合默认判定为 `improvement`（`+0.0.1`）。
-
-## 必须同步更新的文件
-
-使用 `git rev-parse --show-toplevel` 获取仓库根目录，以下路径均相对于根目录：
-
-- `VERSION`
-- `frontend/package.json`（`"version"` 字段）
-- `pyproject.toml`（`version =` 字段）
-- `uv.lock`（**不要手动编辑**，通过 `uv lock` 重新生成）
-- `docs/CHANGELOG.md`
-- `docs/version-history.md`
-
-## 节省上下文规则
-
-发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取：
-
-```bash
-git status --short
-git diff --stat HEAD
-git diff --name-only HEAD
-rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
-```
-
-除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。
-
-## 执行步骤
-
-### Step 1 — 环境检查
-
-```bash
-git branch --show-current          # 确认在 dev 分支
-git status --short                 # 检查是否有无关的未暂存修改
-cat VERSION                        # 读取当前版本
-```
-
-若当前**不在 `dev` 分支**，停下来告知用户，不要继续。
-
-若存在无关的未暂存修改，列出并询问用户是否一并提交，或先 stash。
-
-### Step 2 — 确定发版类型与新版本号
-
-- 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用
-- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
-- 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`）
-- **先输出发版计划供用户确认**：
-
-```
-发版计划：
-  类型：bugfix
-  版本：0.26.2 → 0.26.3
-  分支：dev
-  将更新：VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
-```
-
-### Step 3 — 更新版本号文件
-
-按顺序更新（每步用 Edit 工具，精确替换，不要重写整个文件）：
-
-1. `VERSION` — 直接替换全部内容为新版本号
-2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
-3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
-4. 运行 `uv lock` 重新生成 `uv.lock`（在仓库根目录下执行）
-
-### Step 4 — 更新 CHANGELOG.md
-
-在文件顶部插入新条目，格式：
-
-```markdown
-## [x.x.x] — YYYY-MM-DD
-
-### ✨ Features / 🐛 Fixes / 🔧 Improvements
-- ...（只列高信号条目，最多 5 条）
-- ...
-
----
-```
-
-日期使用 `date +%Y-%m-%d` 获取今天的日期。
-
-### Step 5 — 更新 docs/version-history.md
-
-- 更新文件头部的"当前开发版本"字段
-- 在时间线表格顶部插入新行：`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
-
-### Step 6 — 验证
-
-针对本次变更范围做最小验证：
-
-- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile <changed_files>`
-- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明）
-- 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
-
-```bash
-cat VERSION
-rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
-```
-
-### Step 7 — 提交前预览
-
-展示将要提交的文件列表：
-
-```bash
-git diff --stat HEAD
-```
-
-再次确认所有必须文件都在变更列表中，**不包含**非预期文件（如调试文件、.env 等）。
-
-### Step 8 — Commit & Push（用户确认后）
-
-```bash
-git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
-# 若有代码变更也一并 stage
-git add <code_files>
-
-git commit -m "release: bump version to x.x.x"
-git tag vx.x.x
-git push origin dev
-git push origin vx.x.x
-```
-
-commit message 固定格式：`release: bump version to x.x.x`
-
-### Step 9 — 完成确认
-
-输出摘要：
-
-```
-✓ 版本号已更新：0.26.2 → 0.26.3
-✓ CHANGELOG 已更新
-✓ version-history 已更新
-✓ uv.lock 已重新生成
-✓ 验证通过
-✓ commit: release: bump version to 0.26.3
-✓ tag: v0.26.3
-✓ 已 push 到 origin/dev
-```
-
-## 注意事项
-
-- `uv.lock` 只能通过 `uv lock` 生成，绝不手动编辑
-- 发版 commit 只包含版本文件 + 本次功能代码，不混入无关改动
-- 若环境中 `uv` 不可用，说明原因并跳过 lockfile 更新，提醒用户手动运行

.codex/skills/docs/SKILL.md

@@ -1,126 +1,73 @@
 ---
 name: docs
-description: Analyze current Planet repo changes and create or update technical documentation under docs/technical/zh. Use when the user asks to write docs, update technical docs, summarize implementation changes into documentation, or port the Claude docs-codex workflow into Codex.
+description: Create or update repository documentation from current code changes. Use when the user asks to write docs, update docs, summarize implementation changes into docs, or check documentation coverage. Load repository-specific coverage rules from docs/documentation-coverage-rules.md when present.
 ---
 
 # Docs
 
-Use this skill when the user asks to create or update Planet technical documentation, especially under `docs/technical/zh/`.
+Use this skill when the task is documentation work: creating, updating, checking, or summarizing docs for code or behavior changes.
 
 ## Goal
 
-Write or update technical docs that explain why a change exists, not only what files changed.
+Write documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep the skill generic; repository-specific rules belong in the repository, not in this skill.
 
-Default target directory:
+## Repository Rules
 
-- `docs/technical/zh/`
+Before deciding scope, check whether the repository has a documentation rules file:
+
+```bash
+test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
+```
+
+If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
 
 ## Workflow
 
-1. Gather change context:
+1. Gather focused context:
 
 ```bash
 git diff HEAD --stat
 git diff HEAD --name-only
 git log --oneline -10
-ls docs/technical/zh/
+rg --files docs
 ```
 
-If the user gives a specific topic, focus on that topic. Otherwise infer the documentation topic from the file list and diff stat. Do **not** read the full repository diff by default; inspect focused diffs only for the files that define the doc topic:
+If the user gives a topic, focus on that topic. Otherwise infer the doc topic from changed files. Avoid reading large full diffs by default; inspect focused files and symbols:
 
 ```bash
 git diff HEAD -- <path>
 rg -n "class |def |function |export |router|@router|interface |type " <path>
 ```
 
-2. Decide document scope:
+2. Decide scope:
 
-- Use one document for one coherent topic.
-- Split documents when the changes cross meaningful domains, such as backend performance and ops startup behavior.
 - Prefer updating an existing relevant doc over creating a duplicate.
-- Name new files as lowercase hyphenated `domain-topic-detail.md`, for example:
+- Use one document for one coherent topic.
-  - `backend-datasources-api-performance.md`
+- Split documents only when changes cross meaningful domains.
-  - `ops-planet-sh-startup.md`
+- Keep filenames lowercase and hyphenated.
-  - `earth-bgp-context.md`
 
-3. Apply the documentation coverage checklist before writing:
+3. Write the doc:
 
-- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
+- Explain background/problem, design decisions, constraints, and operational impact.
-- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
+- Keep code snippets short and directly relevant.
-- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
+- List related files only when they help future maintainers navigate.
-- Earth frontend behavior changes must update `docs/technical/zh/earth-frontend-context.md`.
+- Use the repository’s existing language, heading style, and naming conventions.
-- Earth layer additions, `renderOrder`, altitude/radius offsets, depth strategy, pointer picking, legend modes, or layer panel/startup ordering must update `docs/technical/zh/earth-render-layer-order.md`.
+- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
-- Earth layer visual style or legend symbol/color semantics should also update `docs/technical/zh/earth-layer-style-reference.md` when that reference is affected.
-- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
-- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
-- If adding a new technical document, add it to `docs/technical/zh/README.md` when it should be discoverable from the technical docs index.
-- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
 
-4. Write the doc in Chinese:
+4. Verify:
 
-- Write Chinese prose for `docs/technical/zh/`.
+- Read the completed doc once for clarity and stale statements.
-- Keep technical identifiers, API paths, config keys, code symbols, and standard product names in English where appropriate.
+- Verify important referenced paths exist with `test -e` or `rg --files`.
-- Use `##` and `###` headings; avoid going deeper than three levels.
+- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
-- Use fenced code blocks with language tags.
+- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
-- Use tables when comparing options or listing parameters.
-
-5. Required content:
-
-- Background/problem: what was wrong before and why the change was needed.
-- Core design decisions and rationale.
-- Key code snippets, preferably before/after or focused excerpts.
-- Related files and what each file contributes.
-
-6. Verification:
-
-- Read the completed doc and check that the reasoning is clear.
-- Verify important referenced paths exist.
-- Use `rg --files` or `test -e` for path existence instead of relying on memory.
-- Run a quick duplicate-language check when editing bilingual docs:
-
-```bash
-python - <<'PY'
-from pathlib import Path
-same = []
-for en in sorted(Path("docs/technical/en").glob("*.md")):
-    zh = Path("docs/technical/zh") / en.name
-    if zh.exists() and en.read_text() == zh.read_text():
-        same.append(en.name)
-if same:
-    raise SystemExit("identical en/zh docs: " + ", ".join(same))
-print("no identical en/zh docs")
-PY
-```
-
-Also check that Chinese docs do not link to the old language-less technical docs path:
-
-```bash
-rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
-```
-
-This command should return no matches.
-
-If checking many links, prefer deterministic extraction:
-
-```bash
-rg -n "\]\(([^)]+)\)" docs/technical/zh/<doc>.md
-```
-
-Also run focused stale-term searches derived from the change, for example:
-
-```bash
-rg -n "old label|old route purpose|obsolete provider assumption" docs/technical docs/plans
-```
 
 ## Hard Constraints
 
-- A file under `docs/technical/zh/` must not be an English source file copied as a placeholder.
+- Do not leave placeholder docs or copied source text pretending to be documentation.
-- Do not leave a Chinese doc with only an English title and English first-screen content.
+- Do not duplicate bilingual files byte-for-byte.
-- When an English counterpart exists in `docs/technical/en/`, never duplicate it byte-for-byte into `docs/technical/zh/`.
+- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
-- Internal links inside `docs/technical/zh/` should point to `docs/technical/zh/...` for Chinese docs, unless intentionally linking to an English-only file.
+- Do not write changelog-style lists without the reasoning, constraints, and tradeoffs behind the change.
-- Do not reference PR numbers, issue numbers, or the current conversation.
+- Keep docs concise enough to maintain.
-- Do not write changelog-style lists like "changed A, changed B, changed C" without the constraints and tradeoffs behind those changes.
-- Keep code snippets concise and relevant.
 
 ## Recommended Output
 
@@ -128,9 +75,9 @@ After editing, summarize:
 
 ```md
 Updated:
-- docs/technical/zh/example.md — what changed
+- path/to/doc.md — what changed
 
 Verified:
-- no identical en/zh docs
+- checks that passed
-- no language-less docs/technical links in zh docs
+- checks that could not be run, if any
 ```

.dockerignore

@@ -2,9 +2,14 @@
 
 !pyproject.toml
 !uv.lock
+!VERSION
+!backend/
+!backend/**
 !aiprovider/
 !aiprovider/**
 
+backend/.env
+backend/.env.*
 aiprovider/.env
 aiprovider/.env.*
 !aiprovider/.env.example

.gitea/workflows/ci.yaml

@@ -0,0 +1,64 @@
+name: ci
+
+on:
+  push:
+    branches:
+      - dev
+      - main
+  pull_request:
+
+env:
+  REGISTRY: gitea.rclaw.top
+  IMAGE_NAMESPACE: linkong/planet
+
+jobs:
+  backend:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        run: curl -LsSf https://astral.sh/uv/install.sh | sh
+      - name: Sync Python dependencies
+        run: ~/.local/bin/uv sync --group dev
+      - name: Run backend smoke tests
+        working-directory: backend
+        run: PYTHONPATH=. "$GITHUB_WORKSPACE/.venv/bin/python" -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
+
+  frontend:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Bun
+        run: curl -fsSL https://bun.sh/install | bash
+      - name: Build frontend
+        working-directory: frontend
+        run: |
+          ~/.bun/bin/bun install --frozen-lockfile
+          ~/.bun/bin/bun run build
+
+  delivery:
+    runs-on: ubuntu-latest
+    needs:
+      - backend
+      - frontend
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Helm
+        run: |
+          mkdir -p "$HOME/.local/bin"
+          curl -fsSL https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o /tmp/helm.tar.gz
+          tar -xzf /tmp/helm.tar.gz -C /tmp
+          mv /tmp/linux-amd64/helm "$HOME/.local/bin/helm"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+      - name: Docker build smoke
+        run: |
+          docker build -t "$REGISTRY/$IMAGE_NAMESPACE/frontend:${GITHUB_SHA}" ./frontend
+          docker build -t "$REGISTRY/$IMAGE_NAMESPACE/backend:${GITHUB_SHA}" -f backend/Dockerfile .
+          docker build -t "$REGISTRY/$IMAGE_NAMESPACE/aiprovider:${GITHUB_SHA}" -f aiprovider/Dockerfile .
+      - name: Helm template smoke
+        run: |
+          helm lint deploy/helm/planet
+          helm template planet-staging deploy/helm/planet \
+            --namespace planet-staging \
+            -f deploy/helm/planet/values.single-node.yaml \
+            --set image.tag="${GITHUB_SHA}" >/tmp/planet-rendered.yaml

.gitea/workflows/deploy-staging.yaml

@@ -0,0 +1,67 @@
+name: deploy-staging
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+
+env:
+  REGISTRY: gitea.rclaw.top
+  IMAGE_NAMESPACE: linkong/planet
+  RELEASE_NAME: planet-staging
+  NAMESPACE: planet-staging
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install deploy tools
+        run: |
+          mkdir -p "$HOME/.local/bin"
+          curl -fsSL https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o /tmp/helm.tar.gz
+          tar -xzf /tmp/helm.tar.gz -C /tmp
+          mv /tmp/linux-amd64/helm "$HOME/.local/bin/helm"
+          curl -fsSL https://dl.k8s.io/release/v1.30.5/bin/linux/amd64/kubectl -o /tmp/kubectl
+          install -m 0755 /tmp/kubectl "$HOME/.local/bin/kubectl"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+      - name: Configure kubeconfig
+        run: |
+          mkdir -p "$HOME/.kube"
+          printf "%s" "${{ secrets.KUBE_CONFIG_STAGING }}" | base64 -d > "$HOME/.kube/config"
+          chmod 600 "$HOME/.kube/config"
+      - name: Deploy Helm release
+        run: |
+          kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
+          helm upgrade --install "$RELEASE_NAME" deploy/helm/planet \
+            --namespace "$NAMESPACE" \
+            -f deploy/helm/planet/values.single-node.yaml \
+            --set global.imageRegistry="$REGISTRY" \
+            --set global.imageNamespace="$IMAGE_NAMESPACE" \
+            --set image.tag="${GITHUB_SHA}"
+      - name: Wait for rollout
+        run: |
+          kubectl rollout status deployment/planet-frontend -n "$NAMESPACE" --timeout=180s
+          kubectl rollout status deployment/planet-backend -n "$NAMESPACE" --timeout=180s
+          kubectl rollout status deployment/planet-aiprovider -n "$NAMESPACE" --timeout=180s
+      - name: Smoke test services
+        run: |
+          kubectl run planet-smoke-${GITHUB_RUN_NUMBER} \
+            --rm -i --restart=Never \
+            --namespace "$NAMESPACE" \
+            --image=curlimages/curl:8.11.1 \
+            --command -- sh -c '
+              set -eu
+              curl -fsS http://planet-frontend:3000/ >/dev/null
+              curl -fsS http://planet-frontend:3000/health >/dev/null
+              curl -fsS http://planet-frontend:3000/api/health >/dev/null
+              curl -fsS http://planet-backend:8000/health >/dev/null
+              curl -fsS http://planet-aiprovider:8010/health >/dev/null
+            '
+      - name: Collect diagnostics on failure
+        if: failure()
+        run: |
+          kubectl get all -n "$NAMESPACE" -o wide || true
+          kubectl describe pods -n "$NAMESPACE" || true
+          kubectl logs -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE_NAME" --all-containers --tail=200 || true

.gitea/workflows/release.yaml

@@ -0,0 +1,58 @@
+name: release
+
+on:
+  push:
+    branches:
+      - main
+    tags:
+      - "v*"
+  workflow_dispatch:
+
+env:
+  REGISTRY: gitea.rclaw.top
+  IMAGE_NAMESPACE: linkong/planet
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Resolve image tags
+        id: meta
+        run: |
+          echo "sha_tag=${GITHUB_SHA}" >> "$GITHUB_OUTPUT"
+          if printf "%s" "${GITHUB_REF}" | grep -q '^refs/tags/v'; then
+            echo "release_tag=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
+          else
+            echo "release_tag=" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Login to registry
+        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login "$REGISTRY" -u "${{ secrets.REGISTRY_USER }}" --password-stdin
+      - name: Build and push images
+        run: |
+          for service in frontend backend aiprovider; do
+            case "$service" in
+              frontend)
+                dockerfile="./frontend/Dockerfile"
+                context="./frontend"
+                ;;
+              backend)
+                dockerfile="backend/Dockerfile"
+                context="."
+                ;;
+              aiprovider)
+                dockerfile="aiprovider/Dockerfile"
+                context="."
+                ;;
+            esac
+
+            image="$REGISTRY/$IMAGE_NAMESPACE/$service:${{ steps.meta.outputs.sha_tag }}"
+            docker build -t "$image" -f "$dockerfile" "$context"
+            docker push "$image"
+
+            if [ -n "${{ steps.meta.outputs.release_tag }}" ]; then
+              release_image="$REGISTRY/$IMAGE_NAMESPACE/$service:${{ steps.meta.outputs.release_tag }}"
+              docker tag "$image" "$release_image"
+              docker push "$release_image"
+            fi
+          done

.gitignore

@@ -8,6 +8,7 @@
 .env
 .env.local
 .env.*.local
+config/earth-boundary-sources.local.json
 *.pem
 *.key
 *.crt
@@ -24,11 +25,14 @@ __pycache__/
 build/
 develop-eggs/
 dist/
-downloads/
+downloads/*
+!downloads/usbipd-win/
+downloads/usbipd-win/*
+!downloads/usbipd-win/usbipd-win-5.3.0.msi
 eggs/
 .eggs/
-lib/
+/lib/
-lib64/
+/lib64/
 parts/
 sdist/
 var/
@@ -150,3 +154,9 @@ temp/
 # Runtime Data
 # ----------------------
 data/ai/bgp-briefs/
+data/earth-boundary-sources/
+
+# Generated Earth boundary tile artifacts. Keep source configs and builders in
+# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
+# committing thousands of generated tile files.
+frontend/public/earth/data/boundaries/

AGENTS.md

@@ -0,0 +1,180 @@
+# AGENTS.md
+
+**Planet agent harness. Defines behavior for coding agents working in this repository.**
+
+---
+
+## Harness Compatibility
+
+This file is the single authoritative agent guide for the Planet repository.
+The older lowercase `agents.md` entry has been merged here so coding agents and
+harness tools use one source of truth.
+
+### Source Of Truth
+
+- `rules.md` is the mandatory repository rule source. Always load `core`,
+  `security`, and `workflow`; load only task-relevant modules after that.
+- `AGENTS.md` defines the local agent operating mode and evidence gates.
+- `project_context.md` is background, not a rule source. Prefer newer
+  implementation docs when it disagrees with current code.
+- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
+  goal-driven work, and release.
+- Do not duplicate long workflow text across harness files. Durable constraints
+  belong in `rules.md`; task procedures belong in skills or scripts.
+
+Read these files before changing code:
+
+1. `rules.md`
+2. `AGENTS.md`
+3. `project_context.md`
+4. `README.md`
+5. `docs/HARNESS.md`
+6. `CODEMAP.md`
+
+For documentation work, also read `docs/documentation-coverage-rules.md`.
+
+### Start Safely
+
+Before broad edits:
+
+```bash
+git status --short
+scripts/harness/doctor.sh
+```
+
+Use focused context commands before reading large files:
+
+```bash
+rg -n "<symbol-or-term>" <path>
+git diff --stat HEAD
+git diff --name-only HEAD
+git diff --unified=0 HEAD -- <path>
+```
+
+Preserve user changes already present in the worktree.
+
+### Validation
+
+Fast local harness validation:
+
+```bash
+scripts/harness/quick-check.sh
+```
+
+Full local validation:
+
+```bash
+scripts/harness/validate.sh
+```
+
+`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
+unless disabled by its documented environment flags. Docker image smoke builds
+are intentionally opt-in:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
+current non-interactive environment first. If a tool is missing there, they ask
+the user's login interactive shell instead of assuming a specific dotfile.
+
+### High-Risk Areas
+
+- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
+  `destroy` cleanup.
+- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
+- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
+  rendered smoke evidence for public pages, auth guards, authenticated admin
+  route/section availability, safe navigation/search/tab interactions, mobile
+  layout, and 125% / 150% zoom, not only a build.
+- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
+  one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
+  intermediate wrappers keep `min-height: 0`, and only the intended child owns
+  scrolling.
+- `aiprovider` is a protocol/provider adapter; keep business prompts and product
+  workflows in the backend.
+- Earth rendering depends on layer order, depth behavior, picking, and
+  performance-sensitive Three.js code.
+- Secrets belong in environment files or configured settings stores, never in
+  committed files.
+- Backend service code must use structured logging instead of `print()` or
+  debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
+
+### Conflict Policy
+
+Existing project rules and workflows win. If new harness guidance conflicts with
+`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
+behavior and document the compatibility note in `docs/harness-audit.md` or
+`docs/HARNESS.md`.
+
+---
+
+## Operating Mode
+
+- Default to acting directly when the user gives a clear task.
+- Ask before acting only when the missing decision is risky, cannot be
+  discovered from repository context, and no conservative assumption is safe.
+- Read relevant files before editing.
+- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
+  focused file reads, tests, builds, linters, and harness scripts.
+- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
+  release work, and documentation unless the task requires it.
+
+---
+
+## Evidence Gates
+
+- Visual inputs are blocking evidence. If the user provides a screenshot, image,
+  mock, browser capture, or visual reference, obtain evidence from the artifact
+  before interpreting intent or editing code.
+- Path resolution is part of the task. If the path cannot be opened, first try
+  reasonable local equivalents such as WSL/Windows path conversion,
+  workspace-relative lookup, absolute paths, and attached-file locations.
+- Never guess from prompt text, filenames, previous context, logs, OCR, or
+  memory when a visual artifact was provided but cannot be accessed.
+- OCR is acceptable evidence for text-only visual questions or non-multimodal
+  environments; state that OCR was used as the fallback. Layout, color, spacing,
+  pixel, and rendering issues need real visual inspection or a clear limitation
+  note.
+- If a visual artifact still cannot be inspected, say so and pause that
+  visual-dependent part of the work.
+- Claims of completion need evidence: a relevant test, build, lint, screenshot,
+  diff, direct file check, or harness result.
+- For UI and rendering changes, verify the rendered result when local tooling
+  allows it.
+
+---
+
+## Communication
+
+- Match the user's language. Use Chinese for Chinese requests unless the user
+  asks otherwise.
+- Keep updates short and specific: what is being inspected, edited, or verified.
+- Final responses should summarize changed files and verification, with blockers
+  stated plainly.
+- Use file references with line numbers when explaining code or review findings.
+
+---
+
+## Quality Bar
+
+- Prefer existing project patterns over new abstractions.
+- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
+  a stable path exists.
+- Centralize prompts, constants, defaults, and shared request/response handling.
+- Do not add secrets, generated runtime output, or local environment files.
+- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
+- Run the smallest relevant verification for the changed scope and report
+  anything skipped.
+
+---
+
+## Prohibited
+
+- Do not skip visual evidence handling when a visual artifact was provided.
+- Do not preserve obsolete harness files just because they already exist.
+- Do not invent behavior not present in code, docs, or verified external
+  sources.
+- Do not rewrite unrelated files during cleanup.
+- Do not mark a task complete without checking concrete success criteria.

CODEMAP.md

@@ -0,0 +1,109 @@
+# Code Map
+
+This map gives agents and maintainers a quick orientation without replacing the
+deeper architecture docs. Current implementation docs under `docs/technical/`
+are the source of detail for specific subsystems.
+
+## Top-Level Areas
+
+| Path | Role | Notes |
+| --- | --- | --- |
+| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
+| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
+| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
+| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
+| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
+| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
+| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
+| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
+| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
+
+## Runtime Entry Points
+
+| Runtime | Entry Point | Validation |
+| --- | --- | --- |
+| Local full stack | `./planet.sh start` | `./planet.sh health` |
+| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
+| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
+| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
+| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
+| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
+
+## Ownership Boundaries
+
+- Backend owns business state, auth, evidence collection, prompt selection, AI
+  task orchestration, and database persistence.
+- `aiprovider` owns provider identity, request adapter style, model gateway
+  retries, and health/status endpoints only.
+- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
+  and client-side state that mirrors backend truth.
+- Web Earth rendering changes must preserve documented layer order, altitude
+  offsets, picking behavior, legend semantics, and performance constraints.
+- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
+  wrapping it from harness scripts instead of duplicating its internals.
+- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
+  see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
+  interactive command path without hardcoding `.zshrc`.
+
+## Validation Commands
+
+```bash
+scripts/harness/doctor.sh
+scripts/harness/security-check.sh
+scripts/harness/backend-rules-check.sh
+scripts/harness/frontend-rules-check.sh
+scripts/harness/docs-consistency-check.sh
+scripts/harness/quick-check.sh
+scripts/harness/validate.sh
+./planet.sh health
+```
+
+CI-equivalent local checks:
+
+```bash
+cd backend
+uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
+
+cd frontend
+bun install --frozen-lockfile
+bun run build
+PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
+```
+
+The frontend smoke covers public routes, unauthenticated admin guards,
+login-error handling, the Earth iframe entry, and authenticated `super_admin`
+admin route/section rendering with mocked API data. Authenticated admin checks
+run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
+check for accidental global horizontal overflow. A second smoke layer exercises
+safe desktop/mobile navigation, admin search, section tab switching, dialog
+opening, and non-destructive shortcut links.
+
+Optional delivery smoke, when Docker and Helm are available:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+## Deeper Docs
+
+| Topic | Start Here |
+| --- | --- |
+| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
+| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
+| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
+| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
+| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
+| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
+| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
+| Documentation rules | `docs/documentation-coverage-rules.md` |
+| Harness workflow | `docs/HARNESS.md` |
+
+## Known Sharp Edges
+
+- `project_context.md` is static background for agents. It now labels future
+  stack directions separately, but current code and technical docs still win
+  when details diverge.
+- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
+  active local development shape.
+- Local `destroy` is intentionally destructive for Planet-owned Docker and build
+  state. Never run it as a validation shortcut.

README.md

@@ -8,68 +8,54 @@
 
 ## 系统架构
 
+当前仓库的核心形态是“Web Earth 可视化 + React 运维台 + FastAPI 数据与 AI 编排后端 + 独立模型适配层”。物理大屏与 UE 客户端仍是长期方向，但不再作为本地开发和当前发布的必需运行单元。
+
 ```
-┌─────────────────────────────────────────────────────────────────────────┐
+┌─────────────────────────────────────────────────────────────────────┐
-│                           物理大屏展示层                                  │
+│                         浏览器展示与运维层                            │
-│  ┌─────────────────────────────────────────────────────────────────┐    │
+│  ┌──────────────────────────────┐  ┌──────────────────────────────┐ │
-│  │         偏振片3D大屏 (2m×3m, 4K, 120Hz, 眼镜式)                 │    │
+│  │ Web Earth                    │  │ React 运维台                  │ │
-│  │  ┌─────────────────────────────────────────────────────────┐    │    │
+│  │ frontend/public/earth        │  │ frontend/src                  │ │
-│  │  │              虚幻引擎 UE5 客户端                          │    │    │
+│  │ Three.js 地球 / HUD / 新闻    │  │ 数据源 / 告警 / AI 设置       │ │
-│  │  │  ├── 3D地球渲染 (Cesium for UE)                          │    │    │
+│  │ 国界精度 / 品牌内容配置       │  │ 提示词配置 / 用户与系统配置   │ │
-│  │  │  ├── 算力点可视化 (GPU集群、智算中心)                     │    │    │
+│  └──────────────────────────────┘  └──────────────────────────────┘ │
-│  │  │  ├── 连接弧线 (光缆、路由、数据流向)                      │    │    │
+└─────────────────────────────────────────────────────────────────────┘
-│  │  │  ├── 粒子效果 (数据流动、告警提示)                        │    │    │
+                       │ REST / WebSocket
-│  │  │  └── 自动巡航相机 + 交互控制                              │    │    │
+                       ▼
-│  │  └─────────────────────────────────────────────────────────┘    │    │
+┌─────────────────────────────────────────────────────────────────────┐
-│  └─────────────────────────────────────────────────────────────────┘    │
+│                       FastAPI 业务与编排后端                          │
-└─────────────────────────────────────────────────────────────────────────┘
+│  ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
-                                    ▲
+│  │ 数据 API 与认证     │ │ Earth 新闻增强      │ │ 告警与态势简报   │ │
-                                    │ WebSocket (实时推送)
+│  │ JWT / 权限 / 审计   │ │ 位置推断 / 本地化   │ │ BGP / 告警研判   │ │
-                                    │ 120Hz 心跳 / 数据帧同步
+│  └────────────────────┘ └────────────────────┘ └─────────────────┘ │
-                                    ▼
+│  ┌────────────────────┐ ┌────────────────────┐ ┌─────────────────┐ │
-┌─────────────────────────────────────────────────────────────────────────┐
+│  │ 系统运行配置        │ │ 默认提示词注册表     │ │ 未来 Agent Runtime│ │
-│                           数据中台服务层 (FastAPI)                        │
+│  │ system_settings     │ │ 代码发布 + DB 覆盖  │ │ 工具/证据/工作流   │ │
-│  ┌─────────────────────────────────────────────────────────────────┐    │
+│  └────────────────────┘ └────────────────────┘ └─────────────────┘ │
-│  │                      API Gateway (Redis 限流)                    │    │
+└─────────────────────────────────────────────────────────────────────┘
-│  └─────────────────────────────────────────────────────────────────┘    │
+        │ SQLAlchemy / Redis Stream            │ 纯净 LLM 调用
-│                                    │                                    │
+        ▼                                      ▼
-│  ┌───────────────────┬──────────────────────────┬──────────────────┐   │
+┌──────────────────────────────┐      ┌──────────────────────────────┐
-│  │    数据采集服务    │      核心业务服务         │    运维管理服务   │   │
+│ PostgreSQL / Redis            │      │ aiprovider                   │
-│  │  ┌─────────────┐  │  ┌─────────────────┐    │  ┌─────────────┐ │   │
+│ 用户、配置、采集结果、新闻     │      │ provider + protocol adapter  │
-│  │  │ 调度中心    │  │  │ WebSocket 服务  │    │  │ 用户管理    │ │   │
+│ Stream、缓存、运行状态         │      │ OpenAI / MiniMax / Ollama 等 │
-│  │  │ (Celery)   │  │  │ (FastAPI)       │    │  │ (JWT Auth)  │ │   │
+└──────────────────────────────┘      └──────────────────────────────┘
-│  │  └─────────────┘  │  └─────────────────┘    │  └─────────────┘ │   │
+        ▲
-│  │  ┌─────────────┐  │  ┌─────────────────┐    │  ┌─────────────┐ │   │
+        │ 采集器 / 外部数据源
-│  │  │ 采集器池    │  │  │ 数据查询 API    │    │  │ 数据源配置  │ │   │
+        ▼
-│  │  │ (10+源)    │  │  │ (REST)          │    │  │ 监控告警    │ │   │
+┌─────────────────────────────────────────────────────────────────────┐
-│  │  └─────────────┘  │  └─────────────────┘    │  └─────────────┘ │   │
+│ RSS 新闻、BGP 观测、公开数据源、后续 WebSearch/OCR/语音识别等工具      │
-│  │  ┌─────────────┐  │  ┌─────────────────┐    │  ┌─────────────┐ │   │
+└─────────────────────────────────────────────────────────────────────┘
-│  │  │ 消息队列    │  │  │ 态势分析引擎    │    │  │ 系统配置    │ │   │
-│  │  │ (Kafka)    │  │  │ (计算/聚合)     │    │  │ 日志审计    │ │   │
-│  │  └─────────────┘  │  └─────────────────┘    │  └─────────────┘ │   │
-│  └───────────────────┴──────────────────────────┴──────────────────┘   │
-└─────────────────────────────────────────────────────────────────────────┘
-                                    ▲
-                                    │ 内部 API 调用
-                                    ▼
-┌─────────────────────────────────────────────────────────────────────────┐
-│                           Web管理端 (React Admin)                        │
-│  ┌─────────────────────────────────────────────────────────────────┐    │
-│  │  登录页 │ 仪表盘 │ 用户管理 │ 数据源配置 │ 任务监控 │ 系统配置   │    │
-│  └─────────────────────────────────────────────────────────────────┘    │
-└─────────────────────────────────────────────────────────────────────────┘
-                                    ▲
-                                    │ PostgreSQL / Redis
-                                    ▼
-┌─────────────────────────────────────────────────────────────────────────┐
-│                           数据存储层                                      │
-│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
-│  │ PostgreSQL  │  │ TimescaleDB │  │    Redis    │  │  MinIO      │    │
-│  │ (用户/配置) │  │ (时序数据)  │  │ (缓存/会话) │  │ (文件存储) │    │
-│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘    │
-└─────────────────────────────────────────────────────────────────────────┘
 ```
 
+架构边界:
+
+- `backend` 负责业务语义、证据收集、提示词选择、AI 任务编排、权限和数据落库。
+- `aiprovider` 只负责把纯净模型请求适配到不同供应商或协议，不内置具体业务提示词。
+- 默认提示词随代码发布并保存在 `backend/app/ai_tasks/default_prompts.json`，运维台可在数据库中保存覆盖值，重置时回到当前代码版本的默认提示词。
+- Earth 新闻保留英文原文，中文展示结果存入 `localizations`，前端默认展示 `zh-CN` 的 `display_title`、`display_summary` 和中文地域/状态文案。
+- Earth LLM 指令、语音识别、多角色态势研判属于后续 Agent Runtime 方向，计划见 [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)。
+
 ## 四大核心要素
 
 | 层级 | 要素 | 描述 |
@@ -87,21 +73,21 @@
 |------|------|------|
 | FastAPI | 0.109+ | Web 框架 |
 | SQLAlchemy | 2.0+ | ORM |
-| Alembic | - | 数据库迁移 |
+| uv | - | Python 依赖与命令运行 |
-| Celery | 5.3+ | 任务队列 |
+| Redis | 7.0+ | 缓存、Stream 与运行协调 |
-| Redis | 7.0+ | 缓存/消息 |
-| Kafka | 3.0+ | 事件流 |
 | PyJWT | - | 认证 |
+| APScheduler / 后台任务 | - | 采集、增强与运行时任务 |
 
 ### 前端 (React Admin)
 
 | 组件 | 用途 |
 |------|------|
 | React 18 | UI 框架 |
-| Ant Design Pro | 管理后台组件 |
+| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
 | Axios | HTTP 客户端 |
 | Socket.io-client | WebSocket 客户端 |
 | ECharts | 统计图表 |
+| Three.js | Earth 3D 地球渲染 |
 | Bun | 前端包管理与脚本运行 |
 
 前端工程统一使用 Bun：
@@ -110,22 +96,16 @@
 - 运行脚本使用 `bun run <script>`
 - 不使用 `npm`、`pnpm`、`yarn`
 
-### 虚幻引擎客户端
+### 大屏与 3D 展示方向
 
-| 组件 | 版本 | 用途 |
+当前发布优先使用浏览器 Web Earth。UE5 / Cesium for Unreal / Niagara 可作为后续物理大屏方向接入，但不是本地开发闭环的必需组件。
-|------|------|------|
-| Unreal Engine 5 | 5.3+ | 3D 渲染引擎 |
-| Cesium for Unreal | 1.5+ | 地理可视化 |
-| Niagara | - | 粒子系统 |
 
 ### 数据库
 
 | 组件 | 用途 |
 |------|------|
 | PostgreSQL 15+ | 关系数据 |
-| TimescaleDB | 时序数据扩展 |
+| Redis 7+ | 缓存、Stream、运行状态 |
-| Redis 7+ | 缓存/会话 |
-| MinIO | S3 兼容存储 |
 
 ### 部署
 
@@ -152,9 +132,9 @@
 | P0 | Epoch AI | 每小时 |
 | P0 | Hugging Face | 每 2 小时 |
 | P0 | GitHub | 每 4 小时 |
-| P0 每日 |
+| P0 | 海底光缆 / IXP / 卫星等基础设施数据 | 每日或按源刷新 |
 | P0 | PeeringDB | 每 2 小时 |
-| P1 | Cloudflare Radar | | TeleGeography | 每小时 |
+| P1 | Cloudflare Radar / TeleGeography | 每小时 |
 | P1 | CAIDA BGPStream | 每 15 分钟 |
 
 ## 项目结构
@@ -166,20 +146,18 @@
 │   │   ├── core/         # 核心配置
 │   │   ├── models/       # 数据模型
 │   │   ├── schemas/      # Pydantic 模型
-│   │   ├── services/     # 业务逻辑
+│   │   ├── services/     # 业务逻辑与 AI 任务编排
-│   │   └── tasks/        # Celery 任务
+│   │   └── ai_tasks/     # 默认提示词与 AI 任务定义
 │   └── tests/
+├── aiprovider/            # 独立模型供应商适配层
 ├── frontend/              # React 管理后台
 │   ├── src/
 │   │   ├── components/   # 组件
 │   │   ├── pages/        # 页面
 │   │   ├── services/     # API 服务
 │   │   └── store/        # 状态管理
-│   └── tests/
+│   ├── public/earth/      # Web Earth 静态应用
-├── unreal/                # UE5 大屏客户端
+│   └── tests/             # 前端测试
-│   ├── Content/
-│   ├── Source/
-│   └── Plugins/
 ├── data/                  # 数据文件
 ├── docs/                  # 文档
 ├── scripts/               # 脚本
@@ -190,11 +168,14 @@
 
 ## 快速启动
 
+入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上，`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx，启动 Docker 服务并配置当前用户的访问权限；需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
+
 ```bash
-# 新机器首次初始化
+# 新机器或空项目首次初始化
-./scripts/bootstrap-dev.sh
+./planet.sh init
-# 会自动安装/检查 uv、bun，并同步 Python/前端依赖
+# 会先准备 Docker / Compose / Buildx，再安装/检查 uv、bun 并同步 Python/前端依赖
 # 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
+# 会启动 PostgreSQL/Redis，并创建表、默认数据源和本地默认用户
 
 # 启动前后端服务
 ./planet.sh start
@@ -210,6 +191,9 @@
 
 # 查看服务状态
 ./planet.sh health
+
+# 删除容器、卷、镜像和本地编译状态，执行前需要输入 Y 确认
+./planet.sh destroy
 ```
 
 前端命令约定：
@@ -236,13 +220,15 @@ bun run build
 
 推荐按下面顺序排查和配置。
 
+端口占用、`iphlpsvc` / portproxy、摄像头和依赖问题的集中排障入口见 [常见问题](/home/ray/dev/linkong/planet/docs/technical/zh/faq.md)。
+
 ### 1. 在 WSL 中启动服务
 
 ```bash
 ./planet.sh start --allow-lan
 ```
 
-这会让前端监听 `0.0.0.0:3000`，后端监听 `0.0.0.0:8000`。
+这会让前端监听 `0.0.0.0:3000`，后端监听 `0.0.0.0:8000`，AI Provider 通过 Docker 发布到 `0.0.0.0:8010`。启动前脚本会检查这三个端口；如果 WSL/Linux 侧无法释放端口，并检测到 Windows 侧 listener 或旧 `portproxy`，会请求管理员 PowerShell 清理。
 
 ### 2. 先确认 WSL 内部服务正常
 
@@ -251,14 +237,16 @@ bun run build
 ```bash
 curl http://localhost:3000
 curl http://localhost:8000/health
-ss -ltnp | grep -E ':3000|:8000'
+curl http://localhost:8010/health
+ss -ltnp | grep -E ':3000|:8000|:8010'
 ```
 
 预期：
 
 - `3000` 返回前端 HTML
 - `8000/health` 返回健康检查 JSON
-- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
+- `8010/health` 返回 AI Provider 健康检查 JSON
+- `ss` 中能看到 `0.0.0.0:3000`、`0.0.0.0:8000` 和 `0.0.0.0:8010`，或 Docker 已发布 `8010`
 
 如果这一步不通，先不要继续做 Windows 转发。
 
@@ -269,42 +257,31 @@ ss -ltnp | grep -E ':3000|:8000'
 ```powershell
 curl http://localhost:3000
 curl http://localhost:8000/health
+curl http://localhost:8010/health
 ```
 
 在常见的 WSL2 开发环境下，Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
 
-### 4. 如果需要让局域网设备访问，再做 Windows 端口转发
+### 4. 如果需要让局域网设备访问，清理端口和防火墙
 
-注意：下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
+`./planet.sh start --allow-lan` 不再启动额外的 Windows 端口转发进程。它直接让开发服务对 `3000` / `8000` / `8010` 开放，并在启动前尝试释放这些端口。端口被 Windows 侧 listener 或旧 `portproxy` 占用时，脚本会请求一次管理员 PowerShell 清理。
 
-先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`：
+如果以前手动配置过持久 `portproxy`，若自动请求被取消，可以手动清理，避免 `iphlpsvc` 继续占用端口：
 
 ```powershell
 netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
 netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
-
+netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8010
-netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
-netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
 ```
 
-再放行 Windows 防火墙：
+脚本会检测 Windows 防火墙是否已放行 `3000` / `8000` / `8010`。如果缺少规则，会触发一次 Windows UAC 管理员 PowerShell 请求来自动创建。若自动请求被取消，也可以手动执行：
 
 ```powershell
 New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
 New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
+New-NetFirewallRule -DisplayName "WSL Planet 8010" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8010
 ```
 
-检查转发规则是否生效：
-
-```powershell
-netsh interface portproxy show all
-```
-
-预期能看到：
-
-- `0.0.0.0:3000 -> 127.0.0.1:3000`
-- `0.0.0.0:8000 -> 127.0.0.1:8000`
-
 ### 5. 查 Windows 局域网 IP，并让其他设备访问
 
 在 Windows PowerShell 中执行：
@@ -319,6 +296,8 @@ ipconfig
 
 - `http://<Windows局域网IP>:3000/earth`
 - `http://<Windows局域网IP>:3000/admin`
+- `http://<Windows局域网IP>:8000/health`
+- `http://<Windows局域网IP>:8010/health`
 
 例如：
 
@@ -327,7 +306,7 @@ ipconfig
 ### 6. 常见现象与判断
 
 - WSL 中 `curl localhost:3000` 能通，但 Windows 访问 `WSL 的局域网 IP:3000` 不通：这是正常现象之一，优先验证 Windows 的 `localhost:3000`
-- Windows `localhost:3000` 能通，但局域网设备访问 `Windows 局域网 IP:3000` 不通：通常缺少 `portproxy` 或防火墙放行
+- Windows `localhost:3000` 能通，但局域网设备访问 `Windows 局域网 IP:3000` 不通：通常是 Windows 防火墙、网络配置或旧 `portproxy` 残留
 - `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`：说明当前 PowerShell 不是提权管理员窗口
 
 ### 7. 本项目一次性验证顺序
@@ -336,14 +315,16 @@ ipconfig
 
 1. WSL 中执行 `curl http://localhost:3000`
 2. WSL 中执行 `curl http://localhost:8000/health`
-3. Windows 中执行 `curl http://localhost:3000`
+3. WSL 中执行 `curl http://localhost:8010/health`
-4. Windows 中执行 `curl http://localhost:8000/health`
+4. Windows 中执行 `curl http://localhost:3000`
-5. 管理员 PowerShell 配置 `portproxy` 和防火墙
+5. Windows 中执行 `curl http://localhost:8000/health`
-6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
+6. Windows 中执行 `curl http://localhost:8010/health`
+7. 按脚本提示完成 Windows 防火墙或端口清理 UAC 请求
+8. 用手机或其他电脑访问 Windows 对外端口，例如 `http://<Windows局域网IP>:3000/earth`
 
 ## 启动容错参数
 
-`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试，并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
+`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康，再验证后端实际连接；`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待，避免重复消耗健康检查预算。
 
 可通过环境变量临时调整:
 
@@ -363,18 +344,27 @@ DATABASE_RETRY_INTERVAL=10 \
 - `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈，默认 `3` 次、`5` 秒
 - `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数，默认 `3`
 - `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数，默认 `3`
-- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔，默认 `10` 次、`2` 秒
+- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔，默认 `60` 次、`2` 秒
 - `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔，默认 `10` 次、`2` 秒
 - `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔，默认 `10` 次、`2` 秒
 
-## AI 接口预留
+## AI 与智能体接口
 
-项目现在采用“两层”设计:
+项目现在采用“三段式”边界:
 
-- 主后端暴露稳定业务接口: `GET /api/v1/ai/provider/status`、`POST /api/v1/ai/situational-awareness/analyze`
+- `backend`: 暴露业务接口，负责选择任务提示词、组织证据、调用工具、保存 AI 设置和结果。
-- 独立 `aiprovider` 服务负责适配具体模型供应商
+- `aiprovider`: 暴露模型网关接口，只负责 provider / protocol 适配，不写入 BGP、新闻、告警等业务提示词。
+- 模型供应商: OpenAI 兼容、MiniMax、Anthropic、Ollama 或其他兼容网关。
 
-这样前端和业务代码不直接依赖 OpenAI、本地模型网关或其他订阅服务，后续切换部署方式只需要调整环境变量。
+这样前端和业务代码不直接依赖某个模型供应商，后续增加 Agent Runtime、Earth 一键 LLM 指令、语音识别或多角色态势研判时，也可以把业务工作流放在后端，而不是污染模型适配层。
+
+当前已落地的 AI 配置能力:
+
+- 运维台 AI 设置可维护 provider、模型、协议、超时、token 等运行配置。
+- 运维台 AI 设置中的“提示词”页可选择不同功能入口，手动覆盖提示词，并一键重置到默认值。
+- 默认提示词随代码发布，位于 [backend/app/ai_tasks/default_prompts.json](/home/ray/dev/linkong/planet/backend/app/ai_tasks/default_prompts.json)。
+- 覆盖值保存在数据库运行配置中，升级代码后可继续保留现场配置，也可重置到新版本默认提示词。
+- 态势摘要、告警研判、新闻本地化等入口应使用各自任务提示词；调用 `aiprovider` 时只传递当前任务所需的 `prompt` / `system_prompt`。
 
 主后端建议配置:
 
@@ -387,39 +377,33 @@ AI_PROVIDER_TIMEOUT_SECONDS=60
 `aiprovider` 服务建议配置:
 
 ```env
-AI_PROVIDER=openai_compatible
+AI_PROVIDER=minimax
-AI_BASE_URL=https://api.openai.com/v1
+AI_PROVIDER_API=anthropic-messages
+AI_BASE_URL=https://api.minimaxi.com/anthropic
 AI_API_KEY=your_api_key
-AI_MODEL=gpt-4o-mini
+AI_MODEL=MiniMax-M2.7
 AI_TIMEOUT_SECONDS=60
 AI_PROVIDER_SERVICE_TOKEN=change_me
 ```
 
-OpenAI 兼容场景推荐使用:
+推荐映射关系:
 
-- `AI_PROVIDER=openai_compatible`
+- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
+- `MiniMax`: `AI_PROVIDER=minimax` + `AI_PROVIDER_API=anthropic-messages`
+- Claude 兼容网关: `AI_PROVIDER=anthropic` + `AI_PROVIDER_API=anthropic-messages`
+- `Ollama`: `AI_PROVIDER=ollama` + `AI_PROVIDER_API=ollama-generate`
 
-Claude 兼容场景推荐使用:
+比如 MiniMax 可以这样配置:
-
-- `AI_PROVIDER=anthropic`
-- `AI_PROVIDER=anthropic_compatible`
-- `AI_PROVIDER=claude_compatible`
-
-Ollama 原生场景推荐使用:
-
-- `AI_PROVIDER=ollama`
-
-比如 MiniMax 或其他 Claude 兼容网关，可以这样配置:
 
 ```env
-AI_PROVIDER=claude_compatible
+AI_PROVIDER=minimax
-AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
+AI_PROVIDER_API=anthropic-messages
+AI_BASE_URL=https://api.minimaxi.com/anthropic
 AI_API_KEY=your_api_key
-AI_MODEL=your-claude-compatible-model
+AI_MODEL=MiniMax-M2.7
 AI_TIMEOUT_SECONDS=60
 AI_MAX_TOKENS=1200
 AI_ANTHROPIC_VERSION=2023-06-01
-AI_PROVIDER_SERVICE_TOKEN=change_me
 ```
 
 如果你要本地直接起模型适配层，项目里已经补了模板:
@@ -427,12 +411,6 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
 - [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
 - [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml)
 
-推荐映射关系:
-
-- `vLLM` / `LM Studio` / `One API`: `AI_PROVIDER=openai_compatible`
-- `MiniMax` / Claude 兼容网关: `AI_PROVIDER=claude_compatible`
-- `Ollama`: `AI_PROVIDER=ollama`
-
 运行与调用补充:
 
 - `./planet.sh start` 默认会启动 `aiprovider`
@@ -442,11 +420,13 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
 
 详细文档:
 
-- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
+- [docs/technical/zh/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/zh/agents-aiprovider.md)
+- [docs/technical/en/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/en/agents-aiprovider.md)
 - [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
 - [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
 - [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
 - [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
+- [docs/plans/agents-earth-command-runtime-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md)
 
 ## 前端页面布局规范
 

TODO.md

@@ -1,38 +1,85 @@
 # TODO
 
-- [x] 把 BGP 观测站和异常点的 `hover/click` 手感再磨细一点
+This file is the active backlog only. Completed history belongs in `docs/CHANGELOG.md`; detailed designs belong in `docs/plans/`.
-- [x] 开始做 BGP 异常和海缆/区域的关联展示
+
-- [x] 做 Earth 侧的 `BGP activity layer`，让低 incident 密度时地图仍然有持续可感知的观测存在感
+## Earth
-- [x] 给 Earth BGP 补三层状态表达：`平稳观测态 / 局部波动态 / 事件活跃态`
+
-- [x] 把“当前无活跃事件”改造成“观测网络仍在运行、当前未发现聚合级事件”的状态表达
+- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
-- [x] 做 collector / region 近 15 分钟 activity score 聚合接口或动态聚合逻辑
+- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
-- [x] 把 Earth 的 BGP incident 改成 `紧凑事件核 + 向外扩张环形 pulse`，替换当前大面积 glow
+- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
-- [x] 为 BGP incident 建立符号系统：按事件类型用不同 marker，而不是都用同一种亮点
+- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
-- [x] 把 incident 地理定位从 `collector-centric` 改成 `prefix-centric`，优先使用 `prefix_geography`，其次 `prefix_scope`，再次 ASN 区域，最后才回退到观测区域质心
+- [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
-- [x] 新增 `prefix_geography` 数据层，不再把 `prefix_scope` 当成 prefix 地理归属本身
+- [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
-- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
+- [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
-- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
+- [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture.
-- [x] 把 RIR delegated 设计成 prefix geography 的 fallback，而不是主来源
+- [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable.
-- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心，优先落成 `json` 或 `yaml` 文件，运行时按 `provider/model` 读取兼容设置，而不是把专项兼容继续散落在 Python 分支里
+- [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker.
-- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中，至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
+
-- [ ] 在兼容配置中补齐可声明项：`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
+## Compute Centers And Location
-- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
+
-- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
+- [ ] Unknown compute-center locations: continue reducing unresolved records through the shared location pipeline, with confidence, precision, reason, and verification date preserved in GeoJSON/details.
-- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构，拆成 data / markers / overlays / animation，降低后续维护复杂度
+- [ ] Compute-center registry: keep expanding the local canonical location registry with `canonical_name`, aliases, operator, country/region/city, coordinates, confidence, and source notes.
-- [ ] 可选优化（非必做）：将 BGP incident/collector 标点改为 HTML marker（参考 worldmonitor 的 `htmlElementsData` 思路），实现近乎固定屏幕尺寸与更高密度可点击性
+- [ ] Compute-center enrichment: improve source-page parsing for Epoch AI and related collectors by extracting location clues from detail pages, embedded JSON, schema.org, OpenGraph, script variables, PDFs, and press releases.
-- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化：`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置，避免把设备级偏好过早做成全局配置
+- [ ] Compute-center identity normalization: normalize operator / cluster / facility aliases such as `xAI / Colossus / Memphis`, `OpenAI / Stargate`, `CoreWeave`, `Lambda`, and `Crusoe`.
-- [ ] 如果后续明确需要“账号级同步 Earth 偏好”，再单独设计 `Earth user preferences`：优先按用户维度而不是全局系统设置保存，并规划 `localStorage -> backend` 的平滑迁移策略
+- [ ] Compute-center manual review: add an export/review/import workflow for unresolved or estimated locations and feed confirmed results back into the registry.
-- [ ] 为 Planet / Earth 补一个可用的日志查看系统：先明确前后端/AI Provider/采集任务的日志入口、最近日志聚合、筛选与 tail 能力，再决定是先做脚本级统一入口还是控制台内置日志面板
+
-- [ ] 重写控制台 UI，逐步抛弃 Ant Design，建立自有组件体系，并统一采用 `tabler.io` / Tabler Icons 作为控制台主图标库
+## AIS / Vessels
-- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录，优先保持当前“实时聚合”链路不变，只先解决新闻源不可配置的问题
+
-- [ ] 为 Earth 态势新闻设计后续采集器化方案：明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式，再决定何时把新闻从实时抓取升级成正式 collector
+- [ ] AIS aggregation strategy v4: expose source priority, field-level merge rules, freshness windows, and protected dynamic-field rules in configuration, with validation and strategy version returned by vessel APIs.
-- [ ] 为 Earth 地球表面增加一层与基础纹理对齐的材质/纹理 overlay，并在同层叠加国界轮廓参考线；要求国界线与底图稳定对齐，且 hover 到国家轮廓时能高亮当前国家，便于校准地表和增强交互
+- [ ] AIS vessel enrichment v5: add asynchronous vessel profile enrichment for ship type detail, AIS class, flag, dimensions, build year, operator, and cached media. Do not fetch third-party pages in the realtime AIS request path.
-- [ ] 把 Earth 新闻接入通用巡航队列：按新闻发生地和时间排序生成巡航目标，巡航聚焦到新闻事件时显示对应新闻卡片，并保持实现边界为“通用巡航层 + 新闻业务适配层”，不要再把新闻逻辑直接耦合回 `main.js` 状态机
+- [ ] AIS identity cleanup: continue identifying vessels whose display name is only `MMSI <number>` and backfill names from AISStream static messages, BarentsWatch static fields, or enrichment cache.
-- [ ] 为未知位置的算力中心建立分层坐标补全链路：优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`，并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
+
-- [ ] 为算力中心补一份可维护的本地位置注册表，例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`，避免把地点知识长期硬编码在 `visualization.py`
+## AI Provider And Agents
-- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析：即使公开 API 不给坐标，也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
+
-- [ ] 为未知位置算力中心增加外部富化策略评估：可选接入公开知识源或搜索兜底，只抓“站点名/园区名/城市名”级别线索，不直接抓经纬度结论，并把结果作为候选证据而不是真值
+- [ ] Unified integration config schema: implement the shared low-code schema engine for datasource, AI Provider, Web Search, and OCR configuration described in [Integration Config Schema System Plan](/home/ray/dev/linkong/planet/docs/plans/integration-config-schema-system-plan.md).
-- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层，先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
+- [ ] AI provider routing: finish the OpenClaw-style provider/model routing refactor described in [AI Provider OpenClaw-Style Routing Plan](/home/ray/dev/linkong/planet/docs/plans/ai-provider-openclaw-style-routing-plan.md), so model-specific transport rules live in provider metadata rather than runtime hardcoding.
-- [ ] 为估算位置增加更细的视觉和产品表达：除了问号角标，还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”，并允许在设置中单独开关“仅看精确位置”
+- [ ] AI provider catalog: replace the temporary `model_provider_apis` bridge with structured `models_metadata`, discovery descriptors, and incremental model sync with stale marking.
-- [ ] 为国家级估算点设计更合理的落点策略：优先落在“该国主要算力/数据中心城市候选集”而不是几何质心，必要时同国多节点做稳定散列分配，避免大量节点堆在荒漠或海上
+- [ ] AI provider connectivity: keep the plug action as lightweight network/auth/model-directory validation only, and keep real generation tests inside Playground or explicit “trial run” actions.
-- [ ] 为未知位置算力中心建立人工校验工作流：支持导出待核验清单、记录人工确认结果，并把人工确认反哺到位置注册表，逐步减少问号点比例
+- [ ] Agent runtime foundation: add auditable agent runs, steps, evidence, proposals, and the Agent operations UI described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
+- [ ] Agent tool protocol: add backend JSON tool-call fallback, optional provider-native tool compatibility, tool whitelist validation, and policy-gated proposal application.
+- [ ] Speech/ASR integration for agents: add provider-neutral transcription settings and API, defaulting to Whisper-compatible API providers while keeping text commands usable when ASR is unavailable.
+- [ ] Earth voice wake: add device-local configurable wake-word preferences, microphone fallback states, and post-wake instruction upload for Earth commands.
+- [ ] AI provider compatibility center: move provider/model compatibility rules into a JSON/YAML config read by runtime, instead of continuing to scatter provider-specific branches through Python code.
+- [ ] Provider compatibility coverage: add explicit config for OpenAI, Anthropic, MiniMax, Ollama, Moonshot, DeepSeek, Qwen, GLM, Gemini, OpenRouter, vLLM, LM Studio, and One API.
+- [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.
+- [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data.
+
+## Archive
+
+Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason.
+
+### Completed
+
+- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
+- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
+- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
+- [x] Refined BGP observer and anomaly `hover/click` feel.
+- [x] Added BGP anomaly relationship display with cables / regions.
+- [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.
+- [x] Added BGP state expression for stable observation, local fluctuation, and active incident states.
+- [x] Reframed "no active incident" as "observation network is running; no aggregate incident detected".
+- [x] Added collector / region recent activity scoring.
+- [x] Replaced oversized BGP incident glow with compact incident core plus outward pulse rings.
+- [x] Added BGP incident symbol types instead of using one generic bright marker.
+- [x] Switched BGP incident geography from collector-centric to prefix-centric priority.
+- [x] Added `prefix_geography` as a separate data layer instead of treating `prefix_scope` as prefix geography.
+- [x] Added IPtoASN / IPtoCountry as the main prefix-centric geography source.
+- [x] Added OpenGeoFeed as a high-quality prefix geography override source.
+- [x] Made RIR delegated data a prefix geography fallback rather than the primary source.
+- [x] Added route leak and path instability / flap detectors after the activity layer work.
+- [x] Console UI modernization. Admin is now the only console, legacy Ant Design / Admin Next code paths and dependencies have been removed, and current console UI uses Planet-owned components.
+- [x] Earth news cruise adapter. News cruise now uses `news-cruise-adapter.js` and is wired from `main.js` instead of keeping news-specific sequencing directly in the main Earth loop.
+- [x] Presentation controller ownership. `PresentationController` now guards async ownership through active request identity checks, and current callers pass per-request card targets so stale connector/card work cannot overwrite the active presentation.
+- [x] Earth live sync. Database writes now flow through `earth_data_change_events`, `earth_db_change_listener`, layer adapters, cache invalidation, and the `earth_updates` WebSocket channel; the Earth frontend debounces updates and refreshes BGP, cables, compute centers, satellites, vessels, news, and interactables by layer.
+- [x] System logs. Log sources now normalize into `LogEvent`, Admin supports snapshot filtering plus WebSocket tail/follow, task/detail views deep-link into prefiltered logs, and Admin runtime errors report through the `admin-client` log source.
+
+### Obsolete Or Superseded
+
+- [ ] AIS v3.1 old `/geo/vessels` full-merge requirement. Superseded by `/api/v1/vessels/snapshot`, controlled legacy fallback, and diagnostics in the AIS aggregation plan.
+- [ ] AIS v3.2 old framing of AISStream as a batch collector that needed conversion. Superseded by the implemented long-lived AISStream collector and realtime stream UI.
+- [ ] AIS v3.3 old one-shot REST progress semantics for AISStream. Superseded by realtime stream status handling.
+- [ ] AIS v3.4 broad identity cleanup wording. Folded into the active AIS identity cleanup and v5 enrichment tasks.
+- [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map.
+- [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans.
+- [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog.
+- [ ] Earth preferences backend sync scope. Superseded by the current product decision to keep Earth preferences device-local in `localStorage` until account-level synchronization becomes a real requirement.

VERSION

@@ -1 +1 @@
-0.46.0
+0.74.4

agents.md

@@ -1,231 +0,0 @@
-# agents.md
-
-**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
-
----
-
-## Identity
-
-You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
-
-You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
-- Python FastAPI backend
-- React Admin dashboard
-- Unreal Engine 5 3D visualization
-- Multi-source data collection
-- Polarized 3D large display (4K, 120Hz)
-
----
-
-## Communication Style
-
-### Tone
-- **Professional but concise**
-- Technical accuracy with clarity
-- No unnecessary verbosity
-- Use code comments sparingly (explain **why**, not **what**)
-
-### When Responding
-1. **Answer directly** - 1-3 sentences for simple questions
-2. **Use code blocks** for all code snippets
-3. **Include file:line_number** references when discussing code
-4. **Never** start with "I am an AI assistant" or similar phrases
-5. **Never** add unnecessary preambles/postambles
-
-### Examples
-
-**Good:**
-```
-GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
-```
-
-**Bad:**
-```
-Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
-```
-
----
-
-## Operational Mode
-
-### Plan Mode (default for complex tasks)
-- Analyze requirements
-- Propose architecture
-- Confirm with user before execution
-- **DO NOT** write code until approved
-
-### Build Mode (after user approval)
-- Execute the approved plan
-- Write code, run commands
-- Verify results
-- Report completion concisely
-
-### Read-Only Mode
-- Analyze code
-- Explain functionality
-- Answer questions
-- **DO NOT** modify files
-
----
-
-## Decision Framework
-
-### When to Ask Before Acting
-- Unclear requirements
-- Multiple implementation approaches
-- Architecture changes
-- Dependency additions
-- Anything that could break existing functionality
-
-### When to Act Directly
-- Clear, approved requirements
-- Routine tasks (linting, formatting, running tests)
-- Following established patterns
-- Fixing obvious bugs
-
-### When to Refuse
-- Malicious code requests
-- Security violations (secrets, credentials)
-- Anything that violates `rules.md`
-
----
-
-## Working Principles
-
-### 1. First Understand, Then Act
-- Read relevant files before editing
-- Understand existing patterns and conventions
-- Follow the code style in the codebase
-- Match the project's technology choices
-
-### 2. Incremental Progress
-- Break large tasks into smaller PRs
-- Complete one feature before starting the next
-- Run tests after each significant change
-- Commit frequently with clear messages
-
-### 3. Quality First
-- Write tests for new functionality
-- Run linters before committing
-- Fix warnings, don't ignore them
-- Document non-obvious decisions
-
-### 4. Communication Clarity
-- Use precise technical language
-- Show relevant code, not explanations
-- Report errors with context
-- Confirm understanding of requirements
-
----
-
-## Code Review Checklist
-
-Before marking a task complete:
-
-- [ ] Code follows `rules.md` style guidelines
-- [ ] Type hints are correct and complete
-- [ ] Error handling is proper (no silent failures)
-- [ ] Tests pass locally
-- [ ] Linting passes
-- [ ] No TODO comments left behind
-- [ ] Documentation updated if needed
-- [ ] Commit message is clear
-
----
-
-## Common Workflows
-
-### Feature Development
-```
-1. Understand requirements
-2. Check existing patterns in codebase
-3. Design solution (brief mental model)
-4. Write code following rules.md
-5. Write/run tests
-6. Lint and format
-7. Commit with clear message
-8. Report completion
-```
-
-### Bug Fix
-```
-1. Reproduce the bug (write failing test)
-2. Locate the source
-3. Fix the issue
-4. Verify test passes
-5. Check for regressions
-6. Commit fix
-```
-
-### Refactoring
-```
-1. Understand current behavior
-2. Design target state
-3. Make incremental changes
-4. Preserve tests
-5. Verify functionality
-6. Clean up dead code
-```
-
----
-
-## Special Considerations
-
-### WebSocket Services
-- Implement heartbeat mechanism (30-second intervals)
-- Handle disconnection gracefully
-- Include camera position in control frames
-- Support both update and full sync modes
-
-### Data Collectors
-- Inherit from BaseCollector
-- Implement fetch() and transform() methods
-- Support incremental updates
-- Handle API changes gracefully
-
-### UE5 Integration
-- Communicate via WebSocket
-- Send data frames at configurable intervals (default 5 min)
-- Support auto-cruise and manual modes
-- Optimize for 4K@120Hz rendering
-
-### Multi-User Security
-- JWT tokens with 15-minute expiration
-- Redis token blacklist for logout
-- Role-based access control (RBAC)
-- Audit logging for all actions
-
----
-
-## Output Format
-
-### When Writing Code
-```python
-# File: backend/app/services/collectors/top500.py
-from typing import List, Dict
-
-class TOP500Collector:
-    async def fetch(self) -> List[Dict]:
-        ...
-```
-
-### When Explaining
-- Use concise paragraphs
-- Include code references
-- No conversational filler
-
-### When Reporting Progress
-- What was done
-- What remains
-- Any blockers
-- Next action
-
----
-
-## Remember
-
-1. **Rules are hard constraints** - follow `rules.md` absolutely
-2. **Context provides understanding** - use `project_context.md` for background
-3. **Role defines behavior** - follow `agents.md` for how to work
-4. **Quality over speed** - Enterprise systems require precision
-5. **Communicate clearly** - Precision in, precision out

aiprovider/.env.example

@@ -32,6 +32,15 @@ AI_API_KEY=sk-cp-change-me
 AI_MAX_TOKENS=1200
 AI_ANTHROPIC_VERSION=2023-06-01
 
+# Optional provider-specific keys used by Settings fallback before AI_API_KEY
+# MINIMAX_API_KEY=sk-cp-change-me
+# OPENAI_API_KEY=sk-change-me
+# ANTHROPIC_API_KEY=sk-ant-change-me
+# DEEPSEEK_API_KEY=sk-change-me
+# DASHSCOPE_API_KEY=sk-change-me
+# MOONSHOT_API_KEY=sk-change-me
+# OPENROUTER_API_KEY=sk-or-change-me
+
 # OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
 # AI_PROVIDER=openai
 # AI_PROVIDER_API=openai-completions

aiprovider/Dockerfile

@@ -2,6 +2,7 @@
 
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
+ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
 
 FROM ${UV_IMAGE} AS uv
 FROM ${PYTHON_IMAGE}
@@ -15,16 +16,25 @@ ENV PYTHONUNBUFFERED=1
 ENV UV_COMPILE_BYTECODE=1
 ENV UV_LINK_MODE=copy
 
+RUN mkdir -p /root/.config/uv
+
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
 COPY pyproject.toml uv.lock /app/
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv sync --frozen --no-dev
+    --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
+    uv sync --frozen --only-group aiprovider
 
 COPY aiprovider /app/aiprovider
 
+ARG AI_PROVIDER_BUILD_FINGERPRINT
+LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
+
 EXPOSE 8010
 
-CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010", "--reload"]
+HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
+    CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
+
+CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]

aiprovider/config.py

@@ -17,9 +17,6 @@ class Settings(BaseSettings):
     AI_HTTP_RETRY_ATTEMPTS: int = 2
     AI_MAX_TOKENS: int = 1200
     AI_ANTHROPIC_VERSION: str = "2023-06-01"
-    AI_ANALYSIS_SYSTEM_PROMPT: str = (
-        "你是态势感知分析助手。请基于输入的上下文、观测与约束，输出结构化、克制、可执行的分析。"
-    )
 
     AI_PROVIDER_SERVICE_TOKEN: str = ""
 

aiprovider/main.py

@@ -45,6 +45,7 @@ def get_provider_service(
     x_ai_model: str | None = Header(default=None),
     x_ai_max_tokens: str | None = Header(default=None),
     x_ai_anthropic_version: str | None = Header(default=None),
+    x_ai_model_provider_apis: str | None = Header(default=None),
 ) -> ProviderService:
     overrides = {
         "provider": x_ai_provider,
@@ -53,6 +54,7 @@ def get_provider_service(
         "api_key": x_ai_api_key,
         "model": x_ai_model,
         "anthropic_version": x_ai_anthropic_version,
+        "model_provider_apis": x_ai_model_provider_apis,
     }
     if x_ai_max_tokens:
         overrides["max_tokens"] = x_ai_max_tokens

aiprovider/provider_service.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import asyncio
+import json
 from typing import Any
 
 import httpx
@@ -14,7 +15,6 @@ from aiprovider.schemas import (
     SituationalAnalysisResponse,
 )
 
-
 def _normalize_provider(value: str) -> str:
     return (value or "disabled").strip().lower()
 
@@ -62,7 +62,9 @@ class ProviderService:
         self.anthropic_version = str(
             overrides.get("anthropic_version") or settings.AI_ANTHROPIC_VERSION
         )
-        self.system_prompt = settings.AI_ANALYSIS_SYSTEM_PROMPT
+        self.model_provider_apis = self._parse_model_provider_apis(
+            overrides.get("model_provider_apis")
+        )
 
     def get_status(self) -> AIProviderStatusResponse:
         enabled = self.provider != "disabled"
@@ -94,16 +96,23 @@ class ProviderService:
 
         prompt = self._build_prompt(payload)
 
-        if self.provider_api == "openai-completions":
+        provider_api = self._resolve_model_provider_api(model)
-            data = await self._request_openai_compatible(model, prompt)
+
+        if provider_api == "openai-completions":
+            data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
             content = self._extract_openai_content(data)
             content_blocks = self._extract_openai_blocks(data)
-        elif self.provider_api == "anthropic-messages":
+        elif provider_api == "anthropic-messages":
-            data = await self._request_anthropic_messages(model, prompt, payload.thinking)
+            data = await self._request_anthropic_messages(
+                model,
+                prompt,
+                payload.thinking,
+                payload.system_prompt,
+            )
             content = self._extract_anthropic_content(data)
             content_blocks = self._extract_anthropic_blocks(data)
-        elif self.provider_api == "ollama-generate":
+        elif provider_api == "ollama-generate":
-            data = await self._request_ollama(model, prompt)
+            data = await self._request_ollama(model, prompt, payload.system_prompt)
             content = self._extract_ollama_content(data)
             content_blocks = self._extract_ollama_blocks(data)
         else:
@@ -128,6 +137,26 @@ class ProviderService:
     def _requires_api_key(self) -> bool:
         return self.provider_api != "ollama-generate"
 
+    def _resolve_model_provider_api(self, model: str) -> str:
+        return self.model_provider_apis.get(model) or self.provider_api
+
+    def _parse_model_provider_apis(self, value: Any) -> dict[str, str]:
+        if isinstance(value, dict):
+            raw = value
+        elif isinstance(value, str) and value.strip():
+            try:
+                parsed = json.loads(value)
+            except json.JSONDecodeError:
+                return {}
+            raw = parsed if isinstance(parsed, dict) else {}
+        else:
+            raw = {}
+        return {
+            str(model): _normalize_provider_api(str(provider_api))
+            for model, provider_api in raw.items()
+            if model and provider_api
+        }
+
     def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
         sections = [
             f"任务标题:\n{payload.title}",
@@ -139,19 +168,28 @@ class ProviderService:
             sections.append("约束条件:\n" + "\n".join(f"- {item}" for item in payload.constraints))
         if payload.context:
             sections.append(f"附加上下文:\n{payload.context}")
-        sections.append(
-            "请输出: 1) 态势摘要 2) 关键风险 3) 研判依据 4) 建议动作 5) 还缺少的数据。"
-        )
         return "\n\n".join(sections)
 
-    async def _request_openai_compatible(self, model: str, prompt: str) -> dict[str, Any]:
+    def _resolve_system_prompt(self, system_prompt: str | None) -> str | None:
+        resolved = str(system_prompt or "").strip()
+        return resolved or None
+
+    async def _request_openai_compatible(
+        self,
+        model: str,
+        prompt: str,
+        system_prompt: str | None = None,
+    ) -> dict[str, Any]:
+        messages = []
+        resolved_system_prompt = self._resolve_system_prompt(system_prompt)
+        if resolved_system_prompt:
+            messages.append({"role": "system", "content": resolved_system_prompt})
+        messages.append({"role": "user", "content": prompt})
         request_body = {
             "model": model,
-            "messages": [
+            "messages": messages,
-                {"role": "system", "content": self.system_prompt},
-                {"role": "user", "content": prompt},
-            ],
             "temperature": 0.2,
+            "max_tokens": self.max_tokens,
         }
         return await self._post(
             path="/chat/completions",
@@ -167,10 +205,10 @@ class ProviderService:
         model: str,
         prompt: str,
         thinking: dict[str, Any] | None = None,
+        system_prompt: str | None = None,
     ) -> dict[str, Any]:
         request_body = {
             "model": model,
-            "system": self.system_prompt,
             "messages": [
                 {
                     "role": "user",
@@ -185,6 +223,9 @@ class ProviderService:
             "max_tokens": self.max_tokens,
             "temperature": 0.2,
         }
+        resolved_system_prompt = self._resolve_system_prompt(system_prompt)
+        if resolved_system_prompt:
+            request_body["system"] = resolved_system_prompt
         resolved_thinking = self._resolve_anthropic_thinking(thinking)
         if resolved_thinking:
             request_body["thinking"] = resolved_thinking
@@ -218,19 +259,28 @@ class ProviderService:
         model: str,
         prompt: str,
         thinking: dict[str, Any] | None = None,
+        system_prompt: str | None = None,
     ) -> dict[str, Any]:
-        return await self._request_anthropic_messages(model, prompt, thinking)
+        return await self._request_anthropic_messages(model, prompt, thinking, system_prompt)
 
-    async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
+    async def _request_ollama(
+        self,
+        model: str,
+        prompt: str,
+        system_prompt: str | None = None,
+    ) -> dict[str, Any]:
         request_body = {
             "model": model,
             "stream": False,
-            "system": self.system_prompt,
             "prompt": prompt,
             "options": {
                 "temperature": 0.2,
+                "num_predict": self.max_tokens,
             },
         }
+        resolved_system_prompt = self._resolve_system_prompt(system_prompt)
+        if resolved_system_prompt:
+            request_body["system"] = resolved_system_prompt
         return await self._post(
             path="/api/generate",
             headers={
@@ -289,13 +339,19 @@ class ProviderService:
         message = choices[0].get("message") or {}
         content = message.get("content")
         if isinstance(content, str):
-            return content
+            if content:
+                return content
+            reasoning_content = message.get("reasoning_content")
+            return reasoning_content if isinstance(reasoning_content, str) else ""
         if isinstance(content, list):
             return "".join(
                 item.get("text", "")
                 for item in content
                 if isinstance(item, dict)
             )
+        reasoning_content = message.get("reasoning_content")
+        if isinstance(reasoning_content, str):
+            return reasoning_content
         return ""
 
     def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
@@ -306,9 +362,14 @@ class ProviderService:
         message = choices[0].get("message") or {}
         content = message.get("content")
         if isinstance(content, str):
-            return [AIContentBlock(type="text", text=content)]
+            blocks = [AIContentBlock(type="text", text=content)] if content else []
+            reasoning_content = message.get("reasoning_content")
+            if isinstance(reasoning_content, str) and reasoning_content:
+                blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
+            return blocks
         if not isinstance(content, list):
-            return []
+            reasoning_content = message.get("reasoning_content")
+            return [AIContentBlock(type="thinking", thinking=reasoning_content)] if isinstance(reasoning_content, str) and reasoning_content else []
 
         blocks: list[AIContentBlock] = []
         for item in content:
@@ -321,7 +382,11 @@ class ProviderService:
                     metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
                 )
             )
+        reasoning_content = message.get("reasoning_content")
+        if isinstance(reasoning_content, str) and reasoning_content:
+            blocks.append(AIContentBlock(type="thinking", thinking=reasoning_content))
         return blocks
+
     def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
         content = payload.get("content")
         if isinstance(content, str):

aiprovider/schemas.py

@@ -13,10 +13,11 @@ class AIContentBlock(BaseModel):
 
 class SituationalAnalysisRequest(BaseModel):
     title: str = Field(..., min_length=1, max_length=200)
-    objective: str = Field(..., min_length=1, max_length=1000)
+    objective: str = Field(..., min_length=1, max_length=20000)
     context: dict[str, Any] = Field(default_factory=dict)
     observations: list[str] = Field(default_factory=list)
     constraints: list[str] = Field(default_factory=list)
+    system_prompt: str | None = Field(default=None, max_length=8000)
     preferred_model: str | None = Field(default=None, max_length=200)
     thinking: dict[str, Any] | None = None
 

backend/Dockerfile

@@ -1,3 +1,5 @@
+# syntax=docker/dockerfile:1.7
+
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
 
@@ -12,17 +14,25 @@ ENV PYTHONDONTWRITEBYTECODE=1
 ENV PYTHONUNBUFFERED=1
 ENV UV_COMPILE_BYTECODE=1
 ENV UV_LINK_MODE=copy
+ENV PYTHONPATH=/app/backend
+
+RUN mkdir -p /root/.config/uv
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
 COPY pyproject.toml uv.lock /app/
-RUN uv sync --frozen --no-dev
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
+    uv sync --frozen --no-dev
 
 COPY backend /app/backend
 COPY VERSION /app/VERSION
 
 EXPOSE 8000
 
-CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
+HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
+    CMD curl -fsS http://127.0.0.1:8000/health >/dev/null || exit 1
+
+CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

backend/app/ai_tasks/__init__.py

@@ -0,0 +1,2 @@
+"""AI task prompt registry and runtime helpers."""
+

backend/app/ai_tasks/default_prompts.json

@@ -0,0 +1,74 @@
+[
+  {
+    "key": "earth.news.enrich",
+    "label": "Earth 新闻汉化与定位",
+    "group": "Earth 新闻",
+    "version": "2026-05-16.2",
+    "system_prompt": "",
+    "prompt": "Return exactly one strict JSON object with a location object and a localizations object. Infer the most likely physical event location and produce a faithful Simplified Chinese title plus a one-sentence newswire-style Chinese summary based only on the supplied RSS headline, description, source, and date. The summary should read like a concise breaking-news lead, not a label, slogan, or keyword headline."
+  },
+  {
+    "key": "alerts.brief",
+    "label": "系统告警研判",
+    "group": "告警研判",
+    "version": "2026-05-16.1",
+    "system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束，输出结构化、克制、可执行的值班研判；明确区分事实、推断与建议，不要夸大证据不足的风险。",
+    "prompt": "基于当前告警总量、严重度、状态、数据源分布与最近告警摘录，生成一份面向值班人员的简明告警态势简报，突出待处理风险、告警集中点和优先动作。"
+  },
+  {
+    "key": "alerts.situational.brief",
+    "label": "跨模块态势告警研判",
+    "group": "告警研判",
+    "version": "2026-05-16.1",
+    "system_prompt": "你是告警研判助手。请基于输入的告警事实、上下文与约束，输出结构化、克制、可执行的值班研判；明确区分事实、推断与建议，不要夸大证据不足的风险。",
+    "prompt": "综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报，生成一份面向值班人员的态势告警简报，指出当前最需要关注的风险域、跨模块联动迹象和优先动作。"
+  },
+  {
+    "key": "bgp.brief",
+    "label": "BGP 态势简报",
+    "group": "BGP",
+    "version": "2026-05-16.2",
+    "system_prompt": "你是 BGP 值班分析师。请直接输出面向值班人员的中文 Markdown 简报，只写最终研判内容；不要复述用户需求、提示词、写作计划、字段清单或“我将如何回答”。",
+    "prompt": "基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据，生成一份面向操作员的简明态势简报，突出区域热点、观测偏差、当前风险、证据和优先动作。"
+  },
+  {
+    "key": "location.factcheck.normalize",
+    "label": "位置事实核查结构化",
+    "group": "位置解析",
+    "version": "2026-05-16.1",
+    "system_prompt": "",
+    "prompt": "Convert the supplied location factcheck text into exactly one strict JSON object. Extract only facts present in the text or original query."
+  },
+  {
+    "key": "location.factcheck.resolve",
+    "label": "位置事实核查兜底",
+    "group": "位置解析",
+    "version": "2026-05-16.1",
+    "system_prompt": "",
+    "prompt": "Return exactly one JSON object for the most likely physical location. Use only fact-checkable public knowledge; return null fields rather than guessing when evidence is weak."
+  },
+  {
+    "key": "datasource.mapping",
+    "label": "数据源映射生成",
+    "group": "采集配置",
+    "version": "2026-05-16.1",
+    "system_prompt": "",
+    "prompt": "Return only JSON for a deterministic mapping DSL. The JSON must contain source.items_path and fields. Do not include prose or code."
+  },
+  {
+    "key": "credential.guide",
+    "label": "采集器凭据教程",
+    "group": "采集配置",
+    "version": "2026-05-16.1",
+    "system_prompt": "",
+    "prompt": "生成一份中文采集器凭据配置教程。只能根据 context.search_evidence 中的来源生成教程；如果证据不足，明确说明需要以官方页面为准。"
+  },
+  {
+    "key": "ai.connection_test",
+    "label": "AI Provider 连接测试",
+    "group": "运维测试",
+    "version": "2026-05-16.1",
+    "system_prompt": "",
+    "prompt": "Reply OK."
+  }
+]

backend/app/ai_tasks/prompts.py

@@ -0,0 +1,182 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import UTC, datetime
+import json
+from functools import lru_cache
+from pathlib import Path
+from typing import Any
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.system_setting import SystemSetting
+
+AI_PROMPTS_CATEGORY = "ai_prompts"
+DEFAULT_PROMPTS_PATH = Path(__file__).with_name("default_prompts.json")
+
+
+@dataclass(frozen=True)
+class AIPromptDefinition:
+    key: str
+    label: str
+    group: str
+    version: str
+    system_prompt: str
+    prompt: str
+
+
+@dataclass(frozen=True)
+class EffectiveAIPrompt:
+    key: str
+    label: str
+    group: str
+    version: str
+    default_system_prompt: str
+    default_prompt: str
+    system_prompt: str
+    prompt: str
+    is_custom: bool
+    updated_at: str | None = None
+
+
+@lru_cache(maxsize=1)
+def list_prompt_definitions() -> tuple[AIPromptDefinition, ...]:
+    raw_items = json.loads(DEFAULT_PROMPTS_PATH.read_text(encoding="utf-8"))
+    return tuple(
+        AIPromptDefinition(
+            key=str(item["key"]),
+            label=str(item["label"]),
+            group=str(item["group"]),
+            version=str(item["version"]),
+            system_prompt=str(item.get("system_prompt") or ""),
+            prompt=str(item.get("prompt") or ""),
+        )
+        for item in raw_items
+    )
+
+
+def get_prompt_definition(task_key: str) -> AIPromptDefinition:
+    for definition in list_prompt_definitions():
+        if definition.key == task_key:
+            return definition
+    raise KeyError(task_key)
+
+
+async def _get_prompt_setting(db: AsyncSession) -> SystemSetting | None:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == AI_PROMPTS_CATEGORY)
+    )
+    return result.scalar_one_or_none()
+
+
+def _normalize_overrides(payload: dict[str, Any] | None) -> dict[str, dict[str, Any]]:
+    raw = (payload or {}).get("overrides")
+    if not isinstance(raw, dict):
+        return {}
+    return {
+        str(key): dict(value)
+        for key, value in raw.items()
+        if isinstance(value, dict)
+    }
+
+
+async def get_prompt_overrides(db: AsyncSession) -> dict[str, dict[str, Any]]:
+    if not hasattr(db, "execute"):
+        return {}
+    setting = await _get_prompt_setting(db)
+    return _normalize_overrides(setting.payload if setting else None)
+
+
+def _effective_prompt(
+    definition: AIPromptDefinition,
+    override: dict[str, Any] | None,
+) -> EffectiveAIPrompt:
+    override = override or {}
+    custom_system = override.get("system_prompt")
+    custom_prompt = override.get("prompt")
+    has_custom_system = isinstance(custom_system, str)
+    has_custom_prompt = isinstance(custom_prompt, str)
+    return EffectiveAIPrompt(
+        key=definition.key,
+        label=definition.label,
+        group=definition.group,
+        version=definition.version,
+        default_system_prompt=definition.system_prompt,
+        default_prompt=definition.prompt,
+        system_prompt=custom_system if has_custom_system else definition.system_prompt,
+        prompt=custom_prompt if has_custom_prompt else definition.prompt,
+        is_custom=has_custom_system or has_custom_prompt,
+        updated_at=str(override.get("updated_at") or "") or None,
+    )
+
+
+async def list_effective_prompts(db: AsyncSession) -> list[EffectiveAIPrompt]:
+    overrides = await get_prompt_overrides(db)
+    return [
+        _effective_prompt(definition, overrides.get(definition.key))
+        for definition in list_prompt_definitions()
+    ]
+
+
+async def get_effective_prompt(db: AsyncSession | None, task_key: str) -> EffectiveAIPrompt:
+    definition = get_prompt_definition(task_key)
+    if db is None:
+        return _effective_prompt(definition, None)
+    overrides = await get_prompt_overrides(db)
+    return _effective_prompt(definition, overrides.get(task_key))
+
+
+async def save_prompt_override(
+    db: AsyncSession,
+    task_key: str,
+    *,
+    system_prompt: str,
+    prompt: str,
+) -> EffectiveAIPrompt:
+    definition = get_prompt_definition(task_key)
+    setting = await _get_prompt_setting(db)
+    payload = dict(setting.payload or {}) if setting else {}
+    overrides = _normalize_overrides(payload)
+    overrides[definition.key] = {
+        "system_prompt": system_prompt,
+        "prompt": prompt,
+        "updated_at": datetime.now(UTC).isoformat().replace("+00:00", "Z"),
+    }
+    payload["overrides"] = overrides
+    if setting is None:
+        setting = SystemSetting(category=AI_PROMPTS_CATEGORY, payload=payload)
+        db.add(setting)
+    else:
+        setting.payload = payload
+    await db.commit()
+    return _effective_prompt(definition, overrides[definition.key])
+
+
+async def reset_prompt_override(db: AsyncSession, task_key: str) -> EffectiveAIPrompt:
+    definition = get_prompt_definition(task_key)
+    setting = await _get_prompt_setting(db)
+    if setting is None:
+        return _effective_prompt(definition, None)
+    payload = dict(setting.payload or {})
+    overrides = _normalize_overrides(payload)
+    overrides.pop(definition.key, None)
+    payload["overrides"] = overrides
+    setting.payload = payload
+    await db.commit()
+    return _effective_prompt(definition, None)
+
+
+def serialize_effective_prompt(prompt: EffectiveAIPrompt) -> dict[str, Any]:
+    return {
+        "key": prompt.key,
+        "label": prompt.label,
+        "group": prompt.group,
+        "version": prompt.version,
+        "default_system_prompt": prompt.default_system_prompt,
+        "default_prompt": prompt.default_prompt,
+        "system_prompt": prompt.system_prompt,
+        "prompt": prompt.prompt,
+        "is_custom": prompt.is_custom,
+        "updated_at": prompt.updated_at,
+    }

backend/app/api/main.py

@@ -5,15 +5,22 @@ from app.api.v1 import (
     users,
     datasource_config,
     datasources,
+    docs,
+    earth,
     tasks,
     dashboard,
-    websocket,
     alerts,
     settings,
     collected_data,
+    data_products,
+    layers,
     visualization,
+    vessel_aggregation,
+    vessels,
     bgp,
     news,
+    interactables,
+    realtime_sources,
     system_control,
     tv,
 )
@@ -28,12 +35,24 @@ api_router.include_router(
 )
 api_router.include_router(datasources.router, prefix="/datasources", tags=["datasources"])
 api_router.include_router(collected_data.router, prefix="/collected", tags=["collected-data"])
+api_router.include_router(docs.router, prefix="/docs", tags=["docs"])
+api_router.include_router(earth.router, prefix="/earth", tags=["earth"])
 api_router.include_router(tasks.router, prefix="/tasks", tags=["tasks"])
 api_router.include_router(dashboard.router, prefix="/dashboard", tags=["dashboard"])
 api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
 api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
 api_router.include_router(system_control.router, prefix="/system", tags=["system"])
+api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
+api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
 api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
+api_router.include_router(
+    vessel_aggregation.router,
+    prefix="/vessel-aggregation",
+    tags=["vessel-aggregation"],
+)
+api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
 api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
 api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
 api_router.include_router(news.router, prefix="/news", tags=["news"])
+api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
+api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])

backend/app/api/v1/ai.py

@@ -3,6 +3,7 @@ from uuid import uuid4
 from fastapi import APIRouter, Depends, HTTPException, Request, Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.logging import get_logger
 from app.core.security import get_current_user
 from app.db.session import get_db
 from app.models.user import User
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
     stop_message,
 )
 from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
+from app.services.business_logs import emit_business_log, exception_context
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 
 
 @router.get("/provider/status", response_model=AIProviderStatusResponse)
@@ -122,6 +125,16 @@ async def create_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.create",
+        message="Playground message creation requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
+    )
     return await create_turn(
         db,
         user_id=current_user.id,
@@ -136,6 +149,16 @@ async def stop_playground_message(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.stop",
+        message="Playground message stop requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "message_id": payload.message_id},
+    )
     return await stop_message(
         db,
         user_id=current_user.id,
@@ -150,6 +173,16 @@ async def resend_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.resend",
+        message="Playground message resend requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
+    )
     return await resend_turn(
         db,
         user_id=current_user.id,
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
         anomaly_limit=payload.anomaly_limit,
         collector_limit=payload.collector_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.bgp.facts_collected",
+        message="BGP brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={
+            "incident_limit": payload.incident_limit,
+            "anomaly_limit": payload.anomaly_limit,
+            "collector_limit": payload.collector_limit,
+            "fact_count": len(facts or []),
+        },
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    await emit_business_log(
-    return save_bgp_brief_record(
+        logger,
-        analysis,
+        event="ai.brief.bgp.start",
+        message="BGP brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
         request_id=request_id,
-        facts=facts,
+        user_id=current_user.id,
-        context=context,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        record = save_bgp_brief_record(
+            analysis,
+            request_id=request_id,
+            facts=facts,
+            context=context,
+        )
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.completed",
+            message="BGP brief AI analysis saved",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
+        )
+        return record
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.failed",
+            message="BGP brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/alerts/brief", response_model=AlertBriefResponse)
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
         db,
         alert_limit=payload.alert_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.alerts.facts_collected",
+        message="Alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    await emit_business_log(
-    return AlertBriefResponse(
+        logger,
-        **analysis.model_dump(),
+        event="ai.brief.alerts.start",
-        title=brief_request.title,
+        message="Alert brief AI analysis started",
-        objective=brief_request.objective,
+        category="ai",
-        facts=facts,
+        service="api",
-        context=context,
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.completed",
+            message="Alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return AlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.failed",
+            message="Alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
     response.headers["X-Request-ID"] = request_id
 
     brief_request, facts, context = await build_situational_alert_brief_request(db)
+    await emit_business_log(
+        logger,
+        event="ai.brief.situational_alerts.facts_collected",
+        message="Situational alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
+    await emit_business_log(
-    return SituationalAlertBriefResponse(
+        logger,
-        **analysis.model_dump(),
+        event="ai.brief.situational_alerts.start",
-        title=brief_request.title,
+        message="Situational alert brief AI analysis started",
-        objective=brief_request.objective,
+        category="ai",
-        facts=facts,
+        service="api",
-        context=context,
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.completed",
+            message="Situational alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return SituationalAlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.failed",
+            message="Situational alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise

backend/app/api/v1/auth.py

@@ -1,26 +1,86 @@
-from datetime import timedelta
-
 from fastapi import APIRouter, Depends, HTTPException, status
 from fastapi.security import OAuth2PasswordRequestForm
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import OtpPurpose, UserRole
+from app.core.logging import get_logger
 from app.core.security import (
     create_access_token,
     create_refresh_token,
-    blacklist_token,
     get_current_user,
+    get_password_hash,
     verify_password,
 )
 from app.db.session import get_db
 from app.models.user import User
 from app.schemas.token import Token
-from app.schemas.user import UserCreate, UserResponse
+from app.schemas.user import (
+    ForgotPasswordRequest,
+    ResendCodeRequest,
+    ResetPasswordRequest,
+    UserRegister,
+    UserResponse,
+    VerifyEmailRequest,
+)
+from app.services import otp
+from app.services.email import (
+    EmailError,
+    EmailNotConfiguredError,
+    send_verification_email,
+)
+
+logger = get_logger(__name__)
 
 router = APIRouter()
 
 
+def _token_response(user: User) -> dict:
+    access_token = create_access_token(data={"sub": user.id})
+    refresh = create_refresh_token(data={"sub": user.id})
+    expires_in = (
+        settings.ACCESS_TOKEN_EXPIRE_MINUTES * 60
+        if settings.ACCESS_TOKEN_EXPIRE_MINUTES > 0
+        else None
+    )
+    return {
+        "access_token": access_token,
+        "token_type": "bearer",
+        "expires_in": expires_in,
+        "refresh_token": refresh,
+        "user": {
+            "id": user.id,
+            "username": user.username,
+            "role": user.role,
+            "gatekeeper_groups": user.gatekeeper_groups or [],
+        },
+    }
+
+
+async def _load_user_by_email(db: AsyncSession, email: str) -> User | None:
+    result = await db.execute(
+        text(
+            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups, email_verified "
+            "FROM users WHERE email = :email"
+        ),
+        {"email": email},
+    )
+    row = result.fetchone()
+    if row is None:
+        return None
+    user = User()
+    user.id = row[0]
+    user.username = row[1]
+    user.email = row[2]
+    user.password_hash = row[3]
+    user.role = row[4]
+    user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
+    user.email_verified = bool(row[7])
+    return user
+
+
 @router.post("/login", response_model=Token)
 async def login(
     form_data: OAuth2PasswordRequestForm = Depends(),
@@ -28,7 +88,8 @@ async def login(
 ):
     result = await db.execute(
         text(
-            "SELECT id, username, email, password_hash, role, is_active FROM users WHERE username = :username"
+            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups, email_verified "
+            "FROM users WHERE username = :username"
         ),
         {"username": form_data.username},
     )
@@ -46,6 +107,8 @@ async def login(
     user.password_hash = row[3]
     user.role = row[4]
     user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
+    user.email_verified = bool(row[7])
 
     if not verify_password(form_data.password, user.password_hash):
         raise HTTPException(
@@ -57,24 +120,13 @@ async def login(
             status_code=status.HTTP_401_UNAUTHORIZED,
             detail="User is inactive",
         )
+    if not user.email_verified:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail={"code": "EMAIL_NOT_VERIFIED", "email": user.email},
+        )
 
-    access_token = create_access_token(data={"sub": user.id})
+    return _token_response(user)
-    refresh_token = create_refresh_token(data={"sub": user.id})
-
-    expires_in = None
-    if settings.ACCESS_TOKEN_EXPIRE_MINUTES > 0:
-        expires_in = settings.ACCESS_TOKEN_EXPIRE_MINUTES * 60
-
-    return {
-        "access_token": access_token,
-        "token_type": "bearer",
-        "expires_in": expires_in,
-        "user": {
-            "id": user.id,
-            "username": user.username,
-            "role": user.role,
-        },
-    }
 
 
 @router.post("/refresh", response_model=Token)
@@ -95,6 +147,7 @@ async def refresh_token(
             "id": current_user.id,
             "username": current_user.username,
             "role": current_user.role,
+            "gatekeeper_groups": current_user.gatekeeper_groups or [],
         },
     }
 
@@ -111,6 +164,190 @@ async def get_me(current_user: User = Depends(get_current_user)):
         "username": current_user.username,
         "email": current_user.email,
         "role": current_user.role,
+        "gatekeeper_groups": current_user.gatekeeper_groups or [],
         "is_active": current_user.is_active,
+        "email_verified": getattr(current_user, "email_verified", True),
         "created_at": current_user.created_at,
     }
+
+
+async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None:
+    try:
+        await send_verification_email(db, to=email, code=code, purpose=purpose)
+    except EmailNotConfiguredError as exc:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except EmailError as exc:
+        logger.warning_event(
+            "SMTP send failed",
+            event="auth.email.send_failed",
+            context={"email": email, "purpose": purpose, "error": str(exc)},
+        )
+        raise HTTPException(
+            status_code=status.HTTP_502_BAD_GATEWAY,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+
+
+@router.post("/register", status_code=status.HTTP_201_CREATED)
+async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
+    existing = await db.execute(
+        text("SELECT id, email_verified FROM users WHERE username = :u OR email = :e"),
+        {"u": payload.username, "e": payload.email},
+    )
+    row = existing.fetchone()
+    if row is not None:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail={"code": "USER_ALREADY_EXISTS", "message": "Username or email already in use"},
+        )
+
+    user = User(
+        username=payload.username,
+        email=payload.email,
+        password_hash=get_password_hash(payload.password),
+        role=UserRole.VIEWER.value,
+        is_active=True,
+        email_verified=False,
+    )
+    db.add(user)
+    await db.commit()
+
+    try:
+        code = otp.issue_code(payload.email, OtpPurpose.REGISTER)
+    except otp.OtpResendRateLimited as exc:
+        raise HTTPException(
+            status_code=status.HTTP_429_TOO_MANY_REQUESTS,
+            detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
+        ) from exc
+    await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER)
+    return {"status": "pending_verification", "email": payload.email}
+
+
+@router.post("/verify-email", response_model=Token)
+async def verify_email(payload: VerifyEmailRequest, db: AsyncSession = Depends(get_db)):
+    user = await _load_user_by_email(db, payload.email)
+    if user is None:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail={"code": "USER_NOT_FOUND"},
+        )
+    try:
+        otp.verify_code(payload.email, "register", payload.code)
+    except otp.OtpExpired as exc:
+        raise HTTPException(
+            status_code=status.HTTP_410_GONE,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except otp.OtpAttemptsExceeded as exc:
+        raise HTTPException(
+            status_code=status.HTTP_429_TOO_MANY_REQUESTS,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except otp.OtpInvalid as exc:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+
+    await db.execute(
+        text("UPDATE users SET email_verified = TRUE WHERE id = :id"),
+        {"id": user.id},
+    )
+    await db.commit()
+    user.email_verified = True
+    return _token_response(user)
+
+
+@router.post("/resend-code")
+async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get_db)):
+    user = await _load_user_by_email(db, payload.email)
+    if user is None:
+        # Avoid email enumeration; pretend success.
+        return {"status": "ok"}
+    if payload.purpose is OtpPurpose.REGISTER and user.email_verified:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail={"code": "ALREADY_VERIFIED"},
+        )
+    try:
+        code = otp.issue_code(payload.email, payload.purpose)
+    except otp.OtpResendRateLimited as exc:
+        raise HTTPException(
+            status_code=status.HTTP_429_TOO_MANY_REQUESTS,
+            detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
+        ) from exc
+    await _send_code_or_raise(db, payload.email, code, payload.purpose)
+    return {"status": "ok"}
+
+
+@router.post("/forgot-password")
+async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Depends(get_db)):
+    user = await _load_user_by_email(db, payload.email)
+    if user is None:
+        # Don't leak whether an email is registered.
+        return {"status": "ok"}
+    try:
+        code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD)
+    except otp.OtpResendRateLimited:
+        # Silently accept; the user can retry after the cooldown.
+        return {"status": "ok"}
+    try:
+        await send_verification_email(
+            db,
+            to=payload.email,
+            code=code,
+            purpose=OtpPurpose.RESET_PASSWORD,
+        )
+    except EmailNotConfiguredError as exc:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except EmailError as exc:
+        logger.warning_event(
+            "SMTP send failed",
+            event="auth.email.send_failed",
+            context={
+                "email": payload.email,
+                "purpose": OtpPurpose.RESET_PASSWORD.value,
+                "error": str(exc),
+            },
+        )
+    return {"status": "ok"}
+
+
+@router.post("/reset-password")
+async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depends(get_db)):
+    user = await _load_user_by_email(db, payload.email)
+    if user is None:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail={"code": "OTP_INVALID"},
+        )
+    try:
+        otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code)
+    except otp.OtpExpired as exc:
+        raise HTTPException(
+            status_code=status.HTTP_410_GONE,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except otp.OtpAttemptsExceeded as exc:
+        raise HTTPException(
+            status_code=status.HTTP_429_TOO_MANY_REQUESTS,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+    except otp.OtpInvalid as exc:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail={"code": exc.code, "message": str(exc)},
+        ) from exc
+
+    await db.execute(
+        text("UPDATE users SET password_hash = :p, email_verified = TRUE WHERE id = :id"),
+        {"p": get_password_hash(payload.new_password), "id": user.id},
+    )
+    await db.commit()
+    return {"status": "ok"}

backend/app/api/v1/bgp.py

@@ -5,13 +5,26 @@ from fastapi import APIRouter, Depends, HTTPException, Query
 from sqlalchemy import func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from pydantic import BaseModel
+
 from app.core.security import get_current_user
 from app.db.session import get_db
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
 from app.models.user import User
+from app.services.bgp_collector_locations import (
+    build_bgp_collector_location_query,
+    collect_bgp_collector_location_candidates,
+    get_bgp_collector_location_dict,
+)
 from app.services.bgp_collectors import build_bgp_collector_coverage
+from app.services.ai_client import get_ai_provider_client
+from app.api.v1.settings import get_web_search_client
+from app.services.location.llm_fallback import (
+    collect_llm_location_fallback_candidate,
+    collect_location_search_evidence,
+)
 
 router = APIRouter()
 
@@ -264,6 +277,120 @@ async def get_bgp_collector_summary(
     }
 
 
+class CollectBGPCollectorLocationRequest(BaseModel):
+    city: Optional[str] = None
+    country: Optional[str] = None
+    site: Optional[str] = None
+    operator: Optional[str] = None
+
+
+@router.post("/collectors/{collector_id}/collect-location")
+async def collect_bgp_collector_location(
+    collector_id: str,
+    payload: CollectBGPCollectorLocationRequest,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Run the shared location pipeline for a BGP route collector.
+
+    Mirrors ``POST /api/v1/visualization/compute-centers/{source_id}/collect-location``.
+    Returns ranked candidates from source coordinates and Nominatim queries
+    built around the collector's stored context (IXP / city / country). Stored
+    collector locations provide context only; they are not emitted as
+    candidates.
+    """
+    if not collector_id or not collector_id.strip():
+        raise HTTPException(status_code=400, detail="collector_id is required")
+
+    legacy = get_bgp_collector_location_dict(collector_id) or {}
+    site = payload.site or legacy.get("matched_location_name")
+    city = payload.city or legacy.get("city")
+    country = payload.country or legacy.get("country")
+    operator = payload.operator or "RIPE NCC"
+
+    candidates, attempted_queries = collect_bgp_collector_location_candidates(
+        collector=collector_id,
+        site=site,
+        city=city,
+        country=country,
+        operator=operator,
+    )
+    llm_failure_reason = None
+    if not candidates:
+        query = build_bgp_collector_location_query(
+            collector=collector_id,
+            site=site,
+            city=city,
+            country=country,
+            operator=operator,
+        )
+        llm_result = None
+        try:
+            web_search_client = await get_web_search_client(db)
+            search_result = await collect_location_search_evidence(
+                web_search_client=web_search_client,
+                query=query,
+                entity_type="bgp_collector",
+            )
+            attempted_queries = [*attempted_queries, *search_result.attempted_queries]
+            if not search_result.evidence:
+                llm_failure_reason = search_result.failure_reason
+                raise RuntimeError(search_result.failure_reason or "no WebSearch evidence")
+            provider_client = await get_ai_provider_client(db)
+            llm_result = await collect_llm_location_fallback_candidate(
+                provider_client=provider_client,
+                query=query,
+                entity_type="bgp_collector",
+                db=db,
+                attempted_queries=attempted_queries,
+                search_evidence=search_result.evidence,
+            )
+        except Exception as exc:
+            if llm_failure_reason is None:
+                llm_failure_reason = f"LLM location factcheck unavailable: {exc}"
+                attempted_queries = [
+                    *attempted_queries,
+                    f"llm_factcheck:bgp_collector:{collector_id or 'unknown'}",
+                ]
+        if llm_result is not None:
+            attempted_queries = [*attempted_queries, *llm_result.attempted_queries]
+            candidates = llm_result.candidates
+            llm_failure_reason = llm_result.failure_reason
+
+    context = {
+        "collector": collector_id,
+        "site": site,
+        "city": city,
+        "country": country,
+        "operator": operator,
+    }
+
+    if not candidates:
+        return {
+            "collector_id": collector_id,
+            "name": collector_id,
+            "success": False,
+            "failure_reason": (
+                "No source coordinates or online geocoding result reached"
+                " city-level precision for this collector."
+            ),
+            "candidates": [],
+            "attempted_queries": list(attempted_queries),
+            "llm_failure_reason": llm_failure_reason,
+            "context": context,
+        }
+
+    return {
+        "collector_id": collector_id,
+        "name": collector_id,
+        "success": True,
+        "candidates": [candidate.to_dict() for candidate in candidates],
+        "best_candidate": candidates[0].to_dict(),
+        "attempted_queries": list(attempted_queries),
+        "context": context,
+    }
+
+
 @router.get("/overview/summary")
 async def get_bgp_overview_summary(
     current_user: User = Depends(get_current_user),

backend/app/api/v1/collected_data.py

@@ -102,7 +102,12 @@ def build_search_rank_sql(search: Optional[str]) -> str:
     """
 
 
-def serialize_collected_row(row, source_name_map: dict[str, str] | None = None) -> dict:
+def serialize_collected_row(
+    row,
+    source_name_map: dict[str, str] | None = None,
+    *,
+    include_metadata: bool = True,
+) -> dict:
     metadata = row[7]
     source = row[1]
     return {
@@ -120,7 +125,7 @@ def serialize_collected_row(row, source_name_map: dict[str, str] | None = None)
         "longitude": get_metadata_field(metadata, "longitude"),
         "value": get_metadata_field(metadata, "value"),
         "unit": get_metadata_field(metadata, "unit"),
-        "metadata": metadata,
+        "metadata": metadata if include_metadata else None,
         "cores": get_metadata_field(metadata, "cores"),
         "rmax": get_metadata_field(metadata, "rmax"),
         "rpeak": get_metadata_field(metadata, "rpeak"),
@@ -145,6 +150,7 @@ async def list_collected_data(
     search: Optional[str] = Query(None, description="搜索名称"),
     page: int = Query(1, ge=1, description="页码"),
     page_size: int = Query(20, ge=1, le=100, description="每页数量"),
+    include_metadata: bool = Query(True, description="是否返回完整 metadata 字段"),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -201,7 +207,7 @@ async def list_collected_data(
 
     data = []
     for row in rows:
-        data.append(serialize_collected_row(row[:11], source_name_map))
+        data.append(serialize_collected_row(row[:11], source_name_map, include_metadata=include_metadata))
 
     return {
         "total": total,

backend/app/api/v1/data_products.py

@@ -0,0 +1,98 @@
+from datetime import UTC, datetime
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.v1.visualization import get_visualization_geo_summary
+from app.core.time import to_iso8601_utc
+from app.db.session import get_db
+
+router = APIRouter()
+
+PRODUCT_DEFINITIONS: dict[str, dict] = {
+    "vessels": {
+        "name": "船只",
+        "sources": ["aisstream_vessels", "barentswatch_vessels"],
+        "primary_stat_key": "vessel_count",
+        "stat_keys": ["vessel_count", "vessel_raw_unique_mmsi", "vessel_legacy_unique_mmsi"],
+    },
+    "cables": {
+        "name": "海底光缆",
+        "sources": [
+            "arcgis_cables",
+            "arcgis_landing_points",
+            "arcgis_cable_landing_relation",
+            "telegeography_cables",
+            "telegeography_landing",
+            "telegeography_systems",
+            "fao_landing_points",
+        ],
+        "primary_stat_key": "cable_count",
+        "stat_keys": ["cable_count", "landing_point_count"],
+    },
+    "satellites": {
+        "name": "卫星",
+        "sources": ["celestrak_tle", "spacetrack_tle"],
+        "primary_stat_key": "satellite_count",
+        "stat_keys": ["satellite_count"],
+    },
+    "bgp": {
+        "name": "BGP",
+        "sources": [
+            "ris_live_bgp",
+            "bgpstream_bgp",
+            "iptoasn_prefix_geo",
+            "opengeofeed_prefix_geo",
+            "nro_delegated_prefix_geo",
+        ],
+        "primary_stat_key": "bgp_event_count",
+        "stat_keys": ["bgp_event_count", "bgp_incident_count", "bgp_anomaly_count", "bgp_collector_count"],
+    },
+    "compute": {
+        "name": "算力",
+        "sources": ["top500", "epoch_ai_gpu"],
+        "primary_stat_key": "compute_center_count",
+        "stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
+    },
+}
+
+
+def _build_product_status(product_id: str, summary: dict) -> dict:
+    definition = PRODUCT_DEFINITIONS[product_id]
+    stats = summary.get("stats", {})
+    product_stats = {key: stats.get(key, 0) for key in definition["stat_keys"]}
+    total_count = int(product_stats.get(definition["primary_stat_key"]) or 0)
+    return {
+        "product_id": product_id,
+        "name": definition["name"],
+        "sources": definition["sources"],
+        "generated_at": summary.get("generated_at") or to_iso8601_utc(datetime.now(UTC)),
+        "total_count": total_count,
+        "stats": product_stats,
+        "build_state": "ready",
+        "stats_scope": "global",
+        "stats_freshness": "cached_or_indexed",
+    }
+
+
+@router.get("")
+async def list_data_products(db: AsyncSession = Depends(get_db)):
+    summary = await get_visualization_geo_summary(db)
+    return {
+        "generated_at": summary.get("generated_at"),
+        "data": [
+            _build_product_status(product_id, summary)
+            for product_id in PRODUCT_DEFINITIONS
+        ],
+    }
+
+
+@router.get("/{product_id}/status")
+async def get_data_product_status(
+    product_id: str,
+    db: AsyncSession = Depends(get_db),
+):
+    if product_id not in PRODUCT_DEFINITIONS:
+        raise HTTPException(status_code=404, detail="Unknown data product")
+    summary = await get_visualization_geo_summary(db)
+    return _build_product_status(product_id, summary)

backend/app/api/v1/datasource_config.py

@@ -5,47 +5,125 @@ from datetime import datetime
 import base64
 import json
 import re
-from fastapi import APIRouter, Depends, HTTPException, status
+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
-from sqlalchemy import select, func
+from sqlalchemy import delete, select, func
 from sqlalchemy.ext.asyncio import AsyncSession
 from pydantic import BaseModel, Field
 import httpx
 
 from app.core.target_schema_registry import get_target_schema, list_target_schemas
 from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.core.enums import AuthType, MappingValidationStatus, UserRole
 from app.db.session import get_db
 from app.models.user import User
 from app.models.datasource_config import DataSourceConfig
 from app.models.datasource_mapping import DataSourceMappingTemplate
+from app.models.collected_data import CollectedData
+from app.models.vessel import AISRawObservation, AISSourceHealth
 from app.core.security import get_current_user
 from app.core.cache import cache
 from app.core.time import to_iso8601_utc
+from app.ai_tasks.prompts import get_effective_prompt
 from app.schemas.ai import SituationalAnalysisRequest
 from app.services.ai_client import AIProviderClient, get_ai_provider_client
 from app.services.datasource_mapping import (
     MappingError,
     build_heuristic_mapping,
     execute_mapping,
-    persist_mapped_records,
     redact_for_llm,
     stable_payload_hash,
 )
+from app.services.custom_datasource_runtime import (
+    CustomDatasourceRuntimeError,
+    get_custom_stream_status,
+    run_mapped_rest_config,
+    run_mapped_websocket_config,
+    start_custom_stream,
+    stop_custom_stream,
+    test_websocket_config,
+)
 from app.services.datasource_connectivity import (
+    _resolve_aisstream_api_key,
+    _resolve_spacetrack_credentials_with_override,
     get_builtin_connection_status,
     save_connectivity_success,
     strip_connectivity_validation,
     test_builtin_connectivity,
 )
+from app.services.barentswatch import resolve_barentswatch_config
+from app.services.persistent_logs import record_audit_log
 
 router = APIRouter()
 
+DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
+SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
+
+
+def _user_role_value(user: User) -> str:
+    role = getattr(user, "role", "")
+    return str(getattr(role, "value", role) or "").lower()
+
+
+def _user_display_name(user: User) -> str:
+    return str(getattr(user, "username", None) or getattr(user, "email", None) or getattr(user, "id", ""))
+
+
+async def _record_datasource_secret_reveal(
+    *,
+    current_user: User,
+    request: Request,
+    target_id: str,
+    result: str,
+    details: dict[str, Any],
+) -> None:
+    await record_audit_log(
+        action="datasource_config.secret.reveal",
+        actor_id=getattr(current_user, "id", None),
+        actor_name=_user_display_name(current_user),
+        target_type="datasource_config_secret",
+        target_id=target_id,
+        result=result,
+        ip=request.client.host if request.client else None,
+        details=details,
+    )
+
+
+async def _ensure_datasource_secret_reveal_allowed(
+    current_user: User,
+    request: Request,
+    target_id: str,
+    details: dict[str, Any],
+) -> None:
+    if _user_role_value(current_user) in SECRET_REVEAL_ROLES:
+        return
+    await _record_datasource_secret_reveal(
+        current_user=current_user,
+        request=request,
+        target_id=target_id,
+        result="denied",
+        details={**details, "role": _user_role_value(current_user)},
+    )
+    raise HTTPException(
+        status_code=status.HTTP_403_FORBIDDEN,
+        detail="Only administrators can reveal datasource credentials",
+    )
+
+def _default_builtin_config(name: str) -> dict[str, Any]:
+    return {"timeout": 30, "retry": 3}
+
+
+def _default_builtin_source_type(name: str) -> str:
+    if name == "aisstream_vessels":
+        return "websocket"
+    return "http"
+
 
 class DataSourceConfigCreate(BaseModel):
     name: str = Field(..., min_length=1, max_length=100)
     description: Optional[str] = None
-    source_type: str = Field(..., description="http, api, database")
+    source_type: str = Field(..., description="rest, websocket, http, api, database")
     endpoint: str = Field(..., max_length=500)
-    auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
+    auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic")
     auth_config: dict = Field(default={})
     headers: dict = Field(default={})
     config: dict = Field(default={"timeout": 30, "retry": 3})
@@ -56,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel):
     description: Optional[str] = None
     source_type: Optional[str] = None
     endpoint: Optional[str] = Field(None, max_length=500)
-    auth_type: Optional[str] = None
+    auth_type: Optional[AuthType] = None
     auth_config: Optional[dict] = None
     headers: Optional[dict] = None
     config: Optional[dict] = None
@@ -131,7 +209,7 @@ class MappingTemplateCreate(BaseModel):
     mapping_json: dict
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
+    validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT
     is_active: bool = False
 
 
@@ -140,7 +218,7 @@ class MappingTemplateUpdate(BaseModel):
     mapping_json: Optional[dict] = None
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
+    validation_status: Optional[MappingValidationStatus] = None
     is_active: Optional[bool] = None
 
 
@@ -219,6 +297,8 @@ def _build_query_params(auth_type: str, auth_config: dict, config: dict) -> dict
 
 
 async def fetch_custom_sample_from_config(config: DataSourceConfig, limit_bytes: int) -> Any:
+    if str(config.source_type or "").lower() in {"websocket", "ws"}:
+        raise HTTPException(status_code=400, detail="WebSocket sources must use connection test or run-mapped stream.")
     request_config = config.config or {}
     method = str(request_config.get("method") or request_config.get("request_method") or "GET").upper()
     if method not in {"GET", "POST"}:
@@ -318,7 +398,7 @@ async def list_configs(
     """List all user-defined data source configurations"""
     query = select(DataSourceConfig)
     if active_only:
-        query = query.where(DataSourceConfig.is_active == True)
+        query = query.where(DataSourceConfig.is_active)
     query = query.order_by(DataSourceConfig.created_at.desc())
 
     result = await db.execute(query)
@@ -351,7 +431,7 @@ async def list_all_datasources(
     db: AsyncSession = Depends(get_db),
 ):
     """List all data sources: YAML defaults + DB overrides"""
-    from app.core.data_sources import COLLECTOR_URL_KEYS, get_data_sources_config
+    from app.core.data_sources import get_data_sources_config
 
     config = get_data_sources_config()
 
@@ -359,33 +439,144 @@ async def list_all_datasources(
     db_configs = {c.name: c for c in db_query.scalars().all()}
 
     result = []
-    for name, yaml_key in COLLECTOR_URL_KEYS.items():
+    for name, metadata in DEFAULT_DATASOURCES.items():
         yaml_url = config.get_yaml_url(name)
         db_config = db_configs.get(name)
+        default_config = _default_builtin_config(name)
+        default_url = yaml_url
+        db_auth_config = db_config.auth_config or {} if db_config else {}
 
         result.append(
             {
                 "name": name,
-                "default_url": yaml_url,
+                "requires_credentials": bool(metadata.get("requires_credentials", False)),
-                "endpoint": db_config.endpoint if db_config else yaml_url,
+                "credential_provider": metadata.get("credential_provider"),
+                "credential_status": metadata.get("credential_status", "none"),
+                "default_url": default_url,
+                "endpoint": db_config.endpoint if db_config else default_url,
                 "is_overridden": db_config is not None and db_config.endpoint != yaml_url
-                if yaml_url
+                if default_url
                 else db_config is not None,
                 "is_active": db_config.is_active if db_config else True,
-                "source_type": db_config.source_type if db_config else "http",
+                "source_type": db_config.source_type if db_config else _default_builtin_source_type(name),
                 "auth_type": db_config.auth_type if db_config else "none",
+                "auth_config": {
+                    "client_id": db_auth_config.get("client_id") or "",
+                    "username": db_auth_config.get("username") or "",
+                    "key_name": db_auth_config.get("key_name") or db_auth_config.get("param_name") or "",
+                    "param_name": db_auth_config.get("param_name") or db_auth_config.get("key_name") or "",
+                    "location": db_auth_config.get("location") or db_auth_config.get("in") or "",
+                    "in": db_auth_config.get("in") or db_auth_config.get("location") or "",
+                },
+                "auth_configured": {
+                    "api_key": bool(db_auth_config.get("api_key")),
+                    "client_id": bool(db_auth_config.get("client_id")),
+                    "client_secret": bool(db_auth_config.get("client_secret")),
+                    "username": bool(db_auth_config.get("username")),
+                    "password": bool(db_auth_config.get("password")),
+                },
                 "headers": db_config.headers if db_config else {},
-                "config": strip_connectivity_validation(db_config.config if db_config else {}),
+                "config": strip_connectivity_validation(db_config.config if db_config else default_config),
                 "config_id": db_config.id if db_config else None,
                 "description": db_config.description
                 if db_config
-                else f"Data source from YAML: {yaml_key}",
+                else f"内置采集器默认配置：{metadata.get('display_name') or metadata.get('name') or name}",
             }
         )
 
     return {"total": len(result), "data": result}
 
 
+@router.get("/configs/secrets")
+async def reveal_builtin_config_secrets(
+    request: Request,
+    name: str = Query(..., min_length=1),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Reveal configured built-in datasource credentials for admin editing."""
+    source = name.strip()
+    metadata = DEFAULT_DATASOURCES.get(source)
+    if not metadata or not metadata.get("requires_credentials"):
+        raise HTTPException(status_code=404, detail="Credentialed datasource config not found")
+
+    provider = str(metadata.get("credential_provider") or "")
+    target_id = f"datasource_config:{source}"
+    await _ensure_datasource_secret_reveal_allowed(
+        current_user,
+        request,
+        target_id,
+        {"source": source, "provider": provider},
+    )
+
+    result = await db.execute(select(DataSourceConfig).where(DataSourceConfig.name == source))
+    record = result.scalar_one_or_none()
+    auth_config = dict(record.auth_config or {}) if record else {}
+    payload: dict[str, Any] = {
+        "name": source,
+        "provider": provider,
+    }
+    details: dict[str, Any] = {"source": source, "provider": provider}
+
+    if provider == "barentswatch":
+        resolved = await resolve_barentswatch_config(db)
+        client_id = str(auth_config.get("client_id") or resolved.client_id or "")
+        client_secret = str(auth_config.get("client_secret") or resolved.client_secret or "")
+        source_label = "datasource_config" if auth_config.get("client_id") or auth_config.get("client_secret") else resolved.credential_source
+        payload.update(
+            {
+                "client_id": client_id,
+                "client_secret": client_secret,
+                "client_id_source": source_label if client_id else "missing",
+                "client_secret_source": source_label if client_secret else "missing",
+            }
+        )
+        details.update(
+            {
+                "client_id_configured": bool(client_id),
+                "client_secret_configured": bool(client_secret),
+                "credential_source": source_label,
+            }
+        )
+    elif provider == "aisstream":
+        api_key, api_key_source = await _resolve_aisstream_api_key(db)
+        payload.update({"api_key": api_key, "api_key_source": api_key_source})
+        details.update({"api_key_configured": bool(api_key), "api_key_source": api_key_source})
+    elif provider == "spacetrack":
+        if auth_config.get("username") or auth_config.get("password"):
+            username = str(auth_config.get("username") or "")
+            password = str(auth_config.get("password") or "")
+            credential_source = "datasource_config"
+        else:
+            username, password, credential_source = _resolve_spacetrack_credentials_with_override()
+        payload.update(
+            {
+                "username": username,
+                "password": password,
+                "username_source": credential_source if username else "missing",
+                "password_source": credential_source if password else "missing",
+            }
+        )
+        details.update(
+            {
+                "username_configured": bool(username),
+                "password_configured": bool(password),
+                "credential_source": credential_source,
+            }
+        )
+    else:
+        raise HTTPException(status_code=400, detail="Datasource credential provider is not supported")
+
+    await _record_datasource_secret_reveal(
+        current_user=current_user,
+        request=request,
+        target_id=target_id,
+        result="success",
+        details=details,
+    )
+    return payload
+
+
 @router.get("/configs/{config_id}")
 async def get_config(
     config_id: int,
@@ -464,6 +655,8 @@ async def update_config(
     for field, value in update_data.items():
         if field == "config":
             value = strip_connectivity_validation(value)
+        if field == "auth_config" and value == {} and (config.auth_config or {}):
+            continue
         setattr(config, field, value)
 
     await db.commit()
@@ -481,6 +674,8 @@ async def update_config(
 @router.delete("/configs/{config_id}")
 async def delete_config(
     config_id: int,
+    delete_mappings: bool = Query(False),
+    delete_source_data: bool = Query(False),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -491,12 +686,59 @@ async def delete_config(
     if not config:
         raise HTTPException(status_code=404, detail="Configuration not found")
 
+    deleted_mappings = 0
+    deleted_records = {
+        "collected_data": 0,
+        "ais_raw_observations": 0,
+        "ais_source_health": 0,
+    }
+
+    if delete_source_data:
+        collected_result = await db.execute(
+            delete(CollectedData).where(CollectedData.source == config.name)
+        )
+        raw_result = await db.execute(
+            delete(AISRawObservation).where(AISRawObservation.source == config.name)
+        )
+        health_result = await db.execute(
+            delete(AISSourceHealth).where(AISSourceHealth.source == config.name)
+        )
+        deleted_records = {
+            "collected_data": collected_result.rowcount or 0,
+            "ais_raw_observations": raw_result.rowcount or 0,
+            "ais_source_health": health_result.rowcount or 0,
+        }
+
+    if delete_mappings or delete_source_data:
+        mapping_result = await db.execute(
+            delete(DataSourceMappingTemplate).where(
+                DataSourceMappingTemplate.datasource_config_id == config_id
+            )
+        )
+        deleted_mappings = mapping_result.rowcount or 0
+
     await db.delete(config)
     await db.commit()
 
     cache.delete_pattern("datasource_configs:*")
 
-    return {"message": "Configuration deleted successfully"}
+    if delete_source_data and (config.config or {}).get("target_schema") == "vessel_ais":
+        from app.core.websocket.broadcaster import broadcaster
+
+        await broadcaster.broadcast_custom(
+            "vessels",
+            {
+                "action": "reload",
+                "source": config.name,
+                "reason": "custom_source_deleted",
+            },
+        )
+
+    return {
+        "message": "Configuration deleted successfully",
+        "deleted_mappings": deleted_mappings,
+        "deleted_records": deleted_records,
+    }
 
 
 @router.post("/configs/{config_id}/test")
@@ -513,6 +755,8 @@ async def test_config(
         raise HTTPException(status_code=404, detail="Configuration not found")
 
     try:
+        if str(config.source_type or "").lower() in {"websocket", "ws"}:
+            return await test_websocket_config(config)
         result = await test_endpoint(
             endpoint=config.endpoint,
             auth_type=config.auth_type,
@@ -543,6 +787,18 @@ async def test_new_config(
 ):
     """Test a new data source configuration without saving"""
     try:
+        if str(config_data.source_type or "").lower() in {"websocket", "ws"}:
+            config = DataSourceConfig(
+                name=config_data.name,
+                description=config_data.description,
+                source_type=config_data.source_type,
+                endpoint=config_data.endpoint,
+                auth_type=config_data.auth_type,
+                auth_config=config_data.auth_config,
+                headers=config_data.headers,
+                config=config_data.config,
+            )
+            return await test_websocket_config(config)
         result = await test_endpoint(
             endpoint=config_data.endpoint,
             auth_type=config_data.auth_type,
@@ -601,6 +857,7 @@ async def connect_builtin_config(
         config_data.headers,
         config_data.config,
         db,
+        config_data.auth_config,
     )
     if result.get("success") and result.get("checksum"):
         validation = await save_connectivity_success(
@@ -660,6 +917,7 @@ async def get_datasource_target_schemas(
 @router.post("/mappings/propose")
 async def propose_datasource_mapping(
     payload: MappingProposeRequest,
+    db: AsyncSession = Depends(get_db),
     current_user: User = Depends(get_current_user),
     ai_client: AIProviderClient = Depends(get_ai_provider_client),
 ):
@@ -673,14 +931,12 @@ async def propose_datasource_mapping(
     generated_by = "heuristic"
     if payload.use_ai:
         try:
+            prompt = await get_effective_prompt(db, DATASOURCE_MAPPING_PROMPT_KEY)
             response = await ai_client.analyze(
                 SituationalAnalysisRequest(
                     title=f"Generate datasource mapping for {schema.key}",
-                    objective=(
+                    objective=prompt.prompt,
-                        "Return only JSON for a deterministic mapping DSL. "
+                    system_prompt=prompt.system_prompt or None,
-                        "The JSON must contain source.items_path and fields. "
-                        "Do not include prose or code."
-                    ),
                     context={
                         "target_schema": schema.to_dict(),
                         "sample_payload": redacted_sample,
@@ -867,6 +1123,8 @@ async def update_datasource_mapping(
 @router.post("/{config_id}/run-mapped")
 async def run_mapped_datasource(
     config_id: int,
+    background: bool = Query(False, description="For WebSocket sources, start a background stream task."),
+    debug_max_messages: int | None = Query(None, ge=1),
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
@@ -875,20 +1133,24 @@ async def run_mapped_datasource(
     if not datasource:
         raise HTTPException(status_code=404, detail="Configuration not found")
 
-    result = await db.execute(
-        select(DataSourceMappingTemplate)
-        .where(DataSourceMappingTemplate.datasource_config_id == config_id)
-        .where(DataSourceMappingTemplate.is_active.is_(True))
-        .order_by(DataSourceMappingTemplate.version.desc())
-        .limit(1)
-    )
-    mapping = result.scalar_one_or_none()
-    if not mapping:
-        raise HTTPException(status_code=404, detail="No active mapping template found")
-
     try:
-        sample = await fetch_custom_sample_from_config(datasource, 5_000_000)
+        if str(datasource.source_type or "").lower() in {"websocket", "ws"}:
-        mapped = execute_mapping(sample, mapping.mapping_json, mapping.target_schema)
+            if background and debug_max_messages is None:
+                started = start_custom_stream(config_id)
+                if not started:
+                    raise HTTPException(status_code=409, detail="Custom WebSocket source is already running")
+                return {
+                    "status": "started",
+                    "datasource_config_id": config_id,
+                    "stream": get_custom_stream_status(config_id),
+                }
+            return await run_mapped_websocket_config(
+                db,
+                datasource,
+                debug_max_messages=debug_max_messages,
+            )
+
+        return await run_mapped_rest_config(db, datasource)
     except httpx.HTTPStatusError as exc:
         raise HTTPException(
             status_code=exc.response.status_code,
@@ -896,36 +1158,26 @@ async def run_mapped_datasource(
         ) from exc
     except httpx.HTTPError as exc:
         raise HTTPException(status_code=502, detail=f"Datasource request failed: {exc}") from exc
-    except (MappingError, ValueError) as exc:
+    except (CustomDatasourceRuntimeError, MappingError, ValueError) as exc:
         raise HTTPException(status_code=400, detail=f"Mapping failed: {exc}") from exc
 
-    if mapped["failed_count"] > 0:
-        return {
-            "status": "failed",
-            "datasource_config_id": config_id,
-            "mapping_id": mapping.id,
-            "mapping_version": mapping.version,
-            "target_schema": mapping.target_schema,
-            "mapped_count": mapped["mapped_count"],
-            "failed_count": mapped["failed_count"],
-            "errors": mapped["errors"][:20],
-        }
 
-    written_count = await persist_mapped_records(
+@router.post("/{config_id}/stop-mapped")
-        db,
+async def stop_mapped_datasource(
-        datasource_name=datasource.name,
+    config_id: int,
-        datasource_config_id=datasource.id,
+    current_user: User = Depends(get_current_user),
-        target_schema=mapping.target_schema,
+):
-        records=mapped["records"],
+    stopped = await stop_custom_stream(config_id)
-        mapping_version=mapping.version,
-    )
     return {
-        "status": "success",
+        "status": "stopped" if stopped else "not_running",
         "datasource_config_id": config_id,
-        "mapping_id": mapping.id,
+        "stream": get_custom_stream_status(config_id),
-        "mapping_version": mapping.version,
-        "target_schema": mapping.target_schema,
-        "fetched_count": mapped["total_items"],
-        "mapped_count": mapped["mapped_count"],
-        "written_count": written_count,
     }
+
+
+@router.get("/{config_id}/stream-status")
+async def get_mapped_stream_status(
+    config_id: int,
+    current_user: User = Depends(get_current_user),
+):
+    return get_custom_stream_status(config_id)

backend/app/api/v1/docs.py

@@ -0,0 +1,102 @@
+"""Authenticated documentation APIs."""
+
+from __future__ import annotations
+
+from fastapi import APIRouter, Depends, HTTPException, status
+from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
+from sqlalchemy import text
+
+from app.core.security import decode_token
+from app.db.session import async_session_factory
+from app.models.user import User
+from app.services.docs_gatekeeper import (
+    DOCS_BY_SLUG,
+    VALID_DOCS_LANGS,
+    can_read_doc,
+    catalog_for_user,
+    doc_path_for,
+    title_for,
+)
+
+router = APIRouter()
+optional_bearer = HTTPBearer(auto_error=False)
+
+
+async def get_optional_current_user(
+    credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
+) -> User | None:
+    if credentials is None:
+        return None
+
+    payload = decode_token(credentials.credentials)
+    if payload is None or payload.get("type") != "access" or payload.get("sub") is None:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid token",
+        )
+
+    async with async_session_factory() as db:
+        result = await db.execute(
+            text(
+                "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
+            ),
+            {"id": int(payload["sub"])},
+        )
+        row = result.fetchone()
+    if row is None or not row[5]:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="User not found or inactive",
+        )
+
+    user = User()
+    user.id = row[0]
+    user.username = row[1]
+    user.email = row[2]
+    user.password_hash = row[3]
+    user.role = row[4]
+    user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
+    return user
+
+
+@router.get("/catalog")
+async def get_docs_catalog(current_user: User | None = Depends(get_optional_current_user)):
+    return {
+        "items": catalog_for_user(current_user),
+        "authenticated": current_user is not None,
+    }
+
+
+@router.get("/{lang}/{slug}")
+async def get_doc_content(
+    lang: str,
+    slug: str,
+    current_user: User | None = Depends(get_optional_current_user),
+):
+    if lang not in VALID_DOCS_LANGS:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Document not found")
+
+    entry = DOCS_BY_SLUG.get(slug)
+    if entry is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Document not found")
+
+    path = doc_path_for(entry, lang)
+    if not path.exists():
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Document not found")
+
+    if not can_read_doc(entry, current_user):
+        if current_user is None:
+            raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Insufficient Docs permissions")
+
+    return {
+        "slug": entry.slug,
+        "filename": entry.filename,
+        "lang": lang,
+        "title": title_for(entry, lang),
+        "group": entry.group,
+        "order": entry.order,
+        "access": entry.access,
+        "markdown": path.read_text(encoding="utf-8"),
+    }

backend/app/api/v1/earth.py

@@ -0,0 +1,682 @@
+"""Earth asset management APIs."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+from uuid import uuid4
+
+from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
+from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
+from pydantic import BaseModel, Field
+from sqlalchemy import delete, func, select, text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.config import settings as app_settings
+from app.core.security import decode_token, get_current_user, redis_client
+from app.db.session import get_db
+from app.models.collected_data import CollectedData
+from app.models.datasource import DataSource
+from app.models.datasource_config import DataSourceConfig
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.services.tv_streams import get_tv_settings_payload
+from app.services.earth_news import (
+    get_earth_news_sources_payload,
+    reset_earth_news_sources_payload,
+    save_earth_news_sources_payload,
+    test_news_source_config,
+)
+from app.services.earth_news_manual import (
+    broadcast_manual_news_changed,
+    create_manual_news_group,
+    delete_manual_news_item,
+    get_news_record_or_404,
+    import_manual_news_items,
+    list_news_groups,
+    list_news_records,
+    parse_manual_news_import_upload,
+    rename_manual_news_group,
+    reprocess_manual_news_item,
+    serialize_news_record,
+    upsert_manual_news_item,
+)
+from app.services.earth_boundaries import (
+    EarthBoundaryBuildError,
+    get_boundary_build_status,
+    get_boundary_status,
+    save_boundary_config,
+    start_boundary_build_job,
+)
+
+
+router = APIRouter()
+optional_bearer = HTTPBearer(auto_error=False)
+REPO_ROOT = Path(__file__).resolve().parents[4]
+EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
+EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
+EARTH_BRAND_CATEGORY = "earth_brand"
+EARTH_ABOUT_CATEGORY = "earth_about"
+SYSTEM_SETTINGS_CATEGORY = "system"
+MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
+ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
+
+
+def _app_version_label() -> str:
+    version = str(app_settings.VERSION or "").strip() or "0.0.0"
+    return version if version.startswith("v") else f"v{version}"
+
+
+DEFAULT_EARTH_BRAND = {
+    "logo_src": "/earth/assets/brand/earth-logo.png",
+    "title_src": "/earth/assets/brand/title-zh.png",
+    "title_text": "智能星球计划",
+    "subtitle": "现实层宇宙全息感知系统",
+    "description": "卫星 · 海底光缆 · 算力基础设施",
+    "aria_label": "智能星球计划品牌标识",
+    "title_alt": "智能星球计划",
+}
+
+DEFAULT_EARTH_ABOUT = {
+    "logo_src": "/earth/assets/brand/lim-logo.png",
+    "kicker": "About",
+    "title": "智能星球计划",
+    "version": _app_version_label(),
+    "description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。",
+    "meta": [
+        {"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
+        {"label": "策划人", "value": "方兴东、黄柳青"},
+        {"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
+    ],
+}
+EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
+
+
+class EarthBoundaryConfigPayload(BaseModel):
+    config: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthBrandPayload(BaseModel):
+    logo_src: str = Field(default=DEFAULT_EARTH_BRAND["logo_src"], max_length=1000)
+    title_src: str = Field(default=DEFAULT_EARTH_BRAND["title_src"], max_length=1000)
+    title_text: str = Field(default=DEFAULT_EARTH_BRAND["title_text"], max_length=120)
+    subtitle: str = Field(default=DEFAULT_EARTH_BRAND["subtitle"], max_length=160)
+    description: str = Field(default=DEFAULT_EARTH_BRAND["description"], max_length=200)
+    aria_label: str = Field(default=DEFAULT_EARTH_BRAND["aria_label"], max_length=200)
+    title_alt: str = Field(default=DEFAULT_EARTH_BRAND["title_alt"], max_length=200)
+
+
+class EarthAboutMetaItem(BaseModel):
+    label: str = Field(default="", max_length=80)
+    value: str = Field(default="", max_length=240)
+
+
+class EarthAboutPayload(BaseModel):
+    logo_src: str = Field(default=DEFAULT_EARTH_ABOUT["logo_src"], max_length=1000)
+    kicker: str = Field(default=DEFAULT_EARTH_ABOUT["kicker"], max_length=80)
+    title: str = Field(default=DEFAULT_EARTH_ABOUT["title"], max_length=160)
+    version: str = Field(default=DEFAULT_EARTH_ABOUT["version"], max_length=80)
+    description: str = Field(default=DEFAULT_EARTH_ABOUT["description"], max_length=800)
+    meta: list[EarthAboutMetaItem] = Field(default_factory=list)
+
+
+class EarthNewsSourcesPayload(BaseModel):
+    cache_version: int | None = None
+    source_tags: list[dict[str, Any]] = Field(default_factory=list)
+    categories: list[dict[str, Any]] = Field(default_factory=list)
+    item_tag_rules: list[dict[str, Any]] = Field(default_factory=list)
+    sources: list[dict[str, Any]] = Field(default_factory=list)
+    health: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsSourceTestPayload(BaseModel):
+    source: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsManualItemPayload(BaseModel):
+    title: str = Field(default="", max_length=500)
+    summary: str = Field(default="", max_length=1200)
+    content: str = Field(default="", max_length=12000)
+    url: str = Field(default="", max_length=2000)
+    source: str = Field(default="", max_length=255)
+    region: str = Field(default="global", max_length=80)
+    published_at: str | None = None
+    category: str = Field(default="other", max_length=80)
+    tags: list[str] = Field(default_factory=list)
+    location: dict[str, Any] | None = None
+    homepage_url: str = Field(default="", max_length=2000)
+    content_language: str = Field(default="", max_length=32)
+    group_id: str | None = Field(default=None, max_length=120)
+
+
+class EarthNewsManualGroupPayload(BaseModel):
+    name: str = Field(default="", max_length=120)
+
+
+def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
+    merged = DEFAULT_EARTH_BRAND.copy()
+    if payload:
+        for key in DEFAULT_EARTH_BRAND:
+            value = payload.get(key)
+            if value is not None:
+                merged[key] = str(value).strip()
+
+    if not merged["title_text"]:
+        merged["title_text"] = DEFAULT_EARTH_BRAND["title_text"]
+    if not merged["aria_label"]:
+        merged["aria_label"] = merged["title_text"]
+    if not merged["title_alt"]:
+        merged["title_alt"] = merged["title_text"]
+    return merged
+
+
+def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str, Any]:
+    merged: dict[str, Any] = {
+        key: value
+        for key, value in DEFAULT_EARTH_ABOUT.items()
+        if key != "meta"
+    }
+    raw_meta = DEFAULT_EARTH_ABOUT["meta"]
+    if payload:
+        for key in ("logo_src", "kicker", "title", "description"):
+            value = payload.get(key)
+            if value is not None:
+                merged[key] = str(value).strip()
+        raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
+    merged["version"] = _app_version_label()
+
+    for key, default_value in DEFAULT_EARTH_ABOUT.items():
+        if key == "meta":
+            continue
+        if not merged.get(key):
+            merged[key] = default_value
+
+    normalized_meta: list[dict[str, str]] = []
+    for item in raw_meta:
+        if not isinstance(item, dict):
+            continue
+        label = str(item.get("label") or "").strip()
+        value = str(item.get("value") or "").strip()
+        if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
+            value = "方兴东、黄柳青"
+        if label or value:
+            normalized_meta.append({"label": label, "value": value})
+    if not normalized_meta:
+        normalized_meta = [dict(item) for item in DEFAULT_EARTH_ABOUT["meta"]]
+    merged["meta"] = normalized_meta
+    return merged
+
+
+def _is_demo_mode_enabled(payload: Any) -> bool:
+    return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
+
+
+async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
+    )
+    return result.scalar_one_or_none()
+
+
+async def _get_earth_brand_payload(db: AsyncSession) -> dict[str, Any]:
+    record = await _get_earth_brand_record(db)
+    return {
+        "brand": _normalize_earth_brand_payload(record.payload if record else None),
+        "is_default": record is None,
+    }
+
+
+async def _get_earth_about_record(db: AsyncSession) -> SystemSetting | None:
+    result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY)
+    )
+    return result.scalar_one_or_none()
+
+
+async def _get_earth_about_payload(db: AsyncSession) -> dict[str, Any]:
+    record = await _get_earth_about_record(db)
+    return {
+        "about": _normalize_earth_about_payload(record.payload if record else None),
+        "is_default": record is None,
+    }
+
+
+async def _get_optional_current_user(
+    credentials: HTTPAuthorizationCredentials | None = Depends(optional_bearer),
+    db: AsyncSession = Depends(get_db),
+) -> User | None:
+    if credentials is None:
+        return None
+    token = credentials.credentials
+    if redis_client.sismember("blacklisted_tokens", token):
+        return None
+    payload = decode_token(token)
+    if payload is None or payload.get("type") != "access":
+        return None
+    user_id = payload.get("sub")
+    if user_id is None:
+        return None
+    result = await db.execute(
+        text(
+            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
+        ),
+        {"id": int(user_id)},
+    )
+    row = result.fetchone()
+    if row is None or not row[5]:
+        return None
+    user = User()
+    user.id = row[0]
+    user.username = row[1]
+    user.email = row[2]
+    user.password_hash = row[3]
+    user.role = row[4]
+    user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
+    return user
+
+
+@router.get("/brand")
+async def get_earth_brand(db: AsyncSession = Depends(get_db)):
+    return await _get_earth_brand_payload(db)
+
+
+@router.put("/brand")
+async def update_earth_brand(
+    payload: EarthBrandPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    normalized = _normalize_earth_brand_payload(payload.model_dump())
+    record = await _get_earth_brand_record(db)
+    if record is None:
+        record = SystemSetting(category=EARTH_BRAND_CATEGORY, payload=normalized)
+        db.add(record)
+    else:
+        record.payload = normalized
+    await db.commit()
+    await db.refresh(record)
+    return {"status": "updated", "brand": _normalize_earth_brand_payload(record.payload), "is_default": False}
+
+
+@router.delete("/brand")
+@router.post("/brand/reset")
+async def reset_earth_brand(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY))
+    await db.commit()
+    return {"status": "reset", "brand": DEFAULT_EARTH_BRAND.copy(), "is_default": True}
+
+
+@router.post("/brand/assets")
+async def upload_earth_brand_asset(
+    file: UploadFile = File(...),
+    _current_user: User = Depends(get_current_user),
+):
+    original_name = file.filename or ""
+    extension = Path(original_name).suffix.lower()
+    if extension not in ALLOWED_EARTH_BRAND_EXTENSIONS:
+        raise HTTPException(
+            status_code=400,
+            detail={
+                "code": "unsupported_file_type",
+                "message": "Only png, jpg, jpeg, webp, and svg brand assets are supported.",
+            },
+        )
+
+    content = await file.read(MAX_EARTH_BRAND_ASSET_BYTES + 1)
+    if len(content) > MAX_EARTH_BRAND_ASSET_BYTES:
+        raise HTTPException(
+            status_code=400,
+            detail={
+                "code": "file_too_large",
+                "message": "Brand asset must be 3 MB or smaller.",
+            },
+        )
+
+    EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
+    safe_name = f"{uuid4().hex}{extension}"
+    destination = EARTH_BRAND_ASSET_DIR / safe_name
+    destination.write_bytes(content)
+    asset_url = f"{EARTH_BRAND_ASSET_URL_PREFIX}/{safe_name}"
+    return {"url": asset_url, "filename": safe_name, "content_type": file.content_type}
+
+
+@router.get("/about")
+async def get_earth_about(db: AsyncSession = Depends(get_db)):
+    return await _get_earth_about_payload(db)
+
+
+@router.put("/about")
+async def update_earth_about(
+    payload: EarthAboutPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    normalized = _normalize_earth_about_payload(payload.model_dump())
+    record = await _get_earth_about_record(db)
+    if record is None:
+        record = SystemSetting(category=EARTH_ABOUT_CATEGORY, payload=normalized)
+        db.add(record)
+    else:
+        record.payload = normalized
+    await db.commit()
+    await db.refresh(record)
+    return {"status": "updated", "about": _normalize_earth_about_payload(record.payload), "is_default": False}
+
+
+@router.delete("/about")
+async def reset_earth_about(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    await db.execute(delete(SystemSetting).where(SystemSetting.category == EARTH_ABOUT_CATEGORY))
+    await db.commit()
+    return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
+
+
+@router.get("/news-sources")
+async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
+    return await get_earth_news_sources_payload(db)
+
+
+@router.put("/news-sources")
+async def update_earth_news_sources(
+    payload: EarthNewsSourcesPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await save_earth_news_sources_payload(db, payload.model_dump())
+
+
+@router.delete("/news-sources")
+@router.post("/news-sources/reset")
+async def reset_earth_news_sources(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await reset_earth_news_sources_payload(db)
+
+
+@router.post("/news-sources/test")
+async def test_earth_news_source(
+    payload: EarthNewsSourceTestPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await test_news_source_config(payload.source, db=db)
+
+
+@router.get("/news-groups")
+async def list_earth_news_groups_admin(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_groups(db)
+
+
+@router.post("/news-groups")
+async def create_earth_news_group_admin(
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await create_manual_news_group(db, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    return {"status": "ok", "group": group}
+
+
+@router.put("/news-groups/{group_id:path}")
+async def rename_earth_news_group_admin(
+    group_id: str,
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await rename_manual_news_group(db, group_id, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "group": group}
+
+
+@router.get("/news-items")
+async def list_earth_news_items_admin(
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=100),
+    source_type: str | None = Query(None),
+    region: str | None = Query(None),
+    category: str | None = Query(None),
+    status_filter: str | None = Query(None, alias="status"),
+    group_id: str | None = Query(None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_records(
+        db,
+        page=page,
+        page_size=page_size,
+        source_type=source_type,
+        region=region,
+        category=category,
+        status_filter=status_filter,
+        group_id=group_id,
+    )
+
+
+@router.post("/news-items")
+async def create_earth_news_item_admin(
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        result = await upsert_manual_news_item(db, payload.model_dump())
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.post("/news-items/import")
+async def import_earth_news_items_admin(
+    file: UploadFile = File(...),
+    group_id: str | None = Form(default=None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        payload = await parse_manual_news_import_upload(await file.read())
+        result = await import_manual_news_items(db, payload, group_id=group_id)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", **result}
+
+
+@router.put("/news-items/{item_id:path}")
+async def update_earth_news_item_admin(
+    item_id: str,
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        result = await upsert_manual_news_item(
+            db,
+            payload.model_dump(),
+            item_id_override=item_id,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.delete("/news-items/{item_id:path}")
+async def delete_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        deleted = await delete_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    if not deleted:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "deleted", "id": item_id}
+
+
+@router.post("/news-items/{item_id:path}/reprocess")
+async def reprocess_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        queued = await reprocess_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
+
+
+@router.get("/oobe-status")
+async def get_earth_oobe_status(
+    current_user: User | None = Depends(_get_optional_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    current_count_result = await db.execute(
+        select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
+    )
+    current_record_count = int(current_count_result.scalar() or 0)
+    system_result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
+    )
+    system_record = system_result.scalar_one_or_none()
+    demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
+
+    datasource_count_result = await db.execute(select(func.count(DataSource.id)))
+    datasource_count = int(datasource_count_result.scalar() or 0)
+    active_datasource_count_result = await db.execute(
+        select(func.count(DataSource.id)).where(DataSource.is_active.is_(True))
+    )
+    active_datasource_count = int(active_datasource_count_result.scalar() or 0)
+    config_result = await db.execute(select(func.count(DataSourceConfig.id)))
+    custom_config_count = int(config_result.scalar() or 0)
+
+    tv_payload = await get_tv_settings_payload(db)
+    tv_sources = tv_payload.get("sources") if isinstance(tv_payload, dict) else []
+    tv_source_count = len(tv_sources) if isinstance(tv_sources, list) else 0
+
+    boundary_status = get_boundary_status()
+    has_core_layers = bool(boundary_status.get("ready") or boundary_status.get("available") or boundary_status.get("status") in {"ready", "built", "ok"})
+    has_collected_data = current_record_count > 0
+    ready = has_collected_data
+
+    suggestions: list[str] = []
+    if demo_mode:
+        suggestions.append("演示模式已开启")
+    if not current_user:
+        suggestions.append("登录控制台")
+    if not has_collected_data:
+        suggestions.append("触发数据源采集")
+    if not custom_config_count:
+        suggestions.append("确认采集器配置")
+    if not has_core_layers:
+        suggestions.append("构建或启用 Earth 图层")
+
+    return {
+        "ready": ready,
+        "demo_mode": demo_mode,
+        "authenticated": current_user is not None,
+        "needs_login": current_user is None and not ready and not demo_mode,
+        "has_collected_data": has_collected_data,
+        "has_tv_sources": tv_source_count > 0,
+        "has_core_layers": has_core_layers,
+        "current_record_count": current_record_count,
+        "datasource_count": datasource_count,
+        "active_datasource_count": active_datasource_count,
+        "custom_config_count": custom_config_count,
+        "tv_source_count": tv_source_count,
+        "suggestions": suggestions,
+        "login_url": "/login?next=/datasources",
+        "datasources_url": "/datasources",
+        "collection_url": "/collection-management",
+    }
+
+
+@router.get("/boundaries/status")
+async def get_earth_boundary_status():
+    return get_boundary_status()
+
+def _is_loopback_request(request: Request) -> bool:
+    host = request.client.host if request.client else ""
+    return host in {"127.0.0.1", "::1", "localhost"} or host.startswith("127.")
+
+
+def _require_local_or_user(request: Request, user: User | None) -> None:
+    if user is not None or _is_loopback_request(request):
+        return
+    raise HTTPException(
+        status_code=status.HTTP_401_UNAUTHORIZED,
+        detail="Authentication required outside localhost",
+    )
+
+
+@router.put("/boundaries/config")
+async def update_earth_boundary_config(
+    payload: EarthBoundaryConfigPayload,
+    _current_user: User = Depends(get_current_user),
+):
+    try:
+        return save_boundary_config(payload.config)
+    except EarthBoundaryBuildError as exc:
+        raise HTTPException(
+            status_code=400,
+            detail={"code": exc.code, "message": str(exc), "details": exc.details},
+        ) from exc
+
+
+@router.post("/boundaries/build")
+async def build_earth_boundary_assets(
+    request: Request,
+    current_user: User | None = Depends(_get_optional_current_user),
+):
+    _require_local_or_user(request, current_user)
+    try:
+        return await start_boundary_build_job()
+    except EarthBoundaryBuildError as exc:
+        raise HTTPException(
+            status_code=400,
+            detail={"code": exc.code, "message": str(exc), "details": exc.details},
+        ) from exc
+
+
+@router.get("/boundaries/build/status")
+async def get_earth_boundary_build_status():
+    return get_boundary_build_status()

backend/app/api/v1/interactables.py

@@ -0,0 +1,190 @@
+"""CRUD APIs for persistent Earth interactables."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
+from pydantic import BaseModel, Field, field_validator
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.earth_interactable import EarthInteractable
+from app.models.user import User
+from app.services.earth_interactables import (
+    build_interactable_event,
+    interactables_to_geojson,
+    invalidate_interactable_cache,
+    list_interactables,
+    normalize_interactable_id,
+    publish_interactable_event,
+    serialize_interactable,
+)
+from app.services.earth_layer_cache import (
+    EarthLayerCachePolicy,
+    earth_layer_cache,
+    get_or_build_layer_payload,
+)
+
+router = APIRouter()
+INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
+    fresh_ttl_seconds=60,
+    stale_ttl_seconds=10 * 60,
+    max_features=5000,
+)
+
+
+class InteractableCreate(BaseModel):
+    id: str | None = Field(default=None, max_length=160)
+    layer: str = Field(default="default", min_length=1, max_length=80)
+    kind: str = Field(default="default", min_length=1, max_length=80)
+    label: str = Field(default="", max_length=255)
+    description: str = Field(default="", max_length=4000)
+    latitude: float = Field(ge=-90, le=90)
+    longitude: float = Field(ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] = Field(default_factory=dict)
+
+    @field_validator("layer", "kind")
+    @classmethod
+    def normalize_key(cls, value: str) -> str:
+        normalized = str(value or "").strip()
+        if not normalized:
+            raise ValueError("must not be empty")
+        return normalized
+
+
+class InteractableUpdate(BaseModel):
+    layer: str | None = Field(default=None, min_length=1, max_length=80)
+    kind: str | None = Field(default=None, min_length=1, max_length=80)
+    label: str | None = Field(default=None, max_length=255)
+    description: str | None = Field(default=None, max_length=4000)
+    latitude: float | None = Field(default=None, ge=-90, le=90)
+    longitude: float | None = Field(default=None, ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] | None = None
+
+
+@router.get("")
+async def get_interactables(
+    response: Response,
+    layer: str | None = Query(default=None),
+    include_deleted: bool = Query(default=False),
+    db: AsyncSession = Depends(get_db),
+):
+    items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
+    response.headers["X-Planet-Interactables-Count"] = str(len(items))
+    return {"items": [serialize_interactable(item) for item in items]}
+
+
+@router.get("/geojson")
+async def get_interactables_geojson(
+    response: Response,
+    layer: str | None = Query(default=None),
+    db: AsyncSession = Depends(get_db),
+):
+    async def build_payload() -> dict[str, Any]:
+        items = await list_interactables(db, layer=layer)
+        return interactables_to_geojson(items)
+
+    payload = await get_or_build_layer_payload(
+        key=earth_layer_cache.key("interactables", interactable_layer=layer or "all"),
+        policy=INTERACTABLE_CACHE_POLICY,
+        builder=build_payload,
+        response=response,
+    )
+    response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
+    return payload
+
+
+@router.post("", status_code=status.HTTP_201_CREATED)
+async def create_interactable(
+    payload: InteractableCreate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record_id = normalize_interactable_id(payload.id)
+    existing = await db.get(EarthInteractable, record_id)
+    if existing and not existing.is_deleted:
+        raise HTTPException(status_code=409, detail="Interactable already exists")
+
+    if existing is None:
+        record = EarthInteractable(id=record_id)
+        db.add(record)
+    else:
+        record = existing
+        record.is_deleted = False
+        record.deleted_at = None
+        record.revision += 1
+
+    record.layer = payload.layer
+    record.kind = payload.kind
+    record.label = payload.label
+    record.description = payload.description
+    record.latitude = payload.latitude
+    record.longitude = payload.longitude
+    record.altitude = payload.altitude
+    record.properties = payload.properties
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("created", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.get("/{interactable_id}")
+async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+    return {"item": serialize_interactable(record)}
+
+
+@router.patch("/{interactable_id}")
+async def update_interactable(
+    interactable_id: str,
+    payload: InteractableUpdate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    previous_layer = record.layer
+    patch = payload.model_dump(exclude_unset=True)
+    for key, value in patch.items():
+        setattr(record, key, value)
+    record.revision += 1
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(previous_layer)
+    if record.layer != previous_layer:
+        invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("updated", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.delete("/{interactable_id}")
+async def delete_interactable(
+    interactable_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    record.is_deleted = True
+    record.deleted_at = datetime.now(UTC)
+    record.revision += 1
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("deleted", record)
+    event = build_interactable_event(action="deleted", record=record, include_item=True)
+    return {"deleted": True, "event": event}

backend/app/api/v1/layers.py

@@ -0,0 +1,231 @@
+from typing import Any, Optional
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Response
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.v1.visualization import (
+    _parse_bbox,
+    get_bgp_anomalies_geojson,
+    get_bgp_collectors_geojson,
+    get_bgp_incidents_geojson,
+    get_cables_geojson,
+    get_landing_points_geojson,
+    get_satellites_geojson,
+)
+from app.api.v1.vessels import build_vessel_snapshot_response
+from app.db.session import get_db
+
+router = APIRouter()
+
+DEFAULT_LAYER_LIMIT = 1000
+MAX_LAYER_LIMIT = 5000
+LOW_ZOOM_FEATURE_LIMIT = 500
+
+
+def _clamp_limit(limit: int, zoom: int) -> tuple[int, bool]:
+    clamped = min(max(limit, 1), MAX_LAYER_LIMIT)
+    if zoom <= 3:
+        return min(clamped, LOW_ZOOM_FEATURE_LIMIT), clamped != limit or clamped > LOW_ZOOM_FEATURE_LIMIT
+    return clamped, clamped != limit
+
+
+def _coordinate_in_bbox(coord: Any, bbox: tuple[float, float, float, float]) -> bool:
+    if not isinstance(coord, (list, tuple)) or len(coord) < 2:
+        return False
+    try:
+        lon = float(coord[0])
+        lat = float(coord[1])
+    except (TypeError, ValueError):
+        return False
+    lon_min, lat_min, lon_max, lat_max = bbox
+    return lon_min <= lon <= lon_max and lat_min <= lat <= lat_max
+
+
+def _geometry_intersects_bbox(geometry: dict, bbox: tuple[float, float, float, float]) -> bool:
+    coordinates = geometry.get("coordinates")
+    geometry_type = geometry.get("type")
+    if geometry_type == "Point":
+        return _coordinate_in_bbox(coordinates, bbox)
+    if geometry_type in {"LineString", "MultiPoint"}:
+        return any(_coordinate_in_bbox(coord, bbox) for coord in coordinates or [])
+    if geometry_type in {"Polygon", "MultiLineString"}:
+        return any(
+            _coordinate_in_bbox(coord, bbox)
+            for line in coordinates or []
+            for coord in line
+        )
+    if geometry_type == "MultiPolygon":
+        return any(
+            _coordinate_in_bbox(coord, bbox)
+            for polygon in coordinates or []
+            for line in polygon
+            for coord in line
+        )
+    return False
+
+
+def _guard_geojson_layer(
+    geojson: dict,
+    *,
+    bbox: tuple[float, float, float, float],
+    zoom: int,
+    limit: int,
+) -> dict:
+    bounded_limit, limit_clamped = _clamp_limit(limit, zoom)
+    features = [
+        feature
+        for feature in geojson.get("features", [])
+        if _geometry_intersects_bbox(feature.get("geometry") or {}, bbox)
+    ]
+    visible_count = len(features)
+    returned_features = features[:bounded_limit]
+    return {
+        **geojson,
+        "features": returned_features,
+        "visible_count": visible_count,
+        "returned_count": len(returned_features),
+        "diagnostics": {
+            "bbox_limited": True,
+            "limit": bounded_limit,
+            "limit_clamped": limit_clamped,
+            "truncated": visible_count > len(returned_features),
+            "degraded": zoom <= 3 or visible_count > len(returned_features),
+            "stats_scope": "viewport",
+        },
+    }
+
+
+def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]:
+    parsed = _parse_bbox(bbox)
+    if parsed is None:
+        raise HTTPException(status_code=400, detail="bbox is required")
+    return parsed
+
+
+@router.get("/vessels/snapshot")
+async def get_vessel_layer_snapshot(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: float = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    vessel_type: Optional[str] = Query(None, alias="type"),
+    since_minutes: int = Query(60, ge=1, le=1440),
+    db: AsyncSession = Depends(get_db),
+    response: Response = None,
+):
+    parsed_bbox = _parse_layer_bbox(bbox)
+    return await build_vessel_snapshot_response(
+        db,
+        bbox=parsed_bbox,
+        zoom=zoom,
+        limit=limit,
+        type_filter=vessel_type,
+        since_minutes=since_minutes,
+        response=response,
+    )
+
+
+@router.get("/cables")
+async def get_cable_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    db: AsyncSession = Depends(get_db),
+):
+    return _guard_geojson_layer(
+        await get_cables_geojson(db),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )
+
+
+@router.get("/landing-points")
+async def get_landing_point_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    db: AsyncSession = Depends(get_db),
+):
+    return _guard_geojson_layer(
+        await get_landing_points_geojson(db),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )
+
+
+@router.get("/satellites")
+async def get_satellite_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    db: AsyncSession = Depends(get_db),
+):
+    bounded_limit, _ = _clamp_limit(limit, zoom)
+    return _guard_geojson_layer(
+        await get_satellites_geojson(limit=bounded_limit, db=db),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )
+
+
+@router.get("/bgp/anomalies")
+async def get_bgp_anomaly_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    severity: Optional[str] = Query(None),
+    status: Optional[str] = Query("active"),
+    db: AsyncSession = Depends(get_db),
+):
+    bounded_limit, _ = _clamp_limit(limit, zoom)
+    return _guard_geojson_layer(
+        await get_bgp_anomalies_geojson(
+            severity=severity,
+            status=status,
+            limit=bounded_limit,
+            db=db,
+        ),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )
+
+
+@router.get("/bgp/incidents")
+async def get_bgp_incident_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    severity: Optional[str] = Query(None),
+    status: Optional[str] = Query("active"),
+    db: AsyncSession = Depends(get_db),
+):
+    bounded_limit, _ = _clamp_limit(limit, zoom)
+    return _guard_geojson_layer(
+        await get_bgp_incidents_geojson(
+            severity=severity,
+            status=status,
+            limit=min(bounded_limit, 500),
+            db=db,
+        ),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )
+
+
+@router.get("/bgp/collectors")
+async def get_bgp_collector_layer(
+    bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
+    zoom: int = Query(..., ge=1, le=20),
+    limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
+    db: AsyncSession = Depends(get_db),
+):
+    return _guard_geojson_layer(
+        await get_bgp_collectors_geojson(db),
+        bbox=_parse_layer_bbox(bbox),
+        zoom=zoom,
+        limit=limit,
+    )

backend/app/api/v1/news.py

@@ -1,13 +1,92 @@
-from fastapi import APIRouter, Query
+from fastapi import APIRouter, Depends, HTTPException, Query
+from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.services.earth_news import get_earth_news_payload
+from app.db.session import get_db
+from app.services.earth_news import (
+    ALLOWED_NEWS_CATEGORY_KEYS,
+    SUPPORTED_NEWS_LOCALES,
+    REGION_ANCHORS,
+    get_earth_news_payload,
+)
 
 router = APIRouter()
 
 
+def _parse_categories(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    requested = {item.strip().lower() for item in raw.split(",") if item.strip()}
+    invalid = sorted(requested - set(ALLOWED_NEWS_CATEGORY_KEYS))
+    if invalid:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news categories.",
+                "invalid_categories": invalid,
+                "allowed_categories": list(ALLOWED_NEWS_CATEGORY_KEYS),
+            },
+        )
+    return requested or None
+
+
+def _parse_source_ids(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    return {item.strip() for item in raw.split(",") if item.strip()} or None
+
+
+def _parse_limit(raw: int | None) -> int:
+    if raw is None:
+        return 12
+    if raw < 1:
+        raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
+    return min(raw, 100)
+
+
+def _parse_locale(raw: str | None) -> str:
+    if raw is None or not raw.strip():
+        return "zh-CN"
+    requested = raw.strip()
+    if requested not in SUPPORTED_NEWS_LOCALES:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news locale.",
+                "invalid_locale": requested,
+                "allowed_locales": sorted(SUPPORTED_NEWS_LOCALES),
+            },
+        )
+    return requested
+
+
 @router.get("/earth-feed")
 async def get_earth_feed(
     lat: float | None = Query(None, description="Current Earth view center latitude"),
     lon: float | None = Query(None, description="Current Earth view center longitude"),
+    region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
+    categories: str | None = Query(None, description="Comma-separated news category keys"),
+    sources: str | None = Query(None, description="Comma-separated news source ids"),
+    limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
+    locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
+    db: AsyncSession = Depends(get_db),
 ):
-    return await get_earth_news_payload(lat=lat, lon=lon)
+    normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
+    if normalized_region is not None and normalized_region not in REGION_ANCHORS:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news region.",
+                "invalid_region": normalized_region,
+                "allowed_regions": list(REGION_ANCHORS.keys()),
+            },
+        )
+    return await get_earth_news_payload(
+        lat=lat,
+        lon=lon,
+        region=normalized_region,
+        categories=_parse_categories(categories),
+        source_ids=_parse_source_ids(sources),
+        limit=_parse_limit(limit),
+        locale=_parse_locale(locale),
+        db=db,
+    )

backend/app/api/v1/realtime_sources.py

@@ -0,0 +1,280 @@
+"""Realtime datasource operations and runtime statistics."""
+
+from datetime import UTC, datetime, timedelta
+import os
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import distinct, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.data_sources import get_data_sources_config
+from app.core.security import get_current_user
+from app.core.time import to_iso8601_utc
+from app.db.session import get_db
+from app.models.datasource import DataSource
+from app.models.datasource_config import DataSourceConfig
+from app.models.user import User
+from app.models.vessel import AISRawObservation, AISSourceHealth
+from app.services.custom_datasource_runtime import (
+    get_custom_stream_status,
+    start_custom_stream,
+    stop_custom_stream,
+)
+from app.services.scheduler import (
+    cancel_running_collector_now,
+    is_collector_running,
+    run_collector_now,
+)
+from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA, update_ais_source_health
+
+router = APIRouter()
+
+BUILTIN_REALTIME_SOURCES = {"aisstream_vessels"}
+REALTIME_SOURCE_TYPES = {"websocket", "ws"}
+
+
+def _is_realtime_config(config: DataSourceConfig) -> bool:
+    return str(config.source_type or "").lower() in REALTIME_SOURCE_TYPES
+
+
+def _safe_config_dict(value: Any) -> dict[str, Any]:
+    return value if isinstance(value, dict) else {}
+
+
+def _credential_configured(config: DataSourceConfig | None) -> bool:
+    if config is not None:
+        auth_config = _safe_config_dict(config.auth_config)
+        config_payload = _safe_config_dict(config.config)
+        if auth_config.get("api_key") or config_payload.get("api_key"):
+            return True
+    return bool(os.getenv("AISSTREAM_API_KEY"))
+
+
+async def _load_realtime_stats(db: AsyncSession, source: str) -> dict[str, Any]:
+    now = datetime.now(UTC)
+    observed_24h = now - timedelta(hours=24)
+    observed_1h = now - timedelta(hours=1)
+    payload_mmsi = AISRawObservation.entity_key
+
+    result = await db.execute(
+        select(
+            func.count(AISRawObservation.id).label("total_observations"),
+            func.count(AISRawObservation.id)
+            .filter(AISRawObservation.observed_at >= observed_24h)
+            .label("observations_24h"),
+            func.count(AISRawObservation.id)
+            .filter(AISRawObservation.observed_at >= observed_1h)
+            .label("observations_1h"),
+            func.count(distinct(payload_mmsi)).label("unique_mmsi_total"),
+            func.count(distinct(payload_mmsi))
+            .filter(AISRawObservation.observed_at >= observed_24h)
+            .label("unique_mmsi_24h"),
+            func.max(AISRawObservation.observed_at).label("latest_observed_at"),
+            func.max(AISRawObservation.collected_at).label("latest_collected_at"),
+        )
+        .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
+        .where(AISRawObservation.source == source)
+    )
+    row = result.mappings().one()
+    return {
+        "total_observations": int(row["total_observations"] or 0),
+        "observations_24h": int(row["observations_24h"] or 0),
+        "observations_1h": int(row["observations_1h"] or 0),
+        "unique_mmsi_total": int(row["unique_mmsi_total"] or 0),
+        "unique_mmsi_24h": int(row["unique_mmsi_24h"] or 0),
+        "latest_observed_at": to_iso8601_utc(row["latest_observed_at"]),
+        "latest_collected_at": to_iso8601_utc(row["latest_collected_at"]),
+    }
+
+
+def _runtime_status_for_builtin(source: str) -> dict[str, Any]:
+    running = is_collector_running(source)
+    return {
+        "running": running,
+        "done": False,
+        "runtime": "collector",
+    }
+
+
+def _runtime_status_for_custom(config_id: int) -> dict[str, Any]:
+    status = get_custom_stream_status(config_id)
+    return {
+        "running": bool(status.get("running")),
+        "done": bool(status.get("done")),
+        "runtime": "custom_stream",
+    }
+
+
+async def _serialize_builtin_aisstream(
+    db: AsyncSession,
+    datasource: DataSource,
+    config: DataSourceConfig | None,
+) -> dict[str, Any]:
+    health = await db.get(AISSourceHealth, datasource.source)
+    config_payload = _safe_config_dict(config.config if config else {})
+    endpoint = (
+        (config.endpoint if config else None)
+        or get_data_sources_config().get_yaml_url(datasource.source)
+    )
+    return {
+        "source": datasource.source,
+        "name": datasource.name,
+        "display_name": "AISStream 实时船舶",
+        "kind": "builtin",
+        "source_type": "websocket",
+        "endpoint": endpoint,
+        "is_active": bool(datasource.is_active),
+        "credential_configured": _credential_configured(config),
+        "message_types": config_payload.get("message_types") or ["PositionReport", "ShipStaticData"],
+        "bounding_boxes": config_payload.get("bounding_boxes") or [[[-90, -180], [90, 180]]],
+        "config": config_payload,
+        "runtime": _runtime_status_for_builtin(datasource.source),
+        "health": health.to_dict() if health else None,
+        "stats": await _load_realtime_stats(db, datasource.source),
+    }
+
+
+async def _serialize_custom_stream(
+    db: AsyncSession,
+    config: DataSourceConfig,
+) -> dict[str, Any]:
+    health = await db.get(AISSourceHealth, config.name)
+    config_payload = _safe_config_dict(config.config)
+    return {
+        "source": config.name,
+        "name": config.name,
+        "display_name": config.description or config.name,
+        "kind": "custom",
+        "config_id": config.id,
+        "source_type": config.source_type,
+        "endpoint": config.endpoint,
+        "is_active": bool(config.is_active),
+        "credential_configured": config.auth_type == "none" or bool(_safe_config_dict(config.auth_config)),
+        "message_types": config_payload.get("message_types") or [],
+        "bounding_boxes": config_payload.get("bounding_boxes") or [],
+        "config": config_payload,
+        "runtime": _runtime_status_for_custom(config.id),
+        "health": health.to_dict() if health else None,
+        "stats": await _load_realtime_stats(db, config.name),
+    }
+
+
+async def _load_builtin_aisstream(db: AsyncSession) -> tuple[DataSource | None, DataSourceConfig | None]:
+    result = await db.execute(select(DataSource).where(DataSource.source == "aisstream_vessels"))
+    datasource = result.scalar_one_or_none()
+    config_result = await db.execute(
+        select(DataSourceConfig)
+        .where(DataSourceConfig.name == "aisstream_vessels")
+        .where(DataSourceConfig.is_active.is_(True))
+        .order_by(DataSourceConfig.id.desc())
+        .limit(1)
+    )
+    return datasource, config_result.scalar_one_or_none()
+
+
+async def _load_custom_realtime_config(db: AsyncSession, source: str) -> DataSourceConfig | None:
+    result = await db.execute(
+        select(DataSourceConfig)
+        .where(DataSourceConfig.name == source)
+        .order_by(DataSourceConfig.id.desc())
+        .limit(1)
+    )
+    config = result.scalar_one_or_none()
+    return config if config is not None and _is_realtime_config(config) else None
+
+
+@router.get("")
+async def list_realtime_sources(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    sources: list[dict[str, Any]] = []
+    datasource, builtin_config = await _load_builtin_aisstream(db)
+    if datasource is not None:
+        sources.append(await _serialize_builtin_aisstream(db, datasource, builtin_config))
+
+    custom_result = await db.execute(
+        select(DataSourceConfig)
+        .where(func.lower(DataSourceConfig.source_type).in_(REALTIME_SOURCE_TYPES))
+        .order_by(DataSourceConfig.name)
+    )
+    for config in custom_result.scalars().all():
+        if config.name in BUILTIN_REALTIME_SOURCES:
+            continue
+        sources.append(await _serialize_custom_stream(db, config))
+
+    return {"total": len(sources), "data": sources}
+
+
+async def _ensure_builtin_startable(db: AsyncSession) -> DataSourceConfig | None:
+    datasource, config = await _load_builtin_aisstream(db)
+    if datasource is None:
+        raise HTTPException(status_code=404, detail="Realtime source not found")
+    if not datasource.is_active:
+        raise HTTPException(status_code=400, detail="Realtime source is disabled")
+    if not _credential_configured(config):
+        raise HTTPException(status_code=400, detail="AISStream API key is not configured")
+    return config
+
+
+@router.post("/{source}/start")
+async def start_realtime_source(
+    source: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    if source == "aisstream_vessels":
+        await _ensure_builtin_startable(db)
+        if is_collector_running(source):
+            return {"status": "already_running", "source": source, "runtime": _runtime_status_for_builtin(source)}
+        if not run_collector_now(source):
+            raise HTTPException(status_code=409, detail="Realtime source could not be started")
+        return {"status": "started", "source": source, "runtime": _runtime_status_for_builtin(source)}
+
+    config = await _load_custom_realtime_config(db, source)
+    if config is None:
+        raise HTTPException(status_code=404, detail="Realtime source not found")
+    if not config.is_active:
+        raise HTTPException(status_code=400, detail="Realtime source is disabled")
+    started = start_custom_stream(config.id)
+    return {
+        "status": "started" if started else "already_running",
+        "source": source,
+        "runtime": _runtime_status_for_custom(config.id),
+    }
+
+
+@router.post("/{source}/stop")
+async def stop_realtime_source(
+    source: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    if source == "aisstream_vessels":
+        stopped = await cancel_running_collector_now(source)
+        await update_ais_source_health(db, source=source, connection_state="disconnected", last_error=None)
+        await db.commit()
+        return {"status": "stopped" if stopped else "not_running", "source": source, "runtime": _runtime_status_for_builtin(source)}
+
+    config = await _load_custom_realtime_config(db, source)
+    if config is None:
+        raise HTTPException(status_code=404, detail="Realtime source not found")
+    stopped = await stop_custom_stream(config.id)
+    await update_ais_source_health(db, source=source, connection_state="disconnected", last_error=None)
+    await db.commit()
+    return {
+        "status": "stopped" if stopped else "not_running",
+        "source": source,
+        "runtime": _runtime_status_for_custom(config.id),
+    }
+
+
+@router.post("/{source}/restart")
+async def restart_realtime_source(
+    source: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    await stop_realtime_source(source, current_user=current_user, db=db)
+    return await start_realtime_source(source, current_user=current_user, db=db)

backend/app/api/v1/system_control.py

@@ -1,16 +1,19 @@
 from __future__ import annotations
 
 import os
+import secrets
 import subprocess
 import sys
 
 from datetime import datetime
 
-from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status
 from pydantic import BaseModel
+from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.core.config import ROOT_DIR
+from app.core.config import ROOT_DIR, settings
 from app.core.security import get_current_user
+from app.db.session import get_db
 from app.models.user import User
 from app.services.persistent_logs import record_audit_log, record_system_log
 from app.services.system_control import (
@@ -33,8 +36,13 @@ from app.services.system_logs import (
     append_buffer_log,
     list_log_sources,
     normalize_log_level,
+    read_database_log_snapshot,
     read_log_snapshot,
+    read_observability_group_events,
+    read_observability_groups,
+    read_observability_raw_events,
 )
+from app.services.earth_layer_cache import earth_layer_cache
 
 router = APIRouter()
 
@@ -104,12 +112,115 @@ class EarthClientLogEventCreate(BaseModel):
     url: str | None = None
     module: str | None = None
     detail: str | None = None
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    metadata: dict[str, object] | None = None
 
 
 class EarthClientLogEventResponse(BaseModel):
     accepted: bool
     source_id: str
     level: str
+    fingerprint: str | None = None
+
+
+class ServiceLogEventCreate(BaseModel):
+    source: str = "ai-provider"
+    service: str = "ai-provider"
+    module: str | None = None
+    category: str | None = None
+    event: str = "service.runtime_log"
+    level: str = "error"
+    message: str
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    request_id: str | None = None
+    trace_id: str | None = None
+    task_id: str | None = None
+    source_id: int | str | None = None
+    provider: str | None = None
+    context: dict[str, object] | None = None
+
+
+async def ingest_client_log_event(
+    source_id: str,
+    *,
+    service: str,
+    event: str,
+    default_module: str,
+    default_category: str,
+    payload: EarthClientLogEventCreate,
+    request: Request,
+) -> EarthClientLogEventResponse:
+    normalized_level = normalize_log_level(payload.level)
+    append_buffer_log(
+        source_id,
+        level=normalized_level,
+        message=payload.message,
+        context={
+            "category": payload.category or "",
+            "url": payload.url or "",
+            "module": payload.module or "",
+            "detail": payload.detail or "",
+            "fingerprint": payload.fingerprint or "",
+            "occurrence_count": max(1, int(payload.occurrence_count or 1)),
+            "metadata": payload.metadata or {},
+        },
+    )
+    await record_system_log(
+        source=source_id,
+        service=service,
+        module=payload.module or default_module,
+        event=event,
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or default_category,
+        context={
+            "url": payload.url or "",
+            "detail": payload.detail or "",
+            "module": payload.module or "",
+            "client_ip": request.client.host if request.client else "",
+            "metadata": payload.metadata or {},
+        },
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(accepted=True, source_id=source_id, level=normalized_level, fingerprint=payload.fingerprint)
+
+
+def require_observability_ingest_token(
+    authorization: str | None,
+    ingest_token: str | None,
+) -> None:
+    expected_token = settings.OBSERVABILITY_INGEST_TOKEN.strip()
+    if not expected_token:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail="Observability service ingestion is not configured",
+        )
+    provided = ""
+    if ingest_token:
+        provided = ingest_token.strip()
+    elif authorization:
+        scheme, _, token = authorization.partition(" ")
+        if scheme.lower() == "bearer":
+            provided = token.strip()
+    if not provided or not secrets.compare_digest(provided, expected_token):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Invalid observability ingestion token",
+        )
+
+
+class EarthLayerCacheStatusResponse(BaseModel):
+    prefix: str
+    key_count: int
+    memory_bytes: int
+    layers: dict[str, dict[str, int]]
+
+
+class EarthLayerCacheClearResponse(BaseModel):
+    deleted: int
 
 
 def ensure_super_admin(current_user: User) -> None:
@@ -132,6 +243,34 @@ def validate_log_date(raw_value: str | None, field_name: str) -> str | None:
         ) from exc
 
 
+@router.get("/cache/earth-layers", response_model=EarthLayerCacheStatusResponse)
+async def get_earth_layer_cache_status(
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+    try:
+        return earth_layer_cache.status()
+    except Exception as exc:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail=f"Unable to read Earth layer cache status: {exc}",
+        ) from exc
+
+
+@router.delete("/cache/earth-layers", response_model=EarthLayerCacheClearResponse)
+async def clear_earth_layer_cache(
+    current_user: User = Depends(get_current_user),
+):
+    ensure_super_admin(current_user)
+    try:
+        return {"deleted": earth_layer_cache.delete_pattern()}
+    except Exception as exc:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail=f"Unable to clear Earth layer cache: {exc}",
+        ) from exc
+
+
 @router.post("/restart-tasks", response_model=RestartTaskResponse)
 async def create_restart_task(
     payload: RestartTaskCreate,
@@ -270,7 +409,141 @@ async def get_system_log_sources(
     current_user: User = Depends(get_current_user),
 ):
     ensure_super_admin(current_user)
-    return {"items": list_log_sources()}
+    return {
+        "items": [
+            *list_log_sources(),
+            {
+                "source_id": "system-db",
+                "name": "系统事件",
+                "kind": "database",
+                "location": "table://system_logs",
+                "description": "后端持久化系统事件、AI 和采集器操作日志。",
+                "category": "database",
+                "status": "ok",
+            },
+            {
+                "source_id": "audit-db",
+                "name": "审计事件",
+                "kind": "database",
+                "location": "table://audit_logs",
+                "description": "管理员敏感操作和密钥 reveal 审计记录。",
+                "category": "audit",
+                "status": "ok",
+            },
+        ]
+    }
+
+
+@router.get("/logs/observability/groups")
+async def get_observability_log_groups(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
+    return await read_observability_groups(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
+
+
+@router.get("/logs/observability/groups/{fingerprint}/events")
+async def get_observability_group_events(
+    fingerprint: str,
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    payload = await read_observability_group_events(fingerprint, limit=limit, db=db)
+    if payload is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Observability group not found")
+    return payload
+
+
+@router.get("/logs/observability/raw")
+async def get_observability_raw_events(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    return await read_observability_raw_events(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
+
+
+@router.post("/logs/service", response_model=EarthClientLogEventResponse)
+async def ingest_service_log(
+    payload: ServiceLogEventCreate,
+    authorization: str | None = Header(default=None),
+    ingest_token: str | None = Header(default=None, alias="X-Planet-Observability-Token"),
+):
+    require_observability_ingest_token(authorization, ingest_token)
+    normalized_level = normalize_log_level(payload.level)
+    source = (payload.source or "ai-provider").strip() or "ai-provider"
+    context = dict(payload.context or {})
+    if payload.request_id:
+        context["request_id"] = payload.request_id
+    if payload.trace_id:
+        context["trace_id"] = payload.trace_id
+    if payload.task_id:
+        context["task_id"] = payload.task_id
+    if payload.source_id is not None:
+        context["source_id"] = payload.source_id
+    if payload.provider:
+        context["provider"] = payload.provider
+    await record_system_log(
+        source=source,
+        service=(payload.service or source).strip() or source,
+        module=payload.module or source,
+        event=(payload.event or "service.runtime_log").strip() or "service.runtime_log",
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or "service-runtime",
+        context=context,
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(
+        accepted=True,
+        source_id=source,
+        level=normalized_level,
+        fingerprint=payload.fingerprint,
+    )
 
 
 @router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
@@ -283,6 +556,7 @@ async def get_system_log_snapshot(
     end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
     search: str | None = Query(None, description="Case-insensitive substring search"),
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
     ensure_super_admin(current_user)
 
@@ -305,15 +579,26 @@ async def get_system_log_snapshot(
     if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
         raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
 
-    snapshot = read_log_snapshot(
+    snapshot = await read_database_log_snapshot(
         source_id,
-        limit,
+        limit=limit,
         level=level,
         levels=levels,
         start_date=normalized_start_date,
         end_date=normalized_end_date,
         search=search,
+        db=db,
     )
+    if snapshot is None:
+        snapshot = read_log_snapshot(
+            source_id,
+            limit,
+            level=level,
+            levels=levels,
+            start_date=normalized_start_date,
+            end_date=normalized_end_date,
+            search=search,
+        )
     if snapshot is None:
         raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Log source not found")
     return snapshot
@@ -324,31 +609,28 @@ async def ingest_earth_client_log(
     payload: EarthClientLogEventCreate,
     request: Request,
 ):
-    normalized_level = normalize_log_level(payload.level)
+    return await ingest_client_log_event(
-    append_buffer_log(
         "earth-client",
-        level=normalized_level,
-        message=payload.message,
-        context={
-            "category": payload.category or "",
-            "url": payload.url or "",
-            "module": payload.module or "",
-            "detail": payload.detail or "",
-        },
-    )
-    await record_system_log(
-        source="earth-client",
         service="earth",
-        module=payload.module or "earth-client",
         event="earth.client.runtime_log",
-        level=normalized_level,
+        default_module="earth-client",
-        message=payload.message,
+        default_category="client-runtime",
-        category=payload.category or "client-runtime",
+        payload=payload,
-        context={
+        request=request,
-            "url": payload.url or "",
+    )
-            "detail": payload.detail or "",
+
-            "module": payload.module or "",
+
-            "client_ip": request.client.host if request.client else "",
+@router.post("/logs/admin-client", response_model=EarthClientLogEventResponse)
-        },
+async def ingest_admin_client_log(
+    payload: EarthClientLogEventCreate,
+    request: Request,
+):
+    return await ingest_client_log_event(
+        "admin-client",
+        service="admin",
+        event="admin.client.runtime_log",
+        default_module="admin-client",
+        default_category="client-runtime",
+        payload=payload,
+        request=request,
     )
-    return {"accepted": True, "source_id": "earth-client", "level": normalized_level}

backend/app/api/v1/tasks.py

@@ -29,7 +29,8 @@ async def list_tasks(
         SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
                ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
                ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
-               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
+               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress,
+               ct.task_type, ct.source, ds.source as datasource_source
         FROM collection_tasks ct
         JOIN data_sources ds ON ct.datasource_id = ds.id
         WHERE 1=1
@@ -39,12 +40,19 @@ async def list_tasks(
 
     if datasource_id:
         query += " AND ct.datasource_id = :datasource_id"
-        count_query += " WHERE ct.datasource_id = :datasource_id"
+        count_query += " AND ct.datasource_id = :datasource_id"
         params["datasource_id"] = datasource_id
     if status:
-        query += " AND ct.status = :status"
+        statuses = [item.strip() for item in status.split(",") if item.strip()]
-        count_query += " AND ct.status = :status"
+        if len(statuses) > 1:
-        params["status"] = status
+            placeholders = ", ".join(f":status_{index}" for index, _item in enumerate(statuses))
+            query += f" AND ct.status IN ({placeholders})"
+            count_query += f" AND ct.status IN ({placeholders})"
+            params.update({f"status_{index}": item for index, item in enumerate(statuses)})
+        else:
+            query += " AND ct.status = :status"
+            count_query += " AND ct.status = :status"
+            params["status"] = statuses[0] if statuses else status
 
     query += f" ORDER BY ct.created_at DESC LIMIT {page_size} OFFSET {offset}"
 
@@ -76,6 +84,9 @@ async def list_tasks(
                 "phase_unit": t[13],
                 "total_records": t[14],
                 "progress": t[15],
+                "task_type": t[16],
+                "source": t[17] or t[18],
+                "datasource_source": t[18],
             }
             for t in tasks
         ],

backend/app/api/v1/tv.py

@@ -1,3 +1,4 @@
+import re
 from urllib.parse import quote, urljoin
 
 import httpx
@@ -6,16 +7,41 @@ from fastapi.responses import Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
+from app.services.tv_catalog import get_tv_catalog_page
 from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
 
 router = APIRouter()
 
+_HLS_URI_ATTRIBUTE_RE = re.compile(r'URI="([^"]+)"')
+
+
+def _proxied_tv_url(url: str) -> str:
+    return f"/api/v1/tv/proxy?url={quote(url, safe='')}"
+
+
+def _rewrite_hls_uri_attributes(line: str, *, base_url: str) -> str:
+    def replace(match: re.Match[str]) -> str:
+        uri = match.group(1)
+        absolute_url = urljoin(base_url, uri)
+        return f'URI="{_proxied_tv_url(absolute_url)}"'
+
+    return _HLS_URI_ATTRIBUTE_RE.sub(replace, line)
+
+
+def _should_strip_hls_metadata_line(line: str) -> bool:
+    normalized = line.strip().upper()
+    return normalized.startswith("#EXT-X-MEDIA:") and "TYPE=SUBTITLES" in normalized
+
 
 @router.get("/streams")
 async def list_public_tv_streams(
+    offset: int = Query(0, ge=0),
+    limit: int = Query(50, ge=1, le=100),
+    q: str = Query("", max_length=200),
+    selected_id: str | None = Query(None, max_length=200),
     db: AsyncSession = Depends(get_db),
 ):
-    return await get_public_tv_payload(db)
+    return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
 
 
 @router.get("/proxy")
@@ -56,11 +82,16 @@ async def proxy_tv_stream(
         rewritten_lines: list[str] = []
         for line in manifest_text.splitlines():
             stripped = line.strip()
-            if not stripped or stripped.startswith("#"):
+            if not stripped:
                 rewritten_lines.append(line)
                 continue
+            if stripped.startswith("#"):
+                if _should_strip_hls_metadata_line(line):
+                    continue
+                rewritten_lines.append(_rewrite_hls_uri_attributes(line, base_url=response_url))
+                continue
             absolute_url = urljoin(response_url, stripped)
-            rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
+            rewritten_lines.append(_proxied_tv_url(absolute_url))
         return Response(
             content="\n".join(rewritten_lines),
             media_type="application/vnd.apple.mpegurl",

backend/app/api/v1/users.py

@@ -1,16 +1,22 @@
+import json
 from typing import List
 
 from fastapi import APIRouter, Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
+from app.core.enums import UserRole
 from app.core.security import get_current_user, get_password_hash
 from app.db.session import get_db
 from app.models.user import User
-from app.schemas.user import UserCreate, UserResponse, UserUpdate
+from app.schemas.user import UserCreate, UserUpdate
 
 router = APIRouter()
 
+VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"}
+ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value]
+SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value]
+
 
 def check_permission(current_user: User, required_roles: List[str]) -> bool:
     user_role_value = (
@@ -29,7 +35,7 @@ async def list_users(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]):
+    if not check_permission(current_user, ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -52,7 +58,7 @@ async def list_users(
 
     offset = (page - 1) * page_size
     query = text(
-        f"SELECT id, username, email, role, is_active, last_login_at, created_at FROM users WHERE {where_sql} ORDER BY created_at DESC LIMIT {page_size} OFFSET {offset}"
+        f"SELECT id, username, email, role, is_active, last_login_at, created_at, gatekeeper_groups FROM users WHERE {where_sql} ORDER BY created_at DESC LIMIT {page_size} OFFSET {offset}"
     )
     count_query = text(f"SELECT COUNT(*) FROM users WHERE {where_sql}")
 
@@ -75,6 +81,7 @@ async def list_users(
                 "is_active": u[4],
                 "last_login_at": u[5],
                 "created_at": u[6],
+                "gatekeeper_groups": u[7] or [],
             }
             for u in users
         ],
@@ -87,7 +94,7 @@ async def get_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -95,7 +102,7 @@ async def get_user(
 
     result = await db.execute(
         text(
-            "SELECT id, username, email, role, is_active, last_login_at, created_at FROM users WHERE id = :id"
+            "SELECT id, username, email, role, is_active, last_login_at, created_at, gatekeeper_groups FROM users WHERE id = :id"
         ),
         {"id": user_id},
     )
@@ -114,6 +121,7 @@ async def get_user(
         "is_active": user[4],
         "last_login_at": user[5],
         "created_at": user[6],
+        "gatekeeper_groups": user[7] or [],
     }
 
 
@@ -123,11 +131,17 @@ async def create_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can create users",
         )
+    invalid_groups = sorted(set(user_data.gatekeeper_groups) - VALID_GATEKEEPER_GROUPS)
+    if invalid_groups:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Unsupported Gatekeeper groups: {', '.join(invalid_groups)}",
+        )
 
     result = await db.execute(
         text("SELECT id FROM users WHERE username = :username OR email = :email"),
@@ -142,13 +156,14 @@ async def create_user(
     hashed_password = get_password_hash(user_data.password)
 
     await db.execute(
-        text("""INSERT INTO users (username, email, password_hash, role, is_active, created_at, updated_at)
+        text("""INSERT INTO users (username, email, password_hash, role, gatekeeper_groups, is_active, created_at, updated_at)
-           VALUES (:username, :email, :password_hash, :role, :is_active, NOW(), NOW())"""),
+           VALUES (:username, :email, :password_hash, :role, CAST(:gatekeeper_groups AS jsonb), :is_active, NOW(), NOW())"""),
         {
             "username": user_data.username,
             "email": user_data.email,
             "password_hash": hashed_password,
             "role": user_data.role,
+            "gatekeeper_groups": json.dumps(user_data.gatekeeper_groups),
             "is_active": True,
         },
     )
@@ -172,6 +187,7 @@ async def create_user(
         "username": user_data.username,
         "email": user_data.email,
         "role": user_data.role,
+        "gatekeeper_groups": user_data.gatekeeper_groups,
         "is_active": True,
     }
 
@@ -183,17 +199,29 @@ async def update_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
         )
 
-    if not check_permission(current_user, ["super_admin"]) and user_data.role is not None:
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can change user role",
         )
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None:
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Only super_admin can change Gatekeeper groups",
+        )
+    if user_data.gatekeeper_groups is not None:
+        invalid_groups = sorted(set(user_data.gatekeeper_groups) - VALID_GATEKEEPER_GROUPS)
+        if invalid_groups:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=f"Unsupported Gatekeeper groups: {', '.join(invalid_groups)}",
+            )
 
     result = await db.execute(
         text("SELECT id FROM users WHERE id = :id"),
@@ -213,6 +241,9 @@ async def update_user(
     if user_data.role is not None:
         update_fields.append("role = :role")
         params["role"] = user_data.role
+    if user_data.gatekeeper_groups is not None:
+        update_fields.append("gatekeeper_groups = CAST(:gatekeeper_groups AS jsonb)")
+        params["gatekeeper_groups"] = json.dumps(user_data.gatekeeper_groups)
     if user_data.is_active is not None:
         update_fields.append("is_active = :is_active")
         params["is_active"] = user_data.is_active
@@ -232,7 +263,7 @@ async def delete_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can delete users",

backend/app/api/v1/vessel_aggregation.py

@@ -0,0 +1,132 @@
+"""v4 strategy + v5 conflict-promotion + enrichment APIs for vessel_ais."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.user import User
+from app.models.vessel import AISConflictRecord
+from app.services.vessel_aggregation_strategy import (
+    StrategyValidationError,
+    load_strategy,
+    reset_strategy,
+    save_strategy,
+)
+from app.services.vessel_enrichment import (
+    get_vessel_enrichment_bundle,
+    upsert_vessel_media_enrichment,
+    upsert_vessel_profile_enrichment,
+)
+
+router = APIRouter()
+
+
+@router.get("/strategy")
+async def get_aggregation_strategy(db: AsyncSession = Depends(get_db)):
+    return await load_strategy(db)
+
+
+@router.put("/strategy")
+async def put_aggregation_strategy(
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        return await save_strategy(db, payload)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.delete("/strategy")
+async def reset_aggregation_strategy(
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await reset_strategy(db)
+
+
+@router.post("/conflicts/{mmsi}/{field}/promote-to-rule")
+async def promote_conflict_to_rule(
+    mmsi: int,
+    field: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    """Lift the current conflict resolution into a persistent strategy rule."""
+
+    result = await db.execute(
+        select(AISConflictRecord)
+        .where(AISConflictRecord.target_schema == "vessel_ais")
+        .where(AISConflictRecord.entity_key == str(mmsi))
+        .where(AISConflictRecord.field == field)
+        .order_by(AISConflictRecord.updated_at.desc(), AISConflictRecord.id.desc())
+        .limit(1)
+    )
+    record = result.scalar_one_or_none()
+    if record is None or not record.selected_source:
+        raise HTTPException(status_code=404, detail="Conflict record with selected_source not found")
+
+    strategy = await load_strategy(db)
+    vessel_ais = dict(strategy.get("vessel_ais") or {})
+    field_rules = dict(vessel_ais.get("field_rules") or {})
+    field_rules[field] = {"mode": "source_priority", "source_priority": [record.selected_source]}
+    vessel_ais["field_rules"] = field_rules
+
+    incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
+    try:
+        return await save_strategy(db, incoming)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.delete("/conflicts/{mmsi}/{field}/promote-to-rule")
+async def revert_conflict_rule(
+    mmsi: int,
+    field: str,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    strategy = await load_strategy(db)
+    vessel_ais = dict(strategy.get("vessel_ais") or {})
+    field_rules = dict(vessel_ais.get("field_rules") or {})
+    if field in field_rules:
+        del field_rules[field]
+    vessel_ais["field_rules"] = field_rules
+
+    incoming = {"version": int(strategy.get("version") or 0), "vessel_ais": vessel_ais}
+    try:
+        return await save_strategy(db, incoming)
+    except StrategyValidationError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+
+@router.get("/enrichment/{mmsi}")
+async def get_vessel_enrichment(mmsi: int, db: AsyncSession = Depends(get_db)):
+    return await get_vessel_enrichment_bundle(db, mmsi)
+
+
+@router.put("/enrichment/{mmsi}/profile")
+async def put_vessel_profile_enrichment(
+    mmsi: int,
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await upsert_vessel_profile_enrichment(db, mmsi=mmsi, payload=payload)
+
+
+@router.put("/enrichment/{mmsi}/media")
+async def put_vessel_media_enrichment(
+    mmsi: int,
+    payload: dict[str, Any],
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await upsert_vessel_media_enrichment(db, mmsi=mmsi, payload=payload)

backend/app/api/v1/vessels.py

@@ -0,0 +1,41 @@
+"""Bounded vessel snapshot APIs backed by the latest vessel state table."""
+
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Response
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.v1.visualization import _parse_bbox, build_vessel_snapshot_response
+from app.db.session import get_db
+from app.services.vessel_ais_aggregation import MAX_SNAPSHOT_LIMIT
+
+router = APIRouter()
+
+
+@router.get("/snapshot")
+async def get_vessel_snapshot(
+    bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"),
+    zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"),
+    type: Optional[str] = Query(
+        None,
+        description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
+    ),
+    limit: int = Query(1000, ge=1, le=MAX_SNAPSHOT_LIMIT),
+    since_minutes: int = Query(60, ge=1, le=1440),
+    db: AsyncSession = Depends(get_db),
+    response: Response = None,
+):
+    if not bbox:
+        raise HTTPException(status_code=400, detail="bbox is required")
+    parsed_bbox = _parse_bbox(bbox)
+    if parsed_bbox is None:
+        raise HTTPException(status_code=400, detail="bbox is required")
+    return await build_vessel_snapshot_response(
+        db,
+        bbox=parsed_bbox,
+        zoom=zoom,
+        type_filter=type,
+        limit=limit,
+        since_minutes=since_minutes,
+        response=response,
+    )

backend/app/api/v1/websocket.py

@@ -1,20 +1,24 @@
 """WebSocket API endpoints"""
 
 import asyncio
-import json
 from datetime import UTC, datetime
 from typing import Optional
 
 from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
 from jose import jwt, JWTError
+from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import UserRole
 from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
+from app.db.session import async_session_factory
+from app.services.log_tail import LOG_TAIL_CHANNEL, log_tail_manager
 
 logger = get_logger(__name__, service="api")
 router = APIRouter()
+EARTH_UPDATES_CHANNEL = "earth_updates"
 
 
 async def authenticate_token(token: str) -> Optional[dict]:
@@ -37,19 +41,41 @@ async def authenticate_token(token: str) -> Optional[dict]:
         return None
 
 
+async def load_websocket_user_role(user_id: str | None) -> str | None:
+    if not user_id:
+        return None
+    try:
+        async with async_session_factory() as db:
+            result = await db.execute(
+                text("SELECT role, is_active FROM users WHERE id = :id"),
+                {"id": int(user_id)},
+            )
+            row = result.fetchone()
+    except Exception as exc:
+        logger.warning_event(
+            "WebSocket user role lookup failed",
+            event="auth.websocket.role_lookup_failed",
+            context={"user_id": user_id, "error": str(exc)},
+        )
+        return None
+    if row is None or not row[1]:
+        return None
+    return str(row[0] or "")
+
+
 @router.websocket("/ws")
 async def websocket_endpoint(
     websocket: WebSocket,
-    token: str = Query(...),
+    token: str | None = Query(None),
 ):
     """WebSocket endpoint for real-time data"""
     logger.info_event(
         "WebSocket connection attempt",
         event="auth.websocket.connection_attempt",
-        context={"token_preview": f"{token[:8]}..."},
+        context={"token_preview": f"{token[:8]}..." if token else "anonymous"},
     )
-    payload = await authenticate_token(token)
+    payload = await authenticate_token(token) if token else None
-    if payload is None:
+    if token and payload is None:
         logger.warning_event(
             "WebSocket authentication failed, closing connection",
             event="auth.websocket.connection_rejected",
@@ -57,7 +83,22 @@ async def websocket_endpoint(
         await websocket.close(code=4001)
         return
 
-    user_id = str(payload.get("sub"))
+    is_anonymous = payload is None
+    user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
+    user_role = await load_websocket_user_role(user_id) if payload else None
+    supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
+        "gpu_clusters",
+        "submarine_cables",
+        "ixp_nodes",
+        "alerts",
+        "dashboard",
+        "datasource_tasks",
+        "vessels",
+        "earth_news",
+        EARTH_UPDATES_CHANNEL,
+    ]
+    if user_role == UserRole.SUPER_ADMIN.value:
+        supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
     await manager.connect(websocket, user_id)
 
     try:
@@ -68,14 +109,7 @@ async def websocket_endpoint(
                     "connection_id": f"conn_{user_id}",
                     "server_version": settings.VERSION,
                     "heartbeat_interval": 30,
-                    "supported_channels": [
+                    "supported_channels": supported_channels,
-                        "gpu_clusters",
-                        "submarine_cables",
-                        "ixp_nodes",
-                        "alerts",
-                        "dashboard",
-                        "datasource_tasks",
-                    ],
                 },
             }
         )
@@ -92,11 +126,80 @@ async def websocket_endpoint(
                         }
                     )
                 elif data.get("type") == "subscribe":
-                    channels = data.get("data", {}).get("channels", [])
+                    payload_data = data.get("data", {})
+                    if not isinstance(payload_data, dict):
+                        payload_data = {}
+                    log_tail_config = None
+                    channels = payload_data.get("channels", [])
+                    if isinstance(channels, str):
+                        channels = [channels]
+                    elif not isinstance(channels, list):
+                        channels = []
+                    channel = payload_data.get("channel")
+                    if channel and channel not in channels:
+                        channels = [*channels, channel]
+                    if LOG_TAIL_CHANNEL in channels:
+                        if user_role != UserRole.SUPER_ADMIN.value:
+                            await websocket.send_json(
+                                {
+                                    "type": "subscription_error",
+                                    "data": {"channel": LOG_TAIL_CHANNEL, "detail": "Only super_admin can subscribe logs"},
+                                }
+                            )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
+                        else:
+                            try:
+                                log_tail_config = await log_tail_manager.subscribe(websocket, payload_data)
+                            except ValueError as exc:
+                                await websocket.send_json(
+                                    {
+                                        "type": "subscription_error",
+                                        "data": {"channel": LOG_TAIL_CHANNEL, "detail": str(exc)},
+                                    }
+                                )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
+                    if is_anonymous:
+                        channels = [channel for channel in channels if channel in supported_channels]
+                    vessel_subscription = None
+                    if "vessels" in channels:
+                        try:
+                            vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
+                        except ValueError as exc:
+                            await websocket.send_json(
+                                {
+                                    "type": "subscription_error",
+                                    "data": {"channel": "vessels", "detail": str(exc)},
+                                }
+                            )
+                            continue
+                        channels = [channel for channel in channels if channel != "vessels"]
+                    manager.subscribe(websocket, channels)
                     await websocket.send_json(
                         {
                             "type": "subscription_confirmed",
-                            "data": {"action": "subscribe", "channels": channels},
+                            "data": {
+                                "action": "subscribe",
+                                "channels": [
+                                    *channels,
+                                    *([LOG_TAIL_CHANNEL] if log_tail_config else []),
+                                    *(["vessels"] if vessel_subscription else []),
+                                ],
+                                "vessels": vessel_subscription,
+                                "logs_tail": log_tail_config.__dict__ if log_tail_config else None,
+                            },
+                        }
+                    )
+                elif data.get("type") == "unsubscribe":
+                    channels = data.get("data", {}).get("channels", [])
+                    if isinstance(channels, str):
+                        channels = [channels]
+                    if LOG_TAIL_CHANNEL in channels:
+                        await log_tail_manager.unsubscribe(websocket)
+                    manager.unsubscribe(websocket, channels)
+                    await websocket.send_json(
+                        {
+                            "type": "subscription_confirmed",
+                            "data": {"action": "unsubscribe", "channels": channels},
                         }
                     )
                 elif data.get("type") == "control_frame":
@@ -112,4 +215,5 @@ async def websocket_endpoint(
     except WebSocketDisconnect:
         pass
     finally:
+        await log_tail_manager.disconnect(websocket)
         manager.disconnect(websocket, user_id)

backend/app/core/collected_data_fields.py

@@ -4,8 +4,8 @@ from typing import Any, Dict, Optional
 FIELD_ALIASES = {
     "country": ("country",),
     "city": ("city",),
-    "latitude": ("latitude",),
+    "latitude": ("latitude", "lat"),
-    "longitude": ("longitude",),
+    "longitude": ("longitude", "lon", "lng"),
     "value": ("value",),
     "unit": ("unit",),
     "cores": ("cores",),
@@ -14,6 +14,28 @@ FIELD_ALIASES = {
     "power": ("power",),
 }
 
+NESTED_FIELD_ALIASES = {
+    "latitude": (
+        ("location", "latitude"),
+        ("location", "lat"),
+        ("geo", "latitude"),
+        ("geo", "lat"),
+        ("coordinates", "latitude"),
+        ("coordinates", "lat"),
+    ),
+    "longitude": (
+        ("location", "longitude"),
+        ("location", "lon"),
+        ("location", "lng"),
+        ("geo", "longitude"),
+        ("geo", "lon"),
+        ("geo", "lng"),
+        ("coordinates", "longitude"),
+        ("coordinates", "lon"),
+        ("coordinates", "lng"),
+    ),
+}
+
 
 def get_metadata_field(metadata: Optional[Dict[str, Any]], field: str, fallback: Any = None) -> Any:
     if isinstance(metadata, dict):
@@ -21,9 +43,34 @@ def get_metadata_field(metadata: Optional[Dict[str, Any]], field: str, fallback:
             value = metadata.get(key)
             if value not in (None, ""):
                 return value
+        for path in NESTED_FIELD_ALIASES.get(field, ()):
+            current: Any = metadata
+            for key in path:
+                if not isinstance(current, dict):
+                    current = None
+                    break
+                current = current.get(key)
+            if current not in (None, ""):
+                return current
+        if field in {"latitude", "longitude"}:
+            value = _get_coordinate_sequence_value(metadata, field)
+            if value not in (None, ""):
+                return value
     return fallback
 
 
+def _get_coordinate_sequence_value(metadata: Dict[str, Any], field: str) -> Any:
+    for key in ("coordinates", "coord", "coords"):
+        value = metadata.get(key)
+        if not isinstance(value, (list, tuple)) or len(value) < 2:
+            continue
+        # GeoJSON uses [longitude, latitude]. Most raw collector tuples in this
+        # codebase use explicit field names, so only sequence aliases are treated
+        # as GeoJSON-shaped to avoid guessing.
+        return value[1] if field == "latitude" else value[0]
+    return None
+
+
 def build_dynamic_metadata(
     metadata: Optional[Dict[str, Any]],
     *,

backend/app/core/config.py

@@ -41,6 +41,7 @@ class Settings(BaseSettings):
     AI_PROVIDER_SERVICE_TOKEN: str = ""
     AI_PROVIDER_TIMEOUT_SECONDS: int = 60
     AI_PROVIDER_RETRY_ATTEMPTS: int = 2
+    OBSERVABILITY_INGEST_TOKEN: str = ""
 
     @property
     def REDIS_URL(self) -> str:

backend/app/core/data_sources.py

@@ -1,7 +1,6 @@
 import os
 import yaml
 from functools import lru_cache
-from typing import Optional
 
 
 COLLECTOR_URL_KEYS = {
@@ -32,6 +31,7 @@ COLLECTOR_URL_KEYS = {
     "nro_delegated_prefix_geo": "nro.delegated_stats_url",
     "news_live_streams": "news_live_streams.channels_url",
     "barentswatch_vessels": "barentswatch_vessels.url",
+    "aisstream_vessels": "aisstream_vessels.url",
 }
 
 
@@ -74,7 +74,7 @@ class DataSourcesConfig:
             from app.models.datasource_config import DataSourceConfig
 
             query = select(DataSourceConfig).where(
-                DataSourceConfig.name == collector_name, DataSourceConfig.is_active == True
+                DataSourceConfig.name == collector_name, DataSourceConfig.is_active
             )
             result = await db.execute(query)
             db_config = result.scalar_one_or_none()

backend/app/core/data_sources.yaml

@@ -98,3 +98,7 @@ news_live_streams:
 barentswatch_vessels:
   # BarentsWatch Live AIS latest combined endpoint. Requires an AIS bearer token.
   url: "https://live.ais.barentswatch.no/v1/latest/combined"
+
+aisstream_vessels:
+  # AISStream realtime WebSocket endpoint. Requires an AISStream API key.
+  url: "wss://stream.aisstream.io/v0/stream"

backend/app/core/datasource_defaults.py

@@ -245,6 +245,28 @@ DEFAULT_DATASOURCES = {
         "credential_provider": "barentswatch",
         "credential_status": "supported",
     },
+    "aisstream_vessels": {
+        "id": 28,
+        "name": "AISStream Vessels",
+        "display_name": "AISStream 实时船舶",
+        "module": "L4",
+        "priority": "P1",
+        "frequency_minutes": 1,
+        "is_free": True,
+        "requires_credentials": True,
+        "credential_provider": "aisstream",
+        "credential_status": "supported",
+    },
+    "media_news_archive": {
+        "id": 33,
+        "name": "Media News Archive",
+        "display_name": "媒体新闻归档",
+        "module": "L4",
+        "priority": "P2",
+        "frequency_minutes": 720,
+        "is_free": True,
+        "requires_credentials": False,
+    },
 }
 
 ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}

backend/app/core/enums.py

@@ -0,0 +1,226 @@
+"""Stable backend protocol enums.
+
+Database columns and JSON payloads continue to store the enum string values.
+Configurable identifiers, user-authored values, and open-ended taxonomies do
+not belong in this module.
+"""
+
+from __future__ import annotations
+
+import logging
+from enum import StrEnum
+from typing import TypeVar
+
+logger = logging.getLogger(__name__)
+
+EnumT = TypeVar("EnumT", bound=StrEnum)
+
+
+def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT:
+    """Parse an external value without breaking reads of legacy data."""
+
+    if value is None or str(value).strip() == "":
+        return default
+    if isinstance(value, enum_type):
+        return value
+    try:
+        return enum_type(str(value).strip().lower())
+    except (TypeError, ValueError):
+        logger.warning(
+            "Unknown %s value %r; falling back to %s",
+            enum_type.__name__,
+            value,
+            default.value,
+        )
+        return default
+
+
+class NewsImportanceLevel(StrEnum):
+    LOW = "low"
+    MEDIUM = "medium"
+    HIGH = "high"
+    CRITICAL = "critical"
+
+
+class BreakingLevel(StrEnum):
+    NONE = "none"
+    WATCH = "watch"
+    BREAKING = "breaking"
+    CRITICAL = "critical"
+
+
+class BreakingScope(StrEnum):
+    REGIONAL = "regional"
+    GLOBAL = "global"
+
+
+class BreakingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+    MULTI_SOURCE = "multi_source"
+
+
+class NewsSourceType(StrEnum):
+    RSS = "rss"
+    ATOM = "atom"
+    AGGREGATED = "aggregated"
+    REFERENCE = "reference"
+    MANUAL = "manual"
+
+
+class NewsEnrichmentStatus(StrEnum):
+    PENDING = "pending"
+    QUEUED = "queued"
+    ATTEMPTED = "attempted"
+    SUCCESS = "success"
+    CONTENT_ONLY = "content_only"
+    LOCATION_ONLY = "location_only"
+    UNAVAILABLE = "unavailable"
+    PROVIDER_ERROR = "provider_error"
+    PARSE_ERROR = "parse_error"
+    NO_RESULT = "no_result"
+
+
+class NewsMarketImpact(StrEnum):
+    NONE = "none"
+    SECTOR = "sector"
+    NATIONAL = "national"
+    GLOBAL = "global"
+
+
+class NewsTaggingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+
+
+class JobType(StrEnum):
+    COLLECT = "collect"
+    CLEAR_DATA = "clear_data"
+    CLEAR_CACHE = "clear_cache"
+    EARTH_REFRESH = "earth_refresh"
+
+
+class JobStatus(StrEnum):
+    QUEUED = "queued"
+    RUNNING = "running"
+    CANCELLING = "cancelling"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class RollbackPolicy(StrEnum):
+    KEEP_COMMITTED_BATCHES = "keep_committed_batches"
+
+
+class MappingValidationStatus(StrEnum):
+    DRAFT = "draft"
+    VALID = "valid"
+    INVALID = "invalid"
+
+
+class SnapshotStatus(StrEnum):
+    RUNNING = "running"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class DatasourceRunStatus(StrEnum):
+    RUNNING = "running"
+    NOT_RUN = "not_run"
+    COLLECTED = "collected"
+    UNCOLLECTED = "uncollected"
+
+
+class ProviderApi(StrEnum):
+    ANTHROPIC_MESSAGES = "anthropic-messages"
+    OPENAI_COMPLETIONS = "openai-completions"
+    OLLAMA_GENERATE = "ollama-generate"
+
+
+class PlaygroundMessageRole(StrEnum):
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+    TOOL = "tool"
+
+
+class PlaygroundMessageKind(StrEnum):
+    MESSAGE = "message"
+    THINKING = "thinking"
+    ERROR = "error"
+    STATUS = "status"
+
+
+class PlaygroundMessageStatus(StrEnum):
+    PENDING = "pending"
+    THINKING = "thinking"
+    ANSWERING = "answering"
+    DONE = "done"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+    ERROR = "error"
+    STOPPED = "stopped"
+
+
+class OtpPurpose(StrEnum):
+    REGISTER = "register"
+    VERIFY_EMAIL = "verify_email"
+    RESET_PASSWORD = "reset_password"
+
+
+class UserRole(StrEnum):
+    VIEWER = "viewer"
+    ADMIN = "admin"
+    SUPER_ADMIN = "super_admin"
+
+
+class AlertSeverity(StrEnum):
+    CRITICAL = "critical"
+    WARNING = "warning"
+    INFO = "info"
+
+
+class AlertStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class BGPStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class LogLevel(StrEnum):
+    ALL = "all"
+    ERROR = "error"
+    WARNING = "warning"
+    INFO = "info"
+    DEBUG = "debug"
+
+
+class ConnectionState(StrEnum):
+    DISCONNECTED = "disconnected"
+    CONNECTING = "connecting"
+    CONNECTED = "connected"
+    ERROR = "error"
+
+
+class AuthType(StrEnum):
+    NONE = "none"
+    BEARER = "bearer"
+    API_KEY = "api_key"
+    BASIC = "basic"
+
+
+class TVSourceType(StrEnum):
+    IFRAME = "iframe"
+    HLS = "hls"
+    VIDEO = "video"
+    EXTERNAL = "external"
+    YOUTUBE = "youtube"

backend/app/core/security.py

@@ -105,7 +105,7 @@ async def get_current_user(
         )
     result = await db.execute(
         text(
-            "SELECT id, username, email, password_hash, role, is_active FROM users WHERE id = :id"
+            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
         ),
         {"id": int(user_id)},
     )
@@ -122,6 +122,7 @@ async def get_current_user(
     user.password_hash = row[3]
     user.role = row[4]
     user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
     return user
 
 
@@ -144,7 +145,7 @@ async def get_current_user_refresh(
         )
     result = await db.execute(
         text(
-            "SELECT id, username, email, password_hash, role, is_active FROM users WHERE id = :id"
+            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
         ),
         {"id": int(user_id)},
     )
@@ -161,6 +162,7 @@ async def get_current_user_refresh(
     user.password_hash = row[3]
     user.role = row[4]
     user.is_active = row[5]
+    user.gatekeeper_groups = row[6] or []
     return user
 
 

backend/app/core/target_schema_registry.py

@@ -16,8 +16,11 @@ class VesselAISRecord(BaseModel):
     sog: float | None = None
     cog: float | None = Field(default=None, ge=0, le=360)
     heading: int | None = Field(default=None, ge=0, le=511)
+    nav_status: int | None = None
     name: str | None = None
+    callsign: str | None = None
     vessel_type: str | int | None = None
+    vessel_type_name: str | None = None
     received_at: datetime | None = None
 
 
@@ -104,8 +107,11 @@ TARGET_SCHEMAS: dict[str, TargetSchema] = {
             TargetField("sog", "float", False, "对地航速，单位节", 12.4),
             TargetField("cog", "float", False, "对地航向，0-360 度", 184.5),
             TargetField("heading", "integer", False, "船首向，0-511", 186),
+            TargetField("nav_status", "integer", False, "导航状态码", 0),
             TargetField("name", "string", False, "船名", "OSLO EXPRESS"),
-            TargetField("vessel_type", "string", False, "船型", "cargo"),
+            TargetField("callsign", "string", False, "呼号", "LAAB"),
+            TargetField("vessel_type", "string", False, "船型代码", 70),
+            TargetField("vessel_type_name", "string", False, "船型名称", "Cargo"),
             TargetField("received_at", "datetime", False, "数据接收时间", "2026-04-28T00:00:00Z"),
         ),
     ),

backend/app/core/websocket/broadcaster.py

@@ -2,12 +2,17 @@
 
 import asyncio
 from datetime import UTC, datetime
-from typing import Dict, Any, Optional
+from typing import Dict, Any
 
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
 
+EARTH_UPDATES_CHANNEL = "earth_updates"
+VESSEL_STATE_QUERY_BATCH_SIZE = 1000
+logger = get_logger(__name__, service="websocket")
+
 
 class DataBroadcaster:
     """Periodically broadcasts data to connected WebSocket clients"""
@@ -15,6 +20,8 @@ class DataBroadcaster:
     def __init__(self):
         self.running = False
         self.tasks: Dict[str, asyncio.Task] = {}
+        self._pending_vessel_updates: Dict[str, Dict[str, Any]] = {}
+        self._vessel_flush_interval = 1.0
 
     async def get_dashboard_stats(self) -> Dict[str, Any]:
         """Get dashboard statistics"""
@@ -68,6 +75,9 @@ class DataBroadcaster:
 
     async def broadcast_custom(self, channel: str, data: Dict[str, Any]):
         """Broadcast custom data to a specific channel"""
+        if channel == "vessels":
+            self.enqueue_vessel_update(data)
+            return
         await manager.broadcast(
             {
                 "type": "data_frame",
@@ -75,9 +85,90 @@ class DataBroadcaster:
                 "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
-            channel=channel if channel in manager.active_connections else "all",
+            channel=channel,
         )
 
+    async def broadcast_earth_update(self, data: Dict[str, Any]):
+        """Broadcast Earth visualization refresh hints to connected clients."""
+        await self.broadcast_custom(EARTH_UPDATES_CHANNEL, data)
+
+    def enqueue_vessel_update(self, data: Dict[str, Any]):
+        vessels = data.get("vessels") if isinstance(data, dict) else None
+        if not isinstance(vessels, list):
+            return
+        source = data.get("source")
+        action = data.get("action") or "upsert"
+        created = data.get("created")
+        for vessel in vessels:
+            if not isinstance(vessel, dict):
+                continue
+            mmsi = vessel.get("mmsi")
+            if mmsi in (None, ""):
+                continue
+            self._pending_vessel_updates[str(mmsi)] = {
+                **vessel,
+                "_source": source,
+                "_action": action,
+                "_created": created,
+            }
+
+    async def flush_vessel_updates(self):
+        if not self._pending_vessel_updates:
+            return
+        pending = self._pending_vessel_updates
+        self._pending_vessel_updates = {}
+        try:
+            vessels = await self._load_current_vessel_updates(list(pending))
+        except Exception:
+            # Preserve newer updates that arrived during the failed database read.
+            self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
+            raise
+        if not vessels:
+            return
+        vessels = [
+            {
+                **vessel,
+                "action": vessel.get("action", "upsert"),
+                "created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
+            }
+            for vessel in vessels
+        ]
+        await manager.broadcast_vessels(
+            {
+                "action": "upsert",
+                "source": "mixed",
+                "created": None,
+                "vessels": vessels,
+            }
+        )
+
+    async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
+        from app.db.session import async_session_factory
+        from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
+
+        mmsis = [int(key) for key in keys if key.isdigit()]
+        vessels = []
+        async with async_session_factory() as db:
+            for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
+                vessels.extend(await get_current_vessels_by_mmsi(
+                    db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
+                ))
+        present = {int(vessel["mmsi"]) for vessel in vessels}
+        vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
+        return vessels
+
+    async def broadcast_vessels_periodically(self):
+        while self.running:
+            try:
+                await self.flush_vessel_updates()
+            except Exception as exc:
+                logger.exception_event(
+                    "Failed to flush vessel updates",
+                    event="vessels.broadcast.failed",
+                    context={"error": str(exc)},
+                )
+            await asyncio.sleep(self._vessel_flush_interval)
+
     async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
         """Broadcast datasource task progress updates to connected clients."""
         await manager.broadcast(
@@ -87,7 +178,7 @@ class DataBroadcaster:
                 "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
-            channel="all",
+            channel="datasource_tasks",
         )
 
     def start(self):
@@ -95,6 +186,7 @@ class DataBroadcaster:
         if not self.running:
             self.running = True
             self.tasks["dashboard"] = asyncio.create_task(self.broadcast_stats(5))
+            self.tasks["vessels"] = asyncio.create_task(self.broadcast_vessels_periodically())
 
     def stop(self):
         """Stop all broadcasters"""
@@ -102,6 +194,7 @@ class DataBroadcaster:
         for task in self.tasks.values():
             task.cancel()
         self.tasks.clear()
+        self._pending_vessel_updates.clear()
 
 
 broadcaster = DataBroadcaster()

backend/app/core/websocket/manager.py

@@ -1,20 +1,25 @@
 """WebSocket Connection Manager"""
 
-import json
+from datetime import UTC, datetime
-import asyncio
+from typing import Any, Dict, Set, Optional
-from typing import Dict, Set, Optional
-from datetime import datetime
 from fastapi import WebSocket
 import redis.asyncio as redis
 
 from app.core.config import settings
 
+MAX_VESSEL_SUBSCRIPTION_LIMIT = 5000
+MAX_VESSEL_WS_MESSAGE_ITEMS = 1000
+MAX_VESSEL_BBOX_AREA = 2500.0
+
 
 class ConnectionManager:
     """Manages WebSocket connections"""
 
     def __init__(self):
         self.active_connections: Dict[str, Set[WebSocket]] = {}  # user_id -> connections
+        self.channel_subscriptions: Dict[str, Set[WebSocket]] = {}
+        self.websocket_channels: Dict[WebSocket, Set[str]] = {}
+        self.vessel_subscriptions: Dict[WebSocket, dict[str, Any]] = {}
         self.redis_client: Optional[redis.Redis] = None
 
     async def connect(self, websocket: WebSocket, user_id: str):
@@ -40,6 +45,87 @@ class ConnectionManager:
             self.active_connections[user_id].discard(websocket)
             if not self.active_connections[user_id]:
                 del self.active_connections[user_id]
+        self.unsubscribe_all(websocket)
+
+    def subscribe(self, websocket: WebSocket, channels: list[str]):
+        normalized_channels = {
+            str(channel).strip()
+            for channel in channels
+            if str(channel).strip()
+        }
+        if not normalized_channels:
+            return
+
+        socket_channels = self.websocket_channels.setdefault(websocket, set())
+        for channel in normalized_channels:
+            self.channel_subscriptions.setdefault(channel, set()).add(websocket)
+            socket_channels.add(channel)
+
+    def unsubscribe(self, websocket: WebSocket, channels: list[str]):
+        for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
+            if channel == "vessels":
+                self.vessel_subscriptions.pop(websocket, None)
+            subscribers = self.channel_subscriptions.get(channel)
+            if subscribers is not None:
+                subscribers.discard(websocket)
+                if not subscribers:
+                    del self.channel_subscriptions[channel]
+            socket_channels = self.websocket_channels.get(websocket)
+            if socket_channels is not None:
+                socket_channels.discard(channel)
+                if not socket_channels:
+                    del self.websocket_channels[websocket]
+
+    def unsubscribe_all(self, websocket: WebSocket):
+        channels = list(self.websocket_channels.get(websocket, set()))
+        if channels:
+            self.unsubscribe(websocket, channels)
+        self.vessel_subscriptions.pop(websocket, None)
+
+    def subscribe_vessels(self, websocket: WebSocket, config: dict[str, Any]) -> dict[str, Any]:
+        subscription = self._normalize_vessel_subscription(config)
+        self.channel_subscriptions.setdefault("vessels", set()).add(websocket)
+        self.websocket_channels.setdefault(websocket, set()).add("vessels")
+        self.vessel_subscriptions[websocket] = subscription
+        return subscription
+
+    def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
+        global_scope = config.get("scope") == "global"
+        bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
+        if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
+            raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
+        try:
+            lon_min, lat_min, lon_max, lat_max = [float(value) for value in bbox]
+        except (TypeError, ValueError) as exc:
+            raise ValueError("bbox values must be numbers") from exc
+        if lat_min > lat_max:
+            lat_min, lat_max = lat_max, lat_min
+        if lon_min > lon_max:
+            lon_min, lon_max = lon_max, lon_min
+        if not (-180 <= lon_min <= 180 and -180 <= lon_max <= 180):
+            raise ValueError("bbox longitude values must be between -180 and 180")
+        if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
+            raise ValueError("bbox latitude values must be between -90 and 90")
+        if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
+            raise ValueError("bbox is too large; zoom in or request a smaller viewport")
+
+        zoom = int(config.get("zoom") or 1)
+        if zoom < 1 or zoom > 20:
+            raise ValueError("zoom must be between 1 and 20")
+        limit = min(max(int(config.get("limit") or 1000), 1), MAX_VESSEL_SUBSCRIPTION_LIMIT)
+        vessel_types = {
+            str(item).strip().lower()
+            for item in str(config.get("type") or "").split(",")
+            if str(item).strip()
+        }
+        return {
+            "scope": "global" if global_scope else "viewport",
+            "bbox": (lon_min, lat_min, lon_max, lat_max),
+            "zoom": zoom,
+            "limit": limit,
+            "type": sorted(vessel_types),
+            "last_sent_at": None,
+        }
 
     async def send_personal_message(self, message: dict, user_id: str):
         if user_id in self.active_connections:
@@ -54,13 +140,84 @@ class ConnectionManager:
             for user_id in self.active_connections:
                 await self.send_personal_message(message, user_id)
         else:
-            await self.send_personal_message(message, channel)
+            for connection in list(self.channel_subscriptions.get(channel, set())):
+                try:
+                    await connection.send_json(message)
+                except Exception:
+                    self.unsubscribe_all(connection)
+
+    async def broadcast_vessels(self, data: dict[str, Any]):
+        vessels = data.get("vessels") if isinstance(data, dict) else None
+        if not isinstance(vessels, list) or not vessels:
+            return
+
+        for connection, subscription in list(self.vessel_subscriptions.items()):
+            matched = [
+                vessel
+                for vessel in vessels
+                if self._vessel_matches_subscription(vessel, subscription)
+            ]
+            if subscription.get("scope") != "global":
+                matched = matched[:subscription["limit"]]
+            if not matched:
+                continue
+            subscription["last_sent_at"] = datetime.now(UTC)
+            message = {
+                "type": "data_frame",
+                "channel": "vessels",
+                "timestamp": subscription["last_sent_at"].isoformat(),
+                "payload": {
+                    **data,
+                    "vessels": [],
+                    "subscription": {
+                        "bbox": list(subscription["bbox"]),
+                        "zoom": subscription["zoom"],
+                        "limit": subscription["limit"],
+                        "scope": subscription.get("scope", "viewport"),
+                    },
+                },
+            }
+            try:
+                for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
+                    await connection.send_json({
+                        **message,
+                        "payload": {
+                            **message["payload"],
+                            "vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
+                        },
+                    })
+            except Exception:
+                self.unsubscribe_all(connection)
+
+    def _vessel_matches_subscription(
+        self,
+        vessel: dict[str, Any],
+        subscription: dict[str, Any],
+    ) -> bool:
+        if vessel.get("action") == "remove" and subscription.get("scope") == "global":
+            return True
+        try:
+            lon = float(vessel.get("lon"))
+            lat = float(vessel.get("lat"))
+        except (TypeError, ValueError):
+            return False
+        lon_min, lat_min, lon_max, lat_max = subscription["bbox"]
+        if not (lon_min <= lon <= lon_max and lat_min <= lat <= lat_max):
+            return False
+        requested_types = subscription.get("type") or set()
+        if not requested_types:
+            return True
+        type_name = str(vessel.get("vessel_type_name") or "").lower()
+        return any(requested_type in type_name for requested_type in requested_types)
 
     async def close_all(self):
         for user_id in self.active_connections:
             for connection in self.active_connections[user_id]:
                 await connection.close()
         self.active_connections.clear()
+        self.channel_subscriptions.clear()
+        self.websocket_channels.clear()
+        self.vessel_subscriptions.clear()
 
 
 manager = ConnectionManager()

backend/app/data/seeds/ripe_ris_collector_locations_seed.json

@@ -0,0 +1,328 @@
+{
+  "_comment": "Seed payload for the bgp_collector_locations DB table. Coordinates were migrated from the legacy RIPE_RIS_COLLECTOR_COORDS table and default to city-center; seeded rows are unverified and should be upgraded in the database with source evidence when known.",
+  "locations": [
+    {
+      "canonical_name": "RIPE RIS rrc00",
+      "aliases": ["rrc00", "RIPE RIS rrc00", "AMS-IX"],
+      "operator": "RIPE NCC",
+      "site": "AMS-IX",
+      "city": "Amsterdam",
+      "country": "Netherlands",
+      "latitude": 52.3676,
+      "longitude": 4.9041,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc01",
+      "aliases": ["rrc01", "RIPE RIS rrc01", "LINX"],
+      "operator": "RIPE NCC",
+      "site": "LINX",
+      "city": "London",
+      "country": "United Kingdom",
+      "latitude": 51.5072,
+      "longitude": -0.1276,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc03",
+      "aliases": ["rrc03", "RIPE RIS rrc03", "AMS-IX"],
+      "operator": "RIPE NCC",
+      "site": "AMS-IX",
+      "city": "Amsterdam",
+      "country": "Netherlands",
+      "latitude": 52.3676,
+      "longitude": 4.9041,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc04",
+      "aliases": ["rrc04", "RIPE RIS rrc04", "CIXP", "CERN Internet Exchange Point"],
+      "operator": "RIPE NCC",
+      "site": "CIXP",
+      "city": "Geneva",
+      "country": "Switzerland",
+      "latitude": 46.2044,
+      "longitude": 6.1432,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc05",
+      "aliases": ["rrc05", "RIPE RIS rrc05", "VIX", "Vienna Internet Exchange"],
+      "operator": "RIPE NCC",
+      "site": "VIX",
+      "city": "Vienna",
+      "country": "Austria",
+      "latitude": 48.2082,
+      "longitude": 16.3738,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc06",
+      "aliases": ["rrc06", "RIPE RIS rrc06", "JPIX", "Otemachi"],
+      "operator": "RIPE NCC",
+      "site": "JPIX",
+      "city": "Otemachi",
+      "country": "Japan",
+      "latitude": 35.686,
+      "longitude": 139.7671,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc07",
+      "aliases": ["rrc07", "RIPE RIS rrc07", "Netnod", "Netnod Stockholm"],
+      "operator": "RIPE NCC",
+      "site": "Netnod Stockholm",
+      "city": "Stockholm",
+      "country": "Sweden",
+      "latitude": 59.3293,
+      "longitude": 18.0686,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc10",
+      "aliases": ["rrc10", "RIPE RIS rrc10", "MIX", "Milan Internet Exchange"],
+      "operator": "RIPE NCC",
+      "site": "MIX",
+      "city": "Milan",
+      "country": "Italy",
+      "latitude": 45.4642,
+      "longitude": 9.19,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc11",
+      "aliases": ["rrc11", "RIPE RIS rrc11", "NYIIX", "New York International Internet Exchange"],
+      "operator": "RIPE NCC",
+      "site": "NYIIX",
+      "city": "New York",
+      "country": "United States",
+      "latitude": 40.7128,
+      "longitude": -74.006,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc12",
+      "aliases": ["rrc12", "RIPE RIS rrc12", "DE-CIX", "DE-CIX Frankfurt"],
+      "operator": "RIPE NCC",
+      "site": "DE-CIX Frankfurt",
+      "city": "Frankfurt",
+      "country": "Germany",
+      "latitude": 50.1109,
+      "longitude": 8.6821,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc13",
+      "aliases": ["rrc13", "RIPE RIS rrc13", "MSK-IX"],
+      "operator": "RIPE NCC",
+      "site": "MSK-IX",
+      "city": "Moscow",
+      "country": "Russia",
+      "latitude": 55.7558,
+      "longitude": 37.6173,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc14",
+      "aliases": ["rrc14", "RIPE RIS rrc14", "PAIX", "Palo Alto Internet Exchange"],
+      "operator": "RIPE NCC",
+      "site": "PAIX",
+      "city": "Palo Alto",
+      "country": "United States",
+      "latitude": 37.4419,
+      "longitude": -122.143,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc15",
+      "aliases": ["rrc15", "RIPE RIS rrc15", "PTT.br Sao Paulo", "PTTMetro Sao Paulo"],
+      "operator": "RIPE NCC",
+      "site": "PTT.br",
+      "city": "Sao Paulo",
+      "country": "Brazil",
+      "latitude": -23.5558,
+      "longitude": -46.6396,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc16",
+      "aliases": ["rrc16", "RIPE RIS rrc16", "Equinix Miami", "NOTA Miami"],
+      "operator": "RIPE NCC",
+      "site": "Equinix Miami",
+      "city": "Miami",
+      "country": "United States",
+      "latitude": 25.7617,
+      "longitude": -80.1918,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc18",
+      "aliases": ["rrc18", "RIPE RIS rrc18", "CATNIX"],
+      "operator": "RIPE NCC",
+      "site": "CATNIX",
+      "city": "Barcelona",
+      "country": "Spain",
+      "latitude": 41.3874,
+      "longitude": 2.1686,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc19",
+      "aliases": ["rrc19", "RIPE RIS rrc19", "NAPAfrica", "JINX", "NAPAfrica Johannesburg"],
+      "operator": "RIPE NCC",
+      "site": "NAPAfrica Johannesburg",
+      "city": "Johannesburg",
+      "country": "South Africa",
+      "latitude": -26.2041,
+      "longitude": 28.0473,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc20",
+      "aliases": ["rrc20", "RIPE RIS rrc20", "SwissIX"],
+      "operator": "RIPE NCC",
+      "site": "SwissIX",
+      "city": "Zurich",
+      "country": "Switzerland",
+      "latitude": 47.3769,
+      "longitude": 8.5417,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc21",
+      "aliases": ["rrc21", "RIPE RIS rrc21", "France-IX Paris"],
+      "operator": "RIPE NCC",
+      "site": "France-IX Paris",
+      "city": "Paris",
+      "country": "France",
+      "latitude": 48.8566,
+      "longitude": 2.3522,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc22",
+      "aliases": ["rrc22", "RIPE RIS rrc22", "InterLAN Bucharest"],
+      "operator": "RIPE NCC",
+      "site": "InterLAN Bucharest",
+      "city": "Bucharest",
+      "country": "Romania",
+      "latitude": 44.4268,
+      "longitude": 26.1025,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc23",
+      "aliases": ["rrc23", "RIPE RIS rrc23", "Equinix Singapore"],
+      "operator": "RIPE NCC",
+      "site": "Equinix Singapore",
+      "city": "Singapore",
+      "country": "Singapore",
+      "latitude": 1.3521,
+      "longitude": 103.8198,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc24",
+      "aliases": ["rrc24", "RIPE RIS rrc24", "LACNIC Montevideo"],
+      "operator": "RIPE NCC",
+      "site": "LACNIC Montevideo",
+      "city": "Montevideo",
+      "country": "Uruguay",
+      "latitude": -34.9011,
+      "longitude": -56.1645,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc25",
+      "aliases": ["rrc25", "RIPE RIS rrc25", "AMS-IX"],
+      "operator": "RIPE NCC",
+      "site": "AMS-IX",
+      "city": "Amsterdam",
+      "country": "Netherlands",
+      "latitude": 52.3676,
+      "longitude": 4.9041,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    },
+    {
+      "canonical_name": "RIPE RIS rrc26",
+      "aliases": ["rrc26", "RIPE RIS rrc26", "UAE-IX"],
+      "operator": "RIPE NCC",
+      "site": "UAE-IX",
+      "city": "Dubai",
+      "country": "United Arab Emirates",
+      "latitude": 25.2048,
+      "longitude": 55.2708,
+      "precision": "city",
+      "confidence": 0.85,
+      "source_note": "Migrated from RIPE_RIS_COLLECTOR_COORDS legacy table",
+      "verified_at": null
+    }
+  ],
+  "city_fallbacks": []
+}

backend/app/db/session.py

@@ -1,6 +1,6 @@
 from typing import AsyncGenerator
 
-from sqlalchemy import text
+from sqlalchemy import bindparam, text
 from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
 from sqlalchemy.orm import declarative_base
 
@@ -72,25 +72,112 @@ async def seed_default_datasources(session: AsyncSession):
     await session.commit()
 
 
+LEGACY_EARTH_BOUNDARY_SOURCES = (
+    "earth_admin0_boundaries",
+    "earth_coastline",
+    "earth_claim_lines",
+    "earth_boundary_tiles",
+)
+LEGACY_EARTH_BOUNDARY_DATATYPES = (
+    "earth_boundary_source",
+    "earth_boundary_tiles",
+)
+LEGACY_EARTH_BOUNDARY_IDS = (29, 30, 31, 32)
+
+
+async def purge_legacy_earth_boundary_datasources(session: AsyncSession) -> None:
+    source_names = tuple(LEGACY_EARTH_BOUNDARY_SOURCES)
+    source_ids = tuple(LEGACY_EARTH_BOUNDARY_IDS)
+    data_types = tuple(LEGACY_EARTH_BOUNDARY_DATATYPES)
+    await session.execute(
+        text(
+            """
+            DELETE FROM datasource_mapping_templates
+            WHERE target_schema IN :data_types
+               OR datasource_config_id IN (
+                    SELECT id FROM datasource_configs WHERE name IN :source_names
+               )
+            """
+        ).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
+        {"source_names": list(source_names), "data_types": list(data_types)},
+    )
+    await session.execute(
+        text("DELETE FROM datasource_configs WHERE name IN :source_names").bindparams(
+            bindparam("source_names", expanding=True)
+        ),
+        {"source_names": list(source_names)},
+    )
+    await session.execute(
+        text(
+            """
+            DELETE FROM collected_data
+            WHERE source IN :source_names OR data_type IN :data_types
+            """
+        ).bindparams(bindparam("source_names", expanding=True), bindparam("data_types", expanding=True)),
+        {"source_names": list(source_names), "data_types": list(data_types)},
+    )
+    await session.execute(
+        text(
+            """
+            DELETE FROM data_snapshots
+            WHERE source IN :source_names OR datasource_id IN :source_ids
+            """
+        ).bindparams(bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)),
+        {"source_names": list(source_names), "source_ids": list(source_ids)},
+    )
+    await session.execute(
+        text("DELETE FROM collection_tasks WHERE datasource_id IN :source_ids").bindparams(
+            bindparam("source_ids", expanding=True)
+        ),
+        {"source_ids": list(source_ids)},
+    )
+    await session.execute(
+        text("DELETE FROM data_sources WHERE source IN :source_names OR id IN :source_ids").bindparams(
+            bindparam("source_names", expanding=True), bindparam("source_ids", expanding=True)
+        ),
+        {"source_names": list(source_names), "source_ids": list(source_ids)},
+    )
+    await session.commit()
+
+
+DEFAULT_LOGIN_USERS = (
+    {
+        "username": "admin",
+        "email": "admin@planet.local",
+        "password": "admin123",
+        "role": "super_admin",
+    },
+    {
+        "username": "linkong",
+        "email": "linkong@planet.local",
+        "password": "LK12345678",
+        "role": "super_admin",
+    },
+)
+
+
 async def ensure_default_admin_user(session: AsyncSession):
     from app.core.security import get_password_hash
     from app.models.user import User
 
-    result = await session.execute(
+    for default_user in DEFAULT_LOGIN_USERS:
-        text("SELECT id FROM users WHERE username = 'admin'")
+        result = await session.execute(
-    )
+            text("SELECT id FROM users WHERE username = :username"),
-    if result.fetchone():
+            {"username": default_user["username"]},
-        return
+        )
-
+        if result.fetchone():
-    session.add(
+            continue
-        User(
+
-            username="admin",
+        session.add(
-            email="admin@planet.local",
+            User(
-            password_hash=get_password_hash("admin123"),
+                username=default_user["username"],
-            role="super_admin",
+                email=default_user["email"],
-            is_active=True,
+                password_hash=get_password_hash(default_user["password"]),
+                role=default_user["role"],
+                is_active=True,
+                email_verified=True,
+            )
         )
-    )
     await session.commit()
 
 
@@ -103,15 +190,20 @@ async def init_db():
     import app.models.datasource_config  # noqa: F401
     import app.models.alert  # noqa: F401
     import app.models.bgp_anomaly  # noqa: F401
+    import app.models.bgp_collector_location  # noqa: F401
     import app.models.bgp_incident  # noqa: F401
     import app.models.bgp_observation  # noqa: F401
     import app.models.collected_data  # noqa: F401
+    import app.models.compute_center_location  # noqa: F401
     import app.models.system_setting  # noqa: F401
     import app.models.playground_session  # noqa: F401
     import app.models.playground_message  # noqa: F401
     import app.models.system_log  # noqa: F401
     import app.models.vessel  # noqa: F401
+    import app.models.vessel_enrichment  # noqa: F401
     import app.models.datasource_mapping  # noqa: F401
+    import app.models.earth_news  # noqa: F401
+    import app.models.earth_interactable  # noqa: F401
 
     logger.warning_event(
         "Database pool settings active",
@@ -127,6 +219,31 @@ async def init_db():
 
     async with engine.begin() as conn:
         await conn.run_sync(Base.metadata.create_all)
+        users_email_verified_existed = (
+            await conn.execute(
+                text(
+                    """
+                    SELECT 1
+                    FROM information_schema.columns
+                    WHERE table_name = 'users' AND column_name = 'email_verified'
+                    """
+                )
+            )
+        ).fetchone() is not None
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE users
+                ADD COLUMN IF NOT EXISTS gatekeeper_groups JSONB DEFAULT '[]'::jsonb,
+                ADD COLUMN IF NOT EXISTS email_verified BOOLEAN NOT NULL DEFAULT FALSE,
+                ADD COLUMN IF NOT EXISTS pending_email VARCHAR(255)
+                """
+            )
+        )
+        if not users_email_verified_existed:
+            await conn.execute(
+                text("UPDATE users SET email_verified = TRUE WHERE email_verified = FALSE")
+            )
         await conn.execute(
             text(
                 """
@@ -142,6 +259,407 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE TABLE IF NOT EXISTS earth_data_change_events (
+                    id BIGSERIAL PRIMARY KEY,
+                    table_name VARCHAR(128) NOT NULL,
+                    operation VARCHAR(16) NOT NULL,
+                    source VARCHAR(128),
+                    entity_key VARCHAR(255),
+                    payload JSONB NOT NULL,
+                    occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                    consumed_at TIMESTAMPTZ
+                )
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
+                ON earth_data_change_events (consumed_at, id)
+                WHERE consumed_at IS NULL
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
+                    change_table TEXT,
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                DECLARE
+                    change_event_id BIGINT;
+                    change_payload JSONB;
+                BEGIN
+                    change_payload := jsonb_build_object(
+                        'event', 'earth.layer.changed',
+                        'table', change_table,
+                        'operation', change_operation,
+                        'source', change_source,
+                        'entity_key', NULL,
+                        'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
+                        'records_processed', COALESCE(source_record_count, 0),
+                        'occurred_at', NOW()
+                    );
+
+                    INSERT INTO earth_data_change_events (
+                        table_name,
+                        operation,
+                        source,
+                        entity_key,
+                        payload,
+                        occurred_at
+                    ) VALUES (
+                        change_table,
+                        change_operation,
+                        change_source,
+                        NULL,
+                        change_payload,
+                        NOW()
+                    )
+                    RETURNING id INTO change_event_id;
+
+                    change_payload := change_payload || jsonb_build_object(
+                        'event_id', change_event_id
+                    );
+
+                    UPDATE earth_data_change_events
+                    SET payload = change_payload
+                    WHERE id = change_event_id;
+
+                    PERFORM pg_notify(
+                        'planet_earth_data_changes',
+                        change_payload::text
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                BEGIN
+                    PERFORM planet_emit_earth_data_changed_statement(
+                        'collected_data',
+                        change_operation,
+                        change_source,
+                        source_record_count,
+                        source_entity_keys
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (
+                                        SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                        UNION ALL
+                                        SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                                    ) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM new_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM new_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM old_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM old_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM (
+                                SELECT source FROM new_rows
+                                UNION
+                                SELECT source FROM old_rows
+                            ) changed_sources
+                            WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM (
+                                        SELECT id, source_id, entity_key, source FROM new_rows
+                                        UNION ALL
+                                        SELECT id, source_id, entity_key, source FROM old_rows
+                                    ) changed_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT id, source_id, entity_key, source FROM new_rows
+                                UNION ALL
+                                SELECT id, source_id, entity_key, source FROM old_rows
+                            ) changed_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        for statement in (
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
+            "DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_insert
+            AFTER INSERT ON collected_data
+            REFERENCING NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_update
+            AFTER UPDATE ON collected_data
+            REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_delete
+            AFTER DELETE ON collected_data
+            REFERENCING OLD TABLE AS old_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+        ):
+            await conn.execute(text(statement))
+        for table_name in (
+            "bgp_observations",
+            "bgp_anomalies",
+            "bgp_incidents",
+            "bgp_collector_locations",
+            "vessel_static",
+            "vessel_position",
+            "vessel_current_state",
+            "ais_raw_observations",
+            "ais_source_health",
+            "compute_center_locations",
+            "earth_interactables",
+            "earth_news_items",
+        ):
+            for statement in (
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_insert
+                AFTER INSERT ON {table_name}
+                REFERENCING NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_update
+                AFTER UPDATE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_delete
+                AFTER DELETE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+            ):
+                await conn.execute(text(statement))
         await conn.execute(
             text(
                 """
@@ -151,7 +669,39 @@ async def init_db():
                 ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
                 ADD COLUMN IF NOT EXISTS phase_current BIGINT,
                 ADD COLUMN IF NOT EXISTS phase_total BIGINT,
-                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
+                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
+                ADD COLUMN IF NOT EXISTS source VARCHAR(100),
+                ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
+                ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
+                ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
+                ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
+                ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
+                ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS cancel_reason TEXT
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE earth_news_items
+                ADD COLUMN IF NOT EXISTS content_language VARCHAR(32) NOT NULL DEFAULT 'en',
+                ADD COLUMN IF NOT EXISTS localizations JSONB NOT NULL DEFAULT '{}'::jsonb,
+                ADD COLUMN IF NOT EXISTS enrichment_status VARCHAR(80) NOT NULL DEFAULT 'pending',
+                ADD COLUMN IF NOT EXISTS enrichment_error TEXT,
+                ADD COLUMN IF NOT EXISTS enriched_at TIMESTAMPTZ
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE earth_interactables
+                ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
+                ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
+                ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
+                ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
                 """
             )
         )
@@ -163,6 +713,124 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_news_enrichment_status
+                ON earth_news_items (enrichment_status)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_news_enriched_at
+                ON earth_news_items (enriched_at)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
+                ON earth_interactables (layer, is_deleted)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
+                ON earth_interactables (updated_at)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
+                ON collection_tasks (source, status)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
+                ON collection_tasks (status, created_at, id)
+                WHERE status = 'queued'
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
+                ON collection_tasks (dedupe_key)
+                WHERE dedupe_key IS NOT NULL
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collected_data_source_current_id
+                ON collected_data (source, is_current, id)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collected_data_source_task_id
+                ON collected_data (source, task_id, id)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox
+                ON vessel_current_state (lon, lat)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_observed
+                ON vessel_current_state (observed_at DESC)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_ais_raw_schema_observed_entity
+                ON ais_raw_observations (target_schema, observed_at, entity_key)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_ais_raw_schema_observed_desc
+                ON ais_raw_observations (target_schema, observed_at DESC)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_ais_raw_payload_lon_lat
+                ON ais_raw_observations (
+                    ((normalized_payload->>'lon')::double precision),
+                    ((normalized_payload->>'lat')::double precision)
+                )
+                WHERE target_schema = 'vessel_ais'
+                """
+            )
+        )
         await conn.execute(
             text(
                 """
@@ -183,5 +851,15 @@ async def init_db():
         )
 
     async with async_session_factory() as session:
+        from app.services.bgp_collector_locations import (
+            seed_default_bgp_collector_locations,
+        )
+        from app.services.compute_center_locations import (
+            seed_compute_center_locations_from_source_coords,
+        )
+
+        await seed_default_bgp_collector_locations(session)
+        await seed_compute_center_locations_from_source_coords(session)
         await seed_default_datasources(session)
+        await purge_legacy_earth_boundary_datasources(session)
         await ensure_default_admin_user(session)

backend/app/main.py

@@ -1,8 +1,10 @@
 from contextlib import asynccontextmanager
+from pathlib import Path
 from uuid import uuid4
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
 from starlette.middleware.base import BaseHTTPMiddleware
 
 from app.api.main import api_router
@@ -18,6 +20,15 @@ from app.services.scheduler import (
     stop_scheduler,
     sync_scheduler_with_datasources,
 )
+from app.services.earth_news_worker import (
+    start_earth_news_target_worker,
+    stop_earth_news_target_worker,
+)
+from app.services.earth_db_change_listener import (
+    start_earth_db_change_listener,
+    stop_earth_db_change_listener,
+)
+from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
 
 
 configure_logging()
@@ -53,7 +64,13 @@ async def lifespan(app: FastAPI):
     start_scheduler()
     await sync_scheduler_with_datasources()
     broadcaster.start()
+    start_data_job_worker()
+    start_earth_db_change_listener()
+    start_earth_news_target_worker()
     yield
+    await stop_earth_news_target_worker()
+    await stop_earth_db_change_listener()
+    await stop_data_job_worker()
     broadcaster.stop()
     stop_scheduler()
 
@@ -82,6 +99,14 @@ app.add_middleware(WebSocketCORSMiddleware)
 app.include_router(api_router, prefix="/api/v1")
 app.include_router(websocket.router)
 
+EARTH_BRAND_ASSET_DIR = Path(__file__).resolve().parents[2] / "data" / "earth-brand"
+EARTH_BRAND_ASSET_DIR.mkdir(parents=True, exist_ok=True)
+app.mount(
+    "/earth-brand-assets",
+    StaticFiles(directory=str(EARTH_BRAND_ASSET_DIR)),
+    name="earth-brand-assets",
+)
+
 
 @app.get("/health")
 async def health_check():

backend/app/models/__init__.py

@@ -6,14 +6,18 @@ from app.models.datasource import DataSource
 from app.models.datasource_config import DataSourceConfig
 from app.models.alert import Alert, AlertSeverity, AlertStatus
 from app.models.bgp_anomaly import BGPAnomaly
+from app.models.bgp_collector_location import BGPCollectorLocation
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
+from app.models.compute_center_location import ComputeCenterLocationRecord
 from app.models.system_setting import SystemSetting
 from app.models.playground_session import PlaygroundSession
 from app.models.playground_message import PlaygroundMessage
-from app.models.system_log import SystemLog, AuditLog
+from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
-from app.models.vessel import VesselPosition, VesselStatic
+from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
 from app.models.datasource_mapping import DataSourceMappingTemplate
+from app.models.earth_news import EarthNewsItem
+from app.models.earth_interactable import EarthInteractable
 
 __all__ = [
     "User",
@@ -27,11 +31,22 @@ __all__ = [
     "AlertSeverity",
     "AlertStatus",
     "BGPAnomaly",
+    "BGPCollectorLocation",
     "BGPIncident",
     "BGPObservation",
+    "ComputeCenterLocationRecord",
     "SystemLog",
     "AuditLog",
+    "ObservabilityEvent",
+    "ObservabilityEventGroup",
+    "PlaygroundSession",
+    "PlaygroundMessage",
     "VesselPosition",
     "VesselStatic",
+    "AISRawObservation",
+    "AISConflictRecord",
+    "AISSourceHealth",
     "DataSourceMappingTemplate",
+    "EarthNewsItem",
+    "EarthInteractable",
 ]

backend/app/models/alert.py

@@ -1,26 +1,12 @@
 from datetime import datetime
-from enum import Enum
-from typing import Optional
 
-from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
+from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum
-from sqlalchemy.orm import relationship
 
+from app.core.enums import AlertSeverity, AlertStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
 
-class AlertSeverity(str, Enum):
-    CRITICAL = "critical"
-    WARNING = "warning"
-    INFO = "info"
-
-
-class AlertStatus(str, Enum):
-    ACTIVE = "active"
-    ACKNOWLEDGED = "acknowledged"
-    RESOLVED = "resolved"
-
-
 class Alert(Base):
     __tablename__ = "alerts"
 

backend/app/models/bgp_anomaly.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -17,7 +18,7 @@ class BGPAnomaly(Base):
     source = Column(String(100), nullable=False, index=True)
     anomaly_type = Column(String(50), nullable=False, index=True)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     entity_key = Column(String(255), nullable=False, index=True)
     prefix = Column(String(64), nullable=True, index=True)
     origin_asn = Column(Integer, nullable=True, index=True)

backend/app/models/bgp_collector_location.py

@@ -0,0 +1,52 @@
+"""Stored BGP route-collector locations."""
+
+from sqlalchemy import Boolean, Column, DateTime, Float, Integer, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class BGPCollectorLocation(Base):
+    """Current known location for a BGP route collector."""
+
+    __tablename__ = "bgp_collector_locations"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    collector_id = Column(String(100), nullable=False, unique=True, index=True)
+    operator = Column(String(255), nullable=True)
+    site = Column(String(255), nullable=True)
+    city = Column(String(255), nullable=True)
+    country = Column(String(255), nullable=True)
+    latitude = Column(Float, nullable=True)
+    longitude = Column(Float, nullable=True)
+    precision = Column(String(30), nullable=False, default="city")
+    confidence = Column(Float, nullable=True)
+    source = Column(String(80), nullable=False, default="legacy_seed", index=True)
+    source_url = Column(String(500), nullable=True)
+    source_note = Column(Text, nullable=True)
+    raw_payload = Column(JSON, nullable=False, default=dict)
+    needs_confirmation = Column(Boolean, nullable=False, default=True, index=True)
+    verification_status = Column(String(30), nullable=False, default="unverified", index=True)
+    verified_at = Column(DateTime(timezone=True), nullable=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+    def to_location_dict(self) -> dict:
+        return {
+            "city": self.city,
+            "country": self.country,
+            "latitude": self.latitude,
+            "longitude": self.longitude,
+            "precision": self.precision,
+            "source": self.source,
+            "needs_confirmation": self.needs_confirmation,
+            "matched_location_name": self.site or self.collector_id,
+            "verified_at": to_iso8601_utc(self.verified_at),
+            "confidence": self.confidence,
+            "operator": self.operator,
+            "site": self.site,
+            "verification_status": self.verification_status,
+            "source_note": self.source_note,
+            "source_url": self.source_url,
+        }

backend/app/models/bgp_incident.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -20,7 +21,7 @@ class BGPIncident(Base):
     title = Column(String(255), nullable=False)
     summary = Column(Text, nullable=False)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     confidence = Column(Float, nullable=False, default=0.5)
     started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
     ended_at = Column(DateTime(timezone=True), nullable=True)

backend/app/models/collected_data.py

@@ -48,6 +48,8 @@ class CollectedData(Base):
     # Indexes for common queries
     __table_args__ = (
         Index("idx_collected_data_source_collected", "source", "collected_at"),
+        Index("idx_collected_data_source_current_id", "source", "is_current", "id"),
+        Index("idx_collected_data_source_task_id", "source", "task_id", "id"),
         Index("idx_collected_data_source_type", "source", "data_type"),
         Index("idx_collected_data_source_source_id", "source", "source_id"),
     )

backend/app/models/compute_center_location.py

@@ -0,0 +1,60 @@
+"""Stored compute-center locations."""
+
+from sqlalchemy import Boolean, Column, DateTime, Float, Integer, JSON, String, Text, UniqueConstraint
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class ComputeCenterLocationRecord(Base):
+    """Current known location for a compute-center record."""
+
+    __tablename__ = "compute_center_locations"
+    __table_args__ = (
+        UniqueConstraint("source", "source_id", name="uq_compute_center_location_source_id"),
+    )
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    source = Column(String(100), nullable=False, index=True)
+    source_id = Column(String(255), nullable=False, index=True)
+    name = Column(String(500), nullable=True)
+    operator = Column(String(255), nullable=True)
+    site = Column(String(255), nullable=True)
+    city = Column(String(255), nullable=True)
+    country = Column(String(255), nullable=True)
+    latitude = Column(Float, nullable=True)
+    longitude = Column(Float, nullable=True)
+    precision = Column(String(30), nullable=False, default="city")
+    confidence = Column(Float, nullable=True)
+    location_source = Column(String(80), nullable=False, default="stored_compute_center_location", index=True)
+    source_url = Column(String(500), nullable=True)
+    source_note = Column(Text, nullable=True)
+    raw_payload = Column(JSON, nullable=False, default=dict)
+    needs_confirmation = Column(Boolean, nullable=False, default=False, index=True)
+    verification_status = Column(String(30), nullable=False, default="verified", index=True)
+    verified_at = Column(DateTime(timezone=True), nullable=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+    def to_location_dict(self) -> dict:
+        return {
+            "source": self.source,
+            "source_id": self.source_id,
+            "name": self.name,
+            "operator": self.operator,
+            "site": self.site,
+            "city": self.city,
+            "country": self.country,
+            "latitude": self.latitude,
+            "longitude": self.longitude,
+            "precision": self.precision,
+            "confidence": self.confidence,
+            "location_source": self.location_source,
+            "source_url": self.source_url,
+            "source_note": self.source_note,
+            "raw_payload": self.raw_payload or {},
+            "needs_confirmation": self.needs_confirmation,
+            "verification_status": self.verification_status,
+            "verified_at": to_iso8601_utc(self.verified_at),
+        }

backend/app/models/data_snapshot.py

@@ -1,6 +1,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import SnapshotStatus
 from app.db.session import Base
 
 
@@ -16,7 +17,7 @@ class DataSnapshot(Base):
     started_at = Column(DateTime(timezone=True), server_default=func.now())
     completed_at = Column(DateTime(timezone=True), nullable=True)
     record_count = Column(Integer, default=0)
-    status = Column(String(20), nullable=False, default="running")
+    status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value)
     is_current = Column(Boolean, default=True, index=True)
     parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
     summary = Column(JSON, default={})

backend/app/models/datasource_mapping.py

@@ -3,6 +3,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import MappingValidationStatus
 from app.db.session import Base
 
 
@@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base):
     target_schema = Column(String(80), nullable=False, index=True)
     mapping_json = Column(JSON, nullable=False, default={})
     sample_payload_hash = Column(String(64), nullable=True)
-    validation_status = Column(String(30), nullable=False, default="draft")
+    validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value)
     version = Column(Integer, nullable=False, default=1)
     is_active = Column(Boolean, nullable=False, default=False, index=True)
     created_at = Column(DateTime(timezone=True), server_default=func.now())

backend/app/models/earth_interactable.py

@@ -0,0 +1,30 @@
+"""Persistent Earth interactable objects."""
+
+from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class EarthInteractable(Base):
+    __tablename__ = "earth_interactables"
+
+    id = Column(String(160), primary_key=True)
+    layer = Column(String(80), nullable=False, default="interactables", index=True)
+    kind = Column(String(80), nullable=False, default="default", index=True)
+    label = Column(String(255), nullable=False, default="")
+    description = Column(Text, nullable=False, default="")
+    latitude = Column(Float, nullable=False)
+    longitude = Column(Float, nullable=False)
+    altitude = Column(Float, nullable=True)
+    revision = Column(Integer, nullable=False, default=1)
+    properties = Column(JSON, nullable=False, default=dict)
+    is_deleted = Column(Boolean, nullable=False, default=False, index=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+    deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
+
+    __table_args__ = (
+        Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
+        Index("idx_earth_interactables_updated_at", "updated_at"),
+    )

backend/app/models/earth_news.py

@@ -0,0 +1,40 @@
+from sqlalchemy import Boolean, Column, DateTime, Float, Index, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class EarthNewsItem(Base):
+    __tablename__ = "earth_news_items"
+
+    id = Column(String(160), primary_key=True)
+    title = Column(String(500), nullable=False)
+    summary = Column(Text, nullable=False, default="")
+    content_language = Column(String(32), nullable=False, default="en")
+    localizations = Column(JSON, nullable=False, default=dict)
+    url = Column(Text, nullable=False)
+    source = Column(String(255), nullable=False, default="")
+    feed_name = Column(String(255), nullable=False, default="")
+    region = Column(String(80), nullable=False, index=True)
+    homepage_url = Column(Text, nullable=False, default="")
+    published_at = Column(DateTime(timezone=True), nullable=True, index=True)
+
+    latitude = Column(Float, nullable=False)
+    longitude = Column(Float, nullable=False)
+    location_label = Column(String(255), nullable=False)
+    location_source = Column(String(80), nullable=False, default="region_anchor")
+    verified = Column(Boolean, nullable=False, default=False, index=True)
+    location_meta = Column(JSON, nullable=False, default=dict)
+
+    first_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    last_seen_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False, index=True)
+    resolved_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    enrichment_status = Column(String(80), nullable=False, default="pending", index=True)
+    enrichment_error = Column(Text, nullable=True)
+    enriched_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+    __table_args__ = (
+        Index("idx_earth_news_region_published", "region", "published_at"),
+        Index("idx_earth_news_region_seen", "region", "last_seen_at"),
+    )

backend/app/models/playground_message.py

@@ -1,6 +1,7 @@
 from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus
 from app.db.session import Base
 
 
@@ -13,8 +14,8 @@ class PlaygroundMessage(Base):
     user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
     parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
     role = Column(String(20), nullable=False)
-    kind = Column(String(20), nullable=False, default="message")
+    kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value)
-    status = Column(String(20), nullable=False, default="done")
+    status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value)
     title = Column(String(255), nullable=True)
     content = Column(Text, nullable=False, default="")
     thinking_content = Column(Text, nullable=False, default="")

backend/app/models/system_log.py

@@ -38,3 +38,46 @@ class AuditLog(Base):
     ip = Column(String(64), nullable=True)
     details = Column(JSON, nullable=False, default=dict)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEvent(Base):
+    __tablename__ = "observability_events"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    level = Column(String(20), nullable=False, index=True)
+    message = Column(Text, nullable=False)
+    fingerprint = Column(String(80), nullable=False, index=True)
+    request_id = Column(String(64), nullable=True, index=True)
+    trace_id = Column(String(64), nullable=True, index=True)
+    task_id = Column(String(120), nullable=True, index=True)
+    source_ref_id = Column(String(120), nullable=True, index=True)
+    provider = Column(String(120), nullable=True, index=True)
+    user_id = Column(Integer, nullable=True, index=True)
+    context = Column(JSON, nullable=False, default=dict)
+    occurrence_count = Column(Integer, nullable=False, default=1)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEventGroup(Base):
+    __tablename__ = "observability_event_groups"
+
+    fingerprint = Column(String(80), primary_key=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    last_level = Column(String(20), nullable=False, index=True)
+    sample_message = Column(Text, nullable=False)
+    sample_detail = Column(Text, nullable=True)
+    affected_sources = Column(JSON, nullable=False, default=list)
+    count = Column(Integer, nullable=False, default=0)
+    first_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    last_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())

backend/app/models/task.py

@@ -1,8 +1,9 @@
-"""Collection Task model"""
+"""Datasource job model."""
 
-from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
+from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import JobStatus, JobType, RollbackPolicy
 from app.db.session import Base
 
 
@@ -11,8 +12,10 @@ class CollectionTask(Base):
 
     id = Column(Integer, primary_key=True, autoincrement=True)
     datasource_id = Column(Integer, nullable=False, index=True)
-    status = Column(String(20), nullable=False)  # pending, running, success, failed, cancelled
+    source = Column(String(100), nullable=True, index=True)
-    phase = Column(String(30), default="queued")
+    task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True)
+    status = Column(String(20), nullable=False)  # queued, running, cancelling, success, failed, cancelled
+    phase = Column(String(30), default=JobStatus.QUEUED.value)
     phase_progress = Column(Float)
     phase_message = Column(String(255))
     phase_current = Column(BigInteger)
@@ -24,6 +27,13 @@ class CollectionTask(Base):
     total_records = Column(Integer, default=0)  # Total records to process
     progress = Column(Float, default=0.0)  # Progress percentage (0-100)
     error_message = Column(Text)
+    payload = Column(JSON, default=dict)
+    rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value)
+    dedupe_key = Column(String(180), nullable=True, index=True)
+    worker_id = Column(String(120), nullable=True, index=True)
+    locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
+    cancel_reason = Column(Text)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
 
     def __repr__(self):

backend/app/models/user.py

@@ -1,6 +1,7 @@
-from sqlalchemy import Boolean, Column, Integer, String, DateTime
+from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import UserRole
 from app.db.session import Base
 
 
@@ -11,8 +12,11 @@ class User(Base):
     username = Column(String(50), unique=True, index=True, nullable=False)
     email = Column(String(255), unique=True, index=True, nullable=False)
     password_hash = Column(String(255), nullable=False)
-    role = Column(String(20), default="viewer")
+    role = Column(String(20), default=UserRole.VIEWER.value)
+    gatekeeper_groups = Column(JSON, default=list)
     is_active = Column(Boolean, default=True)
+    email_verified = Column(Boolean, default=False, nullable=False)
+    pending_email = Column(String(255), nullable=True)
     last_login_at = Column(DateTime(timezone=True))
     created_at = Column(DateTime(timezone=True), server_default=func.now())
     updated_at = Column(

backend/app/models/vessel.py

@@ -1,8 +1,9 @@
 """Vessel AIS models for live maritime tracking."""
 
-from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, SmallInteger, String
+from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
 from sqlalchemy.sql import func
 
+from app.core.enums import ConnectionState
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -73,3 +74,175 @@ class VesselPosition(Base):
             "nav_status": self.nav_status,
             "received_at": to_iso8601_utc(self.received_at),
         }
+
+
+class VesselCurrentState(Base):
+    """Latest renderable state for one vessel, independent from AIS history."""
+
+    __tablename__ = "vessel_current_state"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    lat = Column(Float, nullable=False)
+    lon = Column(Float, nullable=False)
+    sog = Column(Float, nullable=True)
+    cog = Column(Float, nullable=True)
+    heading = Column(SmallInteger, nullable=True)
+    nav_status = Column(SmallInteger, nullable=True, index=True)
+    name = Column(String(128), nullable=True)
+    callsign = Column(String(16), nullable=True)
+    vessel_type = Column(SmallInteger, nullable=True, index=True)
+    vessel_type_name = Column(String(64), nullable=True, index=True)
+    flag = Column(String(4), nullable=True, index=True)
+    length = Column(Float, nullable=True)
+    width = Column(Float, nullable=True)
+    draught = Column(Float, nullable=True)
+    imo = Column(BigInteger, nullable=True)
+    source = Column(String(100), nullable=False, index=True)
+    observed_at = Column(DateTime(timezone=True), nullable=False)
+    field_sources = Column(JSON, default=dict)
+    selected_reasons = Column(JSON, default=dict)
+    source_summary = Column(JSON, default=dict)
+    quality_flags = Column(JSON, default=list)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    __table_args__ = (
+        Index("idx_vessel_current_bbox", "lon", "lat"),
+        Index("idx_vessel_current_observed", "observed_at"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "lat": self.lat,
+            "lon": self.lon,
+            "sog": self.sog,
+            "cog": self.cog,
+            "heading": self.heading,
+            "nav_status": self.nav_status,
+            "name": self.name,
+            "callsign": self.callsign,
+            "vessel_type": self.vessel_type,
+            "vessel_type_name": self.vessel_type_name,
+            "flag": self.flag,
+            "length": self.length,
+            "width": self.width,
+            "draught": self.draught,
+            "imo": self.imo,
+            "source": self.source,
+            "received_at": self.observed_at,
+            "field_sources": self.field_sources or {},
+            "selected_reasons": self.selected_reasons or {},
+            "source_summary": self.source_summary or {},
+            "quality_flags": self.quality_flags or [],
+        }
+
+
+class AISRawObservation(Base):
+    """Source-level AIS fact before aggregation and conflict resolution."""
+
+    __tablename__ = "ais_raw_observations"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
+    source = Column(String(100), nullable=False, index=True)
+    entity_key = Column(String(64), nullable=False, index=True)
+    delivery_mode = Column(String(32), nullable=False, index=True)
+    transport = Column(String(32), nullable=False, index=True)
+    message_type = Column(String(64), nullable=True, index=True)
+    source_message_id = Column(String(128), nullable=True, index=True)
+    observation_hash = Column(String(64), nullable=False, unique=True, index=True)
+    observed_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    collected_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+    normalized_payload = Column(JSON, default=dict)
+    raw_payload = Column(JSON, default=dict)
+    quality_flags = Column(JSON, default=list)
+
+    __table_args__ = (
+        Index("idx_ais_raw_entity_observed", "target_schema", "entity_key", "observed_at"),
+        Index("idx_ais_raw_schema_observed_entity", "target_schema", "observed_at", "entity_key"),
+        Index("idx_ais_raw_source_entity", "source", "entity_key"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "target_schema": self.target_schema,
+            "source": self.source,
+            "entity_key": self.entity_key,
+            "delivery_mode": self.delivery_mode,
+            "transport": self.transport,
+            "message_type": self.message_type,
+            "source_message_id": self.source_message_id,
+            "observation_hash": self.observation_hash,
+            "observed_at": to_iso8601_utc(self.observed_at),
+            "collected_at": to_iso8601_utc(self.collected_at),
+            "normalized_payload": self.normalized_payload or {},
+            "raw_payload": self.raw_payload or {},
+            "quality_flags": self.quality_flags or [],
+        }
+
+
+class AISConflictRecord(Base):
+    """Recorded field-level disagreement between AIS sources."""
+
+    __tablename__ = "ais_conflict_records"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    target_schema = Column(String(64), nullable=False, default="vessel_ais", index=True)
+    entity_key = Column(String(64), nullable=False, index=True)
+    field = Column(String(64), nullable=False, index=True)
+    candidates = Column(JSON, default=dict)
+    selected_source = Column(String(100), nullable=True, index=True)
+    selected_value = Column(JSON, nullable=True)
+    selected_reason = Column(String(64), nullable=True, index=True)
+    resolved_by = Column(String(32), nullable=False, default="system", index=True)
+    status = Column(String(32), nullable=False, default="open", index=True)
+    created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    __table_args__ = (
+        Index("idx_ais_conflict_entity_field", "target_schema", "entity_key", "field"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "id": self.id,
+            "target_schema": self.target_schema,
+            "entity_key": self.entity_key,
+            "field": self.field,
+            "candidates": self.candidates or {},
+            "selected_source": self.selected_source,
+            "selected_value": self.selected_value,
+            "selected_reason": self.selected_reason,
+            "resolved_by": self.resolved_by,
+            "status": self.status,
+            "created_at": to_iso8601_utc(self.created_at),
+            "updated_at": to_iso8601_utc(self.updated_at),
+        }
+
+
+class AISSourceHealth(Base):
+    """Runtime health signal for an AIS collector source."""
+
+    __tablename__ = "ais_source_health"
+
+    source = Column(String(100), primary_key=True)
+    connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True)
+    last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    last_error = Column(String(500), nullable=True)
+    message_rate = Column(Float, nullable=True)
+    lag_seconds = Column(Float, nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), index=True)
+
+    def to_dict(self) -> dict:
+        return {
+            "source": self.source,
+            "connection_state": self.connection_state,
+            "last_seen_at": to_iso8601_utc(self.last_seen_at),
+            "last_success_at": to_iso8601_utc(self.last_success_at),
+            "last_error": self.last_error,
+            "message_rate": self.message_rate,
+            "lag_seconds": self.lag_seconds,
+            "updated_at": to_iso8601_utc(self.updated_at),
+        }

backend/app/models/vessel_enrichment.py

@@ -0,0 +1,63 @@
+"""Vessel enrichment cache tables (v5).
+
+Profile and media enrichment are stored separately so cache TTLs can differ
+and so the conflict-resolution + display layers can read either independently.
+"""
+
+from sqlalchemy import BigInteger, Column, DateTime, Float, JSON, String
+from sqlalchemy.sql import func
+
+from app.core.time import to_iso8601_utc
+from app.db.session import Base
+
+
+class VesselProfileEnrichment(Base):
+    """Cached static vessel profile (type, flag, dimensions, operator, etc.)."""
+
+    __tablename__ = "vessel_profile_enrichment"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    source = Column(String(100), nullable=False, default="system")
+    payload = Column(JSON, nullable=False, default=dict)
+    fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+    expires_at = Column(DateTime(timezone=True), nullable=True)
+    confidence = Column(Float, nullable=True)
+    reference_url = Column(String(500), nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "source": self.source,
+            "payload": self.payload or {},
+            "fetched_at": to_iso8601_utc(self.fetched_at),
+            "expires_at": to_iso8601_utc(self.expires_at),
+            "confidence": self.confidence,
+            "reference_url": self.reference_url,
+        }
+
+
+class VesselMediaEnrichment(Base):
+    """Cached vessel imagery / external detail references."""
+
+    __tablename__ = "vessel_media_enrichment"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    source = Column(String(100), nullable=False, default="system")
+    payload = Column(JSON, nullable=False, default=dict)
+    fetched_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+    expires_at = Column(DateTime(timezone=True), nullable=True)
+    confidence = Column(Float, nullable=True)
+    reference_url = Column(String(500), nullable=True)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now(), onupdate=func.now())
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "source": self.source,
+            "payload": self.payload or {},
+            "fetched_at": to_iso8601_utc(self.fetched_at),
+            "expires_at": to_iso8601_utc(self.expires_at),
+            "confidence": self.confidence,
+            "reference_url": self.reference_url,
+        }

backend/app/schemas/ai.py

@@ -2,6 +2,7 @@ from typing import Any
 
 from pydantic import BaseModel, Field
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus
 
 class AIContentBlock(BaseModel):
     type: str
@@ -13,10 +14,11 @@ class AIContentBlock(BaseModel):
 
 class SituationalAnalysisRequest(BaseModel):
     title: str = Field(..., min_length=1, max_length=200)
-    objective: str = Field(..., min_length=1, max_length=1000)
+    objective: str = Field(..., min_length=1, max_length=20000)
     context: dict[str, Any] = Field(default_factory=dict)
     observations: list[str] = Field(default_factory=list)
     constraints: list[str] = Field(default_factory=list)
+    system_prompt: str | None = Field(default=None, max_length=8000)
     preferred_model: str | None = Field(default=None, max_length=200)
     thinking: dict[str, Any] | None = None
 
@@ -109,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel):
 
 class PlaygroundMessageRecord(BaseModel):
     id: str
-    role: str
+    role: PlaygroundMessageRole
-    kind: str = "message"
+    kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE
-    status: str = "done"
+    status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE
     title: str | None = None
     content: str = ""
     thinking_content: str = ""

backend/app/schemas/user.py

@@ -3,6 +3,7 @@ from typing import Optional
 
 from pydantic import BaseModel, EmailStr, Field
 
+from app.core.enums import OtpPurpose, UserRole
 
 class UserBase(BaseModel):
     username: str
@@ -11,18 +12,21 @@ class UserBase(BaseModel):
 
 class UserCreate(UserBase):
     password: str = Field(..., min_length=8)
-    role: str = "viewer"
+    role: UserRole = UserRole.VIEWER
+    gatekeeper_groups: list[str] = Field(default_factory=list)
 
 
 class UserUpdate(BaseModel):
     email: Optional[EmailStr] = None
-    role: Optional[str] = None
+    role: Optional[UserRole] = None
+    gatekeeper_groups: Optional[list[str]] = None
     is_active: Optional[bool] = None
 
 
 class UserInDB(UserBase):
     id: int
     role: str
+    gatekeeper_groups: list[str] = Field(default_factory=list)
     is_active: bool
     last_login_at: Optional[datetime]
     created_at: datetime
@@ -34,8 +38,36 @@ class UserInDB(UserBase):
 class UserResponse(UserBase):
     id: int
     role: str
+    gatekeeper_groups: list[str] = Field(default_factory=list)
     is_active: bool
+    email_verified: bool = False
     created_at: datetime
 
     class Config:
         from_attributes = True
+
+
+class UserRegister(BaseModel):
+    username: str = Field(..., min_length=3, max_length=50)
+    email: EmailStr
+    password: str = Field(..., min_length=8, max_length=128)
+
+
+class VerifyEmailRequest(BaseModel):
+    email: EmailStr
+    code: str = Field(..., min_length=6, max_length=6)
+
+
+class ResendCodeRequest(BaseModel):
+    email: EmailStr
+    purpose: OtpPurpose = OtpPurpose.REGISTER
+
+
+class ForgotPasswordRequest(BaseModel):
+    email: EmailStr
+
+
+class ResetPasswordRequest(BaseModel):
+    email: EmailStr
+    code: str = Field(..., min_length=6, max_length=6)
+    new_password: str = Field(..., min_length=8, max_length=128)

backend/app/services/ai_client.py

@@ -1,18 +1,25 @@
 from __future__ import annotations
 
 import asyncio
+import json
+from time import perf_counter
 
 import httpx
 from fastapi import Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
+from app.core.logging import get_logger
 from app.db.session import get_db
 from app.schemas.ai import (
     AIProviderStatusResponse,
     SituationalAnalysisRequest,
     SituationalAnalysisResponse,
 )
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai")
 
 
 class AIProviderClient:
@@ -57,10 +64,25 @@ class AIProviderClient:
             value = self.llm_config.get(key)
             if value not in (None, ""):
                 headers[header_name] = str(value)
+        model_provider_apis = self.llm_config.get("model_provider_apis")
+        if isinstance(model_provider_apis, dict) and model_provider_apis:
+            headers["X-AI-Model-Provider-APIs"] = json.dumps(model_provider_apis)
         return headers
 
     async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
+        context = self._base_log_context(operation="status")
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             return AIProviderStatusResponse(
                 provider="unconfigured",
                 enabled=False,
@@ -69,27 +91,133 @@ class AIProviderClient:
                 base_url=None,
             )
 
-        data = await self._request("GET", "/v1/provider/status", request_id=request_id)
+        started_at = perf_counter()
-        return AIProviderStatusResponse.model_validate(data)
+        await emit_business_log(
+            logger,
+            event="ai.provider.status.start",
+            message="AI provider status request started",
+            category="ai",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=context,
+        )
+        try:
+            data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
+            result = AIProviderStatusResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.success",
+                message="AI provider status request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "configured": result.configured,
+                    "enabled": result.enabled,
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def analyze(
         self,
         payload: SituationalAnalysisRequest,
         request_id: str | None = None,
     ) -> SituationalAnalysisResponse:
+        context = self._base_log_context(
+            operation="analyze",
+            preferred_model=payload.preferred_model,
+            input_summary=self._summarize_analysis_payload(payload),
+        )
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             raise HTTPException(
                 status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                 detail="AI provider service URL is not configured.",
             )
 
-        data = await self._request(
+        started_at = perf_counter()
-            "POST",
+        await emit_business_log(
-            "/v1/analyze",
+            logger,
-            json=payload.model_dump(),
+            event="ai.provider.analyze.start",
+            message="AI provider analyze request started",
+            category="ai",
+            service="ai",
+            module=__name__,
             request_id=request_id,
+            context=context,
         )
-        return SituationalAnalysisResponse.model_validate(data)
+        try:
+            data = await self._request(
+                "POST",
+                "/v1/analyze",
+                json=payload.model_dump(),
+                request_id=request_id,
+                operation="analyze",
+                payload_summary=context["input_summary"],
+            )
+            result = SituationalAnalysisResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.success",
+                message="AI provider analyze request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "content_block_count": len(result.content_blocks or []),
+                    "thinking_block_count": len(result.thinking_blocks or []),
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def _request(
         self,
@@ -97,9 +225,12 @@ class AIProviderClient:
         path: str,
         json: dict | None = None,
         request_id: str | None = None,
+        operation: str = "request",
+        payload_summary: dict | None = None,
     ) -> dict:
         last_error: Exception | None = None
         for attempt in range(1, self.retry_attempts + 1):
+            attempt_started_at = perf_counter()
             try:
                 async with httpx.AsyncClient(timeout=self.timeout) as client:
                     response = await client.request(
@@ -113,6 +244,15 @@ class AIProviderClient:
             except httpx.HTTPStatusError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts and exc.response.status_code >= 500:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        status_code=exc.response.status_code,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 detail = exc.response.text or "AI provider service returned an error"
@@ -123,6 +263,14 @@ class AIProviderClient:
             except httpx.HTTPError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 raise HTTPException(
@@ -135,6 +283,71 @@ class AIProviderClient:
             detail=f"AI provider service request failed: {last_error}",
         )
 
+    def _base_log_context(self, **extra: object) -> dict:
+        llm_provider_apis = self.llm_config.get("model_provider_apis")
+        return {
+            "provider": self.llm_config.get("provider") or "",
+            "provider_api": self.llm_config.get("provider_api") or "",
+            "model": self.llm_config.get("model") or "",
+            "base_url_configured": bool(self.llm_config.get("base_url")),
+            "service_url_configured": bool(self.service_url),
+            "timeout_seconds": self.timeout,
+            "retry_attempts": self.retry_attempts,
+            "model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
+            **extra,
+        }
+
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    @staticmethod
+    def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
+        context = payload.context if isinstance(payload.context, dict) else {}
+        thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
+        return {
+            "title_length": len(payload.title or ""),
+            "objective_length": len(payload.objective or ""),
+            "observation_count": len(payload.observations or []),
+            "constraint_count": len(payload.constraints or []),
+            "has_system_prompt": bool(payload.system_prompt),
+            "thinking_enabled": bool(thinking),
+            "context_keys": sorted(str(key) for key in context.keys()),
+        }
+
+    async def _log_retry(
+        self,
+        *,
+        operation: str,
+        request_id: str | None,
+        attempt: int,
+        duration_ms: int,
+        error: BaseException,
+        status_code: int | None = None,
+        payload_summary: dict | None = None,
+    ) -> None:
+        await emit_business_log(
+            logger,
+            event=f"ai.provider.{operation}.retry",
+            message="AI provider request will retry",
+            category="ai",
+            level="warning",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=exception_context(
+                error,
+                {
+                    **self._base_log_context(operation=operation),
+                    "attempt": attempt,
+                    "next_attempt": attempt + 1,
+                    "status_code": status_code,
+                    "duration_ms": duration_ms,
+                    "input_summary": payload_summary,
+                },
+            ),
+        )
+
 
 async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
     from app.api.v1.settings import get_runtime_ai_provider_config

backend/app/services/ai_tools/__init__.py

@@ -0,0 +1,7 @@
+"""Backend-owned AI tool services.
+
+The services in this package are business tools used by Planet's backend
+orchestrators. They intentionally live outside ``aiprovider`` so model transport
+stays separate from evidence collection and domain policy.
+"""
+

backend/app/services/ai_tools/evidence_store.py

@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+import hashlib
+from typing import Iterable
+
+from app.services.ai_tools.schemas import FetchedEvidence, SearchEvidence
+
+
+def evidence_content_hash(text: str) -> str:
+    return hashlib.sha256(text.encode("utf-8")).hexdigest()
+
+
+def normalize_search_evidence(items: Iterable[SearchEvidence], *, limit: int = 5) -> list[dict]:
+    normalized: list[dict] = []
+    seen_urls: set[str] = set()
+    for item in items:
+        if not item.url or item.url in seen_urls:
+            continue
+        seen_urls.add(item.url)
+        normalized.append(
+            {
+                "title": item.title,
+                "url": item.url,
+                "snippet": item.snippet,
+                "content": item.compact_text(),
+                "score": item.score,
+                "source_provider": item.source_provider,
+                "retrieved_at": item.retrieved_at.isoformat(),
+                "metadata": item.metadata,
+            }
+        )
+        if len(normalized) >= limit:
+            break
+    return normalized
+
+
+def normalize_fetched_evidence(item: FetchedEvidence, *, text_limit: int = 1200) -> dict:
+    text = " ".join(item.text.split())[:text_limit]
+    return {
+        "title": item.title,
+        "url": item.final_url or item.url,
+        "text": text,
+        "content_hash": item.content_hash or evidence_content_hash(item.text),
+        "extractor": item.extractor,
+        "retrieved_at": item.retrieved_at.isoformat(),
+        "metadata": item.metadata,
+    }
+

backend/app/services/ai_tools/schemas.py

@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class SearchEvidence(BaseModel):
+    title: str = ""
+    url: str = ""
+    snippet: str = ""
+    content: str = ""
+    score: float | None = None
+    source_provider: str = ""
+    retrieved_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+    def compact_text(self, limit: int = 700) -> str:
+        text = " ".join((self.content or self.snippet or "").split())
+        return text[:limit]
+
+
+class FetchedEvidence(BaseModel):
+    url: str
+    final_url: str = ""
+    title: str = ""
+    text: str = ""
+    content_hash: str = ""
+    extractor: str = "basic_html"
+    retrieved_at: datetime = Field(default_factory=lambda: datetime.now(UTC))
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class WebSearchProviderConfig(BaseModel):
+    provider: str = "tavily"
+    base_url: str = ""
+    api_key: str = ""
+    max_results: int = Field(default=5, ge=1, le=20)
+    timeout_seconds: int = Field(default=20, ge=3, le=120)
+    endpoint_path: str = ""
+    search_depth: str = "basic"
+    engine: str = "google"
+    include_answer: bool = False
+    include_raw_content: bool = False
+    include_text: bool = False
+    categories: str = "general"
+    engines: list[str] = Field(default_factory=list)
+    search_path: str = ""
+    scrape_path: str = ""
+    scrape_formats: list[str] = Field(default_factory=lambda: ["markdown"])
+
+
+class WebSearchConfig(BaseModel):
+    enabled: bool = False
+    default_provider: str = "tavily"
+    provider: str = "tavily"
+    providers: dict[str, WebSearchProviderConfig] = Field(default_factory=dict)
+
+    @property
+    def active_provider_config(self) -> WebSearchProviderConfig:
+        return self.providers.get(self.default_provider) or self.providers.get(self.provider) or WebSearchProviderConfig(provider=self.default_provider or self.provider)
+

backend/app/services/ai_tools/web_fetch.py

@@ -0,0 +1,122 @@
+from __future__ import annotations
+
+import hashlib
+from time import perf_counter
+from urllib.parse import urlparse
+
+import httpx
+from bs4 import BeautifulSoup
+
+from app.core.logging import get_logger
+from app.services.ai_tools.schemas import FetchedEvidence
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
+
+
+class WebFetchError(RuntimeError):
+    pass
+
+
+def _extract_title_and_text(html: str) -> tuple[str, str]:
+    soup = BeautifulSoup(html, "html.parser")
+    for tag in soup(["script", "style", "noscript", "svg"]):
+        tag.decompose()
+    title = soup.title.get_text(" ", strip=True) if soup.title else ""
+    main = soup.find("main") or soup.find("article") or soup.body or soup
+    text = main.get_text("\n", strip=True)
+    lines = [line.strip() for line in text.splitlines() if line.strip()]
+    return title, "\n".join(lines)
+
+
+async def fetch_url_evidence(
+    url: str,
+    *,
+    timeout_seconds: int = 20,
+    max_bytes: int = 1_500_000,
+) -> FetchedEvidence:
+    started_at = perf_counter()
+    if not url:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch failed because URL is empty",
+            category="ai_tool",
+            level="warning",
+            service="ai_tool",
+            module=__name__,
+            context={"reason": "empty_url"},
+        )
+        raise WebFetchError("url is required")
+    request_host = urlparse(url).netloc
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.start",
+        message="WebFetch request started",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "timeout_seconds": timeout_seconds,
+            "max_bytes": max_bytes,
+        },
+    )
+    try:
+        async with httpx.AsyncClient(
+            timeout=timeout_seconds,
+            follow_redirects=True,
+            headers={"User-Agent": "PlanetEvidenceFetcher/1.0"},
+        ) as client:
+            response = await client.get(url)
+            response.raise_for_status()
+            content = response.content[:max_bytes]
+    except httpx.HTTPError as exc:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch request failed",
+            category="ai_tool",
+            level="error",
+            service="ai_tool",
+            module=__name__,
+            context=exception_context(
+                exc,
+                {
+                    "url_host": request_host,
+                    "status": "failed",
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            ),
+        )
+        raise WebFetchError(f"failed to fetch page: {exc}") from exc
+
+    title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
+    content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.success",
+        message="WebFetch request completed",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "final_url_host": urlparse(str(response.url)).netloc,
+            "status": "success",
+            "status_code": response.status_code,
+            "bytes_read": len(content),
+            "content_hash": content_hash,
+            "duration_ms": int((perf_counter() - started_at) * 1000),
+            "extractor": "beautifulsoup_basic",
+        },
+    )
+    return FetchedEvidence(
+        url=url,
+        final_url=str(response.url),
+        title=title,
+        text=text,
+        content_hash=content_hash,
+        extractor="beautifulsoup_basic",
+    )

backend/app/services/ai_tools/web_search.py

@@ -0,0 +1,476 @@
+from __future__ import annotations
+
+from copy import deepcopy
+import hashlib
+from time import perf_counter
+from typing import Any
+
+import httpx
+
+from app.core.logging import get_logger
+from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
+
+
+WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
+    "tavily": {
+        "provider": "tavily",
+        "label": "Tavily",
+        "api_key_env": "TAVILY_API_KEY",
+        "base_url": "https://api.tavily.com",
+        "endpoint_path": "/search",
+        "max_results": 5,
+        "timeout_seconds": 20,
+        "search_depth": "basic",
+        "include_answer": False,
+        "include_raw_content": False,
+    },
+    "brave": {
+        "provider": "brave",
+        "label": "Brave Search API",
+        "api_key_env": "BRAVE_SEARCH_API_KEY",
+        "base_url": "https://api.search.brave.com",
+        "endpoint_path": "/res/v1/web/search",
+        "max_results": 5,
+        "timeout_seconds": 20,
+    },
+    "serpapi": {
+        "provider": "serpapi",
+        "label": "SerpAPI",
+        "api_key_env": "SERPAPI_API_KEY",
+        "base_url": "https://serpapi.com",
+        "endpoint_path": "/search.json",
+        "engine": "google",
+        "max_results": 5,
+        "timeout_seconds": 20,
+    },
+    "exa": {
+        "provider": "exa",
+        "label": "Exa",
+        "api_key_env": "EXA_API_KEY",
+        "base_url": "https://api.exa.ai",
+        "endpoint_path": "/search",
+        "max_results": 5,
+        "timeout_seconds": 20,
+        "include_text": False,
+    },
+    "firecrawl": {
+        "provider": "firecrawl",
+        "label": "Firecrawl Search / Scrape",
+        "api_key_env": "FIRECRAWL_API_KEY",
+        "base_url": "https://api.firecrawl.dev",
+        "search_path": "/v2/search",
+        "scrape_path": "/v2/scrape",
+        "max_results": 5,
+        "timeout_seconds": 30,
+        "scrape_formats": ["markdown"],
+    },
+    "searxng": {
+        "provider": "searxng",
+        "label": "SearXNG",
+        "api_key_env": "SEARXNG_API_KEY",
+        "base_url": "http://localhost:8080",
+        "endpoint_path": "/",
+        "max_results": 5,
+        "timeout_seconds": 20,
+        "categories": "general",
+        "engines": [],
+    },
+}
+
+
+class WebSearchError(RuntimeError):
+    pass
+
+
+class WebSearchConfigurationError(WebSearchError):
+    pass
+
+
+def normalize_web_search_provider(provider: str | None) -> str:
+    return (provider or "tavily").strip().lower() or "tavily"
+
+
+def get_web_search_provider_preset(provider: str) -> dict[str, Any]:
+    provider_id = normalize_web_search_provider(provider)
+    preset = WEB_SEARCH_PROVIDER_PRESETS.get(provider_id)
+    if not preset:
+        raise ValueError(f"Unsupported web search provider: {provider}")
+    return deepcopy(preset)
+
+
+def list_web_search_provider_presets() -> list[dict[str, Any]]:
+    return [get_web_search_provider_preset(provider) for provider in WEB_SEARCH_PROVIDER_PRESETS]
+
+
+def provider_defaults(provider: str) -> WebSearchProviderConfig:
+    preset = get_web_search_provider_preset(provider)
+    return WebSearchProviderConfig(**{
+        key: value
+        for key, value in preset.items()
+        if key in WebSearchProviderConfig.model_fields
+    })
+
+
+class WebSearchClient:
+    def __init__(self, config: WebSearchConfig) -> None:
+        self.config = config
+
+    async def search(
+        self,
+        query: str,
+        *,
+        max_results: int | None = None,
+        domains: list[str] | None = None,
+        freshness_days: int | None = None,
+    ) -> list[SearchEvidence]:
+        started_at = perf_counter()
+        if not self.config.enabled:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because integration is disabled",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": self.config.default_provider, "reason": "disabled"},
+            )
+            raise WebSearchConfigurationError("WebSearch is disabled.")
+        provider_config = self.config.active_provider_config
+        provider = normalize_web_search_provider(provider_config.provider)
+        if provider != "searxng" and not provider_config.api_key:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because API key is not configured",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "missing_api_key"},
+            )
+            raise WebSearchConfigurationError(f"{provider} API key is not configured.")
+        query = " ".join(str(query or "").split())
+        if not query:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch failed because query is empty",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "empty_query"},
+            )
+            raise WebSearchConfigurationError("search query is required.")
+        limit = max_results or provider_config.max_results
+        context = {
+            "provider": provider,
+            "query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
+            "query_length": len(query),
+            "max_results": limit,
+            "domain_count": len(domains or []),
+            "freshness_days": freshness_days,
+        }
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_search.start",
+            message="WebSearch request started",
+            category="ai_tool",
+            service="ai_tool",
+            module=__name__,
+            context=context,
+        )
+        try:
+            if provider == "tavily":
+                results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
+            elif provider == "brave":
+                results = await self._search_brave(provider_config, query, limit, domains)
+            elif provider == "serpapi":
+                results = await self._search_serpapi(provider_config, query, limit)
+            elif provider == "exa":
+                results = await self._search_exa(provider_config, query, limit, domains)
+            elif provider == "firecrawl":
+                results = await self._search_firecrawl(provider_config, query, limit)
+            elif provider == "searxng":
+                results = await self._search_searxng(provider_config, query, limit, domains)
+            else:
+                raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
+            event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
+            await emit_business_log(
+                logger,
+                event=event,
+                message="WebSearch request completed" if results else "WebSearch returned no results",
+                category="ai_tool",
+                level="info" if results else "warning",
+                service="ai_tool",
+                module=__name__,
+                context={
+                    **context,
+                    "status": "success" if results else "empty",
+                    "result_count": len(results),
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            )
+            return results
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch request failed",
+                category="ai_tool",
+                level="error",
+                service="ai_tool",
+                module=__name__,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
+            )
+            raise
+
+    async def test_connection(self) -> list[SearchEvidence]:
+        return await self.search("Planet WebSearch connectivity test", max_results=1)
+
+    async def _request_json(
+        self,
+        method: str,
+        url: str,
+        *,
+        provider_config: WebSearchProviderConfig,
+        headers: dict[str, str] | None = None,
+        params: dict[str, Any] | None = None,
+        json: dict[str, Any] | None = None,
+    ) -> dict[str, Any]:
+        try:
+            async with httpx.AsyncClient(timeout=provider_config.timeout_seconds) as client:
+                response = await client.request(
+                    method,
+                    url,
+                    headers=headers,
+                    params=params,
+                    json=json,
+                )
+                response.raise_for_status()
+                data = response.json()
+        except httpx.HTTPStatusError as exc:
+            detail = exc.response.text or exc.response.reason_phrase
+            raise WebSearchError(f"{provider_config.provider} request failed: {detail}") from exc
+        except httpx.HTTPError as exc:
+            raise WebSearchError(f"{provider_config.provider} request failed: {exc}") from exc
+        except ValueError as exc:
+            raise WebSearchError(f"{provider_config.provider} returned invalid JSON") from exc
+        return data if isinstance(data, dict) else {}
+
+    async def _search_tavily(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+        domains: list[str] | None,
+        freshness_days: int | None,
+    ) -> list[SearchEvidence]:
+        body: dict[str, Any] = {
+            "api_key": config.api_key,
+            "query": query,
+            "max_results": max_results,
+            "search_depth": config.search_depth or "basic",
+            "include_answer": config.include_answer,
+            "include_raw_content": config.include_raw_content,
+        }
+        if domains:
+            body["include_domains"] = domains
+        if freshness_days:
+            body["days"] = freshness_days
+        data = await self._request_json(
+            "POST",
+            _join_url(config.base_url, config.endpoint_path or "/search"),
+            provider_config=config,
+            json=body,
+        )
+        return [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("url") or ""),
+                snippet=str(item.get("content") or ""),
+                content=str(item.get("raw_content") or ""),
+                score=_float_or_none(item.get("score")),
+                source_provider="tavily",
+                metadata={"query": data.get("query") or query},
+            )
+            for item in data.get("results") or []
+            if isinstance(item, dict) and item.get("url")
+        ]
+
+    async def _search_brave(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+        domains: list[str] | None,
+    ) -> list[SearchEvidence]:
+        search_query = query
+        if domains:
+            search_query = f"{query} " + " ".join(f"site:{domain}" for domain in domains)
+        data = await self._request_json(
+            "GET",
+            _join_url(config.base_url, config.endpoint_path or "/res/v1/web/search"),
+            provider_config=config,
+            headers={"X-Subscription-Token": config.api_key},
+            params={"q": search_query, "count": max_results},
+        )
+        results = (data.get("web") or {}).get("results") or []
+        return [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("url") or ""),
+                snippet=str(item.get("description") or ""),
+                source_provider="brave",
+                metadata={"age": item.get("age")},
+            )
+            for item in results
+            if isinstance(item, dict) and item.get("url")
+        ]
+
+    async def _search_serpapi(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+    ) -> list[SearchEvidence]:
+        data = await self._request_json(
+            "GET",
+            _join_url(config.base_url, config.endpoint_path or "/search.json"),
+            provider_config=config,
+            params={
+                "api_key": config.api_key,
+                "engine": config.engine or "google",
+                "q": query,
+                "num": max_results,
+            },
+        )
+        return [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("link") or ""),
+                snippet=str(item.get("snippet") or ""),
+                source_provider="serpapi",
+                metadata={"position": item.get("position")},
+            )
+            for item in data.get("organic_results") or []
+            if isinstance(item, dict) and item.get("link")
+        ]
+
+    async def _search_exa(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+        domains: list[str] | None,
+    ) -> list[SearchEvidence]:
+        body: dict[str, Any] = {
+            "query": query,
+            "numResults": max_results,
+        }
+        if domains:
+            body["includeDomains"] = domains
+        if config.include_text:
+            body["contents"] = {"text": True}
+        data = await self._request_json(
+            "POST",
+            _join_url(config.base_url, config.endpoint_path or "/search"),
+            provider_config=config,
+            headers={"Authorization": f"Bearer {config.api_key}"},
+            json=body,
+        )
+        return [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("url") or ""),
+                snippet=str(item.get("summary") or ""),
+                content=str(item.get("text") or ""),
+                score=_float_or_none(item.get("score")),
+                source_provider="exa",
+                metadata={"id": item.get("id")},
+            )
+            for item in data.get("results") or []
+            if isinstance(item, dict) and item.get("url")
+        ]
+
+    async def _search_firecrawl(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+    ) -> list[SearchEvidence]:
+        data = await self._request_json(
+            "POST",
+            _join_url(config.base_url, config.search_path or "/v2/search"),
+            provider_config=config,
+            headers={"Authorization": f"Bearer {config.api_key}"},
+            json={"query": query, "limit": max_results},
+        )
+        raw_results = data.get("data") or data.get("results") or []
+        return [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("url") or item.get("sourceURL") or ""),
+                snippet=str(item.get("description") or item.get("markdown") or ""),
+                source_provider="firecrawl",
+                metadata={"status": item.get("status")},
+            )
+            for item in raw_results
+            if isinstance(item, dict) and (item.get("url") or item.get("sourceURL"))
+        ]
+
+    async def _search_searxng(
+        self,
+        config: WebSearchProviderConfig,
+        query: str,
+        max_results: int,
+        domains: list[str] | None,
+    ) -> list[SearchEvidence]:
+        search_query = query
+        if domains:
+            search_query = f"{query} " + " ".join(f"site:{domain}" for domain in domains)
+        params: dict[str, Any] = {
+            "q": search_query,
+            "format": "json",
+            "categories": config.categories or "general",
+        }
+        if config.engines:
+            params["engines"] = ",".join(config.engines)
+        headers = {"Authorization": f"Bearer {config.api_key}"} if config.api_key else None
+        data = await self._request_json(
+            "GET",
+            _join_url(config.base_url, config.endpoint_path or "/"),
+            provider_config=config,
+            headers=headers,
+            params=params,
+        )
+        results = data.get("results") or []
+        evidence = [
+            SearchEvidence(
+                title=str(item.get("title") or ""),
+                url=str(item.get("url") or ""),
+                snippet=str(item.get("content") or ""),
+                score=_float_or_none(item.get("score")),
+                source_provider="searxng",
+                metadata={"engine": item.get("engine")},
+            )
+            for item in results
+            if isinstance(item, dict) and item.get("url")
+        ]
+        return evidence[:max_results]
+
+
+def _join_url(base_url: str, path: str) -> str:
+    return f"{(base_url or '').rstrip('/')}/{(path or '').lstrip('/')}"
+
+
+def _float_or_none(value: Any) -> float | None:
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return None

backend/app/services/alert_ai_brief.py

@@ -8,6 +8,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.models.alert import Alert, AlertSeverity, AlertStatus
 from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
+from app.ai_tasks.prompts import get_effective_prompt
+
+ALERT_BRIEF_PROMPT_KEY = "alerts.brief"
 
 
 def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
@@ -84,11 +87,13 @@ async def build_alert_brief_request(
         "top_datasources": dict(datasource_counts.most_common(6)),
         "top_active_datasources": dict(active_datasource_counts.most_common(5)),
     }
+    prompt = await get_effective_prompt(db, ALERT_BRIEF_PROMPT_KEY)
 
     return (
         SituationalAnalysisRequest(
             title="告警态势 AI 简报",
-            objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录，生成一份面向值班人员的简明告警态势简报，突出待处理风险、告警集中点和优先动作。",
+            objective=prompt.prompt,
+            system_prompt=prompt.system_prompt or None,
             observations=facts,
             constraints=[
                 "明确区分事实、推断与建议。",

backend/app/services/bgp_ai_brief.py

@@ -11,9 +11,12 @@ from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
 from app.schemas.ai import SituationalAnalysisRequest
+from app.ai_tasks.prompts import get_effective_prompt
 from app.services.bgp_collectors import build_bgp_collector_coverage
 from app.services.bgp_enrichment import lookup_prefix_geography
 
+BGP_BRIEF_PROMPT_KEY = "bgp.brief"
+
 
 def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
     if not counter:
@@ -243,12 +246,15 @@ async def build_bgp_brief_request(
             for prefix, item in list(prefix_geographies.items())[:8]
         },
     }
+    prompt = await get_effective_prompt(db, BGP_BRIEF_PROMPT_KEY)
 
     return SituationalAnalysisRequest(
         title="BGP 态势 AI 简报",
-        objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据，生成一份面向操作员的简明态势简报，突出区域热点、观测偏差、当前风险、证据和优先动作。",
+        objective=prompt.prompt,
+        system_prompt=prompt.system_prompt or None,
         observations=observations_lines,
         constraints=[
+            "直接输出中文 Markdown 简报正文，不要输出英文写作计划、提示词复述、字段说明或元评论。",
             "明确区分事实、推断与建议。",
             "优先指出需要立即关注的高严重度 incident 或异常模式。",
             "需要单独指出哪些区域结论来自 prefix geography / affected regions，哪些可能受 collector coverage 偏差影响。",

backend/app/services/bgp_collector_locations.py

@@ -0,0 +1,324 @@
+"""BGP route-collector location resolver.
+
+Collector positions are stored in the ``bgp_collector_locations`` database
+table. The old JSON registry is now only a seed payload used during database
+initialization, not a runtime resolver or candidate source.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Iterator
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.models.bgp_collector_location import BGPCollectorLocation
+from app.services.location import (
+    LocationCandidate,
+    LocationPipeline,
+    LocationQuery,
+    NominatimResolver,
+    ResolutionResult,
+    ResolverOutput,
+    SourceCoordinatesResolver,
+    build_default_nominatim_geocoder,
+    coerce_str,
+    normalize_text,
+)
+
+SEED_PATH = (
+    Path(__file__).resolve().parents[1]
+    / "data"
+    / "seeds"
+    / "ripe_ris_collector_locations_seed.json"
+)
+
+# ── Geocoder (kept at module level for monkeypatching + cache_clear) ──
+
+_geocode_online = build_default_nominatim_geocoder()
+
+
+# ── In-process compatibility cache ──────────────────────────────────
+
+
+RIPE_RIS_COLLECTOR_COORDS: dict[str, dict[str, Any]] = {}
+
+
+def _collector_record_to_dict(record: BGPCollectorLocation) -> dict[str, Any]:
+    return record.to_location_dict()
+
+
+def set_bgp_collector_location_cache(
+    locations: dict[str, dict[str, Any]],
+) -> None:
+    """Replace the legacy compatibility cache in-place."""
+    RIPE_RIS_COLLECTOR_COORDS.clear()
+    RIPE_RIS_COLLECTOR_COORDS.update(
+        {coerce_str(key): dict(value) for key, value in locations.items()}
+    )
+
+
+async def refresh_bgp_collector_location_cache(
+    session: AsyncSession,
+) -> dict[str, dict[str, Any]]:
+    result = await session.execute(select(BGPCollectorLocation))
+    records = result.scalars().all()
+    cache = {
+        record.collector_id: _collector_record_to_dict(record)
+        for record in records
+        if record.collector_id
+    }
+    set_bgp_collector_location_cache(cache)
+    return cache
+
+
+def _load_seed_payload() -> dict[str, Any]:
+    with SEED_PATH.open("r", encoding="utf-8") as handle:
+        return json.load(handle)
+
+
+def _seed_entry_to_record_kwargs(entry: dict[str, Any], collector_id: str) -> dict[str, Any]:
+    return {
+        "collector_id": collector_id,
+        "operator": entry.get("operator") or "RIPE NCC",
+        "site": entry.get("site"),
+        "city": entry.get("city"),
+        "country": entry.get("country"),
+        "latitude": entry.get("latitude"),
+        "longitude": entry.get("longitude"),
+        "precision": entry.get("precision") or "city",
+        "confidence": entry.get("confidence"),
+        "source": "legacy_seed",
+        "source_url": None,
+        "source_note": entry.get("source_note")
+        or "Seeded from legacy RIPE RIS collector coordinates",
+        "raw_payload": entry,
+        "needs_confirmation": True,
+        "verification_status": "unverified",
+        "verified_at": None,
+    }
+
+
+async def seed_default_bgp_collector_locations(session: AsyncSession) -> None:
+    """Seed default RIPE RIS collector locations without overwriting users."""
+    payload = _load_seed_payload()
+    for entry in payload.get("locations", []):
+        aliases = entry.get("aliases") or []
+        collector_ids = [
+            coerce_str(alias)
+            for alias in aliases
+            if coerce_str(alias).startswith("rrc")
+        ]
+        if not collector_ids:
+            continue
+        collector_id = collector_ids[0]
+        existing = await session.scalar(
+            select(BGPCollectorLocation).where(
+                BGPCollectorLocation.collector_id == collector_id
+            )
+        )
+        if existing:
+            continue
+        session.add(
+            BGPCollectorLocation(
+                **_seed_entry_to_record_kwargs(entry, collector_id)
+            )
+        )
+    await session.commit()
+    await refresh_bgp_collector_location_cache(session)
+
+
+def get_bgp_collector_location_dict(collector_name: str) -> dict[str, Any]:
+    """Return the current cached collector location dict, or ``{}`` if unknown."""
+    return dict(RIPE_RIS_COLLECTOR_COORDS.get(coerce_str(collector_name), {}))
+
+
+def iter_known_collector_names() -> Iterator[str]:
+    """Yield every collector technical name (rrcXX) known in the cache."""
+    return iter(sorted(RIPE_RIS_COLLECTOR_COORDS.keys()))
+
+
+# ── Pipeline construction ──────────────────────────────────────────
+
+
+class StoredCollectorLocationResolver:
+    """Resolve a collector through the DB-backed compatibility cache."""
+
+    name = "stored_collector_location"
+
+    def resolve(self, query: LocationQuery) -> ResolverOutput:
+        collector = coerce_str(query.name)
+        if not collector:
+            for alias in query.aliases:
+                collector = coerce_str(alias)
+                if collector:
+                    break
+        if not collector:
+            return ResolverOutput()
+        location = get_bgp_collector_location_dict(collector)
+        if not location:
+            return ResolverOutput()
+        latitude = location.get("latitude")
+        longitude = location.get("longitude")
+        if latitude in (None, 0.0) or longitude in (None, 0.0):
+            return ResolverOutput()
+        return ResolverOutput(
+            candidates=(
+                LocationCandidate(
+                    latitude=float(latitude),
+                    longitude=float(longitude),
+                    display_name=location.get("matched_location_name") or collector,
+                    precision=location.get("precision") or "city",
+                    confidence=float(location.get("confidence") or 0.85),
+                    query=f"stored_collector_location::{collector}",
+                    source=location.get("source") or self.name,
+                    source_note=location.get("source_note"),
+                    matched_fields=("collector",),
+                    needs_confirmation=bool(location.get("needs_confirmation")),
+                    city=location.get("city"),
+                    region=None,
+                    country=location.get("country"),
+                    matched_location_name=(
+                        location.get("matched_location_name") or collector
+                    ),
+                    location_verified_at=location.get("verified_at"),
+                    suggested_registry_entry=None,
+                ),
+            )
+        )
+
+
+def _bgp_collector_query_plan(
+    query: LocationQuery,
+) -> list[tuple[str, tuple[str, ...]]]:
+    """Build the Nominatim query plan for a BGP collector."""
+    extra = query.extra or {}
+    site = str(extra.get("site") or "")
+    operator = str(extra.get("operator") or "")
+    city = query.city or ""
+    country = query.country or ""
+
+    plan: list[tuple[str, tuple[str, ...]]] = []
+
+    def add(parts: list[tuple[str, str]]) -> None:
+        non_empty = [(field, value) for field, value in parts if value]
+        if not non_empty:
+            return
+        seen: set[str] = set()
+        cleaned: list[str] = []
+        fields: list[str] = []
+        for field, value in non_empty:
+            key = normalize_text(value)
+            if not key or key in seen:
+                continue
+            seen.add(key)
+            cleaned.append(value)
+            fields.append(field)
+        if not cleaned:
+            return
+        composed = ", ".join(cleaned)
+        if not any(composed == existing for existing, _ in plan):
+            plan.append((composed, tuple(fields)))
+
+    add([("site", site), ("city", city), ("country", country)])
+    add([("site", site), ("country", country)])
+    add([("operator", operator), ("city", city), ("country", country)])
+    add([("city", city), ("country", country)])
+    return plan
+
+
+BGP_COLLECTOR_PIPELINE = LocationPipeline(
+    [
+        SourceCoordinatesResolver(),
+        StoredCollectorLocationResolver(),
+    ],
+    failure_reason=(
+        "Could not resolve BGP collector to renderable coordinates from"
+        " source coordinates or stored collector location."
+    ),
+)
+
+BGP_COLLECTOR_COLLECTION_PIPELINE = LocationPipeline(
+    [
+        SourceCoordinatesResolver(),
+        NominatimResolver(
+            query_plan_builder=_bgp_collector_query_plan,
+            # Late-binding so tests can monkeypatch ``_geocode_online``.
+            geocoder=lambda q: _geocode_online(q),
+        ),
+    ],
+    failure_reason=(
+        "Could not resolve BGP collector to renderable coordinates from"
+        " source coordinates or online geocoding."
+    ),
+)
+
+
+# ── Public API ─────────────────────────────────────────────────────
+
+
+def resolve_bgp_collector_location(
+    collector_name: str,
+    *,
+    city: str | None = None,
+    country: str | None = None,
+    site: str | None = None,
+    operator: str | None = None,
+) -> ResolutionResult:
+    """Resolve a BGP collector to its best-known stored location."""
+    stored = get_bgp_collector_location_dict(collector_name)
+    name = coerce_str(collector_name) or None
+    query = LocationQuery(
+        name=name,
+        aliases=tuple(filter(None, (collector_name,))),
+        city=coerce_str(city or stored.get("city")) or None,
+        country=coerce_str(country or stored.get("country")) or None,
+        extra={
+            "site": coerce_str(site or stored.get("site")),
+            "operator": coerce_str(operator or stored.get("operator")) or "RIPE NCC",
+        },
+    )
+    return BGP_COLLECTOR_PIPELINE.resolve_best(query)
+
+
+def collect_bgp_collector_location_candidates(
+    *,
+    collector: str | None = None,
+    city: str | None = None,
+    country: str | None = None,
+    site: str | None = None,
+    operator: str | None = None,
+) -> tuple[list[LocationCandidate], list[str]]:
+    query = build_bgp_collector_location_query(
+        collector=collector,
+        city=city,
+        country=country,
+        site=site,
+        operator=operator,
+    )
+    return BGP_COLLECTOR_COLLECTION_PIPELINE.collect_candidates(query)
+
+
+def build_bgp_collector_location_query(
+    *,
+    collector: str | None = None,
+    city: str | None = None,
+    country: str | None = None,
+    site: str | None = None,
+    operator: str | None = None,
+) -> LocationQuery:
+    stored = get_bgp_collector_location_dict(collector or "")
+    name = coerce_str(collector) or None
+    return LocationQuery(
+        name=name,
+        aliases=tuple(filter(None, (collector,))),
+        city=coerce_str(city or stored.get("city")) or None,
+        country=coerce_str(country or stored.get("country")) or None,
+        extra={
+            "site": coerce_str(site or stored.get("site")),
+            "operator": coerce_str(operator or stored.get("operator")) or "RIPE NCC",
+            "collector": coerce_str(collector),
+        },
+    )

backend/app/services/bgp_detectors.py

@@ -6,6 +6,7 @@ from collections import Counter, defaultdict
 from datetime import UTC, datetime
 from typing import Any
 
+from app.core.enums import BGPStatus
 from app.models.bgp_anomaly import BGPAnomaly
 
 
@@ -127,7 +128,7 @@ def detect_origin_change_anomalies(
                     source=source,
                     anomaly_type=anomaly_type,
                     severity=severity,
-                    status="active",
+                    status=BGPStatus.ACTIVE.value,
                     entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
                     prefix=prefix,
                     origin_asn=sorted(historic)[0] if historic else None,
@@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies(
                 source=source,
                 anomaly_type="more_specific_burst",
                 severity="high",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
                 prefix=sample.get("prefix"),
                 origin_asn=sample.get("origin_asn"),
@@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies(
                 source=source,
                 anomaly_type="mass_withdrawal",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
                 prefix=prefix,
                 origin_asn=origin_asn,
@@ -354,7 +355,7 @@ def detect_route_leak_anomalies(
                 source=source,
                 anomaly_type="route_leak_candidate",
                 severity="high" if max_path_length >= dominant_length + 3 else "medium",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),
@@ -435,7 +436,7 @@ def detect_path_flap_anomalies(
                 source=source,
                 anomaly_type="path_flap",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),