release: bump version to 0.76.6

.claude/commands/cleanup.md

@@ -1,139 +0,0 @@
----
-description: 审查当前工作区未提交代码中的垃圾代码，并在不影响逻辑的前提下自动清理
-argument-hint: 可选：指定要检查的文件或目录（默认检查所有未提交修改）
-allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
----
-
-# /cleanup — 垃圾代码审查与清理
-
-分析当前工作区（git diff）中的未提交代码，找出并修复常见垃圾代码，**不得改变任何运行逻辑**。
-
-## 检查范围
-
-若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。
-
-## 节省上下文规则
-
-优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文：
-
-```bash
-git diff --name-only HEAD
-git diff --unified=0 HEAD -- <path>
-git diff --check
-rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
-```
-
-只有 focused diff 不足以安全判断或修改时，才读取完整文件。
-
-## 审查清单
-
-按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）：
-
-### 1. 重复逻辑 (Duplicate Logic)
-- 完全相同或高度相似的代码块在多处出现
-- 同一函数/方法被多个地方各自实现，已有公共版本未被复用
-- 相同的 DOM 查询、正则、模板字符串在同一文件重复
-
-### 2. Magic Numbers / Magic Strings
-- 裸数字直接参与计算（如偏移量、时间、尺寸、阈值），没有命名常量
-- 硬编码字符串（如 id 名、状态值、URL 片段）散落在逻辑中
-- 例外：`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
-
-### 3. 命名问题
-- 含义不明的缩写变量（如 `or_`, `tmp2`, `x2`）
-- 命名与实际用途不符
-- 同一概念在不同地方用不同名字表达
-
-### 4. 死代码 / 无效代码
-- 注释掉的旧代码块（3行以上）
-- 声明后从未使用的变量/参数/导入
-- 永远不会执行的条件分支
-
-### 5. 代码风格问题
-- 尾部空白字符（trailing whitespace）
-- 同一文件内风格不一致（如混用单双引号、缩进不统一）
-- 空行使用不一致（连续多个空行等）
-
-### 6. 其他常见问题
-- 私有辅助函数应被 export 但没有，导致调用方重复实现
-- 类型/接口重复定义
-- 过于冗长的条件表达式可以简化（不改逻辑）
-
-## 执行步骤
-
-### Step 1 — 获取待检查文件列表
-
-```bash
-# 无参数时：获取所有未提交修改
-git diff HEAD --name-only
-
-# 有参数时：用 $ARGUMENTS 过滤
-```
-
-### Step 2 — 逐文件阅读并分析
-
-先从 focused diff 开始：
-
-```bash
-git diff --unified=0 HEAD -- <file>
-```
-
-用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。
-
-### Step 3 — 报告问题清单
-
-在修改前，先以列表形式输出所有发现的问题：
-
-```
-发现 N 个问题：
-
-[文件] js/foo.js
-  · L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询，可提取到 getPanel()
-  · L91: Magic number — 硬编码 14 作为偏移量，应命名为 TOOLTIP_OFFSET
-
-[文件] js/bar.js
-  · L12: 命名问题 — 变量 `or_` 语义不明，应命名为 outerR/outerG/outerB
-  ...
-```
-
-如果没有发现问题，直接输出"未发现垃圾代码，当前代码质量良好。"并停止。
-
-### Step 4 — 执行修复
-
-对每个问题，使用 Edit 工具进行**最小化修改**：
-
-- **重复逻辑**：提取为共享常量/函数，更新所有调用点
-- **Magic number**：在文件顶部或逻辑附近声明 `const NAME = value`，替换所有引用
-- **命名问题**：重命名变量，更新所有使用处
-- **死代码**：直接删除
-- **尾部空白/风格**：修正
-- **未 export 的函数**：添加 `export`，在调用方改为导入（不重复实现）
-
-**修复原则：**
-- 只改在审查清单中发现的问题，不做额外优化
-- 每次 Edit 只修改确实有问题的行，保持 diff 最小
-- 改完后用 `grep` 验证旧的坏代码已消失
-- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具
-
-### Step 5 — 输出总结
-
-```
-清理完成：
-
-修复了 N 个问题：
-  ✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
-  ✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14（4处引用）
-  ✓ controls.js — export updateLayerButtonState，移除 main.js 中的重复实现
-  ...
-
-未修改的问题（需人工确认）：
-  ! foo.js L45 — 注释代码块较长，建议手动确认是否可删除
-```
-
-## 约束
-
-- **禁止**改变函数签名、接口定义、导出 API（除非问题正是私有函数应被 export）
-- **禁止**添加新功能、新抽象、新参数
-- **禁止**修改注释内容（只删除注释掉的死代码）
-- **禁止**修改测试文件逻辑
-- 如果一个 Magic number 的语义不完全确定，**跳过**，在总结中标记为"需人工确认"

.claude/commands/docs.md

@@ -1,104 +0,0 @@
----
-description: Create or update repository documentation from current code changes
-argument-hint: Optional: topic to document, or leave empty to infer from git diff
-allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
----
-
-# /docs — Documentation Workflow
-
-## Goal
-
-Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
-
-## Repository Rules
-
-Before deciding scope, check whether the repository has a documentation rules file:
-
-```bash
-test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
-```
-
-If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
-
-## Workflow
-
-### Step 1 — Understand The Change
-
-```bash
-git diff HEAD --stat
-git diff HEAD --name-only
-git log --oneline -10
-rg --files docs
-```
-
-If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
-
-```bash
-git diff HEAD -- <path>
-rg -n "class |def |function |export |router|@router|interface |type " <path>
-```
-
-### Step 2 — Decide Scope
-
-- Prefer updating an existing relevant document over creating a duplicate.
-- Use one document for one coherent topic.
-- Split documents only when the change crosses meaningful domains.
-- Keep filenames lowercase and hyphenated.
-- Apply the repository-specific rules file before writing.
-
-#### Document Audience Routing (Planet)
-
-In this repository, classify the action's performer before picking a target file:
-
-- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
-- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
-- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
-
-Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
-
-For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
-
-### Step 3 — Write
-
-Explain:
-
-- Background/problem: what was wrong or missing before.
-- Core design decisions and rationale.
-- Operational or user-facing impact.
-- Relevant code paths, only when useful for future maintainers.
-
-Style:
-
-- Follow the repository’s existing language and heading conventions.
-- Use fenced code blocks with language tags.
-- Prefer tables for comparisons or parameter lists.
-- Keep snippets concise and relevant.
-- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
-
-### Step 4 — Verify
-
-- Read the completed docs once for clarity and stale statements.
-- Verify referenced paths exist with `test -e` or `rg --files`.
-- Run applicable checks from `docs/documentation-coverage-rules.md`.
-- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
-
-### Step 5 — Report
-
-Summarize changed docs and verification:
-
-```md
-Updated:
-- path/to/doc.md — what changed
-
-Verified:
-- checks that passed
-- checks that could not be run, if any
-```
-
-## Hard Constraints
-
-- Do not leave placeholder docs.
-- Do not duplicate bilingual files byte-for-byte.
-- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
-- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
-- Keep docs maintainable and concise.

.claude/commands/goal-driven.md

@@ -1,93 +0,0 @@
----
-description: 用 goal-driven 方法推动一个复杂任务持续执行，直到明确成功标准被满足
-argument-hint: 建议填写任务目标；若同时给出成功标准更好
-allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
----
-
-# /goal-driven — 目标驱动执行模式
-
-使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务：先固定目标与成功标准，再持续执行和反复验收，直到标准真正满足。
-
-适用场景：
-
-- 长周期实现任务
-- 高复杂度工程任务
-- 可被明确验收的研究、实现、迁移、验证类工作
-
-不适用场景：
-
-- 纯脑暴
-- 无法定义成功标准的模糊任务
-- 很小的一次性修改
-
-## 输入要求
-
-若 `$ARGUMENTS` 只包含目标，没有成功标准，先补全一版可执行的成功标准再开始。
-
-启动时先输出：
-
-```md
-Goal
-- ...
-
-Criteria for success
-- ...
-
-Plan
-1. ...
-2. ...
-3. ...
-
-Verification
-- ...
-```
-
-## 执行规则
-
-1. 先把任务固化为两个核心块：
-   - `Goal`
-   - `Criteria for success`
-
-2. 成功标准必须尽量客观，可验证，可落地。
-   优先写成：
-   - 需要交付什么
-   - 需要通过哪些测试或验证
-   - 如何判断结果真的完成
-
-3. 进入持续执行循环：
-   - 完成一个阶段
-   - 检查当前结果是否满足成功标准
-   - 若未满足，明确剩余差距并继续推进
-
-4. 任何“完成了”“差不多了”“已实现”之类的结论，都必须经过验证，不能直接接受。
-
-5. 如果验证失败：
-   - 明确指出哪条成功标准没满足
-   - 继续工作，不要把阶段性进展误判为完成
-
-6. 只有在以下情况之一才能停止：
-   - 成功标准已满足
-   - 用户明确要求停止
-
-## 执行风格
-
-- 重证据，轻口头判断
-- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
-- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据
-- 重验收，轻自我感觉
-- 优先用测试、日志、产物、对比结果来证明完成
-- 对长期任务保持“未达标就继续”的节奏
-
-## 简版模板
-
-```md
-Goal: [[[[[在此填写最终目标]]]]]
-
-Criteria for success: [[[[[在此填写成功标准]]]]]
-
-循环执行：
-1. 推进任务
-2. 检查是否满足成功标准
-3. 若未满足，继续工作
-4. 直到满足标准或用户明确停止
-```

.claude/commands/release.md

@@ -1,160 +0,0 @@
----
-description: 发版工作流：根据变更类型决定版本号，更新所有版本文件和 changelog，运行验证，commit 并 push
-argument-hint: 可选：feature | bugfix | 或直接描述本次发布内容
-allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
----
-
-# /release — Planet 发版工作流
-
-## 版本号规则
-
-| 变更类型 | 版本跳动 | 适用场景 |
-|---------|---------|---------|
-| `feature` | `+0.1.0` | 纯新功能，无 bugfix |
-| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合，或以 UI/体验改进为主的迭代 |
-| `bugfix` | `+0.0.1` | 纯 bug 修复，无新功能 |
-| `docs` / `maintenance` / `refactor` | 默认不发版，除非用户明确要求 |
-
-意图混合时以用户明确描述为准；bugfix + 小 feature 混合默认判定为 `improvement`（`+0.0.1`）。
-
-## 必须同步更新的文件
-
-使用 `git rev-parse --show-toplevel` 获取仓库根目录，以下路径均相对于根目录：
-
-- `VERSION`
-- `frontend/package.json`（`"version"` 字段）
-- `pyproject.toml`（`version =` 字段）
-- `uv.lock`（**不要手动编辑**，通过 `uv lock` 重新生成）
-- `docs/CHANGELOG.md`
-- `docs/version-history.md`
-
-## 节省上下文规则
-
-发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取：
-
-```bash
-git status --short
-git diff --stat HEAD
-git diff --name-only HEAD
-rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
-```
-
-除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。
-
-## 执行步骤
-
-### Step 1 — 环境检查
-
-```bash
-git branch --show-current          # 确认在 dev 分支
-git status --short                 # 检查是否有无关的未暂存修改
-cat VERSION                        # 读取当前版本
-```
-
-若当前**不在 `dev` 分支**，停下来告知用户，不要继续。
-
-若存在无关的未暂存修改，列出并询问用户是否一并提交，或先 stash。
-
-### Step 2 — 确定发版类型与新版本号
-
-- 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用
-- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
-- 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`）
-- **先输出发版计划供用户确认**：
-
-```
-发版计划：
-  类型：bugfix
-  版本：0.26.2 → 0.26.3
-  分支：dev
-  将更新：VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
-```
-
-### Step 3 — 更新版本号文件
-
-按顺序更新（每步用 Edit 工具，精确替换，不要重写整个文件）：
-
-1. `VERSION` — 直接替换全部内容为新版本号
-2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
-3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
-4. 运行 `uv lock` 重新生成 `uv.lock`（在仓库根目录下执行）
-
-### Step 4 — 更新 CHANGELOG.md
-
-在文件顶部插入新条目，格式：
-
-```markdown
-## [x.x.x] — YYYY-MM-DD
-
-### ✨ Features / 🐛 Fixes / 🔧 Improvements
-- ...（只列高信号条目，最多 5 条）
-- ...
-
----
-```
-
-日期使用 `date +%Y-%m-%d` 获取今天的日期。
-
-### Step 5 — 更新 docs/version-history.md
-
-- 更新文件头部的"当前开发版本"字段
-- 在时间线表格顶部插入新行：`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
-
-### Step 6 — 验证
-
-针对本次变更范围做最小验证：
-
-- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile <changed_files>`
-- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明）
-- 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
-
-```bash
-cat VERSION
-rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
-```
-
-### Step 7 — 提交前预览
-
-展示将要提交的文件列表：
-
-```bash
-git diff --stat HEAD
-```
-
-再次确认所有必须文件都在变更列表中，**不包含**非预期文件（如调试文件、.env 等）。
-
-### Step 8 — Commit & Push（用户确认后）
-
-```bash
-git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
-# 若有代码变更也一并 stage
-git add <code_files>
-
-git commit -m "release: bump version to x.x.x"
-git tag vx.x.x
-git push origin dev
-git push origin vx.x.x
-```
-
-commit message 固定格式：`release: bump version to x.x.x`
-
-### Step 9 — 完成确认
-
-输出摘要：
-
-```
-✓ 版本号已更新：0.26.2 → 0.26.3
-✓ CHANGELOG 已更新
-✓ version-history 已更新
-✓ uv.lock 已重新生成
-✓ 验证通过
-✓ commit: release: bump version to 0.26.3
-✓ tag: v0.26.3
-✓ 已 push 到 origin/dev
-```
-
-## 注意事项
-
-- `uv.lock` 只能通过 `uv lock` 生成，绝不手动编辑
-- 发版 commit 只包含版本文件 + 本次功能代码，不混入无关改动
-- 若环境中 `uv` 不可用，说明原因并跳过 lockfile 更新，提醒用户手动运行

.dockerignore

@@ -3,8 +3,23 @@
 !pyproject.toml
 !uv.lock
 !VERSION
+!frontend/
+!frontend/public/
+!frontend/public/earth/
+!frontend/public/earth/assets/
+!frontend/public/earth/assets/cards/
+!frontend/public/earth/assets/cards/manifest.json
 !backend/
 !backend/**
+!config/
+!config/earth-boundary-pov-policy.china-v1.json
+!scripts/
+!scripts/build_earth_boundary_china_pov_geojson.py
+!scripts/build_earth_boundary_pmtiles.py
+!scripts/earth_boundary_policy.py
+!resources/
+!resources/earth/
+!resources/earth/boundary-pov-geometry.china-v1.json
 !aiprovider/
 !aiprovider/**
 

.gitignore

@@ -25,11 +25,14 @@ __pycache__/
 build/
 develop-eggs/
 dist/
-downloads/
+downloads/*
+!downloads/usbipd-win/
+downloads/usbipd-win/*
+!downloads/usbipd-win/usbipd-win-5.3.0.msi
 eggs/
 .eggs/
-lib/
-lib64/
+/lib/
+/lib64/
 parts/
 sdist/
 var/
@@ -157,3 +160,8 @@ data/earth-boundary-sources/
 # Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
 # committing thousands of generated tile files.
 frontend/public/earth/data/boundaries/
+
+# Offline comparison originals; runtime 8K maps and provenance remain versioned.
+resources/earth/candidates/nasa-blue-marble/*.jpg
+resources/earth/candidates/natural-earth-3/*.jpg
+*:Zone.Identifier

AGENTS.md

@@ -0,0 +1,183 @@
+# AGENTS.md
+
+**Planet agent harness. Defines behavior for coding agents working in this repository.**
+
+---
+
+## Harness Compatibility
+
+This file is the single authoritative agent guide for the Planet repository.
+The older lowercase `agents.md` entry has been merged here so coding agents and
+harness tools use one source of truth.
+
+### Source Of Truth
+
+- `rules.md` is the mandatory repository rule source. Always load `core`,
+  `security`, and `workflow`; load only task-relevant modules after that.
+- `AGENTS.md` defines the local agent operating mode and evidence gates.
+- `project_context.md` is background, not a rule source. Prefer newer
+  implementation docs when it disagrees with current code.
+- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
+  goal-driven work, and release.
+- Do not duplicate long workflow text across harness files. Durable constraints
+  belong in `rules.md`; task procedures belong in skills or scripts.
+
+Read these files before changing code:
+
+1. `rules.md`
+2. `AGENTS.md`
+3. `project_context.md`
+4. `README.md`
+5. `docs/HARNESS.md`
+6. `CODEMAP.md`
+
+For documentation work, also read `docs/documentation-coverage-rules.md`.
+
+### Start Safely
+
+Before broad edits:
+
+```bash
+git status --short
+scripts/harness/doctor.sh
+```
+
+Use focused context commands before reading large files:
+
+```bash
+rg -n "<symbol-or-term>" <path>
+git diff --stat HEAD
+git diff --name-only HEAD
+git diff --unified=0 HEAD -- <path>
+```
+
+Preserve user changes already present in the worktree.
+
+### Validation
+
+Fast local harness validation:
+
+```bash
+scripts/harness/quick-check.sh
+```
+
+Full local validation:
+
+```bash
+scripts/harness/validate.sh
+```
+
+`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
+unless disabled by its documented environment flags. Docker image smoke builds
+are intentionally opt-in:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
+current non-interactive environment first. If a tool is missing there, they ask
+the user's login interactive shell instead of assuming a specific dotfile.
+
+### High-Risk Areas
+
+- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
+  `destroy` cleanup.
+- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
+- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
+  rendered smoke evidence for public pages, auth guards, authenticated admin
+  route/section availability, safe navigation/search/tab interactions, mobile
+  layout, and 125% / 150% zoom, not only a build.
+- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
+  one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
+  intermediate wrappers keep `min-height: 0`, and only the intended child owns
+  scrolling.
+- `aiprovider` is a protocol/provider adapter; keep business prompts and product
+  workflows in the backend.
+- Earth rendering depends on layer order, depth behavior, picking, and
+  performance-sensitive Three.js code.
+- Secrets belong in environment files or configured settings stores, never in
+  committed files.
+- Backend service code must use structured logging instead of `print()` or
+  debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
+
+### Conflict Policy
+
+Existing project rules and workflows win. If new harness guidance conflicts with
+`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
+behavior and document the compatibility note in `docs/harness-audit.md` or
+`docs/HARNESS.md`.
+
+---
+
+## Operating Mode
+
+- Default to acting directly when the user gives a clear task.
+- For authorized multi-agent work, follow the Coding Agent Model Strategy in
+  `rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
+  and Astra for complex reasoning; the main agent owns final verification.
+- Ask before acting only when the missing decision is risky, cannot be
+  discovered from repository context, and no conservative assumption is safe.
+- Read relevant files before editing.
+- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
+  focused file reads, tests, builds, linters, and harness scripts.
+- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
+  release work, and documentation unless the task requires it.
+
+---
+
+## Evidence Gates
+
+- Visual inputs are blocking evidence. If the user provides a screenshot, image,
+  mock, browser capture, or visual reference, obtain evidence from the artifact
+  before interpreting intent or editing code.
+- Path resolution is part of the task. If the path cannot be opened, first try
+  reasonable local equivalents such as WSL/Windows path conversion,
+  workspace-relative lookup, absolute paths, and attached-file locations.
+- Never guess from prompt text, filenames, previous context, logs, OCR, or
+  memory when a visual artifact was provided but cannot be accessed.
+- OCR is acceptable evidence for text-only visual questions or non-multimodal
+  environments; state that OCR was used as the fallback. Layout, color, spacing,
+  pixel, and rendering issues need real visual inspection or a clear limitation
+  note.
+- If a visual artifact still cannot be inspected, say so and pause that
+  visual-dependent part of the work.
+- Claims of completion need evidence: a relevant test, build, lint, screenshot,
+  diff, direct file check, or harness result.
+- For UI and rendering changes, verify the rendered result when local tooling
+  allows it.
+
+---
+
+## Communication
+
+- Match the user's language. Use Chinese for Chinese requests unless the user
+  asks otherwise.
+- Keep updates short and specific: what is being inspected, edited, or verified.
+- Final responses should summarize changed files and verification, with blockers
+  stated plainly.
+- Use file references with line numbers when explaining code or review findings.
+
+---
+
+## Quality Bar
+
+- Prefer existing project patterns over new abstractions.
+- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
+  a stable path exists.
+- Centralize prompts, constants, defaults, and shared request/response handling.
+- Do not add secrets, generated runtime output, or local environment files.
+- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
+- Run the smallest relevant verification for the changed scope and report
+  anything skipped.
+
+---
+
+## Prohibited
+
+- Do not skip visual evidence handling when a visual artifact was provided.
+- Do not preserve obsolete harness files just because they already exist.
+- Do not invent behavior not present in code, docs, or verified external
+  sources.
+- Do not rewrite unrelated files during cleanup.
+- Do not mark a task complete without checking concrete success criteria.

CODEMAP.md

@@ -0,0 +1,110 @@
+# Code Map
+
+This map gives agents and maintainers a quick orientation without replacing the
+deeper architecture docs. Current implementation docs under `docs/technical/`
+are the source of detail for specific subsystems.
+
+## Top-Level Areas
+
+| Path | Role | Notes |
+| --- | --- | --- |
+| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
+| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
+| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
+| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
+| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
+| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
+| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
+| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
+| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
+| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
+
+## Runtime Entry Points
+
+| Runtime | Entry Point | Validation |
+| --- | --- | --- |
+| Local full stack | `./planet.sh start` | `./planet.sh health` |
+| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
+| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
+| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
+| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
+| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
+
+## Ownership Boundaries
+
+- Backend owns business state, auth, evidence collection, prompt selection, AI
+  task orchestration, and database persistence.
+- `aiprovider` owns provider identity, request adapter style, model gateway
+  retries, and health/status endpoints only.
+- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
+  and client-side state that mirrors backend truth.
+- Web Earth rendering changes must preserve documented layer order, altitude
+  offsets, picking behavior, legend semantics, and performance constraints.
+- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
+  wrapping it from harness scripts instead of duplicating its internals.
+- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
+  see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
+  interactive command path without hardcoding `.zshrc`.
+
+## Validation Commands
+
+```bash
+scripts/harness/doctor.sh
+scripts/harness/security-check.sh
+scripts/harness/backend-rules-check.sh
+scripts/harness/frontend-rules-check.sh
+scripts/harness/docs-consistency-check.sh
+scripts/harness/quick-check.sh
+scripts/harness/validate.sh
+./planet.sh health
+```
+
+CI-equivalent local checks:
+
+```bash
+cd backend
+uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
+
+cd frontend
+bun install --frozen-lockfile
+bun run build
+PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
+```
+
+The frontend smoke covers public routes, unauthenticated admin guards,
+login-error handling, the Earth iframe entry, and authenticated `super_admin`
+admin route/section rendering with mocked API data. Authenticated admin checks
+run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
+check for accidental global horizontal overflow. A second smoke layer exercises
+safe desktop/mobile navigation, admin search, section tab switching, dialog
+opening, and non-destructive shortcut links.
+
+Optional delivery smoke, when Docker and Helm are available:
+
+```bash
+PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
+```
+
+## Deeper Docs
+
+| Topic | Start Here |
+| --- | --- |
+| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
+| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
+| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
+| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
+| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
+| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
+| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
+| Documentation rules | `docs/documentation-coverage-rules.md` |
+| Harness workflow | `docs/HARNESS.md` |
+
+## Known Sharp Edges
+
+- `project_context.md` is static background for agents. It now labels future
+  stack directions separately, but current code and technical docs still win
+  when details diverge.
+- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
+  active local development shape.
+- Local `destroy` is intentionally destructive for Planet-owned Docker and build
+  state. Never run it as a validation shortcut.

README.md

@@ -83,7 +83,7 @@
 | 组件 | 用途 |
 |------|------|
 | React 18 | UI 框架 |
-| Ant Design Pro | 管理后台组件 |
+| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
 | Axios | HTTP 客户端 |
 | Socket.io-client | WebSocket 客户端 |
 | ECharts | 统计图表 |
@@ -168,10 +168,12 @@
 
 ## 快速启动
 
+入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上，`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx，启动 Docker 服务并配置当前用户的访问权限；需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
+
 ```bash
 # 新机器或空项目首次初始化
 ./planet.sh init
-# 会自动安装/检查 uv、bun，同步 Python/前端依赖
+# 会先准备 Docker / Compose / Buildx，再安装/检查 uv、bun 并同步 Python/前端依赖
 # 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
 # 会启动 PostgreSQL/Redis，并创建表、默认数据源和本地默认用户
 
@@ -322,7 +324,7 @@ ipconfig
 
 ## 启动容错参数
 
-`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试，并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
+`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康，再验证后端实际连接；`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待，避免重复消耗健康检查预算。
 
 可通过环境变量临时调整:
 

TODO.md

@@ -4,20 +4,16 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
 
 ## Earth
 
+- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
 - [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
 - [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
 - [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
-- [x] High-precision country boundary tile framework: implement the static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache described in [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md).
-- [x] Add the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
-- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
 - [ ] Replace debug GeoJSON boundary tiles with the real `earth-boundaries-china-pov-v1.pmtiles` production artifact after audited admin-0 / coastline / claim-line sources and the PMTiles toolchain are available.
 - [ ] Import authoritative China POV / coastline / claim-line source packages through the three standard Earth boundary source collectors, then rebuild a versioned PMTiles artifact so highest zoom `8-10` preserves trusted source geometry instead of seed data.
 - [ ] Earth boundary data: acquire or generate auditable China POV geometry for Zangnan, Aksai Chin, Taiwan/Penghu, Diaoyu Dao and affiliated islands, Chiwei Yu, South China Sea islands, Kosovo, Gaza, and the official dashed maritime claim line before implementing final visual changes.
 - [ ] Earth high-resolution basemap tiles: implement the viewport-loaded imagery layer described in [Earth High Resolution Basemap Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-resolution-basemap-tiles-plan.md), using high-precision coastline as the alignment reference instead of replacing the globe with one huge texture.
-- [ ] Presentation controller ownership: replace the singleton card fallback in [presentation-controller.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/presentation-controller.js) with a presentation/card token check before BGP/News migrate onto the shared controller, so connectors only attach to their owning card.
 - [ ] BGP frontend maintainability: split [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by responsibility into data loading, marker rendering, overlays, and animation once the current interaction behavior is stable.
 - [ ] Optional BGP marker experiment: evaluate HTML markers for BGP incident/collector points if WebGL marker density or fixed screen-size clickability becomes a real blocker.
-- [ ] Earth news cruise: connect Earth news to the generic cruise queue via a news adapter rather than coupling news-specific sequencing into `main.js`.
 
 ## Compute Centers And Location
 
@@ -48,19 +44,15 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
 - [ ] Compatibility schema: cover adapter type, base URL pattern, auth header, thinking/reasoning defaults, stream path, tool-call capability, multimodal capability, and provider-specific request patches.
 - [ ] BGP geography fallback: evaluate `inetnum` / `inet6num` whois as a finer fallback layer after `prefix_geography`, `OpenGeoFeed`, and RIR delegated data.
 
-## Platform
-
-- [ ] Earth preferences scope: keep current device-local Earth preferences in `localStorage`; only design backend user preferences if account-level synchronization becomes a real product requirement.
-- [ ] System logs: finish a usable Planet log viewing flow that covers backend, frontend, AI Provider, and collector/task logs, with filtering and tailing.
-- [ ] Console UI modernization: gradually replace Ant Design with Planet-owned components and a consistent Tabler Icons based icon system.
-- [ ] Earth live sync: design a unified realtime invalidation path for summary/BGP/satellite updates if polling and current WebSocket channels become insufficient.
-
 ## Archive
 
 Archived items stay here so old context is not lost. Completed items remain checked; obsolete, invalid, or superseded items stay unchecked and include the reason.
 
 ### Completed
 
+- [x] Implemented the high-precision country boundary tile framework from [Earth High Precision Boundary Tiles Plan](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md): static vector tile builder, versioned seed output, frontend bbox tile loader, debounce, in-flight dedupe, and LRU cache.
+- [x] Added the `pmtiles-mvt` frontend tile provider contract, MVT decoder dependencies, static PMTiles Nginx handling, collector artifact registration, production readiness check, and user operation docs for Earth boundaries.
+- [x] Split Earth boundary ingestion into standard source collectors (`earth_admin0_boundaries`, `earth_coastline`, `earth_claim_lines`) plus the downstream `earth_boundary_tiles` PMTiles builder.
 - [x] Refined BGP observer and anomaly `hover/click` feel.
 - [x] Added BGP anomaly relationship display with cables / regions.
 - [x] Added the Earth BGP activity layer so the map still feels alive when incident density is low.
@@ -75,6 +67,11 @@ Archived items stay here so old context is not lost. Completed items remain chec
 - [x] Added OpenGeoFeed as a high-quality prefix geography override source.
 - [x] Made RIR delegated data a prefix geography fallback rather than the primary source.
 - [x] Added route leak and path instability / flap detectors after the activity layer work.
+- [x] Console UI modernization. Admin is now the only console, legacy Ant Design / Admin Next code paths and dependencies have been removed, and current console UI uses Planet-owned components.
+- [x] Earth news cruise adapter. News cruise now uses `news-cruise-adapter.js` and is wired from `main.js` instead of keeping news-specific sequencing directly in the main Earth loop.
+- [x] Presentation controller ownership. `PresentationController` now guards async ownership through active request identity checks, and current callers pass per-request card targets so stale connector/card work cannot overwrite the active presentation.
+- [x] Earth live sync. Database writes now flow through `earth_data_change_events`, `earth_db_change_listener`, layer adapters, cache invalidation, and the `earth_updates` WebSocket channel; the Earth frontend debounces updates and refreshes BGP, cables, compute centers, satellites, vessels, news, and interactables by layer.
+- [x] System logs. Log sources now normalize into `LogEvent`, Admin supports snapshot filtering plus WebSocket tail/follow, task/detail views deep-link into prefiltered logs, and Admin runtime errors report through the `admin-client` log source.
 
 ### Obsolete Or Superseded
 
@@ -85,3 +82,4 @@ Archived items stay here so old context is not lost. Completed items remain chec
 - [ ] Earth surface material overlay for boundary calibration. Superseded by the high-precision boundary tile plan; future work must use source-faithful boundary/coastline data rather than overlay calibration against the coarse base map.
 - [ ] Hardcoded Earth news source extraction as a standalone task. Superseded by the broader Earth news source configuration and collector plans.
 - [ ] Country-level compute-center fallback placement as a standalone task. Superseded by the shared location pipeline and registry/manual-review backlog.
+- [ ] Earth preferences backend sync scope. Superseded by the current product decision to keep Earth preferences device-local in `localStorage` until account-level synchronization becomes a real requirement.

VERSION

@@ -1 +1 @@
-0.65.2
+0.76.6

agents.md

@@ -1,231 +0,0 @@
-# agents.md
-
-**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
-
----
-
-## Identity
-
-You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
-
-You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
-- Python FastAPI backend
-- React Admin dashboard
-- Unreal Engine 5 3D visualization
-- Multi-source data collection
-- Polarized 3D large display (4K, 120Hz)
-
----
-
-## Communication Style
-
-### Tone
-- **Professional but concise**
-- Technical accuracy with clarity
-- No unnecessary verbosity
-- Use code comments sparingly (explain **why**, not **what**)
-
-### When Responding
-1. **Answer directly** - 1-3 sentences for simple questions
-2. **Use code blocks** for all code snippets
-3. **Include file:line_number** references when discussing code
-4. **Never** start with "I am an AI assistant" or similar phrases
-5. **Never** add unnecessary preambles/postambles
-
-### Examples
-
-**Good:**
-```
-GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
-```
-
-**Bad:**
-```
-Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
-```
-
----
-
-## Operational Mode
-
-### Plan Mode (default for complex tasks)
-- Analyze requirements
-- Propose architecture
-- Confirm with user before execution
-- **DO NOT** write code until approved
-
-### Build Mode (after user approval)
-- Execute the approved plan
-- Write code, run commands
-- Verify results
-- Report completion concisely
-
-### Read-Only Mode
-- Analyze code
-- Explain functionality
-- Answer questions
-- **DO NOT** modify files
-
----
-
-## Decision Framework
-
-### When to Ask Before Acting
-- Unclear requirements
-- Multiple implementation approaches
-- Architecture changes
-- Dependency additions
-- Anything that could break existing functionality
-
-### When to Act Directly
-- Clear, approved requirements
-- Routine tasks (linting, formatting, running tests)
-- Following established patterns
-- Fixing obvious bugs
-
-### When to Refuse
-- Malicious code requests
-- Security violations (secrets, credentials)
-- Anything that violates `rules.md`
-
----
-
-## Working Principles
-
-### 1. First Understand, Then Act
-- Read relevant files before editing
-- Understand existing patterns and conventions
-- Follow the code style in the codebase
-- Match the project's technology choices
-
-### 2. Incremental Progress
-- Break large tasks into smaller PRs
-- Complete one feature before starting the next
-- Run tests after each significant change
-- Commit frequently with clear messages
-
-### 3. Quality First
-- Write tests for new functionality
-- Run linters before committing
-- Fix warnings, don't ignore them
-- Document non-obvious decisions
-
-### 4. Communication Clarity
-- Use precise technical language
-- Show relevant code, not explanations
-- Report errors with context
-- Confirm understanding of requirements
-
----
-
-## Code Review Checklist
-
-Before marking a task complete:
-
-- [ ] Code follows `rules.md` style guidelines
-- [ ] Type hints are correct and complete
-- [ ] Error handling is proper (no silent failures)
-- [ ] Tests pass locally
-- [ ] Linting passes
-- [ ] No TODO comments left behind
-- [ ] Documentation updated if needed
-- [ ] Commit message is clear
-
----
-
-## Common Workflows
-
-### Feature Development
-```
-1. Understand requirements
-2. Check existing patterns in codebase
-3. Design solution (brief mental model)
-4. Write code following rules.md
-5. Write/run tests
-6. Lint and format
-7. Commit with clear message
-8. Report completion
-```
-
-### Bug Fix
-```
-1. Reproduce the bug (write failing test)
-2. Locate the source
-3. Fix the issue
-4. Verify test passes
-5. Check for regressions
-6. Commit fix
-```
-
-### Refactoring
-```
-1. Understand current behavior
-2. Design target state
-3. Make incremental changes
-4. Preserve tests
-5. Verify functionality
-6. Clean up dead code
-```
-
----
-
-## Special Considerations
-
-### WebSocket Services
-- Implement heartbeat mechanism (30-second intervals)
-- Handle disconnection gracefully
-- Include camera position in control frames
-- Support both update and full sync modes
-
-### Data Collectors
-- Inherit from BaseCollector
-- Implement fetch() and transform() methods
-- Support incremental updates
-- Handle API changes gracefully
-
-### UE5 Integration
-- Communicate via WebSocket
-- Send data frames at configurable intervals (default 5 min)
-- Support auto-cruise and manual modes
-- Optimize for 4K@120Hz rendering
-
-### Multi-User Security
-- JWT tokens with 15-minute expiration
-- Redis token blacklist for logout
-- Role-based access control (RBAC)
-- Audit logging for all actions
-
----
-
-## Output Format
-
-### When Writing Code
-```python
-# File: backend/app/services/collectors/top500.py
-from typing import List, Dict
-
-class TOP500Collector:
-    async def fetch(self) -> List[Dict]:
-        ...
-```
-
-### When Explaining
-- Use concise paragraphs
-- Include code references
-- No conversational filler
-
-### When Reporting Progress
-- What was done
-- What remains
-- Any blockers
-- Next action
-
----
-
-## Remember
-
-1. **Rules are hard constraints** - follow `rules.md` absolutely
-2. **Context provides understanding** - use `project_context.md` for background
-3. **Role defines behavior** - follow `agents.md` for how to work
-4. **Quality over speed** - Enterprise systems require precision
-5. **Communicate clearly** - Precision in, precision out

aiprovider/Dockerfile

@@ -1,4 +1,4 @@
-# syntax=docker/dockerfile:1.7
+# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
 
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
 FROM ${UV_IMAGE} AS uv
 FROM ${PYTHON_IMAGE}
 
-ARG AI_PROVIDER_BUILD_FINGERPRINT
-LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
-
 COPY --from=uv /uv /uvx /bin/
 
 WORKDIR /app
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 COPY pyproject.toml uv.lock /app/
 RUN --mount=type=cache,target=/root/.cache/uv \
     --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
-    uv sync --frozen --no-dev
+    uv sync --frozen --only-group aiprovider
 
 COPY aiprovider /app/aiprovider
 
+ARG AI_PROVIDER_BUILD_FINGERPRINT
+LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
+
 EXPOSE 8010
 
 HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
     CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
 
-CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
+CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]

aiprovider/README.md

@@ -4,7 +4,7 @@
 
 完整使用说明见:
 
-- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
+- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
 
 当前支持:
 
@@ -15,6 +15,7 @@
   - `AI_PROVIDER=ollama`
 - request adapter:
   - `AI_PROVIDER_API=openai-completions`
+  - `AI_PROVIDER_API=openai-responses`
   - `AI_PROVIDER_API=anthropic-messages`
   - `AI_PROVIDER_API=ollama-generate`
 

aiprovider/provider_service.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import asyncio
 import json
 from typing import Any
+from uuid import NAMESPACE_URL, uuid4, uuid5
 
 import httpx
 from fastapi import HTTPException, status
@@ -65,6 +66,7 @@ class ProviderService:
         self.model_provider_apis = self._parse_model_provider_apis(
             overrides.get("model_provider_apis")
         )
+        self.session_id = str(uuid4())
 
     def get_status(self) -> AIProviderStatusResponse:
         enabled = self.provider != "disabled"
@@ -95,6 +97,8 @@ class ProviderService:
             )
 
         prompt = self._build_prompt(payload)
+        if payload.context.get("session_id") is not None:
+            self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
 
         provider_api = self._resolve_model_provider_api(model)
 
@@ -102,6 +106,10 @@ class ProviderService:
             data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
             content = self._extract_openai_content(data)
             content_blocks = self._extract_openai_blocks(data)
+        elif provider_api == "openai-responses":
+            data = await self._request_openai_responses(model, prompt, payload.system_prompt)
+            content_blocks = self._extract_responses_blocks(data)
+            content = "".join(block.text for block in content_blocks if block.text)
         elif provider_api == "anthropic-messages":
             data = await self._request_anthropic_messages(
                 model,
@@ -200,6 +208,39 @@ class ProviderService:
             request_body=request_body,
         )
 
+    async def _request_openai_responses(
+        self, model: str, prompt: str, system_prompt: str | None = None,
+    ) -> dict[str, Any]:
+        request_body: dict[str, Any] = {
+            "model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
+        }
+        resolved_system_prompt = self._resolve_system_prompt(system_prompt)
+        if resolved_system_prompt:
+            request_body["instructions"] = resolved_system_prompt
+        return await self._post(
+            path="/responses",
+            headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
+            request_body=request_body,
+        )
+
+    def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
+        blocks: list[AIContentBlock] = []
+        for item in payload.get("output") or []:
+            if not isinstance(item, dict):
+                continue
+            if item.get("type") == "message":
+                for part in item.get("content") or []:
+                    if not isinstance(part, dict):
+                        continue
+                    text = part.get("text") or part.get("refusal")
+                    if isinstance(text, str) and text:
+                        blocks.append(AIContentBlock(type="text", text=text))
+            elif item.get("type") == "reasoning":
+                for part in item.get("summary") or []:
+                    if isinstance(part, dict) and isinstance(part.get("text"), str):
+                        blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
+        return blocks
+
     async def _request_anthropic_messages(
         self,
         model: str,
@@ -226,7 +267,7 @@ class ProviderService:
         resolved_system_prompt = self._resolve_system_prompt(system_prompt)
         if resolved_system_prompt:
             request_body["system"] = resolved_system_prompt
-        resolved_thinking = self._resolve_anthropic_thinking(thinking)
+        resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
         if resolved_thinking:
             request_body["thinking"] = resolved_thinking
         if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
@@ -243,8 +284,12 @@ class ProviderService:
             request_body=request_body,
         )
 
-    def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
+    def _resolve_anthropic_thinking(
+        self, thinking: dict[str, Any] | None, model: str,
+    ) -> dict[str, Any] | None:
         if thinking:
+            if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
+                return {"type": "adaptive"}
             return thinking
 
         # OpenClaw treats MiniMax's Anthropic-compatible path specially:
@@ -295,6 +340,9 @@ class ProviderService:
         headers: dict[str, str],
         request_body: dict[str, Any],
     ) -> dict[str, Any]:
+        headers = {"User-Agent": "Planet/1.0", **headers}
+        if self.provider == "opencode-go":
+            headers["x-opencode-session"] = self.session_id
         last_error: Exception | None = None
         for attempt in range(1, self.http_retry_attempts + 1):
             try:

backend/Dockerfile

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
     uv sync --frozen --no-dev
 
 COPY backend /app/backend
+COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
+COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
+COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
+COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
+COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
+COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
 COPY VERSION /app/VERSION
 
 EXPOSE 8000

backend/app/api/main.py

@@ -1,3 +1,4 @@
+from app.api.v1 import card_images
 from fastapi import APIRouter
 from app.api.v1 import (
     ai,
@@ -12,6 +13,7 @@ from app.api.v1 import (
     alerts,
     settings,
     collected_data,
+    compute_intelligence,
     data_products,
     layers,
     visualization,
@@ -19,6 +21,7 @@ from app.api.v1 import (
     vessels,
     bgp,
     news,
+    interactables,
     realtime_sources,
     system_control,
     tv,
@@ -42,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
 api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
 api_router.include_router(system_control.router, prefix="/system", tags=["system"])
 api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
+api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
 api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
 api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
 api_router.include_router(
@@ -53,4 +57,7 @@ api_router.include_router(vessels.router, prefix="/vessels", tags=["vessels"])
 api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
 api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
 api_router.include_router(news.router, prefix="/news", tags=["news"])
+api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
 api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
+
+api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])

backend/app/api/v1/ai.py

@@ -3,6 +3,7 @@ from uuid import uuid4
 from fastapi import APIRouter, Depends, HTTPException, Request, Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.logging import get_logger
 from app.core.security import get_current_user
 from app.db.session import get_db
 from app.models.user import User
@@ -47,8 +48,10 @@ from app.services.playground_chat_service import (
     stop_message,
 )
 from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
+from app.services.business_logs import emit_business_log, exception_context
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 
 
 @router.get("/provider/status", response_model=AIProviderStatusResponse)
@@ -122,6 +125,16 @@ async def create_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.create",
+        message="Playground message creation requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "preset": payload.selected_preset_key},
+    )
     return await create_turn(
         db,
         user_id=current_user.id,
@@ -136,6 +149,16 @@ async def stop_playground_message(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.stop",
+        message="Playground message stop requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "message_id": payload.message_id},
+    )
     return await stop_message(
         db,
         user_id=current_user.id,
@@ -150,6 +173,16 @@ async def resend_playground_message(
     provider_client: AIProviderClient = Depends(get_ai_provider_client),
     db: AsyncSession = Depends(get_db),
 ):
+    await emit_business_log(
+        logger,
+        event="ai.playground.message.resend",
+        message="Playground message resend requested",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"session_key": payload.session_key, "user_message_id": payload.user_message_id},
+    )
     return await resend_turn(
         db,
         user_id=current_user.id,
@@ -214,16 +247,70 @@ async def analyze_bgp_brief(
         anomaly_limit=payload.anomaly_limit,
         collector_limit=payload.collector_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.bgp.facts_collected",
+        message="BGP brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={
+            "incident_limit": payload.incident_limit,
+            "anomaly_limit": payload.anomaly_limit,
+            "collector_limit": payload.collector_limit,
+            "fact_count": len(facts or []),
+        },
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return save_bgp_brief_record(
-        analysis,
+    await emit_business_log(
+        logger,
+        event="ai.brief.bgp.start",
+        message="BGP brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
         request_id=request_id,
-        facts=facts,
-        context=context,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        record = save_bgp_brief_record(
+            analysis,
+            request_id=request_id,
+            facts=facts,
+            context=context,
+        )
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.completed",
+            message="BGP brief AI analysis saved",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model, "brief_id": record.id},
+        )
+        return record
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.bgp.failed",
+            message="BGP brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/alerts/brief", response_model=AlertBriefResponse)
@@ -242,17 +329,65 @@ async def analyze_alert_brief(
         db,
         alert_limit=payload.alert_limit,
     )
+    await emit_business_log(
+        logger,
+        event="ai.brief.alerts.facts_collected",
+        message="Alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"alert_limit": payload.alert_limit, "fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return AlertBriefResponse(
-        **analysis.model_dump(),
-        title=brief_request.title,
-        objective=brief_request.objective,
-        facts=facts,
-        context=context,
+    await emit_business_log(
+        logger,
+        event="ai.brief.alerts.start",
+        message="Alert brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.completed",
+            message="Alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return AlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.alerts.failed",
+            message="Alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise
 
 
 @router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
@@ -268,14 +403,62 @@ async def analyze_situational_alert_brief(
     response.headers["X-Request-ID"] = request_id
 
     brief_request, facts, context = await build_situational_alert_brief_request(db)
+    await emit_business_log(
+        logger,
+        event="ai.brief.situational_alerts.facts_collected",
+        message="Situational alert brief facts collected",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"fact_count": len(facts or [])},
+    )
     brief_request.preferred_model = payload.preferred_model
     brief_request.thinking = payload.thinking
 
-    analysis = await provider_client.analyze(brief_request, request_id=request_id)
-    return SituationalAlertBriefResponse(
-        **analysis.model_dump(),
-        title=brief_request.title,
-        objective=brief_request.objective,
-        facts=facts,
-        context=context,
+    await emit_business_log(
+        logger,
+        event="ai.brief.situational_alerts.start",
+        message="Situational alert brief AI analysis started",
+        category="ai",
+        service="api",
+        module=__name__,
+        request_id=request_id,
+        user_id=current_user.id,
+        context={"preferred_model": payload.preferred_model},
     )
+    try:
+        analysis = await provider_client.analyze(brief_request, request_id=request_id)
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.completed",
+            message="Situational alert brief AI analysis completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context={"provider": analysis.provider, "model": analysis.model},
+        )
+        return SituationalAlertBriefResponse(
+            **analysis.model_dump(),
+            title=brief_request.title,
+            objective=brief_request.objective,
+            facts=facts,
+            context=context,
+        )
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="ai.brief.situational_alerts.failed",
+            message="Situational alert brief AI analysis failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            request_id=request_id,
+            user_id=current_user.id,
+            context=exception_context(exc, {"preferred_model": payload.preferred_model}),
+        )
+        raise

backend/app/api/v1/auth.py

@@ -4,6 +4,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import OtpPurpose, UserRole
 from app.core.logging import get_logger
 from app.core.security import (
     create_access_token,
@@ -170,7 +171,7 @@ async def get_me(current_user: User = Depends(get_current_user)):
     }
 
 
-async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: str) -> None:
+async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None:
     try:
         await send_verification_email(db, to=email, code=code, purpose=purpose)
     except EmailNotConfiguredError as exc:
@@ -207,7 +208,7 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
         username=payload.username,
         email=payload.email,
         password_hash=get_password_hash(payload.password),
-        role="viewer",
+        role=UserRole.VIEWER.value,
         is_active=True,
         email_verified=False,
     )
@@ -215,13 +216,13 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
     await db.commit()
 
     try:
-        code = otp.issue_code(payload.email, "register")
+        code = otp.issue_code(payload.email, OtpPurpose.REGISTER)
     except otp.OtpResendRateLimited as exc:
         raise HTTPException(
             status_code=status.HTTP_429_TOO_MANY_REQUESTS,
             detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
         ) from exc
-    await _send_code_or_raise(db, payload.email, code, "register")
+    await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER)
     return {"status": "pending_verification", "email": payload.email}
 
 
@@ -266,7 +267,7 @@ async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get
     if user is None:
         # Avoid email enumeration; pretend success.
         return {"status": "ok"}
-    if payload.purpose == "register" and user.email_verified:
+    if payload.purpose is OtpPurpose.REGISTER and user.email_verified:
         raise HTTPException(
             status_code=status.HTTP_409_CONFLICT,
             detail={"code": "ALREADY_VERIFIED"},
@@ -289,12 +290,17 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
         # Don't leak whether an email is registered.
         return {"status": "ok"}
     try:
-        code = otp.issue_code(payload.email, "reset_password")
+        code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD)
     except otp.OtpResendRateLimited:
         # Silently accept; the user can retry after the cooldown.
         return {"status": "ok"}
     try:
-        await send_verification_email(db, to=payload.email, code=code, purpose="reset_password")
+        await send_verification_email(
+            db,
+            to=payload.email,
+            code=code,
+            purpose=OtpPurpose.RESET_PASSWORD,
+        )
     except EmailNotConfiguredError as exc:
         raise HTTPException(
             status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
@@ -304,7 +310,11 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
         logger.warning_event(
             "SMTP send failed",
             event="auth.email.send_failed",
-            context={"email": payload.email, "purpose": "reset_password", "error": str(exc)},
+            context={
+                "email": payload.email,
+                "purpose": OtpPurpose.RESET_PASSWORD.value,
+                "error": str(exc),
+            },
         )
     return {"status": "ok"}
 
@@ -318,7 +328,7 @@ async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depen
             detail={"code": "OTP_INVALID"},
         )
     try:
-        otp.verify_code(payload.email, "reset_password", payload.code)
+        otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code)
     except otp.OtpExpired as exc:
         raise HTTPException(
             status_code=status.HTTP_410_GONE,

backend/app/api/v1/card_images.py

@@ -0,0 +1,108 @@
+"""Card-image configuration, upload, catalog selection and public resolution."""
+
+from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
+from fastapi.responses import FileResponse
+from sqlalchemy.ext.asyncio import AsyncSession
+from starlette.concurrency import run_in_threadpool
+
+from app.core.enums import UserRole
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.user import User
+from app.schemas.card_images import (
+    CardImageConfiguration,
+    CardImagePreview,
+    CardImagePublish,
+    EntityType,
+    MatchScope,
+)
+from app.services.card_image_configuration import (
+    publish_configuration,
+    read_configuration,
+    resolve_image,
+)
+from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
+from app.services.card_image_targets import get_target, search_targets
+
+router = APIRouter()
+
+
+async def image_admin(user: User = Depends(get_current_user)) -> User:
+    if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
+        raise HTTPException(403, "Administrator access required")
+    return user
+
+
+@router.get("/configuration")
+async def get_configuration(
+    response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
+):
+    response.headers["Cache-Control"] = "no-store"
+    return await read_configuration(db)
+
+
+@router.put("/configuration")
+async def save_configuration(
+    payload: CardImagePublish,
+    actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return await publish_configuration(db, payload.revision, payload.config, actor)
+
+
+@router.post("/upload")
+async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
+    content = await file.read(MAX_UPLOAD_BYTES + 1)
+    await file.close()
+    return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
+
+
+@router.get("/files/{filename}")
+async def image_file(filename: str):
+    path = uploaded_path(filename)
+    if not path.is_file():
+        raise HTTPException(404, "Image not found")
+    return FileResponse(
+        path,
+        headers={
+            "Cache-Control": "public, max-age=31536000, immutable",
+            "X-Content-Type-Options": "nosniff",
+        },
+    )
+
+
+@router.get("/targets")
+async def targets(
+    entity_type: EntityType,
+    scope: MatchScope = "object",
+    q: str = Query(default="", max_length=100),
+    limit: int = Query(default=25, ge=1, le=50),
+    _actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
+
+
+@router.get("/resolve")
+async def resolve(
+    response: Response,
+    entity_type: EntityType,
+    entity_id: str = Query(min_length=1, max_length=300),
+    db: AsyncSession = Depends(get_db),
+):
+    response.headers["Cache-Control"] = "no-store"
+    envelope = await read_configuration(db)
+    config = CardImageConfiguration.model_validate(envelope["config"])
+    context = await get_target(db, entity_type, entity_id)
+    result = resolve_image(config, context)
+    return {**result, "revision": envelope["revision"], "entity": context}
+
+
+@router.post("/preview")
+async def preview(
+    payload: CardImagePreview,
+    _actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    context = await get_target(db, payload.entity_type, payload.entity_id)
+    return {**resolve_image(payload.config, context), "entity": context}

backend/app/api/v1/compute_intelligence.py

@@ -0,0 +1,235 @@
+"""Compute observations, administrator configuration and independent facility samples."""
+
+from datetime import UTC, datetime
+from typing import Literal
+
+from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
+from app.api.v1.visualization import convert_compute_centers_to_geojson
+from app.core.earth_geography_policy import display_country
+from app.core.enums import UserRole
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.collected_data import CollectedData
+from app.models.data_snapshot import DataSnapshot
+from app.models.user import User
+from app.schemas.compute_configuration import (
+    ComputeConfiguration,
+    ComputeConfigurationReset,
+    ComputeConfigurationWrite,
+)
+from app.services.compute_catalog import (
+    PARTICIPANTS,
+    participant_members,
+    source_country_names,
+    source_country_participants,
+)
+from app.services.compute_center_locations import refresh_compute_center_location_cache
+from app.services.compute_configuration import (
+    DEFAULT_CONFIGURATION,
+    configuration_response,
+    publish_configuration,
+    read_configuration,
+)
+from app.services.compute_history import compute_history
+from fastapi import APIRouter, Depends, HTTPException, Query, Response
+from sqlalchemy import and_, case, func, or_, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+router = APIRouter()
+
+
+def selected_countries(countries: str) -> list[str]:
+    codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
+    if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
+        raise HTTPException(422, "Select 1–12 participants from the compute catalog")
+    members = participant_members(codes)
+    flattened = [country for participant in members.values() for country in participant]
+    if len(flattened) != len(set(flattened)):
+        raise HTTPException(422, "Comparison participants cannot have overlapping country members")
+    return codes
+
+
+@router.get("/series")
+async def get_series(
+    response: Response,
+    countries: str = "CN,US",
+    mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
+    db: AsyncSession = Depends(get_db),
+):
+    codes = selected_countries(countries)
+    envelope = await read_configuration(db)
+    response.headers["Cache-Control"] = "no-store"
+    result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
+    result.update(
+        version=envelope["revision"],
+        provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
+    )
+    return result
+
+
+async def configuration_admin(user: User = Depends(get_current_user)) -> User:
+    if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
+        raise HTTPException(403, "Administrator access required")
+    return user
+
+
+@router.get("/configuration")
+async def get_configuration(
+    response: Response,
+    _actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    response.headers["Cache-Control"] = "no-store"
+    return configuration_response(await read_configuration(db))
+
+
+@router.put("/configuration")
+async def update_configuration(
+    payload: ComputeConfigurationWrite,
+    actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    envelope = await publish_configuration(
+        db, revision=payload.revision, config=payload.config, actor=actor,
+    )
+    return configuration_response(envelope)
+
+
+@router.post("/configuration/reset")
+async def reset_configuration(
+    payload: ComputeConfigurationReset,
+    actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    envelope = await publish_configuration(
+        db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
+    )
+    return configuration_response(envelope)
+
+
+async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
+    now = datetime.now(UTC)
+    cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
+    ranked = (
+        select(
+            DataSnapshot.id,
+            func.row_number()
+            .over(
+                partition_by=DataSnapshot.source,
+                order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
+            )
+            .label("rank"),
+        )
+        .where(
+            DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
+            DataSnapshot.status == "success",
+            DataSnapshot.completed_at < cutoff,
+        )
+        .subquery()
+    )
+    snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
+    infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
+    # Old collector rows have ambiguous units/identity and must not become
+    # national evidence merely because a new UI can display them.
+    query = select(CollectedData).where(
+        compute_record_filter(CollectedData),
+        CollectedData.deleted_at.is_(None),
+        CollectedData.snapshot_id.in_(snapshots),
+        CollectedData.is_valid == 1,
+        or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
+        or_(
+            infrastructure,
+            CollectedData.source == "top500",
+            and_(
+                CollectedData.extra_data["status"].as_string() == "Existing",
+                func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
+                CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
+            ),
+        ),
+        CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
+    )
+    return query
+
+
+@router.get("/facilities")
+async def get_facilities(
+    countries: str = "CN,US",
+    year: int | None = Query(None, ge=2000, le=2100),
+    limit: int = Query(1000, ge=1, le=2000),
+    db: AsyncSession = Depends(get_db),
+):
+    codes = selected_countries(countries)
+    query = await sample_query(db, codes, year)
+    country = CollectedData.extra_data["country"].as_string()
+    source_participants = source_country_participants(codes)
+    # Canonical source subdivisions retain their raw label on each record but
+    # share one owner in the comparison, using the same policy as the globe.
+    owner_cases = [
+        (country == name, display_country(name))
+        for name in source_country_names(codes)
+        if display_country(name) != name
+    ]
+    display = case(*owner_cases, else_=country) if owner_cases else country
+    participant_cases = [(country == name, code) for name, code in source_participants.items()]
+    participant = case(*participant_cases, else_=None)
+    counts = (
+        (
+            await db.execute(
+                query.with_only_columns(
+                    display.label("country"),
+                    participant.label("participant_code"),
+                    CollectedData.source,
+                    func.count().label("count"),
+                ).group_by(display, participant, CollectedData.source)
+            )
+        )
+        .mappings()
+        .all()
+    )
+    participant_counts = (
+        (
+            await db.execute(
+                query.with_only_columns(
+                    participant.label("participant_code"),
+                    CollectedData.source,
+                    func.count().label("count"),
+                ).group_by(participant, CollectedData.source)
+            )
+        )
+        .mappings()
+        .all()
+    )
+    records = (
+        (await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
+        .scalars()
+        .all()
+    )
+    if year:
+        await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
+    else:
+        await refresh_compute_center_location_cache(db)
+    payload = convert_compute_centers_to_geojson(records)
+    for feature in payload["features"]:
+        feature["properties"]["historical_snapshot"] = year is not None
+        properties = feature["properties"]
+        properties["source_country"] = properties.get("country")
+        properties["country"] = display_country(properties.get("country"))
+    total = sum(row["count"] for row in counts)
+    payload.update(
+        {
+            "counts": [dict(row) for row in counts],
+            "participant_counts": [dict(row) for row in participant_counts],
+            "participants": [PARTICIPANTS[code] for code in codes],
+            "total_count": total,
+            "returned_count": len(records),
+            "truncated": total > limit,
+            "reference_year": year,
+            "scope": "public_source_records",
+            "history_basis": (
+                "snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
+            ),
+            "coverage": "available" if total else "no_verified_snapshot",
+            "cross_source_deduplicated": False,
+        }
+    )
+    return payload

backend/app/api/v1/data_products.py

@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
         "name": "算力",
         "sources": ["top500", "epoch_ai_gpu"],
         "primary_stat_key": "compute_center_count",
-        "stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
+        "stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
     },
 }
 

backend/app/api/v1/datasource_config.py

@@ -13,6 +13,7 @@ import httpx
 
 from app.core.target_schema_registry import get_target_schema, list_target_schemas
 from app.core.datasource_defaults import DEFAULT_DATASOURCES
+from app.core.enums import AuthType, MappingValidationStatus, UserRole
 from app.db.session import get_db
 from app.models.user import User
 from app.models.datasource_config import DataSourceConfig
@@ -34,7 +35,6 @@ from app.services.datasource_mapping import (
 )
 from app.services.custom_datasource_runtime import (
     CustomDatasourceRuntimeError,
-    fetch_rest_payload,
     get_custom_stream_status,
     run_mapped_rest_config,
     run_mapped_websocket_config,
@@ -42,8 +42,6 @@ from app.services.custom_datasource_runtime import (
     stop_custom_stream,
     test_websocket_config,
 )
-
-DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
 from app.services.datasource_connectivity import (
     _resolve_aisstream_api_key,
     _resolve_spacetrack_credentials_with_override,
@@ -57,7 +55,8 @@ from app.services.persistent_logs import record_audit_log
 
 router = APIRouter()
 
-SECRET_REVEAL_ROLES = {"admin", "super_admin"}
+DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
+SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
 
 
 def _user_role_value(user: User) -> str:
@@ -124,7 +123,7 @@ class DataSourceConfigCreate(BaseModel):
     description: Optional[str] = None
     source_type: str = Field(..., description="rest, websocket, http, api, database")
     endpoint: str = Field(..., max_length=500)
-    auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
+    auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic")
     auth_config: dict = Field(default={})
     headers: dict = Field(default={})
     config: dict = Field(default={"timeout": 30, "retry": 3})
@@ -135,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel):
     description: Optional[str] = None
     source_type: Optional[str] = None
     endpoint: Optional[str] = Field(None, max_length=500)
-    auth_type: Optional[str] = None
+    auth_type: Optional[AuthType] = None
     auth_config: Optional[dict] = None
     headers: Optional[dict] = None
     config: Optional[dict] = None
@@ -210,7 +209,7 @@ class MappingTemplateCreate(BaseModel):
     mapping_json: dict
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
+    validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT
     is_active: bool = False
 
 
@@ -219,7 +218,7 @@ class MappingTemplateUpdate(BaseModel):
     mapping_json: Optional[dict] = None
     sample_payload: Any | None = None
     sample_payload_hash: Optional[str] = None
-    validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
+    validation_status: Optional[MappingValidationStatus] = None
     is_active: Optional[bool] = None
 
 
@@ -918,6 +917,7 @@ async def get_datasource_target_schemas(
 @router.post("/mappings/propose")
 async def propose_datasource_mapping(
     payload: MappingProposeRequest,
+    db: AsyncSession = Depends(get_db),
     current_user: User = Depends(get_current_user),
     ai_client: AIProviderClient = Depends(get_ai_provider_client),
 ):

backend/app/api/v1/datasources.py

@@ -1,4 +1,3 @@
-import asyncio
 from datetime import datetime, timedelta, timezone
 from typing import Optional
 
@@ -7,7 +6,8 @@ from pydantic import BaseModel, Field
 from sqlalchemy import func, or_, select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.core.cache import cache
+from app.core.logging import get_logger
+from app.core.enums import JobStatus, SnapshotStatus
 from app.core.time import to_iso8601_utc
 from app.core.security import get_current_user
 from app.core.data_sources import get_data_sources_config
@@ -20,18 +20,27 @@ from app.models.datasource_config import DataSourceConfig
 from app.models.task import CollectionTask
 from app.models.user import User
 from app.models.vessel import AISRawObservation
-from app.services.vessel_ais_aggregation import VESSEL_AIS_SCHEMA
 from app.services.scheduler import (
-    cancel_running_collector_now,
-    get_latest_task_id_for_datasource,
-    run_collector_now,
     sync_datasource_job,
 )
-from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
+from app.services.data_jobs import (
+    JOB_STATUS_CANCELLING,
+    JOB_STATUS_QUEUED,
+    JOB_STATUS_RUNNING,
+    JOB_TYPE_CLEAR_CACHE,
+    JOB_TYPE_CLEAR_DATA,
+    JOB_TYPE_COLLECT,
+    enqueue_datasource_job,
+    get_active_datasource_job,
+    request_cancel_datasource_task,
+)
+from app.services.business_logs import emit_business_log
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
 
+
 PRODUCT_SOURCE_KEYWORDS: tuple[tuple[str, tuple[str, ...]], ...] = (
     ("vessels", ("vessel", "ais")),
     ("cables", ("cable", "landing", "telegeography", "arcgis", "fao")),
@@ -115,7 +124,7 @@ async def _load_latest_running_tasks(
             _task_rank_column(CollectionTask.started_at),
         )
         .where(CollectionTask.datasource_id.in_(datasource_ids))
-        .where(CollectionTask.status == "running")
+        .where(CollectionTask.status.in_((JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)))
         .subquery()
     )
     result = await db.execute(
@@ -126,32 +135,6 @@ async def _load_latest_running_tasks(
     return {task.datasource_id: task for task in result.scalars().all()}
 
 
-async def _load_latest_task_ids(
-    db: AsyncSession,
-    datasource_ids: list[int],
-) -> dict[int, int]:
-    if not datasource_ids:
-        return {}
-
-    ranked_tasks = (
-        select(
-            CollectionTask.id.label("task_id"),
-            CollectionTask.datasource_id.label("datasource_id"),
-            func.row_number().over(
-                partition_by=CollectionTask.datasource_id,
-                order_by=CollectionTask.id.desc(),
-            ).label("row_num"),
-        )
-        .where(CollectionTask.datasource_id.in_(datasource_ids))
-        .subquery()
-    )
-    result = await db.execute(
-        select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
-        .where(ranked_tasks.c.row_num == 1)
-    )
-    return {datasource_id: task_id for datasource_id, task_id in result.all()}
-
-
 async def _load_latest_tasks(
     db: AsyncSession,
     datasource_ids: list[int],
@@ -182,6 +165,8 @@ async def _load_latest_tasks(
 async def _load_collected_record_counts(
     db: AsyncSession,
     sources: list[str],
+    *,
+    exact_vessel_counts: bool = False,
 ) -> dict[str, int]:
     if not sources:
         return {}
@@ -202,14 +187,46 @@ async def _load_collected_record_counts(
         or "ais" in source
     ]
     if vessel_sources:
-        raw_result = await db.execute(
-            select(AISRawObservation.source, func.count(AISRawObservation.id))
-            .where(AISRawObservation.target_schema == VESSEL_AIS_SCHEMA)
-            .where(AISRawObservation.source.in_(vessel_sources))
-            .group_by(AISRawObservation.source)
+        if exact_vessel_counts:
+            exact_result = await db.execute(
+                select(AISRawObservation.source, func.count(AISRawObservation.id))
+                .where(AISRawObservation.source.in_(vessel_sources))
+                .group_by(AISRawObservation.source)
+            )
+            for source, count in exact_result.all():
+                counts[source] = max(counts.get(source, 0), int(count or 0))
+            return counts
+
+        # AIS raw observations can be tens of millions of rows. Use planner
+        # statistics for the datasource list instead of blocking page load on
+        # source-level count(*) scans.
+        stats_result = await db.execute(
+            text(
+                """
+                SELECT
+                    COALESCE(pg_class.reltuples, 0)::bigint AS total_rows,
+                    pg_stats.most_common_vals::text AS source_values,
+                    pg_stats.most_common_freqs::text AS source_freqs
+                FROM pg_class
+                LEFT JOIN pg_stats
+                  ON pg_stats.schemaname = 'public'
+                 AND pg_stats.tablename = 'ais_raw_observations'
+                 AND pg_stats.attname = 'source'
+                WHERE pg_class.relname = 'ais_raw_observations'
+                LIMIT 1
+                """
+            )
         )
-        for source, count in raw_result.all():
-            counts[source] = max(counts.get(source, 0), int(count or 0))
+        stats = stats_result.mappings().first()
+        if stats:
+            total_rows = int(stats["total_rows"] or 0)
+            values = str(stats["source_values"] or "").strip("{}")
+            freqs = str(stats["source_freqs"] or "").strip("{}")
+            source_values = [value.strip('"') for value in values.split(",") if value]
+            source_freqs = [float(value) for value in freqs.split(",") if value]
+            for source, freq in zip(source_values, source_freqs):
+                if source in vessel_sources:
+                    counts[source] = max(counts.get(source, 0), int(round(total_rows * freq)))
 
     return counts
 
@@ -303,8 +320,11 @@ def serialize_datasource_row(
         "last_run": to_iso8601_utc(last_run_at),
         "last_run_at": to_iso8601_utc(last_run_at),
         "last_status": last_status,
-        "is_running": running_task is not None,
+        "is_running": running_task is not None and running_task.task_type == JOB_TYPE_COLLECT,
+        "is_task_active": running_task is not None,
+        "task_status": running_task.status if running_task else None,
         "task_id": display_task.id if display_task else None,
+        "task_type": display_task.task_type if display_task else None,
         "progress": display_task.progress if display_task else None,
         "phase": display_task.phase if display_task else None,
         "phase_progress": display_task.phase_progress if display_task else None,
@@ -400,9 +420,25 @@ async def _trigger_datasource_batch(
     datasources: list[DataSource],
     *,
     force: bool,
+    actor_id: int | None = None,
+    trigger_kind: str = "batch",
 ) -> dict:
+    await emit_business_log(
+        logger,
+        event=f"collector.trigger.{trigger_kind}.start",
+        message="Datasource batch trigger started",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=actor_id,
+        context={
+            "trigger_kind": trigger_kind,
+            "force": force,
+            "requested_count": len(datasources),
+        },
+    )
     if not datasources:
-        return {
+        result = {
             "status": "noop",
             "message": "No matching data sources to trigger",
             "force": force,
@@ -410,8 +446,18 @@ async def _trigger_datasource_batch(
             "skipped": [],
             "failed": [],
         }
+        await emit_business_log(
+            logger,
+            event=f"collector.trigger.{trigger_kind}.completed",
+            message="Datasource batch trigger completed with no matching sources",
+            category="collector",
+            service="api",
+            module=__name__,
+            user_id=actor_id,
+            context={"trigger_kind": trigger_kind, "force": force, "status": "noop", "triggered_count": 0},
+        )
+        return result
 
-    previous_task_ids: dict[int, Optional[int]] = {}
     triggered_sources: list[dict] = []
     skipped_sources: list[dict] = []
     failed_sources: list[dict] = []
@@ -446,9 +492,11 @@ async def _trigger_datasource_batch(
                     }
                 )
                 continue
-            cancelled = await cancel_running_collector_now(datasource.source)
-            if not cancelled:
-                await rollback_orphaned_running_task(db, datasource, running_task)
+            await request_cancel_datasource_task(
+                db,
+                running_task,
+                reason="superseded_by_forced_collection",
+            )
 
         if not force and not is_due_for_collection(datasource, now):
             skipped_sources.append(
@@ -465,57 +513,51 @@ async def _trigger_datasource_batch(
             )
             continue
 
-        previous_task_ids[datasource.id] = None
-        success = run_collector_now(datasource.source)
-        if not success:
-            failed_sources.append(
-                {
-                    "id": datasource.id,
-                    "source": datasource.source,
-                    "name": datasource.name,
-                    "reason": "trigger_failed",
-                }
-            )
-            continue
+        task = await enqueue_datasource_job(
+            db,
+            datasource,
+            JOB_TYPE_COLLECT,
+            payload={"force": force, "trigger": "batch"},
+        )
 
         triggered_sources.append(
             {
                 "id": datasource.id,
                 "source": datasource.source,
                 "name": datasource.name,
-                "task_id": None,
+                "task_id": task.id,
             }
         )
 
-    latest_task_ids = await _load_latest_task_ids(
-        db,
-        [datasource.id for datasource in datasources],
-    )
-    for datasource_id in previous_task_ids:
-        previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
-
-    for _ in range(20):
-        await asyncio.sleep(0.1)
-        pending = [item for item in triggered_sources if item["task_id"] is None]
-        if not pending:
-            break
-        latest_task_ids = await _load_latest_task_ids(
-            db,
-            [item["id"] for item in pending],
-        )
-        for item in pending:
-            task_id = latest_task_ids.get(item["id"])
-            if task_id is not None and task_id != previous_task_ids.get(item["id"]):
-                item["task_id"] = task_id
-
-    return {
-        "status": "triggered" if triggered_sources else "partial",
-        "message": f"Triggered {len(triggered_sources)} data sources",
+    result = {
+        "status": "queued" if triggered_sources else "partial",
+        "message": f"Queued {len(triggered_sources)} data source jobs",
         "force": force,
         "triggered": triggered_sources,
         "skipped": skipped_sources,
         "failed": failed_sources,
     }
+    await emit_business_log(
+        logger,
+        event=f"collector.trigger.{trigger_kind}.completed",
+        message="Datasource batch trigger completed",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=actor_id,
+        context={
+            "trigger_kind": trigger_kind,
+            "force": force,
+            "status": result["status"],
+            "requested_count": len(datasources),
+            "triggered_count": len(triggered_sources),
+            "skipped_count": len(skipped_sources),
+            "failed_count": len(failed_sources),
+            "triggered_sources": [item["source"] for item in triggered_sources],
+            "skipped_reasons": [item["reason"] for item in skipped_sources],
+        },
+    )
+    return result
 
 
 async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
@@ -540,7 +582,7 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
     result = await db.execute(
         select(CollectionTask)
         .where(CollectionTask.datasource_id == datasource_id)
-        .where(CollectionTask.status == "running")
+        .where(CollectionTask.status == JobStatus.RUNNING.value)
         .order_by(CollectionTask.started_at.desc())
         .limit(1)
     )
@@ -568,8 +610,8 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
             f"Marked failed automatically after stale running timeout "
             f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
         )
-        task.status = "failed"
-        task.phase = "failed"
+        task.status = JobStatus.FAILED.value
+        task.phase = JobStatus.FAILED.value
         task.completed_at = now
         task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
         await db.commit()
@@ -606,7 +648,7 @@ async def rollback_orphaned_running_task(
     )
 
     if snapshot is not None:
-        snapshot.status = "cancelled"
+        snapshot.status = SnapshotStatus.CANCELLED.value
         snapshot.is_current = False
         snapshot.completed_at = datetime.now(timezone.utc)
         summary = dict(snapshot.summary or {})
@@ -629,13 +671,13 @@ async def rollback_orphaned_running_task(
                 {"snapshot_id": snapshot.parent_snapshot_id},
             )
 
-    running_task.status = "cancelled"
-    running_task.phase = "cancelled"
+    running_task.status = JobStatus.CANCELLED.value
+    running_task.phase = JobStatus.CANCELLED.value
     running_task.completed_at = datetime.now(timezone.utc)
     existing_error = (running_task.error_message or "").strip()
     cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
     running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
-    datasource.last_status = "cancelled"
+    datasource.last_status = JobStatus.CANCELLED.value
     datasource.last_run_at = datetime.now(timezone.utc)
     await db.commit()
 
@@ -670,7 +712,7 @@ async def fail_and_rollback_stale_running_task(
     )
 
     if snapshot is not None:
-        snapshot.status = "failed"
+        snapshot.status = SnapshotStatus.FAILED.value
         snapshot.is_current = False
         snapshot.completed_at = datetime.now(timezone.utc)
         summary = dict(snapshot.summary or {})
@@ -698,11 +740,11 @@ async def fail_and_rollback_stale_running_task(
         f"Marked failed automatically after stale running timeout "
         f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
     )
-    running_task.status = "failed"
-    running_task.phase = "failed"
+    running_task.status = JobStatus.FAILED.value
+    running_task.phase = JobStatus.FAILED.value
     running_task.completed_at = datetime.now(timezone.utc)
     running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
-    datasource.last_status = "failed"
+    datasource.last_status = JobStatus.FAILED.value
     datasource.last_run_at = datetime.now(timezone.utc)
     await db.commit()
 
@@ -780,7 +822,13 @@ async def trigger_all_datasources(
         .order_by(DataSource.module, DataSource.id)
     )
     datasources = result.scalars().all()
-    return await _trigger_datasource_batch(db, datasources, force=force)
+    return await _trigger_datasource_batch(
+        db,
+        datasources,
+        force=force,
+        actor_id=current_user.id,
+        trigger_kind="all",
+    )
 
 
 @router.post("/trigger-batch")
@@ -816,7 +864,13 @@ async def trigger_datasource_batch(
         collected=None if payload.source_ids else payload.collected,
         credential_status=None if payload.source_ids else payload.credential_status,
     )
-    return await _trigger_datasource_batch(db, datasources, force=payload.force)
+    return await _trigger_datasource_batch(
+        db,
+        datasources,
+        force=payload.force,
+        actor_id=current_user.id,
+        trigger_kind="batch",
+    )
 
 
 @router.get("/snapshots")
@@ -909,7 +963,7 @@ async def get_datasource_row(
         [datasource],
         include_endpoint=include_endpoint,
     )
-    record_counts = await _load_collected_record_counts(db, [datasource.source])
+    record_counts = await _load_collected_record_counts(db, [datasource.source], exact_vessel_counts=True)
     return {
         "data": serialize_datasource_row(
             datasource,
@@ -992,8 +1046,24 @@ async def trigger_datasource(
     if not datasource.is_active:
         raise HTTPException(status_code=400, detail="Data source is disabled")
 
-    running_task = await get_running_task(db, datasource.id)
+    running_task = await get_active_datasource_job(db, datasource.id, task_types=(JOB_TYPE_COLLECT,))
     if running_task is not None and not force:
+        await emit_business_log(
+            logger,
+            event="collector.trigger.single.skipped_already_running",
+            message="Datasource trigger skipped because a task is already running",
+            category="collector",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={
+                "collector_name": datasource.source,
+                "datasource_id": datasource.id,
+                "task_id": running_task.id,
+                "status": "skipped",
+            },
+        )
         raise HTTPException(
             status_code=409,
             detail={
@@ -1013,31 +1083,42 @@ async def trigger_datasource(
         )
 
     if running_task is not None and force:
-        cancelled = await cancel_running_collector_now(datasource.source)
-        if not cancelled:
-            await rollback_orphaned_running_task(db, datasource, running_task)
+        await request_cancel_datasource_task(
+            db,
+            running_task,
+            reason="superseded_by_forced_collection",
+        )
 
-    previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
-    success = run_collector_now(datasource.source)
-    if not success:
-        raise HTTPException(status_code=500, detail=f"Failed to trigger collector '{datasource.source}'")
-
-    task_id = None
-    for _ in range(20):
-        await asyncio.sleep(0.1)
-        task_id = await get_latest_task_id_for_datasource(datasource.id)
-        if task_id is not None and task_id != previous_task_id:
-            break
-    if task_id == previous_task_id:
-        task_id = None
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_COLLECT,
+        payload={"force": force, "trigger": "single"},
+    )
+    await emit_business_log(
+        logger,
+        event="collector.trigger.single.completed",
+        message="Datasource trigger queued",
+        category="collector",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={
+            "collector_name": datasource.source,
+            "datasource_id": datasource.id,
+            "task_id": task.id,
+            "force": force,
+            "status": "queued",
+        },
+    )
 
     return {
-        "status": "triggered",
+        "status": "queued",
         "source_id": datasource.id,
-        "task_id": task_id,
+        "task_id": task.id,
         "collector_name": datasource.source,
         "force": force,
-        "message": f"Collector '{datasource.source}' has been triggered",
+        "message": f"Collector '{datasource.source}' has been queued",
     }
 
 
@@ -1051,22 +1132,30 @@ async def clear_datasource_data(
     if not datasource:
         raise HTTPException(status_code=404, detail="Data source not found")
 
-    result = await db.execute(
-        select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
+    active_task = await get_active_datasource_job(db, datasource.id)
+    if active_task is not None:
+        raise HTTPException(
+            status_code=409,
+            detail={
+                "reason": "datasource_job_in_progress",
+                "message": "当前数据源已有任务在执行，请等待完成或先取消任务。",
+                "task_id": active_task.id,
+                "task_type": active_task.task_type,
+                "status": active_task.status,
+            },
+        )
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_CLEAR_DATA,
+        payload={"source": datasource.source},
     )
-    count = result.scalar() or 0
-
-    if count == 0:
-        return {"status": "success", "message": "No data to clear", "deleted_count": 0}
-
-    delete_query = CollectedData.__table__.delete().where(CollectedData.source == datasource.source)
-    await db.execute(delete_query)
-    await db.commit()
 
     return {
-        "status": "success",
-        "message": f"Cleared {count} records for data source '{datasource.name}'",
-        "deleted_count": count,
+        "status": "queued",
+        "message": f"Queued data clearing for data source '{datasource.name}'",
+        "task_id": task.id,
+        "deleted_count": None,
     }
 
 
@@ -1080,16 +1169,44 @@ async def clear_datasource_cache(
     if not datasource:
         raise HTTPException(status_code=404, detail="Data source not found")
 
-    earth_deleted_count = invalidate_earth_layer_cache_for_source(datasource.source)
-    dashboard_deleted_count = int(cache.delete("dashboard:stats")) + int(cache.delete("dashboard:summary"))
-    deleted_count = earth_deleted_count + dashboard_deleted_count
+    task = await enqueue_datasource_job(
+        db,
+        datasource,
+        JOB_TYPE_CLEAR_CACHE,
+        payload={"source": datasource.source},
+        dedupe_key=f"clear_cache:{datasource.source}",
+    )
 
     return {
-        "status": "success",
-        "message": f"Cleared {deleted_count} cache keys for data source '{datasource.name}'",
-        "deleted_count": deleted_count,
-        "earth_layer_deleted_count": earth_deleted_count,
-        "dashboard_deleted_count": dashboard_deleted_count,
+        "status": "queued",
+        "message": f"Queued cache clearing for data source '{datasource.name}'",
+        "task_id": task.id,
+        "deleted_count": None,
+    }
+
+
+@router.post("/{source_id}/tasks/{task_id}/cancel")
+async def cancel_datasource_task(
+    source_id: str,
+    task_id: int,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    datasource = await get_datasource_record(db, source_id)
+    if not datasource:
+        raise HTTPException(status_code=404, detail="Data source not found")
+
+    task = await db.get(CollectionTask, task_id)
+    if not task or task.datasource_id != datasource.id:
+        raise HTTPException(status_code=404, detail="Task not found")
+
+    task = await request_cancel_datasource_task(db, task)
+    return {
+        "status": "cancelled" if task.completed_at else "cancelling",
+        "task_id": task.id,
+        "task_type": task.task_type,
+        "phase": task.phase,
+        "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
     }
 
 
@@ -1109,7 +1226,7 @@ async def get_task_status(
         if not task or task.datasource_id != datasource.id:
             raise HTTPException(status_code=404, detail="Task not found")
     else:
-        task = await get_running_task(db, datasource.id)
+        task = await get_active_datasource_job(db, datasource.id)
         if task is None:
             result = await db.execute(
                 select(CollectionTask)
@@ -1134,8 +1251,12 @@ async def get_task_status(
         }
 
     return {
-        "is_running": task.status == "running",
+        "is_running": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING}
+        and task.task_type == JOB_TYPE_COLLECT,
+        "is_task_active": task.status in {JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING},
         "task_id": task.id,
+        "task_type": task.task_type,
+        "task_status": task.status,
         "progress": task.progress,
         "phase": task.phase,
         "phase_progress": task.phase_progress,
@@ -1146,5 +1267,6 @@ async def get_task_status(
         "records_processed": task.records_processed,
         "total_records": task.total_records,
         "status": task.status,
+        "requested_cancel_at": to_iso8601_utc(task.requested_cancel_at),
         "error_message": task.error_message,
     }

backend/app/api/v1/earth.py

@@ -6,12 +6,13 @@ from pathlib import Path
 from typing import Any
 from uuid import uuid4
 
-from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
+from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from pydantic import BaseModel, Field
 from sqlalchemy import delete, func, select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.config import settings as app_settings
 from app.core.security import decode_token, get_current_user, redis_client
 from app.db.session import get_db
 from app.models.collected_data import CollectedData
@@ -20,6 +21,26 @@ from app.models.datasource_config import DataSourceConfig
 from app.models.system_setting import SystemSetting
 from app.models.user import User
 from app.services.tv_streams import get_tv_settings_payload
+from app.services.earth_news import (
+    get_earth_news_sources_payload,
+    reset_earth_news_sources_payload,
+    save_earth_news_sources_payload,
+    test_news_source_config,
+)
+from app.services.earth_news_manual import (
+    broadcast_manual_news_changed,
+    create_manual_news_group,
+    delete_manual_news_item,
+    get_news_record_or_404,
+    import_manual_news_items,
+    list_news_groups,
+    list_news_records,
+    parse_manual_news_import_upload,
+    rename_manual_news_group,
+    reprocess_manual_news_item,
+    serialize_news_record,
+    upsert_manual_news_item,
+)
 from app.services.earth_boundaries import (
     EarthBoundaryBuildError,
     get_boundary_build_status,
@@ -36,9 +57,16 @@ EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
 EARTH_BRAND_ASSET_URL_PREFIX = "/earth-brand-assets"
 EARTH_BRAND_CATEGORY = "earth_brand"
 EARTH_ABOUT_CATEGORY = "earth_about"
+SYSTEM_SETTINGS_CATEGORY = "system"
 MAX_EARTH_BRAND_ASSET_BYTES = 3 * 1024 * 1024
 ALLOWED_EARTH_BRAND_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".svg"}
 
+
+def _app_version_label() -> str:
+    version = str(app_settings.VERSION or "").strip() or "0.0.0"
+    return version if version.startswith("v") else f"v{version}"
+
+
 DEFAULT_EARTH_BRAND = {
     "logo_src": "/earth/assets/brand/earth-logo.png",
     "title_src": "/earth/assets/brand/title-zh.png",
@@ -53,14 +81,15 @@ DEFAULT_EARTH_ABOUT = {
     "logo_src": "/earth/assets/brand/lim-logo.png",
     "kicker": "About",
     "title": "智能星球计划",
-    "version": "v0.64.0",
+    "version": _app_version_label(),
     "description": "面向临空场景下的智能媒体研究、全球态势感知与多源开放数据巡航，提供可视化观测、事件聚合与交互式探索能力。",
     "meta": [
         {"label": "出品方", "value": "浙江大学临空智能媒体研究院"},
-        {"label": "策划人", "value": "黄柳青"},
+        {"label": "策划人", "value": "方兴东、黄柳青"},
         {"label": "产品兼开发者", "value": "钱坤、张鸽、齐鹏"},
     ],
 }
+EARTH_ABOUT_LEGACY_PLANNER_VALUE = "黄柳青"
 
 
 class EarthBoundaryConfigPayload(BaseModel):
@@ -91,6 +120,39 @@ class EarthAboutPayload(BaseModel):
     meta: list[EarthAboutMetaItem] = Field(default_factory=list)
 
 
+class EarthNewsSourcesPayload(BaseModel):
+    cache_version: int | None = None
+    source_tags: list[dict[str, Any]] = Field(default_factory=list)
+    categories: list[dict[str, Any]] = Field(default_factory=list)
+    item_tag_rules: list[dict[str, Any]] = Field(default_factory=list)
+    sources: list[dict[str, Any]] = Field(default_factory=list)
+    health: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsSourceTestPayload(BaseModel):
+    source: dict[str, Any] = Field(default_factory=dict)
+
+
+class EarthNewsManualItemPayload(BaseModel):
+    title: str = Field(default="", max_length=500)
+    summary: str = Field(default="", max_length=1200)
+    content: str = Field(default="", max_length=12000)
+    url: str = Field(default="", max_length=2000)
+    source: str = Field(default="", max_length=255)
+    region: str = Field(default="global", max_length=80)
+    published_at: str | None = None
+    category: str = Field(default="other", max_length=80)
+    tags: list[str] = Field(default_factory=list)
+    location: dict[str, Any] | None = None
+    homepage_url: str = Field(default="", max_length=2000)
+    content_language: str = Field(default="", max_length=32)
+    group_id: str | None = Field(default=None, max_length=120)
+
+
+class EarthNewsManualGroupPayload(BaseModel):
+    name: str = Field(default="", max_length=120)
+
+
 def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
     merged = DEFAULT_EARTH_BRAND.copy()
     if payload:
@@ -116,11 +178,12 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
     }
     raw_meta = DEFAULT_EARTH_ABOUT["meta"]
     if payload:
-        for key in ("logo_src", "kicker", "title", "version", "description"):
+        for key in ("logo_src", "kicker", "title", "description"):
             value = payload.get(key)
             if value is not None:
                 merged[key] = str(value).strip()
         raw_meta = payload.get("meta") if isinstance(payload.get("meta"), list) else raw_meta
+    merged["version"] = _app_version_label()
 
     for key, default_value in DEFAULT_EARTH_ABOUT.items():
         if key == "meta":
@@ -134,6 +197,8 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
             continue
         label = str(item.get("label") or "").strip()
         value = str(item.get("value") or "").strip()
+        if label == "策划人" and value == EARTH_ABOUT_LEGACY_PLANNER_VALUE:
+            value = "方兴东、黄柳青"
         if label or value:
             normalized_meta.append({"label": label, "value": value})
     if not normalized_meta:
@@ -142,6 +207,10 @@ def _normalize_earth_about_payload(payload: dict[str, Any] | None) -> dict[str,
     return merged
 
 
+def _is_demo_mode_enabled(payload: Any) -> bool:
+    return bool(payload.get("demo_mode")) if isinstance(payload, dict) else False
+
+
 async def _get_earth_brand_record(db: AsyncSession) -> SystemSetting | None:
     result = await db.execute(
         select(SystemSetting).where(SystemSetting.category == EARTH_BRAND_CATEGORY)
@@ -308,6 +377,194 @@ async def reset_earth_about(
     return {"status": "reset", "about": _normalize_earth_about_payload(None), "is_default": True}
 
 
+@router.get("/news-sources")
+async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
+    return await get_earth_news_sources_payload(db)
+
+
+@router.put("/news-sources")
+async def update_earth_news_sources(
+    payload: EarthNewsSourcesPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await save_earth_news_sources_payload(db, payload.model_dump())
+
+
+@router.delete("/news-sources")
+@router.post("/news-sources/reset")
+async def reset_earth_news_sources(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await reset_earth_news_sources_payload(db)
+
+
+@router.post("/news-sources/test")
+async def test_earth_news_source(
+    payload: EarthNewsSourceTestPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await test_news_source_config(payload.source, db=db)
+
+
+@router.get("/news-groups")
+async def list_earth_news_groups_admin(
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_groups(db)
+
+
+@router.post("/news-groups")
+async def create_earth_news_group_admin(
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await create_manual_news_group(db, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    return {"status": "ok", "group": group}
+
+
+@router.put("/news-groups/{group_id:path}")
+async def rename_earth_news_group_admin(
+    group_id: str,
+    payload: EarthNewsManualGroupPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        group = await rename_manual_news_group(db, group_id, payload.name)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "group": group}
+
+
+@router.get("/news-items")
+async def list_earth_news_items_admin(
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=100),
+    source_type: str | None = Query(None),
+    region: str | None = Query(None),
+    category: str | None = Query(None),
+    status_filter: str | None = Query(None, alias="status"),
+    group_id: str | None = Query(None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    return await list_news_records(
+        db,
+        page=page,
+        page_size=page_size,
+        source_type=source_type,
+        region=region,
+        category=category,
+        status_filter=status_filter,
+        group_id=group_id,
+    )
+
+
+@router.post("/news-items")
+async def create_earth_news_item_admin(
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        result = await upsert_manual_news_item(db, payload.model_dump())
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.post("/news-items/import")
+async def import_earth_news_items_admin(
+    file: UploadFile = File(...),
+    group_id: str | None = Form(default=None),
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        payload = await parse_manual_news_import_upload(await file.read())
+        result = await import_manual_news_items(db, payload, group_id=group_id)
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", **result}
+
+
+@router.put("/news-items/{item_id:path}")
+async def update_earth_news_item_admin(
+    item_id: str,
+    payload: EarthNewsManualItemPayload,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        result = await upsert_manual_news_item(
+            db,
+            payload.model_dump(),
+            item_id_override=item_id,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
+
+
+@router.delete("/news-items/{item_id:path}")
+async def delete_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    try:
+        deleted = await delete_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    if not deleted:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "deleted", "id": item_id}
+
+
+@router.post("/news-items/{item_id:path}/reprocess")
+async def reprocess_earth_news_item_admin(
+    item_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    existing = await get_news_record_or_404(db, item_id)
+    if existing is None:
+        raise HTTPException(status_code=404, detail="News item not found.")
+    try:
+        queued = await reprocess_manual_news_item(db, item_id)
+    except PermissionError as exc:
+        raise HTTPException(status_code=403, detail=str(exc)) from exc
+    await db.commit()
+    await broadcast_manual_news_changed()
+    return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
+
+
 @router.get("/oobe-status")
 async def get_earth_oobe_status(
     current_user: User | None = Depends(_get_optional_current_user),
@@ -317,6 +574,11 @@ async def get_earth_oobe_status(
         select(func.count(CollectedData.id)).where(CollectedData.is_current.is_(True))
     )
     current_record_count = int(current_count_result.scalar() or 0)
+    system_result = await db.execute(
+        select(SystemSetting).where(SystemSetting.category == SYSTEM_SETTINGS_CATEGORY)
+    )
+    system_record = system_result.scalar_one_or_none()
+    demo_mode = _is_demo_mode_enabled(system_record.payload if system_record else None)
 
     datasource_count_result = await db.execute(select(func.count(DataSource.id)))
     datasource_count = int(datasource_count_result.scalar() or 0)
@@ -337,6 +599,8 @@ async def get_earth_oobe_status(
     ready = has_collected_data
 
     suggestions: list[str] = []
+    if demo_mode:
+        suggestions.append("演示模式已开启")
     if not current_user:
         suggestions.append("登录控制台")
     if not has_collected_data:
@@ -348,8 +612,9 @@ async def get_earth_oobe_status(
 
     return {
         "ready": ready,
+        "demo_mode": demo_mode,
         "authenticated": current_user is not None,
-        "needs_login": current_user is None and not ready,
+        "needs_login": current_user is None and not ready and not demo_mode,
         "has_collected_data": has_collected_data,
         "has_tv_sources": tv_source_count > 0,
         "has_core_layers": has_core_layers,

backend/app/api/v1/interactables.py

@@ -0,0 +1,190 @@
+"""CRUD APIs for persistent Earth interactables."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Response, status
+from pydantic import BaseModel, Field, field_validator
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.earth_interactable import EarthInteractable
+from app.models.user import User
+from app.services.earth_interactables import (
+    build_interactable_event,
+    interactables_to_geojson,
+    invalidate_interactable_cache,
+    list_interactables,
+    normalize_interactable_id,
+    publish_interactable_event,
+    serialize_interactable,
+)
+from app.services.earth_layer_cache import (
+    EarthLayerCachePolicy,
+    earth_layer_cache,
+    get_or_build_layer_payload,
+)
+
+router = APIRouter()
+INTERACTABLE_CACHE_POLICY = EarthLayerCachePolicy(
+    fresh_ttl_seconds=60,
+    stale_ttl_seconds=10 * 60,
+    max_features=5000,
+)
+
+
+class InteractableCreate(BaseModel):
+    id: str | None = Field(default=None, max_length=160)
+    layer: str = Field(default="default", min_length=1, max_length=80)
+    kind: str = Field(default="default", min_length=1, max_length=80)
+    label: str = Field(default="", max_length=255)
+    description: str = Field(default="", max_length=4000)
+    latitude: float = Field(ge=-90, le=90)
+    longitude: float = Field(ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] = Field(default_factory=dict)
+
+    @field_validator("layer", "kind")
+    @classmethod
+    def normalize_key(cls, value: str) -> str:
+        normalized = str(value or "").strip()
+        if not normalized:
+            raise ValueError("must not be empty")
+        return normalized
+
+
+class InteractableUpdate(BaseModel):
+    layer: str | None = Field(default=None, min_length=1, max_length=80)
+    kind: str | None = Field(default=None, min_length=1, max_length=80)
+    label: str | None = Field(default=None, max_length=255)
+    description: str | None = Field(default=None, max_length=4000)
+    latitude: float | None = Field(default=None, ge=-90, le=90)
+    longitude: float | None = Field(default=None, ge=-180, le=180)
+    altitude: float | None = None
+    properties: dict[str, Any] | None = None
+
+
+@router.get("")
+async def get_interactables(
+    response: Response,
+    layer: str | None = Query(default=None),
+    include_deleted: bool = Query(default=False),
+    db: AsyncSession = Depends(get_db),
+):
+    items = await list_interactables(db, layer=layer, include_deleted=include_deleted)
+    response.headers["X-Planet-Interactables-Count"] = str(len(items))
+    return {"items": [serialize_interactable(item) for item in items]}
+
+
+@router.get("/geojson")
+async def get_interactables_geojson(
+    response: Response,
+    layer: str | None = Query(default=None),
+    db: AsyncSession = Depends(get_db),
+):
+    async def build_payload() -> dict[str, Any]:
+        items = await list_interactables(db, layer=layer)
+        return interactables_to_geojson(items)
+
+    payload = await get_or_build_layer_payload(
+        key=earth_layer_cache.key("interactables", interactable_layer=layer or "all"),
+        policy=INTERACTABLE_CACHE_POLICY,
+        builder=build_payload,
+        response=response,
+    )
+    response.headers["X-Planet-Interactables-Count"] = str(len(payload.get("features") or []))
+    return payload
+
+
+@router.post("", status_code=status.HTTP_201_CREATED)
+async def create_interactable(
+    payload: InteractableCreate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record_id = normalize_interactable_id(payload.id)
+    existing = await db.get(EarthInteractable, record_id)
+    if existing and not existing.is_deleted:
+        raise HTTPException(status_code=409, detail="Interactable already exists")
+
+    if existing is None:
+        record = EarthInteractable(id=record_id)
+        db.add(record)
+    else:
+        record = existing
+        record.is_deleted = False
+        record.deleted_at = None
+        record.revision += 1
+
+    record.layer = payload.layer
+    record.kind = payload.kind
+    record.label = payload.label
+    record.description = payload.description
+    record.latitude = payload.latitude
+    record.longitude = payload.longitude
+    record.altitude = payload.altitude
+    record.properties = payload.properties
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("created", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.get("/{interactable_id}")
+async def get_interactable(interactable_id: str, db: AsyncSession = Depends(get_db)):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+    return {"item": serialize_interactable(record)}
+
+
+@router.patch("/{interactable_id}")
+async def update_interactable(
+    interactable_id: str,
+    payload: InteractableUpdate,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    previous_layer = record.layer
+    patch = payload.model_dump(exclude_unset=True)
+    for key, value in patch.items():
+        setattr(record, key, value)
+    record.revision += 1
+
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(previous_layer)
+    if record.layer != previous_layer:
+        invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("updated", record)
+    return {"item": serialize_interactable(record)}
+
+
+@router.delete("/{interactable_id}")
+async def delete_interactable(
+    interactable_id: str,
+    _current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    record = await db.get(EarthInteractable, interactable_id)
+    if record is None or record.is_deleted:
+        raise HTTPException(status_code=404, detail="Interactable not found")
+
+    record.is_deleted = True
+    record.deleted_at = datetime.now(UTC)
+    record.revision += 1
+    await db.commit()
+    await db.refresh(record)
+    invalidate_interactable_cache(record.layer)
+    await publish_interactable_event("deleted", record)
+    event = build_interactable_event(action="deleted", record=record, include_item=True)
+    return {"deleted": True, "event": event}

backend/app/api/v1/layers.py

@@ -105,7 +105,7 @@ def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]:
 @router.get("/vessels/snapshot")
 async def get_vessel_layer_snapshot(
     bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
-    zoom: int = Query(..., ge=1, le=20),
+    zoom: float = Query(..., ge=1, le=20),
     limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
     vessel_type: Optional[str] = Query(None, alias="type"),
     since_minutes: int = Query(60, ge=1, le=1440),

backend/app/api/v1/news.py

@@ -1,16 +1,92 @@
-from fastapi import APIRouter, Depends, Query
+from fastapi import APIRouter, Depends, HTTPException, Query
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
-from app.services.earth_news import get_earth_news_payload
+from app.services.earth_news import (
+    ALLOWED_NEWS_CATEGORY_KEYS,
+    SUPPORTED_NEWS_LOCALES,
+    REGION_ANCHORS,
+    get_earth_news_payload,
+)
 
 router = APIRouter()
 
 
+def _parse_categories(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    requested = {item.strip().lower() for item in raw.split(",") if item.strip()}
+    invalid = sorted(requested - set(ALLOWED_NEWS_CATEGORY_KEYS))
+    if invalid:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news categories.",
+                "invalid_categories": invalid,
+                "allowed_categories": list(ALLOWED_NEWS_CATEGORY_KEYS),
+            },
+        )
+    return requested or None
+
+
+def _parse_source_ids(raw: str | None) -> set[str] | None:
+    if raw is None or not raw.strip():
+        return None
+    return {item.strip() for item in raw.split(",") if item.strip()} or None
+
+
+def _parse_limit(raw: int | None) -> int:
+    if raw is None:
+        return 12
+    if raw < 1:
+        raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
+    return min(raw, 100)
+
+
+def _parse_locale(raw: str | None) -> str:
+    if raw is None or not raw.strip():
+        return "zh-CN"
+    requested = raw.strip()
+    if requested not in SUPPORTED_NEWS_LOCALES:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news locale.",
+                "invalid_locale": requested,
+                "allowed_locales": sorted(SUPPORTED_NEWS_LOCALES),
+            },
+        )
+    return requested
+
+
 @router.get("/earth-feed")
 async def get_earth_feed(
     lat: float | None = Query(None, description="Current Earth view center latitude"),
     lon: float | None = Query(None, description="Current Earth view center longitude"),
+    region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
+    categories: str | None = Query(None, description="Comma-separated news category keys"),
+    sources: str | None = Query(None, description="Comma-separated news source ids"),
+    limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
+    locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
     db: AsyncSession = Depends(get_db),
 ):
-    return await get_earth_news_payload(lat=lat, lon=lon, db=db)
+    normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
+    if normalized_region is not None and normalized_region not in REGION_ANCHORS:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news region.",
+                "invalid_region": normalized_region,
+                "allowed_regions": list(REGION_ANCHORS.keys()),
+            },
+        )
+    return await get_earth_news_payload(
+        lat=lat,
+        lon=lon,
+        region=normalized_region,
+        categories=_parse_categories(categories),
+        source_ids=_parse_source_ids(sources),
+        limit=_parse_limit(limit),
+        locale=_parse_locale(locale),
+        db=db,
+    )

backend/app/api/v1/settings.py

@@ -3,14 +3,16 @@ from datetime import UTC, datetime
 import os
 from pathlib import Path
 from typing import Optional
+from urllib.parse import urlsplit
 
 from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
-import httpx
 from pydantic import BaseModel, EmailStr, Field
 from dotenv import dotenv_values
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.logging import get_logger
+from app.core.enums import ProviderApi, TVSourceType, UserRole
 from app.core.security import get_current_user
 from app.core.time import to_iso8601_utc
 from app.core.config import settings as app_settings
@@ -63,14 +65,18 @@ from app.services.llm_provider_catalog import (
     list_fallback_llm_provider_presets,
     refresh_llm_provider_preset,
 )
+from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
 from app.services.scheduler import sync_datasource_job
 from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
 from app.services.persistent_logs import record_audit_log
+from app.services.business_logs import emit_business_log, exception_context
 
 router = APIRouter()
+logger = get_logger(__name__, service="api")
 AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
 AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
-SECRET_REVEAL_ROLES = {"admin", "super_admin"}
+SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
+LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
 
 DEFAULT_SETTINGS = {
     "system": {
@@ -79,6 +85,7 @@ DEFAULT_SETTINGS = {
         "auto_refresh": True,
         "data_retention_days": 30,
         "max_concurrent_tasks": 5,
+        "demo_mode": False,
     },
     "notifications": {
         "email_enabled": False,
@@ -203,6 +210,7 @@ class SystemSettingsUpdate(BaseModel):
     auto_refresh: bool = True
     data_retention_days: int = Field(default=30, ge=1, le=3650)
     max_concurrent_tasks: int = Field(default=5, ge=1, le=50)
+    demo_mode: bool = False
 
 
 class NotificationSettingsUpdate(BaseModel):
@@ -231,7 +239,7 @@ class TVStreamSourceUpdate(BaseModel):
     provider: str = Field(default="Unknown", max_length=100)
     region: str = Field(default="Global", max_length=100)
     language: str = Field(default="und", max_length=32)
-    source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
+    source_type: TVSourceType = TVSourceType.IFRAME
     embed_url: str = ""
     stream_url: str = ""
     homepage_url: str = ""
@@ -274,7 +282,7 @@ class AIProviderIntegrationUpdate(BaseModel):
     service_token: Optional[str] = None
     default_provider: Optional[str] = None
     provider: str = Field(default="minimax", max_length=80)
-    provider_api: str = Field(default="anthropic-messages", max_length=80)
+    provider_api: ProviderApi = ProviderApi.ANTHROPIC_MESSAGES
     base_url: str = Field(default="", max_length=500)
     model: str = Field(default="", max_length=200)
     api_key: Optional[str] = None
@@ -339,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
 
 
 def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
-    merged = deepcopy(DEFAULT_SETTINGS[category])
+    if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
+        defaults = get_fallback_llm_provider_preset(
+            category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
+        )
+    else:
+        defaults = DEFAULT_SETTINGS[category]
+    merged = deepcopy(defaults)
     if payload:
         merged.update(payload)
     return merged
@@ -376,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
 
 
 async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
+    merged = merge_with_defaults(category, payload)
     record = await get_setting_record(db, category)
     if record is None:
         record = SystemSetting(category=category, payload=payload)
@@ -385,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
 
     await db.commit()
     await db.refresh(record)
-    return merge_with_defaults(category, record.payload)
+    return merged
 
 
 AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
@@ -418,7 +433,7 @@ def _get_provider_preset(provider: str) -> dict:
     except ValueError:
         return {
             "provider": provider,
-            "provider_api": "openai-completions",
+            "provider_api": ProviderApi.OPENAI_COMPLETIONS.value,
             "base_url": "",
         "model": "",
         "models": [],
@@ -482,12 +497,12 @@ def _provider_defaults(provider: str) -> dict:
     preset = _get_provider_preset(provider)
     return {
         "provider": provider,
-        "provider_api": preset.get("provider_api") or "openai-completions",
+        "provider_api": preset.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
         "base_url": preset.get("base_url") or "",
         "model": preset.get("model") or "",
         "api_key": "",
         "max_tokens": (
-            1200 if preset.get("provider_api") == "anthropic-messages" else 4096
+            1200 if preset.get("provider_api") == ProviderApi.ANTHROPIC_MESSAGES.value else 4096
         ),
         "anthropic_version": "2023-06-01",
         "model_provider_apis": preset.get("model_provider_apis") or {},
@@ -649,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
         normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
     )
     api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
+    model_provider_apis = {
+        **(provider_config.get("model_provider_apis") or {}),
+        **(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
+    }
+    if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
+        model_provider_apis = {}
     return {
         "service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
         "service_token": _resolve_service_token(normalized_ai)[0],
@@ -660,14 +681,13 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
         ),
         "llm_config": {
             "provider": default_provider,
-            "provider_api": provider_config.get("provider_api") or "anthropic-messages",
+            "provider_api": provider_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
             "base_url": provider_config.get("base_url") or "",
             "model": provider_config.get("model") or "",
             "api_key": api_key,
             "max_tokens": int(provider_config.get("max_tokens") or 1200),
             "anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
-            "model_provider_apis": provider_config.get("model_provider_apis") or {},
-            "preset_models": _get_provider_preset(default_provider).get("models") or [],
+            "model_provider_apis": model_provider_apis,
         },
     }
 
@@ -699,6 +719,18 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
         retry_attempts=runtime_config["retry_attempts"],
         llm_config=runtime_config.get("llm_config") or {},
     )
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.full_connection.start",
+        message="AI provider full connection validation started",
+        category="ai",
+        service="api",
+        module=__name__,
+        context={
+            "provider": runtime_config.get("llm_config", {}).get("provider"),
+            "model": runtime_config.get("llm_config", {}).get("model"),
+        },
+    )
     status_result = await client.get_status()
     if not status_result.configured:
         raise HTTPException(
@@ -715,6 +747,19 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
             constraints=["回复尽量简短。"],
         )
     )
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.full_connection.success",
+        message="AI provider full connection validation completed",
+        category="ai",
+        service="api",
+        module=__name__,
+        context={
+            "provider": analysis_result.provider,
+            "model": analysis_result.model,
+            "configured": status_result.configured,
+        },
+    )
     return {
         "status": status_result.model_dump(),
         "provider": analysis_result.provider,
@@ -722,156 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
     }
 
 
-def _join_provider_url(base_url: str, path: str) -> str:
-    return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
-
-
-def _extract_model_ids(payload: dict) -> list[str]:
-    data = payload.get("data") if isinstance(payload, dict) else None
-    if isinstance(data, list):
-        return [
-            str(item.get("id"))
-            for item in data
-            if isinstance(item, dict) and item.get("id")
-        ]
-    models = payload.get("models") if isinstance(payload, dict) else None
-    if isinstance(models, list):
-        return [
-            str(item.get("name") or item.get("model") or item.get("id") or item)
-            for item in models
-            if item
-        ]
-    return []
-
-
-def _contains_model(model_ids: list[str], model: str) -> bool:
-    normalized_model = model.strip().lower()
-    return any(str(item).strip().lower() == normalized_model for item in model_ids)
-
-
 async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
     provider = _normalize_provider_id(llm_config.get("provider") or "")
-    configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
+    provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
     model = str(llm_config.get("model") or "").strip()
-    base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
+    base_url = str(llm_config.get("base_url") or "").strip()
     api_key = str(llm_config.get("api_key") or "").strip()
-    provider_api = configured_api
-    model_provider_apis = llm_config.get("model_provider_apis")
-    if isinstance(model_provider_apis, dict):
-        provider_api = str(model_provider_apis.get(model) or provider_api)
-    preset_models = [
-        str(item)
-        for item in (llm_config.get("preset_models") or [])
-        if str(item).strip()
-    ]
-
-    if not provider or not base_url or not model:
-        return {
-            "success": False,
-            "connected": False,
-            "message": "当前 provider/base_url/model 未完整配置。",
-            "mode": "lightweight_config",
-        }
-    if provider_api != "ollama-generate" and not api_key:
-        return {
-            "success": False,
-            "connected": False,
-            "message": "当前 provider 未配置 API Key。",
-            "mode": "lightweight_config",
-        }
-
-    if provider == "opencode-go":
-        url = _join_provider_url(base_url, "/models")
-        headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == "ollama-generate":
-        url = _join_provider_url(base_url, "/api/tags")
-        headers: dict[str, str] = {}
-    elif provider_api == "openai-completions":
-        url = _join_provider_url(base_url, "/models")
-        headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == "anthropic-messages":
-        url = _join_provider_url(base_url, "/models")
-        headers = {
-            "x-api-key": api_key,
-            "anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
-        }
-    else:
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
-            "mode": "lightweight_unsupported",
-        }
-
+    result = {
+        "success": False, "connected": False, "mode": "lightweight_models",
+        "provider": provider, "model": model,
+    }
+    if not base_url or not model:
+        return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
+    if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
+        return {**result, "message": "当前 provider 未配置 API Key。"}
     try:
-        async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
-            response = await client.get(url, headers=headers)
-            response.raise_for_status()
-            payload = response.json()
-    except httpx.HTTPStatusError as exc:
-        detail = exc.response.text or exc.response.reason_phrase
-        if exc.response.status_code == 404 and _contains_model(preset_models, model):
-            return {
-                "success": True,
-                "connected": True,
-                "message": "轻量连通性测试通过；当前 provider 不提供可用的模型目录，已按内置模型预设确认。",
-                "mode": "lightweight_preset",
-                "provider": provider,
-                "provider_api": provider_api,
-                "model": model,
-                "url": url,
-            }
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
-            "mode": "lightweight_models",
-            "url": url,
-        }
+        catalog = await fetch_model_catalog(
+            provider, base_url, provider_api, api_key,
+            str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
+        )
     except Exception as exc:
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"轻量连通性测试失败: {exc}",
-            "mode": "lightweight_models",
-            "url": url,
-        }
-
-    model_ids = _extract_model_ids(payload)
-    if model_ids and not _contains_model(model_ids, model):
-        if _contains_model(preset_models, model):
-            return {
-                "success": True,
-                "connected": True,
-                "message": "轻量连通性测试通过；provider 模型目录未返回当前别名，已按内置模型预设确认。",
-                "mode": "lightweight_models_with_preset_alias",
-                "provider": provider,
-                "provider_api": provider_api,
-                "model": model,
-                "models_count": len(model_ids),
-                "url": url,
-            }
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"连接可用，但模型目录中没有当前模型: {model}",
-            "mode": "lightweight_models",
-            "provider": provider,
-            "model": model,
-            "models_count": len(model_ids),
-            "url": url,
-        }
-
+        return {**result, "message": catalog_error_message(exc)}
+    result.update({"url": catalog.url, "models_count": len(catalog.models)})
+    if model.casefold() not in {item.casefold() for item in catalog.models}:
+        return {**result, "message": f"模型目录查询成功，但当前模型不在目录中：{model}"}
+    model_apis = llm_config.get("model_provider_apis") or {}
     return {
-        "success": True,
-        "connected": True,
-        "message": "轻量连通性测试通过",
-        "mode": "lightweight_models",
-        "provider": provider,
-        "provider_api": provider_api,
-        "model": model,
-        "models_count": len(model_ids),
-        "url": url,
+        **result, "success": True, "connected": True,
+        "provider_api": model_apis.get(model) or provider_api,
+        "message": "模型目录查询成功，当前模型已找到；尚未执行生成调用。",
     }
 
 
@@ -1135,7 +1059,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
         api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
         providers_payload[provider_id] = {
             "provider": provider_id,
-            "provider_api": provider_config.get("provider_api") or "openai-completions",
+            "provider_api": provider_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
             "base_url": provider_config.get("base_url") or "",
             "model": provider_config.get("model") or "",
             "api_key": _mask_secret(api_key, api_key_source),
@@ -1185,7 +1109,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
             "service_token": _mask_secret(*_resolve_service_token(normalized_ai)),
             "default_provider": default_provider,
             "provider": default_provider,
-            "provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
+            "provider_api": display_llm_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
             "base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
             "model": display_llm_config.get("model") or "MiniMax-M2.7",
             "api_key": display_llm_config.get("api_key") or _mask_secret(None),
@@ -1428,7 +1352,7 @@ async def update_smtp_settings(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if current_user.role not in ("admin", "super_admin"):
+    if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
         raise HTTPException(status_code=403, detail="Only administrators can change SMTP settings")
     current = await get_setting_payload(db, "smtp")
     merged = _build_smtp_payload(current, payload)
@@ -1442,7 +1366,7 @@ async def test_smtp_settings(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if current_user.role not in ("admin", "super_admin"):
+    if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
         raise HTTPException(status_code=403, detail="Only administrators can test SMTP settings")
     from app.services.email import EmailError, send_email
 
@@ -1612,9 +1536,34 @@ async def connect_ai_provider_integration(
         llm_config=quick_llm_config,
     )
 
+    await emit_business_log(
+        logger,
+        event="settings.ai_provider.connect.start",
+        message="AI provider connection test started",
+        category="ai",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={
+            "provider": payload.provider,
+            "model": payload.model,
+            "timeout_seconds": min(int(runtime_config["timeout_seconds"] or 60), AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS),
+        },
+    )
     try:
         status_result = await client.get_status()
         if not status_result.configured:
+            await emit_business_log(
+                logger,
+                event="settings.ai_provider.connect.failed",
+                message="AI provider connection test failed because provider is incomplete",
+                category="ai",
+                level="warning",
+                service="api",
+                module=__name__,
+                user_id=current_user.id,
+                context={"provider": payload.provider, "model": payload.model, "configured": False},
+            )
             return {
                 "success": False,
                 "connected": False,
@@ -1628,6 +1577,22 @@ async def connect_ai_provider_integration(
                 AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS,
             ),
         )
+        await emit_business_log(
+            logger,
+            event=("settings.ai_provider.connect.success" if lightweight_result["success"]
+                   else "settings.ai_provider.connect.failed"),
+            message="AI provider connection test completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={
+                "provider": payload.provider,
+                "model": payload.model,
+                "configured": True,
+                "connected": lightweight_result["connected"],
+            },
+        )
         return {
             **lightweight_result,
             "status": status_result.model_dump(),
@@ -1639,6 +1604,17 @@ async def connect_ai_provider_integration(
             "message": str(exc.detail),
         }
     except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.ai_provider.connect.failed",
+            message="AI provider connection test failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": payload.provider, "model": payload.model}),
+        )
         return {
             "success": False,
             "connected": False,
@@ -1786,8 +1762,28 @@ async def connect_web_search_integration(
     runtime_config = _runtime_config_from_web_search_payload(draft_web_search_payload)
     client = WebSearchClient(runtime_config)
 
+    await emit_business_log(
+        logger,
+        event="settings.web_search.connect.start",
+        message="WebSearch connection test started",
+        category="ai_tool",
+        service="api",
+        module=__name__,
+        user_id=current_user.id,
+        context={"provider": runtime_config.default_provider},
+    )
     try:
         results = await client.test_connection()
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.success",
+            message="WebSearch connection test completed",
+            category="ai_tool",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": runtime_config.default_provider, "result_count": len(results)},
+        )
         return {
             "success": True,
             "connected": True,
@@ -1796,18 +1792,51 @@ async def connect_web_search_integration(
             "results": [item.model_dump(mode="json") for item in results[:3]],
         }
     except WebSearchConfigurationError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed because configuration is incomplete",
+            category="ai_tool",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
             "message": str(exc),
         }
     except WebSearchError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed",
+            category="ai_tool",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
             "message": str(exc),
         }
     except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.web_search.connect.failed",
+            message="WebSearch connection test failed",
+            category="ai_tool",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": runtime_config.default_provider}),
+        )
         return {
             "success": False,
             "connected": False,
@@ -1835,17 +1864,60 @@ async def generate_provider_credential_guide(
     ai_client: AIProviderClient = Depends(get_ai_provider_client),
 ):
     try:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.start",
+            message="Credential guide generation started",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": provider},
+        )
         web_search_client = await get_web_search_client(db)
-        return {
-            "guide": await generate_credential_guide(
+        guide = await generate_credential_guide(
                 db,
                 provider,
                 ai_client,
                 web_search_client,
             )
-        }
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.success",
+            message="Credential guide generation completed",
+            category="ai",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context={"provider": provider},
+        )
+        return {"guide": guide}
     except ValueError as exc:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.failed",
+            message="Credential guide generation failed",
+            category="ai",
+            level="warning",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": provider}),
+        )
         raise HTTPException(status_code=404, detail=str(exc)) from exc
+    except Exception as exc:
+        await emit_business_log(
+            logger,
+            event="settings.credential_guide.generate.failed",
+            message="Credential guide generation failed",
+            category="ai",
+            level="error",
+            service="api",
+            module=__name__,
+            user_id=current_user.id,
+            context=exception_context(exc, {"provider": provider}),
+        )
+        raise
 
 
 @router.post("/credential-guides/{provider}/reset")
@@ -1863,8 +1935,18 @@ async def reset_provider_credential_guide(
 @router.get("/integrations/ai-provider/presets")
 async def get_ai_provider_presets(
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
-    return {"data": list_fallback_llm_provider_presets()}
+    presets = list_fallback_llm_provider_presets()
+    saved = await get_setting_payloads(
+        db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
+    )
+    return {
+        "data": [
+            {**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
+            for preset in presets
+        ]
+    }
 
 
 @router.post("/integrations/ai-provider/presets/{provider}/refresh")
@@ -1872,22 +1954,52 @@ async def refresh_ai_provider_preset(
     provider: str,
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
+    payload: AIProviderIntegrationUpdate | None = None,
 ):
     try:
         provider_id = _normalize_provider_id(provider)
-        api_key = None
-        if provider_id == "opencode-go":
-            current_payload = await get_setting_payload(db, "external_integrations")
-            ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
-            provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
-            api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
-        return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
+        get_fallback_llm_provider_preset(provider_id)
     except ValueError as exc:
         raise HTTPException(status_code=404, detail=str(exc)) from exc
+
+    current_payload = await get_setting_payload(db, "external_integrations")
+    if payload is not None:
+        if _normalize_provider_id(payload.provider) != provider_id:
+            raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
+        ai_payload = _build_ai_provider_payload(current_payload, payload)
+    else:
+        ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
+    provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
+    api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
+    if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
+        raise HTTPException(
+            status_code=400, detail="请先配置当前供应商的 API Key，再刷新模型列表。",
+        )
+    try:
+        refreshed = await refresh_llm_provider_preset(
+            provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
+            provider_api=provider_config.get("provider_api"),
+            anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
+        )
     except Exception as exc:
-        fallback = get_fallback_llm_provider_preset(provider)
-        fallback["refresh_error"] = str(exc)
-        return {"data": fallback}
+        logger.warning(
+            "LLM provider catalog refresh failed",
+            extra={
+                "event": "settings.ai_provider.catalog.failed",
+                "context": {
+                    "provider": provider_id,
+                    "error_type": type(exc).__name__,
+                },
+            },
+        )
+        raise HTTPException(
+            status_code=502,
+            detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
+        ) from exc
+
+    refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
+    await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
+    return {"data": refreshed}
 
 
 @router.put("/integrations")

backend/app/api/v1/system_control.py

@@ -1,20 +1,19 @@
 from __future__ import annotations
 
 import os
+import secrets
 import subprocess
 import sys
 
 from datetime import datetime
 
-from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, status
 from pydantic import BaseModel
-from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.core.config import ROOT_DIR
+from app.core.config import ROOT_DIR, settings
 from app.core.security import get_current_user
 from app.db.session import get_db
-from app.models.system_log import AuditLog, SystemLog
 from app.models.user import User
 from app.services.persistent_logs import record_audit_log, record_system_log
 from app.services.system_control import (
@@ -37,7 +36,11 @@ from app.services.system_logs import (
     append_buffer_log,
     list_log_sources,
     normalize_log_level,
+    read_database_log_snapshot,
     read_log_snapshot,
+    read_observability_group_events,
+    read_observability_groups,
+    read_observability_raw_events,
 )
 from app.services.earth_layer_cache import earth_layer_cache
 
@@ -109,12 +112,104 @@ class EarthClientLogEventCreate(BaseModel):
     url: str | None = None
     module: str | None = None
     detail: str | None = None
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    metadata: dict[str, object] | None = None
 
 
 class EarthClientLogEventResponse(BaseModel):
     accepted: bool
     source_id: str
     level: str
+    fingerprint: str | None = None
+
+
+class ServiceLogEventCreate(BaseModel):
+    source: str = "ai-provider"
+    service: str = "ai-provider"
+    module: str | None = None
+    category: str | None = None
+    event: str = "service.runtime_log"
+    level: str = "error"
+    message: str
+    fingerprint: str | None = None
+    occurrence_count: int = 1
+    request_id: str | None = None
+    trace_id: str | None = None
+    task_id: str | None = None
+    source_id: int | str | None = None
+    provider: str | None = None
+    context: dict[str, object] | None = None
+
+
+async def ingest_client_log_event(
+    source_id: str,
+    *,
+    service: str,
+    event: str,
+    default_module: str,
+    default_category: str,
+    payload: EarthClientLogEventCreate,
+    request: Request,
+) -> EarthClientLogEventResponse:
+    normalized_level = normalize_log_level(payload.level)
+    append_buffer_log(
+        source_id,
+        level=normalized_level,
+        message=payload.message,
+        context={
+            "category": payload.category or "",
+            "url": payload.url or "",
+            "module": payload.module or "",
+            "detail": payload.detail or "",
+            "fingerprint": payload.fingerprint or "",
+            "occurrence_count": max(1, int(payload.occurrence_count or 1)),
+            "metadata": payload.metadata or {},
+        },
+    )
+    await record_system_log(
+        source=source_id,
+        service=service,
+        module=payload.module or default_module,
+        event=event,
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or default_category,
+        context={
+            "url": payload.url or "",
+            "detail": payload.detail or "",
+            "module": payload.module or "",
+            "client_ip": request.client.host if request.client else "",
+            "metadata": payload.metadata or {},
+        },
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(accepted=True, source_id=source_id, level=normalized_level, fingerprint=payload.fingerprint)
+
+
+def require_observability_ingest_token(
+    authorization: str | None,
+    ingest_token: str | None,
+) -> None:
+    expected_token = settings.OBSERVABILITY_INGEST_TOKEN.strip()
+    if not expected_token:
+        raise HTTPException(
+            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+            detail="Observability service ingestion is not configured",
+        )
+    provided = ""
+    if ingest_token:
+        provided = ingest_token.strip()
+    elif authorization:
+        scheme, _, token = authorization.partition(" ")
+        if scheme.lower() == "bearer":
+            provided = token.strip()
+    if not provided or not secrets.compare_digest(provided, expected_token):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Invalid observability ingestion token",
+        )
 
 
 class EarthLayerCacheStatusResponse(BaseModel):
@@ -339,95 +434,116 @@ async def get_system_log_sources(
     }
 
 
-async def read_database_log_snapshot(
-    source_id: str,
-    *,
-    limit: int,
-    level: str,
-    levels: str | None,
-    start_date: str | None,
-    end_date: str | None,
-    search: str | None,
-    db: AsyncSession,
-) -> dict | None:
-    selected_levels = set(normalize_log_level(item) for item in (levels or level).split(",") if item.strip())
-    selected_levels.discard("all")
-    search_query = (search or "").strip().lower()
-    lines: list[str] = []
+@router.get("/logs/observability/groups")
+async def get_observability_log_groups(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    if normalized_start_date and normalized_end_date and normalized_start_date > normalized_end_date:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="start_date must be earlier than or equal to end_date")
+    return await read_observability_groups(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
 
-    if source_id == "system-db":
-        query = select(SystemLog).order_by(SystemLog.occurred_at.desc().nullslast(), SystemLog.id.desc()).limit(limit * 5)
-        result = await db.execute(query)
-        records = result.scalars().all()
-        for record in records:
-            record_level = normalize_log_level(record.level)
-            if selected_levels and record_level not in selected_levels:
-                continue
-            occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
-            if start_date and occurred_at and occurred_at < start_date:
-                continue
-            if end_date and occurred_at and occurred_at > end_date:
-                continue
-            line = " ".join(
-                part
-                for part in [
-                    record.occurred_at.isoformat() if record.occurred_at else "",
-                    record_level.upper(),
-                    record.source,
-                    record.event or "",
-                    record.message,
-                ]
-                if part
-            )
-            if search_query and search_query not in line.lower():
-                continue
-            lines.append(line)
-    elif source_id == "audit-db":
-        query = select(AuditLog).order_by(AuditLog.occurred_at.desc().nullslast(), AuditLog.id.desc()).limit(limit * 5)
-        result = await db.execute(query)
-        records = result.scalars().all()
-        for record in records:
-            occurred_at = record.occurred_at.date().isoformat() if record.occurred_at else ""
-            if start_date and occurred_at and occurred_at < start_date:
-                continue
-            if end_date and occurred_at and occurred_at > end_date:
-                continue
-            line = " ".join(
-                part
-                for part in [
-                    record.occurred_at.isoformat() if record.occurred_at else "",
-                    "INFO",
-                    record.action,
-                    record.target_type or "",
-                    record.target_id or "",
-                    record.result or "",
-                ]
-                if part
-            )
-            if search_query and search_query not in line.lower():
-                continue
-            lines.append(line)
-    else:
-        return None
 
-    lines = list(reversed(lines[:limit]))
-    return {
-        "source_id": source_id,
-        "name": "系统事件" if source_id == "system-db" else "审计事件",
-        "kind": "database",
-        "location": "table://system_logs" if source_id == "system-db" else "table://audit_logs",
-        "description": "数据库持久化日志",
-        "category": "database" if source_id == "system-db" else "audit",
-        "status": "ok" if lines else "empty",
-        "level": level,
-        "selected_levels": sorted(selected_levels),
-        "search_query": search or "",
-        "available_levels": ["all", "error", "warning", "info", "debug"],
-        "daily_markers": [],
-        "line_limit": limit,
-        "line_count": len(lines),
-        "lines": lines,
-    }
+@router.get("/logs/observability/groups/{fingerprint}/events")
+async def get_observability_group_events(
+    fingerprint: str,
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    payload = await read_observability_group_events(fingerprint, limit=limit, db=db)
+    if payload is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Observability group not found")
+    return payload
+
+
+@router.get("/logs/observability/raw")
+async def get_observability_raw_events(
+    limit: int = DEFAULT_LOG_LINE_LIMIT,
+    level: str = "all",
+    levels: str | None = Query(None, description="Comma-separated log levels"),
+    start_date: str | None = Query(None, description="Filter logs from this date (YYYY-MM-DD)"),
+    end_date: str | None = Query(None, description="Filter logs until this date (YYYY-MM-DD)"),
+    search: str | None = Query(None, description="Case-insensitive substring search"),
+    current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
+):
+    ensure_super_admin(current_user)
+    if limit < 1 or limit > MAX_LOG_LINE_LIMIT:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"limit must be between 1 and {MAX_LOG_LINE_LIMIT}")
+    normalized_start_date = validate_log_date(start_date, "start_date")
+    normalized_end_date = validate_log_date(end_date, "end_date")
+    return await read_observability_raw_events(
+        limit=limit,
+        level=level,
+        levels=levels,
+        start_date=normalized_start_date,
+        end_date=normalized_end_date,
+        search=search,
+        db=db,
+    )
+
+
+@router.post("/logs/service", response_model=EarthClientLogEventResponse)
+async def ingest_service_log(
+    payload: ServiceLogEventCreate,
+    authorization: str | None = Header(default=None),
+    ingest_token: str | None = Header(default=None, alias="X-Planet-Observability-Token"),
+):
+    require_observability_ingest_token(authorization, ingest_token)
+    normalized_level = normalize_log_level(payload.level)
+    source = (payload.source or "ai-provider").strip() or "ai-provider"
+    context = dict(payload.context or {})
+    if payload.request_id:
+        context["request_id"] = payload.request_id
+    if payload.trace_id:
+        context["trace_id"] = payload.trace_id
+    if payload.task_id:
+        context["task_id"] = payload.task_id
+    if payload.source_id is not None:
+        context["source_id"] = payload.source_id
+    if payload.provider:
+        context["provider"] = payload.provider
+    await record_system_log(
+        source=source,
+        service=(payload.service or source).strip() or source,
+        module=payload.module or source,
+        event=(payload.event or "service.runtime_log").strip() or "service.runtime_log",
+        level=normalized_level,
+        message=payload.message,
+        category=payload.category or "service-runtime",
+        context=context,
+        fingerprint=payload.fingerprint,
+        occurrence_count=max(1, int(payload.occurrence_count or 1)),
+    )
+    return EarthClientLogEventResponse(
+        accepted=True,
+        source_id=source,
+        level=normalized_level,
+        fingerprint=payload.fingerprint,
+    )
 
 
 @router.get("/logs/{source_id}", response_model=SystemLogSnapshotResponse)
@@ -493,31 +609,28 @@ async def ingest_earth_client_log(
     payload: EarthClientLogEventCreate,
     request: Request,
 ):
-    normalized_level = normalize_log_level(payload.level)
-    append_buffer_log(
+    return await ingest_client_log_event(
         "earth-client",
-        level=normalized_level,
-        message=payload.message,
-        context={
-            "category": payload.category or "",
-            "url": payload.url or "",
-            "module": payload.module or "",
-            "detail": payload.detail or "",
-        },
-    )
-    await record_system_log(
-        source="earth-client",
         service="earth",
-        module=payload.module or "earth-client",
         event="earth.client.runtime_log",
-        level=normalized_level,
-        message=payload.message,
-        category=payload.category or "client-runtime",
-        context={
-            "url": payload.url or "",
-            "detail": payload.detail or "",
-            "module": payload.module or "",
-            "client_ip": request.client.host if request.client else "",
-        },
+        default_module="earth-client",
+        default_category="client-runtime",
+        payload=payload,
+        request=request,
+    )
+
+
+@router.post("/logs/admin-client", response_model=EarthClientLogEventResponse)
+async def ingest_admin_client_log(
+    payload: EarthClientLogEventCreate,
+    request: Request,
+):
+    return await ingest_client_log_event(
+        "admin-client",
+        service="admin",
+        event="admin.client.runtime_log",
+        default_module="admin-client",
+        default_category="client-runtime",
+        payload=payload,
+        request=request,
     )
-    return {"accepted": True, "source_id": "earth-client", "level": normalized_level}

backend/app/api/v1/tasks.py

@@ -29,7 +29,8 @@ async def list_tasks(
         SELECT ct.id, ct.datasource_id, ds.name as datasource_name, ct.status,
                ct.started_at, ct.completed_at, ct.records_processed, ct.error_message,
                ct.phase, ct.phase_progress, ct.phase_message, ct.phase_current,
-               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress
+               ct.phase_total, ct.phase_unit, ct.total_records, ct.progress,
+               ct.task_type, ct.source, ds.source as datasource_source
         FROM collection_tasks ct
         JOIN data_sources ds ON ct.datasource_id = ds.id
         WHERE 1=1
@@ -39,12 +40,19 @@ async def list_tasks(
 
     if datasource_id:
         query += " AND ct.datasource_id = :datasource_id"
-        count_query += " WHERE ct.datasource_id = :datasource_id"
+        count_query += " AND ct.datasource_id = :datasource_id"
         params["datasource_id"] = datasource_id
     if status:
-        query += " AND ct.status = :status"
-        count_query += " AND ct.status = :status"
-        params["status"] = status
+        statuses = [item.strip() for item in status.split(",") if item.strip()]
+        if len(statuses) > 1:
+            placeholders = ", ".join(f":status_{index}" for index, _item in enumerate(statuses))
+            query += f" AND ct.status IN ({placeholders})"
+            count_query += f" AND ct.status IN ({placeholders})"
+            params.update({f"status_{index}": item for index, item in enumerate(statuses)})
+        else:
+            query += " AND ct.status = :status"
+            count_query += " AND ct.status = :status"
+            params["status"] = statuses[0] if statuses else status
 
     query += f" ORDER BY ct.created_at DESC LIMIT {page_size} OFFSET {offset}"
 
@@ -76,6 +84,9 @@ async def list_tasks(
                 "phase_unit": t[13],
                 "total_records": t[14],
                 "progress": t[15],
+                "task_type": t[16],
+                "source": t[17] or t[18],
+                "datasource_source": t[18],
             }
             for t in tasks
         ],

backend/app/api/v1/tv.py

@@ -1,3 +1,4 @@
+import re
 from urllib.parse import quote, urljoin
 
 import httpx
@@ -6,16 +7,41 @@ from fastapi.responses import Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
+from app.services.tv_catalog import get_tv_catalog_page
 from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
 
 router = APIRouter()
 
+_HLS_URI_ATTRIBUTE_RE = re.compile(r'URI="([^"]+)"')
+
+
+def _proxied_tv_url(url: str) -> str:
+    return f"/api/v1/tv/proxy?url={quote(url, safe='')}"
+
+
+def _rewrite_hls_uri_attributes(line: str, *, base_url: str) -> str:
+    def replace(match: re.Match[str]) -> str:
+        uri = match.group(1)
+        absolute_url = urljoin(base_url, uri)
+        return f'URI="{_proxied_tv_url(absolute_url)}"'
+
+    return _HLS_URI_ATTRIBUTE_RE.sub(replace, line)
+
+
+def _should_strip_hls_metadata_line(line: str) -> bool:
+    normalized = line.strip().upper()
+    return normalized.startswith("#EXT-X-MEDIA:") and "TYPE=SUBTITLES" in normalized
+
 
 @router.get("/streams")
 async def list_public_tv_streams(
+    offset: int = Query(0, ge=0),
+    limit: int = Query(50, ge=1, le=100),
+    q: str = Query("", max_length=200),
+    selected_id: str | None = Query(None, max_length=200),
     db: AsyncSession = Depends(get_db),
 ):
-    return await get_public_tv_payload(db)
+    return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
 
 
 @router.get("/proxy")
@@ -56,11 +82,16 @@ async def proxy_tv_stream(
         rewritten_lines: list[str] = []
         for line in manifest_text.splitlines():
             stripped = line.strip()
-            if not stripped or stripped.startswith("#"):
+            if not stripped:
                 rewritten_lines.append(line)
                 continue
+            if stripped.startswith("#"):
+                if _should_strip_hls_metadata_line(line):
+                    continue
+                rewritten_lines.append(_rewrite_hls_uri_attributes(line, base_url=response_url))
+                continue
             absolute_url = urljoin(response_url, stripped)
-            rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
+            rewritten_lines.append(_proxied_tv_url(absolute_url))
         return Response(
             content="\n".join(rewritten_lines),
             media_type="application/vnd.apple.mpegurl",

backend/app/api/v1/users.py

@@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import text
 
+from app.core.enums import UserRole
 from app.core.security import get_current_user, get_password_hash
 from app.db.session import get_db
 from app.models.user import User
@@ -13,6 +14,8 @@ from app.schemas.user import UserCreate, UserUpdate
 router = APIRouter()
 
 VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"}
+ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value]
+SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value]
 
 
 def check_permission(current_user: User, required_roles: List[str]) -> bool:
@@ -32,7 +35,7 @@ async def list_users(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]):
+    if not check_permission(current_user, ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -91,7 +94,7 @@ async def get_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
@@ -128,7 +131,7 @@ async def create_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can create users",
@@ -196,18 +199,18 @@ async def update_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
+    if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Insufficient permissions",
         )
 
-    if not check_permission(current_user, ["super_admin"]) and user_data.role is not None:
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can change user role",
         )
-    if not check_permission(current_user, ["super_admin"]) and user_data.gatekeeper_groups is not None:
+    if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None:
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can change Gatekeeper groups",
@@ -260,7 +263,7 @@ async def delete_user(
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    if not check_permission(current_user, ["super_admin"]):
+    if not check_permission(current_user, SUPER_ADMIN_ROLES):
         raise HTTPException(
             status_code=status.HTTP_403_FORBIDDEN,
             detail="Only super_admin can delete users",

backend/app/api/v1/vessels.py

@@ -1,4 +1,4 @@
-"""Bounded vessel snapshot APIs for viewport-first consumers."""
+"""Bounded vessel snapshot APIs backed by the latest vessel state table."""
 
 from typing import Optional
 
@@ -14,8 +14,8 @@ router = APIRouter()
 
 @router.get("/snapshot")
 async def get_vessel_snapshot(
-    bbox: Optional[str] = Query(None, description="Viewport bbox as lon_min,lat_min,lon_max,lat_max"),
-    zoom: int = Query(..., ge=1, le=20, description="Current map zoom level"),
+    bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"),
+    zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"),
     type: Optional[str] = Query(
         None,
         description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",

backend/app/api/v1/visualization.py

@@ -17,15 +17,18 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import select, func
 from typing import List, Dict, Any, Optional
 
+from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
 from app.core.collected_data_fields import get_record_field
+from app.core.enums import BGPStatus
 from app.core.satellite_tle import build_tle_lines_from_elements
 from app.core.time import to_iso8601_utc
 from app.db.session import get_db
+from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
 from app.models.collected_data import CollectedData
-from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
+from app.models.vessel import AISSourceHealth, VesselCurrentState, VesselPosition, VesselStatic
 from app.services.bgp_collectors import build_bgp_collector_coverage
 from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
 from app.services.compute_center_locations import (
@@ -38,7 +41,7 @@ from app.services.compute_center_locations import (
     upsert_compute_center_location,
 )
 from app.services.ai_client import get_ai_provider_client
-from app.api.v1.settings import get_web_search_client
+from app.api.v1.settings import get_runtime_web_search_config, get_web_search_client
 from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
 from app.services.location.llm_fallback import (
     collect_llm_location_fallback_candidate,
@@ -47,11 +50,10 @@ from app.services.location.llm_fallback import (
 from app.services.persistent_logs import record_system_log
 from app.services.vessel_ais_aggregation import (
     build_field_conflict_candidates,
-    count_unique_raw_vessel_mmsi,
     get_aggregated_vessel,
     get_aggregated_vessel_track,
     get_aggregated_vessels,
-    get_aggregated_vessels_snapshot,
+    get_current_vessels_snapshot,
     get_vessel_conflict_records,
     get_vessel_raw_observations,
     MAX_SNAPSHOT_LIMIT,
@@ -75,7 +77,6 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128
 TERRAIN_TILE_BATCH_CONCURRENCY = 16
 _terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
 VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
-VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
 SECONDS_PER_MINUTE = 60
 BYTES_PER_MIB = 1024 * 1024
 CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
@@ -105,8 +106,8 @@ LANDING_POINT_CACHE_POLICY = EarthLayerCachePolicy(
 SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
     SATELLITE_CACHE_FRESH_SECONDS,
     SATELLITE_CACHE_STALE_SECONDS,
-    max_features=8000,
-    max_bytes=10 * BYTES_PER_MIB,
+    max_features=25000,
+    max_bytes=32 * BYTES_PER_MIB,
 )
 COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
     COMPUTE_CENTER_CACHE_FRESH_SECONDS,
@@ -338,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
     return {"type": "FeatureCollection", "features": features}
 
 
-def _normalize_satellite_constellation_group(
-    raw_group: Any,
-    name: Optional[str],
-) -> Optional[str]:
-    normalized_group = str(raw_group or "").strip().lower()
-    if normalized_group:
-        return normalized_group
-
-    normalized_name = str(name or "").strip().upper()
-    if normalized_name.startswith("STARLINK"):
-        return "starlink"
-    if normalized_name.startswith("IRIDIUM"):
-        return "iridium-next"
-
-    return None
-
 
 def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
     if constellation_group == "starlink":
@@ -506,16 +491,19 @@ async def _count_current_or_latest_task_data(
 async def _load_current_collected_data_by_sources(
     db: AsyncSession,
     sources: List[str],
+    *, include_compute_types: bool = False,
 ) -> Dict[str, List[CollectedData]]:
     if not sources:
         return {}
 
     stmt = (
         select(CollectedData)
-        .where(CollectedData.source.in_(sources))
+        .where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
         .where(CollectedData.is_current.is_(True))
         .order_by(CollectedData.source.asc(), CollectedData.id.desc())
     )
+    if include_compute_types:
+        stmt = stmt.where(CollectedData.deleted_at.is_(None))
     result = await db.execute(stmt)
 
     grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
@@ -645,7 +633,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
         return "unknown"
 
     unit = str(capacity_unit or "").strip().lower()
-    if unit in {"pflop/s", "pflops", "pflop"}:
+    if unit in {"flop/s", "flops"}:
+        normalized_tflops = capacity_value / 1e12
+    elif unit in {"pflop/s", "pflops", "pflop"}:
         normalized_tflops = capacity_value * 1000
     elif unit in {"gflop/s", "gflops", "gflop"}:
         normalized_tflops = capacity_value
@@ -673,15 +663,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
     """
     features: List[Dict[str, Any]] = []
     unresolved: List[Dict[str, Any]] = []
+    withheld_count = 0
 
     for record in records:
         metadata = record.extra_data or {}
+        if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
+            withheld_count += 1
+            continue
         result = resolve_compute_center_location_full(record, metadata)
-        site_type = (
-            "supercomputer"
-            if record.source == "top500" or record.data_type == "supercomputer"
-            else "gpu_cluster"
-        )
+        site_type = compute_site_type(record)
 
         if not result.is_resolved:
             diagnostic = result.diagnostic or ResolutionDiagnostic(
@@ -707,7 +697,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
         latitude = location.latitude
         longitude = location.longitude
 
-        if site_type == "supercomputer":
+        if metadata.get("compute_contract") == "v1":
+            capacity_value = _parse_float(get_record_field(record, "value"))
+            capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
+        elif site_type == "supercomputer":
             capacity_value = _parse_float(get_record_field(record, "rmax"))
             capacity_unit = "GFlops"
         else:
@@ -768,7 +761,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
             }
         )
 
-    return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
+    return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
 
 
 VESSEL_TYPE_FILTERS = {
@@ -839,9 +832,14 @@ def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict
             continue
         source_summary = {}
         for source, summary in (vessel.get("source_summary") or {}).items():
+            latest_observed_at = summary.get("latest_observed_at")
             source_summary[source] = {
                 **summary,
-                "latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
+                "latest_observed_at": (
+                    to_iso8601_utc(latest_observed_at)
+                    if isinstance(latest_observed_at, datetime)
+                    else latest_observed_at
+                ),
             }
         props = {
             "mmsi": vessel["mmsi"],
@@ -1075,7 +1073,7 @@ async def build_vessel_snapshot_response(
     db: AsyncSession,
     *,
     bbox: tuple[float, float, float, float] | None,
-    zoom: int | None,
+    zoom: float | None,
     type_filter: str | None,
     limit: int | None,
     since_minutes: int = 60,
@@ -1117,6 +1115,7 @@ async def build_vessel_snapshot_response(
     safe_limit = _safe_vessel_limit(limit)
     safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
     observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
+    snapshot_started_at = datetime.now(UTC)
     features, diagnostics = await _load_raw_vessel_snapshot_features(
         db,
         bbox=bbox,
@@ -1133,6 +1132,7 @@ async def build_vessel_snapshot_response(
         "features": features,
         "count": len(features),
         "stats": _build_vessel_stats(features),
+        "generated_at": to_iso8601_utc(snapshot_started_at),
         "diagnostics": {
             **diagnostics,
             "filtered_count": len(features),
@@ -1563,15 +1563,7 @@ async def _build_cables_geojson(db: AsyncSession) -> dict[str, Any]:
     try:
         records = await _load_current_collected_data(db, "arcgis_cables")
 
-        if not records:
-            raise HTTPException(
-                status_code=404,
-                detail="No cable data found. Please run the arcgis_cables collector first.",
-            )
-
         return convert_cable_to_geojson(records)
-    except HTTPException:
-        raise
     except Exception as e:
         logger.exception_event(
             "Failed to build cables GeoJSON response",
@@ -1625,16 +1617,8 @@ async def _build_landing_points_geojson(db: AsyncSession) -> dict[str, Any]:
             relation_records,
             cable_records,
         )
-        
-        if not records:
-            raise HTTPException(
-                status_code=404,
-                detail="No landing point data found. Please run the arcgis_landing_points collector first.",
-            )
-        
+
         return convert_landing_point_to_geojson(records, city_to_cable_ids_map, cable_id_to_name_map)
-    except HTTPException:
-        raise
     except Exception as e:
         logger.exception_event(
             "Failed to build landing points GeoJSON response",
@@ -1965,9 +1949,10 @@ async def _build_compute_centers_geojson(
     records_by_source = await _load_current_collected_data_by_sources(
         db,
         ["top500", "epoch_ai_gpu"],
+        include_compute_types=True,
     )
     records = _filter_known_records(
-        records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
+        [record for group in records_by_source.values() for record in group],
     )
     if limit is not None:
         records = records[:limit]
@@ -1982,6 +1967,7 @@ async def _build_compute_centers_geojson(
                 "total": 0,
                 "supercomputers": 0,
                 "gpu_clusters": 0,
+                "by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
                 "unresolved": 0,
             },
         }
@@ -2024,6 +2010,7 @@ async def _build_compute_centers_geojson(
         "count": len(sanitized_features),
         "stats": {
             "total": len(sanitized_features),
+            "by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
             "supercomputers": sum(
                 1 for feature in sanitized_features
                 if feature.get("properties", {}).get("site_type") == "supercomputer"
@@ -2071,6 +2058,44 @@ class SaveComputeCenterLocationRequest(BaseModel):
     model_config = {"populate_by_name": True}
 
 
+async def _compute_center_location_web_search_capability(db: AsyncSession) -> Dict[str, Any]:
+    try:
+        config = await get_runtime_web_search_config(db)
+    except Exception as exc:
+        return {
+            "enabled": False,
+            "provider": None,
+            "reason": f"WebSearch 配置读取失败：{exc}",
+        }
+    provider_config = config.active_provider_config
+    has_api_key = bool((provider_config.api_key or "").strip())
+    if not config.enabled:
+        return {
+            "enabled": False,
+            "provider": config.default_provider,
+            "reason": "WebSearch 未开启，无法进行事实核查定位。",
+        }
+    if not has_api_key:
+        return {
+            "enabled": False,
+            "provider": config.default_provider,
+            "reason": f"WebSearch Provider {config.default_provider} 未配置 API Key。",
+        }
+    return {
+        "enabled": True,
+        "provider": config.default_provider,
+        "reason": "",
+    }
+
+
+@router.get("/compute-centers/location-capability")
+async def get_compute_center_location_capability(
+    db: AsyncSession = Depends(get_db),
+):
+    """Return whether fact-checked compute-center location collection can run."""
+    return await _compute_center_location_web_search_capability(db)
+
+
 @router.post("/compute-centers/{source_id}/collect-location")
 async def collect_compute_center_location(
     source_id: str,
@@ -2089,6 +2114,9 @@ async def collect_compute_center_location(
     """
     if not source_id or not source_id.strip():
         raise HTTPException(status_code=400, detail="source_id is required")
+    capability = await _compute_center_location_web_search_capability(db)
+    if not capability.get("enabled"):
+        raise HTTPException(status_code=409, detail=capability)
 
     record = await _load_compute_center_record(db, source_id)
     name = payload.name or (record.name if record else None)
@@ -2299,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
     stmt = (
         select(CollectedData)
         .where(CollectedData.source_id == source_id)
-        .where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
+        .where(compute_record_filter(CollectedData))
+        .where(CollectedData.deleted_at.is_(None))
         .order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
         .limit(1)
     )
@@ -2315,58 +2344,35 @@ async def _load_raw_vessel_snapshot_features(
     observed_since: datetime,
 ) -> tuple[list[dict[str, Any]], dict[str, Any]]:
     if bbox is None:
-        aggregated_vessels = await get_aggregated_vessels(
-            db,
-            limit=limit,
-            observed_since=observed_since,
-        )
-    else:
-        aggregated_vessels = await get_aggregated_vessels_snapshot(
-            db,
-            bbox=bbox,
-            limit=limit,
-            observed_since=observed_since,
-        )
-    raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
-    raw_features = raw_geojson.get("features", [])
-    features = raw_features
-    legacy_features: list[dict[str, Any]] = []
-    legacy_fallback_used = False
-    if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED:
-        legacy_features = await _load_legacy_vessel_snapshot_features(
-            db,
-            bbox=bbox,
-            limit=limit,
-        )
-        features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features)
-        legacy_fallback_used = bool(legacy_features)
-
+        return [], {
+            "source": "vessel_current_state",
+            "current_state_count": 0,
+            "final_unique_mmsi": 0,
+        }
+    current_vessels = await get_current_vessels_snapshot(
+        db,
+        bbox=bbox,
+        limit=limit,
+        observed_since=observed_since,
+    )
+    features = convert_aggregated_vessels_to_geojson(current_vessels).get("features", [])
+    unique_mmsi = len(
+        {
+            key
+            for key in (_feature_mmsi_key(feature) for feature in features)
+            if key is not None
+        }
+    )
     return features, {
-        "raw_feature_count": len(raw_features),
-        "raw_unique_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in raw_features)
-                if key is not None
-            }
-        ),
-        "legacy_feature_count": len(legacy_features),
-        "legacy_backfilled_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in legacy_features)
-                if key is not None
-            }
-        ),
-        "legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
-        "legacy_fallback_used": legacy_fallback_used,
-        "final_unique_mmsi": len(
-            {
-                key
-                for key in (_feature_mmsi_key(feature) for feature in features)
-                if key is not None
-            }
-        ),
+        "source": "vessel_current_state",
+        "current_state_count": len(features),
+        "final_unique_mmsi": unique_mmsi,
+        "raw_feature_count": 0,
+        "raw_unique_mmsi": 0,
+        "legacy_feature_count": 0,
+        "legacy_backfilled_mmsi": 0,
+        "legacy_fallback_enabled": False,
+        "legacy_fallback_used": False,
     }
 
 @router.get("/vessels/custom-supplements")
@@ -2696,6 +2702,8 @@ async def _build_bgp_collectors_geojson(db: AsyncSession) -> dict[str, Any]:
         db,
         source_filter=("ris_live_bgp", "bgpstream_bgp"),
     )
+    if not any(int(item.get("observation_count") or 0) > 0 for item in coverage):
+        return {"type": "FeatureCollection", "features": [], "count": 0}
     coverage_by_collector = {
         item["collector"]: item
         for item in coverage
@@ -2719,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
     )
 
 
+async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
+    kind = compute_site_type_expression(CollectedData)
+    rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
+        .where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
+               CollectedData.deleted_at.is_(None))
+        .group_by(CollectedData.source, kind))).all()
+    counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
+    present_sources = set()
+    for source, site_type, count in rows:
+        present_sources.add(source)
+        counts[site_type] += int(count or 0)
+    # Retain the existing last-task fallback for the two collected catalogs.
+    for source in COMPUTE_SOURCES:
+        if source not in present_sources:
+            key = "supercomputer" if source == "top500" else "gpu_cluster"
+            counts[key] += await _count_current_or_latest_task_data(db, source)
+    return counts
+
+
 async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
     cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
     landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
@@ -2727,15 +2754,16 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
         "celestrak_tle",
         exclude_unknown_name=True,
     )
-    supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
-    gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
-    compute_center_count = supercomputer_count + gpu_cluster_count
+    compute_counts = await _compute_facility_counts(db)
+    supercomputer_count = compute_counts["supercomputer"]
+    gpu_cluster_count = compute_counts["gpu_cluster"]
+    compute_center_count = sum(compute_counts.values())
 
     active_incident_result = await db.execute(
-        select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
+        select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
     )
     active_anomaly_result = await db.execute(
-        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
+        select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value),
     )
     active_incident_count = int(active_incident_result.scalar() or 0)
     active_anomaly_count = int(active_anomaly_result.scalar() or 0)
@@ -2756,21 +2784,14 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
         )
     else:
         bgp_collector_count = int(bgp_collector_scalar or 0)
-    raw_unique_window_hours = 24
-    raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
-        db,
-        observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
+    vessel_current_window_minutes = 60
+    vessel_current_result = await db.execute(
+        select(func.count(VesselCurrentState.mmsi)).where(
+            VesselCurrentState.observed_at
+            >= datetime.now(UTC) - timedelta(minutes=vessel_current_window_minutes)
+        )
     )
-    legacy_unique_result = await db.execute(
-        select(func.count(func.distinct(VesselPosition.mmsi)))
-    )
-    legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
-    legacy_fallback_active = (
-        VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED
-        and raw_unique_mmsi == 0
-        and legacy_unique_mmsi > 0
-    )
-    vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi
+    vessel_count = int(vessel_current_result.scalar() or 0)
     aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
 
     return {
@@ -2781,17 +2802,19 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
             "satellite_count": satellite_count,
             "compute_center_count": compute_center_count,
             "vessel_count": vessel_count,
-            "vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent",
-            "vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
-            "vessel_raw_unique_mmsi": raw_unique_mmsi,
-            "vessel_raw_unique_window_hours": raw_unique_window_hours,
-            "vessel_legacy_unique_mmsi": legacy_unique_mmsi,
+            "vessel_count_source": "vessel_current_state",
+            "vessel_current_window_minutes": vessel_current_window_minutes,
+            "vessel_raw_unique_mmsi": 0,
+            "vessel_legacy_unique_mmsi": 0,
             "aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
             "aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
             "aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,
             "aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
             "supercomputer_count": supercomputer_count,
             "gpu_cluster_count": gpu_cluster_count,
+            "data_center_count": compute_counts["data_center"],
+            "cloud_center_count": compute_counts["cloud_center"],
+            "edge_node_count": compute_counts["edge_node"],
             "bgp_event_count": active_incident_count or active_anomaly_count,
             "bgp_incident_count": active_incident_count,
             "bgp_anomaly_count": active_anomaly_count,

backend/app/api/v1/websocket.py

@@ -6,11 +6,15 @@ from typing import Optional
 
 from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Query
 from jose import jwt, JWTError
+from sqlalchemy import text
 
 from app.core.config import settings
+from app.core.enums import UserRole
 from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
+from app.db.session import async_session_factory
+from app.services.log_tail import LOG_TAIL_CHANNEL, log_tail_manager
 
 logger = get_logger(__name__, service="api")
 router = APIRouter()
@@ -37,6 +41,28 @@ async def authenticate_token(token: str) -> Optional[dict]:
         return None
 
 
+async def load_websocket_user_role(user_id: str | None) -> str | None:
+    if not user_id:
+        return None
+    try:
+        async with async_session_factory() as db:
+            result = await db.execute(
+                text("SELECT role, is_active FROM users WHERE id = :id"),
+                {"id": int(user_id)},
+            )
+            row = result.fetchone()
+    except Exception as exc:
+        logger.warning_event(
+            "WebSocket user role lookup failed",
+            event="auth.websocket.role_lookup_failed",
+            context={"user_id": user_id, "error": str(exc)},
+        )
+        return None
+    if row is None or not row[1]:
+        return None
+    return str(row[0] or "")
+
+
 @router.websocket("/ws")
 async def websocket_endpoint(
     websocket: WebSocket,
@@ -59,6 +85,7 @@ async def websocket_endpoint(
 
     is_anonymous = payload is None
     user_id = str(payload.get("sub")) if payload else f"anonymous:{id(websocket)}"
+    user_role = await load_websocket_user_role(user_id) if payload else None
     supported_channels = ["vessels", "earth_news", EARTH_UPDATES_CHANNEL] if is_anonymous else [
         "gpu_clusters",
         "submarine_cables",
@@ -70,6 +97,8 @@ async def websocket_endpoint(
         "earth_news",
         EARTH_UPDATES_CHANNEL,
     ]
+    if user_role == UserRole.SUPER_ADMIN.value:
+        supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
     await manager.connect(websocket, user_id)
 
     try:
@@ -100,6 +129,7 @@ async def websocket_endpoint(
                     payload_data = data.get("data", {})
                     if not isinstance(payload_data, dict):
                         payload_data = {}
+                    log_tail_config = None
                     channels = payload_data.get("channels", [])
                     if isinstance(channels, str):
                         channels = [channels]
@@ -108,10 +138,30 @@ async def websocket_endpoint(
                     channel = payload_data.get("channel")
                     if channel and channel not in channels:
                         channels = [*channels, channel]
+                    if LOG_TAIL_CHANNEL in channels:
+                        if user_role != UserRole.SUPER_ADMIN.value:
+                            await websocket.send_json(
+                                {
+                                    "type": "subscription_error",
+                                    "data": {"channel": LOG_TAIL_CHANNEL, "detail": "Only super_admin can subscribe logs"},
+                                }
+                            )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
+                        else:
+                            try:
+                                log_tail_config = await log_tail_manager.subscribe(websocket, payload_data)
+                            except ValueError as exc:
+                                await websocket.send_json(
+                                    {
+                                        "type": "subscription_error",
+                                        "data": {"channel": LOG_TAIL_CHANNEL, "detail": str(exc)},
+                                    }
+                                )
+                            channels = [item for item in channels if item != LOG_TAIL_CHANNEL]
                     if is_anonymous:
                         channels = [channel for channel in channels if channel in supported_channels]
                     vessel_subscription = None
-                    if "vessels" in channels and "bbox" in payload_data:
+                    if "vessels" in channels:
                         try:
                             vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
                         except ValueError as exc:
@@ -131,14 +181,20 @@ async def websocket_endpoint(
                                 "action": "subscribe",
                                 "channels": [
                                     *channels,
+                                    *([LOG_TAIL_CHANNEL] if log_tail_config else []),
                                     *(["vessels"] if vessel_subscription else []),
                                 ],
                                 "vessels": vessel_subscription,
+                                "logs_tail": log_tail_config.__dict__ if log_tail_config else None,
                             },
                         }
                     )
                 elif data.get("type") == "unsubscribe":
                     channels = data.get("data", {}).get("channels", [])
+                    if isinstance(channels, str):
+                        channels = [channels]
+                    if LOG_TAIL_CHANNEL in channels:
+                        await log_tail_manager.unsubscribe(websocket)
                     manager.unsubscribe(websocket, channels)
                     await websocket.send_json(
                         {
@@ -159,4 +215,5 @@ async def websocket_endpoint(
     except WebSocketDisconnect:
         pass
     finally:
+        await log_tail_manager.disconnect(websocket)
         manager.disconnect(websocket, user_id)

backend/app/core/compute_types.py

@@ -0,0 +1,35 @@
+"""Compute facility classification shared by the map and image resolver."""
+
+from sqlalchemy import case, or_
+
+COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
+COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
+
+
+def compute_site_type(record) -> str:
+    metadata = record.extra_data or {}
+    for value in (metadata.get("site_type"), record.data_type):
+        if value in COMPUTE_SITE_TYPES:
+            return value
+    return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
+        record.source, "data_center"
+    )
+
+
+def compute_site_type_expression(record):
+    declared = record.extra_data["site_type"].as_string()
+    return case(
+        (declared.in_(COMPUTE_SITE_TYPES), declared),
+        (record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
+        (record.source == "top500", "supercomputer"),
+        (record.source == "epoch_ai_gpu", "gpu_cluster"),
+        else_="data_center",
+    )
+
+
+def compute_record_filter(record):
+    return or_(
+        record.source.in_(COMPUTE_SOURCES),
+        record.data_type.in_(COMPUTE_SITE_TYPES),
+        record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
+    )

backend/app/core/config.py

@@ -41,6 +41,7 @@ class Settings(BaseSettings):
     AI_PROVIDER_SERVICE_TOKEN: str = ""
     AI_PROVIDER_TIMEOUT_SECONDS: int = 60
     AI_PROVIDER_RETRY_ATTEMPTS: int = 2
+    OBSERVABILITY_INGEST_TOKEN: str = ""
 
     @property
     def REDIS_URL(self) -> str:

backend/app/core/earth_geography_policy.py

@@ -0,0 +1,57 @@
+"""Country display ownership for Earth comparisons and cards.
+
+Keep source-country evidence separate; use this projection only for display
+grouping so regional source records cannot become competing national keys.
+"""
+
+from __future__ import annotations
+
+import json
+from functools import lru_cache
+from pathlib import Path
+
+from app.core.countries import COUNTRY_ENTRIES, normalize_country
+
+
+_DISPLAY_ISO_A3 = {
+    canonical: next(
+        (alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
+        None,
+    )
+    for canonical, aliases in COUNTRY_ENTRIES
+}
+_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
+_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
+
+
+@lru_cache(maxsize=1)
+def _policy_country_by_code() -> dict[str, str]:
+    try:
+        content = _POLICY_PATH.read_text(encoding="utf-8")
+    except FileNotFoundError as error:
+        raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
+    policy = json.loads(content)
+    if policy.get("profile") != "china-pov-v1":
+        raise ValueError("unsupported Earth display-policy profile")
+    mappings = {}
+    for rule in policy["rules"]:
+        target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
+        if target not in _COUNTRY_BY_ISO_A3:
+            continue
+        for code in rule.get("sourceIsoA3", []):
+            mappings[code] = _COUNTRY_BY_ISO_A3[target]
+    return mappings
+
+
+def display_country(value: str | None) -> str | None:
+    country = normalize_country(value)
+    code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
+    owner = _policy_country_by_code().get(code)
+    if owner:
+        return owner
+    return country
+
+
+def display_iso_a3(value: str | None) -> str | None:
+    country = display_country(value)
+    return _DISPLAY_ISO_A3.get(country) if country else None

backend/app/core/enums.py

@@ -0,0 +1,227 @@
+"""Stable backend protocol enums.
+
+Database columns and JSON payloads continue to store the enum string values.
+Configurable identifiers, user-authored values, and open-ended taxonomies do
+not belong in this module.
+"""
+
+from __future__ import annotations
+
+import logging
+from enum import StrEnum
+from typing import TypeVar
+
+logger = logging.getLogger(__name__)
+
+EnumT = TypeVar("EnumT", bound=StrEnum)
+
+
+def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT:
+    """Parse an external value without breaking reads of legacy data."""
+
+    if value is None or str(value).strip() == "":
+        return default
+    if isinstance(value, enum_type):
+        return value
+    try:
+        return enum_type(str(value).strip().lower())
+    except (TypeError, ValueError):
+        logger.warning(
+            "Unknown %s value %r; falling back to %s",
+            enum_type.__name__,
+            value,
+            default.value,
+        )
+        return default
+
+
+class NewsImportanceLevel(StrEnum):
+    LOW = "low"
+    MEDIUM = "medium"
+    HIGH = "high"
+    CRITICAL = "critical"
+
+
+class BreakingLevel(StrEnum):
+    NONE = "none"
+    WATCH = "watch"
+    BREAKING = "breaking"
+    CRITICAL = "critical"
+
+
+class BreakingScope(StrEnum):
+    REGIONAL = "regional"
+    GLOBAL = "global"
+
+
+class BreakingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+    MULTI_SOURCE = "multi_source"
+
+
+class NewsSourceType(StrEnum):
+    RSS = "rss"
+    ATOM = "atom"
+    AGGREGATED = "aggregated"
+    REFERENCE = "reference"
+    MANUAL = "manual"
+
+
+class NewsEnrichmentStatus(StrEnum):
+    PENDING = "pending"
+    QUEUED = "queued"
+    ATTEMPTED = "attempted"
+    SUCCESS = "success"
+    CONTENT_ONLY = "content_only"
+    LOCATION_ONLY = "location_only"
+    UNAVAILABLE = "unavailable"
+    PROVIDER_ERROR = "provider_error"
+    PARSE_ERROR = "parse_error"
+    NO_RESULT = "no_result"
+
+
+class NewsMarketImpact(StrEnum):
+    NONE = "none"
+    SECTOR = "sector"
+    NATIONAL = "national"
+    GLOBAL = "global"
+
+
+class NewsTaggingSource(StrEnum):
+    RULES = "rules"
+    AI = "ai"
+    MANUAL = "manual"
+
+
+class JobType(StrEnum):
+    COLLECT = "collect"
+    CLEAR_DATA = "clear_data"
+    CLEAR_CACHE = "clear_cache"
+    EARTH_REFRESH = "earth_refresh"
+
+
+class JobStatus(StrEnum):
+    QUEUED = "queued"
+    RUNNING = "running"
+    CANCELLING = "cancelling"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class RollbackPolicy(StrEnum):
+    KEEP_COMMITTED_BATCHES = "keep_committed_batches"
+
+
+class MappingValidationStatus(StrEnum):
+    DRAFT = "draft"
+    VALID = "valid"
+    INVALID = "invalid"
+
+
+class SnapshotStatus(StrEnum):
+    RUNNING = "running"
+    SUCCESS = "success"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+
+
+class DatasourceRunStatus(StrEnum):
+    RUNNING = "running"
+    NOT_RUN = "not_run"
+    COLLECTED = "collected"
+    UNCOLLECTED = "uncollected"
+
+
+class ProviderApi(StrEnum):
+    ANTHROPIC_MESSAGES = "anthropic-messages"
+    OPENAI_COMPLETIONS = "openai-completions"
+    OPENAI_RESPONSES = "openai-responses"
+    OLLAMA_GENERATE = "ollama-generate"
+
+
+class PlaygroundMessageRole(StrEnum):
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+    TOOL = "tool"
+
+
+class PlaygroundMessageKind(StrEnum):
+    MESSAGE = "message"
+    THINKING = "thinking"
+    ERROR = "error"
+    STATUS = "status"
+
+
+class PlaygroundMessageStatus(StrEnum):
+    PENDING = "pending"
+    THINKING = "thinking"
+    ANSWERING = "answering"
+    DONE = "done"
+    FAILED = "failed"
+    CANCELLED = "cancelled"
+    ERROR = "error"
+    STOPPED = "stopped"
+
+
+class OtpPurpose(StrEnum):
+    REGISTER = "register"
+    VERIFY_EMAIL = "verify_email"
+    RESET_PASSWORD = "reset_password"
+
+
+class UserRole(StrEnum):
+    VIEWER = "viewer"
+    ADMIN = "admin"
+    SUPER_ADMIN = "super_admin"
+
+
+class AlertSeverity(StrEnum):
+    CRITICAL = "critical"
+    WARNING = "warning"
+    INFO = "info"
+
+
+class AlertStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class BGPStatus(StrEnum):
+    ACTIVE = "active"
+    ACKNOWLEDGED = "acknowledged"
+    RESOLVED = "resolved"
+
+
+class LogLevel(StrEnum):
+    ALL = "all"
+    ERROR = "error"
+    WARNING = "warning"
+    INFO = "info"
+    DEBUG = "debug"
+
+
+class ConnectionState(StrEnum):
+    DISCONNECTED = "disconnected"
+    CONNECTING = "connecting"
+    CONNECTED = "connected"
+    ERROR = "error"
+
+
+class AuthType(StrEnum):
+    NONE = "none"
+    BEARER = "bearer"
+    API_KEY = "api_key"
+    BASIC = "basic"
+
+
+class TVSourceType(StrEnum):
+    IFRAME = "iframe"
+    HLS = "hls"
+    VIDEO = "video"
+    EXTERNAL = "external"
+    YOUTUBE = "youtube"

backend/app/core/satellite_presentation.py

@@ -0,0 +1,13 @@
+"""Shared source-group normalization; never infer hardware generations."""
+
+
+def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
+    group = str(raw_group or "").strip().lower()
+    if group:
+        return group
+    normalized_name = str(name or "").strip().upper()
+    if normalized_name.startswith("STARLINK"):
+        return "starlink"
+    if normalized_name.startswith("IRIDIUM"):
+        return "iridium-next"
+    return None

backend/app/core/security.py

@@ -6,7 +6,7 @@ import redis
 from fastapi import Depends, HTTPException, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from jose import JWTError, jwt
-from sqlalchemy import text
+from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
         return None
 
 
+async def _load_active_user(user_id: str, db: AsyncSession) -> User:
+    try:
+        parsed_id = int(user_id)
+    except (TypeError, ValueError):
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
+    result = await db.execute(select(User).where(User.id == parsed_id))
+    user = result.scalar_one_or_none()
+    if user is None or not user.is_active:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="User not found or inactive",
+        )
+    return user
+
+
 async def get_current_user(
     credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
     db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
             status_code=status.HTTP_401_UNAUTHORIZED,
             detail="Invalid token",
         )
-    result = await db.execute(
-        text(
-            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
-        ),
-        {"id": int(user_id)},
-    )
-    row = result.fetchone()
-    if row is None or not row[5]:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="User not found or inactive",
-        )
-    user = User()
-    user.id = row[0]
-    user.username = row[1]
-    user.email = row[2]
-    user.password_hash = row[3]
-    user.role = row[4]
-    user.is_active = row[5]
-    user.gatekeeper_groups = row[6] or []
-    return user
+    return await _load_active_user(user_id, db)
 
 
 async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
             status_code=status.HTTP_401_UNAUTHORIZED,
             detail="Invalid token",
         )
-    result = await db.execute(
-        text(
-            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
-        ),
-        {"id": int(user_id)},
-    )
-    row = result.fetchone()
-    if row is None or not row[5]:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="User not found or inactive",
-        )
-    user = User()
-    user.id = row[0]
-    user.username = row[1]
-    user.email = row[2]
-    user.password_hash = row[3]
-    user.role = row[4]
-    user.is_active = row[5]
-    user.gatekeeper_groups = row[6] or []
-    return user
+    return await _load_active_user(user_id, db)
 
 
 def blacklist_token(token: str) -> None:

backend/app/core/websocket/broadcaster.py

@@ -4,11 +4,14 @@ import asyncio
 from datetime import UTC, datetime
 from typing import Dict, Any
 
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
 
 EARTH_UPDATES_CHANNEL = "earth_updates"
+VESSEL_STATE_QUERY_BATCH_SIZE = 1000
+logger = get_logger(__name__, service="websocket")
 
 
 class DataBroadcaster:
@@ -114,16 +117,22 @@ class DataBroadcaster:
             return
         pending = self._pending_vessel_updates
         self._pending_vessel_updates = {}
-        vessels = []
-        for item in pending.values():
-            vessel = dict(item)
-            source = vessel.pop("_source", None)
-            action = vessel.pop("_action", "upsert")
-            created = vessel.pop("_created", None)
-            vessel["source"] = source
-            vessel["action"] = action
-            vessel["created"] = created
-            vessels.append(vessel)
+        try:
+            vessels = await self._load_current_vessel_updates(list(pending))
+        except Exception:
+            # Preserve newer updates that arrived during the failed database read.
+            self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
+            raise
+        if not vessels:
+            return
+        vessels = [
+            {
+                **vessel,
+                "action": vessel.get("action", "upsert"),
+                "created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
+            }
+            for vessel in vessels
+        ]
         await manager.broadcast_vessels(
             {
                 "action": "upsert",
@@ -133,12 +142,31 @@ class DataBroadcaster:
             }
         )
 
+    async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
+        from app.db.session import async_session_factory
+        from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
+
+        mmsis = [int(key) for key in keys if key.isdigit()]
+        vessels = []
+        async with async_session_factory() as db:
+            for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
+                vessels.extend(await get_current_vessels_by_mmsi(
+                    db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
+                ))
+        present = {int(vessel["mmsi"]) for vessel in vessels}
+        vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
+        return vessels
+
     async def broadcast_vessels_periodically(self):
         while self.running:
             try:
                 await self.flush_vessel_updates()
-            except Exception:
-                pass
+            except Exception as exc:
+                logger.exception_event(
+                    "Failed to flush vessel updates",
+                    event="vessels.broadcast.failed",
+                    context={"error": str(exc)},
+                )
             await asyncio.sleep(self._vessel_flush_interval)
 
     async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
@@ -150,7 +178,7 @@ class DataBroadcaster:
                 "timestamp": to_iso8601_utc(datetime.now(UTC)),
                 "payload": data,
             },
-            channel="all",
+            channel="datasource_tasks",
         )
 
     def start(self):

backend/app/core/websocket/manager.py

@@ -63,6 +63,8 @@ class ConnectionManager:
 
     def unsubscribe(self, websocket: WebSocket, channels: list[str]):
         for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
+            if channel == "vessels":
+                self.vessel_subscriptions.pop(websocket, None)
             subscribers = self.channel_subscriptions.get(channel)
             if subscribers is not None:
                 subscribers.discard(websocket)
@@ -88,7 +90,8 @@ class ConnectionManager:
         return subscription
 
     def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
-        bbox = config.get("bbox")
+        global_scope = config.get("scope") == "global"
+        bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
         if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
             raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
         try:
@@ -103,7 +106,7 @@ class ConnectionManager:
             raise ValueError("bbox longitude values must be between -180 and 180")
         if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
             raise ValueError("bbox latitude values must be between -90 and 90")
-        if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
+        if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
             raise ValueError("bbox is too large; zoom in or request a smaller viewport")
 
         zoom = int(config.get("zoom") or 1)
@@ -116,10 +119,11 @@ class ConnectionManager:
             if str(item).strip()
         }
         return {
+            "scope": "global" if global_scope else "viewport",
             "bbox": (lon_min, lat_min, lon_max, lat_max),
             "zoom": zoom,
             "limit": limit,
-            "type": vessel_types,
+            "type": sorted(vessel_types),
             "last_sent_at": None,
         }
 
@@ -152,7 +156,9 @@ class ConnectionManager:
                 vessel
                 for vessel in vessels
                 if self._vessel_matches_subscription(vessel, subscription)
-            ][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
+            ]
+            if subscription.get("scope") != "global":
+                matched = matched[:subscription["limit"]]
             if not matched:
                 continue
             subscription["last_sent_at"] = datetime.now(UTC)
@@ -162,16 +168,24 @@ class ConnectionManager:
                 "timestamp": subscription["last_sent_at"].isoformat(),
                 "payload": {
                     **data,
-                    "vessels": matched,
+                    "vessels": [],
                     "subscription": {
                         "bbox": list(subscription["bbox"]),
                         "zoom": subscription["zoom"],
                         "limit": subscription["limit"],
+                        "scope": subscription.get("scope", "viewport"),
                     },
                 },
             }
             try:
-                await connection.send_json(message)
+                for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
+                    await connection.send_json({
+                        **message,
+                        "payload": {
+                            **message["payload"],
+                            "vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
+                        },
+                    })
             except Exception:
                 self.unsubscribe_all(connection)
 
@@ -180,6 +194,8 @@ class ConnectionManager:
         vessel: dict[str, Any],
         subscription: dict[str, Any],
     ) -> bool:
+        if vessel.get("action") == "remove" and subscription.get("scope") == "global":
+            return True
         try:
             lon = float(vessel.get("lon"))
             lat = float(vessel.get("lat"))

backend/app/db/session.py

@@ -203,6 +203,7 @@ async def init_db():
     import app.models.vessel_enrichment  # noqa: F401
     import app.models.datasource_mapping  # noqa: F401
     import app.models.earth_news  # noqa: F401
+    import app.models.earth_interactable  # noqa: F401
 
     logger.warning_event(
         "Database pool settings active",
@@ -258,6 +259,407 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE TABLE IF NOT EXISTS earth_data_change_events (
+                    id BIGSERIAL PRIMARY KEY,
+                    table_name VARCHAR(128) NOT NULL,
+                    operation VARCHAR(16) NOT NULL,
+                    source VARCHAR(128),
+                    entity_key VARCHAR(255),
+                    payload JSONB NOT NULL,
+                    occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                    consumed_at TIMESTAMPTZ
+                )
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_data_change_events_unconsumed
+                ON earth_data_change_events (consumed_at, id)
+                WHERE consumed_at IS NULL
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_earth_data_changed_statement(
+                    change_table TEXT,
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                DECLARE
+                    change_event_id BIGINT;
+                    change_payload JSONB;
+                BEGIN
+                    change_payload := jsonb_build_object(
+                        'event', 'earth.layer.changed',
+                        'table', change_table,
+                        'operation', change_operation,
+                        'source', change_source,
+                        'entity_key', NULL,
+                        'entity_keys', COALESCE(to_jsonb(source_entity_keys), '[]'::jsonb),
+                        'records_processed', COALESCE(source_record_count, 0),
+                        'occurred_at', NOW()
+                    );
+
+                    INSERT INTO earth_data_change_events (
+                        table_name,
+                        operation,
+                        source,
+                        entity_key,
+                        payload,
+                        occurred_at
+                    ) VALUES (
+                        change_table,
+                        change_operation,
+                        change_source,
+                        NULL,
+                        change_payload,
+                        NOW()
+                    )
+                    RETURNING id INTO change_event_id;
+
+                    change_payload := change_payload || jsonb_build_object(
+                        'event_id', change_event_id
+                    );
+
+                    UPDATE earth_data_change_events
+                    SET payload = change_payload
+                    WHERE id = change_event_id;
+
+                    PERFORM pg_notify(
+                        'planet_earth_data_changes',
+                        change_payload::text
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_emit_collected_data_changed_statement(
+                    change_operation TEXT,
+                    change_source TEXT,
+                    source_record_count INTEGER,
+                    source_entity_keys TEXT[]
+                )
+                RETURNS VOID AS $$
+                BEGIN
+                    PERFORM planet_emit_earth_data_changed_statement(
+                        'collected_data',
+                        change_operation,
+                        change_source,
+                        source_record_count,
+                        source_entity_keys
+                    );
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_earth_table_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM new_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (SELECT to_jsonb(t) AS row_data FROM old_rows AS t) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) changed_rows
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(
+                                        NULLIF(row_data->>'entity_key', ''),
+                                        NULLIF(row_data->>'source_id', ''),
+                                        NULLIF(row_data->>'incident_key', ''),
+                                        NULLIF(row_data->>'id', ''),
+                                        NULLIF(row_data->>'mmsi', '')
+                                    )
+                                    FROM (
+                                        SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                        UNION ALL
+                                        SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                                    ) rows_for_keys
+                                    WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT to_jsonb(t) AS row_data FROM new_rows AS t
+                                UNION ALL
+                                SELECT to_jsonb(t) AS row_data FROM old_rows AS t
+                            ) rows_for_count
+                            WHERE COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME) = change_source;
+
+                            PERFORM planet_emit_earth_data_changed_statement(
+                                TG_TABLE_NAME,
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE OR REPLACE FUNCTION planet_notify_collected_data_changed_statement()
+                RETURNS trigger AS $$
+                DECLARE
+                    change_source TEXT;
+                    source_record_count INTEGER;
+                    source_entity_keys TEXT[];
+                BEGIN
+                    IF TG_OP = 'INSERT' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM new_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM new_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM new_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'DELETE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM old_rows WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM old_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM old_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    ELSIF TG_OP = 'UPDATE' THEN
+                        FOR change_source IN
+                            SELECT DISTINCT source FROM (
+                                SELECT source FROM new_rows
+                                UNION
+                                SELECT source FROM old_rows
+                            ) changed_sources
+                            WHERE source IS NOT NULL
+                        LOOP
+                            SELECT
+                                COUNT(*),
+                                ARRAY(
+                                    SELECT DISTINCT COALESCE(entity_key, source_id, id::text)
+                                    FROM (
+                                        SELECT id, source_id, entity_key, source FROM new_rows
+                                        UNION ALL
+                                        SELECT id, source_id, entity_key, source FROM old_rows
+                                    ) changed_rows
+                                    WHERE source = change_source
+                                    LIMIT 20
+                                )
+                            INTO source_record_count, source_entity_keys
+                            FROM (
+                                SELECT id, source_id, entity_key, source FROM new_rows
+                                UNION ALL
+                                SELECT id, source_id, entity_key, source FROM old_rows
+                            ) changed_rows
+                            WHERE source = change_source;
+
+                            PERFORM planet_emit_collected_data_changed_statement(
+                                TG_OP,
+                                change_source,
+                                source_record_count,
+                                source_entity_keys
+                            );
+                        END LOOP;
+                    END IF;
+
+                    RETURN NULL;
+                END;
+                $$ LANGUAGE plpgsql;
+                """
+            )
+        )
+        for statement in (
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_insert ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_update ON collected_data",
+            "DROP TRIGGER IF EXISTS tr_planet_collected_data_changed_delete ON collected_data",
+            "DROP FUNCTION IF EXISTS planet_notify_collected_data_changed()",
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_insert
+            AFTER INSERT ON collected_data
+            REFERENCING NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_update
+            AFTER UPDATE ON collected_data
+            REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+            """
+            CREATE TRIGGER tr_planet_collected_data_changed_delete
+            AFTER DELETE ON collected_data
+            REFERENCING OLD TABLE AS old_rows
+            FOR EACH STATEMENT
+            EXECUTE FUNCTION planet_notify_collected_data_changed_statement()
+            """,
+        ):
+            await conn.execute(text(statement))
+        for table_name in (
+            "bgp_observations",
+            "bgp_anomalies",
+            "bgp_incidents",
+            "bgp_collector_locations",
+            "vessel_static",
+            "vessel_position",
+            "vessel_current_state",
+            "ais_raw_observations",
+            "ais_source_health",
+            "compute_center_locations",
+            "earth_interactables",
+            "earth_news_items",
+        ):
+            for statement in (
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_insert ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_update ON {table_name}",
+                f"DROP TRIGGER IF EXISTS tr_planet_{table_name}_changed_delete ON {table_name}",
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_insert
+                AFTER INSERT ON {table_name}
+                REFERENCING NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_update
+                AFTER UPDATE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+                f"""
+                CREATE TRIGGER tr_planet_{table_name}_changed_delete
+                AFTER DELETE ON {table_name}
+                REFERENCING OLD TABLE AS old_rows
+                FOR EACH STATEMENT
+                EXECUTE FUNCTION planet_notify_earth_table_changed_statement()
+                """,
+            ):
+                await conn.execute(text(statement))
         await conn.execute(
             text(
                 """
@@ -267,7 +669,16 @@ async def init_db():
                 ADD COLUMN IF NOT EXISTS phase_message VARCHAR(255),
                 ADD COLUMN IF NOT EXISTS phase_current BIGINT,
                 ADD COLUMN IF NOT EXISTS phase_total BIGINT,
-                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30)
+                ADD COLUMN IF NOT EXISTS phase_unit VARCHAR(30),
+                ADD COLUMN IF NOT EXISTS source VARCHAR(100),
+                ADD COLUMN IF NOT EXISTS task_type VARCHAR(30) NOT NULL DEFAULT 'collect',
+                ADD COLUMN IF NOT EXISTS payload JSONB NOT NULL DEFAULT '{}'::jsonb,
+                ADD COLUMN IF NOT EXISTS rollback_policy VARCHAR(40) NOT NULL DEFAULT 'keep_committed_batches',
+                ADD COLUMN IF NOT EXISTS dedupe_key VARCHAR(180),
+                ADD COLUMN IF NOT EXISTS worker_id VARCHAR(120),
+                ADD COLUMN IF NOT EXISTS locked_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS requested_cancel_at TIMESTAMPTZ,
+                ADD COLUMN IF NOT EXISTS cancel_reason TEXT
                 """
             )
         )
@@ -283,6 +694,17 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                ALTER TABLE earth_interactables
+                ADD COLUMN IF NOT EXISTS altitude DOUBLE PRECISION,
+                ADD COLUMN IF NOT EXISTS revision INTEGER NOT NULL DEFAULT 1,
+                ADD COLUMN IF NOT EXISTS is_deleted BOOLEAN NOT NULL DEFAULT FALSE,
+                ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ
+                """
+            )
+        )
         await conn.execute(
             text(
                 """
@@ -307,6 +729,48 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_layer_deleted
+                ON earth_interactables (layer, is_deleted)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_earth_interactables_updated_at
+                ON earth_interactables (updated_at)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_source_status
+                ON collection_tasks (source, status)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_queue
+                ON collection_tasks (status, created_at, id)
+                WHERE status = 'queued'
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_collection_tasks_dedupe
+                ON collection_tasks (dedupe_key)
+                WHERE dedupe_key IS NOT NULL
+                """
+            )
+        )
         await conn.execute(
             text(
                 """
@@ -323,6 +787,22 @@ async def init_db():
                 """
             )
         )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox
+                ON vessel_current_state (lon, lat)
+                """
+            )
+        )
+        await conn.execute(
+            text(
+                """
+                CREATE INDEX IF NOT EXISTS idx_vessel_current_observed
+                ON vessel_current_state (observed_at DESC)
+                """
+            )
+        )
         await conn.execute(
             text(
                 """

backend/app/main.py

@@ -24,6 +24,11 @@ from app.services.earth_news_worker import (
     start_earth_news_target_worker,
     stop_earth_news_target_worker,
 )
+from app.services.earth_db_change_listener import (
+    start_earth_db_change_listener,
+    stop_earth_db_change_listener,
+)
+from app.services.data_jobs import start_data_job_worker, stop_data_job_worker
 
 
 configure_logging()
@@ -59,9 +64,13 @@ async def lifespan(app: FastAPI):
     start_scheduler()
     await sync_scheduler_with_datasources()
     broadcaster.start()
+    start_data_job_worker()
+    start_earth_db_change_listener()
     start_earth_news_target_worker()
     yield
     await stop_earth_news_target_worker()
+    await stop_earth_db_change_listener()
+    await stop_data_job_worker()
     broadcaster.stop()
     stop_scheduler()
 

backend/app/models/__init__.py

@@ -13,10 +13,11 @@ from app.models.compute_center_location import ComputeCenterLocationRecord
 from app.models.system_setting import SystemSetting
 from app.models.playground_session import PlaygroundSession
 from app.models.playground_message import PlaygroundMessage
-from app.models.system_log import SystemLog, AuditLog
+from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
 from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth, VesselPosition, VesselStatic
 from app.models.datasource_mapping import DataSourceMappingTemplate
 from app.models.earth_news import EarthNewsItem
+from app.models.earth_interactable import EarthInteractable
 
 __all__ = [
     "User",
@@ -36,6 +37,8 @@ __all__ = [
     "ComputeCenterLocationRecord",
     "SystemLog",
     "AuditLog",
+    "ObservabilityEvent",
+    "ObservabilityEventGroup",
     "PlaygroundSession",
     "PlaygroundMessage",
     "VesselPosition",
@@ -45,4 +48,5 @@ __all__ = [
     "AISSourceHealth",
     "DataSourceMappingTemplate",
     "EarthNewsItem",
+    "EarthInteractable",
 ]

backend/app/models/alert.py

@@ -1,26 +1,12 @@
 from datetime import datetime
-from enum import Enum
-from typing import Optional
 
-from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
-from sqlalchemy.orm import relationship
+from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum
 
+from app.core.enums import AlertSeverity, AlertStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
 
-class AlertSeverity(str, Enum):
-    CRITICAL = "critical"
-    WARNING = "warning"
-    INFO = "info"
-
-
-class AlertStatus(str, Enum):
-    ACTIVE = "active"
-    ACKNOWLEDGED = "acknowledged"
-    RESOLVED = "resolved"
-
-
 class Alert(Base):
     __tablename__ = "alerts"
 

backend/app/models/bgp_anomaly.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -17,7 +18,7 @@ class BGPAnomaly(Base):
     source = Column(String(100), nullable=False, index=True)
     anomaly_type = Column(String(50), nullable=False, index=True)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     entity_key = Column(String(255), nullable=False, index=True)
     prefix = Column(String(64), nullable=True, index=True)
     origin_asn = Column(Integer, nullable=True, index=True)

backend/app/models/bgp_incident.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
 
+from app.core.enums import BGPStatus
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -20,7 +21,7 @@ class BGPIncident(Base):
     title = Column(String(255), nullable=False)
     summary = Column(Text, nullable=False)
     severity = Column(String(20), nullable=False, index=True)
-    status = Column(String(20), nullable=False, default="active", index=True)
+    status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
     confidence = Column(Float, nullable=False, default=0.5)
     started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
     ended_at = Column(DateTime(timezone=True), nullable=True)

backend/app/models/data_snapshot.py

@@ -1,6 +1,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import SnapshotStatus
 from app.db.session import Base
 
 
@@ -16,7 +17,7 @@ class DataSnapshot(Base):
     started_at = Column(DateTime(timezone=True), server_default=func.now())
     completed_at = Column(DateTime(timezone=True), nullable=True)
     record_count = Column(Integer, default=0)
-    status = Column(String(20), nullable=False, default="running")
+    status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value)
     is_current = Column(Boolean, default=True, index=True)
     parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
     summary = Column(JSON, default={})

backend/app/models/datasource_mapping.py

@@ -3,6 +3,7 @@
 from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import MappingValidationStatus
 from app.db.session import Base
 
 
@@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base):
     target_schema = Column(String(80), nullable=False, index=True)
     mapping_json = Column(JSON, nullable=False, default={})
     sample_payload_hash = Column(String(64), nullable=True)
-    validation_status = Column(String(30), nullable=False, default="draft")
+    validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value)
     version = Column(Integer, nullable=False, default=1)
     is_active = Column(Boolean, nullable=False, default=False, index=True)
     created_at = Column(DateTime(timezone=True), server_default=func.now())

backend/app/models/earth_interactable.py

@@ -0,0 +1,30 @@
+"""Persistent Earth interactable objects."""
+
+from sqlalchemy import Boolean, Column, DateTime, Float, Index, Integer, JSON, String, Text
+from sqlalchemy.sql import func
+
+from app.db.session import Base
+
+
+class EarthInteractable(Base):
+    __tablename__ = "earth_interactables"
+
+    id = Column(String(160), primary_key=True)
+    layer = Column(String(80), nullable=False, default="interactables", index=True)
+    kind = Column(String(80), nullable=False, default="default", index=True)
+    label = Column(String(255), nullable=False, default="")
+    description = Column(Text, nullable=False, default="")
+    latitude = Column(Float, nullable=False)
+    longitude = Column(Float, nullable=False)
+    altitude = Column(Float, nullable=True)
+    revision = Column(Integer, nullable=False, default=1)
+    properties = Column(JSON, nullable=False, default=dict)
+    is_deleted = Column(Boolean, nullable=False, default=False, index=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+    deleted_at = Column(DateTime(timezone=True), nullable=True, index=True)
+
+    __table_args__ = (
+        Index("idx_earth_interactables_layer_deleted", "layer", "is_deleted"),
+        Index("idx_earth_interactables_updated_at", "updated_at"),
+    )

backend/app/models/playground_message.py

@@ -1,6 +1,7 @@
 from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus
 from app.db.session import Base
 
 
@@ -13,8 +14,8 @@ class PlaygroundMessage(Base):
     user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
     parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
     role = Column(String(20), nullable=False)
-    kind = Column(String(20), nullable=False, default="message")
-    status = Column(String(20), nullable=False, default="done")
+    kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value)
+    status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value)
     title = Column(String(255), nullable=True)
     content = Column(Text, nullable=False, default="")
     thinking_content = Column(Text, nullable=False, default="")

backend/app/models/system_log.py

@@ -38,3 +38,46 @@ class AuditLog(Base):
     ip = Column(String(64), nullable=True)
     details = Column(JSON, nullable=False, default=dict)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEvent(Base):
+    __tablename__ = "observability_events"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    occurred_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    level = Column(String(20), nullable=False, index=True)
+    message = Column(Text, nullable=False)
+    fingerprint = Column(String(80), nullable=False, index=True)
+    request_id = Column(String(64), nullable=True, index=True)
+    trace_id = Column(String(64), nullable=True, index=True)
+    task_id = Column(String(120), nullable=True, index=True)
+    source_ref_id = Column(String(120), nullable=True, index=True)
+    provider = Column(String(120), nullable=True, index=True)
+    user_id = Column(Integer, nullable=True, index=True)
+    context = Column(JSON, nullable=False, default=dict)
+    occurrence_count = Column(Integer, nullable=False, default=1)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+
+
+class ObservabilityEventGroup(Base):
+    __tablename__ = "observability_event_groups"
+
+    fingerprint = Column(String(80), primary_key=True)
+    source = Column(String(50), nullable=False, index=True)
+    service = Column(String(50), nullable=True, index=True)
+    module = Column(String(120), nullable=True, index=True)
+    category = Column(String(80), nullable=True, index=True)
+    event = Column(String(160), nullable=True, index=True)
+    last_level = Column(String(20), nullable=False, index=True)
+    sample_message = Column(Text, nullable=False)
+    sample_detail = Column(Text, nullable=True)
+    affected_sources = Column(JSON, nullable=False, default=list)
+    count = Column(Integer, nullable=False, default=0)
+    first_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    last_seen_at = Column(DateTime(timezone=True), nullable=False, index=True)
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())

backend/app/models/task.py

@@ -1,8 +1,9 @@
-"""Collection Task model"""
+"""Datasource job model."""
 
-from sqlalchemy import BigInteger, Column, DateTime, Integer, String, Text, Float
+from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
 from sqlalchemy.sql import func
 
+from app.core.enums import JobStatus, JobType, RollbackPolicy
 from app.db.session import Base
 
 
@@ -11,8 +12,10 @@ class CollectionTask(Base):
 
     id = Column(Integer, primary_key=True, autoincrement=True)
     datasource_id = Column(Integer, nullable=False, index=True)
-    status = Column(String(20), nullable=False)  # pending, running, success, failed, cancelled
-    phase = Column(String(30), default="queued")
+    source = Column(String(100), nullable=True, index=True)
+    task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True)
+    status = Column(String(20), nullable=False)  # queued, running, cancelling, success, failed, cancelled
+    phase = Column(String(30), default=JobStatus.QUEUED.value)
     phase_progress = Column(Float)
     phase_message = Column(String(255))
     phase_current = Column(BigInteger)
@@ -24,6 +27,13 @@ class CollectionTask(Base):
     total_records = Column(Integer, default=0)  # Total records to process
     progress = Column(Float, default=0.0)  # Progress percentage (0-100)
     error_message = Column(Text)
+    payload = Column(JSON, default=dict)
+    rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value)
+    dedupe_key = Column(String(180), nullable=True, index=True)
+    worker_id = Column(String(120), nullable=True, index=True)
+    locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
+    requested_cancel_at = Column(DateTime(timezone=True), nullable=True)
+    cancel_reason = Column(Text)
     created_at = Column(DateTime(timezone=True), server_default=func.now())
 
     def __repr__(self):

backend/app/models/user.py

@@ -1,6 +1,7 @@
 from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String
 from sqlalchemy.sql import func
 
+from app.core.enums import UserRole
 from app.db.session import Base
 
 
@@ -11,7 +12,7 @@ class User(Base):
     username = Column(String(50), unique=True, index=True, nullable=False)
     email = Column(String(255), unique=True, index=True, nullable=False)
     password_hash = Column(String(255), nullable=False)
-    role = Column(String(20), default="viewer")
+    role = Column(String(20), default=UserRole.VIEWER.value)
     gatekeeper_groups = Column(JSON, default=list)
     is_active = Column(Boolean, default=True)
     email_verified = Column(Boolean, default=False, nullable=False)

backend/app/models/vessel.py

@@ -3,6 +3,7 @@
 from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
 from sqlalchemy.sql import func
 
+from app.core.enums import ConnectionState
 from app.core.time import to_iso8601_utc
 from app.db.session import Base
 
@@ -75,6 +76,67 @@ class VesselPosition(Base):
         }
 
 
+class VesselCurrentState(Base):
+    """Latest renderable state for one vessel, independent from AIS history."""
+
+    __tablename__ = "vessel_current_state"
+
+    mmsi = Column(BigInteger, primary_key=True)
+    lat = Column(Float, nullable=False)
+    lon = Column(Float, nullable=False)
+    sog = Column(Float, nullable=True)
+    cog = Column(Float, nullable=True)
+    heading = Column(SmallInteger, nullable=True)
+    nav_status = Column(SmallInteger, nullable=True, index=True)
+    name = Column(String(128), nullable=True)
+    callsign = Column(String(16), nullable=True)
+    vessel_type = Column(SmallInteger, nullable=True, index=True)
+    vessel_type_name = Column(String(64), nullable=True, index=True)
+    flag = Column(String(4), nullable=True, index=True)
+    length = Column(Float, nullable=True)
+    width = Column(Float, nullable=True)
+    draught = Column(Float, nullable=True)
+    imo = Column(BigInteger, nullable=True)
+    source = Column(String(100), nullable=False, index=True)
+    observed_at = Column(DateTime(timezone=True), nullable=False)
+    field_sources = Column(JSON, default=dict)
+    selected_reasons = Column(JSON, default=dict)
+    source_summary = Column(JSON, default=dict)
+    quality_flags = Column(JSON, default=list)
+    updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
+
+    __table_args__ = (
+        Index("idx_vessel_current_bbox", "lon", "lat"),
+        Index("idx_vessel_current_observed", "observed_at"),
+    )
+
+    def to_dict(self) -> dict:
+        return {
+            "mmsi": self.mmsi,
+            "lat": self.lat,
+            "lon": self.lon,
+            "sog": self.sog,
+            "cog": self.cog,
+            "heading": self.heading,
+            "nav_status": self.nav_status,
+            "name": self.name,
+            "callsign": self.callsign,
+            "vessel_type": self.vessel_type,
+            "vessel_type_name": self.vessel_type_name,
+            "flag": self.flag,
+            "length": self.length,
+            "width": self.width,
+            "draught": self.draught,
+            "imo": self.imo,
+            "source": self.source,
+            "received_at": self.observed_at,
+            "field_sources": self.field_sources or {},
+            "selected_reasons": self.selected_reasons or {},
+            "source_summary": self.source_summary or {},
+            "quality_flags": self.quality_flags or [],
+        }
+
+
 class AISRawObservation(Base):
     """Source-level AIS fact before aggregation and conflict resolution."""
 
@@ -165,7 +227,7 @@ class AISSourceHealth(Base):
     __tablename__ = "ais_source_health"
 
     source = Column(String(100), primary_key=True)
-    connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
+    connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True)
     last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
     last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
     last_error = Column(String(500), nullable=True)

backend/app/schemas/ai.py

@@ -2,6 +2,7 @@ from typing import Any
 
 from pydantic import BaseModel, Field
 
+from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus
 
 class AIContentBlock(BaseModel):
     type: str
@@ -110,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel):
 
 class PlaygroundMessageRecord(BaseModel):
     id: str
-    role: str
-    kind: str = "message"
-    status: str = "done"
+    role: PlaygroundMessageRole
+    kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE
+    status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE
     title: str | None = None
     content: str = ""
     thinking_content: str = ""

backend/app/schemas/card_images.py

@@ -0,0 +1,128 @@
+"""Typed, validated configuration for optional object-card images."""
+
+from typing import Literal, Self
+from urllib.parse import urlsplit
+
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+
+EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
+MatchScope = Literal["object", "model", "group", "category"]
+
+
+def safe_url(value: str, *, local: bool = False) -> str:
+    if not value:
+        return value
+    if any(ord(char) < 32 for char in value) or "\\" in value:
+        raise ValueError("Invalid URL")
+    parts = urlsplit(value)
+    if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
+        if (
+            ".." not in parts.path
+            and "%" not in parts.path
+            and not parts.netloc
+            and not parts.query
+            and not parts.fragment
+        ):
+            return value
+    if (
+        parts.scheme in {"http", "https"}
+        and parts.hostname
+        and not parts.username
+        and not parts.password
+    ):
+        return value
+    raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
+
+
+class CardImageAsset(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+    id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
+    title: str = Field(min_length=1, max_length=200)
+    url: str = Field(min_length=1, max_length=2000)
+    variants: dict[str, str] = Field(default_factory=dict, max_length=4)
+    width: int = Field(default=0, ge=0, le=20000)
+    height: int = Field(default=0, ge=0, le=20000)
+    credit: str = Field(default="", max_length=500)
+    source_url: str = Field(default="", max_length=2000)
+    license: str = Field(default="", max_length=300)
+    license_url: str = Field(default="", max_length=2000)
+    kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
+    rights_status: Literal["available", "permission_pending"] = "available"
+    enabled: bool = True
+    focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
+    focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
+
+    @field_validator("url")
+    @classmethod
+    def validate_image_url(cls, value: str) -> str:
+        return safe_url(value, local=True)
+
+    @field_validator("source_url", "license_url")
+    @classmethod
+    def validate_source_url(cls, value: str) -> str:
+        return safe_url(value)
+
+    @field_validator("variants")
+    @classmethod
+    def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
+        for width, url in value.items():
+            if width not in {"640", "1280"}:
+                raise ValueError("Unsupported image variant")
+            safe_url(url, local=True)
+        return value
+
+
+class CardImageRule(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+    id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
+    entity_type: EntityType
+    scope: MatchScope
+    value: str = Field(min_length=1, max_length=300)
+    label: str = Field(default="", max_length=500)
+    asset_id: str | None = Field(default=None, max_length=100)
+    suppress: bool = False
+    enabled: bool = True
+
+    @model_validator(mode="after")
+    def validate_action(self) -> Self:
+        if self.scope != "object":
+            self.value = self.value.casefold()
+        if self.suppress and (self.scope != "object" or self.asset_id):
+            raise ValueError("Only an object rule may suppress its image, without an asset")
+        if not self.suppress and not self.asset_id:
+            raise ValueError("Choose an image or suppress this object")
+        return self
+
+    def target(self) -> tuple[str, str, str]:
+        return self.entity_type, self.scope, self.value
+
+
+class CardImageConfiguration(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+    assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
+    rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
+
+    @model_validator(mode="after")
+    def validate_relations(self) -> Self:
+        assets = {asset.id for asset in self.assets}
+        if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
+            self.rules
+        ):
+            raise ValueError("Image and rule IDs must be unique")
+        targets = [rule.target() for rule in self.rules]
+        if len(set(targets)) != len(targets):
+            raise ValueError("Each target may have only one rule")
+        if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
+            raise ValueError("Remove image associations before deleting the image")
+        return self
+
+
+class CardImagePublish(BaseModel):
+    revision: str = Field(max_length=200)
+    config: CardImageConfiguration
+
+
+class CardImagePreview(BaseModel):
+    entity_type: EntityType
+    entity_id: str = Field(min_length=1, max_length=300)
+    config: CardImageConfiguration

backend/app/schemas/compute_configuration.py

@@ -0,0 +1,94 @@
+"""Validated, source-backed observations for the compute comparison cards."""
+
+from datetime import date
+from typing import Annotated, Literal, Self
+
+from app.services.compute_catalog import COUNTRY_PARTICIPANTS
+from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
+
+Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
+Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
+Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
+Text = Annotated[str, Field(min_length=1, max_length=500)]
+
+
+class ConfigurationModel(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+
+
+class ComputeSource(ConfigurationModel):
+    id: Identifier
+    title: Text
+    url: HttpUrl
+    publisher: Text
+    report_year: Year
+    published_at: date | None = None
+    locator: Text
+
+    @field_validator("url")
+    @classmethod
+    def public_reference(cls, value: HttpUrl) -> HttpUrl:
+        if value.username or value.password or len(str(value)) > 2000:
+            raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
+        return value
+
+
+class ComputeObservation(ConfigurationModel):
+    year: Year
+    metric: Literal["global_share", "absolute"]
+    source_id: Identifier
+    values: dict[str, Value] = Field(min_length=1, max_length=250)
+
+    @model_validator(mode="after")
+    def comparable_values(self) -> Self:
+        if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
+            raise ValueError(
+                "Use countries from the canonical compute catalog, not regional totals"
+            )
+        if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
+            raise ValueError("Global shares for one year must total at most 100 percent")
+        return self
+
+
+class InnovationObservation(ConfigurationModel):
+    year: Year
+    value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
+    source_id: Identifier
+
+
+class ComputeConfiguration(ConfigurationModel):
+    reviewed_at: date
+    sources: list[ComputeSource] = Field(max_length=200)
+    observations: list[ComputeObservation] = Field(max_length=202)
+    innovation: list[InnovationObservation] = Field(max_length=101)
+
+    @model_validator(mode="after")
+    def consistent_evidence(self) -> Self:
+        ids = [source.id for source in self.sources]
+        if len(ids) != len(set(ids)):
+            raise ValueError("Report source IDs must be unique")
+        keys = [(row.metric, row.year) for row in self.observations]
+        if len(keys) != len(set(keys)):
+            raise ValueError("Each metric/year may have only one observation")
+        years = [row.year for row in self.innovation]
+        if len(years) != len(set(years)):
+            raise ValueError("Innovation observation years must be unique")
+        sources = {source.id: source for source in self.sources}
+        for row in [*self.observations, *self.innovation]:
+            source = sources.get(row.source_id)
+            if source is None:
+                raise ValueError(f"Missing report source: {row.source_id}")
+            if row.year > source.report_year:
+                raise ValueError("Observation year must not be after its report year")
+        if self.reviewed_at > date.today():
+            raise ValueError("Review date must not be in the future")
+        return self
+
+
+class ComputeConfigurationWrite(ConfigurationModel):
+    revision: str = Field(min_length=1, max_length=80)
+    config: ComputeConfiguration
+
+
+class ComputeConfigurationReset(ConfigurationModel):
+    revision: str = Field(min_length=1, max_length=80)

backend/app/schemas/user.py

@@ -3,6 +3,7 @@ from typing import Optional
 
 from pydantic import BaseModel, EmailStr, Field
 
+from app.core.enums import OtpPurpose, UserRole
 
 class UserBase(BaseModel):
     username: str
@@ -11,13 +12,13 @@ class UserBase(BaseModel):
 
 class UserCreate(UserBase):
     password: str = Field(..., min_length=8)
-    role: str = "viewer"
+    role: UserRole = UserRole.VIEWER
     gatekeeper_groups: list[str] = Field(default_factory=list)
 
 
 class UserUpdate(BaseModel):
     email: Optional[EmailStr] = None
-    role: Optional[str] = None
+    role: Optional[UserRole] = None
     gatekeeper_groups: Optional[list[str]] = None
     is_active: Optional[bool] = None
 
@@ -35,6 +36,9 @@ class UserInDB(UserBase):
 
 
 class UserResponse(UserBase):
+    # Bootstrap accounts have reserved .local email addresses. Validate email
+    # on writes, while allowing existing local identities to be serialized.
+    email: str
     id: int
     role: str
     gatekeeper_groups: list[str] = Field(default_factory=list)
@@ -59,7 +63,7 @@ class VerifyEmailRequest(BaseModel):
 
 class ResendCodeRequest(BaseModel):
     email: EmailStr
-    purpose: str = Field(default="register", pattern="^(register|verify_email|reset_password)$")
+    purpose: OtpPurpose = OtpPurpose.REGISTER
 
 
 class ForgotPasswordRequest(BaseModel):

backend/app/services/ai_client.py

@@ -2,18 +2,24 @@ from __future__ import annotations
 
 import asyncio
 import json
+from time import perf_counter
 
 import httpx
 from fastapi import Depends, HTTPException, status
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
+from app.core.logging import get_logger
 from app.db.session import get_db
 from app.schemas.ai import (
     AIProviderStatusResponse,
     SituationalAnalysisRequest,
     SituationalAnalysisResponse,
 )
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai")
 
 
 class AIProviderClient:
@@ -64,7 +70,19 @@ class AIProviderClient:
         return headers
 
     async def get_status(self, request_id: str | None = None) -> AIProviderStatusResponse:
+        context = self._base_log_context(operation="status")
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             return AIProviderStatusResponse(
                 provider="unconfigured",
                 enabled=False,
@@ -73,27 +91,133 @@ class AIProviderClient:
                 base_url=None,
             )
 
-        data = await self._request("GET", "/v1/provider/status", request_id=request_id)
-        return AIProviderStatusResponse.model_validate(data)
+        started_at = perf_counter()
+        await emit_business_log(
+            logger,
+            event="ai.provider.status.start",
+            message="AI provider status request started",
+            category="ai",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=context,
+        )
+        try:
+            data = await self._request("GET", "/v1/provider/status", request_id=request_id, operation="status")
+            result = AIProviderStatusResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.success",
+                message="AI provider status request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "configured": result.configured,
+                    "enabled": result.enabled,
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.status.failed",
+                message="AI provider status request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def analyze(
         self,
         payload: SituationalAnalysisRequest,
         request_id: str | None = None,
     ) -> SituationalAnalysisResponse:
+        context = self._base_log_context(
+            operation="analyze",
+            preferred_model=payload.preferred_model,
+            input_summary=self._summarize_analysis_payload(payload),
+        )
         if not self.service_url:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze skipped because service URL is not configured",
+                category="ai",
+                level="warning",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={**context, "status": "unconfigured"},
+            )
             raise HTTPException(
                 status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                 detail="AI provider service URL is not configured.",
             )
 
-        data = await self._request(
-            "POST",
-            "/v1/analyze",
-            json=payload.model_dump(),
+        started_at = perf_counter()
+        await emit_business_log(
+            logger,
+            event="ai.provider.analyze.start",
+            message="AI provider analyze request started",
+            category="ai",
+            service="ai",
+            module=__name__,
             request_id=request_id,
+            context=context,
         )
-        return SituationalAnalysisResponse.model_validate(data)
+        try:
+            data = await self._request(
+                "POST",
+                "/v1/analyze",
+                json=payload.model_dump(),
+                request_id=request_id,
+                operation="analyze",
+                payload_summary=context["input_summary"],
+            )
+            result = SituationalAnalysisResponse.model_validate(data)
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.success",
+                message="AI provider analyze request completed",
+                category="ai",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context={
+                    **context,
+                    "status": "success",
+                    "duration_ms": self._duration_ms(started_at),
+                    "result_provider": result.provider,
+                    "result_model": result.model,
+                    "content_block_count": len(result.content_blocks or []),
+                    "thinking_block_count": len(result.thinking_blocks or []),
+                },
+            )
+            return result
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai.provider.analyze.failed",
+                message="AI provider analyze request failed",
+                category="ai",
+                level="error",
+                service="ai",
+                module=__name__,
+                request_id=request_id,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": self._duration_ms(started_at)}),
+            )
+            raise
 
     async def _request(
         self,
@@ -101,9 +225,12 @@ class AIProviderClient:
         path: str,
         json: dict | None = None,
         request_id: str | None = None,
+        operation: str = "request",
+        payload_summary: dict | None = None,
     ) -> dict:
         last_error: Exception | None = None
         for attempt in range(1, self.retry_attempts + 1):
+            attempt_started_at = perf_counter()
             try:
                 async with httpx.AsyncClient(timeout=self.timeout) as client:
                     response = await client.request(
@@ -117,6 +244,15 @@ class AIProviderClient:
             except httpx.HTTPStatusError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts and exc.response.status_code >= 500:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        status_code=exc.response.status_code,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 detail = exc.response.text or "AI provider service returned an error"
@@ -127,6 +263,14 @@ class AIProviderClient:
             except httpx.HTTPError as exc:
                 last_error = exc
                 if attempt < self.retry_attempts:
+                    await self._log_retry(
+                        operation=operation,
+                        request_id=request_id,
+                        attempt=attempt,
+                        duration_ms=self._duration_ms(attempt_started_at),
+                        error=exc,
+                        payload_summary=payload_summary,
+                    )
                     await asyncio.sleep(0.3 * attempt)
                     continue
                 raise HTTPException(
@@ -139,6 +283,71 @@ class AIProviderClient:
             detail=f"AI provider service request failed: {last_error}",
         )
 
+    def _base_log_context(self, **extra: object) -> dict:
+        llm_provider_apis = self.llm_config.get("model_provider_apis")
+        return {
+            "provider": self.llm_config.get("provider") or "",
+            "provider_api": self.llm_config.get("provider_api") or "",
+            "model": self.llm_config.get("model") or "",
+            "base_url_configured": bool(self.llm_config.get("base_url")),
+            "service_url_configured": bool(self.service_url),
+            "timeout_seconds": self.timeout,
+            "retry_attempts": self.retry_attempts,
+            "model_provider_api_count": len(llm_provider_apis or {}) if isinstance(llm_provider_apis, dict) else 0,
+            **extra,
+        }
+
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    @staticmethod
+    def _summarize_analysis_payload(payload: SituationalAnalysisRequest) -> dict:
+        context = payload.context if isinstance(payload.context, dict) else {}
+        thinking = payload.thinking if isinstance(payload.thinking, dict) else payload.thinking
+        return {
+            "title_length": len(payload.title or ""),
+            "objective_length": len(payload.objective or ""),
+            "observation_count": len(payload.observations or []),
+            "constraint_count": len(payload.constraints or []),
+            "has_system_prompt": bool(payload.system_prompt),
+            "thinking_enabled": bool(thinking),
+            "context_keys": sorted(str(key) for key in context.keys()),
+        }
+
+    async def _log_retry(
+        self,
+        *,
+        operation: str,
+        request_id: str | None,
+        attempt: int,
+        duration_ms: int,
+        error: BaseException,
+        status_code: int | None = None,
+        payload_summary: dict | None = None,
+    ) -> None:
+        await emit_business_log(
+            logger,
+            event=f"ai.provider.{operation}.retry",
+            message="AI provider request will retry",
+            category="ai",
+            level="warning",
+            service="ai",
+            module=__name__,
+            request_id=request_id,
+            context=exception_context(
+                error,
+                {
+                    **self._base_log_context(operation=operation),
+                    "attempt": attempt,
+                    "next_attempt": attempt + 1,
+                    "status_code": status_code,
+                    "duration_ms": duration_ms,
+                    "input_summary": payload_summary,
+                },
+            ),
+        )
+
 
 async def get_ai_provider_client(db: AsyncSession = Depends(get_db)) -> AIProviderClient:
     from app.api.v1.settings import get_runtime_ai_provider_config

backend/app/services/ai_tools/web_fetch.py

@@ -1,11 +1,18 @@
 from __future__ import annotations
 
 import hashlib
+from time import perf_counter
+from urllib.parse import urlparse
 
 import httpx
 from bs4 import BeautifulSoup
 
+from app.core.logging import get_logger
 from app.services.ai_tools.schemas import FetchedEvidence
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
 
 
 class WebFetchError(RuntimeError):
@@ -29,8 +36,33 @@ async def fetch_url_evidence(
     timeout_seconds: int = 20,
     max_bytes: int = 1_500_000,
 ) -> FetchedEvidence:
+    started_at = perf_counter()
     if not url:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch failed because URL is empty",
+            category="ai_tool",
+            level="warning",
+            service="ai_tool",
+            module=__name__,
+            context={"reason": "empty_url"},
+        )
         raise WebFetchError("url is required")
+    request_host = urlparse(url).netloc
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.start",
+        message="WebFetch request started",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "timeout_seconds": timeout_seconds,
+            "max_bytes": max_bytes,
+        },
+    )
     try:
         async with httpx.AsyncClient(
             timeout=timeout_seconds,
@@ -41,10 +73,45 @@ async def fetch_url_evidence(
             response.raise_for_status()
             content = response.content[:max_bytes]
     except httpx.HTTPError as exc:
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_fetch.failed",
+            message="WebFetch request failed",
+            category="ai_tool",
+            level="error",
+            service="ai_tool",
+            module=__name__,
+            context=exception_context(
+                exc,
+                {
+                    "url_host": request_host,
+                    "status": "failed",
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            ),
+        )
         raise WebFetchError(f"failed to fetch page: {exc}") from exc
 
     title, text = _extract_title_and_text(content.decode(response.encoding or "utf-8", errors="ignore"))
     content_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()
+    await emit_business_log(
+        logger,
+        event="ai_tool.web_fetch.success",
+        message="WebFetch request completed",
+        category="ai_tool",
+        service="ai_tool",
+        module=__name__,
+        context={
+            "url_host": request_host,
+            "final_url_host": urlparse(str(response.url)).netloc,
+            "status": "success",
+            "status_code": response.status_code,
+            "bytes_read": len(content),
+            "content_hash": content_hash,
+            "duration_ms": int((perf_counter() - started_at) * 1000),
+            "extractor": "beautifulsoup_basic",
+        },
+    )
     return FetchedEvidence(
         url=url,
         final_url=str(response.url),
@@ -53,4 +120,3 @@ async def fetch_url_evidence(
         content_hash=content_hash,
         extractor="beautifulsoup_basic",
     )
-

backend/app/services/ai_tools/web_search.py

@@ -1,11 +1,18 @@
 from __future__ import annotations
 
 from copy import deepcopy
+import hashlib
+from time import perf_counter
 from typing import Any
 
 import httpx
 
+from app.core.logging import get_logger
 from app.services.ai_tools.schemas import SearchEvidence, WebSearchConfig, WebSearchProviderConfig
+from app.services.business_logs import emit_business_log, exception_context
+
+
+logger = get_logger(__name__, service="ai_tool")
 
 
 WEB_SEARCH_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
@@ -120,29 +127,108 @@ class WebSearchClient:
         domains: list[str] | None = None,
         freshness_days: int | None = None,
     ) -> list[SearchEvidence]:
+        started_at = perf_counter()
         if not self.config.enabled:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because integration is disabled",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": self.config.default_provider, "reason": "disabled"},
+            )
             raise WebSearchConfigurationError("WebSearch is disabled.")
         provider_config = self.config.active_provider_config
         provider = normalize_web_search_provider(provider_config.provider)
         if provider != "searxng" and not provider_config.api_key:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.unavailable",
+                message="WebSearch skipped because API key is not configured",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "missing_api_key"},
+            )
             raise WebSearchConfigurationError(f"{provider} API key is not configured.")
         query = " ".join(str(query or "").split())
         if not query:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch failed because query is empty",
+                category="ai_tool",
+                level="warning",
+                service="ai_tool",
+                module=__name__,
+                context={"provider": provider, "reason": "empty_query"},
+            )
             raise WebSearchConfigurationError("search query is required.")
         limit = max_results or provider_config.max_results
-        if provider == "tavily":
-            return await self._search_tavily(provider_config, query, limit, domains, freshness_days)
-        if provider == "brave":
-            return await self._search_brave(provider_config, query, limit, domains)
-        if provider == "serpapi":
-            return await self._search_serpapi(provider_config, query, limit)
-        if provider == "exa":
-            return await self._search_exa(provider_config, query, limit, domains)
-        if provider == "firecrawl":
-            return await self._search_firecrawl(provider_config, query, limit)
-        if provider == "searxng":
-            return await self._search_searxng(provider_config, query, limit, domains)
-        raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
+        context = {
+            "provider": provider,
+            "query_hash": hashlib.sha256(query.encode("utf-8")).hexdigest(),
+            "query_length": len(query),
+            "max_results": limit,
+            "domain_count": len(domains or []),
+            "freshness_days": freshness_days,
+        }
+        await emit_business_log(
+            logger,
+            event="ai_tool.web_search.start",
+            message="WebSearch request started",
+            category="ai_tool",
+            service="ai_tool",
+            module=__name__,
+            context=context,
+        )
+        try:
+            if provider == "tavily":
+                results = await self._search_tavily(provider_config, query, limit, domains, freshness_days)
+            elif provider == "brave":
+                results = await self._search_brave(provider_config, query, limit, domains)
+            elif provider == "serpapi":
+                results = await self._search_serpapi(provider_config, query, limit)
+            elif provider == "exa":
+                results = await self._search_exa(provider_config, query, limit, domains)
+            elif provider == "firecrawl":
+                results = await self._search_firecrawl(provider_config, query, limit)
+            elif provider == "searxng":
+                results = await self._search_searxng(provider_config, query, limit, domains)
+            else:
+                raise WebSearchConfigurationError(f"Unsupported web search provider: {provider}")
+            event = "ai_tool.web_search.success" if results else "ai_tool.web_search.empty"
+            await emit_business_log(
+                logger,
+                event=event,
+                message="WebSearch request completed" if results else "WebSearch returned no results",
+                category="ai_tool",
+                level="info" if results else "warning",
+                service="ai_tool",
+                module=__name__,
+                context={
+                    **context,
+                    "status": "success" if results else "empty",
+                    "result_count": len(results),
+                    "duration_ms": int((perf_counter() - started_at) * 1000),
+                },
+            )
+            return results
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="ai_tool.web_search.failed",
+                message="WebSearch request failed",
+                category="ai_tool",
+                level="error",
+                service="ai_tool",
+                module=__name__,
+                context=exception_context(exc, {**context, "status": "failed", "duration_ms": int((perf_counter() - started_at) * 1000)}),
+            )
+            raise
 
     async def test_connection(self) -> list[SearchEvidence]:
         return await self.search("Planet WebSearch connectivity test", max_results=1)
@@ -388,4 +474,3 @@ def _float_or_none(value: Any) -> float | None:
         return float(value)
     except (TypeError, ValueError):
         return None
-

backend/app/services/bgp_detectors.py

@@ -6,6 +6,7 @@ from collections import Counter, defaultdict
 from datetime import UTC, datetime
 from typing import Any
 
+from app.core.enums import BGPStatus
 from app.models.bgp_anomaly import BGPAnomaly
 
 
@@ -127,7 +128,7 @@ def detect_origin_change_anomalies(
                     source=source,
                     anomaly_type=anomaly_type,
                     severity=severity,
-                    status="active",
+                    status=BGPStatus.ACTIVE.value,
                     entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
                     prefix=prefix,
                     origin_asn=sorted(historic)[0] if historic else None,
@@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies(
                 source=source,
                 anomaly_type="more_specific_burst",
                 severity="high",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
                 prefix=sample.get("prefix"),
                 origin_asn=sample.get("origin_asn"),
@@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies(
                 source=source,
                 anomaly_type="mass_withdrawal",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
                 prefix=prefix,
                 origin_asn=origin_asn,
@@ -354,7 +355,7 @@ def detect_route_leak_anomalies(
                 source=source,
                 anomaly_type="route_leak_candidate",
                 severity="high" if max_path_length >= dominant_length + 3 else "medium",
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),
@@ -435,7 +436,7 @@ def detect_path_flap_anomalies(
                 source=source,
                 anomaly_type="path_flap",
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
                 prefix=prefix,
                 origin_asn=sample_metadata.get("origin_asn"),

backend/app/services/bgp_incidents.py

@@ -9,6 +9,7 @@ from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.collected_data_fields import get_record_field
+from app.core.enums import BGPStatus
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.collected_data import CollectedData
@@ -290,7 +291,7 @@ async def create_bgp_incidents_for_anomalies(
             existing.title = title
             existing.summary = summary
             existing.severity = severity
-            existing.status = "active"
+            existing.status = BGPStatus.ACTIVE.value
             existing.confidence = confidence
             existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC)
             existing.ended_at = None
@@ -313,7 +314,7 @@ async def create_bgp_incidents_for_anomalies(
                 title=title,
                 summary=summary,
                 severity=severity,
-                status="active",
+                status=BGPStatus.ACTIVE.value,
                 confidence=confidence,
                 started_at=primary.started_at or datetime.now(UTC),
                 affected_prefixes=prefixes,

backend/app/services/business_logs.py

@@ -0,0 +1,117 @@
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Mapping
+from typing import Any
+
+from app.core.logging import PlanetLoggerAdapter, sanitize_log_value
+from app.core.request_context import get_request_id
+from app.services.persistent_logs import record_system_log
+
+
+LEVEL_METHODS = {
+    "debug": "debug_event",
+    "info": "info_event",
+    "warning": "warning_event",
+    "error": "error_event",
+}
+
+
+def normalize_business_level(level: str | None) -> str:
+    normalized = str(level or "info").strip().lower()
+    if normalized in {"warn", "warning"}:
+        return "warning"
+    if normalized in {"err", "error", "critical", "fatal"}:
+        return "error"
+    if normalized == "debug":
+        return "debug"
+    return "info"
+
+
+def build_business_context(
+    context: Mapping[str, Any] | None = None,
+    **fields: Any,
+) -> dict[str, Any]:
+    payload = dict(context or {})
+    for key, value in fields.items():
+        if value is not None:
+            payload[key] = value
+    return sanitize_log_value(payload)
+
+
+async def emit_business_log(
+    logger: PlanetLoggerAdapter,
+    *,
+    event: str,
+    message: str,
+    category: str,
+    level: str = "info",
+    source: str = "backend",
+    service: str | None = None,
+    module: str | None = None,
+    request_id: str | None = None,
+    user_id: int | None = None,
+    context: Mapping[str, Any] | None = None,
+) -> None:
+    normalized_level = normalize_business_level(level)
+    safe_context = build_business_context(context)
+    log_method = getattr(logger, LEVEL_METHODS[normalized_level])
+    log_method(message, event=event, context=safe_context)
+    await record_system_log(
+        source=source,
+        level=normalized_level,
+        message=message,
+        service=service,
+        module=module,
+        event=event,
+        request_id=request_id or get_request_id(),
+        user_id=user_id,
+        category=category,
+        context=safe_context,
+    )
+
+
+def emit_business_log_background(
+    logger: PlanetLoggerAdapter,
+    *,
+    event: str,
+    message: str,
+    category: str,
+    level: str = "info",
+    source: str = "backend",
+    service: str | None = None,
+    module: str | None = None,
+    request_id: str | None = None,
+    user_id: int | None = None,
+    context: Mapping[str, Any] | None = None,
+) -> None:
+    normalized_level = normalize_business_level(level)
+    safe_context = build_business_context(context)
+    log_method = getattr(logger, LEVEL_METHODS[normalized_level])
+    log_method(message, event=event, context=safe_context)
+    try:
+        loop = asyncio.get_running_loop()
+    except RuntimeError:
+        return
+    loop.create_task(
+        record_system_log(
+            source=source,
+            level=normalized_level,
+            message=message,
+            service=service,
+            module=module,
+            event=event,
+            request_id=request_id or get_request_id(),
+            user_id=user_id,
+            category=category,
+            context=safe_context,
+        )
+    )
+
+
+def exception_context(exc: BaseException, context: Mapping[str, Any] | None = None) -> dict[str, Any]:
+    return build_business_context(
+        context,
+        error_type=type(exc).__name__,
+        error=str(exc),
+    )

backend/app/services/card_image_configuration.py

@@ -0,0 +1,174 @@
+"""Revision-checked image settings; new bundled assets never replace user edits."""
+
+from uuid import uuid4
+
+from fastapi import HTTPException
+from sqlalchemy import select
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.request_context import get_request_id
+from app.models.system_log import AuditLog
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.schemas.card_images import CardImageConfiguration
+from app.services.card_image_defaults import (
+    bundled_configuration,
+    previous_bundled_assets,
+    previous_bundled_rules,
+)
+
+CATEGORY = "earth_card_images"
+
+
+def effective_envelope(payload: dict | None) -> dict:
+    defaults, version = bundled_configuration()
+    config = (
+        CardImageConfiguration.model_validate(payload["config"])
+        if payload
+        else defaults.model_copy(deep=True)
+    )
+    if payload:
+        previous = previous_bundled_assets()
+        latest = {asset.id: asset for asset in defaults.assets}
+        for index, asset in enumerate(config.assets):
+            old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
+            # Replace the old bundled source while preserving fields the user
+            # changed (crop, title, enabled state, etc.). Custom URLs are untouched.
+            if old is not None and asset.url == old.url and asset.id in latest:
+                overrides = {
+                    key: value
+                    for key, value in asset.model_dump().items()
+                    if value != getattr(old, key)
+                }
+                config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
+        known_assets = set(payload.get("known_assets", []))
+        asset_ids = {asset.id for asset in config.assets}
+        config.assets.extend(
+            asset.model_copy(deep=True)
+            for asset in defaults.assets
+            if asset.id not in known_assets and asset.id not in asset_ids
+        )
+        prior_rules = previous_bundled_rules()
+        latest_rules = {rule.id: rule for rule in defaults.rules}
+        for index, rule in enumerate(config.rules):
+            old_rule = prior_rules.get(rule.id)
+            if (
+                old_rule
+                and rule.id in latest_rules
+                and rule.target() == old_rule.target()
+                and rule.asset_id == old_rule.asset_id
+            ):
+                overrides = {
+                    key: value
+                    for key, value in rule.model_dump().items()
+                    if value != getattr(old_rule, key)
+                }
+                config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
+        known_rules = set(payload.get("known_rules", []))
+        rule_ids = {rule.id for rule in config.rules}
+        targets = {rule.target() for rule in config.rules}
+        available = {asset.id for asset in config.assets}
+        config.rules.extend(
+            rule.model_copy(deep=True)
+            for rule in defaults.rules
+            if rule.id not in known_rules
+            and rule.id not in rule_ids
+            and rule.target() not in targets
+            and rule.asset_id in available
+        )
+    return {
+        "config": config.model_dump(mode="json"),
+        "revision": f"{version}:{payload['revision'] if payload else 'default'}",
+        "is_default": payload is None,
+    }
+
+
+async def read_configuration(db: AsyncSession) -> dict:
+    record = (
+        await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
+    ).scalar_one_or_none()
+    return effective_envelope(record.payload if record else None)
+
+
+async def publish_configuration(
+    db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
+) -> dict:
+    defaults, _ = bundled_configuration()
+    pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
+    if any(
+        asset.id in pending and asset.rights_status != "permission_pending"
+        for asset in config.assets
+    ):
+        raise HTTPException(422, "Bundled image permission is still unverified")
+    initial = {
+        "config": defaults.model_dump(mode="json"),
+        "revision": "default",
+        "known_assets": [],
+        "known_rules": [],
+    }
+    await db.execute(
+        insert(SystemSetting)
+        .values(category=CATEGORY, payload=initial)
+        .on_conflict_do_nothing(index_elements=[SystemSetting.category])
+    )
+    record = (
+        await db.execute(
+            select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
+        )
+    ).scalar_one()
+    if effective_envelope(record.payload)["revision"] != revision:
+        await db.rollback()
+        raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
+    before = record.payload
+    record.payload = {
+        "config": config.model_dump(mode="json"),
+        "revision": uuid4().hex,
+        "known_assets": [asset.id for asset in defaults.assets],
+        "known_rules": [rule.id for rule in defaults.rules],
+    }
+    db.add(
+        AuditLog(
+            actor_id=actor.id,
+            actor_name=actor.username,
+            action="card_images.update",
+            target_type="system_setting",
+            target_id=CATEGORY,
+            result="success",
+            request_id=get_request_id(),
+            details={"before": before, "after": record.payload},
+        )
+    )
+    await db.commit()
+    return effective_envelope(record.payload)
+
+
+def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
+    assets = {asset.id: asset for asset in config.assets}
+    for scope in ("object", "model", "group", "category"):
+        value = context.get("entity_id" if scope == "object" else scope)
+        values = [value] if value else []
+        if scope == "category" and value != "*":
+            values.append("*")
+        for value in values:
+            value = value if scope == "object" else value.casefold()
+            rule = next(
+                (
+                    rule
+                    for rule in config.rules
+                    if rule.enabled
+                    and rule.entity_type == context["entity_type"]
+                    and rule.scope == scope
+                    and rule.value == value
+                ),
+                None,
+            )
+            if not rule:
+                continue
+            match = {"scope": scope, "value": value, "rule_id": rule.id}
+            if rule.suppress:
+                return {"image": None, "match": match, "suppressed": True}
+            asset = assets.get(rule.asset_id)
+            if asset and asset.enabled:
+                return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
+    return {"image": None, "match": None, "suppressed": False}

backend/app/services/card_image_defaults.py

@@ -0,0 +1,108 @@
+"""Read immutable bundled assets without inserting machine-specific database rows."""
+
+import hashlib
+import json
+from functools import lru_cache
+from pathlib import Path
+
+from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
+
+
+def _asset_from_manifest(item: dict) -> CardImageAsset:
+    pending = item.get("redistribution_status") == "permission_not_verified"
+    return CardImageAsset(
+        id=item["id"],
+        title=item["label"],
+        url=item["url"],
+        width=item["width"],
+        height=item["height"],
+        credit=item["credit"],
+        source_url=item.get("source_page") or "",
+        license=item["license"],
+        license_url=item.get("license_url") or "",
+        kind=item["kind"],
+        rights_status="permission_pending" if pending else "available",
+        enabled=item.get("enabled", not pending),
+        focus_x=item.get("focus_x", 50),
+        focus_y=item.get("focus_y", 50),
+        variants=item.get("variants", {}),
+    )
+
+
+@lru_cache(maxsize=1)
+def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
+    manifest = json.loads(MANIFEST_PATH.read_bytes())
+    return {
+        key: [CardImageAsset.model_validate(item) for item in value]
+        for key, value in manifest.get("previous_defaults", {}).items()
+    }
+
+
+@lru_cache(maxsize=1)
+def previous_bundled_rules() -> dict[str, CardImageRule]:
+    manifest = json.loads(MANIFEST_PATH.read_bytes())
+    return {
+        key: CardImageRule.model_validate(value)
+        for key, value in manifest.get("previous_rules", {}).items()
+    }
+
+
+@lru_cache(maxsize=1)
+def bundled_configuration() -> tuple[CardImageConfiguration, str]:
+    content = MANIFEST_PATH.read_bytes()
+    manifest = json.loads(content)
+    assets, rules = [], []
+    for item in manifest["assets"]:
+        asset = _asset_from_manifest(item)
+        assets.append(asset)
+        if not asset.enabled:
+            continue
+        hints = item.get("binding_hints", {})
+        for index, target in enumerate(hints.get("defaults", [])):
+            rules.append(
+                CardImageRule(
+                    id=target.get("id", f"builtin-{asset.id}-default-{index}"),
+                    entity_type=target["entity_type"],
+                    scope=target["scope"],
+                    value=target["value"],
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+        for index, entity_id in enumerate(hints.get("entity_ids", [])):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-entity-{index}",
+                    entity_type=item["entity_type"],
+                    scope="object",
+                    value=entity_id,
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+        for norad_id in hints.get("norad_ids", []):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-{norad_id}",
+                    entity_type="satellite",
+                    scope="object",
+                    value=str(norad_id),
+                    label=f"{asset.title} · NORAD {norad_id}",
+                    asset_id=asset.id,
+                )
+            )
+        if group := hints.get("constellation_group"):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-group",
+                    entity_type="satellite",
+                    scope="group",
+                    value=group,
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+    return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()

backend/app/services/card_image_files.py

@@ -0,0 +1,95 @@
+"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
+
+import hashlib
+import io
+import re
+import warnings
+from uuid import uuid4
+from pathlib import Path
+import os
+
+from fastapi import HTTPException
+from PIL import Image, ImageOps, UnidentifiedImageError
+
+from app.schemas.card_images import CardImageAsset
+from app.services.card_image_defaults import REPO_ROOT
+
+ASSET_DIR = Path(
+    os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
+).resolve()
+MAX_UPLOAD_BYTES = 12 * 1024 * 1024
+MAX_PIXELS = 32_000_000
+FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
+FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
+
+
+def uploaded_path(filename: str) -> Path:
+    if not FILE_PATTERN.fullmatch(filename):
+        raise HTTPException(404, "Image not found")
+    return ASSET_DIR / filename
+
+
+def store_image(content: bytes, title: str) -> CardImageAsset:
+    if len(content) > MAX_UPLOAD_BYTES:
+        raise HTTPException(413, "Image must be 12 MiB or smaller")
+    try:
+        with warnings.catch_warnings():
+            warnings.simplefilter("error", Image.DecompressionBombWarning)
+            image = Image.open(io.BytesIO(content))
+            extension = FORMATS.get(image.format)
+            if (
+                not extension
+                or image.width * image.height > MAX_PIXELS
+                or getattr(image, "is_animated", False)
+            ):
+                raise ValueError("Unsupported or oversized raster")
+            image.load()
+            image = ImageOps.exif_transpose(image).convert(
+                "RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
+            )
+    except (
+        UnidentifiedImageError,
+        OSError,
+        ValueError,
+        Image.DecompressionBombError,
+        Image.DecompressionBombWarning,
+    ) as error:
+        raise HTTPException(
+            422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
+        ) from error
+    digest = hashlib.sha256(content).hexdigest()
+    ASSET_DIR.mkdir(parents=True, exist_ok=True)
+    original = f"{digest}-original.{extension}"
+    original_path = uploaded_path(original)
+    if not original_path.exists():
+        temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
+        temporary.write_bytes(content)
+        temporary.replace(original_path)
+    variants = {}
+    for size, content in render_variants(image).items():
+        filename = f"{digest}-{size}.webp"
+        path = uploaded_path(filename)
+        if not path.exists():
+            temporary = path.with_suffix(f".{uuid4().hex}.tmp")
+            temporary.write_bytes(content)
+            temporary.replace(path)
+        variants[size] = f"/api/v1/earth/card-images/files/{filename}"
+    return CardImageAsset(
+        id=f"upload-{digest[:24]}",
+        title=title or "Image",
+        url=f"/api/v1/earth/card-images/files/{original}",
+        variants=variants,
+        width=image.width,
+        height=image.height,
+    )
+
+
+def render_variants(image: Image.Image) -> dict[str, bytes]:
+    result = {}
+    for size in (640, 1280):
+        resized = image.copy()
+        resized.thumbnail((size, size), Image.Resampling.LANCZOS)
+        buffer = io.BytesIO()
+        resized.save(buffer, format="WEBP", quality=93, method=4)
+        result[str(size)] = buffer.getvalue()
+    return result

backend/app/services/card_image_targets.py

@@ -0,0 +1,180 @@
+"""Bounded catalog queries and type defaults for every Earth object card."""
+
+from sqlalchemy import String, case, cast, func, literal, or_, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.compute_types import (
+    COMPUTE_SITE_TYPES,
+    compute_site_type_expression,
+    compute_record_filter,
+)
+from app.models.bgp_incident import BGPIncident
+from app.models.bgp_collector_location import BGPCollectorLocation
+from app.models.collected_data import CollectedData
+from app.models.earth_interactable import EarthInteractable
+from app.models.vessel import VesselCurrentState
+from app.schemas.card_images import EntityType, MatchScope
+
+SOURCES = {
+    "satellite": ("celestrak_tle", "spacetrack_tle"),
+    "compute_center": ("top500", "epoch_ai_gpu"),
+    "cable": ("arcgis_cables",),
+    "landing_point": ("arcgis_landing_points",),
+}
+# Named constellation families, not inferred hardware models or generations.
+SATELLITE_GROUP_PATTERNS = (
+    ("STARLINK%", "starlink"),
+    ("IRIDIUM%", "iridium-next"),
+    ("BEIDOU-3%", "beidou-3"),
+    ("BEIDOU%", "beidou"),
+    ("QIANFAN%", "qianfan"),
+    ("ONEWEB%", "oneweb"),
+    ("KUIPER%", "kuiper"),
+)
+
+
+def collected_query(entity_type: EntityType):
+    record, metadata = CollectedData, CollectedData.extra_data
+    model = func.coalesce(
+        metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
+    )
+    category = (
+        compute_site_type_expression(record)
+        if entity_type == "compute_center"
+        else literal(entity_type)
+    )
+    if entity_type == "satellite":
+        group = func.coalesce(
+            func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
+            case(
+                *[
+                    (func.upper(record.name).like(pattern), value)
+                    for pattern, value in SATELLITE_GROUP_PATTERNS
+                ],
+                else_="",
+            ),
+        )
+        entity_id = metadata["norad_cat_id"].as_string()
+    else:
+        group = func.coalesce(
+            metadata["operator"].as_string(), metadata["organization"].as_string(), ""
+        )
+        entity_id = record.source + ":" + record.source_id
+    return select(
+        record.name.label("name"),
+        cast(entity_id, String).label("entity_id"),
+        model.label("model"),
+        group.label("group"),
+        category.label("category"),
+    ).where(
+        (
+            compute_record_filter(record)
+            if entity_type == "compute_center"
+            else record.source.in_(SOURCES[entity_type])
+        ),
+        record.is_current.is_(True),
+        record.deleted_at.is_(None),
+        metadata["anonymous"].as_boolean().is_not(True),
+    )
+
+
+def target_query(entity_type: EntityType):
+    if entity_type in SOURCES:
+        return collected_query(entity_type)
+    if entity_type == "vessel":
+        model = VesselCurrentState
+        identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
+        conditions = []
+    elif entity_type == "bgp":
+        model = BGPIncident
+        identity, name, group, category = (
+            model.incident_key,
+            model.title,
+            model.source,
+            model.incident_type,
+        )
+        conditions = []
+    elif entity_type == "bgp_collector":
+        model = BGPCollectorLocation
+        identity, name, group, category = (
+            model.collector_id,
+            model.collector_id,
+            model.operator,
+            literal("bgp_collector"),
+        )
+        conditions = []
+    else:
+        model = EarthInteractable
+        identity, name, group, category = model.id, model.label, model.layer, model.kind
+        conditions = [model.is_deleted.is_(False)]
+    return select(
+        name.label("name"),
+        cast(identity, String).label("entity_id"),
+        literal("").label("model"),
+        func.coalesce(group, "").label("group"),
+        func.coalesce(category, entity_type).label("category"),
+    ).where(*conditions)
+
+
+def category_context(entity_type: EntityType) -> dict[str, str]:
+    return {
+        "entity_type": entity_type,
+        "entity_id": "",
+        "name": "",
+        "model": "",
+        "group": "",
+        "category": entity_type,
+    }
+
+
+async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
+    if entity_type == "compute_center" and entity_id.startswith("__category__:"):
+        category = entity_id.partition(":")[2]
+        if category in COMPUTE_SITE_TYPES:
+            return {**category_context(entity_type), "category": category}
+    if entity_id == "__category__":
+        return category_context(entity_type)
+    if entity_type in SOURCES and entity_type != "satellite":
+        source, separator, source_id = entity_id.partition(":")
+        if not separator or not source or not source_id:
+            return category_context(entity_type)
+        # Match the indexed identity columns before projecting JSON metadata.
+        statement = collected_query(entity_type).where(
+            CollectedData.source == source, CollectedData.source_id == source_id
+        )
+    else:
+        catalog = target_query(entity_type).subquery()
+        statement = select(catalog).where(catalog.c.entity_id == entity_id)
+    row = (await db.execute(statement.limit(1))).mappings().first()
+    if not row:
+        # Old map snapshots can outlive a catalog refresh. They may use the type
+        # illustration, but must not claim an unverified exact-object identity.
+        return category_context(entity_type)
+    context = {key: str(value or "") for key, value in row.items()}
+    return {**context, "entity_type": entity_type}
+
+
+async def search_targets(
+    db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
+) -> list[dict[str, str]]:
+    catalog = target_query(entity_type).subquery()
+    value = catalog.c["entity_id" if scope == "object" else scope]
+    label = catalog.c.name if scope == "object" else value
+    statement = select(value.label("value"), label.label("label")).where(
+        value.is_not(None), value != ""
+    )
+    if query:
+        escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
+        statement = statement.where(
+            or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
+        )
+    rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
+    items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
+    if scope == "category" and not query:
+        known = {item["value"] for item in items}
+        if entity_type == "compute_center":
+            items.extend(
+                {"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
+            )
+        items.insert(0, {"value": "*", "label": "*"})
+    return items

backend/app/services/collectors/aisstream.py

@@ -322,6 +322,21 @@ class AISStreamCollector(BaseCollector):
             last_success_at=now if data else None,
             lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
         )
+        if snapshot_id is not None:
+            from app.models.data_snapshot import DataSnapshot
+
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                snapshot.record_count = records_added
+                snapshot.status = "success"
+                snapshot.completed_at = now
+                snapshot.summary = {
+                    "created": records_added,
+                    "updated": 0,
+                    "unchanged": 0,
+                    "deleted": 0,
+                    "storage": "ais_raw_observations",
+                }
         await db.commit()
         await self.update_progress(records_added, force=True)
         return records_added

backend/app/services/collectors/base.py

@@ -4,42 +4,22 @@ import asyncio
 from abc import ABC, abstractmethod
 from typing import Dict, List, Any, Optional
 from datetime import UTC, datetime
+from time import perf_counter
+from urllib.parse import urlparse
 import httpx
 from sqlalchemy import select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
 from app.core.countries import normalize_country
+from app.core.enums import JobStatus, SnapshotStatus
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.broadcaster import broadcaster
-from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
-
-EARTH_UPDATE_LAYER_HINTS: dict[str, list[str]] = {
-    "ris_live_bgp": ["bgp"],
-    "bgpstream_bgp": ["bgp"],
-    "top500_supercomputers": ["computeCenters"],
-    "epoch_ai_gpu": ["computeCenters"],
-    "huggingface_models": ["computeCenters"],
-    "huggingface_datasets": ["computeCenters"],
-    "huggingface_spaces": ["computeCenters"],
-    "telegeography_cables": ["cables"],
-    "telegeography_landing_points": ["cables"],
-    "telegeography_cable_systems": ["cables"],
-    "arcgis_cables": ["cables"],
-    "fao_landing_points": ["cables"],
-    "arcgis_landing_points": ["cables"],
-    "arcgis_cable_landing_relations": ["cables"],
-    "spacetrack_tle": ["satellites"],
-    "celestrak_tle": ["satellites"],
-    "barentswatch_vessels": ["vessels"],
-    "aisstream_vessels": ["vessels"],
-    "news_live_streams": ["media"],
-    "media_news_archive": ["news"],
-}
+from app.services.business_logs import emit_business_log, exception_context
 
 
-def get_earth_update_layers_for_source(source: str) -> list[str]:
-    return EARTH_UPDATE_LAYER_HINTS.get(source, [])
+logger = get_logger(__name__, service="collector")
 
 
 class BaseCollector(ABC):
@@ -51,6 +31,7 @@ class BaseCollector(ABC):
     frequency_hours: int = 4
     data_type: str = "generic"
     fail_on_empty: bool = False
+    preserve_snapshot_on_failure: bool = False
 
     def __init__(self):
         self._current_task = None
@@ -58,6 +39,7 @@ class BaseCollector(ABC):
         self._datasource_id = 1
         self._resolved_url: Optional[str] = None
         self._last_broadcast_progress: Optional[int] = None
+        self._last_save_summary: dict[str, int] = {}
 
     async def resolve_url(self, db: AsyncSession) -> None:
         from app.core.data_sources import get_data_sources_config
@@ -96,29 +78,6 @@ class BaseCollector(ABC):
         )
         self._last_broadcast_progress = rounded_progress
 
-    async def _publish_earth_update(
-        self,
-        *,
-        action: str,
-        records_processed: int,
-        task_id: int | None = None,
-    ) -> None:
-        layers = get_earth_update_layers_for_source(self.name)
-        if not layers:
-            return
-        await broadcaster.broadcast_earth_update(
-            {
-                "action": action,
-                "source": self.name,
-                "data_type": self.data_type,
-                "layers": layers,
-                "datasource_id": getattr(self, "_datasource_id", None),
-                "task_id": task_id,
-                "records_processed": records_processed,
-                "timestamp": to_iso8601_utc(datetime.now(UTC)),
-            }
-        )
-
     async def update_progress(self, records_processed: int, *, commit: bool = False, force: bool = False):
         """Update task progress - call this during data processing"""
         if self._current_task and self._db_session:
@@ -280,7 +239,7 @@ class BaseCollector(ABC):
             snapshot = await db.get(DataSnapshot, snapshot_id)
             if snapshot:
                 parent_snapshot_id = snapshot.parent_snapshot_id
-                snapshot.status = "cancelled"
+                snapshot.status = SnapshotStatus.CANCELLED.value
                 snapshot.is_current = False
                 snapshot.completed_at = datetime.now(UTC)
                 summary = dict(snapshot.summary or {})
@@ -322,19 +281,39 @@ class BaseCollector(ABC):
         from app.models.data_snapshot import DataSnapshot
 
         start_time = datetime.now(UTC)
+        started_at = perf_counter()
         datasource_id = getattr(self, "_datasource_id", 1)
         snapshot_id: Optional[int] = None
 
         if not collector_registry.is_active(self.name):
+            await self._log_collection_event(
+                "collector.run.skipped_disabled",
+                "Collector skipped because it is disabled",
+                level="info",
+                context={"status": "skipped", "reason": "disabled"},
+            )
             return {"status": "skipped", "reason": "Collector is disabled"}
 
-        task = CollectionTask(
-            datasource_id=datasource_id,
-            status="running",
-            phase="queued",
-            started_at=start_time,
-        )
-        db.add(task)
+        task = self._current_task if isinstance(self._current_task, CollectionTask) else None
+        if task is None:
+            task = CollectionTask(
+                datasource_id=datasource_id,
+                source=self.name,
+                task_type="collect",
+                status="running",
+                phase="queued",
+                started_at=start_time,
+            )
+            db.add(task)
+        else:
+            task.datasource_id = datasource_id
+            task.source = task.source or self.name
+            task.task_type = task.task_type or "collect"
+            task.status = JobStatus.RUNNING.value
+            task.phase = "queued"
+            task.started_at = task.started_at or start_time
+            task.completed_at = None
+            task.error_message = None
         await db.commit()
         task_id = task.id
 
@@ -344,25 +323,80 @@ class BaseCollector(ABC):
 
         await self.resolve_url(db)
         await self._publish_task_update(force=True)
+        await self._log_collection_event(
+            "collector.run.started",
+            "Collector run started",
+            context={"status": "running", "task_id": task_id},
+        )
 
         try:
+            phase_started_at = perf_counter()
             await self.set_phase("fetching", message="正在拉取原始数据")
+            await self._log_collection_event(
+                "collector.phase.fetching.start",
+                "Collector fetch phase started",
+                context={"task_id": task_id, "snapshot_id": snapshot_id},
+            )
             raw_data = await self.fetch()
             task.total_records = len(raw_data)
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.phase.fetching.success",
+                "Collector fetch phase completed",
+                context={
+                    "task_id": task_id,
+                    "raw_count": len(raw_data),
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
 
             if self.fail_on_empty and not raw_data:
                 raise RuntimeError(f"Collector {self.name} returned no data")
 
+            phase_started_at = perf_counter()
             await self.set_phase("transforming", message="正在转换采集数据")
+            await self._log_collection_event(
+                "collector.phase.transforming.start",
+                "Collector transform phase started",
+                context={"task_id": task_id, "raw_count": len(raw_data)},
+            )
             data = self.transform(raw_data)
+            if self.fail_on_empty and not data:
+                raise RuntimeError(f"Collector {self.name} produced no valid records")
+            await self._log_collection_event(
+                "collector.phase.transforming.success",
+                "Collector transform phase completed",
+                context={
+                    "task_id": task_id,
+                    "raw_count": len(raw_data),
+                    "transformed_count": len(data),
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
             snapshot_id = await self._create_snapshot(db, task_id, data, start_time)
 
+            phase_started_at = perf_counter()
             await self.set_phase("saving", message="正在保存采集数据")
+            await self._log_collection_event(
+                "collector.phase.saving.start",
+                "Collector save phase started",
+                context={"task_id": task_id, "snapshot_id": snapshot_id, "transformed_count": len(data)},
+            )
             records_count = await self._save_data(db, data, task_id=task_id, snapshot_id=snapshot_id)
+            await self._log_collection_event(
+                "collector.phase.saving.success",
+                "Collector save phase completed",
+                context={
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "saved_count": records_count,
+                    **self._last_save_summary,
+                    "duration_ms": self._duration_ms(phase_started_at),
+                },
+            )
 
-            task.status = "success"
+            task.status = JobStatus.SUCCESS.value
             task.phase = "completed"
             task.phase_progress = 100.0
             task.phase_message = "采集完成"
@@ -374,10 +408,19 @@ class BaseCollector(ABC):
             task.completed_at = datetime.now(UTC)
             await db.commit()
             await self._publish_task_update(force=True)
-            await self._publish_earth_update(
-                action="collector_completed",
-                records_processed=records_count,
-                task_id=task_id,
+            await self._log_collection_event(
+                "collector.run.completed",
+                "Collector run completed",
+                context={
+                    "status": "success",
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "raw_count": len(raw_data),
+                    "transformed_count": len(data),
+                    "saved_count": records_count,
+                    **self._last_save_summary,
+                    "duration_ms": self._duration_ms(started_at),
+                },
             )
 
             return {
@@ -388,7 +431,7 @@ class BaseCollector(ABC):
             }
         except asyncio.CancelledError:
             await db.rollback()
-            task.status = "cancelled"
+            task.status = JobStatus.CANCELLED.value
             task.phase = "cancelled"
             task.phase_message = "采集已取消"
             task.error_message = "Collection cancelled by operator and rolled back"
@@ -402,10 +445,25 @@ class BaseCollector(ABC):
                 )
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.run.cancelled",
+                "Collector run cancelled",
+                level="warning",
+                context={
+                    "status": "cancelled",
+                    "task_id": task_id,
+                    "snapshot_id": snapshot_id,
+                    "duration_ms": self._duration_ms(started_at),
+                },
+            )
             raise
         except Exception as e:
             await db.rollback()
-            task.status = "failed"
+            if snapshot_id is not None and self.preserve_snapshot_on_failure:
+                await self._rollback_incomplete_run(
+                    db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
+                )
+            task.status = JobStatus.FAILED.value
             task.phase = "failed"
             task.phase_message = str(e)
             task.error_message = str(e)
@@ -413,11 +471,27 @@ class BaseCollector(ABC):
             if snapshot_id is not None:
                 snapshot = await db.get(DataSnapshot, snapshot_id)
                 if snapshot:
-                    snapshot.status = "failed"
+                    snapshot.status = SnapshotStatus.FAILED.value
+                    if self.preserve_snapshot_on_failure:
+                        snapshot.is_current = False
                     snapshot.completed_at = datetime.now(UTC)
                     snapshot.summary = {"error": str(e)}
             await db.commit()
             await self._publish_task_update(force=True)
+            await self._log_collection_event(
+                "collector.run.failed",
+                "Collector run failed",
+                level="error",
+                context=exception_context(
+                    e,
+                    {
+                        "status": "failed",
+                        "task_id": task_id,
+                        "snapshot_id": snapshot_id,
+                        "duration_ms": self._duration_ms(started_at),
+                    },
+                ),
+            )
 
             return {
                 "status": "failed",
@@ -438,12 +512,13 @@ class BaseCollector(ABC):
         from app.models.data_snapshot import DataSnapshot
 
         if not data:
+            self._last_save_summary = {"created": 0, "updated": 0, "unchanged": 0, "deleted": 0}
             if snapshot_id is not None:
                 snapshot = await db.get(DataSnapshot, snapshot_id)
                 if snapshot:
                     snapshot.record_count = 0
                     snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0}
-                    snapshot.status = "success"
+                    snapshot.status = SnapshotStatus.SUCCESS.value
                     snapshot.completed_at = datetime.now(UTC)
                     await db.commit()
             return 0
@@ -576,7 +651,7 @@ class BaseCollector(ABC):
             snapshot = await db.get(DataSnapshot, snapshot_id)
             if snapshot:
                 snapshot.record_count = records_added
-                snapshot.status = "success"
+                snapshot.status = SnapshotStatus.SUCCESS.value
                 snapshot.completed_at = datetime.now(UTC)
                 snapshot.summary = {
                     "created": created_count,
@@ -584,12 +659,51 @@ class BaseCollector(ABC):
                     "unchanged": unchanged_count,
                     "deleted": len(deleted_keys),
                 }
+            self._last_save_summary = {
+                "created": created_count,
+                "updated": updated_count,
+                "unchanged": unchanged_count,
+                "deleted": len(deleted_keys),
+            }
+        else:
+            self._last_save_summary = {
+                "created": created_count,
+                "updated": updated_count,
+                "unchanged": unchanged_count,
+                "deleted": 0,
+            }
 
         await db.commit()
-        invalidate_earth_layer_cache_for_source(self.name)
         await self.update_progress(len(data), force=True)
         return records_added
 
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    async def _log_collection_event(
+        self,
+        event: str,
+        message: str,
+        *,
+        level: str = "info",
+        context: Dict[str, Any] | None = None,
+    ) -> None:
+        await emit_business_log(
+            logger,
+            event=event,
+            message=message,
+            category="collector",
+            level=level,
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                **(context or {}),
+            },
+        )
+
     async def save(self, db: AsyncSession, data: List[Dict[str, Any]]) -> int:
         """Save data to database (legacy method, use _save_data instead)"""
         return await self._save_data(db, data)
@@ -602,10 +716,65 @@ class HTTPCollector(BaseCollector):
     headers: Dict[str, str] = {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
+        started_at = perf_counter()
+        request_host = urlparse(self.base_url).netloc
+        await emit_business_log(
+            logger,
+            event="collector.http.fetch.start",
+            message="Collector HTTP request started",
+            category="collector",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "url_host": request_host,
+            },
+        )
         async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(self.base_url, headers=self.headers)
-            response.raise_for_status()
-            return self.parse_response(response.json())
+            try:
+                response = await client.get(self.base_url, headers=self.headers)
+                response.raise_for_status()
+                payload = response.json()
+                parsed = self.parse_response(payload)
+                await emit_business_log(
+                    logger,
+                    event="collector.http.fetch.success",
+                    message="Collector HTTP request completed",
+                    category="collector",
+                    service="collector",
+                    module=__name__,
+                    context={
+                        "collector_name": self.name,
+                        "datasource_id": getattr(self, "_datasource_id", None),
+                        "url_host": request_host,
+                        "status_code": response.status_code,
+                        "response_bytes": len(response.content or b""),
+                        "parsed_count": len(parsed),
+                        "duration_ms": BaseCollector._duration_ms(started_at),
+                    },
+                )
+                return parsed
+            except Exception as exc:
+                await emit_business_log(
+                    logger,
+                    event="collector.http.fetch.failed",
+                    message="Collector HTTP request failed",
+                    category="collector",
+                    level="error",
+                    service="collector",
+                    module=__name__,
+                    context=exception_context(
+                        exc,
+                        {
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "url_host": request_host,
+                            "duration_ms": BaseCollector._duration_ms(started_at),
+                        },
+                    ),
+                )
+                raise
 
     @abstractmethod
     def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:

backend/app/services/collectors/celestrak.py

@@ -1,15 +1,45 @@
-"""CelesTrak TLE Collector
+"""CelesTrak TLE Collector.
 
-Collects satellite TLE (Two-Line Element) data from CelesTrak.org.
-Free, no authentication required.
+Collects the full active satellite GP element set from CelesTrak.
 """
 
+import asyncio
 import json
-from typing import Dict, Any, List
+from pathlib import Path
+from time import perf_counter
+from typing import Any, Dict, List
+from urllib.parse import urlencode, urlparse
+
 import httpx
 
+from app.core.logging import get_logger
 from app.core.satellite_tle import build_tle_lines_from_elements
+from app.services.business_logs import emit_business_log, exception_context
 from app.services.collectors.base import BaseCollector
+from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
+
+logger = get_logger(__name__, service="collector")
+ACTIVE_GROUP = "active"
+FALLBACK_GROUPS = (
+    "starlink",
+    "gps-ops",
+    "galileo",
+    "glo-ops",
+    "beidou",
+    "geo",
+    "iridium-next",
+    "stations",
+    "visual",
+    "weather",
+    "science",
+    "cubesat",
+    "amateur",
+    "last-30-days",
+)
+FETCH_RETRY_ATTEMPTS = 3
+FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
+CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
+CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
 
 
 class CelesTrakTLECollector(BaseCollector):
@@ -18,55 +48,367 @@ class CelesTrakTLECollector(BaseCollector):
     module = "L3"
     frequency_hours = 24
     data_type = "satellite_tle"
+    _downloader = ResumableFileDownloader(
+        cache_namespace="celestrak",
+        default_accept="application/json",
+        preflight_head=False,
+        cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
+    )
 
     @property
     def base_url(self) -> str:
         return self._resolved_url or ""
 
+    def _active_url(self) -> str:
+        return self._group_url(ACTIVE_GROUP)
+
+    def _group_url(self, group: str) -> str:
+        if not self.base_url:
+            raise RuntimeError("CelesTrak base URL is not configured")
+        return f"{self.base_url}?{urlencode({'GROUP': group, 'FORMAT': 'json'})}"
+
     async def fetch(self) -> List[Dict[str, Any]]:
-        satellite_groups = [
-            "starlink",
-            "gps-ops",
-            "galileo",
-            "glonass",
-            "beidou",
-            "leo",
-            "geo",
-            "iridium-next",
-        ]
+        url = self._active_url()
+        last_error: Exception | None = None
 
-        all_satellites = []
-
-        async with httpx.AsyncClient(timeout=120.0) as client:
-            for group in satellite_groups:
+        async with httpx.AsyncClient(timeout=180.0, follow_redirects=True) as client:
+            for attempt in range(1, FETCH_RETRY_ATTEMPTS + 1):
+                started_at = perf_counter()
                 try:
-                    url = f"{self.base_url}?GROUP={group}&FORMAT=json"
-                    response = await client.get(url)
+                    await emit_business_log(
+                        logger,
+                        event="collector.celestrak.download.start",
+                        message="CelesTrak active satellite download started",
+                        category="collector",
+                        service="collector",
+                        module=__name__,
+                        context={
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "group": ACTIVE_GROUP,
+                            "attempt": attempt,
+                            "url_host": urlparse(url).netloc,
+                        },
+                    )
+                    body_path = await self._downloader.download_file(
+                        client,
+                        url,
+                        extension=".json",
+                        accept="application/json",
+                        progress_callback=self._report_download_progress,
+                        validate_existing=self._validate_json_file,
+                    )
+                    data = await self._load_downloaded_payload(body_path, url)
+                    await emit_business_log(
+                        logger,
+                        event="collector.celestrak.download.success",
+                        message="CelesTrak active satellite download completed",
+                        category="collector",
+                        service="collector",
+                        module=__name__,
+                        context={
+                            "collector_name": self.name,
+                            "datasource_id": getattr(self, "_datasource_id", None),
+                            "group": ACTIVE_GROUP,
+                            "attempt": attempt,
+                            "record_count": len(data),
+                            "duration_ms": self._duration_ms(started_at),
+                        },
+                    )
+                    return data
+                except DownloadHTTPStatusError as exc:
+                    if self._is_not_updated_response(exc):
+                        cached_path = self._downloader.get_cached_file(
+                            url,
+                            ".json",
+                            validate_existing=self._validate_json_file,
+                        )
+                        if cached_path is not None:
+                            data = await self._load_downloaded_payload(cached_path, url)
+                            await emit_business_log(
+                                logger,
+                                event="collector.celestrak.download.cached_not_updated",
+                                message="CelesTrak active satellite data has not changed; using cached download",
+                                category="collector",
+                                level="warning",
+                                service="collector",
+                                module=__name__,
+                                context={
+                                    "collector_name": self.name,
+                                    "datasource_id": getattr(self, "_datasource_id", None),
+                                    "group": ACTIVE_GROUP,
+                                    "attempt": attempt,
+                                    "record_count": len(data),
+                                    "duration_ms": self._duration_ms(started_at),
+                                },
+                            )
+                            return data
+                        await emit_business_log(
+                            logger,
+                            event="collector.celestrak.download.not_updated_no_cache",
+                            message="CelesTrak active satellite data has not changed; trying fallback groups",
+                            category="collector",
+                            level="warning",
+                            service="collector",
+                            module=__name__,
+                            context=exception_context(
+                                exc,
+                                {
+                                    "collector_name": self.name,
+                                    "datasource_id": getattr(self, "_datasource_id", None),
+                                    "group": ACTIVE_GROUP,
+                                    "attempt": attempt,
+                                    "duration_ms": self._duration_ms(started_at),
+                                },
+                            ),
+                        )
+                        return await self._fetch_fallback_groups(client, active_error=exc)
+                    raise
+                except Exception as exc:
+                    last_error = exc
+                    is_final_attempt = attempt >= FETCH_RETRY_ATTEMPTS
+                    await emit_business_log(
+                        logger,
+                        event=(
+                            "collector.celestrak.download.failed"
+                            if is_final_attempt
+                            else "collector.celestrak.download.retry"
+                        ),
+                        message=(
+                            "CelesTrak active satellite download failed"
+                            if is_final_attempt
+                            else "CelesTrak active satellite download will retry"
+                        ),
+                        category="collector",
+                        level="error" if is_final_attempt else "warning",
+                        service="collector",
+                        module=__name__,
+                        context=exception_context(
+                            exc,
+                            {
+                                "collector_name": self.name,
+                                "datasource_id": getattr(self, "_datasource_id", None),
+                                "group": ACTIVE_GROUP,
+                                "attempt": attempt,
+                                "duration_ms": self._duration_ms(started_at),
+                            },
+                        ),
+                    )
+                    if not is_final_attempt:
+                        await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
 
-                    if response.status_code == 200:
-                        data = response.json()
-                        if isinstance(data, list):
-                            for item in data:
-                                if isinstance(item, dict):
-                                    item["_celestrak_group"] = group
-                            all_satellites.extend(data)
-                            print(f"CelesTrak: Fetched {len(data)} satellites from group '{group}'")
-                except Exception as e:
-                    print(f"CelesTrak: Error fetching group '{group}': {e}")
+        raise RuntimeError(
+            f"CelesTrak active satellite download failed after retries: {last_error}"
+        )
 
-        if not all_satellites:
-            return self._get_sample_data()
+    async def _fetch_fallback_groups(
+        self,
+        client: httpx.AsyncClient,
+        *,
+        active_error: DownloadHTTPStatusError,
+    ) -> List[Dict[str, Any]]:
+        started_at = perf_counter()
+        records_by_norad: dict[str, Dict[str, Any]] = {}
+        group_counts: dict[str, int] = {}
 
-        print(f"CelesTrak: Total satellites fetched: {len(all_satellites)}")
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.fallback_groups.start",
+            message="CelesTrak fallback group download started",
+            category="collector",
+            level="warning",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "groups": list(FALLBACK_GROUPS),
+                "reason": "active_not_updated_without_cache",
+            },
+        )
 
-        # Return raw data - base.run() will call transform()
-        return all_satellites
+        try:
+            for group in FALLBACK_GROUPS:
+                group_url = self._group_url(group)
+                try:
+                    body_path = await self._downloader.download_file(
+                        client,
+                        group_url,
+                        extension=".json",
+                        accept="application/json",
+                        validate_existing=self._validate_json_file,
+                    )
+                except DownloadHTTPStatusError as exc:
+                    if not self._is_not_updated_response(exc):
+                        raise RuntimeError(
+                            f"CelesTrak fallback group '{group}' download failed: {exc}"
+                        ) from exc
+                    body_path = self._downloader.get_cached_file(
+                        group_url,
+                        ".json",
+                        validate_existing=self._validate_json_file,
+                    )
+                    if body_path is None:
+                        raise RuntimeError(
+                            f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
+                        ) from exc
+
+                group_records = await self._load_downloaded_payload(
+                    body_path,
+                    group_url,
+                    query_group=group,
+                    constellation_group=group,
+                )
+                group_counts[group] = len(group_records)
+                for item in group_records:
+                    norad_cat_id = item.get("NORAD_CAT_ID")
+                    if norad_cat_id is None:
+                        continue
+                    records_by_norad.setdefault(str(norad_cat_id), item)
+        except Exception as exc:
+            await emit_business_log(
+                logger,
+                event="collector.celestrak.fallback_groups.failed",
+                message="CelesTrak fallback group download failed",
+                category="collector",
+                level="error",
+                service="collector",
+                module=__name__,
+                context=exception_context(
+                    exc,
+                    {
+                        "collector_name": self.name,
+                        "datasource_id": getattr(self, "_datasource_id", None),
+                        "groups": list(FALLBACK_GROUPS),
+                        "completed_groups": list(group_counts),
+                        "duration_ms": self._duration_ms(started_at),
+                    },
+                ),
+            )
+            raise RuntimeError(
+                "CelesTrak active data has not updated since this network's last successful download, "
+                "no active cache is available, and fallback group mode failed. Wait until CelesTrak "
+                "publishes the next GP update, restore the Planet download cache, or use Space-Track."
+            ) from active_error
+
+        records = list(records_by_norad.values())
+        if not records:
+            raise RuntimeError("CelesTrak fallback group mode produced no satellite records")
+
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.fallback_groups.success",
+            message="CelesTrak fallback group download completed",
+            category="collector",
+            level="warning",
+            service="collector",
+            module=__name__,
+            context={
+                "collector_name": self.name,
+                "datasource_id": getattr(self, "_datasource_id", None),
+                "groups": list(FALLBACK_GROUPS),
+                "group_counts": group_counts,
+                "record_count": len(records),
+                "duration_ms": self._duration_ms(started_at),
+            },
+        )
+        return records
+
+    async def _load_downloaded_payload(
+        self,
+        body_path: Path,
+        url: str,
+        *,
+        query_group: str = ACTIVE_GROUP,
+        constellation_group: str | None = None,
+    ) -> List[Dict[str, Any]]:
+        try:
+            data = self._load_active_payload(body_path)
+        except RuntimeError as exc:
+            await self._log_parse_failure(exc)
+            raise
+        for item in data:
+            item["_celestrak_query_group"] = query_group
+            item["_celestrak_source_url"] = url
+            if constellation_group:
+                item["_celestrak_group"] = constellation_group
+        return data
+
+    @staticmethod
+    def _is_not_updated_response(exc: DownloadHTTPStatusError) -> bool:
+        return exc.status_code == 403 and CELESTRAK_NOT_UPDATED_MARKER in exc.body
+
+    @staticmethod
+    def _duration_ms(started_at: float) -> int:
+        return int((perf_counter() - started_at) * 1000)
+
+    async def _report_download_progress(self, downloaded: int, total: int | None) -> None:
+        if total and total > 0:
+            await self.update_phase_progress(
+                current=min(downloaded, total),
+                total=total,
+                unit="bytes",
+                message=f"正在下载 CelesTrak active 卫星数据 {downloaded}/{total} bytes",
+                commit=True,
+            )
+
+    @classmethod
+    def _validate_json_file(cls, path: Path) -> bool:
+        try:
+            cls._load_active_payload(path)
+        except RuntimeError:
+            return False
+        return True
+
+    async def _log_parse_failure(self, exc: Exception) -> None:
+        await emit_business_log(
+            logger,
+            event="collector.celestrak.parse.failed",
+            message="CelesTrak active satellite JSON parsing failed",
+            category="collector",
+            level="error",
+            service="collector",
+            module=__name__,
+            context=exception_context(
+                exc,
+                {
+                    "collector_name": self.name,
+                    "datasource_id": getattr(self, "_datasource_id", None),
+                    "group": ACTIVE_GROUP,
+                },
+            ),
+        )
+
+    @staticmethod
+    def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
+        try:
+            raw = json.loads(path.read_text(encoding="utf-8"))
+        except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
+            raise RuntimeError(f"CelesTrak active payload is not valid JSON: {exc}") from exc
+        if not isinstance(raw, list):
+            raise RuntimeError("CelesTrak active payload is not a JSON array")
+
+        records: List[Dict[str, Any]] = []
+        invalid_count = 0
+        for item in raw:
+            if isinstance(item, dict) and item.get("NORAD_CAT_ID") is not None:
+                records.append(item)
+            else:
+                invalid_count += 1
+        if invalid_count:
+            raise RuntimeError(
+                f"CelesTrak active payload contains {invalid_count} invalid record(s)"
+            )
+        if not records:
+            raise RuntimeError("CelesTrak active payload contains no satellite records")
+        return records
 
     def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
         transformed = []
         for item in raw_data:
+            norad_cat_id = item.get("NORAD_CAT_ID")
             tle_line1, tle_line2 = build_tle_lines_from_elements(
-                norad_cat_id=item.get("NORAD_CAT_ID"),
+                norad_cat_id=norad_cat_id,
                 epoch=item.get("EPOCH"),
                 inclination=item.get("INCLINATION"),
                 raan=item.get("RA_OF_ASC_NODE"),
@@ -75,14 +417,18 @@ class CelesTrakTLECollector(BaseCollector):
                 mean_anomaly=item.get("MEAN_ANOMALY"),
                 mean_motion=item.get("MEAN_MOTION"),
             )
+            constellation_group = self._infer_constellation_group(item)
 
             transformed.append(
                 {
+                    "source_id": str(norad_cat_id),
                     "name": item.get("OBJECT_NAME", "Unknown"),
                     "reference_date": item.get("EPOCH", ""),
                     "metadata": {
-                        "constellation_group": item.get("_celestrak_group"),
-                        "norad_cat_id": item.get("NORAD_CAT_ID"),
+                        "constellation_group": constellation_group,
+                        "celestrak_query_group": item.get("_celestrak_query_group") or ACTIVE_GROUP,
+                        "celestrak_source_url": item.get("_celestrak_source_url"),
+                        "norad_cat_id": norad_cat_id,
                         "international_designator": item.get("OBJECT_ID"),
                         "epoch": item.get("EPOCH"),
                         "mean_motion": item.get("MEAN_MOTION"),
@@ -105,6 +451,19 @@ class CelesTrakTLECollector(BaseCollector):
             )
         return transformed
 
+    @staticmethod
+    def _infer_constellation_group(item: Dict[str, Any]) -> str | None:
+        explicit_group = str(item.get("_celestrak_group") or "").strip().lower()
+        if explicit_group and explicit_group != ACTIVE_GROUP:
+            return explicit_group
+
+        name = str(item.get("OBJECT_NAME") or "").strip().upper()
+        if name.startswith("STARLINK"):
+            return "starlink"
+        if name.startswith("IRIDIUM"):
+            return "iridium-next"
+        return None
+
     def _get_sample_data(self) -> List[Dict[str, Any]]:
         return [
             {

backend/app/services/collectors/compute_base.py

@@ -0,0 +1,82 @@
+"""Quality gates shared by the two compute sources."""
+
+from datetime import datetime
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.countries import normalize_country
+from app.models.compute_center_location import ComputeCenterLocationRecord
+from app.models.data_snapshot import DataSnapshot
+from app.services.collectors.base import BaseCollector
+
+
+class ComputeCollector(BaseCollector):
+    fail_on_empty = True
+    preserve_snapshot_on_failure = True
+
+    async def _create_snapshot(
+        self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
+    ) -> int:
+        previous_count = await db.scalar(
+            select(DataSnapshot.record_count)
+            .where(
+                DataSnapshot.source == self.name,
+                DataSnapshot.is_current.is_(True),
+                DataSnapshot.status == "success",
+            )
+            .order_by(DataSnapshot.id.desc())
+            .limit(1)
+        )
+        if previous_count and len(data) < previous_count * 0.6:
+            raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
+        return await super()._create_snapshot(db, task_id, data, started_at)
+
+    async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
+        if self.name == "top500":
+            await self._copy_verified_locations(db, data)
+        return await super()._save_data(db, data, **kwargs)
+
+    async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
+        """Keep historical location keys; copy only unambiguous exact identities.
+
+        Never migrate by rank: the machine occupying rank 1 changes over time.
+        """
+        names = [item["name"] for item in data]
+        locations = (
+            (
+                await db.execute(
+                    select(ComputeCenterLocationRecord).where(
+                        ComputeCenterLocationRecord.source == self.name,
+                        ComputeCenterLocationRecord.name.in_(names),
+                    )
+                )
+            )
+            .scalars()
+            .all()
+        )
+        existing = {location.source_id for location in locations}
+        for item in data:
+            if item["source_id"] in existing:
+                continue
+            country = normalize_country(item["country"])
+            if not country:
+                continue
+            matches = [
+                location
+                for location in locations
+                if location.name == item["name"]
+                and normalize_country(location.country) == country
+                and (not location.site or location.site == item.get("metadata", {}).get("site"))
+                and not location.source_id.startswith("top500_system_")
+            ]
+            if len(matches) != 1:
+                continue
+            old = matches[0]
+            values = {
+                column.name: getattr(old, column.name)
+                for column in old.__table__.columns
+                if column.name not in {"id", "source_id", "created_at", "updated_at"}
+            }
+            db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
+            existing.add(item["source_id"])

backend/app/services/collectors/downloads.py

@@ -4,7 +4,7 @@ from __future__ import annotations
 
 import hashlib
 import json
-import tempfile
+import os
 import time
 from datetime import UTC, datetime
 from pathlib import Path
@@ -12,11 +12,35 @@ from typing import Any, Awaitable, Callable
 
 import httpx
 
-
 ProgressCallback = Callable[[int, int | None], Awaitable[None]]
 ValidateCallback = Callable[[Path], bool]
 
 
+class DownloadHTTPStatusError(RuntimeError):
+    """HTTP status error that keeps the upstream response body for caller-specific handling."""
+
+    def __init__(self, *, url: str, status_code: int, body: str) -> None:
+        self.url = url
+        self.status_code = status_code
+        self.body = body
+        preview = body.strip().replace("\r", " ").replace("\n", " ")[:240]
+        suffix = f": {preview}" if preview else ""
+        super().__init__(f"HTTP {status_code} while downloading {url}{suffix}")
+
+
+def default_download_cache_root() -> Path:
+    configured = os.getenv("PLANET_DOWNLOAD_CACHE_DIR")
+    if configured:
+        return Path(configured).expanduser()
+    planet_cache = os.getenv("PLANET_CACHE_DIR")
+    if planet_cache:
+        return Path(planet_cache).expanduser() / "downloads"
+    xdg_cache = os.getenv("XDG_CACHE_HOME")
+    if xdg_cache:
+        return Path(xdg_cache).expanduser() / "planet" / "downloads"
+    return Path.home() / ".cache" / "planet" / "downloads"
+
+
 class ResumableFileDownloader:
     """Download files with cache validators and byte-range resume support."""
 
@@ -26,10 +50,15 @@ class ResumableFileDownloader:
         cache_namespace: str,
         user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
         default_accept: str = "*/*",
+        cache_root: Path | None = None,
+        preflight_head: bool = True,
+        cache_ttl_seconds: float = 0,
     ) -> None:
-        self._cache_dir = Path(tempfile.gettempdir()) / "planet-download-cache" / cache_namespace
+        self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
         self._user_agent = user_agent
         self._default_accept = default_accept
+        self._preflight_head = preflight_head
+        self._cache_ttl_seconds = cache_ttl_seconds
 
     @staticmethod
     def _cache_key(url: str) -> str:
@@ -43,13 +72,33 @@ class ResumableFileDownloader:
         meta_path = self._cache_dir / f"{key}.meta.json"
         return final_path, part_path, meta_path
 
+    def cached_file_path(self, url: str, extension: str) -> Path:
+        final_path, _, _ = self._cache_paths(url, extension)
+        return final_path
+
+    def get_cached_file(
+        self,
+        url: str,
+        extension: str,
+        *,
+        validate_existing: ValidateCallback | None = None,
+    ) -> Path | None:
+        final_path = self.cached_file_path(url, extension)
+        if not final_path.exists():
+            return None
+        if validate_existing and not validate_existing(final_path):
+            final_path.unlink(missing_ok=True)
+            return None
+        return final_path
+
     @staticmethod
     def _load_meta(meta_path: Path) -> dict[str, Any]:
         if not meta_path.exists():
             return {}
         try:
-            return json.loads(meta_path.read_text(encoding="utf-8"))
-        except (json.JSONDecodeError, OSError):
+            data = json.loads(meta_path.read_text(encoding="utf-8"))
+            return data if isinstance(data, dict) else {}
+        except json.JSONDecodeError, OSError:
             return {}
 
     @staticmethod
@@ -66,20 +115,25 @@ class ResumableFileDownloader:
             return last_modified == str(meta.get("last_modified") or "").strip()
         return True
 
+    @staticmethod
+    def _response_info(response: httpx.Response) -> dict[str, Any]:
+        # httpx yields decoded bytes; encoded Content-Length cannot validate them.
+        raw_length = response.headers.get("content-length")
+        encoded = response.headers.get("content-encoding", "identity") != "identity"
+        return {
+            "etag": response.headers.get("etag"),
+            "last_modified": response.headers.get("last-modified"),
+            "content_length": int(raw_length) if raw_length and not encoded else None,
+            "accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
+        }
+
     async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
         try:
             response = await client.head(url)
             if response.status_code >= 400:
                 return {}
-            content_length_raw = response.headers.get("content-length")
-            content_length = int(content_length_raw) if content_length_raw else None
-            return {
-                "etag": response.headers.get("etag"),
-                "last_modified": response.headers.get("last-modified"),
-                "content_length": content_length,
-                "accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
-            }
-        except (httpx.HTTPError, ValueError):
+            return self._response_info(response)
+        except httpx.HTTPError, ValueError:
             return {}
 
     async def download_file(
@@ -94,14 +148,24 @@ class ResumableFileDownloader:
     ) -> Path:
         self._cache_dir.mkdir(parents=True, exist_ok=True)
         final_path, part_path, meta_path = self._cache_paths(url, extension)
+        cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
+        if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
+            size = cached_path.stat().st_size
+            if progress_callback:
+                await progress_callback(size, size)
+            return cached_path
         meta = self._load_meta(meta_path)
-        remote = await self.fetch_remote_info(client, url)
+        remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
         expected_size = remote.get("content_length")
 
         if final_path.exists():
             local_size = final_path.stat().st_size
             size_match = expected_size is None or local_size == expected_size
-            if self._validators_match(meta, remote) and size_match:
+            if (
+                (remote.get("etag") or remote.get("last_modified"))
+                and self._validators_match(meta, remote)
+                and size_match
+            ):
                 if validate_existing and not validate_existing(final_path):
                     final_path.unlink(missing_ok=True)
                 else:
@@ -136,23 +200,33 @@ class ResumableFileDownloader:
                 headers["If-Range"] = str(remote.get("last_modified"))
 
         async with client.stream("GET", url, headers=headers) as response:
-            if response.status_code == 304 and final_path.exists():
-                if progress_callback and expected_size and expected_size > 0:
-                    await progress_callback(expected_size, expected_size)
+            if response.status_code == 304:
+                if cached_path is None:
+                    raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
+                final_path.touch()
+                size = final_path.stat().st_size
+                if progress_callback:
+                    await progress_callback(size, size)
                 return final_path
-            response.raise_for_status()
+            if response.status_code >= 400:
+                body = (await response.aread()).decode("utf-8", errors="replace")
+                raise DownloadHTTPStatusError(url=url, status_code=response.status_code, body=body)
 
             if response.status_code == 206 and resume_from > 0:
                 mode = "ab"
             else:
                 mode = "wb"
                 resume_from = 0
+                remote = self._response_info(response)
+                expected_size = remote.get("content_length")
 
             downloaded = resume_from
             last_emit_bytes = 0
             last_emit_time = time.monotonic()
             min_emit_bytes = (
-                max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
+                max(expected_size // 150, 512 * 1024)
+                if expected_size and expected_size > 0
+                else 1024 * 1024
             )
 
             with part_path.open(mode) as f:
@@ -183,6 +257,10 @@ class ResumableFileDownloader:
                 f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
             )
 
+        # Never replace the last good payload with an HTML error or invalid JSON.
+        if validate_existing and not validate_existing(part_path):
+            part_path.unlink(missing_ok=True)
+            raise RuntimeError(f"Downloaded file validation failed for {url}")
         part_path.replace(final_path)
         self._save_meta(
             meta_path,
@@ -195,9 +273,6 @@ class ResumableFileDownloader:
             },
         )
 
-        if validate_existing and not validate_existing(final_path):
-            raise RuntimeError(f"Downloaded file validation failed for {url}")
-
         if progress_callback and expected_size and expected_size > 0:
             await progress_callback(expected_size, expected_size)
 

backend/app/services/collectors/epoch_ai.py

@@ -1,119 +1,135 @@
-"""Epoch AI GPU Clusters Collector
-
-Collects data from Epoch AI GPU clusters tracking.
-https://epoch.ai/data/gpu-clusters
-"""
+"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
 
+import csv
+import hashlib
+import io
+import json
+import math
 import re
-from typing import Dict, Any, List
-from datetime import UTC, datetime
-from bs4 import BeautifulSoup
+from urllib.parse import urlparse
+
 import httpx
 
-from app.services.collectors.base import BaseCollector
+from app.core.countries import normalize_country
+from app.services.collectors.compute_base import ComputeCollector
+from app.services.epoch_location import assess_epoch_location
+
+EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
 
 
-
-class EpochAIGPUCollector(BaseCollector):
+class EpochAIGPUCollector(ComputeCollector):
     name = "epoch_ai_gpu"
     priority = "P0"
     module = "L1"
     frequency_hours = 6
     data_type = "gpu_cluster"
 
-    async def fetch(self) -> List[Dict[str, Any]]:
-        """Fetch Epoch AI GPU clusters data from webpage"""
-        url = self._resolved_url or ""
-
-        async with httpx.AsyncClient(timeout=60.0) as client:
+    async def fetch(self) -> list[dict]:
+        configured = self._resolved_url or EPOCH_CSV_URL
+        parsed = urlparse(configured)
+        url = (
+            EPOCH_CSV_URL
+            if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
+            else configured
+        )
+        async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
             response = await client.get(url)
             response.raise_for_status()
-            return self.parse_response(response.text)
+        return self.parse_response(response.text)
 
-    def parse_response(self, html: str) -> List[Dict[str, Any]]:
-        """Parse Epoch AI webpage to extract GPU cluster data"""
-        data = []
-        soup = BeautifulSoup(html, "html.parser")
+    def parse_response(self, text: str) -> list[dict]:
+        reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
+        required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
+        if not required.issubset(reader.fieldnames or []):
+            raise ValueError(
+                "Epoch CSV schema changed or HTML returned; previous snapshot retained"
+            )
+        entries = []
+        seen = set()
+        anonymous_counts = {}
+        for row in reader:
+            name = (row.get("Name") or "").strip()
+            country = normalize_country(row.get("Country"))
+            if not name:
+                raise ValueError("Epoch row has missing identity")
+            source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
+            anonymous = "anonym" in name.lower()
+            if anonymous:
+                # Public anonymous rows have no facility identity. Preserve
+                # multiplicity within snapshots, without guessing names.
+                digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
+                anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
+                source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
+            if source_id in seen:
+                raise ValueError("Epoch identity collision; previous snapshot retained")
+            seen.add(source_id)
+            raw_date = row.get("First Operational Date", "")
+            reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
+            value = None
+            raw_value = row.get("16-bit OP/s (log)", "")
+            if raw_value:
+                exponent = float(raw_value)
+                if not math.isfinite(exponent) or not 0 <= exponent <= 30:
+                    raise ValueError("Epoch performance exponent out of bounds")
+                value = 10**exponent
+            anonymous = anonymous or (country == "中国" and not row.get("Location"))
+            location = row.get("Location")
+            location_assessment = assess_epoch_location(
+                location,
+                country=country,
+                raw_country=row.get("Country"),
+            )
+            entries.append(
+                {
+                    "source_id": source_id,
+                    "name": name,
+                    "country": country,
+                    "value": value,
+                    "unit": "FLOP/s",
+                    "reference_date": reference_date,
+                    "latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
+                    "longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
+                    "metadata": {
+                        "compute_contract": "v1",
+                        "precision": "FP16",
+                        "measurement": "theoretical",
+                        "source_url": EPOCH_CSV_URL,
+                        "raw_reference_date": raw_date,
+                        "status": row.get("Status"),
+                        "certainty": row.get("Certainty"),
+                        "operator": row.get("Owner"),
+                        "gpu_type": row.get("Chip type (primary)"),
+                        "gpu_count": row.get("Chip quantity (primary)"),
+                        "location": location,
+                        "location_precision": location_assessment.precision,
+                        "location_precision_reason": location_assessment.reason,
+                        "raw_country": row.get("Country"),
+                        "raw_performance": raw_value,
+                        "performance_encoding": "log10",
+                        "builds_upon": row.get("Builds Upon"),
+                        "superseded_by": row.get("Superseded by"),
+                        "possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
+                        "include_in_standard_analysis": row.get(
+                            "Include in Standard Analysis", ""
+                        ).lower()
+                        == "true",
+                        "rounded": country == "中国",
+                        "anonymous": anonymous,
+                        "identity_scope": "snapshot_record" if anonymous else "named_cluster",
+                        "country_unresolved": country is None,
+                        "sources": [
+                            row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
+                        ],
+                    },
+                }
+            )
+        return entries
 
-        # Try to find data table on the page
-        tables = soup.find_all("table")
-        for table in tables:
-            rows = table.find_all("tr")
-            for row in rows[1:]:  # Skip header
-                cells = row.find_all(["td", "th"])
-                if len(cells) >= 5:
-                    try:
-                        cluster_name = cells[0].get_text(strip=True)
-                        if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
-                            continue
-
-                        location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
-                        country, city = self._parse_location(location_cell)
-
-                        perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
-
-                        entry = {
-                            "source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
-                            "name": cluster_name,
-                            "country": country,
-                            "city": city,
-                            "latitude": "",
-                            "longitude": "",
-                            "value": self._parse_performance(perf_cell),
-                            "unit": "TFlop/s",
-                            "metadata": {
-                                "raw_data": perf_cell,
-                            },
-                            "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
-                        }
-                        data.append(entry)
-                    except (ValueError, IndexError, AttributeError):
-                        continue
-
-        # If no table found, return sample data
-        if not data:
-            data = self._get_sample_data()
-
-        return data
-
-    def _parse_location(self, location: str) -> tuple:
-        """Parse location string into country and city"""
-        if not location:
-            return "", ""
-        if "," in location:
-            parts = location.rsplit(",", 1)
-            city = parts[0].strip()
-            country = parts[1].strip() if len(parts) > 1 else ""
-            return country, city
-        return location, ""
-
-    def _parse_performance(self, perf: str) -> str:
-        """Parse performance string to extract value"""
-        if not perf:
-            return "0"
-        match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
-        if match:
-            return match.group(1).replace(",", "")
-        match = re.search(r"([\d,.]+)", perf)
-        if match:
-            return match.group(1).replace(",", "")
-        return "0"
-
-    def _get_sample_data(self) -> List[Dict[str, Any]]:
-        """Return sample data for testing when scraping fails"""
-        return [
-            {
-                "source_id": "epoch_sample_1",
-                "name": "Sample GPU Cluster",
-                "country": "United States",
-                "city": "San Francisco, CA",
-                "latitude": "",
-                "longitude": "",
-                "value": "1000",
-                "unit": "TFlop/s",
-                "metadata": {
-                    "note": "Sample data - Epoch AI page structure may vary",
-                },
-                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
-            },
-        ]
+    @staticmethod
+    def coordinate(value: str | None, bound: int) -> float | None:
+        if not value:
+            return None
+        number = float(value)
+        if not math.isfinite(number) or abs(number) > bound:
+            raise ValueError("Epoch coordinate out of bounds")
+        return number

backend/app/services/collectors/news_live_streams.py

@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
     DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
     DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
     DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
-    DEFAULT_IPTV_ORG_MAX_SOURCES = 120
+    DEFAULT_IPTV_ORG_MAX_SOURCES = 0  # Zero keeps the complete matching channel catalog.
 
     async def fetch(self) -> list[dict[str, Any]]:
         request_url = (self._resolved_url or "").strip()
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
                     "reference_date": datetime.now(UTC).isoformat(),
                 }
             )
-            if len(normalized) >= max_sources:
+            if max_sources > 0 and len(normalized) >= max_sources:
                 break
 
         return normalized

backend/app/services/collectors/peeringdb.py

@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
 """
 
 import asyncio
-import os
-from typing import Dict, Any, List
 from datetime import UTC, datetime
+import os
+from typing import Any, Dict, List
+from urllib.parse import urlencode
 
 import httpx
-from urllib.parse import urlencode
+
+from app.core.logging import get_logger
 from app.services.collectors.base import HTTPCollector
 
 
 # PeeringDB API key - read from environment variable
 PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
+logger = get_logger(__name__, service="collector")
 
 
 class PeeringDBIXPCollector(HTTPCollector):
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
                     if response.status_code == 429:
                         # Rate limited - wait and retry with exponential backoff
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
             "User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
             "Accept": "application/json",
         }
+
     @property
     def request_url(self) -> str:
         base = self._resolved_url or self.base_url
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
 
                     if response.status_code == 429:
                         delay = base_delay * (2**attempt)
-                        print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                        logger.warning_event(
+                            "PeeringDB rate limited; retrying after delay",
+                            event="collector.peeringdb.rate_limited",
+                            context={"delay_seconds": delay, "attempt": attempt + 1},
+                        )
                         await asyncio.sleep(delay)
                         last_error = "Rate limited"
                         continue
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
             except httpx.HTTPStatusError as e:
                 if e.response.status_code == 429:
                     delay = base_delay * (2**attempt)
-                    print(f"PeeringDB rate limited, waiting {delay}s before retry...")
+                    logger.warning_event(
+                        "PeeringDB rate limited; retrying after delay",
+                        event="collector.peeringdb.rate_limited",
+                        context={"delay_seconds": delay, "attempt": attempt + 1},
+                    )
                     await asyncio.sleep(delay)
                     last_error = "Rate limited"
                     continue
                 raise
 
-        print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
+        logger.warning_event(
+            "PeeringDB collection failed after retries",
+            event="collector.peeringdb.retries_exhausted",
+            context={"max_retries": max_retries, "last_error": last_error},
+        )
         return {}
 
     async def fetch(self) -> List[Dict[str, Any]]:

backend/app/services/collectors/spacetrack.py

@@ -1,17 +1,21 @@
-"""Space-Track TLE Collector
+"""Space-Track TLE Collector.
 
 Collects satellite TLE (Two-Line Element) data from Space-Track.org.
 API documentation: https://www.space-track.org/documentation
 """
 
-import json
-from typing import Dict, Any, List
-import httpx
+from typing import Any, Dict, List
 from urllib.parse import urlparse
 
-from app.services.collectors.base import BaseCollector
+import httpx
+
 from app.core.data_sources import get_data_sources_config
+from app.core.logging import get_logger
 from app.core.satellite_tle import build_tle_lines_from_elements
+from app.services.collectors.base import BaseCollector
+
+
+logger = get_logger(__name__, service="collector")
 
 
 class SpaceTrackTLECollector(BaseCollector):
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
         password = settings.SPACETRACK_PASSWORD
 
         if not username or not password:
-            print("SPACETRACK: No credentials configured, using sample data")
+            logger.warning_event(
+                "Space-Track credentials are not configured; using sample data",
+                event="collector.spacetrack.credentials_missing",
+            )
             return self._get_sample_data()
 
-        print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
+        logger.info_event(
+            "Space-Track TLE fetch started",
+            event="collector.spacetrack.fetch.start",
+        )
 
         try:
             async with httpx.AsyncClient(
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
                         "password": password,
                     },
                 )
-                print(f"SPACETRACK: Login response status: {login_response.status_code}")
-                print(f"SPACETRACK: Login response URL: {login_response.url}")
+                logger.info_event(
+                    "Space-Track login response received",
+                    event="collector.spacetrack.login.response",
+                    context={"status_code": login_response.status_code},
+                )
 
                 if login_response.status_code == 403:
-                    print("SPACETRACK: Trying alternate login method...")
+                    logger.warning_event(
+                        "Space-Track login returned forbidden; trying alternate method",
+                        event="collector.spacetrack.login.forbidden",
+                    )
 
                     async with httpx.AsyncClient(
                         timeout=120.0,
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
                     ) as alt_client:
                         await alt_client.get(f"{self.site_root}/")
 
-                        form_data = {
-                            "username": username,
-                            "password": password,
-                            "query": "class/gp/NORAD_CAT_ID/25544/format/json",
-                        }
                         alt_login = await alt_client.post(
                             self.login_url,
                             data={
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
                                 "password": password,
                             },
                         )
-                        print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
+                        logger.info_event(
+                            "Space-Track alternate login response received",
+                            event="collector.spacetrack.alt_login.response",
+                            context={"status_code": alt_login.status_code},
+                        )
 
                         if alt_login.status_code == 200:
                             tle_response = await alt_client.get(self.probe_url)
                             if tle_response.status_code == 200:
                                 data = tle_response.json()
-                                print(f"SPACETRACK: Received {len(data)} records via alt method")
+                                logger.info_event(
+                                    "Space-Track alternate query completed",
+                                    event="collector.spacetrack.alt_query.completed",
+                                    context={"record_count": len(data)},
+                                )
                                 return data
 
                 if login_response.status_code != 200:
-                    print(f"SPACETRACK: Login failed, using sample data")
+                    logger.warning_event(
+                        "Space-Track login failed; using sample data",
+                        event="collector.spacetrack.login.failed",
+                        context={"status_code": login_response.status_code},
+                    )
                     return self._get_sample_data()
 
                 tle_response = await client.get(self.probe_url)
-                print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
-
-                if tle_response.status_code != 200:
-                    print(f"SPACETRACK: Query failed, using sample data")
-                    return self._get_sample_data()
-
-                data = tle_response.json()
-                print(f"SPACETRACK: Received {len(data)} records")
-                return data
-        except Exception as e:
-            print(f"SPACETRACK: Error - {e}, using sample data")
-            return self._get_sample_data()
-
-        print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
-
-        try:
-            async with httpx.AsyncClient(
-                timeout=120.0,
-                follow_redirects=True,
-                headers={
-                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
-                    "Accept": "application/json, text/html, */*",
-                    "Accept-Language": "en-US,en;q=0.9",
-                },
-            ) as client:
-                # First, visit the main page to get any cookies
-                await client.get(f"{self.site_root}/")
-
-                # Login to get session cookie
-                login_response = await client.post(
-                    self.login_url,
-                    data={
-                        "identity": username,
-                        "password": password,
-                    },
+                logger.info_event(
+                    "Space-Track TLE query response received",
+                    event="collector.spacetrack.query.response",
+                    context={"status_code": tle_response.status_code},
                 )
-                print(f"SPACETRACK: Login response status: {login_response.status_code}")
-                print(f"SPACETRACK: Login response URL: {login_response.url}")
-                print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
-
-                if login_response.status_code != 200:
-                    print(f"SPACETRACK: Login failed, using sample data")
-                    return self._get_sample_data()
-
-                # Query for TLE data (get first 1000 satellites)
-                tle_response = await client.get(self.query_url)
-                print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
 
                 if tle_response.status_code != 200:
-                    print(f"SPACETRACK: Query failed, using sample data")
+                    logger.warning_event(
+                        "Space-Track TLE query failed; using sample data",
+                        event="collector.spacetrack.query.failed",
+                        context={"status_code": tle_response.status_code},
+                    )
                     return self._get_sample_data()
 
                 data = tle_response.json()
-                print(f"SPACETRACK: Received {len(data)} records")
+                logger.info_event(
+                    "Space-Track TLE fetch completed",
+                    event="collector.spacetrack.fetch.completed",
+                    context={"record_count": len(data)},
+                )
                 return data
         except Exception as e:
-            print(f"SPACETRACK: Error - {e}, using sample data")
+            logger.warning_event(
+                "Space-Track TLE fetch failed; using sample data",
+                event="collector.spacetrack.fetch.failed",
+                context={"error": str(e)},
+            )
             return self._get_sample_data()
 
     def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:

backend/app/services/collectors/telegeography.py

@@ -163,32 +163,64 @@ class TeleGeographyLandingPointCollector(BaseCollector):
     data_type = "landing_point"
 
     async def fetch(self) -> List[Dict[str, Any]]:
-        """Fetch landing point data from GitHub mirror"""
-        url = self._resolved_url or ""
+        """Fetch landing point data, falling back when the old mirror disappears."""
+        config = get_data_sources_config()
+        sources = [
+            self._resolved_url or "",
+            str(config.get_yaml_value("telegeography.landing_point_url") or ""),
+            str(config.get_yaml_value("arcgis.landing_point_url") or ""),
+        ]
 
-        async with httpx.AsyncClient(timeout=60.0) as client:
-            response = await client.get(url)
-            response.raise_for_status()
-            return self.parse_response(response.json())
+        last_error: Exception | None = None
+        async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
+            for url in dict.fromkeys(source for source in sources if source):
+                try:
+                    params = (
+                        {"where": "1=1", "outFields": "*", "returnGeometry": "true", "f": "geojson"}
+                        if "FeatureServer" in url or url.endswith("/query")
+                        else None
+                    )
+                    response = await client.get(url, params=params)
+                    response.raise_for_status()
+                    records = self.parse_response(response.json())
+                    if records:
+                        return records
+                except Exception as exc:
+                    last_error = exc
+                    continue
 
-    def parse_response(self, data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+        if last_error:
+            raise last_error
+        return self._get_sample_data()
+
+    def parse_response(self, data: List[Dict[str, Any]] | Dict[str, Any]) -> List[Dict[str, Any]]:
         """Parse landing point data"""
         result = []
+        items = data.get("features", []) if isinstance(data, dict) else data
 
-        for item in data:
+        for item in items:
+            props = item.get("properties", {}) if isinstance(item, dict) else {}
+            geometry = item.get("geometry", {}) if isinstance(item, dict) else {}
+            source = props or item
+            coords = geometry.get("coordinates", []) if isinstance(geometry, dict) else []
+            longitude = coords[0] if len(coords) > 0 else source.get("longitude")
+            latitude = coords[1] if len(coords) > 1 else source.get("latitude")
+            source_id = source.get("id") or source.get("OBJECTID") or source.get("city_id") or ""
             try:
                 entry = {
-                    "source_id": f"telegeo_lp_{item.get('id', '')}",
-                    "name": item.get("name", "Unknown"),
-                    "country": item.get("country", "Unknown"),
-                    "city": item.get("city", item.get("name", "")),
-                    "latitude": str(item.get("latitude", "")),
-                    "longitude": str(item.get("longitude", "")),
+                    "source_id": f"telegeo_lp_{source_id}",
+                    "name": source.get("name", source.get("Name", "Unknown")),
+                    "country": source.get("country", "Unknown"),
+                    "city": source.get("city", source.get("Name", source.get("name", ""))),
+                    "latitude": str(latitude or ""),
+                    "longitude": str(longitude or ""),
                     "value": "",
                     "unit": "",
                     "metadata": {
-                        "cable_count": len(item.get("cables", [])),
-                        "url": item.get("url"),
+                        "cable_count": len(source.get("cables", [])),
+                        "url": source.get("url"),
+                        "objectid": source.get("OBJECTID"),
+                        "city_id": source.get("city_id"),
                     },
                     "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
                 }

backend/app/services/collectors/top500.py

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
 """
 
 import asyncio
+import math
 import re
+from urllib.parse import urljoin
 from typing import Dict, Any, List
 from bs4 import BeautifulSoup
 import httpx
 
 from app.core.data_sources import get_data_sources_config
-from app.services.collectors.base import BaseCollector
+from app.services.collectors.compute_base import ComputeCollector
 
 
-class TOP500Collector(BaseCollector):
+class TOP500Collector(ComputeCollector):
     name = "top500"
     priority = "P0"
     module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
         async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
             response = await client.get(url)
             response.raise_for_status()
-            entries = self.parse_response(response.text)
+            entries = self.parse_response(response.text, str(response.url))
 
             semaphore = asyncio.Semaphore(8)
 
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
         system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
         detail_url = ""
         if link and link.get("href"):
-            detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
+            detail_url = urljoin(str(top500_base_url), link.get("href"))
 
         manufacturer = ""
         if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
 
         return detail_map
 
-    def parse_response(self, html: str) -> List[Dict[str, Any]]:
+    def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
         """Parse TOP500 HTML response"""
         data = []
         soup = BeautifulSoup(html, "html.parser")
+        canonical = soup.find("link", rel="canonical")
+        date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
+        date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
+        reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
 
         # Find the table with TOP500 data
         table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
             table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
 
         if table:
+            headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
+            if not reference_date:
+                raise ValueError("TOP500 list date missing; previous snapshot retained")
             rows = table.select("tr")
             for row in rows[1:]:  # Skip header row
                 cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
                         cores = cells[2].get_text(strip=True).replace(",", "")
 
                         rmax_text = cells[3].get_text(strip=True)
-                        rmax = self._parse_performance(rmax_text)
+                        rmax = self._parse_performance(rmax_text, headers[3])
 
                         rpeak_text = cells[4].get_text(strip=True)
-                        rpeak = self._parse_performance(rpeak_text)
+                        rpeak = self._parse_performance(rpeak_text, headers[4])
 
                         power = ""
                         if len(cells) >= 6:
                             power = cells[5].get_text(strip=True).replace(",", "")
 
+                        identity = re.search(r"/system/(\d+)", detail_url)
+                        if not identity:
+                            raise ValueError("TOP500 stable system ID missing")
                         entry = {
-                            "source_id": f"top500_{rank}",
+                            "source_id": f"top500_system_{identity[1]}",
                             "name": system_name,
                             "country": country,
                             "city": city,
-                            "latitude": 0.0,
-                            "longitude": 0.0,
+                            "latitude": None,
+                            "longitude": None,
                             "value": str(rmax),
-                            "unit": "PFlop/s",
+                            "unit": "FLOP/s",
                             "metadata": {
+                                "compute_contract": "v1",
+                                "precision": "FP64", "measurement": "LINPACK Rmax",
+                                "source_url": detail_url, "list_url": date_url,
+                                "status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
                                 "rank": rank,
                                 "cores": cores,
                                 "rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
                                 "site": site,
                             },
                             "_detail_url": detail_url,
-                            "reference_date": "2025-11-01",
+                            "reference_date": reference_date,
                         }
                         data.append(entry)
                     except (ValueError, IndexError, AttributeError) as e:
-                        continue
-
-        # If scraping failed, return sample data for testing
-        if not data:
-            data = self._get_sample_data()
+                        raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
 
         return data
 
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
                 return 0.0
         return 0.0
 
-    def _parse_performance(self, text: str) -> float:
+    def _parse_performance(self, text: str, header: str = "") -> float:
         """Parse performance value from text (handles E, P, T suffixes)"""
-        text = text.strip().upper()
+        text = text.strip().upper().replace(",", "")
+        if not re.search(r"[EPTGMK]F", text):
+            unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
+            if not unit:
+                raise ValueError("TOP500 performance unit missing")
+            text = f"{text} {unit[1]}FLOP/S"
         multipliers = {
             "E": 1e18,
             "P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
 
         # Try simple float parsing
         try:
-            return float(text.replace(",", ""))
+            value = float(text.replace(",", ""))
+            if not math.isfinite(value) or value < 0:
+                raise ValueError("Invalid TOP500 performance")
+            return value
         except ValueError:
-            return 0.0
-
-    def _get_sample_data(self) -> List[Dict[str, Any]]:
-        """Return sample data for testing when scraping fails"""
-        return [
-            {
-                "source_id": "top500_1",
-                "name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
-                "country": "United States",
-                "city": "Livermore, CA",
-                "latitude": 37.6819,
-                "longitude": -121.7681,
-                "value": "1742.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 1,
-                    "cores": "11039616",
-                    "rmax": "1742.00",
-                    "rpeak": "2746.38",
-                    "power": "29581",
-                    "manufacturer": "HPE",
-                    "site": "DOE/NNSA/LLNL",
-                    "processor": "AMD 4th Gen EPYC 24C 1.8GHz",
-                    "interconnect": "Slingshot-11",
-                    "installation_year": "2025",
-                },
-                "reference_date": "2025-11-01",
-            },
-            {
-                "source_id": "top500_2",
-                "name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
-                "country": "United States",
-                "city": "Oak Ridge, TN",
-                "latitude": 36.0107,
-                "longitude": -84.2663,
-                "value": "1353.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 2,
-                    "cores": "9066176",
-                    "rmax": "1353.00",
-                    "rpeak": "2055.72",
-                    "power": "24607",
-                    "manufacturer": "HPE",
-                    "site": "DOE/SC/Oak Ridge National Laboratory",
-                },
-                "reference_date": "2025-11-01",
-            },
-            {
-                "source_id": "top500_3",
-                "name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
-                "country": "United States",
-                "city": "Argonne, IL",
-                "latitude": 41.3784,
-                "longitude": -87.8600,
-                "value": "1012.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 3,
-                    "cores": "9264128",
-                    "rmax": "1012.00",
-                    "rpeak": "1980.01",
-                    "power": "38698",
-                    "manufacturer": "Intel",
-                },
-                "reference_date": "2025-11-01",
-            },
-        ]
+            raise ValueError("Invalid TOP500 performance") from None

backend/app/services/collectors/vessel_ais.py

@@ -6,6 +6,7 @@ from typing import Any
 import httpx
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.enums import SnapshotStatus
 from app.core.time import to_iso8601_utc
 from app.core.websocket.broadcaster import broadcaster
 from app.services.barentswatch import (
@@ -119,6 +120,21 @@ class VesselAISCollector(BaseCollector):
             last_success_at=now if data else None,
             lag_seconds=max((now - latest_observed_at).total_seconds(), 0),
         )
+        if snapshot_id is not None:
+            from app.models.data_snapshot import DataSnapshot
+
+            snapshot = await db.get(DataSnapshot, snapshot_id)
+            if snapshot:
+                snapshot.record_count = records_added
+                snapshot.status = SnapshotStatus.SUCCESS.value
+                snapshot.completed_at = now
+                snapshot.summary = {
+                    "created": records_added,
+                    "updated": 0,
+                    "unchanged": 0,
+                    "deleted": 0,
+                    "storage": "ais_raw_observations",
+                }
         await db.commit()
         await self._broadcast_vessel_snapshot(data)
         await self.update_progress(records_added, force=True)

backend/app/services/compute_catalog.py

@@ -0,0 +1,200 @@
+"""Canonical country and mutually exclusive regional compute participants."""
+
+import re
+from collections.abc import Iterable
+
+from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
+from app.core.earth_geography_policy import display_country
+
+PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
+DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
+
+# Explicit membership prevents a regional total from double-counting a country.
+# APAC deliberately excludes China.
+COMPARISON_GROUPS = (
+    ("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
+    ("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
+    (
+        "EUROPE",
+        "欧洲",
+        "Europe",
+        (
+            "GB",
+            "DE",
+            "FR",
+            "CH",
+            "ES",
+            "FI",
+            "HU",
+            "IE",
+            "IT",
+            "LU",
+            "MD",
+            "NL",
+            "PL",
+            "RO",
+            "RU",
+            "UA",
+        ),
+        {"latitude": 54.0, "longitude": 20.0},
+    ),
+    (
+        "APAC",
+        "亚太（不含中国）",
+        "Asia Pacific (excluding China)",
+        ("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
+        {"latitude": 36.0, "longitude": 130.5},
+    ),
+    (
+        "OTHER",
+        "其他地区",
+        "Other",
+        (
+            "CA",
+            "AO",
+            "EG",
+            "KE",
+            "MZ",
+            "NG",
+            "UG",
+            "ZA",
+            "BR",
+            "CO",
+            "CR",
+            "DO",
+            "MX",
+            "AE",
+            "SY",
+            "TR",
+        ),
+        {"latitude": 12.0, "longitude": 10.0},
+    ),
+)
+
+
+def _country_code(canonical: str, aliases: list[str]) -> str | None:
+    code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
+    return "GB" if canonical == "英国" else code
+
+
+def build_country_catalog() -> list[dict]:
+    """Build the policy-projected canonical country drilldown catalog."""
+    countries = []
+    for canonical, aliases in COUNTRY_ENTRIES:
+        if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
+            continue
+        code = _country_code(canonical, aliases)
+        if code:
+            countries.append(
+                {
+                    "code": code,
+                    "name": canonical,
+                    "name_en": aliases[0],
+                    "kind": "country",
+                    "members": [code],
+                    "anchor": dict(COUNTRY_CENTROIDS[canonical]),
+                    "anchor_kind": "regional_label",
+                }
+            )
+    return countries
+
+
+COUNTRY_CATALOG = build_country_catalog()
+COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
+COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
+
+
+def _validate_group_members() -> None:
+    members = [
+        member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
+    ]
+    if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
+        raise ValueError("compute comparison groups must cover country catalog exactly once")
+
+
+_validate_group_members()
+GROUP_BY_MEMBER = {
+    member: code
+    for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
+    for member in members
+}
+
+
+def build_catalog() -> list[dict]:
+    """Return the five default participants before country drilldown entries."""
+    catalog: list[dict] = []
+    for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
+        entry = (
+            dict(COUNTRY_PARTICIPANTS[code])
+            if code in COUNTRY_PARTICIPANTS
+            else {
+                "code": code,
+                "name": name,
+                "name_en": name_en,
+                "kind": "region",
+                "members": list(members),
+                "anchor": dict(anchor),
+                "anchor_kind": "regional_label",
+            }
+        )
+        entry.update(
+            {
+                "members": list(members),
+                "parent_group": name,
+                "parent_group_code": code,
+                "order": index,
+                "default_selected": True,
+                "color": PALETTE[index % len(PALETTE)],
+            }
+        )
+        catalog.append(entry)
+
+    for entry in COUNTRY_CATALOG:
+        if entry["code"] in DEFAULT_PARTICIPANTS:
+            continue
+        country = dict(entry)
+        group_code = GROUP_BY_MEMBER[country["code"]]
+        group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
+        country.update(
+            {
+                "parent_group": group_name,
+                "parent_group_code": group_code,
+                "order": len(catalog),
+                "default_selected": False,
+                "color": PALETTE[len(catalog) % len(PALETTE)],
+            }
+        )
+        catalog.append(country)
+    return catalog
+
+
+CATALOG = build_catalog()
+PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
+
+
+def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
+    """Resolve country or region participant codes to canonical country codes."""
+    return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
+
+
+def source_country_names(codes: Iterable[str]) -> list[str]:
+    """Return unique canonical source labels, respecting China and Serbia ownership."""
+    selected_members = {
+        member for members in participant_members(codes).values() for member in members
+    }
+    selected_names = {COUNTRIES[code] for code in selected_members}
+    return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
+
+
+def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
+    """Map stored source-country labels to their requested participant code."""
+    owner_by_country = {
+        member: participant
+        for participant, members in participant_members(codes).items()
+        for member in members
+    }
+    code_by_name = {name: code for code, name in COUNTRIES.items()}
+    return {
+        name: owner_by_country[code_by_name[display_country(name)]]
+        for name in source_country_names(codes)
+    }

backend/app/services/compute_center_locations.py

@@ -3,7 +3,12 @@
 This module is a thin domain wrapper that wires up
 :mod:`app.services.location` for compute centers:
 
-    SourceCoordinates
+    Confirmed stored location -> trustworthy source coordinates -> exact
+    official facility registration -> source city/site coordinates.
+
+Epoch AI coordinates are admitted only at the grain asserted by its official
+``Location`` column. Country/region/empty locations remain unresolved even
+when the CSV happens to contain a latitude/longitude pair.
 
 The online Nominatim step is intentionally reserved for the user-triggered
 ``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -11,7 +16,7 @@ startup, so it must stay local and deterministic.
 
 For the full design and the reason behind the abstraction (compute centers,
 BGP collectors, BGP events, and future entities all share one pipeline),
-see ``docs/plans/location-resolver-shared-pipeline-plan.md``.
+see ``docs/technical/zh/location-pipeline-development.md``.
 
 The ``ComputeCenterLocation`` dataclass and the public function signatures are
 preserved verbatim so existing callers and tests do not need to change.
@@ -28,9 +33,12 @@ import httpx
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.compute_types import compute_record_filter
 from app.core.collected_data_fields import get_record_field
 from app.models.collected_data import CollectedData
 from app.models.compute_center_location import ComputeCenterLocationRecord
+from app.services.compute_center_verified_sites import get_verified_compute_site
+from app.services.epoch_location import assess_epoch_location
 
 from app.services.location import (
     LocationCandidate,
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
 
 async def refresh_compute_center_location_cache(
     session: AsyncSession,
+    *,
+    as_of: datetime | None = None,
 ) -> dict[str, dict[str, Any]]:
-    result = await session.execute(select(ComputeCenterLocationRecord))
+    query = select(ComputeCenterLocationRecord)
+    if as_of is not None:
+        query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
+    result = await session.execute(query)
     records = result.scalars().all()
     cache = {}
     for record in records:
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
         )
 
 
+class ComputeCenterSourceCoordinatesResolver:
+    """Use source coordinates only at the grain asserted by the source text."""
+
+    name = "source_coordinates"
+
+    def __init__(self) -> None:
+        self._default = SourceCoordinatesResolver()
+
+    def resolve(self, query: LocationQuery) -> ResolverOutput:
+        extra = query.extra or {}
+        if extra.get("source") != "epoch_ai_gpu":
+            return self._default.resolve(query)
+        if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
+            return ResolverOutput()
+        assessment = assess_epoch_location(
+            extra.get("source_location"),
+            country=query.country,
+            raw_country=extra.get("source_raw_country"),
+        )
+        if not assessment.is_renderable:
+            return ResolverOutput(
+                attempted_queries=(f"source_coordinates:{assessment.precision}",)
+            )
+        output = self._default.resolve(query)
+        if not output.candidates:
+            return output
+        candidate = output.candidates[0]
+        source_location = extra.get("source_location")
+        source_location_text = (
+            source_location.strip() if isinstance(source_location, str) else ""
+        )
+        source_note = assessment.reason
+        if source_location_text:
+            source_note = f"{source_note}; Epoch Location: {source_location_text}"
+        return ResolverOutput(
+            candidates=(
+                LocationCandidate(
+                    **{
+                        **candidate.__dict__,
+                        "precision": assessment.precision,
+                        "confidence": 1.0 if assessment.precision == "precise" else 0.9,
+                        "source_note": source_note,
+                        "needs_confirmation": assessment.precision != "precise",
+                        "matched_location_name": source_location_text
+                        or candidate.matched_location_name,
+                    }
+                ),
+            ),
+            attempted_queries=output.attempted_queries,
+        )
+
+
 class StoredComputeCenterLocationResolver:
     """Resolve a compute center through the DB-backed current-location cache."""
 
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
         longitude = parse_float(stored.get("longitude"))
         if latitude in (None, 0.0) or longitude in (None, 0.0):
             return ResolverOutput()
+        precision = stored.get("precision") or "city"
+        if (
+            extra.get("source") == "epoch_ai_gpu"
+            and stored.get("location_source") == "source_coordinates"
+        ):
+            assessment = assess_epoch_location(
+                extra.get("source_location"),
+                country=query.country,
+                raw_country=extra.get("source_raw_country"),
+            )
+            if not assessment.is_renderable:
+                return ResolverOutput()
+            precision = assessment.precision
         return ResolverOutput(
             candidates=(
                 LocationCandidate(
                     latitude=latitude,
                     longitude=longitude,
                     display_name=stored.get("name") or query.name or "Compute center",
-                    precision=stored.get("precision") or "city",
+                    precision=precision,
                     confidence=float(stored.get("confidence") or 0.85),
                     query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
                     source=self.name,
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
                     matched_location_name=stored.get("site") or stored.get("name") or query.name,
                     location_verified_at=stored.get("verified_at"),
                     suggested_registry_entry=None,
+                    raw_payload={
+                        "stored_location_source": stored.get("location_source"),
+                        "verification_status": stored.get("verification_status"),
+                    },
+                ),
+            )
+        )
+
+
+class VerifiedComputeSiteResolver:
+    """Resolve an exact source identity to an evidence-backed facility site."""
+
+    name = "official_facility_registry"
+
+    def resolve(self, query: LocationQuery) -> ResolverOutput:
+        extra = query.extra or {}
+        source = coerce_str(extra.get("source"))
+        source_id = coerce_str(extra.get("source_id"))
+        if not source or not source_id:
+            return ResolverOutput()
+        site = get_verified_compute_site(source, source_id)
+        if not site:
+            return ResolverOutput()
+        source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
+        source_note = coerce_str(site.get("source_note"))
+        if source_urls:
+            source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
+        return ResolverOutput(
+            candidates=(
+                LocationCandidate(
+                    latitude=float(site["latitude"]),
+                    longitude=float(site["longitude"]),
+                    display_name=coerce_str(site.get("matched_location_name"))
+                    or query.name
+                    or "Compute center",
+                    precision=coerce_str(site.get("precision")) or "site",
+                    confidence=1.0,
+                    query=f"official_facility_registry::{source}:{source_id}",
+                    source=self.name,
+                    source_note=source_note or None,
+                    matched_fields=("source", "source_id"),
+                    needs_confirmation=False,
+                    city=coerce_str(site.get("city")) or None,
+                    region=None,
+                    country=coerce_str(site.get("country")) or query.country,
+                    matched_location_name=coerce_str(site.get("matched_location_name"))
+                    or None,
+                    location_verified_at=coerce_str(site.get("verified_at")) or None,
+                    suggested_registry_entry=None,
+                    raw_payload={
+                        "address": site.get("address"),
+                        "coordinate_basis": site.get("coordinate_basis"),
+                        "sources": source_urls,
+                    },
                 ),
             )
         )
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
     return head or text
 
 
-def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
+def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
     name = coerce_str(getattr(record, "name", None))
     return {
         "source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
             or metadata.get("manufacturer")
         ),
         "organization": coerce_str(metadata.get("organization")),
+        "source_location": metadata.get("location"),
+        "source_raw_country": metadata.get("raw_country"),
     }
 
 
 def _context_to_query(
-    context: dict[str, str],
+    context: dict[str, Any],
     *,
     source_lat: float | None = None,
     source_lon: float | None = None,
@@ -444,6 +578,8 @@ def _context_to_query(
             "site": context.get("site") or "",
             "operator": context.get("operator") or "",
             "organization": context.get("organization") or "",
+            "source_location": context.get("source_location"),
+            "source_raw_country": context.get("source_raw_country"),
         },
     )
 
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
     return plan
 
 
+def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
+    stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
+    stored_confirmed = (
+        candidate.source == "stored_compute_center_location"
+        and not candidate.needs_confirmation
+        and stored_origin != "source_coordinates"
+    )
+    if stored_confirmed:
+        source_rank = 0
+    elif candidate.source == "source_coordinates" and candidate.precision == "precise":
+        source_rank = 1
+    elif candidate.source == "official_facility_registry":
+        source_rank = 2
+    elif candidate.source == "source_coordinates":
+        source_rank = 3
+    elif candidate.source == "stored_compute_center_location":
+        source_rank = 4
+    else:
+        source_rank = 5
+    precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
+    return (source_rank, precision_rank, -float(candidate.confidence or 0))
+
+
 COMPUTE_CENTER_PIPELINE = LocationPipeline(
     [
-        SourceCoordinatesResolver(),
+        ComputeCenterSourceCoordinatesResolver(),
         StoredComputeCenterLocationResolver(),
+        VerifiedComputeSiteResolver(),
     ],
+    sort_key=_compute_center_candidate_sort_key,
     failure_reason=(
         "Could not resolve to city-level coordinates from source coords"
         " or stored compute-center location."
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
 
 COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
     [
-        SourceCoordinatesResolver(),
+        ComputeCenterSourceCoordinatesResolver(),
+        VerifiedComputeSiteResolver(),
         ROROrganizationResolver(),
         NominatimResolver(
             query_plan_builder=_compute_center_query_plan,
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
             geocoder=lambda q: _geocode_online(q),
         ),
     ],
+    sort_key=_compute_center_candidate_sort_key,
     failure_reason=(
         "Could not resolve to city-level coordinates from source coords"
         ", ROR organization lookup, or online geocoding."
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
 _GEOGRAPHY_MODE_BY_SOURCE = {
     "source_coordinates": "source_coordinates",
     "stored_compute_center_location": "stored_compute_center_location",
+    "official_facility_registry": "official_facility_registry",
     "ror_organization_registry": "ror_organization",
     "nominatim_online_geocode": "online_geocode",
 }
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
 def _candidate_to_location(
     candidate: LocationCandidate,
     *,
-    context: dict[str, str],
+    context: dict[str, Any],
 ) -> ComputeCenterLocation:
     geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
     is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +758,7 @@ def _candidate_to_location(
 
 def _diagnostic_for(
     record: Any,
-    context: dict[str, str],
+    context: dict[str, Any],
     *,
     failure_reason: str,
     attempted_queries: tuple[str, ...] = (),
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
     organization: str | None = None,
 ) -> LocationQuery:
     name_value = coerce_str(name)
-    context: dict[str, str] = {
+    context: dict[str, Any] = {
         "source": coerce_str(source),
         "source_id": coerce_str(source_id),
         "name": name_value,
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
     """Seed stored compute-center locations only from real source coordinates."""
     stmt = (
         select(CollectedData)
-        .where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
+        .where(compute_record_filter(CollectedData))
         .where(CollectedData.is_current.is_(True))
     )
     result = await session.execute(stmt)
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
         longitude = parse_float(get_record_field(record, "longitude"))
         if latitude in (None, 0.0) or longitude in (None, 0.0):
             continue
+        metadata = record.extra_data or {}
+        precision = "precise"
+        confidence = 1.0
+        needs_confirmation = False
+        source_note = "Seeded from source-provided compute-center coordinates"
+        if source_value == "epoch_ai_gpu":
+            assessment = assess_epoch_location(
+                metadata.get("location"),
+                country=get_record_field(record, "country"),
+                raw_country=metadata.get("raw_country"),
+            )
+            if not assessment.is_renderable:
+                continue
+            precision = assessment.precision
+            confidence = 1.0 if precision == "precise" else 0.9
+            needs_confirmation = precision != "precise"
+            source_note = assessment.reason
         existing = await session.scalar(
             select(ComputeCenterLocationRecord)
             .where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
         )
         if existing:
             continue
-        metadata = record.extra_data or {}
         session.add(
             ComputeCenterLocationRecord(
                 source=source_value,
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
                 country=coerce_str(get_record_field(record, "country")) or None,
                 latitude=latitude,
                 longitude=longitude,
-                precision="precise",
-                confidence=1.0,
+                precision=precision,
+                confidence=confidence,
                 location_source="source_coordinates",
-                source_note="Seeded from source-provided compute-center coordinates",
+                source_note=source_note,
                 raw_payload={
                     "record_id": getattr(record, "id", None),
                     "source": source_value,
                     "source_id": source_id,
                 },
-                needs_confirmation=False,
+                needs_confirmation=needs_confirmation,
                 verification_status="source_provided",
                 verified_at=None,
             )

backend/app/services/compute_center_verified_sites.json

@@ -0,0 +1,44 @@
+{
+  "version": 1,
+  "sites": [
+    {
+      "source": "epoch_ai_gpu",
+      "source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
+      "latitude": 35.380924,
+      "longitude": 139.625008,
+      "precision": "site",
+      "location_source": "official_facility_registry",
+      "country": "日本",
+      "city": "Yokohama",
+      "matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
+      "address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
+      "source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
+      "verified_at": "2026-09-21T17:06:31Z",
+      "coordinate_basis": "official_access_map_center",
+      "sources": [
+        "https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
+        "https://www.jamstec.go.jp/e/about/access/yokohama.html"
+      ]
+    },
+    {
+      "source": "epoch_ai_gpu",
+      "source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
+      "latitude": 41.718109383287235,
+      "longitude": -87.984498648496,
+      "precision": "site",
+      "location_source": "official_facility_registry",
+      "country": "美国",
+      "city": "Argonne",
+      "matched_location_name": "Argonne Leadership Computing Facility",
+      "address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
+      "source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
+      "verified_at": "2026-09-21T17:06:31Z",
+      "coordinate_basis": "official_access_map_center",
+      "sources": [
+        "https://www.alcf.anl.gov/systems/sophia",
+        "https://docs.alcf.anl.gov/sophia/",
+        "https://www.alcf.anl.gov/about/visiting-alcf"
+      ]
+    }
+  ]
+}

backend/app/services/compute_center_verified_sites.py

@@ -0,0 +1,26 @@
+"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
+
+import json
+from copy import deepcopy
+from functools import lru_cache
+from pathlib import Path
+
+
+@lru_cache(maxsize=1)
+def _site_index() -> dict[tuple[str, str], dict]:
+    payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
+    index = {}
+    for site in payload["sites"]:
+        if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
+            raise ValueError("Invalid verified compute facility coordinate")
+        for source_id in site["source_ids"]:
+            key = (site["source"], source_id)
+            if key in index:
+                raise ValueError("Duplicate verified compute facility identity")
+            index[key] = site
+    return index
+
+
+def get_verified_compute_site(source: str, source_id: str) -> dict | None:
+    site = _site_index().get((source, source_id))
+    return deepcopy(site) if site else None

backend/app/services/compute_configuration.py

@@ -0,0 +1,99 @@
+"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
+
+import hashlib
+from pathlib import Path
+from uuid import uuid4
+
+from app.core.request_context import get_request_id
+from app.models.system_log import AuditLog
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.schemas.compute_configuration import ComputeConfiguration
+from app.services.compute_catalog import COUNTRY_CATALOG
+from fastapi import HTTPException
+from sqlalchemy import select
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+CATEGORY = "compute_comparison"
+DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
+DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
+DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
+
+
+def default_envelope() -> dict:
+    return {
+        "config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
+        "revision": DEFAULT_REVISION,
+        "is_default": True,
+    }
+
+
+async def read_configuration(db: AsyncSession) -> dict:
+    record = (
+        await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
+    ).scalar_one_or_none()
+    return dict(record.payload) if record else default_envelope()
+
+
+def configuration_response(envelope: dict) -> dict:
+    return {
+        **envelope,
+        "countries": [
+            {key: country[key] for key in ("code", "name", "name_en")}
+            for country in COUNTRY_CATALOG
+        ],
+    }
+
+
+async def publish_configuration(
+    db: AsyncSession,
+    *,
+    revision: str,
+    config: ComputeConfiguration,
+    actor: User,
+    reset: bool = False,
+) -> dict:
+    # The unique category plus row lock also serializes concurrent first writes.
+    await db.execute(
+        insert(SystemSetting)
+        .values(category=CATEGORY, payload=default_envelope())
+        .on_conflict_do_nothing(index_elements=[SystemSetting.category])
+    )
+    record = (
+        await db.execute(
+            select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
+        )
+    ).scalar_one()
+    if record.payload["revision"] != revision:
+        await db.rollback()
+        raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
+    envelope = (
+        default_envelope()
+        if reset
+        else {
+            "config": config.model_dump(mode="json"),
+            "revision": uuid4().hex,
+            "is_default": reset,
+        }
+    )
+    before = record.payload
+    if reset:
+        await db.delete(record)
+    else:
+        record.payload = envelope
+    # Audit and configuration commit together: a published edit must be attributable.
+    db.add(
+        AuditLog(
+            actor_id=actor.id,
+            actor_name=actor.username,
+            action="compute_comparison.reset" if reset else "compute_comparison.update",
+            target_type="system_setting",
+            target_id=CATEGORY,
+            result="success",
+            request_id=get_request_id(),
+            details={"before": before, "after": envelope},
+        )
+    )
+    await db.commit()
+    return envelope

backend/app/services/compute_history.py

@@ -0,0 +1,122 @@
+"""Versioned, source-backed annual observations for compute comparisons."""
+
+from copy import deepcopy
+from datetime import date
+
+from app.schemas.compute_configuration import ComputeConfiguration
+from app.services.compute_catalog import (
+    CATALOG,
+    DEFAULT_PARTICIPANTS,
+    PARTICIPANTS,
+    participant_members,
+)
+from app.services.compute_configuration import DEFAULT_CONFIGURATION
+
+
+def _participant_value(
+    members: tuple[str, ...], observations: dict[str, float | int], mode: str
+) -> tuple[float | int | None, dict]:
+    covered = [member for member in members if member in observations]
+    missing = [member for member in members if member not in observations]
+    coverage = {
+        "status": "complete" if not missing else "incomplete",
+        "required_members": list(members),
+        "covered_members": covered,
+        "missing_members": missing,
+    }
+    if missing:
+        return None, coverage
+    if mode == "bilateral_share":
+        if (
+            not set(members).issubset({"CN", "US"})
+            or not {"CN", "US"}.issubset(observations)
+            or observations["US"] + observations["CN"] <= 0
+        ):
+            coverage.update(
+                {
+                    "status": "incompatible",
+                    "reason": "bilateral_share_requires_CN_and_US_observations",
+                }
+            )
+            return None, coverage
+        return (
+            round(
+                sum(observations[member] for member in members)
+                / (observations["US"] + observations["CN"])
+                * 100,
+                2,
+            ),
+            coverage,
+        )
+    return sum(observations[member] for member in members), coverage
+
+
+def compute_history(
+    countries: list[str],
+    mode: str = "global_share",
+    config: ComputeConfiguration | None = None,
+) -> dict:
+    """Aggregate only fully covered, same-year compatible observations."""
+    config = config if config is not None else DEFAULT_CONFIGURATION
+    metric = "global_share" if mode == "bilateral_share" else mode
+    annual = {row.year: row for row in config.observations if row.metric == metric}
+    gaps = {row.year: row for row in config.innovation}
+    observed_years = [row.year for row in config.observations] + list(gaps)
+    years = tuple(
+        range(
+            min(observed_years, default=date.today().year),
+            max([date.today().year, *observed_years]) + 1,
+        )
+    )
+    members_by_participant = participant_members(countries)
+    rows = []
+    for year in years:
+        record = annual.get(year)
+        observations = record.values if record else {}
+        values, coverage = {}, {}
+        for code, members in members_by_participant.items():
+            values[code], coverage[code] = _participant_value(members, observations, mode)
+        evidence_id = record.source_id if record else None
+        rows.append(
+            {"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
+        )
+
+    selected_members = {member for members in members_by_participant.values() for member in members}
+    has_innovation_scope = {"CN", "US"}.issubset(selected_members)
+    return {
+        "version": f"compute-history-{config.reviewed_at}",
+        "provenance": "curated_reports",
+        "reviewed_at": config.reviewed_at.isoformat(),
+        "time_basis": "retrospective",
+        "years": years,
+        "mode": mode,
+        "unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
+        "denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
+            mode
+        ],
+        "catalog": deepcopy(CATALOG),
+        "default_participants": list(DEFAULT_PARTICIPANTS),
+        "countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
+        "compute": rows,
+        "innovation": [
+            {
+                "year": year,
+                "value": gaps[year].value if year in gaps and has_innovation_scope else None,
+                "evidence_id": gaps[year].source_id
+                if year in gaps and has_innovation_scope
+                else None,
+            }
+            for year in years
+        ],
+        "evidence": {
+            source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
+        },
+        "innovation_scope": ["US", "CN"],
+        "limitations": [
+            "公开设施样本不代表全国总算力；FP64、FP16 与 FP32 不可直接相加。",
+            "区域值只在全部成员有同年、同指标、可比较观测时汇总；缺失成员时返回 null。",
+            "全球份额不从中美或其他局部成员反推剩余地区。",
+            "年度资料按已配置报告引用展示，观测年与报告年份分列；跨年方法可比性仍需核验。",
+            "创新分差为美国减中国；只有分差，不反推单国得分，不证明算力与创新的因果关系。",
+        ],
+    }

backend/app/services/compute_history_defaults.json

@@ -0,0 +1,125 @@
+{
+  "reviewed_at": "2026-09-22",
+  "sources": [
+    {
+      "id": "caict-2021",
+      "title": "中国算力发展指数白皮书（2021）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2021,
+      "published_at": null,
+      "locator": "正文 11 页图 4（2020 年全球份额）"
+    },
+    {
+      "id": "caict-2022",
+      "title": "中国算力发展指数白皮书（2022）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2022,
+      "published_at": null,
+      "locator": "正文 13 页图 5（2021 年全球份额）"
+    },
+    {
+      "id": "caict-2023",
+      "title": "中国算力发展指数白皮书（2023）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2023,
+      "published_at": null,
+      "locator": "正文 13 页图 5（2022 年全球份额）；16 页（中国总规模）"
+    },
+    {
+      "id": "caict-2024",
+      "title": "先进计算暨算力发展指数蓝皮书（2024年）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2024,
+      "published_at": null,
+      "locator": "正文 10 页图 3（2023 年全球份额）；12 页（中国总规模）"
+    },
+    {
+      "id": "innovation-2026",
+      "title": "全球人工智能创新指数报告 2026：科技日报报道",
+      "url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
+      "publisher": "科技日报（新浪转载）",
+      "report_year": 2026,
+      "published_at": "2026-07-17",
+      "locator": "2023–2025 年美国减中国的指数分差"
+    }
+  ],
+  "observations": [
+    {
+      "year": 2020,
+      "metric": "global_share",
+      "source_id": "caict-2021",
+      "values": {
+        "US": 36,
+        "CN": 31,
+        "JP": 6
+      }
+    },
+    {
+      "year": 2021,
+      "metric": "global_share",
+      "source_id": "caict-2022",
+      "values": {
+        "US": 34,
+        "CN": 33,
+        "JP": 5
+      }
+    },
+    {
+      "year": 2022,
+      "metric": "global_share",
+      "source_id": "caict-2023",
+      "values": {
+        "US": 34,
+        "CN": 33,
+        "JP": 4
+      }
+    },
+    {
+      "year": 2023,
+      "metric": "global_share",
+      "source_id": "caict-2024",
+      "values": {
+        "US": 41,
+        "CN": 31,
+        "JP": 4
+      }
+    },
+    {
+      "year": 2022,
+      "metric": "absolute",
+      "source_id": "caict-2023",
+      "values": {
+        "CN": 302
+      }
+    },
+    {
+      "year": 2023,
+      "metric": "absolute",
+      "source_id": "caict-2024",
+      "values": {
+        "CN": 435
+      }
+    }
+  ],
+  "innovation": [
+    {
+      "year": 2023,
+      "value": 22.02,
+      "source_id": "innovation-2026"
+    },
+    {
+      "year": 2024,
+      "value": 19.96,
+      "source_id": "innovation-2026"
+    },
+    {
+      "year": 2025,
+      "value": 17.95,
+      "source_id": "innovation-2026"
+    }
+  ]
+}