release: bump version to 0.76.19

.codex/skills/cleanup/SKILL.md

@@ -19,9 +19,9 @@ Do not refactor architecture, add features, or change behavior.
 
 If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
 
-Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
+Only report and fix issues present in **newly added or modified** lines of this diff — do not audit unchanged code, and do not rewrite unrelated files during cleanup.
 
-## Token-Saving Rule
+## Token-Saving Rule & Harness Checks
 
 Prefer deterministic CLI checks before reading files into model context:
 
@@ -29,27 +29,36 @@ Prefer deterministic CLI checks before reading files into model context:
 git diff --name-only HEAD
 git diff --unified=0 HEAD -- <path>
 git diff --check
-rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
 ```
 
+Use the repository's focused harness checks for the touched surfaces:
+
+- Backend Python changes: `scripts/harness/backend-rules-check.sh` (catches direct `print()`, `breakpoint()`, and debugger calls)
+- Frontend changes: `scripts/harness/frontend-rules-check.sh` (catches console logs, missing button types, Card nesting, layout height-chain issues)
+- Secrets / credentials: `scripts/harness/security-check.sh`
+
 Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
 
 ## Checklist
 
 ### 1. Duplicate Logic
-- Identical or near-identical code blocks appearing in multiple places
-- A function/helper that already exists but is re-implemented elsewhere instead of being reused
-- Repeated DOM queries, regex literals, or template strings within the same file
+- Identical or near-identical code blocks appearing in multiple places within the modified scope
+- A helper that already exists but is re-implemented elsewhere instead of being reused
+- Repeated complex queries or template strings that cause actual maintenance hazard
+- **Anti-Over-Defensiveness**: Keep one-off logic at its call site. Do not prematurely abstract single-use or 2-use trivial helpers/templates that are clearer inline.
 
 ### 2. Magic Numbers / Magic Strings
-- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
-- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
-- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
+- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) where the intent is completely opaque and naming adds clear value
+- Hardcoded domain strings (IDs, status values, protocol fragments) scattered through logic
+- **Anti-Over-Defensiveness (AGENTS.md Quality Bar)**:
+  - Idiomatically clear values such as `0`, `1`, `-1`, `100`, `""` are fine inline.
+  - **Do NOT over-extract local single-use layout offsets, Three.js coordinates, padding, or math coefficients into module-level globals.**
+  - Keep one-off logic at its call site unless extraction clarifies a complex operation or lets callers share the same behavior.
 
 ### 3. Naming Issues
 - Cryptic abbreviations (`or_`, `tmp2`, `x2`)
 - Names that do not match actual behavior
-- The same concept referred to by different names in different places
+- The same concept referred to by different names in different places within the changed scope
 
 ### 4. Dead Code
 - Commented-out code blocks (3+ lines)
@@ -58,12 +67,12 @@ Read full files only when the focused diff does not provide enough surrounding c
 
 ### 5. Style Inconsistencies
 - Trailing whitespace
-- Mixed quote styles or indentation within the same file
+- Mixed quote styles or indentation within the same file (Python: 4 spaces Black style; TypeScript: 2 spaces Prettier style)
 - Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
 
 ### 6. Other
-- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
 - Overly verbose conditions that can be simplified without changing logic
+- Keep fixes strictly scoped to the modified files; do not rewrite external callers or expand cleanup beyond the changed scope.
 
 ## Steps
 
@@ -107,30 +116,39 @@ If no issues are found, output "No code smells detected. Code quality looks good
 
 Use the Edit tool for **minimal, targeted changes**:
 
-- **Duplicate logic**: extract to a shared constant or function; update all call sites
-- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
-- **Naming**: rename the variable/function; update all references
+- **Duplicate logic**: extract to a shared constant or helper within the owned file or component scope; update call sites strictly within scope
+- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope only when the value is reused across places or its meaning is genuinely opaque; replace usages. Keep one-off UI/math offsets inline per `AGENTS.md`
+- **Naming**: rename the variable/function; update all references within the modified scope
 - **Dead code**: delete it
 - **Trailing whitespace / style**: fix in place
-- **Unexported helper**: add `export`; update callers to import instead of re-implementing
+- **Unexported helper**: add `export` only when requested; do not modify unrelated callers
 
 Principles:
-- Only fix issues identified in the checklist — no extra improvements
+- Only fix issues identified in the checklist — no extra improvements or speculative refactorings
 - Keep each Edit as small as possible
-- After fixing, verify the old bad pattern is gone with grep
-- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
+- After fixing, verify the old bad pattern is gone with grep or deterministic harness checks
+- Prefer `apply_patch` or targeted edit tools; use formatters only when the repository already uses them for the touched file type
 
-### Step 5 — Summary
+### Step 5 — Verify and summarize
+
+Run the smallest relevant harness checks for the modified surfaces:
+- Python files: `uv run python -m py_compile <changed_py_files>` and `scripts/harness/backend-rules-check.sh`
+- Frontend files: `scripts/harness/frontend-rules-check.sh` (and `bun run --cwd frontend build` if imports/types were adjusted)
+- Formatting / whitespace: `git diff --check`
 
 ```
 Cleanup complete:
 
 Fixed N issues:
   ✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
-  ✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
-  ✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
+  ✓ main.js — clarified variable naming in news event payload handler
+  ✓ controls.js — removed unused dead import
   ...
 
+Validation:
+  ✓ git diff --check passed
+  ✓ scripts/harness/frontend-rules-check.sh passed
+
 Skipped (needs manual review):
   ! foo.js L45 — large commented-out block; confirm it is safe to delete
 ```
@@ -141,4 +159,6 @@ Skipped (needs manual review):
 - **Do not** add new features, abstractions, or parameters
 - **Do not** rewrite comments (only delete commented-out dead code)
 - **Do not** touch test file logic
+- **Do not** rewrite unrelated files during cleanup (per `AGENTS.md` Prohibited)
+- **Do not** over-extract local single-use coordinates, layout numbers, or inline math coefficients into global constants (preventing defensive over-engineering)
 - If a magic number's intent is uncertain, skip it and flag it in the summary

.codex/skills/docs/SKILL.md

@@ -48,18 +48,22 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
 
 3. Write the doc:
 
-- Explain background/problem, design decisions, constraints, and operational impact.
+- Include the context, rationale, or operational impact the reader needs for this change.
 - Keep code snippets short and directly relevant.
 - List related files only when they help future maintainers navigate.
 - Use the repository’s existing language, heading style, and naming conventions.
-- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
+- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. Only update the glossary when introducing new, ambiguous, or cross-language public concepts; avoid unnecessary glossary churn for ordinary bug fixes or internal refactors.
 
 4. Verify:
 
 - Read the completed doc once for clarity and stale statements.
 - Verify important referenced paths exist with `test -e` or `rg --files`.
-- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
-- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
+- Run the repository's centralized documentation harness gate:
+  ```bash
+  scripts/harness/docs-consistency-check.sh
+  ```
+  This automatically verifies bilingual pairs, public registry whitelist, link anchors, language-scoped links, section deep-links, and stale admin route terms in a single pass.
+- Run any additional domain-specific checks from `docs/documentation-coverage-rules.md` when applicable.
 
 ## Hard Constraints
 

.codex/skills/release/SKILL.md

@@ -19,9 +19,11 @@ Do not use this skill for ordinary commits that are not being released.
 ## Versioning Rules
 
 - `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
-- `bugfix` -> bump `+0.0.1`
+- `improvement` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
+- `bugfix` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
 - `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
 
+Mixed bugfix and small feature/UI work defaults to `improvement` unless the user explicitly chooses another release type.
 When intent is mixed, prefer the user's stated release intent.
 
 ## Required Files
@@ -53,28 +55,32 @@ Do not inspect full diffs unless deciding whether changed code belongs in the re
 ### Step 1 — Environment check
 
 ```bash
-git branch --show-current   # must be on dev
+git branch --show-current   # check current branch
 git status --short           # check for unrelated uncommitted changes
 cat VERSION                  # read current version
 ```
 
-If not on `dev`, stop and tell the user. Do not proceed.
+Default release branch is `dev`. If not on `dev` (e.g. on a dedicated maintenance or release branch), report the active branch and confirm before proceeding, rather than hard-failing.
 
 If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
 
 ### Step 2 — Determine release type and next version
 
-- If the user provided an explicit type (`feature` / `bugfix`), use it
-- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
+- If the user provided an explicit type (`feature` / `improvement` / `bugfix`), use it
+- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`:
+  - New features or major capabilities -> `feature`
+  - Mixed bugfix and small feature/UI work -> default to `improvement` (per `rules.md`)
+  - Bugfixes only -> `bugfix`
 - Compute the next version:
   - `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
-  - `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
+  - `improvement`: increment patch only (e.g. `0.76.18` → `0.76.19`)
+  - `bugfix`: increment patch only (e.g. `0.76.18` → `0.76.19`)
 - **Show the release plan before making any changes:**
 
 ```
 Release plan:
-  Type:    bugfix
-  Version: 0.26.2 → 0.26.3
+  Type:    improvement
+  Version: 0.76.18 → 0.76.19
   Branch:  dev
   Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
 ```
@@ -117,17 +123,25 @@ Get today's date with `date +%Y-%m-%d`.
 
 ### Step 6 — Validate
 
-Run the smallest relevant validation for the changes in scope:
+Run the smallest relevant harness validation for the changes in scope:
 
-- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
-- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
-- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
+- **Version consistency**: confirm `VERSION`, `frontend/package.json`, `pyproject.toml`, and `uv.lock` all show the identical target version:
 
 ```bash
 cat VERSION
 rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
 ```
 
+- **Security scan**: run `scripts/harness/security-check.sh`
+- **Python files changed**:
+  - Compile: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `uv run python -m py_compile <changed_files>`
+  - Service rules: run `scripts/harness/backend-rules-check.sh`
+- **Frontend files changed**:
+  - Rules: run `scripts/harness/frontend-rules-check.sh`
+  - Build: run `bun run --cwd frontend build` (Bun only; never skip)
+- **Docs changed**: run `scripts/harness/docs-consistency-check.sh`
+- **Broad / cross-subsystem changes**: run `scripts/harness/quick-check.sh`
+
 ### Step 7 — Pre-commit preview
 
 Show what will be committed:
@@ -141,29 +155,36 @@ Confirm all required files are present and no unexpected files (debug files, `.e
 ### Step 8 — Commit, tag, and push
 
 ```bash
+branch=$(git branch --show-current)
 git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
 # also stage any code changes included in this release
 git add <code_files>
 
 git commit -m "release: bump version to x.x.x"
 git tag vx.x.x
-git push origin dev
-git push origin vx.x.x
 ```
 
 Commit message format is fixed: `release: bump version to x.x.x`
 
+Push to remote:
+- When remote publication is requested:
+  ```bash
+  git push origin "$branch"
+  git push origin vx.x.x
+  ```
+- If remote publication was not requested or remote access is restricted, report the commit and tag as complete and output the push commands for manual execution.
+
 ### Step 9 — Completion summary
 
 ```
-✓ Version bumped: 0.26.2 → 0.26.3
+✓ Version bumped: 0.76.18 → 0.76.19
 ✓ CHANGELOG updated
 ✓ version-history updated
 ✓ uv.lock regenerated
 ✓ Validation passed
-✓ commit: release: bump version to 0.26.3
-✓ tag: v0.26.3
-✓ Pushed to origin/dev
+✓ commit: release: bump version to 0.76.19
+✓ tag: v0.76.19
+✓ Pushed to origin/dev (or reported pending manual push)
 ```
 
 ## Notes

.dockerignore

@@ -3,8 +3,23 @@
 !pyproject.toml
 !uv.lock
 !VERSION
+!frontend/
+!frontend/public/
+!frontend/public/earth/
+!frontend/public/earth/assets/
+!frontend/public/earth/assets/cards/
+!frontend/public/earth/assets/cards/manifest.json
 !backend/
 !backend/**
+!config/
+!config/earth-boundary-pov-policy.china-v1.json
+!scripts/
+!scripts/build_earth_boundary_china_pov_geojson.py
+!scripts/build_earth_boundary_pmtiles.py
+!scripts/earth_boundary_policy.py
+!resources/
+!resources/earth/
+!resources/earth/boundary-pov-geometry.china-v1.json
 !aiprovider/
 !aiprovider/**
 

.gitignore

@@ -160,3 +160,8 @@ data/earth-boundary-sources/
 # Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
 # committing thousands of generated tile files.
 frontend/public/earth/data/boundaries/
+
+# Offline comparison originals; runtime 8K maps and provenance remain versioned.
+resources/earth/candidates/nasa-blue-marble/*.jpg
+resources/earth/candidates/natural-earth-3/*.jpg
+*:Zone.Identifier

AGENTS.md

@@ -55,13 +55,19 @@ Preserve user changes already present in the worktree.
 
 ### Validation
 
-Fast local harness validation:
+For an isolated change, run the smallest check that exercises the affected
+behavior, along with any gates required for that area. For example, run the
+docs consistency check for a link change, or reproduce a bug at the entry
+point where it occurred.
+
+Use the quick suite when a change affects shared harness wiring or needs
+regression checks across areas:
 
 ```bash
 scripts/harness/quick-check.sh
 ```
 
-Full local validation:
+Use full validation for larger cross-subsystem changes:
 
 ```bash
 scripts/harness/validate.sh
@@ -113,6 +119,9 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
 ## Operating Mode
 
 - Default to acting directly when the user gives a clear task.
+- For authorized multi-agent work, follow the Coding Agent Model Strategy in
+  `rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
+  and Astra for complex reasoning; the main agent owns final verification.
 - Ask before acting only when the missing decision is risky, cannot be
   discovered from repository context, and no conservative assumption is safe.
 - Read relevant files before editing.
@@ -120,6 +129,8 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
   focused file reads, tests, builds, linters, and harness scripts.
 - Keep changes scoped to the requested area. Do not mix cleanup, feature work,
   release work, and documentation unless the task requires it.
+- When the user corrects a detail, apply it to the original request. Carry the
+  correction into the result without turning discarded ideas into new requirements.
 
 ---
 
@@ -141,6 +152,12 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
   visual-dependent part of the work.
 - Claims of completion need evidence: a relevant test, build, lint, screenshot,
   diff, direct file check, or harness result.
+- Verify the outcome through the entry point the user relies on. A source check
+  can confirm a code convention. A UI test with a mocked API can confirm what
+  the user sees for that response.
+- When a test fails, identify the behavior or invariant it protects. Fix the
+  implementation if that regressed. If the behavior is correct and the assertion
+  pins an obsolete detail, update the test to exercise the behavior.
 - For UI and rendering changes, verify the rendered result when local tooling
   allows it.
 
@@ -159,7 +176,14 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
 
 ## Quality Bar
 
-- Prefer existing project patterns over new abstractions.
+- Prefer existing project patterns and the smallest change that satisfies the
+  request. Keep one-off logic at its call site unless extraction clarifies a
+  complex operation or lets callers share the same behavior.
+- Validate and normalize external data at the boundary that owns it. Recheck a
+  guarantee when data crosses a new trust boundary or an operation can invalidate it.
+- Before adding a retry, lock, or fallback, identify the failure it addresses
+  from a requirement, an existing contract, or observed behavior. Catch errors
+  where the caller can recover, translate them, or add context and propagate them.
 - Remove stale branches, mocks, compatibility paths, and duplicated helpers once
   a stable path exists.
 - Centralize prompts, constants, defaults, and shared request/response handling.

CODEMAP.md

@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
 | `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
 | `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
 | `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
+| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
 
 ## Runtime Entry Points
 

README.md

@@ -324,7 +324,7 @@ ipconfig
 
 ## 启动容错参数
 
-`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试，并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
+`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康，再验证后端实际连接；`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待，避免重复消耗健康检查预算。
 
 可通过环境变量临时调整:
 

VERSION

@@ -1 +1 @@
-0.74.3
+0.76.19

aiprovider/Dockerfile

@@ -1,4 +1,4 @@
-# syntax=docker/dockerfile:1.7
+# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
 
 ARG PYTHON_IMAGE=python:3.14-slim
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
 FROM ${UV_IMAGE} AS uv
 FROM ${PYTHON_IMAGE}
 
-ARG AI_PROVIDER_BUILD_FINGERPRINT
-LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
-
 COPY --from=uv /uv /uvx /bin/
 
 WORKDIR /app
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 COPY pyproject.toml uv.lock /app/
 RUN --mount=type=cache,target=/root/.cache/uv \
     --mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
-    uv sync --frozen --no-dev
+    uv sync --frozen --only-group aiprovider
 
 COPY aiprovider /app/aiprovider
 
+ARG AI_PROVIDER_BUILD_FINGERPRINT
+LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
+
 EXPOSE 8010
 
 HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
     CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
 
-CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
+CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]

aiprovider/README.md

@@ -4,7 +4,7 @@
 
 完整使用说明见:
 
-- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
+- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
 
 当前支持:
 
@@ -15,6 +15,7 @@
   - `AI_PROVIDER=ollama`
 - request adapter:
   - `AI_PROVIDER_API=openai-completions`
+  - `AI_PROVIDER_API=openai-responses`
   - `AI_PROVIDER_API=anthropic-messages`
   - `AI_PROVIDER_API=ollama-generate`
 

aiprovider/provider_service.py

@@ -3,6 +3,7 @@ from __future__ import annotations
 import asyncio
 import json
 from typing import Any
+from uuid import NAMESPACE_URL, uuid4, uuid5
 
 import httpx
 from fastapi import HTTPException, status
@@ -65,6 +66,7 @@ class ProviderService:
         self.model_provider_apis = self._parse_model_provider_apis(
             overrides.get("model_provider_apis")
         )
+        self.session_id = str(uuid4())
 
     def get_status(self) -> AIProviderStatusResponse:
         enabled = self.provider != "disabled"
@@ -95,6 +97,8 @@ class ProviderService:
             )
 
         prompt = self._build_prompt(payload)
+        if payload.context.get("session_id") is not None:
+            self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
 
         provider_api = self._resolve_model_provider_api(model)
 
@@ -102,6 +106,10 @@ class ProviderService:
             data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
             content = self._extract_openai_content(data)
             content_blocks = self._extract_openai_blocks(data)
+        elif provider_api == "openai-responses":
+            data = await self._request_openai_responses(model, prompt, payload.system_prompt)
+            content_blocks = self._extract_responses_blocks(data)
+            content = "".join(block.text for block in content_blocks if block.text)
         elif provider_api == "anthropic-messages":
             data = await self._request_anthropic_messages(
                 model,
@@ -200,6 +208,39 @@ class ProviderService:
             request_body=request_body,
         )
 
+    async def _request_openai_responses(
+        self, model: str, prompt: str, system_prompt: str | None = None,
+    ) -> dict[str, Any]:
+        request_body: dict[str, Any] = {
+            "model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
+        }
+        resolved_system_prompt = self._resolve_system_prompt(system_prompt)
+        if resolved_system_prompt:
+            request_body["instructions"] = resolved_system_prompt
+        return await self._post(
+            path="/responses",
+            headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
+            request_body=request_body,
+        )
+
+    def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
+        blocks: list[AIContentBlock] = []
+        for item in payload.get("output") or []:
+            if not isinstance(item, dict):
+                continue
+            if item.get("type") == "message":
+                for part in item.get("content") or []:
+                    if not isinstance(part, dict):
+                        continue
+                    text = part.get("text") or part.get("refusal")
+                    if isinstance(text, str) and text:
+                        blocks.append(AIContentBlock(type="text", text=text))
+            elif item.get("type") == "reasoning":
+                for part in item.get("summary") or []:
+                    if isinstance(part, dict) and isinstance(part.get("text"), str):
+                        blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
+        return blocks
+
     async def _request_anthropic_messages(
         self,
         model: str,
@@ -226,7 +267,7 @@ class ProviderService:
         resolved_system_prompt = self._resolve_system_prompt(system_prompt)
         if resolved_system_prompt:
             request_body["system"] = resolved_system_prompt
-        resolved_thinking = self._resolve_anthropic_thinking(thinking)
+        resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
         if resolved_thinking:
             request_body["thinking"] = resolved_thinking
         if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
@@ -243,8 +284,12 @@ class ProviderService:
             request_body=request_body,
         )
 
-    def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
+    def _resolve_anthropic_thinking(
+        self, thinking: dict[str, Any] | None, model: str,
+    ) -> dict[str, Any] | None:
         if thinking:
+            if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
+                return {"type": "adaptive"}
             return thinking
 
         # OpenClaw treats MiniMax's Anthropic-compatible path specially:
@@ -295,6 +340,9 @@ class ProviderService:
         headers: dict[str, str],
         request_body: dict[str, Any],
     ) -> dict[str, Any]:
+        headers = {"User-Agent": "Planet/1.0", **headers}
+        if self.provider == "opencode-go":
+            headers["x-opencode-session"] = self.session_id
         last_error: Exception | None = None
         for attempt in range(1, self.http_retry_attempts + 1):
             try:

backend/Dockerfile

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
     uv sync --frozen --no-dev
 
 COPY backend /app/backend
+COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
+COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
+COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
+COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
+COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
+COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
 COPY VERSION /app/VERSION
 
 EXPOSE 8000

backend/app/api/main.py

@@ -1,3 +1,4 @@
+from app.api.v1 import card_images
 from fastapi import APIRouter
 from app.api.v1 import (
     ai,
@@ -12,6 +13,7 @@ from app.api.v1 import (
     alerts,
     settings,
     collected_data,
+    compute_intelligence,
     data_products,
     layers,
     visualization,
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
 api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
 api_router.include_router(system_control.router, prefix="/system", tags=["system"])
 api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
+api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
 api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
 api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
 api_router.include_router(
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
 api_router.include_router(news.router, prefix="/news", tags=["news"])
 api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
 api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
+
+api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])

backend/app/api/v1/card_images.py

@@ -0,0 +1,108 @@
+"""Card-image configuration, upload, catalog selection and public resolution."""
+
+from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
+from fastapi.responses import FileResponse
+from sqlalchemy.ext.asyncio import AsyncSession
+from starlette.concurrency import run_in_threadpool
+
+from app.core.enums import UserRole
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.user import User
+from app.schemas.card_images import (
+    CardImageConfiguration,
+    CardImagePreview,
+    CardImagePublish,
+    EntityType,
+    MatchScope,
+)
+from app.services.card_image_configuration import (
+    publish_configuration,
+    read_configuration,
+    resolve_image,
+)
+from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
+from app.services.card_image_targets import get_target, search_targets
+
+router = APIRouter()
+
+
+async def image_admin(user: User = Depends(get_current_user)) -> User:
+    if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
+        raise HTTPException(403, "Administrator access required")
+    return user
+
+
+@router.get("/configuration")
+async def get_configuration(
+    response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
+):
+    response.headers["Cache-Control"] = "no-store"
+    return await read_configuration(db)
+
+
+@router.put("/configuration")
+async def save_configuration(
+    payload: CardImagePublish,
+    actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return await publish_configuration(db, payload.revision, payload.config, actor)
+
+
+@router.post("/upload")
+async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
+    content = await file.read(MAX_UPLOAD_BYTES + 1)
+    await file.close()
+    return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
+
+
+@router.get("/files/{filename}")
+async def image_file(filename: str):
+    path = uploaded_path(filename)
+    if not path.is_file():
+        raise HTTPException(404, "Image not found")
+    return FileResponse(
+        path,
+        headers={
+            "Cache-Control": "public, max-age=31536000, immutable",
+            "X-Content-Type-Options": "nosniff",
+        },
+    )
+
+
+@router.get("/targets")
+async def targets(
+    entity_type: EntityType,
+    scope: MatchScope = "object",
+    q: str = Query(default="", max_length=100),
+    limit: int = Query(default=25, ge=1, le=50),
+    _actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
+
+
+@router.get("/resolve")
+async def resolve(
+    response: Response,
+    entity_type: EntityType,
+    entity_id: str = Query(min_length=1, max_length=300),
+    db: AsyncSession = Depends(get_db),
+):
+    response.headers["Cache-Control"] = "no-store"
+    envelope = await read_configuration(db)
+    config = CardImageConfiguration.model_validate(envelope["config"])
+    context = await get_target(db, entity_type, entity_id)
+    result = resolve_image(config, context)
+    return {**result, "revision": envelope["revision"], "entity": context}
+
+
+@router.post("/preview")
+async def preview(
+    payload: CardImagePreview,
+    _actor: User = Depends(image_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    context = await get_target(db, payload.entity_type, payload.entity_id)
+    return {**resolve_image(payload.config, context), "entity": context}

backend/app/api/v1/compute_intelligence.py

@@ -0,0 +1,235 @@
+"""Compute observations, administrator configuration and independent facility samples."""
+
+from datetime import UTC, datetime
+from typing import Literal
+
+from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
+from app.api.v1.visualization import convert_compute_centers_to_geojson
+from app.core.earth_geography_policy import display_country
+from app.core.enums import UserRole
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.collected_data import CollectedData
+from app.models.data_snapshot import DataSnapshot
+from app.models.user import User
+from app.schemas.compute_configuration import (
+    ComputeConfiguration,
+    ComputeConfigurationReset,
+    ComputeConfigurationWrite,
+)
+from app.services.compute_catalog import (
+    PARTICIPANTS,
+    participant_members,
+    source_country_names,
+    source_country_participants,
+)
+from app.services.compute_center_locations import refresh_compute_center_location_cache
+from app.services.compute_configuration import (
+    DEFAULT_CONFIGURATION,
+    configuration_response,
+    publish_configuration,
+    read_configuration,
+)
+from app.services.compute_history import compute_history
+from fastapi import APIRouter, Depends, HTTPException, Query, Response
+from sqlalchemy import and_, case, func, or_, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+router = APIRouter()
+
+
+def selected_countries(countries: str) -> list[str]:
+    codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
+    if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
+        raise HTTPException(422, "Select 1–12 participants from the compute catalog")
+    members = participant_members(codes)
+    flattened = [country for participant in members.values() for country in participant]
+    if len(flattened) != len(set(flattened)):
+        raise HTTPException(422, "Comparison participants cannot have overlapping country members")
+    return codes
+
+
+@router.get("/series")
+async def get_series(
+    response: Response,
+    countries: str = "CN,US",
+    mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
+    db: AsyncSession = Depends(get_db),
+):
+    codes = selected_countries(countries)
+    envelope = await read_configuration(db)
+    response.headers["Cache-Control"] = "no-store"
+    result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
+    result.update(
+        version=envelope["revision"],
+        provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
+    )
+    return result
+
+
+async def configuration_admin(user: User = Depends(get_current_user)) -> User:
+    if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
+        raise HTTPException(403, "Administrator access required")
+    return user
+
+
+@router.get("/configuration")
+async def get_configuration(
+    response: Response,
+    _actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    response.headers["Cache-Control"] = "no-store"
+    return configuration_response(await read_configuration(db))
+
+
+@router.put("/configuration")
+async def update_configuration(
+    payload: ComputeConfigurationWrite,
+    actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    envelope = await publish_configuration(
+        db, revision=payload.revision, config=payload.config, actor=actor,
+    )
+    return configuration_response(envelope)
+
+
+@router.post("/configuration/reset")
+async def reset_configuration(
+    payload: ComputeConfigurationReset,
+    actor: User = Depends(configuration_admin),
+    db: AsyncSession = Depends(get_db),
+):
+    envelope = await publish_configuration(
+        db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
+    )
+    return configuration_response(envelope)
+
+
+async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
+    now = datetime.now(UTC)
+    cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
+    ranked = (
+        select(
+            DataSnapshot.id,
+            func.row_number()
+            .over(
+                partition_by=DataSnapshot.source,
+                order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
+            )
+            .label("rank"),
+        )
+        .where(
+            DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
+            DataSnapshot.status == "success",
+            DataSnapshot.completed_at < cutoff,
+        )
+        .subquery()
+    )
+    snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
+    infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
+    # Old collector rows have ambiguous units/identity and must not become
+    # national evidence merely because a new UI can display them.
+    query = select(CollectedData).where(
+        compute_record_filter(CollectedData),
+        CollectedData.deleted_at.is_(None),
+        CollectedData.snapshot_id.in_(snapshots),
+        CollectedData.is_valid == 1,
+        or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
+        or_(
+            infrastructure,
+            CollectedData.source == "top500",
+            and_(
+                CollectedData.extra_data["status"].as_string() == "Existing",
+                func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
+                CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
+            ),
+        ),
+        CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
+    )
+    return query
+
+
+@router.get("/facilities")
+async def get_facilities(
+    countries: str = "CN,US",
+    year: int | None = Query(None, ge=2000, le=2100),
+    limit: int = Query(1000, ge=1, le=2000),
+    db: AsyncSession = Depends(get_db),
+):
+    codes = selected_countries(countries)
+    query = await sample_query(db, codes, year)
+    country = CollectedData.extra_data["country"].as_string()
+    source_participants = source_country_participants(codes)
+    # Canonical source subdivisions retain their raw label on each record but
+    # share one owner in the comparison, using the same policy as the globe.
+    owner_cases = [
+        (country == name, display_country(name))
+        for name in source_country_names(codes)
+        if display_country(name) != name
+    ]
+    display = case(*owner_cases, else_=country) if owner_cases else country
+    participant_cases = [(country == name, code) for name, code in source_participants.items()]
+    participant = case(*participant_cases, else_=None)
+    counts = (
+        (
+            await db.execute(
+                query.with_only_columns(
+                    display.label("country"),
+                    participant.label("participant_code"),
+                    CollectedData.source,
+                    func.count().label("count"),
+                ).group_by(display, participant, CollectedData.source)
+            )
+        )
+        .mappings()
+        .all()
+    )
+    participant_counts = (
+        (
+            await db.execute(
+                query.with_only_columns(
+                    participant.label("participant_code"),
+                    CollectedData.source,
+                    func.count().label("count"),
+                ).group_by(participant, CollectedData.source)
+            )
+        )
+        .mappings()
+        .all()
+    )
+    records = (
+        (await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
+        .scalars()
+        .all()
+    )
+    if year:
+        await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
+    else:
+        await refresh_compute_center_location_cache(db)
+    payload = convert_compute_centers_to_geojson(records)
+    for feature in payload["features"]:
+        feature["properties"]["historical_snapshot"] = year is not None
+        properties = feature["properties"]
+        properties["source_country"] = properties.get("country")
+        properties["country"] = display_country(properties.get("country"))
+    total = sum(row["count"] for row in counts)
+    payload.update(
+        {
+            "counts": [dict(row) for row in counts],
+            "participant_counts": [dict(row) for row in participant_counts],
+            "participants": [PARTICIPANTS[code] for code in codes],
+            "total_count": total,
+            "returned_count": len(records),
+            "truncated": total > limit,
+            "reference_year": year,
+            "scope": "public_source_records",
+            "history_basis": (
+                "snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
+            ),
+            "coverage": "available" if total else "no_verified_snapshot",
+            "cross_source_deduplicated": False,
+        }
+    )
+    return payload

backend/app/api/v1/data_products.py

@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
         "name": "算力",
         "sources": ["top500", "epoch_ai_gpu"],
         "primary_stat_key": "compute_center_count",
-        "stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
+        "stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
     },
 }
 

backend/app/api/v1/earth.py

@@ -13,6 +13,7 @@ from sqlalchemy import delete, func, select, text
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings as app_settings
+from app.core.logging import get_logger
 from app.core.security import decode_token, get_current_user, redis_client
 from app.db.session import get_db
 from app.models.collected_data import CollectedData
@@ -51,6 +52,7 @@ from app.services.earth_boundaries import (
 
 
 router = APIRouter()
+logger = get_logger(__name__)
 optional_bearer = HTTPBearer(auto_error=False)
 REPO_ROOT = Path(__file__).resolve().parents[4]
 EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
@@ -382,13 +384,28 @@ async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
     return await get_earth_news_sources_payload(db)
 
 
+def _schedule_earth_news_source_refresh() -> None:
+    try:
+        from app.services.scheduler import schedule_earth_news_refresh_for_all_regions
+
+        schedule_earth_news_refresh_for_all_regions(force=True)
+    except Exception as exc:
+        logger.warning_event(
+            "Failed to schedule Earth news refresh after source configuration change",
+            event="earth_news.rss.config_refresh_schedule_failed",
+            context={"error": str(exc)},
+        )
+
+
 @router.put("/news-sources")
 async def update_earth_news_sources(
     payload: EarthNewsSourcesPayload,
     _current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    return await save_earth_news_sources_payload(db, payload.model_dump())
+    result = await save_earth_news_sources_payload(db, payload.model_dump())
+    _schedule_earth_news_source_refresh()
+    return result
 
 
 @router.delete("/news-sources")
@@ -397,7 +414,9 @@ async def reset_earth_news_sources(
     _current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
 ):
-    return await reset_earth_news_sources_payload(db)
+    result = await reset_earth_news_sources_payload(db)
+    _schedule_earth_news_source_refresh()
+    return result
 
 
 @router.post("/news-sources/test")

backend/app/api/v1/news.py

@@ -1,17 +1,26 @@
-from fastapi import APIRouter, Depends, HTTPException, Query
+from fastapi import APIRouter, Depends, HTTPException, Query, status
+from pydantic import BaseModel, Field
 from sqlalchemy.ext.asyncio import AsyncSession
+from uuid import UUID
 
 from app.db.session import get_db
 from app.services.earth_news import (
     ALLOWED_NEWS_CATEGORY_KEYS,
     SUPPORTED_NEWS_LOCALES,
     REGION_ANCHORS,
+    determine_focus_region,
     get_earth_news_payload,
 )
 
 router = APIRouter()
 
 
+class EarthNewsRefreshRequest(BaseModel):
+    lat: float | None = Field(default=None, ge=-90, le=90)
+    lon: float | None = Field(default=None, ge=-180, le=180)
+    region: str | None = Field(default=None, max_length=64)
+
+
 def _parse_categories(raw: str | None) -> set[str] | None:
     if raw is None or not raw.strip():
         return None
@@ -39,7 +48,9 @@ def _parse_limit(raw: int | None) -> int:
     if raw is None:
         return 12
     if raw < 1:
-        raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
+        raise HTTPException(
+            status_code=422, detail={"message": "News limit must be greater than 0."}
+        )
     return min(raw, 100)
 
 
@@ -59,18 +70,76 @@ def _parse_locale(raw: str | None) -> str:
     return requested
 
 
+@router.post("/refresh", status_code=status.HTTP_202_ACCEPTED)
+async def request_earth_news_refresh(payload: EarthNewsRefreshRequest):
+    requested_region = payload.region.strip().lower() if payload.region else None
+    if requested_region and requested_region not in REGION_ANCHORS:
+        raise HTTPException(
+            status_code=422,
+            detail={
+                "message": "Unsupported news region.",
+                "invalid_region": requested_region,
+                "allowed_regions": list(REGION_ANCHORS.keys()),
+            },
+        )
+    region = requested_region or determine_focus_region(payload.lat, payload.lon)
+    from app.services.scheduler import request_earth_news_region_refresh
+
+    try:
+        return request_earth_news_region_refresh(region)
+    except Exception as exc:
+        raise HTTPException(
+            status_code=503,
+            detail={
+                "code": "refresh_failed",
+                "message": "Could not start a news refresh. Please try again.",
+            },
+        ) from exc
+
+
+@router.get("/refresh/{refresh_id}")
+async def get_earth_news_refresh(refresh_id: UUID):
+    from app.services.earth_news_refresh import get_earth_news_refresh_status
+
+    try:
+        result = get_earth_news_refresh_status(refresh_id.hex)
+    except Exception as exc:
+        raise HTTPException(
+            status_code=503,
+            detail={
+                "code": "refresh_failed",
+                "message": "Refresh status is temporarily unavailable. Please retry.",
+            },
+        ) from exc
+    if result is not None:
+        return result
+    return {
+        "refresh_id": refresh_id.hex,
+        "region": None,
+        "status": "failed",
+        "error": {
+            "code": "expired",
+            "message": "Refresh status expired. Submit a new refresh.",
+        },
+    }
+
+
 @router.get("/earth-feed")
 async def get_earth_feed(
     lat: float | None = Query(None, description="Current Earth view center latitude"),
     lon: float | None = Query(None, description="Current Earth view center longitude"),
-    region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
+    region: str | None = Query(
+        None, description="Explicit Earth news region for UE/client integrations"
+    ),
     categories: str | None = Query(None, description="Comma-separated news category keys"),
     sources: str | None = Query(None, description="Comma-separated news source ids"),
     limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
     locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
     db: AsyncSession = Depends(get_db),
 ):
-    normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
+    normalized_region = (
+        region.strip().lower() if isinstance(region, str) and region.strip() else None
+    )
     if normalized_region is not None and normalized_region not in REGION_ANCHORS:
         raise HTTPException(
             status_code=422,

backend/app/api/v1/settings.py

@@ -3,9 +3,9 @@ from datetime import UTC, datetime
 import os
 from pathlib import Path
 from typing import Optional
+from urllib.parse import urlsplit
 
 from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
-import httpx
 from pydantic import BaseModel, EmailStr, Field
 from dotenv import dotenv_values
 from sqlalchemy import select
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
     list_fallback_llm_provider_presets,
     refresh_llm_provider_preset,
 )
+from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
 from app.services.scheduler import sync_datasource_job
 from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
 from app.services.persistent_logs import record_audit_log
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
 AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
 AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
 SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
+LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
 
 DEFAULT_SETTINGS = {
     "system": {
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
 
 
 def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
-    merged = deepcopy(DEFAULT_SETTINGS[category])
+    if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
+        defaults = get_fallback_llm_provider_preset(
+            category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
+        )
+    else:
+        defaults = DEFAULT_SETTINGS[category]
+    merged = deepcopy(defaults)
     if payload:
         merged.update(payload)
     return merged
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
 
 
 async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
+    merged = merge_with_defaults(category, payload)
     record = await get_setting_record(db, category)
     if record is None:
         record = SystemSetting(category=category, payload=payload)
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
 
     await db.commit()
     await db.refresh(record)
-    return merge_with_defaults(category, record.payload)
+    return merged
 
 
 AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
         normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
     )
     api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
+    model_provider_apis = {
+        **(provider_config.get("model_provider_apis") or {}),
+        **(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
+    }
+    if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
+        model_provider_apis = {}
     return {
         "service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
         "service_token": _resolve_service_token(normalized_ai)[0],
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
             "api_key": api_key,
             "max_tokens": int(provider_config.get("max_tokens") or 1200),
             "anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
-            "model_provider_apis": provider_config.get("model_provider_apis") or {},
-            "preset_models": _get_provider_preset(default_provider).get("models") or [],
+            "model_provider_apis": model_provider_apis,
         },
     }
 
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
     }
 
 
-def _join_provider_url(base_url: str, path: str) -> str:
-    return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
-
-
-def _extract_model_ids(payload: dict) -> list[str]:
-    data = payload.get("data") if isinstance(payload, dict) else None
-    if isinstance(data, list):
-        return [
-            str(item.get("id"))
-            for item in data
-            if isinstance(item, dict) and item.get("id")
-        ]
-    models = payload.get("models") if isinstance(payload, dict) else None
-    if isinstance(models, list):
-        return [
-            str(item.get("name") or item.get("model") or item.get("id") or item)
-            for item in models
-            if item
-        ]
-    return []
-
-
-def _contains_model(model_ids: list[str], model: str) -> bool:
-    normalized_model = model.strip().lower()
-    return any(str(item).strip().lower() == normalized_model for item in model_ids)
-
-
 async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
     provider = _normalize_provider_id(llm_config.get("provider") or "")
-    configured_api = (
-        str(llm_config.get("provider_api") or "").strip()
-        or ProviderApi.OPENAI_COMPLETIONS.value
-    )
+    provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
     model = str(llm_config.get("model") or "").strip()
-    base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
+    base_url = str(llm_config.get("base_url") or "").strip()
     api_key = str(llm_config.get("api_key") or "").strip()
-    provider_api = configured_api
-    model_provider_apis = llm_config.get("model_provider_apis")
-    if isinstance(model_provider_apis, dict):
-        provider_api = str(model_provider_apis.get(model) or provider_api)
-    preset_models = [
-        str(item)
-        for item in (llm_config.get("preset_models") or [])
-        if str(item).strip()
-    ]
-
-    if not provider or not base_url or not model:
-        return {
-            "success": False,
-            "connected": False,
-            "message": "当前 provider/base_url/model 未完整配置。",
-            "mode": "lightweight_config",
-        }
+    result = {
+        "success": False, "connected": False, "mode": "lightweight_models",
+        "provider": provider, "model": model,
+    }
+    if not base_url or not model:
+        return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
     if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
-        return {
-            "success": False,
-            "connected": False,
-            "message": "当前 provider 未配置 API Key。",
-            "mode": "lightweight_config",
-        }
-
-    if provider == "opencode-go":
-        url = _join_provider_url(base_url, "/models")
-        headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
-        url = _join_provider_url(base_url, "/api/tags")
-        headers: dict[str, str] = {}
-    elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
-        url = _join_provider_url(base_url, "/models")
-        headers = {"Authorization": f"Bearer {api_key}"}
-    elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
-        url = _join_provider_url(base_url, "/models")
-        headers = {
-            "x-api-key": api_key,
-            "anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
-        }
-    else:
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
-            "mode": "lightweight_unsupported",
-        }
-
+        return {**result, "message": "当前 provider 未配置 API Key。"}
     try:
-        async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
-            response = await client.get(url, headers=headers)
-            response.raise_for_status()
-            payload = response.json()
-    except httpx.HTTPStatusError as exc:
-        detail = exc.response.text or exc.response.reason_phrase
-        if exc.response.status_code == 404 and _contains_model(preset_models, model):
-            return {
-                "success": True,
-                "connected": True,
-                "message": "轻量连通性测试通过；当前 provider 不提供可用的模型目录，已按内置模型预设确认。",
-                "mode": "lightweight_preset",
-                "provider": provider,
-                "provider_api": provider_api,
-                "model": model,
-                "url": url,
-            }
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
-            "mode": "lightweight_models",
-            "url": url,
-        }
+        catalog = await fetch_model_catalog(
+            provider, base_url, provider_api, api_key,
+            str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
+        )
     except Exception as exc:
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"轻量连通性测试失败: {exc}",
-            "mode": "lightweight_models",
-            "url": url,
-        }
-
-    model_ids = _extract_model_ids(payload)
-    if model_ids and not _contains_model(model_ids, model):
-        if _contains_model(preset_models, model):
-            return {
-                "success": True,
-                "connected": True,
-                "message": "轻量连通性测试通过；provider 模型目录未返回当前别名，已按内置模型预设确认。",
-                "mode": "lightweight_models_with_preset_alias",
-                "provider": provider,
-                "provider_api": provider_api,
-                "model": model,
-                "models_count": len(model_ids),
-                "url": url,
-            }
-        return {
-            "success": False,
-            "connected": False,
-            "message": f"连接可用，但模型目录中没有当前模型: {model}",
-            "mode": "lightweight_models",
-            "provider": provider,
-            "model": model,
-            "models_count": len(model_ids),
-            "url": url,
-        }
-
+        return {**result, "message": catalog_error_message(exc)}
+    result.update({"url": catalog.url, "models_count": len(catalog.models)})
+    if model.casefold() not in {item.casefold() for item in catalog.models}:
+        return {**result, "message": f"模型目录查询成功，但当前模型不在目录中：{model}"}
+    model_apis = llm_config.get("model_provider_apis") or {}
     return {
-        "success": True,
-        "connected": True,
-        "message": "轻量连通性测试通过",
-        "mode": "lightweight_models",
-        "provider": provider,
-        "provider_api": provider_api,
-        "model": model,
-        "models_count": len(model_ids),
-        "url": url,
+        **result, "success": True, "connected": True,
+        "provider_api": model_apis.get(model) or provider_api,
+        "message": "模型目录查询成功，当前模型已找到；尚未执行生成调用。",
     }
 
 
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
         )
         await emit_business_log(
             logger,
-            event="settings.ai_provider.connect.success",
+            event=("settings.ai_provider.connect.success" if lightweight_result["success"]
+                   else "settings.ai_provider.connect.failed"),
             message="AI provider connection test completed",
             category="ai",
             service="api",
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
                 "provider": payload.provider,
                 "model": payload.model,
                 "configured": True,
-                "lightweight_status": lightweight_result.get("status"),
+                "connected": lightweight_result["connected"],
             },
         )
         return {
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
 @router.get("/integrations/ai-provider/presets")
 async def get_ai_provider_presets(
     current_user: User = Depends(get_current_user),
+    db: AsyncSession = Depends(get_db),
 ):
-    return {"data": list_fallback_llm_provider_presets()}
+    presets = list_fallback_llm_provider_presets()
+    saved = await get_setting_payloads(
+        db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
+    )
+    return {
+        "data": [
+            {**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
+            for preset in presets
+        ]
+    }
 
 
 @router.post("/integrations/ai-provider/presets/{provider}/refresh")
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
     provider: str,
     current_user: User = Depends(get_current_user),
     db: AsyncSession = Depends(get_db),
+    payload: AIProviderIntegrationUpdate | None = None,
 ):
     try:
         provider_id = _normalize_provider_id(provider)
-        api_key = None
-        if provider_id == "opencode-go":
-            current_payload = await get_setting_payload(db, "external_integrations")
-            ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
-            provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
-            api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
-        return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
+        get_fallback_llm_provider_preset(provider_id)
     except ValueError as exc:
         raise HTTPException(status_code=404, detail=str(exc)) from exc
+
+    current_payload = await get_setting_payload(db, "external_integrations")
+    if payload is not None:
+        if _normalize_provider_id(payload.provider) != provider_id:
+            raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
+        ai_payload = _build_ai_provider_payload(current_payload, payload)
+    else:
+        ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
+    provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
+    api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
+    if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
+        raise HTTPException(
+            status_code=400, detail="请先配置当前供应商的 API Key，再刷新模型列表。",
+        )
+    try:
+        refreshed = await refresh_llm_provider_preset(
+            provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
+            provider_api=provider_config.get("provider_api"),
+            anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
+        )
     except Exception as exc:
-        fallback = get_fallback_llm_provider_preset(provider)
-        fallback["refresh_error"] = str(exc)
-        return {"data": fallback}
+        logger.warning(
+            "LLM provider catalog refresh failed",
+            extra={
+                "event": "settings.ai_provider.catalog.failed",
+                "context": {
+                    "provider": provider_id,
+                    "error_type": type(exc).__name__,
+                },
+            },
+        )
+        raise HTTPException(
+            status_code=502,
+            detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
+        ) from exc
+
+    refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
+    await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
+    return {"data": refreshed}
 
 
 @router.put("/integrations")

backend/app/api/v1/tv.py

@@ -7,6 +7,7 @@ from fastapi.responses import Response
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.session import get_db
+from app.services.tv_catalog import get_tv_catalog_page
 from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
 
 router = APIRouter()
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
 
 @router.get("/streams")
 async def list_public_tv_streams(
+    offset: int = Query(0, ge=0),
+    limit: int = Query(50, ge=1, le=100),
+    q: str = Query("", max_length=200),
+    selected_id: str | None = Query(None, max_length=200),
     db: AsyncSession = Depends(get_db),
 ):
-    return await get_public_tv_payload(db)
+    return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
 
 
 @router.get("/proxy")

backend/app/api/v1/visualization.py

@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy import select, func
 from typing import List, Dict, Any, Optional
 
+from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
 from app.core.collected_data_fields import get_record_field
 from app.core.enums import BGPStatus
 from app.core.satellite_tle import build_tle_lines_from_elements
 from app.core.time import to_iso8601_utc
 from app.db.session import get_db
+from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.bgp_incident import BGPIncident
 from app.models.bgp_observation import BGPObservation
@@ -337,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
     return {"type": "FeatureCollection", "features": features}
 
 
-def _normalize_satellite_constellation_group(
-    raw_group: Any,
-    name: Optional[str],
-) -> Optional[str]:
-    normalized_group = str(raw_group or "").strip().lower()
-    if normalized_group:
-        return normalized_group
-
-    normalized_name = str(name or "").strip().upper()
-    if normalized_name.startswith("STARLINK"):
-        return "starlink"
-    if normalized_name.startswith("IRIDIUM"):
-        return "iridium-next"
-
-    return None
-
 
 def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
     if constellation_group == "starlink":
@@ -505,16 +491,19 @@ async def _count_current_or_latest_task_data(
 async def _load_current_collected_data_by_sources(
     db: AsyncSession,
     sources: List[str],
+    *, include_compute_types: bool = False,
 ) -> Dict[str, List[CollectedData]]:
     if not sources:
         return {}
 
     stmt = (
         select(CollectedData)
-        .where(CollectedData.source.in_(sources))
+        .where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
         .where(CollectedData.is_current.is_(True))
         .order_by(CollectedData.source.asc(), CollectedData.id.desc())
     )
+    if include_compute_types:
+        stmt = stmt.where(CollectedData.deleted_at.is_(None))
     result = await db.execute(stmt)
 
     grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
@@ -644,7 +633,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
         return "unknown"
 
     unit = str(capacity_unit or "").strip().lower()
-    if unit in {"pflop/s", "pflops", "pflop"}:
+    if unit in {"flop/s", "flops"}:
+        normalized_tflops = capacity_value / 1e12
+    elif unit in {"pflop/s", "pflops", "pflop"}:
         normalized_tflops = capacity_value * 1000
     elif unit in {"gflop/s", "gflops", "gflop"}:
         normalized_tflops = capacity_value
@@ -672,15 +663,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
     """
     features: List[Dict[str, Any]] = []
     unresolved: List[Dict[str, Any]] = []
+    withheld_count = 0
 
     for record in records:
         metadata = record.extra_data or {}
+        if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
+            withheld_count += 1
+            continue
         result = resolve_compute_center_location_full(record, metadata)
-        site_type = (
-            "supercomputer"
-            if record.source == "top500" or record.data_type == "supercomputer"
-            else "gpu_cluster"
-        )
+        site_type = compute_site_type(record)
 
         if not result.is_resolved:
             diagnostic = result.diagnostic or ResolutionDiagnostic(
@@ -706,7 +697,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
         latitude = location.latitude
         longitude = location.longitude
 
-        if site_type == "supercomputer":
+        if metadata.get("compute_contract") == "v1":
+            capacity_value = _parse_float(get_record_field(record, "value"))
+            capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
+        elif site_type == "supercomputer":
             capacity_value = _parse_float(get_record_field(record, "rmax"))
             capacity_unit = "GFlops"
         else:
@@ -767,7 +761,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
             }
         )
 
-    return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
+    return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
 
 
 VESSEL_TYPE_FILTERS = {
@@ -1121,6 +1115,7 @@ async def build_vessel_snapshot_response(
     safe_limit = _safe_vessel_limit(limit)
     safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
     observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
+    snapshot_started_at = datetime.now(UTC)
     features, diagnostics = await _load_raw_vessel_snapshot_features(
         db,
         bbox=bbox,
@@ -1137,6 +1132,7 @@ async def build_vessel_snapshot_response(
         "features": features,
         "count": len(features),
         "stats": _build_vessel_stats(features),
+        "generated_at": to_iso8601_utc(snapshot_started_at),
         "diagnostics": {
             **diagnostics,
             "filtered_count": len(features),
@@ -1953,9 +1949,10 @@ async def _build_compute_centers_geojson(
     records_by_source = await _load_current_collected_data_by_sources(
         db,
         ["top500", "epoch_ai_gpu"],
+        include_compute_types=True,
     )
     records = _filter_known_records(
-        records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
+        [record for group in records_by_source.values() for record in group],
     )
     if limit is not None:
         records = records[:limit]
@@ -1970,6 +1967,7 @@ async def _build_compute_centers_geojson(
                 "total": 0,
                 "supercomputers": 0,
                 "gpu_clusters": 0,
+                "by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
                 "unresolved": 0,
             },
         }
@@ -2012,6 +2010,7 @@ async def _build_compute_centers_geojson(
         "count": len(sanitized_features),
         "stats": {
             "total": len(sanitized_features),
+            "by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
             "supercomputers": sum(
                 1 for feature in sanitized_features
                 if feature.get("properties", {}).get("site_type") == "supercomputer"
@@ -2328,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
     stmt = (
         select(CollectedData)
         .where(CollectedData.source_id == source_id)
-        .where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
+        .where(compute_record_filter(CollectedData))
+        .where(CollectedData.deleted_at.is_(None))
         .order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
         .limit(1)
     )
@@ -2727,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
     )
 
 
+async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
+    kind = compute_site_type_expression(CollectedData)
+    rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
+        .where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
+               CollectedData.deleted_at.is_(None))
+        .group_by(CollectedData.source, kind))).all()
+    counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
+    present_sources = set()
+    for source, site_type, count in rows:
+        present_sources.add(source)
+        counts[site_type] += int(count or 0)
+    # Retain the existing last-task fallback for the two collected catalogs.
+    for source in COMPUTE_SOURCES:
+        if source not in present_sources:
+            key = "supercomputer" if source == "top500" else "gpu_cluster"
+            counts[key] += await _count_current_or_latest_task_data(db, source)
+    return counts
+
+
 async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
     cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
     landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
@@ -2735,9 +2754,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
         "celestrak_tle",
         exclude_unknown_name=True,
     )
-    supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
-    gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
-    compute_center_count = supercomputer_count + gpu_cluster_count
+    compute_counts = await _compute_facility_counts(db)
+    supercomputer_count = compute_counts["supercomputer"]
+    gpu_cluster_count = compute_counts["gpu_cluster"]
+    compute_center_count = sum(compute_counts.values())
 
     active_incident_result = await db.execute(
         select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
@@ -2792,6 +2812,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
             "aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
             "supercomputer_count": supercomputer_count,
             "gpu_cluster_count": gpu_cluster_count,
+            "data_center_count": compute_counts["data_center"],
+            "cloud_center_count": compute_counts["cloud_center"],
+            "edge_node_count": compute_counts["edge_node"],
             "bgp_event_count": active_incident_count or active_anomaly_count,
             "bgp_incident_count": active_incident_count,
             "bgp_anomaly_count": active_anomaly_count,

backend/app/api/v1/websocket.py

@@ -161,7 +161,7 @@ async def websocket_endpoint(
                     if is_anonymous:
                         channels = [channel for channel in channels if channel in supported_channels]
                     vessel_subscription = None
-                    if "vessels" in channels and "bbox" in payload_data:
+                    if "vessels" in channels:
                         try:
                             vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
                         except ValueError as exc:

backend/app/core/compute_types.py

@@ -0,0 +1,35 @@
+"""Compute facility classification shared by the map and image resolver."""
+
+from sqlalchemy import case, or_
+
+COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
+COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
+
+
+def compute_site_type(record) -> str:
+    metadata = record.extra_data or {}
+    for value in (metadata.get("site_type"), record.data_type):
+        if value in COMPUTE_SITE_TYPES:
+            return value
+    return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
+        record.source, "data_center"
+    )
+
+
+def compute_site_type_expression(record):
+    declared = record.extra_data["site_type"].as_string()
+    return case(
+        (declared.in_(COMPUTE_SITE_TYPES), declared),
+        (record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
+        (record.source == "top500", "supercomputer"),
+        (record.source == "epoch_ai_gpu", "gpu_cluster"),
+        else_="data_center",
+    )
+
+
+def compute_record_filter(record):
+    return or_(
+        record.source.in_(COMPUTE_SOURCES),
+        record.data_type.in_(COMPUTE_SITE_TYPES),
+        record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
+    )

backend/app/core/earth_geography_policy.py

@@ -0,0 +1,57 @@
+"""Country display ownership for Earth comparisons and cards.
+
+Keep source-country evidence separate; use this projection only for display
+grouping so regional source records cannot become competing national keys.
+"""
+
+from __future__ import annotations
+
+import json
+from functools import lru_cache
+from pathlib import Path
+
+from app.core.countries import COUNTRY_ENTRIES, normalize_country
+
+
+_DISPLAY_ISO_A3 = {
+    canonical: next(
+        (alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
+        None,
+    )
+    for canonical, aliases in COUNTRY_ENTRIES
+}
+_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
+_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
+
+
+@lru_cache(maxsize=1)
+def _policy_country_by_code() -> dict[str, str]:
+    try:
+        content = _POLICY_PATH.read_text(encoding="utf-8")
+    except FileNotFoundError as error:
+        raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
+    policy = json.loads(content)
+    if policy.get("profile") != "china-pov-v1":
+        raise ValueError("unsupported Earth display-policy profile")
+    mappings = {}
+    for rule in policy["rules"]:
+        target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
+        if target not in _COUNTRY_BY_ISO_A3:
+            continue
+        for code in rule.get("sourceIsoA3", []):
+            mappings[code] = _COUNTRY_BY_ISO_A3[target]
+    return mappings
+
+
+def display_country(value: str | None) -> str | None:
+    country = normalize_country(value)
+    code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
+    owner = _policy_country_by_code().get(code)
+    if owner:
+        return owner
+    return country
+
+
+def display_iso_a3(value: str | None) -> str | None:
+    country = display_country(value)
+    return _DISPLAY_ISO_A3.get(country) if country else None

backend/app/core/enums.py

@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
 class ProviderApi(StrEnum):
     ANTHROPIC_MESSAGES = "anthropic-messages"
     OPENAI_COMPLETIONS = "openai-completions"
+    OPENAI_RESPONSES = "openai-responses"
     OLLAMA_GENERATE = "ollama-generate"
 
 

backend/app/core/satellite_presentation.py

@@ -0,0 +1,13 @@
+"""Shared source-group normalization; never infer hardware generations."""
+
+
+def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
+    group = str(raw_group or "").strip().lower()
+    if group:
+        return group
+    normalized_name = str(name or "").strip().upper()
+    if normalized_name.startswith("STARLINK"):
+        return "starlink"
+    if normalized_name.startswith("IRIDIUM"):
+        return "iridium-next"
+    return None

backend/app/core/security.py

@@ -6,7 +6,7 @@ import redis
 from fastapi import Depends, HTTPException, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
 from jose import JWTError, jwt
-from sqlalchemy import text
+from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
         return None
 
 
+async def _load_active_user(user_id: str, db: AsyncSession) -> User:
+    try:
+        parsed_id = int(user_id)
+    except (TypeError, ValueError):
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
+    result = await db.execute(select(User).where(User.id == parsed_id))
+    user = result.scalar_one_or_none()
+    if user is None or not user.is_active:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="User not found or inactive",
+        )
+    return user
+
+
 async def get_current_user(
     credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
     db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
             status_code=status.HTTP_401_UNAUTHORIZED,
             detail="Invalid token",
         )
-    result = await db.execute(
-        text(
-            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
-        ),
-        {"id": int(user_id)},
-    )
-    row = result.fetchone()
-    if row is None or not row[5]:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="User not found or inactive",
-        )
-    user = User()
-    user.id = row[0]
-    user.username = row[1]
-    user.email = row[2]
-    user.password_hash = row[3]
-    user.role = row[4]
-    user.is_active = row[5]
-    user.gatekeeper_groups = row[6] or []
-    return user
+    return await _load_active_user(user_id, db)
 
 
 async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
             status_code=status.HTTP_401_UNAUTHORIZED,
             detail="Invalid token",
         )
-    result = await db.execute(
-        text(
-            "SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
-        ),
-        {"id": int(user_id)},
-    )
-    row = result.fetchone()
-    if row is None or not row[5]:
-        raise HTTPException(
-            status_code=status.HTTP_401_UNAUTHORIZED,
-            detail="User not found or inactive",
-        )
-    user = User()
-    user.id = row[0]
-    user.username = row[1]
-    user.email = row[2]
-    user.password_hash = row[3]
-    user.role = row[4]
-    user.is_active = row[5]
-    user.gatekeeper_groups = row[6] or []
-    return user
+    return await _load_active_user(user_id, db)
 
 
 def blacklist_token(token: str) -> None:

backend/app/core/websocket/broadcaster.py

@@ -4,11 +4,14 @@ import asyncio
 from datetime import UTC, datetime
 from typing import Dict, Any
 
+from app.core.logging import get_logger
 from app.core.time import to_iso8601_utc
 from app.core.websocket.manager import manager
 
 
 EARTH_UPDATES_CHANNEL = "earth_updates"
+VESSEL_STATE_QUERY_BATCH_SIZE = 1000
+logger = get_logger(__name__, service="websocket")
 
 
 class DataBroadcaster:
@@ -114,16 +117,22 @@ class DataBroadcaster:
             return
         pending = self._pending_vessel_updates
         self._pending_vessel_updates = {}
-        vessels = []
-        for item in pending.values():
-            vessel = dict(item)
-            source = vessel.pop("_source", None)
-            action = vessel.pop("_action", "upsert")
-            created = vessel.pop("_created", None)
-            vessel["source"] = source
-            vessel["action"] = action
-            vessel["created"] = created
-            vessels.append(vessel)
+        try:
+            vessels = await self._load_current_vessel_updates(list(pending))
+        except Exception:
+            # Preserve newer updates that arrived during the failed database read.
+            self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
+            raise
+        if not vessels:
+            return
+        vessels = [
+            {
+                **vessel,
+                "action": vessel.get("action", "upsert"),
+                "created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
+            }
+            for vessel in vessels
+        ]
         await manager.broadcast_vessels(
             {
                 "action": "upsert",
@@ -133,12 +142,31 @@ class DataBroadcaster:
             }
         )
 
+    async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
+        from app.db.session import async_session_factory
+        from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
+
+        mmsis = [int(key) for key in keys if key.isdigit()]
+        vessels = []
+        async with async_session_factory() as db:
+            for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
+                vessels.extend(await get_current_vessels_by_mmsi(
+                    db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
+                ))
+        present = {int(vessel["mmsi"]) for vessel in vessels}
+        vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
+        return vessels
+
     async def broadcast_vessels_periodically(self):
         while self.running:
             try:
                 await self.flush_vessel_updates()
-            except Exception:
-                pass
+            except Exception as exc:
+                logger.exception_event(
+                    "Failed to flush vessel updates",
+                    event="vessels.broadcast.failed",
+                    context={"error": str(exc)},
+                )
             await asyncio.sleep(self._vessel_flush_interval)
 
     async def broadcast_datasource_task_update(self, data: Dict[str, Any]):

backend/app/core/websocket/manager.py

@@ -63,6 +63,8 @@ class ConnectionManager:
 
     def unsubscribe(self, websocket: WebSocket, channels: list[str]):
         for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
+            if channel == "vessels":
+                self.vessel_subscriptions.pop(websocket, None)
             subscribers = self.channel_subscriptions.get(channel)
             if subscribers is not None:
                 subscribers.discard(websocket)
@@ -88,7 +90,8 @@ class ConnectionManager:
         return subscription
 
     def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
-        bbox = config.get("bbox")
+        global_scope = config.get("scope") == "global"
+        bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
         if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
             raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
         try:
@@ -103,7 +106,7 @@ class ConnectionManager:
             raise ValueError("bbox longitude values must be between -180 and 180")
         if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
             raise ValueError("bbox latitude values must be between -90 and 90")
-        if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
+        if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
             raise ValueError("bbox is too large; zoom in or request a smaller viewport")
 
         zoom = int(config.get("zoom") or 1)
@@ -116,10 +119,11 @@ class ConnectionManager:
             if str(item).strip()
         }
         return {
+            "scope": "global" if global_scope else "viewport",
             "bbox": (lon_min, lat_min, lon_max, lat_max),
             "zoom": zoom,
             "limit": limit,
-            "type": vessel_types,
+            "type": sorted(vessel_types),
             "last_sent_at": None,
         }
 
@@ -152,7 +156,9 @@ class ConnectionManager:
                 vessel
                 for vessel in vessels
                 if self._vessel_matches_subscription(vessel, subscription)
-            ][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
+            ]
+            if subscription.get("scope") != "global":
+                matched = matched[:subscription["limit"]]
             if not matched:
                 continue
             subscription["last_sent_at"] = datetime.now(UTC)
@@ -162,16 +168,24 @@ class ConnectionManager:
                 "timestamp": subscription["last_sent_at"].isoformat(),
                 "payload": {
                     **data,
-                    "vessels": matched,
+                    "vessels": [],
                     "subscription": {
                         "bbox": list(subscription["bbox"]),
                         "zoom": subscription["zoom"],
                         "limit": subscription["limit"],
+                        "scope": subscription.get("scope", "viewport"),
                     },
                 },
             }
             try:
-                await connection.send_json(message)
+                for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
+                    await connection.send_json({
+                        **message,
+                        "payload": {
+                            **message["payload"],
+                            "vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
+                        },
+                    })
             except Exception:
                 self.unsubscribe_all(connection)
 
@@ -180,6 +194,8 @@ class ConnectionManager:
         vessel: dict[str, Any],
         subscription: dict[str, Any],
     ) -> bool:
+        if vessel.get("action") == "remove" and subscription.get("scope") == "global":
+            return True
         try:
             lon = float(vessel.get("lon"))
             lat = float(vessel.get("lat"))

backend/app/db/bgp_prefix_geo.py

@@ -0,0 +1,51 @@
+"""SQL objects supporting indexed BGP prefix geography lookup."""
+
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import AsyncConnection
+
+BGP_PREFIX_GEO_FUNCTION_SQL = (
+    """
+    CREATE OR REPLACE FUNCTION planet_bgp_try_inet(value text)
+    RETURNS inet
+    LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
+    AS $$
+    BEGIN
+        RETURN value::inet;
+    EXCEPTION WHEN data_exception THEN
+        RETURN NULL;
+    END;
+    $$
+    """,
+    """
+    CREATE OR REPLACE FUNCTION planet_bgp_try_masklen(value text)
+    RETURNS integer
+    LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
+    AS $$
+    BEGIN
+        RETURN masklen(value::cidr);
+    EXCEPTION WHEN data_exception THEN
+        RETURN NULL;
+    END;
+    $$
+    """,
+)
+
+BGP_PREFIX_GEO_INDEX_NAME = "idx_bgp_prefix_geo_range_current"
+BGP_PREFIX_GEO_INDEX_SQL = f"""
+CREATE INDEX CONCURRENTLY IF NOT EXISTS {BGP_PREFIX_GEO_INDEX_NAME}
+ON collected_data USING gist (
+    planet_bgp_try_inet(metadata->>'range_start') inet_ops,
+    planet_bgp_try_inet(metadata->>'range_end') inet_ops
+)
+WHERE source IN (
+    'opengeofeed_prefix_geo',
+    'iptoasn_prefix_geo',
+    'nro_delegated_prefix_geo'
+)
+  AND COALESCE(is_current, TRUE) = TRUE
+"""
+
+
+async def install_bgp_prefix_geo_functions(connection: AsyncConnection) -> None:
+    for statement in BGP_PREFIX_GEO_FUNCTION_SQL:
+        await connection.execute(text(statement))

backend/app/db/session.py

@@ -6,6 +6,7 @@ from sqlalchemy.orm import declarative_base
 
 from app.core.config import settings
 from app.core.logging import get_logger
+from app.db.bgp_prefix_geo import install_bgp_prefix_geo_functions
 
 logger = get_logger(__name__)
 
@@ -219,6 +220,7 @@ async def init_db():
 
     async with engine.begin() as conn:
         await conn.run_sync(Base.metadata.create_all)
+        await install_bgp_prefix_geo_functions(conn)
         users_email_verified_existed = (
             await conn.execute(
                 text(
@@ -299,8 +301,12 @@ async def init_db():
                     change_event_id BIGINT;
                     change_payload JSONB;
                 BEGIN
+                    change_event_id := nextval(
+                        pg_get_serial_sequence('earth_data_change_events', 'id')::regclass
+                    );
                     change_payload := jsonb_build_object(
                         'event', 'earth.layer.changed',
+                        'event_id', change_event_id,
                         'table', change_table,
                         'operation', change_operation,
                         'source', change_source,
@@ -311,6 +317,7 @@ async def init_db():
                     );
 
                     INSERT INTO earth_data_change_events (
+                        id,
                         table_name,
                         operation,
                         source,
@@ -318,23 +325,15 @@ async def init_db():
                         payload,
                         occurred_at
                     ) VALUES (
+                        change_event_id,
                         change_table,
                         change_operation,
                         change_source,
                         NULL,
                         change_payload,
                         NOW()
-                    )
-                    RETURNING id INTO change_event_id;
-
-                    change_payload := change_payload || jsonb_build_object(
-                        'event_id', change_event_id
                     );
 
-                    UPDATE earth_data_change_events
-                    SET payload = change_payload
-                    WHERE id = change_event_id;
-
                     PERFORM pg_notify(
                         'planet_earth_data_changes',
                         change_payload::text
@@ -440,6 +439,16 @@ async def init_db():
                             );
                         END LOOP;
                     ELSIF TG_OP = 'UPDATE' THEN
+                        IF TG_TABLE_NAME = 'earth_news_items' AND NOT EXISTS (
+                            SELECT 1
+                            FROM new_rows AS current_row
+                            FULL JOIN old_rows AS previous_row USING (id)
+                            WHERE (to_jsonb(current_row) - 'last_seen_at' - 'updated_at')
+                                IS DISTINCT FROM
+                                  (to_jsonb(previous_row) - 'last_seen_at' - 'updated_at')
+                        ) THEN
+                            RETURN NULL;
+                        END IF;
                         FOR change_source IN
                             SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
                             FROM (

backend/app/schemas/card_images.py

@@ -0,0 +1,128 @@
+"""Typed, validated configuration for optional object-card images."""
+
+from typing import Literal, Self
+from urllib.parse import urlsplit
+
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+
+EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
+MatchScope = Literal["object", "model", "group", "category"]
+
+
+def safe_url(value: str, *, local: bool = False) -> str:
+    if not value:
+        return value
+    if any(ord(char) < 32 for char in value) or "\\" in value:
+        raise ValueError("Invalid URL")
+    parts = urlsplit(value)
+    if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
+        if (
+            ".." not in parts.path
+            and "%" not in parts.path
+            and not parts.netloc
+            and not parts.query
+            and not parts.fragment
+        ):
+            return value
+    if (
+        parts.scheme in {"http", "https"}
+        and parts.hostname
+        and not parts.username
+        and not parts.password
+    ):
+        return value
+    raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
+
+
+class CardImageAsset(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+    id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
+    title: str = Field(min_length=1, max_length=200)
+    url: str = Field(min_length=1, max_length=2000)
+    variants: dict[str, str] = Field(default_factory=dict, max_length=4)
+    width: int = Field(default=0, ge=0, le=20000)
+    height: int = Field(default=0, ge=0, le=20000)
+    credit: str = Field(default="", max_length=500)
+    source_url: str = Field(default="", max_length=2000)
+    license: str = Field(default="", max_length=300)
+    license_url: str = Field(default="", max_length=2000)
+    kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
+    rights_status: Literal["available", "permission_pending"] = "available"
+    enabled: bool = True
+    focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
+    focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
+
+    @field_validator("url")
+    @classmethod
+    def validate_image_url(cls, value: str) -> str:
+        return safe_url(value, local=True)
+
+    @field_validator("source_url", "license_url")
+    @classmethod
+    def validate_source_url(cls, value: str) -> str:
+        return safe_url(value)
+
+    @field_validator("variants")
+    @classmethod
+    def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
+        for width, url in value.items():
+            if width not in {"640", "1280"}:
+                raise ValueError("Unsupported image variant")
+            safe_url(url, local=True)
+        return value
+
+
+class CardImageRule(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+    id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
+    entity_type: EntityType
+    scope: MatchScope
+    value: str = Field(min_length=1, max_length=300)
+    label: str = Field(default="", max_length=500)
+    asset_id: str | None = Field(default=None, max_length=100)
+    suppress: bool = False
+    enabled: bool = True
+
+    @model_validator(mode="after")
+    def validate_action(self) -> Self:
+        if self.scope != "object":
+            self.value = self.value.casefold()
+        if self.suppress and (self.scope != "object" or self.asset_id):
+            raise ValueError("Only an object rule may suppress its image, without an asset")
+        if not self.suppress and not self.asset_id:
+            raise ValueError("Choose an image or suppress this object")
+        return self
+
+    def target(self) -> tuple[str, str, str]:
+        return self.entity_type, self.scope, self.value
+
+
+class CardImageConfiguration(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+    assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
+    rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
+
+    @model_validator(mode="after")
+    def validate_relations(self) -> Self:
+        assets = {asset.id for asset in self.assets}
+        if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
+            self.rules
+        ):
+            raise ValueError("Image and rule IDs must be unique")
+        targets = [rule.target() for rule in self.rules]
+        if len(set(targets)) != len(targets):
+            raise ValueError("Each target may have only one rule")
+        if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
+            raise ValueError("Remove image associations before deleting the image")
+        return self
+
+
+class CardImagePublish(BaseModel):
+    revision: str = Field(max_length=200)
+    config: CardImageConfiguration
+
+
+class CardImagePreview(BaseModel):
+    entity_type: EntityType
+    entity_id: str = Field(min_length=1, max_length=300)
+    config: CardImageConfiguration

backend/app/schemas/compute_configuration.py

@@ -0,0 +1,94 @@
+"""Validated, source-backed observations for the compute comparison cards."""
+
+from datetime import date
+from typing import Annotated, Literal, Self
+
+from app.services.compute_catalog import COUNTRY_PARTICIPANTS
+from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
+
+Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
+Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
+Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
+Text = Annotated[str, Field(min_length=1, max_length=500)]
+
+
+class ConfigurationModel(BaseModel):
+    model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
+
+
+class ComputeSource(ConfigurationModel):
+    id: Identifier
+    title: Text
+    url: HttpUrl
+    publisher: Text
+    report_year: Year
+    published_at: date | None = None
+    locator: Text
+
+    @field_validator("url")
+    @classmethod
+    def public_reference(cls, value: HttpUrl) -> HttpUrl:
+        if value.username or value.password or len(str(value)) > 2000:
+            raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
+        return value
+
+
+class ComputeObservation(ConfigurationModel):
+    year: Year
+    metric: Literal["global_share", "absolute"]
+    source_id: Identifier
+    values: dict[str, Value] = Field(min_length=1, max_length=250)
+
+    @model_validator(mode="after")
+    def comparable_values(self) -> Self:
+        if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
+            raise ValueError(
+                "Use countries from the canonical compute catalog, not regional totals"
+            )
+        if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
+            raise ValueError("Global shares for one year must total at most 100 percent")
+        return self
+
+
+class InnovationObservation(ConfigurationModel):
+    year: Year
+    value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
+    source_id: Identifier
+
+
+class ComputeConfiguration(ConfigurationModel):
+    reviewed_at: date
+    sources: list[ComputeSource] = Field(max_length=200)
+    observations: list[ComputeObservation] = Field(max_length=202)
+    innovation: list[InnovationObservation] = Field(max_length=101)
+
+    @model_validator(mode="after")
+    def consistent_evidence(self) -> Self:
+        ids = [source.id for source in self.sources]
+        if len(ids) != len(set(ids)):
+            raise ValueError("Report source IDs must be unique")
+        keys = [(row.metric, row.year) for row in self.observations]
+        if len(keys) != len(set(keys)):
+            raise ValueError("Each metric/year may have only one observation")
+        years = [row.year for row in self.innovation]
+        if len(years) != len(set(years)):
+            raise ValueError("Innovation observation years must be unique")
+        sources = {source.id: source for source in self.sources}
+        for row in [*self.observations, *self.innovation]:
+            source = sources.get(row.source_id)
+            if source is None:
+                raise ValueError(f"Missing report source: {row.source_id}")
+            if row.year > source.report_year:
+                raise ValueError("Observation year must not be after its report year")
+        if self.reviewed_at > date.today():
+            raise ValueError("Review date must not be in the future")
+        return self
+
+
+class ComputeConfigurationWrite(ConfigurationModel):
+    revision: str = Field(min_length=1, max_length=80)
+    config: ComputeConfiguration
+
+
+class ComputeConfigurationReset(ConfigurationModel):
+    revision: str = Field(min_length=1, max_length=80)

backend/app/schemas/user.py

@@ -36,6 +36,9 @@ class UserInDB(UserBase):
 
 
 class UserResponse(UserBase):
+    # Bootstrap accounts have reserved .local email addresses. Validate email
+    # on writes, while allowing existing local identities to be serialized.
+    email: str
     id: int
     role: str
     gatekeeper_groups: list[str] = Field(default_factory=list)

backend/app/services/bgp_enrichment.py

@@ -111,15 +111,24 @@ async def _lookup_prefix_geography(
         result = await db.execute(
             text(
                 """
+                WITH candidates AS MATERIALIZED (
+                    SELECT source, id, metadata
+                    FROM collected_data
+                    WHERE source IN (
+                        'opengeofeed_prefix_geo',
+                        'iptoasn_prefix_geo',
+                        'nro_delegated_prefix_geo'
+                    )
+                      AND COALESCE(is_current, TRUE) = TRUE
+                      AND planet_bgp_try_inet(metadata->>'range_start') <= CAST(:range_start AS inet)
+                      AND planet_bgp_try_inet(metadata->>'range_end') >= CAST(:range_end AS inet)
+                )
                 SELECT metadata
-                FROM collected_data
+                FROM candidates
                 WHERE source = :source
-                  AND COALESCE(is_current, TRUE) = TRUE
                   AND metadata->>'family' = :family
-                  AND CAST(metadata->>'range_start' AS inet) <= CAST(:range_start AS inet)
-                  AND CAST(metadata->>'range_end' AS inet) >= CAST(:range_end AS inet)
                 ORDER BY
-                  masklen(CAST(metadata->>'prefix' AS cidr)) DESC NULLS LAST,
+                  planet_bgp_try_masklen(metadata->>'prefix') DESC NULLS LAST,
                   id DESC
                 LIMIT 1
                 """
@@ -146,6 +155,10 @@ async def _lookup_prefix_geography(
 
     results: dict[str, dict[str, Any]] = {}
 
+    # The GiST range predicate is selective, but PostgreSQL 15 greatly overestimates its cost.
+    # Avoid spending more time compiling JIT code than fetching the small candidate set.
+    await db.execute(text("SET LOCAL jit = off"))
+
     for prefix in prefix_values:
         try:
             network = ipaddress.ip_network(prefix, strict=False)

backend/app/services/card_image_configuration.py

@@ -0,0 +1,174 @@
+"""Revision-checked image settings; new bundled assets never replace user edits."""
+
+from uuid import uuid4
+
+from fastapi import HTTPException
+from sqlalchemy import select
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.request_context import get_request_id
+from app.models.system_log import AuditLog
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.schemas.card_images import CardImageConfiguration
+from app.services.card_image_defaults import (
+    bundled_configuration,
+    previous_bundled_assets,
+    previous_bundled_rules,
+)
+
+CATEGORY = "earth_card_images"
+
+
+def effective_envelope(payload: dict | None) -> dict:
+    defaults, version = bundled_configuration()
+    config = (
+        CardImageConfiguration.model_validate(payload["config"])
+        if payload
+        else defaults.model_copy(deep=True)
+    )
+    if payload:
+        previous = previous_bundled_assets()
+        latest = {asset.id: asset for asset in defaults.assets}
+        for index, asset in enumerate(config.assets):
+            old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
+            # Replace the old bundled source while preserving fields the user
+            # changed (crop, title, enabled state, etc.). Custom URLs are untouched.
+            if old is not None and asset.url == old.url and asset.id in latest:
+                overrides = {
+                    key: value
+                    for key, value in asset.model_dump().items()
+                    if value != getattr(old, key)
+                }
+                config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
+        known_assets = set(payload.get("known_assets", []))
+        asset_ids = {asset.id for asset in config.assets}
+        config.assets.extend(
+            asset.model_copy(deep=True)
+            for asset in defaults.assets
+            if asset.id not in known_assets and asset.id not in asset_ids
+        )
+        prior_rules = previous_bundled_rules()
+        latest_rules = {rule.id: rule for rule in defaults.rules}
+        for index, rule in enumerate(config.rules):
+            old_rule = prior_rules.get(rule.id)
+            if (
+                old_rule
+                and rule.id in latest_rules
+                and rule.target() == old_rule.target()
+                and rule.asset_id == old_rule.asset_id
+            ):
+                overrides = {
+                    key: value
+                    for key, value in rule.model_dump().items()
+                    if value != getattr(old_rule, key)
+                }
+                config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
+        known_rules = set(payload.get("known_rules", []))
+        rule_ids = {rule.id for rule in config.rules}
+        targets = {rule.target() for rule in config.rules}
+        available = {asset.id for asset in config.assets}
+        config.rules.extend(
+            rule.model_copy(deep=True)
+            for rule in defaults.rules
+            if rule.id not in known_rules
+            and rule.id not in rule_ids
+            and rule.target() not in targets
+            and rule.asset_id in available
+        )
+    return {
+        "config": config.model_dump(mode="json"),
+        "revision": f"{version}:{payload['revision'] if payload else 'default'}",
+        "is_default": payload is None,
+    }
+
+
+async def read_configuration(db: AsyncSession) -> dict:
+    record = (
+        await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
+    ).scalar_one_or_none()
+    return effective_envelope(record.payload if record else None)
+
+
+async def publish_configuration(
+    db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
+) -> dict:
+    defaults, _ = bundled_configuration()
+    pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
+    if any(
+        asset.id in pending and asset.rights_status != "permission_pending"
+        for asset in config.assets
+    ):
+        raise HTTPException(422, "Bundled image permission is still unverified")
+    initial = {
+        "config": defaults.model_dump(mode="json"),
+        "revision": "default",
+        "known_assets": [],
+        "known_rules": [],
+    }
+    await db.execute(
+        insert(SystemSetting)
+        .values(category=CATEGORY, payload=initial)
+        .on_conflict_do_nothing(index_elements=[SystemSetting.category])
+    )
+    record = (
+        await db.execute(
+            select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
+        )
+    ).scalar_one()
+    if effective_envelope(record.payload)["revision"] != revision:
+        await db.rollback()
+        raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
+    before = record.payload
+    record.payload = {
+        "config": config.model_dump(mode="json"),
+        "revision": uuid4().hex,
+        "known_assets": [asset.id for asset in defaults.assets],
+        "known_rules": [rule.id for rule in defaults.rules],
+    }
+    db.add(
+        AuditLog(
+            actor_id=actor.id,
+            actor_name=actor.username,
+            action="card_images.update",
+            target_type="system_setting",
+            target_id=CATEGORY,
+            result="success",
+            request_id=get_request_id(),
+            details={"before": before, "after": record.payload},
+        )
+    )
+    await db.commit()
+    return effective_envelope(record.payload)
+
+
+def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
+    assets = {asset.id: asset for asset in config.assets}
+    for scope in ("object", "model", "group", "category"):
+        value = context.get("entity_id" if scope == "object" else scope)
+        values = [value] if value else []
+        if scope == "category" and value != "*":
+            values.append("*")
+        for value in values:
+            value = value if scope == "object" else value.casefold()
+            rule = next(
+                (
+                    rule
+                    for rule in config.rules
+                    if rule.enabled
+                    and rule.entity_type == context["entity_type"]
+                    and rule.scope == scope
+                    and rule.value == value
+                ),
+                None,
+            )
+            if not rule:
+                continue
+            match = {"scope": scope, "value": value, "rule_id": rule.id}
+            if rule.suppress:
+                return {"image": None, "match": match, "suppressed": True}
+            asset = assets.get(rule.asset_id)
+            if asset and asset.enabled:
+                return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
+    return {"image": None, "match": None, "suppressed": False}

backend/app/services/card_image_defaults.py

@@ -0,0 +1,108 @@
+"""Read immutable bundled assets without inserting machine-specific database rows."""
+
+import hashlib
+import json
+from functools import lru_cache
+from pathlib import Path
+
+from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
+
+
+def _asset_from_manifest(item: dict) -> CardImageAsset:
+    pending = item.get("redistribution_status") == "permission_not_verified"
+    return CardImageAsset(
+        id=item["id"],
+        title=item["label"],
+        url=item["url"],
+        width=item["width"],
+        height=item["height"],
+        credit=item["credit"],
+        source_url=item.get("source_page") or "",
+        license=item["license"],
+        license_url=item.get("license_url") or "",
+        kind=item["kind"],
+        rights_status="permission_pending" if pending else "available",
+        enabled=item.get("enabled", not pending),
+        focus_x=item.get("focus_x", 50),
+        focus_y=item.get("focus_y", 50),
+        variants=item.get("variants", {}),
+    )
+
+
+@lru_cache(maxsize=1)
+def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
+    manifest = json.loads(MANIFEST_PATH.read_bytes())
+    return {
+        key: [CardImageAsset.model_validate(item) for item in value]
+        for key, value in manifest.get("previous_defaults", {}).items()
+    }
+
+
+@lru_cache(maxsize=1)
+def previous_bundled_rules() -> dict[str, CardImageRule]:
+    manifest = json.loads(MANIFEST_PATH.read_bytes())
+    return {
+        key: CardImageRule.model_validate(value)
+        for key, value in manifest.get("previous_rules", {}).items()
+    }
+
+
+@lru_cache(maxsize=1)
+def bundled_configuration() -> tuple[CardImageConfiguration, str]:
+    content = MANIFEST_PATH.read_bytes()
+    manifest = json.loads(content)
+    assets, rules = [], []
+    for item in manifest["assets"]:
+        asset = _asset_from_manifest(item)
+        assets.append(asset)
+        if not asset.enabled:
+            continue
+        hints = item.get("binding_hints", {})
+        for index, target in enumerate(hints.get("defaults", [])):
+            rules.append(
+                CardImageRule(
+                    id=target.get("id", f"builtin-{asset.id}-default-{index}"),
+                    entity_type=target["entity_type"],
+                    scope=target["scope"],
+                    value=target["value"],
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+        for index, entity_id in enumerate(hints.get("entity_ids", [])):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-entity-{index}",
+                    entity_type=item["entity_type"],
+                    scope="object",
+                    value=entity_id,
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+        for norad_id in hints.get("norad_ids", []):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-{norad_id}",
+                    entity_type="satellite",
+                    scope="object",
+                    value=str(norad_id),
+                    label=f"{asset.title} · NORAD {norad_id}",
+                    asset_id=asset.id,
+                )
+            )
+        if group := hints.get("constellation_group"):
+            rules.append(
+                CardImageRule(
+                    id=f"builtin-{asset.id}-group",
+                    entity_type="satellite",
+                    scope="group",
+                    value=group,
+                    label=asset.title,
+                    asset_id=asset.id,
+                )
+            )
+    return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()

backend/app/services/card_image_files.py

@@ -0,0 +1,95 @@
+"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
+
+import hashlib
+import io
+import re
+import warnings
+from uuid import uuid4
+from pathlib import Path
+import os
+
+from fastapi import HTTPException
+from PIL import Image, ImageOps, UnidentifiedImageError
+
+from app.schemas.card_images import CardImageAsset
+from app.services.card_image_defaults import REPO_ROOT
+
+ASSET_DIR = Path(
+    os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
+).resolve()
+MAX_UPLOAD_BYTES = 12 * 1024 * 1024
+MAX_PIXELS = 32_000_000
+FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
+FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
+
+
+def uploaded_path(filename: str) -> Path:
+    if not FILE_PATTERN.fullmatch(filename):
+        raise HTTPException(404, "Image not found")
+    return ASSET_DIR / filename
+
+
+def store_image(content: bytes, title: str) -> CardImageAsset:
+    if len(content) > MAX_UPLOAD_BYTES:
+        raise HTTPException(413, "Image must be 12 MiB or smaller")
+    try:
+        with warnings.catch_warnings():
+            warnings.simplefilter("error", Image.DecompressionBombWarning)
+            image = Image.open(io.BytesIO(content))
+            extension = FORMATS.get(image.format)
+            if (
+                not extension
+                or image.width * image.height > MAX_PIXELS
+                or getattr(image, "is_animated", False)
+            ):
+                raise ValueError("Unsupported or oversized raster")
+            image.load()
+            image = ImageOps.exif_transpose(image).convert(
+                "RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
+            )
+    except (
+        UnidentifiedImageError,
+        OSError,
+        ValueError,
+        Image.DecompressionBombError,
+        Image.DecompressionBombWarning,
+    ) as error:
+        raise HTTPException(
+            422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
+        ) from error
+    digest = hashlib.sha256(content).hexdigest()
+    ASSET_DIR.mkdir(parents=True, exist_ok=True)
+    original = f"{digest}-original.{extension}"
+    original_path = uploaded_path(original)
+    if not original_path.exists():
+        temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
+        temporary.write_bytes(content)
+        temporary.replace(original_path)
+    variants = {}
+    for size, content in render_variants(image).items():
+        filename = f"{digest}-{size}.webp"
+        path = uploaded_path(filename)
+        if not path.exists():
+            temporary = path.with_suffix(f".{uuid4().hex}.tmp")
+            temporary.write_bytes(content)
+            temporary.replace(path)
+        variants[size] = f"/api/v1/earth/card-images/files/{filename}"
+    return CardImageAsset(
+        id=f"upload-{digest[:24]}",
+        title=title or "Image",
+        url=f"/api/v1/earth/card-images/files/{original}",
+        variants=variants,
+        width=image.width,
+        height=image.height,
+    )
+
+
+def render_variants(image: Image.Image) -> dict[str, bytes]:
+    result = {}
+    for size in (640, 1280):
+        resized = image.copy()
+        resized.thumbnail((size, size), Image.Resampling.LANCZOS)
+        buffer = io.BytesIO()
+        resized.save(buffer, format="WEBP", quality=93, method=4)
+        result[str(size)] = buffer.getvalue()
+    return result

backend/app/services/card_image_targets.py

@@ -0,0 +1,180 @@
+"""Bounded catalog queries and type defaults for every Earth object card."""
+
+from sqlalchemy import String, case, cast, func, literal, or_, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.compute_types import (
+    COMPUTE_SITE_TYPES,
+    compute_site_type_expression,
+    compute_record_filter,
+)
+from app.models.bgp_incident import BGPIncident
+from app.models.bgp_collector_location import BGPCollectorLocation
+from app.models.collected_data import CollectedData
+from app.models.earth_interactable import EarthInteractable
+from app.models.vessel import VesselCurrentState
+from app.schemas.card_images import EntityType, MatchScope
+
+SOURCES = {
+    "satellite": ("celestrak_tle", "spacetrack_tle"),
+    "compute_center": ("top500", "epoch_ai_gpu"),
+    "cable": ("arcgis_cables",),
+    "landing_point": ("arcgis_landing_points",),
+}
+# Named constellation families, not inferred hardware models or generations.
+SATELLITE_GROUP_PATTERNS = (
+    ("STARLINK%", "starlink"),
+    ("IRIDIUM%", "iridium-next"),
+    ("BEIDOU-3%", "beidou-3"),
+    ("BEIDOU%", "beidou"),
+    ("QIANFAN%", "qianfan"),
+    ("ONEWEB%", "oneweb"),
+    ("KUIPER%", "kuiper"),
+)
+
+
+def collected_query(entity_type: EntityType):
+    record, metadata = CollectedData, CollectedData.extra_data
+    model = func.coalesce(
+        metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
+    )
+    category = (
+        compute_site_type_expression(record)
+        if entity_type == "compute_center"
+        else literal(entity_type)
+    )
+    if entity_type == "satellite":
+        group = func.coalesce(
+            func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
+            case(
+                *[
+                    (func.upper(record.name).like(pattern), value)
+                    for pattern, value in SATELLITE_GROUP_PATTERNS
+                ],
+                else_="",
+            ),
+        )
+        entity_id = metadata["norad_cat_id"].as_string()
+    else:
+        group = func.coalesce(
+            metadata["operator"].as_string(), metadata["organization"].as_string(), ""
+        )
+        entity_id = record.source + ":" + record.source_id
+    return select(
+        record.name.label("name"),
+        cast(entity_id, String).label("entity_id"),
+        model.label("model"),
+        group.label("group"),
+        category.label("category"),
+    ).where(
+        (
+            compute_record_filter(record)
+            if entity_type == "compute_center"
+            else record.source.in_(SOURCES[entity_type])
+        ),
+        record.is_current.is_(True),
+        record.deleted_at.is_(None),
+        metadata["anonymous"].as_boolean().is_not(True),
+    )
+
+
+def target_query(entity_type: EntityType):
+    if entity_type in SOURCES:
+        return collected_query(entity_type)
+    if entity_type == "vessel":
+        model = VesselCurrentState
+        identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
+        conditions = []
+    elif entity_type == "bgp":
+        model = BGPIncident
+        identity, name, group, category = (
+            model.incident_key,
+            model.title,
+            model.source,
+            model.incident_type,
+        )
+        conditions = []
+    elif entity_type == "bgp_collector":
+        model = BGPCollectorLocation
+        identity, name, group, category = (
+            model.collector_id,
+            model.collector_id,
+            model.operator,
+            literal("bgp_collector"),
+        )
+        conditions = []
+    else:
+        model = EarthInteractable
+        identity, name, group, category = model.id, model.label, model.layer, model.kind
+        conditions = [model.is_deleted.is_(False)]
+    return select(
+        name.label("name"),
+        cast(identity, String).label("entity_id"),
+        literal("").label("model"),
+        func.coalesce(group, "").label("group"),
+        func.coalesce(category, entity_type).label("category"),
+    ).where(*conditions)
+
+
+def category_context(entity_type: EntityType) -> dict[str, str]:
+    return {
+        "entity_type": entity_type,
+        "entity_id": "",
+        "name": "",
+        "model": "",
+        "group": "",
+        "category": entity_type,
+    }
+
+
+async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
+    if entity_type == "compute_center" and entity_id.startswith("__category__:"):
+        category = entity_id.partition(":")[2]
+        if category in COMPUTE_SITE_TYPES:
+            return {**category_context(entity_type), "category": category}
+    if entity_id == "__category__":
+        return category_context(entity_type)
+    if entity_type in SOURCES and entity_type != "satellite":
+        source, separator, source_id = entity_id.partition(":")
+        if not separator or not source or not source_id:
+            return category_context(entity_type)
+        # Match the indexed identity columns before projecting JSON metadata.
+        statement = collected_query(entity_type).where(
+            CollectedData.source == source, CollectedData.source_id == source_id
+        )
+    else:
+        catalog = target_query(entity_type).subquery()
+        statement = select(catalog).where(catalog.c.entity_id == entity_id)
+    row = (await db.execute(statement.limit(1))).mappings().first()
+    if not row:
+        # Old map snapshots can outlive a catalog refresh. They may use the type
+        # illustration, but must not claim an unverified exact-object identity.
+        return category_context(entity_type)
+    context = {key: str(value or "") for key, value in row.items()}
+    return {**context, "entity_type": entity_type}
+
+
+async def search_targets(
+    db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
+) -> list[dict[str, str]]:
+    catalog = target_query(entity_type).subquery()
+    value = catalog.c["entity_id" if scope == "object" else scope]
+    label = catalog.c.name if scope == "object" else value
+    statement = select(value.label("value"), label.label("label")).where(
+        value.is_not(None), value != ""
+    )
+    if query:
+        escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
+        statement = statement.where(
+            or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
+        )
+    rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
+    items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
+    if scope == "category" and not query:
+        known = {item["value"] for item in items}
+        if entity_type == "compute_center":
+            items.extend(
+                {"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
+            )
+        items.insert(0, {"value": "*", "label": "*"})
+    return items

backend/app/services/collectors/base.py

@@ -31,6 +31,7 @@ class BaseCollector(ABC):
     frequency_hours: int = 4
     data_type: str = "generic"
     fail_on_empty: bool = False
+    preserve_snapshot_on_failure: bool = False
 
     def __init__(self):
         self._current_task = None
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
                 context={"task_id": task_id, "raw_count": len(raw_data)},
             )
             data = self.transform(raw_data)
+            if self.fail_on_empty and not data:
+                raise RuntimeError(f"Collector {self.name} produced no valid records")
             await self._log_collection_event(
                 "collector.phase.transforming.success",
                 "Collector transform phase completed",
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
             raise
         except Exception as e:
             await db.rollback()
+            if snapshot_id is not None and self.preserve_snapshot_on_failure:
+                await self._rollback_incomplete_run(
+                    db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
+                )
             task.status = JobStatus.FAILED.value
             task.phase = "failed"
             task.phase_message = str(e)
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
                 snapshot = await db.get(DataSnapshot, snapshot_id)
                 if snapshot:
                     snapshot.status = SnapshotStatus.FAILED.value
+                    if self.preserve_snapshot_on_failure:
+                        snapshot.is_current = False
                     snapshot.completed_at = datetime.now(UTC)
                     snapshot.summary = {"error": str(e)}
             await db.commit()

backend/app/services/collectors/celestrak.py

@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
 from app.services.collectors.base import BaseCollector
 from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
 
-
 logger = get_logger(__name__, service="collector")
 ACTIVE_GROUP = "active"
 FALLBACK_GROUPS = (
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
 )
 FETCH_RETRY_ATTEMPTS = 3
 FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
+CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
 CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
 
 
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
     _downloader = ResumableFileDownloader(
         cache_namespace="celestrak",
         default_accept="application/json",
+        preflight_head=False,
+        cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
     )
 
     @property
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
                     if not is_final_attempt:
                         await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
 
-        raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
+        raise RuntimeError(
+            f"CelesTrak active satellite download failed after retries: {last_error}"
+        )
 
     async def _fetch_fallback_groups(
         self,
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
         try:
             for group in FALLBACK_GROUPS:
                 group_url = self._group_url(group)
-                cached_path = self._downloader.get_cached_file(
-                    group_url,
-                    ".json",
-                    validate_existing=self._validate_json_file,
-                )
-                if cached_path is not None:
-                    body_path = cached_path
-                else:
-                    try:
-                        body_path = await self._downloader.download_file(
-                            client,
-                            group_url,
-                            extension=".json",
-                            accept="application/json",
-                            validate_existing=self._validate_json_file,
-                        )
-                    except DownloadHTTPStatusError as exc:
-                        if not self._is_not_updated_response(exc):
-                            raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
+                try:
+                    body_path = await self._downloader.download_file(
+                        client,
+                        group_url,
+                        extension=".json",
+                        accept="application/json",
+                        validate_existing=self._validate_json_file,
+                    )
+                except DownloadHTTPStatusError as exc:
+                    if not self._is_not_updated_response(exc):
+                        raise RuntimeError(
+                            f"CelesTrak fallback group '{group}' download failed: {exc}"
+                        ) from exc
+                    body_path = self._downloader.get_cached_file(
+                        group_url,
+                        ".json",
+                        validate_existing=self._validate_json_file,
+                    )
+                    if body_path is None:
                         raise RuntimeError(
                             f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
                         ) from exc
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
                 commit=True,
             )
 
-    @staticmethod
-    def _validate_json_file(path: Path) -> bool:
+    @classmethod
+    def _validate_json_file(cls, path: Path) -> bool:
         try:
-            data = json.loads(path.read_text(encoding="utf-8"))
-        except (OSError, json.JSONDecodeError, UnicodeDecodeError):
+            cls._load_active_payload(path)
+        except RuntimeError:
             return False
-        return isinstance(data, list)
+        return True
 
     async def _log_parse_failure(self, exc: Exception) -> None:
         await emit_business_log(
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
             ),
         )
 
-    def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
+    @staticmethod
+    def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
         try:
             raw = json.loads(path.read_text(encoding="utf-8"))
         except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
             else:
                 invalid_count += 1
         if invalid_count:
-            raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
+            raise RuntimeError(
+                f"CelesTrak active payload contains {invalid_count} invalid record(s)"
+            )
         if not records:
             raise RuntimeError("CelesTrak active payload contains no satellite records")
         return records

backend/app/services/collectors/compute_base.py

@@ -0,0 +1,82 @@
+"""Quality gates shared by the two compute sources."""
+
+from datetime import datetime
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.countries import normalize_country
+from app.models.compute_center_location import ComputeCenterLocationRecord
+from app.models.data_snapshot import DataSnapshot
+from app.services.collectors.base import BaseCollector
+
+
+class ComputeCollector(BaseCollector):
+    fail_on_empty = True
+    preserve_snapshot_on_failure = True
+
+    async def _create_snapshot(
+        self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
+    ) -> int:
+        previous_count = await db.scalar(
+            select(DataSnapshot.record_count)
+            .where(
+                DataSnapshot.source == self.name,
+                DataSnapshot.is_current.is_(True),
+                DataSnapshot.status == "success",
+            )
+            .order_by(DataSnapshot.id.desc())
+            .limit(1)
+        )
+        if previous_count and len(data) < previous_count * 0.6:
+            raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
+        return await super()._create_snapshot(db, task_id, data, started_at)
+
+    async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
+        if self.name == "top500":
+            await self._copy_verified_locations(db, data)
+        return await super()._save_data(db, data, **kwargs)
+
+    async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
+        """Keep historical location keys; copy only unambiguous exact identities.
+
+        Never migrate by rank: the machine occupying rank 1 changes over time.
+        """
+        names = [item["name"] for item in data]
+        locations = (
+            (
+                await db.execute(
+                    select(ComputeCenterLocationRecord).where(
+                        ComputeCenterLocationRecord.source == self.name,
+                        ComputeCenterLocationRecord.name.in_(names),
+                    )
+                )
+            )
+            .scalars()
+            .all()
+        )
+        existing = {location.source_id for location in locations}
+        for item in data:
+            if item["source_id"] in existing:
+                continue
+            country = normalize_country(item["country"])
+            if not country:
+                continue
+            matches = [
+                location
+                for location in locations
+                if location.name == item["name"]
+                and normalize_country(location.country) == country
+                and (not location.site or location.site == item.get("metadata", {}).get("site"))
+                and not location.source_id.startswith("top500_system_")
+            ]
+            if len(matches) != 1:
+                continue
+            old = matches[0]
+            values = {
+                column.name: getattr(old, column.name)
+                for column in old.__table__.columns
+                if column.name not in {"id", "source_id", "created_at", "updated_at"}
+            }
+            db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
+            existing.add(item["source_id"])

backend/app/services/collectors/downloads.py

@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
 
 import httpx
 
-
 ProgressCallback = Callable[[int, int | None], Awaitable[None]]
 ValidateCallback = Callable[[Path], bool]
 
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
         user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
         default_accept: str = "*/*",
         cache_root: Path | None = None,
+        preflight_head: bool = True,
+        cache_ttl_seconds: float = 0,
     ) -> None:
         self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
         self._user_agent = user_agent
         self._default_accept = default_accept
+        self._preflight_head = preflight_head
+        self._cache_ttl_seconds = cache_ttl_seconds
 
     @staticmethod
     def _cache_key(url: str) -> str:
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
         if not meta_path.exists():
             return {}
         try:
-            return json.loads(meta_path.read_text(encoding="utf-8"))
-        except (json.JSONDecodeError, OSError):
+            data = json.loads(meta_path.read_text(encoding="utf-8"))
+            return data if isinstance(data, dict) else {}
+        except json.JSONDecodeError, OSError:
             return {}
 
     @staticmethod
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
             return last_modified == str(meta.get("last_modified") or "").strip()
         return True
 
+    @staticmethod
+    def _response_info(response: httpx.Response) -> dict[str, Any]:
+        # httpx yields decoded bytes; encoded Content-Length cannot validate them.
+        raw_length = response.headers.get("content-length")
+        encoded = response.headers.get("content-encoding", "identity") != "identity"
+        return {
+            "etag": response.headers.get("etag"),
+            "last_modified": response.headers.get("last-modified"),
+            "content_length": int(raw_length) if raw_length and not encoded else None,
+            "accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
+        }
+
     async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
         try:
             response = await client.head(url)
             if response.status_code >= 400:
                 return {}
-            content_length_raw = response.headers.get("content-length")
-            content_length = int(content_length_raw) if content_length_raw else None
-            return {
-                "etag": response.headers.get("etag"),
-                "last_modified": response.headers.get("last-modified"),
-                "content_length": content_length,
-                "accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
-            }
-        except (httpx.HTTPError, ValueError):
+            return self._response_info(response)
+        except httpx.HTTPError, ValueError:
             return {}
 
     async def download_file(
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
     ) -> Path:
         self._cache_dir.mkdir(parents=True, exist_ok=True)
         final_path, part_path, meta_path = self._cache_paths(url, extension)
+        cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
+        if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
+            size = cached_path.stat().st_size
+            if progress_callback:
+                await progress_callback(size, size)
+            return cached_path
         meta = self._load_meta(meta_path)
-        remote = await self.fetch_remote_info(client, url)
+        remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
         expected_size = remote.get("content_length")
 
         if final_path.exists():
             local_size = final_path.stat().st_size
             size_match = expected_size is None or local_size == expected_size
-            if self._validators_match(meta, remote) and size_match:
+            if (
+                (remote.get("etag") or remote.get("last_modified"))
+                and self._validators_match(meta, remote)
+                and size_match
+            ):
                 if validate_existing and not validate_existing(final_path):
                     final_path.unlink(missing_ok=True)
                 else:
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
                 headers["If-Range"] = str(remote.get("last_modified"))
 
         async with client.stream("GET", url, headers=headers) as response:
-            if response.status_code == 304 and final_path.exists():
-                if progress_callback and expected_size and expected_size > 0:
-                    await progress_callback(expected_size, expected_size)
+            if response.status_code == 304:
+                if cached_path is None:
+                    raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
+                final_path.touch()
+                size = final_path.stat().st_size
+                if progress_callback:
+                    await progress_callback(size, size)
                 return final_path
             if response.status_code >= 400:
                 body = (await response.aread()).decode("utf-8", errors="replace")
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
             else:
                 mode = "wb"
                 resume_from = 0
+                remote = self._response_info(response)
+                expected_size = remote.get("content_length")
 
             downloaded = resume_from
             last_emit_bytes = 0
             last_emit_time = time.monotonic()
             min_emit_bytes = (
-                max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
+                max(expected_size // 150, 512 * 1024)
+                if expected_size and expected_size > 0
+                else 1024 * 1024
             )
 
             with part_path.open(mode) as f:
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
                 f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
             )
 
+        # Never replace the last good payload with an HTML error or invalid JSON.
+        if validate_existing and not validate_existing(part_path):
+            part_path.unlink(missing_ok=True)
+            raise RuntimeError(f"Downloaded file validation failed for {url}")
         part_path.replace(final_path)
         self._save_meta(
             meta_path,
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
             },
         )
 
-        if validate_existing and not validate_existing(final_path):
-            raise RuntimeError(f"Downloaded file validation failed for {url}")
-
         if progress_callback and expected_size and expected_size > 0:
             await progress_callback(expected_size, expected_size)
 

backend/app/services/collectors/epoch_ai.py

@@ -1,119 +1,135 @@
-"""Epoch AI GPU Clusters Collector
-
-Collects data from Epoch AI GPU clusters tracking.
-https://epoch.ai/data/gpu-clusters
-"""
+"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
 
+import csv
+import hashlib
+import io
+import json
+import math
 import re
-from typing import Dict, Any, List
-from datetime import UTC, datetime
-from bs4 import BeautifulSoup
+from urllib.parse import urlparse
+
 import httpx
 
-from app.services.collectors.base import BaseCollector
+from app.core.countries import normalize_country
+from app.services.collectors.compute_base import ComputeCollector
+from app.services.epoch_location import assess_epoch_location
+
+EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
 
 
-
-class EpochAIGPUCollector(BaseCollector):
+class EpochAIGPUCollector(ComputeCollector):
     name = "epoch_ai_gpu"
     priority = "P0"
     module = "L1"
     frequency_hours = 6
     data_type = "gpu_cluster"
 
-    async def fetch(self) -> List[Dict[str, Any]]:
-        """Fetch Epoch AI GPU clusters data from webpage"""
-        url = self._resolved_url or ""
-
-        async with httpx.AsyncClient(timeout=60.0) as client:
+    async def fetch(self) -> list[dict]:
+        configured = self._resolved_url or EPOCH_CSV_URL
+        parsed = urlparse(configured)
+        url = (
+            EPOCH_CSV_URL
+            if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
+            else configured
+        )
+        async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
             response = await client.get(url)
             response.raise_for_status()
-            return self.parse_response(response.text)
+        return self.parse_response(response.text)
 
-    def parse_response(self, html: str) -> List[Dict[str, Any]]:
-        """Parse Epoch AI webpage to extract GPU cluster data"""
-        data = []
-        soup = BeautifulSoup(html, "html.parser")
+    def parse_response(self, text: str) -> list[dict]:
+        reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
+        required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
+        if not required.issubset(reader.fieldnames or []):
+            raise ValueError(
+                "Epoch CSV schema changed or HTML returned; previous snapshot retained"
+            )
+        entries = []
+        seen = set()
+        anonymous_counts = {}
+        for row in reader:
+            name = (row.get("Name") or "").strip()
+            country = normalize_country(row.get("Country"))
+            if not name:
+                raise ValueError("Epoch row has missing identity")
+            source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
+            anonymous = "anonym" in name.lower()
+            if anonymous:
+                # Public anonymous rows have no facility identity. Preserve
+                # multiplicity within snapshots, without guessing names.
+                digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
+                anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
+                source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
+            if source_id in seen:
+                raise ValueError("Epoch identity collision; previous snapshot retained")
+            seen.add(source_id)
+            raw_date = row.get("First Operational Date", "")
+            reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
+            value = None
+            raw_value = row.get("16-bit OP/s (log)", "")
+            if raw_value:
+                exponent = float(raw_value)
+                if not math.isfinite(exponent) or not 0 <= exponent <= 30:
+                    raise ValueError("Epoch performance exponent out of bounds")
+                value = 10**exponent
+            anonymous = anonymous or (country == "中国" and not row.get("Location"))
+            location = row.get("Location")
+            location_assessment = assess_epoch_location(
+                location,
+                country=country,
+                raw_country=row.get("Country"),
+            )
+            entries.append(
+                {
+                    "source_id": source_id,
+                    "name": name,
+                    "country": country,
+                    "value": value,
+                    "unit": "FLOP/s",
+                    "reference_date": reference_date,
+                    "latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
+                    "longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
+                    "metadata": {
+                        "compute_contract": "v1",
+                        "precision": "FP16",
+                        "measurement": "theoretical",
+                        "source_url": EPOCH_CSV_URL,
+                        "raw_reference_date": raw_date,
+                        "status": row.get("Status"),
+                        "certainty": row.get("Certainty"),
+                        "operator": row.get("Owner"),
+                        "gpu_type": row.get("Chip type (primary)"),
+                        "gpu_count": row.get("Chip quantity (primary)"),
+                        "location": location,
+                        "location_precision": location_assessment.precision,
+                        "location_precision_reason": location_assessment.reason,
+                        "raw_country": row.get("Country"),
+                        "raw_performance": raw_value,
+                        "performance_encoding": "log10",
+                        "builds_upon": row.get("Builds Upon"),
+                        "superseded_by": row.get("Superseded by"),
+                        "possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
+                        "include_in_standard_analysis": row.get(
+                            "Include in Standard Analysis", ""
+                        ).lower()
+                        == "true",
+                        "rounded": country == "中国",
+                        "anonymous": anonymous,
+                        "identity_scope": "snapshot_record" if anonymous else "named_cluster",
+                        "country_unresolved": country is None,
+                        "sources": [
+                            row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
+                        ],
+                    },
+                }
+            )
+        return entries
 
-        # Try to find data table on the page
-        tables = soup.find_all("table")
-        for table in tables:
-            rows = table.find_all("tr")
-            for row in rows[1:]:  # Skip header
-                cells = row.find_all(["td", "th"])
-                if len(cells) >= 5:
-                    try:
-                        cluster_name = cells[0].get_text(strip=True)
-                        if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
-                            continue
-
-                        location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
-                        country, city = self._parse_location(location_cell)
-
-                        perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
-
-                        entry = {
-                            "source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
-                            "name": cluster_name,
-                            "country": country,
-                            "city": city,
-                            "latitude": "",
-                            "longitude": "",
-                            "value": self._parse_performance(perf_cell),
-                            "unit": "TFlop/s",
-                            "metadata": {
-                                "raw_data": perf_cell,
-                            },
-                            "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
-                        }
-                        data.append(entry)
-                    except (ValueError, IndexError, AttributeError):
-                        continue
-
-        # If no table found, return sample data
-        if not data:
-            data = self._get_sample_data()
-
-        return data
-
-    def _parse_location(self, location: str) -> tuple:
-        """Parse location string into country and city"""
-        if not location:
-            return "", ""
-        if "," in location:
-            parts = location.rsplit(",", 1)
-            city = parts[0].strip()
-            country = parts[1].strip() if len(parts) > 1 else ""
-            return country, city
-        return location, ""
-
-    def _parse_performance(self, perf: str) -> str:
-        """Parse performance string to extract value"""
-        if not perf:
-            return "0"
-        match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
-        if match:
-            return match.group(1).replace(",", "")
-        match = re.search(r"([\d,.]+)", perf)
-        if match:
-            return match.group(1).replace(",", "")
-        return "0"
-
-    def _get_sample_data(self) -> List[Dict[str, Any]]:
-        """Return sample data for testing when scraping fails"""
-        return [
-            {
-                "source_id": "epoch_sample_1",
-                "name": "Sample GPU Cluster",
-                "country": "United States",
-                "city": "San Francisco, CA",
-                "latitude": "",
-                "longitude": "",
-                "value": "1000",
-                "unit": "TFlop/s",
-                "metadata": {
-                    "note": "Sample data - Epoch AI page structure may vary",
-                },
-                "reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
-            },
-        ]
+    @staticmethod
+    def coordinate(value: str | None, bound: int) -> float | None:
+        if not value:
+            return None
+        number = float(value)
+        if not math.isfinite(number) or abs(number) > bound:
+            raise ValueError("Epoch coordinate out of bounds")
+        return number

backend/app/services/collectors/news_live_streams.py

@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
     DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
     DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
     DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
-    DEFAULT_IPTV_ORG_MAX_SOURCES = 120
+    DEFAULT_IPTV_ORG_MAX_SOURCES = 0  # Zero keeps the complete matching channel catalog.
 
     async def fetch(self) -> list[dict[str, Any]]:
         request_url = (self._resolved_url or "").strip()
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
                     "reference_date": datetime.now(UTC).isoformat(),
                 }
             )
-            if len(normalized) >= max_sources:
+            if max_sources > 0 and len(normalized) >= max_sources:
                 break
 
         return normalized

backend/app/services/collectors/top500.py

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
 """
 
 import asyncio
+import math
 import re
+from urllib.parse import urljoin
 from typing import Dict, Any, List
 from bs4 import BeautifulSoup
 import httpx
 
 from app.core.data_sources import get_data_sources_config
-from app.services.collectors.base import BaseCollector
+from app.services.collectors.compute_base import ComputeCollector
 
 
-class TOP500Collector(BaseCollector):
+class TOP500Collector(ComputeCollector):
     name = "top500"
     priority = "P0"
     module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
         async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
             response = await client.get(url)
             response.raise_for_status()
-            entries = self.parse_response(response.text)
+            entries = self.parse_response(response.text, str(response.url))
 
             semaphore = asyncio.Semaphore(8)
 
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
         system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
         detail_url = ""
         if link and link.get("href"):
-            detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
+            detail_url = urljoin(str(top500_base_url), link.get("href"))
 
         manufacturer = ""
         if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
 
         return detail_map
 
-    def parse_response(self, html: str) -> List[Dict[str, Any]]:
+    def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
         """Parse TOP500 HTML response"""
         data = []
         soup = BeautifulSoup(html, "html.parser")
+        canonical = soup.find("link", rel="canonical")
+        date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
+        date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
+        reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
 
         # Find the table with TOP500 data
         table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
             table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
 
         if table:
+            headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
+            if not reference_date:
+                raise ValueError("TOP500 list date missing; previous snapshot retained")
             rows = table.select("tr")
             for row in rows[1:]:  # Skip header row
                 cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
                         cores = cells[2].get_text(strip=True).replace(",", "")
 
                         rmax_text = cells[3].get_text(strip=True)
-                        rmax = self._parse_performance(rmax_text)
+                        rmax = self._parse_performance(rmax_text, headers[3])
 
                         rpeak_text = cells[4].get_text(strip=True)
-                        rpeak = self._parse_performance(rpeak_text)
+                        rpeak = self._parse_performance(rpeak_text, headers[4])
 
                         power = ""
                         if len(cells) >= 6:
                             power = cells[5].get_text(strip=True).replace(",", "")
 
+                        identity = re.search(r"/system/(\d+)", detail_url)
+                        if not identity:
+                            raise ValueError("TOP500 stable system ID missing")
                         entry = {
-                            "source_id": f"top500_{rank}",
+                            "source_id": f"top500_system_{identity[1]}",
                             "name": system_name,
                             "country": country,
                             "city": city,
-                            "latitude": 0.0,
-                            "longitude": 0.0,
+                            "latitude": None,
+                            "longitude": None,
                             "value": str(rmax),
-                            "unit": "PFlop/s",
+                            "unit": "FLOP/s",
                             "metadata": {
+                                "compute_contract": "v1",
+                                "precision": "FP64", "measurement": "LINPACK Rmax",
+                                "source_url": detail_url, "list_url": date_url,
+                                "status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
                                 "rank": rank,
                                 "cores": cores,
                                 "rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
                                 "site": site,
                             },
                             "_detail_url": detail_url,
-                            "reference_date": "2025-11-01",
+                            "reference_date": reference_date,
                         }
                         data.append(entry)
                     except (ValueError, IndexError, AttributeError) as e:
-                        continue
-
-        # If scraping failed, return sample data for testing
-        if not data:
-            data = self._get_sample_data()
+                        raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
 
         return data
 
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
                 return 0.0
         return 0.0
 
-    def _parse_performance(self, text: str) -> float:
+    def _parse_performance(self, text: str, header: str = "") -> float:
         """Parse performance value from text (handles E, P, T suffixes)"""
-        text = text.strip().upper()
+        text = text.strip().upper().replace(",", "")
+        if not re.search(r"[EPTGMK]F", text):
+            unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
+            if not unit:
+                raise ValueError("TOP500 performance unit missing")
+            text = f"{text} {unit[1]}FLOP/S"
         multipliers = {
             "E": 1e18,
             "P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
 
         # Try simple float parsing
         try:
-            return float(text.replace(",", ""))
+            value = float(text.replace(",", ""))
+            if not math.isfinite(value) or value < 0:
+                raise ValueError("Invalid TOP500 performance")
+            return value
         except ValueError:
-            return 0.0
-
-    def _get_sample_data(self) -> List[Dict[str, Any]]:
-        """Return sample data for testing when scraping fails"""
-        return [
-            {
-                "source_id": "top500_1",
-                "name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
-                "country": "United States",
-                "city": "Livermore, CA",
-                "latitude": 37.6819,
-                "longitude": -121.7681,
-                "value": "1742.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 1,
-                    "cores": "11039616",
-                    "rmax": "1742.00",
-                    "rpeak": "2746.38",
-                    "power": "29581",
-                    "manufacturer": "HPE",
-                    "site": "DOE/NNSA/LLNL",
-                    "processor": "AMD 4th Gen EPYC 24C 1.8GHz",
-                    "interconnect": "Slingshot-11",
-                    "installation_year": "2025",
-                },
-                "reference_date": "2025-11-01",
-            },
-            {
-                "source_id": "top500_2",
-                "name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
-                "country": "United States",
-                "city": "Oak Ridge, TN",
-                "latitude": 36.0107,
-                "longitude": -84.2663,
-                "value": "1353.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 2,
-                    "cores": "9066176",
-                    "rmax": "1353.00",
-                    "rpeak": "2055.72",
-                    "power": "24607",
-                    "manufacturer": "HPE",
-                    "site": "DOE/SC/Oak Ridge National Laboratory",
-                },
-                "reference_date": "2025-11-01",
-            },
-            {
-                "source_id": "top500_3",
-                "name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
-                "country": "United States",
-                "city": "Argonne, IL",
-                "latitude": 41.3784,
-                "longitude": -87.8600,
-                "value": "1012.00",
-                "unit": "PFlop/s",
-                "metadata": {
-                    "rank": 3,
-                    "cores": "9264128",
-                    "rmax": "1012.00",
-                    "rpeak": "1980.01",
-                    "power": "38698",
-                    "manufacturer": "Intel",
-                },
-                "reference_date": "2025-11-01",
-            },
-        ]
+            raise ValueError("Invalid TOP500 performance") from None

backend/app/services/compute_catalog.py

@@ -0,0 +1,200 @@
+"""Canonical country and mutually exclusive regional compute participants."""
+
+import re
+from collections.abc import Iterable
+
+from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
+from app.core.earth_geography_policy import display_country
+
+PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
+DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
+
+# Explicit membership prevents a regional total from double-counting a country.
+# APAC deliberately excludes China.
+COMPARISON_GROUPS = (
+    ("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
+    ("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
+    (
+        "EUROPE",
+        "欧洲",
+        "Europe",
+        (
+            "GB",
+            "DE",
+            "FR",
+            "CH",
+            "ES",
+            "FI",
+            "HU",
+            "IE",
+            "IT",
+            "LU",
+            "MD",
+            "NL",
+            "PL",
+            "RO",
+            "RU",
+            "UA",
+        ),
+        {"latitude": 54.0, "longitude": 20.0},
+    ),
+    (
+        "APAC",
+        "亚太（不含中国）",
+        "Asia Pacific (excluding China)",
+        ("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
+        {"latitude": 36.0, "longitude": 130.5},
+    ),
+    (
+        "OTHER",
+        "其他地区",
+        "Other",
+        (
+            "CA",
+            "AO",
+            "EG",
+            "KE",
+            "MZ",
+            "NG",
+            "UG",
+            "ZA",
+            "BR",
+            "CO",
+            "CR",
+            "DO",
+            "MX",
+            "AE",
+            "SY",
+            "TR",
+        ),
+        {"latitude": 12.0, "longitude": 10.0},
+    ),
+)
+
+
+def _country_code(canonical: str, aliases: list[str]) -> str | None:
+    code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
+    return "GB" if canonical == "英国" else code
+
+
+def build_country_catalog() -> list[dict]:
+    """Build the policy-projected canonical country drilldown catalog."""
+    countries = []
+    for canonical, aliases in COUNTRY_ENTRIES:
+        if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
+            continue
+        code = _country_code(canonical, aliases)
+        if code:
+            countries.append(
+                {
+                    "code": code,
+                    "name": canonical,
+                    "name_en": aliases[0],
+                    "kind": "country",
+                    "members": [code],
+                    "anchor": dict(COUNTRY_CENTROIDS[canonical]),
+                    "anchor_kind": "regional_label",
+                }
+            )
+    return countries
+
+
+COUNTRY_CATALOG = build_country_catalog()
+COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
+COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
+
+
+def _validate_group_members() -> None:
+    members = [
+        member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
+    ]
+    if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
+        raise ValueError("compute comparison groups must cover country catalog exactly once")
+
+
+_validate_group_members()
+GROUP_BY_MEMBER = {
+    member: code
+    for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
+    for member in members
+}
+
+
+def build_catalog() -> list[dict]:
+    """Return the five default participants before country drilldown entries."""
+    catalog: list[dict] = []
+    for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
+        entry = (
+            dict(COUNTRY_PARTICIPANTS[code])
+            if code in COUNTRY_PARTICIPANTS
+            else {
+                "code": code,
+                "name": name,
+                "name_en": name_en,
+                "kind": "region",
+                "members": list(members),
+                "anchor": dict(anchor),
+                "anchor_kind": "regional_label",
+            }
+        )
+        entry.update(
+            {
+                "members": list(members),
+                "parent_group": name,
+                "parent_group_code": code,
+                "order": index,
+                "default_selected": True,
+                "color": PALETTE[index % len(PALETTE)],
+            }
+        )
+        catalog.append(entry)
+
+    for entry in COUNTRY_CATALOG:
+        if entry["code"] in DEFAULT_PARTICIPANTS:
+            continue
+        country = dict(entry)
+        group_code = GROUP_BY_MEMBER[country["code"]]
+        group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
+        country.update(
+            {
+                "parent_group": group_name,
+                "parent_group_code": group_code,
+                "order": len(catalog),
+                "default_selected": False,
+                "color": PALETTE[len(catalog) % len(PALETTE)],
+            }
+        )
+        catalog.append(country)
+    return catalog
+
+
+CATALOG = build_catalog()
+PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
+
+
+def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
+    """Resolve country or region participant codes to canonical country codes."""
+    return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
+
+
+def source_country_names(codes: Iterable[str]) -> list[str]:
+    """Return unique canonical source labels, respecting China and Serbia ownership."""
+    selected_members = {
+        member for members in participant_members(codes).values() for member in members
+    }
+    selected_names = {COUNTRIES[code] for code in selected_members}
+    return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
+
+
+def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
+    """Map stored source-country labels to their requested participant code."""
+    owner_by_country = {
+        member: participant
+        for participant, members in participant_members(codes).items()
+        for member in members
+    }
+    code_by_name = {name: code for code, name in COUNTRIES.items()}
+    return {
+        name: owner_by_country[code_by_name[display_country(name)]]
+        for name in source_country_names(codes)
+    }

backend/app/services/compute_center_locations.py

@@ -3,7 +3,12 @@
 This module is a thin domain wrapper that wires up
 :mod:`app.services.location` for compute centers:
 
-    SourceCoordinates
+    Confirmed stored location -> trustworthy source coordinates -> exact
+    official facility registration -> source city/site coordinates.
+
+Epoch AI coordinates are admitted only at the grain asserted by its official
+``Location`` column. Country/region/empty locations remain unresolved even
+when the CSV happens to contain a latitude/longitude pair.
 
 The online Nominatim step is intentionally reserved for the user-triggered
 ``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -28,9 +33,12 @@ import httpx
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
+from app.core.compute_types import compute_record_filter
 from app.core.collected_data_fields import get_record_field
 from app.models.collected_data import CollectedData
 from app.models.compute_center_location import ComputeCenterLocationRecord
+from app.services.compute_center_verified_sites import get_verified_compute_site
+from app.services.epoch_location import assess_epoch_location
 
 from app.services.location import (
     LocationCandidate,
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
 
 async def refresh_compute_center_location_cache(
     session: AsyncSession,
+    *,
+    as_of: datetime | None = None,
 ) -> dict[str, dict[str, Any]]:
-    result = await session.execute(select(ComputeCenterLocationRecord))
+    query = select(ComputeCenterLocationRecord)
+    if as_of is not None:
+        query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
+    result = await session.execute(query)
     records = result.scalars().all()
     cache = {}
     for record in records:
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
         )
 
 
+class ComputeCenterSourceCoordinatesResolver:
+    """Use source coordinates only at the grain asserted by the source text."""
+
+    name = "source_coordinates"
+
+    def __init__(self) -> None:
+        self._default = SourceCoordinatesResolver()
+
+    def resolve(self, query: LocationQuery) -> ResolverOutput:
+        extra = query.extra or {}
+        if extra.get("source") != "epoch_ai_gpu":
+            return self._default.resolve(query)
+        if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
+            return ResolverOutput()
+        assessment = assess_epoch_location(
+            extra.get("source_location"),
+            country=query.country,
+            raw_country=extra.get("source_raw_country"),
+        )
+        if not assessment.is_renderable:
+            return ResolverOutput(
+                attempted_queries=(f"source_coordinates:{assessment.precision}",)
+            )
+        output = self._default.resolve(query)
+        if not output.candidates:
+            return output
+        candidate = output.candidates[0]
+        source_location = extra.get("source_location")
+        source_location_text = (
+            source_location.strip() if isinstance(source_location, str) else ""
+        )
+        source_note = assessment.reason
+        if source_location_text:
+            source_note = f"{source_note}; Epoch Location: {source_location_text}"
+        return ResolverOutput(
+            candidates=(
+                LocationCandidate(
+                    **{
+                        **candidate.__dict__,
+                        "precision": assessment.precision,
+                        "confidence": 1.0 if assessment.precision == "precise" else 0.9,
+                        "source_note": source_note,
+                        "needs_confirmation": assessment.precision != "precise",
+                        "matched_location_name": source_location_text
+                        or candidate.matched_location_name,
+                    }
+                ),
+            ),
+            attempted_queries=output.attempted_queries,
+        )
+
+
 class StoredComputeCenterLocationResolver:
     """Resolve a compute center through the DB-backed current-location cache."""
 
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
         longitude = parse_float(stored.get("longitude"))
         if latitude in (None, 0.0) or longitude in (None, 0.0):
             return ResolverOutput()
+        precision = stored.get("precision") or "city"
+        if (
+            extra.get("source") == "epoch_ai_gpu"
+            and stored.get("location_source") == "source_coordinates"
+        ):
+            assessment = assess_epoch_location(
+                extra.get("source_location"),
+                country=query.country,
+                raw_country=extra.get("source_raw_country"),
+            )
+            if not assessment.is_renderable:
+                return ResolverOutput()
+            precision = assessment.precision
         return ResolverOutput(
             candidates=(
                 LocationCandidate(
                     latitude=latitude,
                     longitude=longitude,
                     display_name=stored.get("name") or query.name or "Compute center",
-                    precision=stored.get("precision") or "city",
+                    precision=precision,
                     confidence=float(stored.get("confidence") or 0.85),
                     query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
                     source=self.name,
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
                     matched_location_name=stored.get("site") or stored.get("name") or query.name,
                     location_verified_at=stored.get("verified_at"),
                     suggested_registry_entry=None,
+                    raw_payload={
+                        "stored_location_source": stored.get("location_source"),
+                        "verification_status": stored.get("verification_status"),
+                    },
+                ),
+            )
+        )
+
+
+class VerifiedComputeSiteResolver:
+    """Resolve an exact source identity to an evidence-backed facility site."""
+
+    name = "official_facility_registry"
+
+    def resolve(self, query: LocationQuery) -> ResolverOutput:
+        extra = query.extra or {}
+        source = coerce_str(extra.get("source"))
+        source_id = coerce_str(extra.get("source_id"))
+        if not source or not source_id:
+            return ResolverOutput()
+        site = get_verified_compute_site(source, source_id)
+        if not site:
+            return ResolverOutput()
+        source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
+        source_note = coerce_str(site.get("source_note"))
+        if source_urls:
+            source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
+        return ResolverOutput(
+            candidates=(
+                LocationCandidate(
+                    latitude=float(site["latitude"]),
+                    longitude=float(site["longitude"]),
+                    display_name=coerce_str(site.get("matched_location_name"))
+                    or query.name
+                    or "Compute center",
+                    precision=coerce_str(site.get("precision")) or "site",
+                    confidence=1.0,
+                    query=f"official_facility_registry::{source}:{source_id}",
+                    source=self.name,
+                    source_note=source_note or None,
+                    matched_fields=("source", "source_id"),
+                    needs_confirmation=False,
+                    city=coerce_str(site.get("city")) or None,
+                    region=None,
+                    country=coerce_str(site.get("country")) or query.country,
+                    matched_location_name=coerce_str(site.get("matched_location_name"))
+                    or None,
+                    location_verified_at=coerce_str(site.get("verified_at")) or None,
+                    suggested_registry_entry=None,
+                    raw_payload={
+                        "address": site.get("address"),
+                        "coordinate_basis": site.get("coordinate_basis"),
+                        "sources": source_urls,
+                    },
                 ),
             )
         )
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
     return head or text
 
 
-def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
+def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
     name = coerce_str(getattr(record, "name", None))
     return {
         "source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
             or metadata.get("manufacturer")
         ),
         "organization": coerce_str(metadata.get("organization")),
+        "source_location": metadata.get("location"),
+        "source_raw_country": metadata.get("raw_country"),
     }
 
 
 def _context_to_query(
-    context: dict[str, str],
+    context: dict[str, Any],
     *,
     source_lat: float | None = None,
     source_lon: float | None = None,
@@ -444,6 +578,8 @@ def _context_to_query(
             "site": context.get("site") or "",
             "operator": context.get("operator") or "",
             "organization": context.get("organization") or "",
+            "source_location": context.get("source_location"),
+            "source_raw_country": context.get("source_raw_country"),
         },
     )
 
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
     return plan
 
 
+def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
+    stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
+    stored_confirmed = (
+        candidate.source == "stored_compute_center_location"
+        and not candidate.needs_confirmation
+        and stored_origin != "source_coordinates"
+    )
+    if stored_confirmed:
+        source_rank = 0
+    elif candidate.source == "source_coordinates" and candidate.precision == "precise":
+        source_rank = 1
+    elif candidate.source == "official_facility_registry":
+        source_rank = 2
+    elif candidate.source == "source_coordinates":
+        source_rank = 3
+    elif candidate.source == "stored_compute_center_location":
+        source_rank = 4
+    else:
+        source_rank = 5
+    precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
+    return (source_rank, precision_rank, -float(candidate.confidence or 0))
+
+
 COMPUTE_CENTER_PIPELINE = LocationPipeline(
     [
-        SourceCoordinatesResolver(),
+        ComputeCenterSourceCoordinatesResolver(),
         StoredComputeCenterLocationResolver(),
+        VerifiedComputeSiteResolver(),
     ],
+    sort_key=_compute_center_candidate_sort_key,
     failure_reason=(
         "Could not resolve to city-level coordinates from source coords"
         " or stored compute-center location."
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
 
 COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
     [
-        SourceCoordinatesResolver(),
+        ComputeCenterSourceCoordinatesResolver(),
+        VerifiedComputeSiteResolver(),
         ROROrganizationResolver(),
         NominatimResolver(
             query_plan_builder=_compute_center_query_plan,
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
             geocoder=lambda q: _geocode_online(q),
         ),
     ],
+    sort_key=_compute_center_candidate_sort_key,
     failure_reason=(
         "Could not resolve to city-level coordinates from source coords"
         ", ROR organization lookup, or online geocoding."
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
 _GEOGRAPHY_MODE_BY_SOURCE = {
     "source_coordinates": "source_coordinates",
     "stored_compute_center_location": "stored_compute_center_location",
+    "official_facility_registry": "official_facility_registry",
     "ror_organization_registry": "ror_organization",
     "nominatim_online_geocode": "online_geocode",
 }
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
 def _candidate_to_location(
     candidate: LocationCandidate,
     *,
-    context: dict[str, str],
+    context: dict[str, Any],
 ) -> ComputeCenterLocation:
     geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
     is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +758,7 @@ def _candidate_to_location(
 
 def _diagnostic_for(
     record: Any,
-    context: dict[str, str],
+    context: dict[str, Any],
     *,
     failure_reason: str,
     attempted_queries: tuple[str, ...] = (),
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
     organization: str | None = None,
 ) -> LocationQuery:
     name_value = coerce_str(name)
-    context: dict[str, str] = {
+    context: dict[str, Any] = {
         "source": coerce_str(source),
         "source_id": coerce_str(source_id),
         "name": name_value,
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
     """Seed stored compute-center locations only from real source coordinates."""
     stmt = (
         select(CollectedData)
-        .where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
+        .where(compute_record_filter(CollectedData))
         .where(CollectedData.is_current.is_(True))
     )
     result = await session.execute(stmt)
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
         longitude = parse_float(get_record_field(record, "longitude"))
         if latitude in (None, 0.0) or longitude in (None, 0.0):
             continue
+        metadata = record.extra_data or {}
+        precision = "precise"
+        confidence = 1.0
+        needs_confirmation = False
+        source_note = "Seeded from source-provided compute-center coordinates"
+        if source_value == "epoch_ai_gpu":
+            assessment = assess_epoch_location(
+                metadata.get("location"),
+                country=get_record_field(record, "country"),
+                raw_country=metadata.get("raw_country"),
+            )
+            if not assessment.is_renderable:
+                continue
+            precision = assessment.precision
+            confidence = 1.0 if precision == "precise" else 0.9
+            needs_confirmation = precision != "precise"
+            source_note = assessment.reason
         existing = await session.scalar(
             select(ComputeCenterLocationRecord)
             .where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
         )
         if existing:
             continue
-        metadata = record.extra_data or {}
         session.add(
             ComputeCenterLocationRecord(
                 source=source_value,
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
                 country=coerce_str(get_record_field(record, "country")) or None,
                 latitude=latitude,
                 longitude=longitude,
-                precision="precise",
-                confidence=1.0,
+                precision=precision,
+                confidence=confidence,
                 location_source="source_coordinates",
-                source_note="Seeded from source-provided compute-center coordinates",
+                source_note=source_note,
                 raw_payload={
                     "record_id": getattr(record, "id", None),
                     "source": source_value,
                     "source_id": source_id,
                 },
-                needs_confirmation=False,
+                needs_confirmation=needs_confirmation,
                 verification_status="source_provided",
                 verified_at=None,
             )

backend/app/services/compute_center_verified_sites.json

@@ -0,0 +1,44 @@
+{
+  "version": 1,
+  "sites": [
+    {
+      "source": "epoch_ai_gpu",
+      "source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
+      "latitude": 35.380924,
+      "longitude": 139.625008,
+      "precision": "site",
+      "location_source": "official_facility_registry",
+      "country": "日本",
+      "city": "Yokohama",
+      "matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
+      "address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
+      "source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
+      "verified_at": "2026-09-21T17:06:31Z",
+      "coordinate_basis": "official_access_map_center",
+      "sources": [
+        "https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
+        "https://www.jamstec.go.jp/e/about/access/yokohama.html"
+      ]
+    },
+    {
+      "source": "epoch_ai_gpu",
+      "source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
+      "latitude": 41.718109383287235,
+      "longitude": -87.984498648496,
+      "precision": "site",
+      "location_source": "official_facility_registry",
+      "country": "美国",
+      "city": "Argonne",
+      "matched_location_name": "Argonne Leadership Computing Facility",
+      "address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
+      "source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
+      "verified_at": "2026-09-21T17:06:31Z",
+      "coordinate_basis": "official_access_map_center",
+      "sources": [
+        "https://www.alcf.anl.gov/systems/sophia",
+        "https://docs.alcf.anl.gov/sophia/",
+        "https://www.alcf.anl.gov/about/visiting-alcf"
+      ]
+    }
+  ]
+}

backend/app/services/compute_center_verified_sites.py

@@ -0,0 +1,26 @@
+"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
+
+import json
+from copy import deepcopy
+from functools import lru_cache
+from pathlib import Path
+
+
+@lru_cache(maxsize=1)
+def _site_index() -> dict[tuple[str, str], dict]:
+    payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
+    index = {}
+    for site in payload["sites"]:
+        if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
+            raise ValueError("Invalid verified compute facility coordinate")
+        for source_id in site["source_ids"]:
+            key = (site["source"], source_id)
+            if key in index:
+                raise ValueError("Duplicate verified compute facility identity")
+            index[key] = site
+    return index
+
+
+def get_verified_compute_site(source: str, source_id: str) -> dict | None:
+    site = _site_index().get((source, source_id))
+    return deepcopy(site) if site else None

backend/app/services/compute_configuration.py

@@ -0,0 +1,99 @@
+"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
+
+import hashlib
+from pathlib import Path
+from uuid import uuid4
+
+from app.core.request_context import get_request_id
+from app.models.system_log import AuditLog
+from app.models.system_setting import SystemSetting
+from app.models.user import User
+from app.schemas.compute_configuration import ComputeConfiguration
+from app.services.compute_catalog import COUNTRY_CATALOG
+from fastapi import HTTPException
+from sqlalchemy import select
+from sqlalchemy.dialects.postgresql import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+CATEGORY = "compute_comparison"
+DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
+DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
+DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
+
+
+def default_envelope() -> dict:
+    return {
+        "config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
+        "revision": DEFAULT_REVISION,
+        "is_default": True,
+    }
+
+
+async def read_configuration(db: AsyncSession) -> dict:
+    record = (
+        await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
+    ).scalar_one_or_none()
+    return dict(record.payload) if record else default_envelope()
+
+
+def configuration_response(envelope: dict) -> dict:
+    return {
+        **envelope,
+        "countries": [
+            {key: country[key] for key in ("code", "name", "name_en")}
+            for country in COUNTRY_CATALOG
+        ],
+    }
+
+
+async def publish_configuration(
+    db: AsyncSession,
+    *,
+    revision: str,
+    config: ComputeConfiguration,
+    actor: User,
+    reset: bool = False,
+) -> dict:
+    # The unique category plus row lock also serializes concurrent first writes.
+    await db.execute(
+        insert(SystemSetting)
+        .values(category=CATEGORY, payload=default_envelope())
+        .on_conflict_do_nothing(index_elements=[SystemSetting.category])
+    )
+    record = (
+        await db.execute(
+            select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
+        )
+    ).scalar_one()
+    if record.payload["revision"] != revision:
+        await db.rollback()
+        raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
+    envelope = (
+        default_envelope()
+        if reset
+        else {
+            "config": config.model_dump(mode="json"),
+            "revision": uuid4().hex,
+            "is_default": reset,
+        }
+    )
+    before = record.payload
+    if reset:
+        await db.delete(record)
+    else:
+        record.payload = envelope
+    # Audit and configuration commit together: a published edit must be attributable.
+    db.add(
+        AuditLog(
+            actor_id=actor.id,
+            actor_name=actor.username,
+            action="compute_comparison.reset" if reset else "compute_comparison.update",
+            target_type="system_setting",
+            target_id=CATEGORY,
+            result="success",
+            request_id=get_request_id(),
+            details={"before": before, "after": envelope},
+        )
+    )
+    await db.commit()
+    return envelope

backend/app/services/compute_history.py

@@ -0,0 +1,122 @@
+"""Versioned, source-backed annual observations for compute comparisons."""
+
+from copy import deepcopy
+from datetime import date
+
+from app.schemas.compute_configuration import ComputeConfiguration
+from app.services.compute_catalog import (
+    CATALOG,
+    DEFAULT_PARTICIPANTS,
+    PARTICIPANTS,
+    participant_members,
+)
+from app.services.compute_configuration import DEFAULT_CONFIGURATION
+
+
+def _participant_value(
+    members: tuple[str, ...], observations: dict[str, float | int], mode: str
+) -> tuple[float | int | None, dict]:
+    covered = [member for member in members if member in observations]
+    missing = [member for member in members if member not in observations]
+    coverage = {
+        "status": "complete" if not missing else "incomplete",
+        "required_members": list(members),
+        "covered_members": covered,
+        "missing_members": missing,
+    }
+    if missing:
+        return None, coverage
+    if mode == "bilateral_share":
+        if (
+            not set(members).issubset({"CN", "US"})
+            or not {"CN", "US"}.issubset(observations)
+            or observations["US"] + observations["CN"] <= 0
+        ):
+            coverage.update(
+                {
+                    "status": "incompatible",
+                    "reason": "bilateral_share_requires_CN_and_US_observations",
+                }
+            )
+            return None, coverage
+        return (
+            round(
+                sum(observations[member] for member in members)
+                / (observations["US"] + observations["CN"])
+                * 100,
+                2,
+            ),
+            coverage,
+        )
+    return sum(observations[member] for member in members), coverage
+
+
+def compute_history(
+    countries: list[str],
+    mode: str = "global_share",
+    config: ComputeConfiguration | None = None,
+) -> dict:
+    """Aggregate only fully covered, same-year compatible observations."""
+    config = config if config is not None else DEFAULT_CONFIGURATION
+    metric = "global_share" if mode == "bilateral_share" else mode
+    annual = {row.year: row for row in config.observations if row.metric == metric}
+    gaps = {row.year: row for row in config.innovation}
+    observed_years = [row.year for row in config.observations] + list(gaps)
+    years = tuple(
+        range(
+            min(observed_years, default=date.today().year),
+            max([date.today().year, *observed_years]) + 1,
+        )
+    )
+    members_by_participant = participant_members(countries)
+    rows = []
+    for year in years:
+        record = annual.get(year)
+        observations = record.values if record else {}
+        values, coverage = {}, {}
+        for code, members in members_by_participant.items():
+            values[code], coverage[code] = _participant_value(members, observations, mode)
+        evidence_id = record.source_id if record else None
+        rows.append(
+            {"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
+        )
+
+    selected_members = {member for members in members_by_participant.values() for member in members}
+    has_innovation_scope = {"CN", "US"}.issubset(selected_members)
+    return {
+        "version": f"compute-history-{config.reviewed_at}",
+        "provenance": "curated_reports",
+        "reviewed_at": config.reviewed_at.isoformat(),
+        "time_basis": "retrospective",
+        "years": years,
+        "mode": mode,
+        "unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
+        "denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
+            mode
+        ],
+        "catalog": deepcopy(CATALOG),
+        "default_participants": list(DEFAULT_PARTICIPANTS),
+        "countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
+        "compute": rows,
+        "innovation": [
+            {
+                "year": year,
+                "value": gaps[year].value if year in gaps and has_innovation_scope else None,
+                "evidence_id": gaps[year].source_id
+                if year in gaps and has_innovation_scope
+                else None,
+            }
+            for year in years
+        ],
+        "evidence": {
+            source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
+        },
+        "innovation_scope": ["US", "CN"],
+        "limitations": [
+            "公开设施样本不代表全国总算力；FP64、FP16 与 FP32 不可直接相加。",
+            "区域值只在全部成员有同年、同指标、可比较观测时汇总；缺失成员时返回 null。",
+            "全球份额不从中美或其他局部成员反推剩余地区。",
+            "年度资料按已配置报告引用展示，观测年与报告年份分列；跨年方法可比性仍需核验。",
+            "创新分差为美国减中国；只有分差，不反推单国得分，不证明算力与创新的因果关系。",
+        ],
+    }

backend/app/services/compute_history_defaults.json

@@ -0,0 +1,125 @@
+{
+  "reviewed_at": "2026-09-22",
+  "sources": [
+    {
+      "id": "caict-2021",
+      "title": "中国算力发展指数白皮书（2021）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2021,
+      "published_at": null,
+      "locator": "正文 11 页图 4（2020 年全球份额）"
+    },
+    {
+      "id": "caict-2022",
+      "title": "中国算力发展指数白皮书（2022）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2022,
+      "published_at": null,
+      "locator": "正文 13 页图 5（2021 年全球份额）"
+    },
+    {
+      "id": "caict-2023",
+      "title": "中国算力发展指数白皮书（2023）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2023,
+      "published_at": null,
+      "locator": "正文 13 页图 5（2022 年全球份额）；16 页（中国总规模）"
+    },
+    {
+      "id": "caict-2024",
+      "title": "先进计算暨算力发展指数蓝皮书（2024年）",
+      "url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
+      "publisher": "中国信息通信研究院",
+      "report_year": 2024,
+      "published_at": null,
+      "locator": "正文 10 页图 3（2023 年全球份额）；12 页（中国总规模）"
+    },
+    {
+      "id": "innovation-2026",
+      "title": "全球人工智能创新指数报告 2026：科技日报报道",
+      "url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
+      "publisher": "科技日报（新浪转载）",
+      "report_year": 2026,
+      "published_at": "2026-07-17",
+      "locator": "2023–2025 年美国减中国的指数分差"
+    }
+  ],
+  "observations": [
+    {
+      "year": 2020,
+      "metric": "global_share",
+      "source_id": "caict-2021",
+      "values": {
+        "US": 36,
+        "CN": 31,
+        "JP": 6
+      }
+    },
+    {
+      "year": 2021,
+      "metric": "global_share",
+      "source_id": "caict-2022",
+      "values": {
+        "US": 34,
+        "CN": 33,
+        "JP": 5
+      }
+    },
+    {
+      "year": 2022,
+      "metric": "global_share",
+      "source_id": "caict-2023",
+      "values": {
+        "US": 34,
+        "CN": 33,
+        "JP": 4
+      }
+    },
+    {
+      "year": 2023,
+      "metric": "global_share",
+      "source_id": "caict-2024",
+      "values": {
+        "US": 41,
+        "CN": 31,
+        "JP": 4
+      }
+    },
+    {
+      "year": 2022,
+      "metric": "absolute",
+      "source_id": "caict-2023",
+      "values": {
+        "CN": 302
+      }
+    },
+    {
+      "year": 2023,
+      "metric": "absolute",
+      "source_id": "caict-2024",
+      "values": {
+        "CN": 435
+      }
+    }
+  ],
+  "innovation": [
+    {
+      "year": 2023,
+      "value": 22.02,
+      "source_id": "innovation-2026"
+    },
+    {
+      "year": 2024,
+      "value": 19.96,
+      "source_id": "innovation-2026"
+    },
+    {
+      "year": 2025,
+      "value": 17.95,
+      "source_id": "innovation-2026"
+    }
+  ]
+}

backend/app/services/earth_boundaries.py

@@ -6,8 +6,10 @@ import asyncio
 import hashlib
 import json
 import shutil
+import subprocess
 import sys
 from datetime import UTC, datetime
+from functools import lru_cache
 from pathlib import Path
 from typing import Any
 from uuid import uuid4
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
 PMTILES_ARTIFACT_PATH = (
     REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
 )
-LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
+POLICY_FALLBACK_PATH = (
+    REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
+)
 POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
+POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
 LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
 EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
 
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
     return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
 
 
+@lru_cache(maxsize=16)
+def _read_versioned_asset_cached(
+    path: Path, mtime_ns: int, size_bytes: int
+) -> tuple[dict[str, Any], str]:
+    payload = _read_json(path)
+    return payload, _stable_json_hash(payload)
+
+
+def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
+    if not path.exists():
+        return {}, ""
+    stat = path.stat()
+    return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
+
+
 def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
     suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
     if suffix in {".json", ".geojson", ".zip", ".pbf"}:
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
     return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
 
 
-def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
-    path = REPO_ROOT / source["path"]
-    payload = _read_json(path)
-    features = payload.get("features") if isinstance(payload, dict) else None
-    return {
-        "type": "FeatureCollection",
-        "features": features if isinstance(features, list) else [],
-    }
-
-
 def _write_high_precision_geojson_manifest(
     sources: list[dict[str, Any]],
     build_input_hash: str,
     missing_tools: list[str],
 ) -> dict[str, Any]:
-    BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
     admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
     coastline = next(source for source in sources if source["kind"] == "coastline")
     claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
-
-    admin0_payload = _load_source_feature_collection(admin0)
-    coastline_payload = _load_source_feature_collection(coastline)
-    claim_payload = _load_source_feature_collection(claim_lines)
-    for feature in coastline_payload["features"]:
-        props = feature.setdefault("properties", {})
-        if isinstance(props, dict):
-            props["PLANET_LAYER"] = "coastline"
-
-    base_payload = {
-        "type": "FeatureCollection",
-        "features": [*admin0_payload["features"], *coastline_payload["features"]],
-    }
-    base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
-    hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
-    claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
-    _write_json(base_path, base_payload)
-    _write_json(hover_path, admin0_payload)
-    _write_json(claim_path, claim_payload)
-
-    manifest = {
-        "version": "natural-earth-v1",
-        "builtAt": _utc_now_iso(),
-        "tileProvider": "geojson-high-precision",
-        "format": "geojson-directory",
-        "buildInputHash": build_input_hash,
-        "base": base_path.name,
-        "hoverIndex": hover_path.name,
-        "claimLine": claim_path.name,
-        "sourceFeatureCount": {
-            "admin0": len(admin0_payload["features"]),
-            "coastline": len(coastline_payload["features"]),
-            "claimLines": len(claim_payload["features"]),
-        },
-        "pmtiles": None,
-        "missingTools": missing_tools,
-    }
+    result = subprocess.run(
+        [
+            sys.executable,
+            str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
+            "--admin0-source", admin0["path"],
+            "--coastline-source", coastline["path"],
+            "--claims-source", claim_lines["path"],
+            "--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
+            "--build-input-hash", build_input_hash,
+            "--pov-policy", _relative(POV_POLICY_PATH),
+            "--pov-geometry", str(POV_GEOMETRY_PATH),
+        ],
+        cwd=REPO_ROOT,
+        capture_output=True,
+        text=True,
+    )
+    if result.returncode:
+        raise EarthBoundaryBuildError(
+            result.stderr.strip() or "Earth boundary policy build failed",
+            code="policy_build_failed",
+        )
+    manifest = _read_json(BOUNDARY_MANIFEST_PATH)
+    manifest["missingTools"] = missing_tools
     _write_json(BOUNDARY_MANIFEST_PATH, manifest)
     return manifest
 
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
                 for source in source_manifest.get("sources", [])
             ],
             "pov_policy": source_manifest.get("povPolicy"),
+            "pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
             "build_config": BUILD_CONFIG,
         }
     )
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
         bool(boundary_manifest)
         and boundary_manifest.get("buildInputHash") == build_input_hash
         and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
+        and _policy_verified(boundary_manifest)
         and PMTILES_ARTIFACT_PATH.exists()
+        and _pmtiles_hover_exists(boundary_manifest)
+    )
+
+
+def _policy_verified(manifest: dict[str, Any]) -> bool:
+    policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
+    geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
+    coverage = manifest.get("povPolicy") or {}
+    return (
+        policy.get("profile") == "china-pov-v1"
+        and coverage.get("profile") == policy["profile"]
+        and coverage.get("sha256") == policy_hash
+        and coverage.get("reviewedGeometrySourceCommit")
+        == geometry.get("sourceCommit")
+        and coverage.get("geometrySha256") == geometry_hash
+        and coverage.get("applied") is True
+        and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
+        <= set(coverage.get("appliedRuleIds") or [])
+    )
+
+
+def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
+    paths = [manifest.get("base"), manifest.get("hoverIndex")]
+    if manifest.get("chinaClaims", {}).get("available"):
+        paths.append(manifest.get("claimLine"))
+    for asset in paths:
+        if not isinstance(asset, str) or Path(asset).name != asset:
+            return False
+        if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
+            return False
+    return True
+
+
+def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
+    hover = manifest.get("hoverIndex")
+    return (
+        isinstance(hover, str)
+        and Path(hover).name == hover
+        and (BOUNDARY_OUTPUT_DIR / hover).is_file()
     )
 
 
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
     manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
     high_precision_ready = (
         manifest_exists
+        and _policy_verified(boundary_manifest)
         and (
             (
                 boundary_manifest.get("tileProvider") == "pmtiles-mvt"
                 and pmtiles_exists
+                and _pmtiles_hover_exists(boundary_manifest)
+            )
+            or (
+                boundary_manifest.get("tileProvider") == "geojson-high-precision"
+                and _geojson_artifacts_exist(boundary_manifest)
             )
-            or boundary_manifest.get("tileProvider") == "geojson-high-precision"
         )
     )
-    legacy_exists = LEGACY_GEOJSON_PATH.exists()
+    legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
+        _versioned_asset(POLICY_FALLBACK_PATH)[0]
+    )
     provider = (
         boundary_manifest.get("tileProvider")
         if high_precision_ready
-        else "legacy-geojson" if legacy_exists else "missing"
+        else "policy-geojson-fallback" if legacy_exists else "missing"
     )
     return {
         "provider": provider,
         "high_precision_ready": high_precision_ready,
         "fallback_available": legacy_exists,
+        "policy_publishable": bool(
+            high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
+        ),
         "config_source": config_source,
         "config_path": _relative(LOCAL_CONFIG_PATH),
         "config_exists": LOCAL_CONFIG_PATH.exists(),
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
             "tileProvider": boundary_manifest.get("tileProvider"),
             "buildInputHash": boundary_manifest.get("buildInputHash"),
             "builtAt": boundary_manifest.get("builtAt"),
+            "povPolicy": boundary_manifest.get("povPolicy"),
+            "publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
         },
         "pmtiles": {
             "path": _relative(PMTILES_ARTIFACT_PATH),
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
             "size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
         },
         "legacy": {
-            "path": _relative(LEGACY_GEOJSON_PATH),
+            "path": _relative(POLICY_FALLBACK_PATH),
             "exists": legacy_exists,
-            "size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
+            "size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
         },
         "source_manifest": {
             "path": _relative(SOURCE_MANIFEST_PATH),
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
                 build_input_hash,
                 "--pov-policy",
                 _relative(POV_POLICY_PATH),
+                "--pov-geometry",
+                str(POV_GEOMETRY_PATH),
             ]
         )
         if progress_callback:

backend/app/services/earth_db_change_listener.py

@@ -22,6 +22,12 @@ from app.services.earth_layer_adapters import (
     get_earth_update_layers_for_source,
 )
 from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
+from app.services.earth_outbox_maintenance import (
+    configured_retention_days,
+    purge_consumed_events,
+    release_maintenance_lock,
+    try_maintenance_lock,
+)
 
 logger = get_logger(__name__)
 
@@ -30,6 +36,7 @@ DEFAULT_DEBOUNCE_SECONDS = 0.25
 DEFAULT_MAX_WAIT_SECONDS = 1.5
 DELETE_FAST_FLUSH_SECONDS = 0.05
 LISTEN_KEEPALIVE_SECONDS = 5.0
+OUTBOX_RETENTION_CHECK_SECONDS = 60.0
 OUTBOX_POLL_LIMIT = 5000
 MAX_ENTITY_KEY_SAMPLES = 20
 MAX_SEEN_EVENT_IDS = 20000
@@ -56,6 +63,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
     source_has_adapter = bool(get_earth_update_layers_for_source(source))
     effective_source = source if source_has_adapter else (table_name if table_name else source)
     operation = payload.get("operation")
+    if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
+        refresh_strategy = "reload"
     update: dict[str, Any] = {
         "event": "earth.layer.changed",
         "action": "database_changed",
@@ -113,6 +122,8 @@ class PendingEarthDbChange:
         operation = payload.get("operation")
         if operation:
             self.operations.add(str(operation))
+            if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
+                self.refresh_strategy = "reload"
         entity_keys = payload.get("entity_keys")
         if not isinstance(entity_keys, list):
             entity_key = payload.get("entity_key")
@@ -176,6 +187,17 @@ class EarthDbChangeDispatcher:
         if not update:
             return False
 
+        if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
+            from app.core.websocket.broadcaster import broadcaster
+
+            keys = payload.get("entity_keys")
+            if not isinstance(keys, list):
+                keys = [payload.get("entity_key")]
+            broadcaster.enqueue_vessel_update({
+                "action": "remove",
+                "vessels": [{"mmsi": key} for key in keys if key is not None],
+            })
+
         source = update["source"]
         pending = self._pending.get(source)
         if pending is None:
@@ -331,6 +353,7 @@ class EarthDbChangeListener:
         self._stop_event: asyncio.Event | None = None
         self._connection: asyncpg.Connection | None = None
         self._loop: asyncio.AbstractEventLoop | None = None
+        self._next_retention_check = 0.0
 
     def start(self) -> None:
         if self._task and not self._task.done():
@@ -370,6 +393,7 @@ class EarthDbChangeListener:
                         )
                     except asyncio.TimeoutError:
                         await self._poll_outbox()
+                        await self._maybe_purge_outbox()
             except asyncio.CancelledError:
                 raise
             except Exception as exc:
@@ -444,6 +468,27 @@ class EarthDbChangeListener:
                 context={"events": len(rows), "accepted": accepted_count},
             )
 
+    async def _maybe_purge_outbox(self) -> None:
+        if self._connection is None or monotonic() < self._next_retention_check:
+            return
+        self._next_retention_check = monotonic() + OUTBOX_RETENTION_CHECK_SECONDS
+        try:
+            if not await try_maintenance_lock(self._connection):
+                return
+            try:
+                await purge_consumed_events(
+                    self._connection,
+                    retention_days=configured_retention_days(),
+                )
+            finally:
+                await release_maintenance_lock(self._connection)
+        except Exception as exc:
+            logger.exception_event(
+                "Earth outbox retention failed",
+                event="earth.outbox.retention_failed",
+                context={"error": str(exc)},
+            )
+
     def _on_notification(
         self,
         _connection: asyncpg.Connection,

backend/app/services/earth_layer_adapters.py

@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
         layers=("vessels",),
         cache_patterns=("vessels*", "summary*"),
         derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
+        refresh_strategy="delta",
     ),
     EarthLayerAdapter(
         sources=frozenset(
@@ -98,7 +99,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
         sources=frozenset({"media_news_archive", "earth_news_items"}),
         tables=frozenset({"earth_news_items"}),
         layers=("news",),
-        cache_patterns=("summary*",),
+        cache_patterns=(),
         refresh_strategy="reload",
     ),
     EarthLayerAdapter(

backend/app/services/earth_layer_cache.py

@@ -2,25 +2,27 @@ from __future__ import annotations
 
 import asyncio
 from collections.abc import Awaitable, Callable
-from dataclasses import dataclass
+from contextlib import suppress
+from dataclasses import dataclass, replace
 import hashlib
 import json
 import time
 from typing import Any
+from uuid import uuid4
 
-from fastapi import Response
+from fastapi import HTTPException, Response
 
 from app.core.cache import _RedisClient
 from app.core.config import settings
 from app.core.logging import get_logger
 
-
 logger = get_logger(__name__, service="earth_layer_cache")
 
 EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
 EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
 DEFAULT_LOCK_TTL_SECONDS = 10
 DEFAULT_LOCK_WAIT_SECONDS = 0.2
+DEFAULT_BUILD_WAIT_SECONDS = 30
 DEFAULT_MAX_FEATURES = 5000
 DEFAULT_MAX_BYTES = 5 * 1024 * 1024
 DEFAULT_BBOX_PRECISION_DEGREES = 0.1
@@ -35,6 +37,7 @@ class EarthLayerCachePolicy:
     max_bytes: int = DEFAULT_MAX_BYTES
     lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
     lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
+    build_wait_seconds: float = DEFAULT_BUILD_WAIT_SECONDS
 
 
 @dataclass(frozen=True)
@@ -49,6 +52,9 @@ class EarthLayerCacheResult:
 class EarthLayerCache:
     def __init__(self) -> None:
         self._client = None
+        self._inflight: dict[
+            tuple[asyncio.AbstractEventLoop, str], asyncio.Future[EarthLayerCacheResult | None]
+        ] = {}
 
     @property
     def client(self):
@@ -82,17 +88,62 @@ class EarthLayerCache:
         value = json.loads(raw)
         return value if isinstance(value, dict) else None
 
-    def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
-        self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
+    def acquire_lock(self, key: str, ttl_seconds: int) -> str | None:
+        token = uuid4().hex
+        return (
+            token if self.client.set(self.lock_key(key), token, nx=True, ex=ttl_seconds) else None
+        )
 
-    def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
-        return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
+    def renew_lock(self, key: str, token: str, ttl_seconds: int) -> bool:
+        return bool(
+            self.client.eval(
+                """
+            if redis.call('get', KEYS[1]) == ARGV[1] then
+                return redis.call('expire', KEYS[1], ARGV[2])
+            end
+            return 0
+            """,
+                1,
+                self.lock_key(key),
+                token,
+                ttl_seconds,
+            )
+        )
 
-    def release_lock(self, key: str) -> None:
-        try:
-            self.client.delete(self.lock_key(key))
-        except Exception:
-            pass
+    def release_lock(self, key: str, token: str) -> None:
+        self.client.eval(
+            """
+            if redis.call('get', KEYS[1]) == ARGV[1] then
+                return redis.call('del', KEYS[1])
+            end
+            return 0
+            """,
+            1,
+            self.lock_key(key),
+            token,
+        )
+
+    def publish(
+        self, key: str, token: str, payload: dict[str, Any], policy: EarthLayerCachePolicy
+    ) -> bool:
+        return bool(
+            self.client.eval(
+                """
+            if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
+            redis.call('setex', KEYS[2], ARGV[2], ARGV[4])
+            redis.call('setex', KEYS[3], ARGV[3], ARGV[4])
+            return 1
+            """,
+                3,
+                self.lock_key(key),
+                key,
+                self.stale_key(key),
+                token,
+                policy.fresh_ttl_seconds,
+                policy.stale_ttl_seconds,
+                json.dumps(payload, ensure_ascii=False, default=str),
+            )
+        )
 
     def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
         keys = list(self.client.scan_iter(match=pattern))
@@ -169,56 +220,122 @@ async def resolve_layer_payload(
     policy: EarthLayerCachePolicy,
     builder: Callable[[], Awaitable[dict[str, Any]]],
 ) -> EarthLayerCacheResult:
-    started = time.perf_counter()
-    try:
-        cached = earth_layer_cache.get_json(key)
-        if cached is not None:
-            return _result(cached, state="hit", key=key)
-
-        lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
-        if lock_acquired:
-            try:
-                payload = await _build_budgeted_payload(builder, policy)
-                _write_fresh_and_stale(key, payload, policy)
-                _log_cache_event("refresh", key, payload, started)
-                return _result(payload, state="refresh", key=key)
-            except Exception as exc:
-                stale = _read_stale(key)
-                if stale is not None:
-                    logger.warning_event(
-                        "Earth layer cache builder failed; returning stale payload",
-                        event="earth_layer_cache.stale_after_builder_error",
-                        context={"key": key, "error": str(exc)},
-                    )
-                    return _result(stale, state="stale", key=key)
-                raise
-            finally:
-                earth_layer_cache.release_lock(key)
-
+    flight_key = (asyncio.get_running_loop(), key)
+    while flight_key in earth_layer_cache._inflight:
         stale = _read_stale(key)
         if stale is not None:
             return _result(stale, state="stale", key=key)
+        future = earth_layer_cache._inflight[flight_key]
+        done, _ = await asyncio.wait({future}, timeout=policy.build_wait_seconds)
+        if not done:
+            raise HTTPException(503, "Earth layer cache rebuild is still in progress")
+        result = future.result()
+        if result is not None:
+            state = "hit" if result.state == "refresh" else result.state
+            return replace(result, state=state)
 
-        await asyncio.sleep(policy.lock_wait_seconds)
-        cached_after_wait = earth_layer_cache.get_json(key)
-        if cached_after_wait is not None:
-            return _result(cached_after_wait, state="hit", key=key)
-
-        payload = await _build_budgeted_payload(builder, policy)
-        _log_cache_event("miss", key, payload, started)
-        return _result(payload, state="miss", key=key)
+    future = asyncio.get_running_loop().create_future()
+    earth_layer_cache._inflight[flight_key] = future
+    try:
+        result = await _resolve_layer_payload(key=key, policy=policy, builder=builder)
+        future.set_result(result)
+        return result
+    except asyncio.CancelledError:
+        # A waiting request can take over using its own still-open DB session.
+        future.set_result(None)
+        raise
+    except Exception as exc:
+        future.set_exception(exc)
+        future.exception()
+        raise
+    finally:
+        earth_layer_cache._inflight.pop(flight_key)
+
+
+async def _resolve_layer_payload(
+    *,
+    key: str,
+    policy: EarthLayerCachePolicy,
+    builder: Callable[[], Awaitable[dict[str, Any]]],
+) -> EarthLayerCacheResult:
+    started = time.perf_counter()
+    try:
+        async with asyncio.timeout(policy.build_wait_seconds):
+            while True:
+                cached = earth_layer_cache.get_json(key)
+                if cached is not None:
+                    return _result(cached, state="hit", key=key)
+                token = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
+                if token is not None:
+                    break
+                stale = _read_stale(key)
+                if stale is not None:
+                    return _result(stale, state="stale", key=key)
+                await asyncio.sleep(policy.lock_wait_seconds)
+    except TimeoutError:
+        raise HTTPException(503, "Earth layer cache rebuild is still in progress") from None
     except Exception as exc:
-        try:
-            payload = await _build_budgeted_payload(builder, policy)
-        except Exception:
-            raise exc
         logger.warning_event(
             "Earth layer cache bypassed",
             event="earth_layer_cache.bypass",
             context={"key": key, "error": str(exc)},
         )
+        payload = await _build_budgeted_payload(builder, policy)
         return _result(payload, state="bypass", key=key)
 
+    renewal = asyncio.create_task(_renew_cache_lock(key, token, policy))
+    try:
+        try:
+            payload = await _build_budgeted_payload(builder, policy)
+        except Exception as exc:
+            stale = _read_stale(key)
+            if stale is None:
+                raise
+            logger.warning_event(
+                "Earth layer cache builder failed; returning stale payload",
+                event="earth_layer_cache.stale_after_builder_error",
+                context={"key": key, "error": str(exc)},
+            )
+            return _result(stale, state="stale", key=key)
+        try:
+            published = earth_layer_cache.publish(key, token, payload, policy)
+        except Exception as exc:
+            logger.warning_event(
+                "Earth layer cache publish failed",
+                event="earth_layer_cache.bypass",
+                context={"key": key, "error": str(exc)},
+            )
+            published = False
+        state = "refresh" if published else "bypass"
+        _log_cache_event(state, key, payload, started)
+        return _result(payload, state=state, key=key)
+    finally:
+        renewal.cancel()
+        with suppress(asyncio.CancelledError):
+            await renewal
+        try:
+            earth_layer_cache.release_lock(key, token)
+        except Exception as exc:
+            logger.warning_event(
+                "Earth layer cache lock release failed",
+                event="earth_layer_cache.lock_release_failed",
+                context={"key": key, "error": str(exc)},
+            )
+
+
+async def _renew_cache_lock(key: str, token: str, policy: EarthLayerCachePolicy) -> None:
+    try:
+        while True:
+            await asyncio.sleep(policy.lock_ttl_seconds / 3)
+            if not earth_layer_cache.renew_lock(key, token, policy.lock_ttl_seconds):
+                raise RuntimeError("Earth layer cache lock ownership lost")
+    except Exception as exc:
+        logger.warning_event(
+            "Earth layer cache lock renewal stopped",
+            event="earth_layer_cache.lock_renewal_failed",
+            context={"key": key, "error": str(exc)},
+        )
+
 
 def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
     budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
@@ -286,11 +403,6 @@ async def _build_budgeted_payload(
     return apply_payload_budget(payload, policy)
 
 
-def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
-    earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
-    earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
-
-
 def _read_stale(key: str) -> dict[str, Any] | None:
     try:
         return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
@@ -359,14 +471,16 @@ def _payload_size(payload: dict[str, Any]) -> int:
 
 def _safe_key_part(value: Any) -> str:
     raw = str(value).strip().lower()
-    return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
+    return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[
+        :160
+    ]
 
 
 def _layer_from_key(key: str) -> str:
     prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
     if not key.startswith(prefix):
         return "unknown"
-    remainder = key[len(prefix):]
+    remainder = key[len(prefix) :]
     return remainder.split(":", 1)[0]
 
 

backend/app/services/earth_news.py

@@ -1,8 +1,9 @@
 from __future__ import annotations
 
 import asyncio
+import codecs
 from dataclasses import dataclass, field
-from datetime import UTC, datetime, timedelta
+from datetime import UTC, datetime
 from email.utils import parsedate_to_datetime
 import hashlib
 import html
@@ -1638,10 +1639,15 @@ async def record_earth_news_sources_health(
     await db.commit()
 
 
-async def get_configured_sources_for_region(db: AsyncSession | None, region: str) -> list[NewsFeedSource]:
+async def get_configured_sources_for_region(
+    db: AsyncSession | None,
+    region: str,
+    *,
+    config_payload: dict[str, Any] | None = None,
+) -> list[NewsFeedSource]:
     if db is None:
         return get_sources_for_region(region)
-    payload = await get_earth_news_sources_payload(db)
+    payload = config_payload or await get_earth_news_sources_payload(db)
     sources = [
         source
         for raw in payload.get("sources", [])
@@ -1800,6 +1806,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
     return ""
 
 
+def _decode_feed_response(response: httpx.Response) -> str:
+    content = response.content
+    for marker, encoding in (
+        (codecs.BOM_UTF32_LE, "utf-32"),
+        (codecs.BOM_UTF32_BE, "utf-32"),
+        (codecs.BOM_UTF16_LE, "utf-16"),
+        (codecs.BOM_UTF16_BE, "utf-16"),
+        (codecs.BOM_UTF8, "utf-8-sig"),
+    ):
+        if content.startswith(marker):
+            return content.decode(encoding)
+
+    declaration = re.match(
+        br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
+        content,
+        re.IGNORECASE,
+    )
+    encoding = response.charset_encoding or (
+        declaration.group(1).decode("ascii") if declaration else "utf-8"
+    )
+    # Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
+    return content.decode(encoding)
+
+
 def _parse_feed_entries(
     xml_text: str,
     source: NewsFeedSource,
@@ -2453,7 +2483,9 @@ async def _fetch_single_feed_url(
                 feed=feed,
             )
         try:
-            items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
+            items = _parse_feed_entries(
+                _decode_feed_response(response), source, feed=feed, config_payload=config_payload
+            )
         except Exception as exc:
             error, status = _format_source_fetch_error(
                 status_code=status_code,
@@ -2461,7 +2493,7 @@ async def _fetch_single_feed_url(
                 body=response.text,
                 parsed_count=0,
             )
-            if error is None:
+            if status == "empty":
                 error = str(exc)
                 status = "format_error"
             return source, [], error, _source_health_result(
@@ -2704,7 +2736,6 @@ async def get_earth_news_payload(
     del provider_client
     active_region = region if region in REGION_ANCHORS else determine_focus_region(lat, lon)
     has_settings_db = db is not None and callable(getattr(db, "execute", None))
-    source_config_payload = await get_earth_news_sources_payload(db) if has_settings_db else None
     sources = (
         await get_configured_sources_for_region(db, active_region)
         if has_settings_db
@@ -2723,39 +2754,7 @@ async def get_earth_news_payload(
             locale=locale,
         )
 
-    from app.services.earth_news_store import (
-        get_earth_news_feed_coverage,
-        get_earth_news_freshness,
-        list_earth_news_cruise_items,
-        list_earth_news_items,
-        upsert_earth_news_items,
-    )
-
-    errors: list[str] = []
-    item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
-    should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at)
-    if not should_supplement and callable(getattr(db, "execute", None)):
-        expected_feed_keys = _expected_feed_keys(sources, active_region=active_region)
-        if expected_feed_keys:
-            covered_feed_keys = await get_earth_news_feed_coverage(
-                db,
-                active_region=active_region,
-                recent_after=datetime.now(UTC) - timedelta(seconds=RSS_SUPPLEMENT_MAX_AGE_SECONDS),
-            )
-            should_supplement = bool(expected_feed_keys - covered_feed_keys)
-    if should_supplement:
-        if source_config_payload is None:
-            fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region)
-        else:
-            fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(
-                sources,
-                active_region=active_region,
-                config_payload=source_config_payload,
-            )
-        await record_earth_news_sources_health(db, health_by_source)
-        ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
-        await _enqueue_unverified_locations(ranked_fetched_items)
-        await upsert_earth_news_items(db, ranked_fetched_items)
+    from app.services.earth_news_store import list_earth_news_cruise_items, list_earth_news_items
 
     items = await _call_store_list_items(
         list_earth_news_items,
@@ -2765,7 +2764,7 @@ async def get_earth_news_payload(
         categories=categories,
         source_ids=source_ids,
     )
-    if not source_ids:
+    if items and not source_ids:
         ready_sources = {
             _news_item_source_id(item)
             for item in items
@@ -2806,10 +2805,6 @@ async def get_earth_news_payload(
         )
     else:
         cruise_items = items
-    enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
-    await _enqueue_unverified_locations(list(enqueue_candidates.values()))
-    stale = bool(errors and items)
-
     return _build_payload(
         lat=lat,
         lon=lon,
@@ -2817,8 +2812,8 @@ async def get_earth_news_payload(
         items=items,
         cruise_items=cruise_items,
         sources=sources,
-        errors=errors,
-        stale=stale,
+        errors=[],
+        stale=False,
         categories=categories,
         source_ids=source_ids,
         limit=limit,

backend/app/services/earth_news_refresh.py

@@ -0,0 +1,480 @@
+"""Periodic RSS supplementation for the read-only Earth news feed."""
+
+from __future__ import annotations
+
+import asyncio
+from contextlib import suppress
+from dataclasses import replace
+from datetime import UTC, datetime, timedelta
+from hashlib import sha256
+import json
+from typing import Any
+from uuid import uuid4
+
+from app.core.cache import _RedisClient
+from app.core.logging import get_logger
+from app.db.session import async_session_factory
+from app.services import earth_news
+from app.services.earth_news_store import (
+    get_earth_news_feed_coverage,
+    get_earth_news_freshness,
+    upsert_earth_news_items,
+)
+
+logger = get_logger(__name__, service="earth_news")
+
+REFRESH_INTERVAL_SECONDS = 10 * 60
+REFRESH_LOCK_TTL_SECONDS = 120
+REFRESH_LOCK_RENEW_SECONDS = 30
+SOURCE_FAILURE_COOLDOWN_SECONDS = 30 * 60
+REFRESH_LOCK_PREFIX = "earth:news:rss-refresh:v1"
+SOURCE_COOLDOWN_PREFIX = "earth:news:rss-cooldown:v1"
+REFRESH_STATUS_PREFIX = "earth:news:rss-refresh-status:v1"
+REFRESH_ACTIVE_PREFIX = "earth:news:rss-refresh-active:v1"
+REFRESH_STATUS_TTL_SECONDS = 10 * 60
+REFRESH_ACTIVE_TTL_SECONDS = 120
+REFRESH_STATUS_HEARTBEAT_SECONDS = 30
+
+
+def _refresh_status_key(refresh_id: str) -> str:
+    return f"{REFRESH_STATUS_PREFIX}:{refresh_id}"
+
+
+def _refresh_active_key(region: str) -> str:
+    return f"{REFRESH_ACTIVE_PREFIX}:{region}"
+
+
+def create_earth_news_refresh_status(region: str) -> tuple[dict[str, Any], bool]:
+    """Create one per-region active refresh status, or return the current one."""
+    refresh_id = uuid4().hex
+    status = {"refresh_id": refresh_id, "region": region, "status": "pending"}
+    script = """
+        local active_id = redis.call('get', KEYS[1])
+        if active_id then
+        local active_status = redis.call('get', ARGV[5] .. active_id)
+            if active_status then
+                local decoded = cjson.decode(active_status)
+                if decoded.status == 'pending' or decoded.status == 'running' then
+                    return {active_id, active_status, 0}
+                end
+            end
+            redis.call('del', KEYS[1])
+        end
+        redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[3])
+        redis.call('set', KEYS[2], ARGV[2], 'EX', ARGV[4])
+        return {ARGV[1], ARGV[2], 1}
+    """
+    raw = _RedisClient.get_client().eval(
+        script,
+        2,
+        _refresh_active_key(region),
+        _refresh_status_key(refresh_id),
+        refresh_id,
+        json.dumps(status, separators=(",", ":")),
+        REFRESH_ACTIVE_TTL_SECONDS,
+        REFRESH_STATUS_TTL_SECONDS,
+        f"{REFRESH_STATUS_PREFIX}:",
+    )
+    _, encoded, created = raw
+    payload = json.loads(encoded)
+    return payload, bool(int(created))
+
+
+def set_earth_news_refresh_status(
+    refresh_id: str,
+    region: str,
+    status: str,
+    *,
+    error: dict[str, str] | None = None,
+) -> dict[str, Any]:
+    payload: dict[str, Any] = {
+        "refresh_id": refresh_id,
+        "region": region,
+        "status": status,
+    }
+    if error:
+        payload["error"] = error
+    script = """
+        if redis.call('get', KEYS[1]) ~= ARGV[2] then return 0 end
+        redis.call('set', KEYS[2], ARGV[1], 'EX', ARGV[3])
+        if ARGV[4] == 'succeeded' or ARGV[4] == 'failed' then
+            redis.call('del', KEYS[1])
+        else
+            redis.call('expire', KEYS[1], ARGV[5])
+        end
+        return 1
+    """
+    _RedisClient.get_client().eval(
+        script,
+        2,
+        _refresh_active_key(region),
+        _refresh_status_key(refresh_id),
+        json.dumps(payload, separators=(",", ":")),
+        refresh_id,
+        REFRESH_STATUS_TTL_SECONDS,
+        status,
+        REFRESH_ACTIVE_TTL_SECONDS,
+    )
+    return payload
+
+
+def touch_earth_news_refresh_status(refresh_id: str, region: str) -> bool:
+    script = """
+        if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
+        redis.call('expire', KEYS[1], ARGV[2])
+        redis.call('expire', KEYS[2], ARGV[3])
+        return 1
+    """
+    return bool(
+        _RedisClient.get_client().eval(
+            script,
+            2,
+            _refresh_active_key(region),
+            _refresh_status_key(refresh_id),
+            refresh_id,
+            REFRESH_ACTIVE_TTL_SECONDS,
+            REFRESH_STATUS_TTL_SECONDS,
+        )
+    )
+
+
+def get_earth_news_refresh_status(refresh_id: str) -> dict[str, Any] | None:
+    raw = _RedisClient.get_client().get(_refresh_status_key(refresh_id))
+    if not raw:
+        return None
+    value = json.loads(raw)
+    if not isinstance(value, dict):
+        return None
+    if value.get("status") in {"pending", "running"}:
+        region = str(value.get("region") or "")
+        active_id = _RedisClient.get_client().get(_refresh_active_key(region)) if region else None
+        if active_id != refresh_id:
+            return {
+                **value,
+                "status": "failed",
+                "error": {
+                    "code": "expired",
+                    "message": "Refresh status expired. Submit a new refresh.",
+                },
+            }
+    return value
+
+
+def _lock_key(region: str) -> str:
+    return f"{REFRESH_LOCK_PREFIX}:{region}"
+
+
+def _cooldown_key(region: str, source_id: str, feed_id: str) -> str:
+    identity = f"{region}\0{source_id}\0{feed_id}".encode("utf-8")
+    return f"{SOURCE_COOLDOWN_PREFIX}:{sha256(identity).hexdigest()}"
+
+
+def _source_failure_cooldown_seconds(source: earth_news.NewsFeedSource | None) -> int:
+    if source is None:
+        return SOURCE_FAILURE_COOLDOWN_SECONDS
+    configured_minutes = source.health_policy.get("cooldown_minutes")
+    try:
+        return max(60, int(float(configured_minutes) * 60))
+    except TypeError, ValueError:
+        return SOURCE_FAILURE_COOLDOWN_SECONDS
+
+
+def _select_feeds(
+    sources: list[earth_news.NewsFeedSource],
+    *,
+    active_region: str,
+    required_keys: set[tuple[str, str]],
+    redis_client: Any | None,
+) -> tuple[list[earth_news.NewsFeedSource], set[tuple[str, str]]]:
+    selected_sources: list[earth_news.NewsFeedSource] = []
+    cooling_keys: set[tuple[str, str]] = set()
+    for source in sources:
+        selected_feeds = []
+        source_cooling = False
+        if redis_client is not None:
+            try:
+                source_cooling = bool(
+                    redis_client.exists(_cooldown_key(active_region, source.id, "__source__"))
+                )
+            except Exception as exc:
+                logger.warning_event(
+                    "Earth news source cooldown lookup failed",
+                    event="earth_news.rss.cooldown_lookup_failed",
+                    context={"region": active_region, "source_id": source.id, "error": str(exc)},
+                )
+        for feed in earth_news._source_feeds(source):
+            feed_key = (source.id, feed.id)
+            if feed_key not in required_keys:
+                continue
+            if source_cooling:
+                cooling_keys.add(feed_key)
+                continue
+            if redis_client is not None:
+                try:
+                    if redis_client.exists(_cooldown_key(active_region, *feed_key)):
+                        cooling_keys.add(feed_key)
+                        continue
+                except Exception as exc:
+                    logger.warning_event(
+                        "Earth news source cooldown lookup failed",
+                        event="earth_news.rss.cooldown_lookup_failed",
+                        context={
+                            "region": active_region,
+                            "source_id": source.id,
+                            "error": str(exc),
+                        },
+                    )
+            selected_feeds.append(feed)
+        if selected_feeds:
+            selected_sources.append(
+                replace(
+                    source,
+                    feed_url=selected_feeds[0].url,
+                    feed_urls=tuple(feed.url for feed in selected_feeds),
+                    feeds=tuple(selected_feeds),
+                )
+            )
+    return selected_sources, cooling_keys
+
+
+def _record_failed_feed_cooldowns(
+    redis_client: Any | None,
+    *,
+    active_region: str,
+    health_by_source: dict[str, dict[str, Any]],
+    sources: list[earth_news.NewsFeedSource],
+) -> None:
+    if redis_client is None:
+        return
+    cooldown_by_source = {source.id: source for source in sources}
+    for source_id, health in health_by_source.items():
+        feed_results = health.get("feed_results")
+        if isinstance(feed_results, list) and feed_results:
+            failed_feed_ids = [
+                str(result.get("feed_id") or "")
+                for result in feed_results
+                if isinstance(result, dict) and not result.get("ok") and result.get("feed_id")
+            ]
+        else:
+            failed_feed_ids = [str(health.get("feed_id") or "__source__")]
+            if health.get("ok"):
+                failed_feed_ids = []
+        for feed_id in failed_feed_ids:
+            try:
+                redis_client.set(
+                    _cooldown_key(active_region, source_id, feed_id),
+                    "1",
+                    ex=_source_failure_cooldown_seconds(cooldown_by_source.get(source_id)),
+                )
+            except Exception as exc:
+                logger.warning_event(
+                    "Earth news source failure cooldown write failed",
+                    event="earth_news.rss.cooldown_write_failed",
+                    context={"region": active_region, "source_id": source_id, "error": str(exc)},
+                )
+
+
+async def _renew_lock(redis_client: Any, key: str, token: str, lease_lost: asyncio.Event) -> None:
+    renew_script = """
+        if redis.call('get', KEYS[1]) == ARGV[1] then
+            return redis.call('expire', KEYS[1], ARGV[2])
+        end
+        return 0
+    """
+    try:
+        while True:
+            await asyncio.sleep(REFRESH_LOCK_RENEW_SECONDS)
+            if not redis_client.eval(renew_script, 1, key, token, REFRESH_LOCK_TTL_SECONDS):
+                lease_lost.set()
+                return
+    except asyncio.CancelledError:
+        raise
+    except Exception as exc:
+        lease_lost.set()
+        logger.warning_event(
+            "Earth news refresh lock renewal failed",
+            event="earth_news.rss.refresh_lock_renewal_failed",
+            context={"error": str(exc)},
+        )
+
+
+async def refresh_earth_news_region(active_region: str, *, force: bool = False) -> dict[str, Any]:
+    """Fetch only stale or uncovered feeds for one region, outside the request path."""
+    if active_region not in earth_news.REGION_ANCHORS:
+        raise ValueError(f"Unsupported Earth news region: {active_region}")
+
+    lock_key = _lock_key(active_region)
+    token = uuid4().hex
+    redis_client = None
+    lock_acquired = False
+    lock_available = False
+    try:
+        redis_client = _RedisClient.get_client()
+        lock_acquired = bool(
+            redis_client.set(lock_key, token, nx=True, ex=REFRESH_LOCK_TTL_SECONDS)
+        )
+        lock_available = True
+    except Exception as exc:
+        logger.warning_event(
+            "Earth news refresh coordination unavailable",
+            event="earth_news.rss.refresh_lock_unavailable",
+            context={"region": active_region, "error": str(exc)},
+        )
+
+    if not lock_available or not lock_acquired:
+        state = "busy" if lock_available else "coordination_unavailable"
+        return {"state": state, "fetched": 0, "upserted": 0}
+
+    lease_lost = asyncio.Event()
+    renewal: asyncio.Task[None] | None = None
+    if lock_acquired and redis_client is not None:
+        renewal = asyncio.create_task(_renew_lock(redis_client, lock_key, token, lease_lost))
+
+    try:
+        async with async_session_factory() as db:
+            source_config = await earth_news.get_earth_news_sources_payload(db)
+            sources = await earth_news.get_configured_sources_for_region(
+                db, active_region, config_payload=source_config
+            )
+            if not sources:
+                return {
+                    "state": "failed",
+                    "error_code": "source_unavailable",
+                    "fetched": 0,
+                    "upserted": 0,
+                }
+
+            item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
+            expected_keys = earth_news._expected_feed_keys(sources, active_region=active_region)
+            if not expected_keys:
+                return {
+                    "state": "failed",
+                    "error_code": "source_unavailable",
+                    "fetched": 0,
+                    "upserted": 0,
+                }
+
+            now = datetime.now(UTC)
+            needs_full_refresh = force or earth_news._needs_rss_supplement(
+                item_count=item_count, newest_at=newest_at
+            )
+            covered_keys = await get_earth_news_feed_coverage(
+                db,
+                active_region=active_region,
+                recent_after=now - timedelta(seconds=earth_news.RSS_SUPPLEMENT_MAX_AGE_SECONDS),
+            )
+            required_keys = expected_keys if needs_full_refresh else expected_keys - covered_keys
+            if not required_keys:
+                return {
+                    "state": "completed",
+                    "successful_sources": 0,
+                    "failed_sources": 0,
+                    "fetched": 0,
+                    "upserted": 0,
+                }
+
+            selected_sources, cooling_keys = _select_feeds(
+                sources,
+                active_region=active_region,
+                required_keys=required_keys,
+                redis_client=redis_client,
+            )
+            if cooling_keys:
+                logger.info_event(
+                    "Skipping failed Earth news feeds during cooldown",
+                    event="earth_news.rss.cooldown_active",
+                    context={"region": active_region, "feed_count": len(cooling_keys)},
+                )
+            if not selected_sources:
+                return {
+                    "state": "failed",
+                    "error_code": "source_unavailable",
+                    "fetched": 0,
+                    "upserted": 0,
+                }
+
+            fetched_items, _errors, health_by_source = (
+                await earth_news._fetch_rss_items_for_sources(
+                    selected_sources,
+                    active_region=active_region,
+                    config_payload=source_config,
+                )
+            )
+            if lease_lost.is_set():
+                await db.rollback()
+                return {"state": "busy", "fetched": 0, "upserted": 0}
+            _record_failed_feed_cooldowns(
+                redis_client,
+                active_region=active_region,
+                health_by_source=health_by_source,
+                sources=selected_sources,
+            )
+            await earth_news.record_earth_news_sources_health(db, health_by_source)
+            if lease_lost.is_set():
+                await db.rollback()
+                return {"state": "busy", "fetched": 0, "upserted": 0}
+            ranked_items = earth_news._rank_and_trim_items(
+                fetched_items, active_region=active_region
+            )
+            await earth_news._enqueue_unverified_locations(ranked_items)
+            upserted = await upsert_earth_news_items(db, ranked_items)
+            if lease_lost.is_set():
+                await db.rollback()
+                return {"state": "busy", "fetched": len(fetched_items), "upserted": 0}
+            await db.commit()
+            successful_sources = sum(bool(health.get("ok")) for health in health_by_source.values())
+            failed_sources = max(0, len(selected_sources) - successful_sources)
+            state = "completed" if successful_sources else "failed"
+            failed_statuses = {
+                str(health.get("status") or "")
+                for health in health_by_source.values()
+                if not health.get("ok")
+            }
+            error_code = None
+            if not successful_sources:
+                error_code = "timeout" if failed_statuses == {"timeout"} else "source_unavailable"
+            return {
+                "state": state,
+                "error_code": error_code,
+                "successful_sources": successful_sources,
+                "failed_sources": failed_sources,
+                "fetched": len(fetched_items),
+                "upserted": upserted,
+            }
+    except Exception as exc:
+        logger.warning_event(
+            "Earth news periodic RSS refresh failed",
+            event="earth_news.rss.refresh_failed",
+            context={"region": active_region, "error": str(exc)},
+        )
+        raise
+    finally:
+        if renewal is not None:
+            renewal.cancel()
+            with suppress(asyncio.CancelledError):
+                await renewal
+        if lock_acquired and redis_client is not None:
+            release_script = """
+                if redis.call('get', KEYS[1]) == ARGV[1] then
+                    return redis.call('del', KEYS[1])
+                end
+                return 0
+            """
+            try:
+                redis_client.eval(release_script, 1, lock_key, token)
+            except Exception as exc:
+                logger.warning_event(
+                    "Earth news refresh lock release failed",
+                    event="earth_news.rss.refresh_lock_release_failed",
+                    context={"region": active_region, "error": str(exc)},
+                )
+
+
+async def refresh_earth_news_all_regions() -> None:
+    for active_region in earth_news.REGION_ANCHORS:
+        try:
+            await refresh_earth_news_region(active_region)
+        except asyncio.CancelledError:
+            raise
+        except Exception:
+            continue

backend/app/services/earth_news_store.py

@@ -278,22 +278,25 @@ async def get_earth_news_feed_coverage(
     active_region: str,
     recent_after: datetime | None = None,
 ) -> set[tuple[str, str]]:
-    query = select(
-        EarthNewsItem.id,
-        EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
-        EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
+    news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
+    source_id = func.coalesce(
+        func.nullif(news_meta.op("->>")("source_id"), ""),
+        func.split_part(EarthNewsItem.id, ":", 1),
     )
+    feed_id = func.nullif(news_meta.op("->>")("feed_id"), "")
+    query = select(
+        source_id,
+        feed_id,
+    ).where(source_id != "", feed_id.is_not(None)).distinct()
     if active_region != "global":
         query = query.where(EarthNewsItem.region.in_({"global", active_region}))
     if recent_after is not None:
         query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
     result = await db.execute(query)
     coverage: set[tuple[str, str]] = set()
-    for item_id, source_id, feed_id in result.all():
+    for source_id, feed_id in result.all():
         normalized_source_id = str(source_id or "").strip()
         normalized_feed_id = str(feed_id or "").strip()
-        if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
-            normalized_source_id = item_id.split(":", 1)[0]
         if normalized_source_id and normalized_feed_id:
             coverage.add((normalized_source_id, normalized_feed_id))
     return coverage
@@ -340,30 +343,49 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
             changed += 1
             continue
 
-        record.title = item.title
-        record.summary = item.summary
-        record.url = item.url
-        record.source = item.source
-        record.feed_name = item.feed_name
-        record.region = item.feed_region
-        record.homepage_url = item.homepage_url
-        record.published_at = item.published_at
+        visible_changed = False
+        for field_name, value in (
+            ("title", item.title),
+            ("summary", item.summary),
+            ("url", item.url),
+            ("source", item.source),
+            ("feed_name", item.feed_name),
+            ("region", item.feed_region),
+            ("homepage_url", item.homepage_url),
+            ("published_at", item.published_at),
+        ):
+            if getattr(record, field_name) != value:
+                setattr(record, field_name, value)
+                visible_changed = True
         record.last_seen_at = now
         location_meta = dict(record.location_meta or {})
-        location_meta["news_meta"] = _news_meta_patch(item)
-        record.location_meta = location_meta
+        news_meta = _news_meta_patch(item)
+        if location_meta.get("news_meta") != news_meta:
+            location_meta["news_meta"] = news_meta
+            record.location_meta = location_meta
+            visible_changed = True
         if item.localizations:
             merged_localizations = {
                 **dict(record.localizations or {}),
                 **dict(item.localizations or {}),
             }
-            record.content_language = item.content_language
-            record.localizations = merged_localizations
+            if record.content_language != item.content_language:
+                record.content_language = item.content_language
+                visible_changed = True
+            if dict(record.localizations or {}) != merged_localizations:
+                record.localizations = merged_localizations
+                visible_changed = True
             if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
-                record.enrichment_status = item.enrichment_status
-                record.enrichment_error = item.enrichment_error
-                record.enriched_at = item.enriched_at
-        changed += 1
+                for field_name, value in (
+                    ("enrichment_status", item.enrichment_status),
+                    ("enrichment_error", item.enrichment_error),
+                    ("enriched_at", item.enriched_at),
+                ):
+                    if getattr(record, field_name) != value:
+                        setattr(record, field_name, value)
+                        visible_changed = True
+        if visible_changed:
+            changed += 1
     await db.flush()
     return changed
 

backend/app/services/earth_outbox_maintenance.py

@@ -0,0 +1,184 @@
+"""Bounded retention for consumed Earth change events."""
+
+from __future__ import annotations
+
+import asyncio
+import os
+from datetime import UTC, datetime, timedelta
+from typing import AsyncContextManager, Protocol
+
+from app.core.logging import get_logger
+
+logger = get_logger(__name__)
+
+INDEX_NAME = "idx_earth_data_change_events_consumed_retention"
+DEFAULT_RETENTION_DAYS = 1
+DEFAULT_BATCH_SIZE = 1000
+DEFAULT_PAUSE_SECONDS = 0.1
+MAINTENANCE_LOCK_KEY = (17001613, 1)
+
+DELETE_BATCH_SQL = """
+WITH doomed AS (
+    SELECT id
+    FROM earth_data_change_events
+    WHERE consumed_at IS NOT NULL AND consumed_at < $1
+    ORDER BY consumed_at, id
+    LIMIT $2
+    FOR UPDATE SKIP LOCKED
+), deleted AS (
+    DELETE FROM earth_data_change_events AS events
+    USING doomed
+    WHERE events.id = doomed.id
+    RETURNING events.id
+)
+SELECT count(*) FROM deleted
+"""
+
+
+class OutboxConnection(Protocol):
+    async def fetchval(self, query: str, *args: object) -> object: ...
+
+    async def fetchrow(self, query: str, *args: object) -> object: ...
+
+    async def execute(self, query: str, *args: object) -> object: ...
+
+    def transaction(self) -> AsyncContextManager[object]: ...
+
+
+def configured_retention_days() -> int:
+    raw = os.getenv("PLANET_EARTH_OUTBOX_RETENTION_DAYS", str(DEFAULT_RETENTION_DAYS))
+    try:
+        days = int(raw)
+        if 1 <= days <= 3650:
+            return days
+    except ValueError:
+        pass
+    logger.warning_event(
+        "Invalid Earth outbox retention; using default",
+        event="earth.outbox.retention_invalid",
+        context={"value": raw},
+    )
+    return DEFAULT_RETENTION_DAYS
+
+
+async def try_maintenance_lock(conn: OutboxConnection) -> bool:
+    return bool(await conn.fetchval("SELECT pg_try_advisory_lock($1, $2)", *MAINTENANCE_LOCK_KEY))
+
+
+async def release_maintenance_lock(conn: OutboxConnection) -> None:
+    await conn.fetchval("SELECT pg_advisory_unlock($1, $2)", *MAINTENANCE_LOCK_KEY)
+
+
+async def retention_index_state(conn: OutboxConnection) -> str:
+    row = await conn.fetchrow(
+        """
+        SELECT indexes.indisvalid, indexes.indisready
+        FROM pg_index AS indexes
+        JOIN pg_class AS index_table ON index_table.oid = indexes.indexrelid
+        WHERE indexes.indrelid = 'earth_data_change_events'::regclass
+          AND index_table.relname = $1
+        """,
+        INDEX_NAME,
+    )
+    if row is None:
+        return "missing"
+    return "ready" if row["indisvalid"] and row["indisready"] else "invalid"
+
+
+async def ensure_retention_index(conn: OutboxConnection) -> None:
+    """Call on an autocommit connection, never inside app init_db's transaction."""
+    state = await retention_index_state(conn)
+    if state == "invalid":
+        await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}")
+    if state != "ready":
+        await conn.execute(
+            f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {INDEX_NAME} "
+            "ON earth_data_change_events (consumed_at, id) WHERE consumed_at IS NOT NULL"
+        )
+    if await retention_index_state(conn) != "ready":
+        raise RuntimeError("Earth outbox retention index is not valid and ready")
+
+
+async def purge_consumed_events(
+    conn: OutboxConnection,
+    *,
+    retention_days: int,
+    batch_size: int = DEFAULT_BATCH_SIZE,
+    max_batches: int = 1,
+    pause_seconds: float = DEFAULT_PAUSE_SECONDS,
+) -> int:
+    if not 1 <= retention_days <= 3650:
+        raise ValueError("retention_days must be between 1 and 3650")
+    if batch_size < 1 or max_batches < 1 or pause_seconds < 0:
+        raise ValueError("batch_size/max_batches must be positive and pause_seconds non-negative")
+    if await retention_index_state(conn) != "ready":
+        logger.warning_event(
+            "Skipping Earth outbox retention because its index is unavailable",
+            event="earth.outbox.retention_index_unavailable",
+        )
+        return 0
+
+    cutoff = datetime.now(UTC) - timedelta(days=retention_days)
+    deleted_total = 0
+    for batch_number in range(max_batches):
+        deleted = int(await conn.fetchval(DELETE_BATCH_SQL, cutoff, batch_size) or 0)
+        deleted_total += deleted
+        if deleted < batch_size:
+            break
+        if batch_number + 1 < max_batches and pause_seconds:
+            await asyncio.sleep(pause_seconds)
+    if deleted_total:
+        logger.info_event(
+            "Purged consumed Earth outbox events",
+            event="earth.outbox.retention_purged",
+            context={"deleted": deleted_total, "retention_days": retention_days},
+        )
+    return deleted_total
+
+
+async def compact_consumed_events(
+    conn: OutboxConnection,
+    *,
+    retention_days: int,
+    lock_timeout_seconds: int = 10,
+    statement_timeout_seconds: int = 900,
+) -> tuple[int, int]:
+    """One-time atomic rewrite; blocks table access until the transaction commits."""
+    if not 1 <= retention_days <= 3650:
+        raise ValueError("retention_days must be between 1 and 3650")
+    if lock_timeout_seconds < 1 or statement_timeout_seconds < 1:
+        raise ValueError("timeouts must be positive")
+    cutoff = datetime.now(UTC) - timedelta(days=retention_days)
+    async with conn.transaction():
+        await conn.fetchval(
+            "SELECT set_config('lock_timeout', $1, true)", f"{lock_timeout_seconds}s"
+        )
+        await conn.fetchval(
+            "SELECT set_config('statement_timeout', $1, true)", f"{statement_timeout_seconds}s"
+        )
+        await conn.execute("LOCK TABLE earth_data_change_events IN ACCESS EXCLUSIVE MODE")
+        await conn.execute(
+            """
+            CREATE TEMP TABLE earth_outbox_retained ON COMMIT DROP AS
+            SELECT * FROM earth_data_change_events
+            WHERE consumed_at IS NULL OR consumed_at >= $1
+            """,
+            cutoff,
+        )
+        retained = await conn.fetchrow("""
+            SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
+            FROM earth_outbox_retained
+            """)
+        if retained is None:
+            raise RuntimeError("Unable to verify retained Earth outbox rows")
+        await conn.execute("TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT")
+        await conn.execute(
+            "INSERT INTO earth_data_change_events SELECT * FROM earth_outbox_retained"
+        )
+        restored = await conn.fetchrow("""
+            SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
+            FROM earth_data_change_events
+            """)
+        if restored is None or dict(retained) != dict(restored):
+            raise RuntimeError("Earth outbox compact verification failed; transaction rolled back")
+    return int(retained["total"]), int(retained["pending"])

backend/app/services/epoch_location.py

@@ -0,0 +1,198 @@
+"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import math
+import re
+from typing import Any
+
+from app.core.countries import normalize_country
+
+
+_COUNTRY_LABEL_OVERRIDES = {
+    "korea republic of": "韩国",
+}
+_REGION_ONLY_LABELS = {
+    "europe",
+    "iowa",
+    "maryland",
+    "new mexico",
+    "texas",
+    "the nordics",
+    "virginia",
+    "washington",
+}
+_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
+_HEMISPHERE_COORDINATE_PATTERN = re.compile(
+    r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
+    r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
+    re.IGNORECASE,
+)
+_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
+    r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
+    r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
+    r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
+    re.IGNORECASE | re.DOTALL,
+)
+_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
+    r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
+    r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
+)
+_NUMBERED_STREET_PATTERN = re.compile(
+    r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
+    r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
+    r"highway|hwy|way|parkway|pkwy|chome)\b|"
+    r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
+    re.IGNORECASE,
+)
+_POSTAL_CODE_PATTERN = re.compile(
+    r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
+    r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
+    re.IGNORECASE,
+)
+_SITE_CONTEXT_PATTERN = re.compile(
+    r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
+    r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
+    re.IGNORECASE,
+)
+_COORDINATE_CUE_PATTERN = re.compile(
+    r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
+    re.IGNORECASE | re.DOTALL,
+)
+
+
+@dataclass(frozen=True)
+class EpochLocationAssessment:
+    precision: str
+    reason: str
+
+    @property
+    def is_renderable(self) -> bool:
+        return self.precision in {"precise", "site", "city"}
+
+
+def _normalized_label(value: Any) -> str:
+    return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
+
+
+def _country_from_label(value: Any) -> str | None:
+    return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
+        _normalized_label(value)
+    )
+
+
+def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
+    raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
+    if isinstance(raw, bool):
+        return None
+    try:
+        number = float(raw)
+    except (TypeError, ValueError):
+        return None
+    return number if math.isfinite(number) and abs(number) <= bound else None
+
+
+def _has_explicit_coordinate_pair(text: str) -> bool:
+    hemispheres = {}
+    for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
+        hemisphere = match.group("hemisphere").upper()
+        degrees = int(match.group("degrees"))
+        minutes = int(match.group("minutes"))
+        seconds = float(match.group("seconds") or 0)
+        bound = 90 if hemisphere in {"N", "S"} else 180
+        if degrees <= bound and minutes < 60 and seconds < 60:
+            hemispheres[hemisphere] = degrees
+    if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
+        return True
+    labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
+    if labeled_pair:
+        latitude = float(labeled_pair.group("latitude"))
+        longitude = float(labeled_pair.group("longitude"))
+        return abs(latitude) <= 90 and abs(longitude) <= 180
+    decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
+    if not decimal_pair:
+        return False
+    latitude, longitude = map(float, decimal_pair.groups())
+    return abs(latitude) <= 90 and abs(longitude) <= 180
+
+
+def _has_address_evidence(text: str) -> bool:
+    if _NUMBERED_STREET_PATTERN.search(text):
+        return True
+    return bool(
+        _POSTAL_CODE_PATTERN.search(text)
+        and ("\n" in text or text.count(",") >= 2)
+        and _SITE_CONTEXT_PATTERN.search(text)
+    )
+
+
+def assess_epoch_location(
+    location: Any,
+    *,
+    country: Any = None,
+    raw_country: Any = None,
+) -> EpochLocationAssessment:
+    """Classify source coordinates without upgrading their source text.
+
+    Epoch's CSV supplies coordinates even when ``Location`` contains only a
+    country. Those coordinates are useful source evidence, but are not a
+    facility point and must not enter the renderable location pipeline.
+    """
+    if isinstance(location, dict):
+        latitude = _coordinate(location, ("lat", "latitude"), 90)
+        longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
+        has_coordinates = latitude is not None and longitude is not None
+        return EpochLocationAssessment(
+            "precise" if has_coordinates else "unknown",
+            "complete explicit coordinate object"
+            if has_coordinates
+            else "coordinate object lacks a complete valid latitude/longitude pair",
+        )
+
+    text = str(location or "").strip()
+    if not text:
+        return EpochLocationAssessment("unknown", "Epoch Location is empty")
+
+    normalized = _normalized_label(text)
+    normalized_country = normalize_country(country)
+    source_country = normalize_country(raw_country)
+    location_country = _country_from_label(text)
+    if location_country or (
+        raw_country and normalized == _normalized_label(raw_country)
+    ):
+        matched = location_country or source_country or normalized_country
+        return EpochLocationAssessment(
+            "country",
+            f"Epoch Location identifies country only: {matched or text}",
+        )
+    if normalized in _REGION_ONLY_LABELS:
+        return EpochLocationAssessment(
+            "region", f"Epoch Location identifies region/state only: {text}"
+        )
+    comma_parts = [part.strip() for part in text.split(",") if part.strip()]
+    if (
+        len(comma_parts) == 2
+        and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
+        and _country_from_label(comma_parts[1])
+    ):
+        return EpochLocationAssessment(
+            "region", f"Epoch Location identifies region/state only: {text}"
+        )
+    if normalized in _UNLOCATABLE_LABELS:
+        return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
+    if normalized.isdigit():
+        return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
+    if _has_explicit_coordinate_pair(text):
+        return EpochLocationAssessment(
+            "precise", "Epoch Location contains a complete valid coordinate pair"
+        )
+    if _COORDINATE_CUE_PATTERN.search(text):
+        return EpochLocationAssessment(
+            "unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
+        )
+    if _has_address_evidence(text):
+        return EpochLocationAssessment("precise", "Epoch Location contains an address")
+    if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
+        return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
+    return EpochLocationAssessment("city", "Epoch Location identifies a locality")

backend/app/services/llm_model_catalog.py

@@ -0,0 +1,186 @@
+"""Authenticated model discovery shared by refresh and connection checks."""
+
+from __future__ import annotations
+
+import asyncio
+from dataclasses import dataclass
+from datetime import datetime
+from urllib.parse import urlsplit, urlunsplit
+
+import httpx
+
+CATALOG_TIMEOUT_SECONDS = 30
+CATALOG_MAX_PAGES = 100
+CATALOG_PAGE_SIZE = 100
+CATALOG_REQUEST_ATTEMPTS = 2
+CATALOG_RETRY_DELAY_SECONDS = 0.2
+SUPPORTED_PROVIDER_APIS = {
+    "openai-completions",
+    "openai-responses",
+    "anthropic-messages",
+    "ollama-generate",
+}
+
+
+class LLMProviderCatalogError(RuntimeError):
+    """A safe, user-facing catalog failure with no upstream response or credentials."""
+
+
+@dataclass(frozen=True)
+class ModelCatalog:
+    url: str
+    models: list[str]
+
+
+def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
+    parts = urlsplit(base_url.strip())
+    if parts.scheme not in {"http", "https"} or not parts.hostname:
+        raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
+    if parts.username or parts.password or parts.query or parts.fragment:
+        raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
+    if provider_api not in SUPPORTED_PROVIDER_APIS:
+        raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
+    path = parts.path.rstrip("/")
+    if provider_api == "ollama-generate":
+        path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
+    elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
+        path = "/api/v1/models"
+    else:
+        if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
+            path += "/v1"
+        path += "/models"
+    return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
+
+
+def catalog_error_message(exc: Exception) -> str:
+    if isinstance(exc, LLMProviderCatalogError):
+        return str(exc)
+    if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
+        return "模型目录请求超时，请检查网络后重试。"
+    if isinstance(exc, httpx.HTTPStatusError):
+        code = exc.response.status_code
+        messages = {
+            401: "API Key 验证失败，请检查当前供应商的凭证。",
+            403: "当前 API Key 无权访问该模型目录，请检查账号权限和服务地域。",
+            404: "模型目录接口不存在，请检查基础地址、地域和接口协议。",
+            429: "供应商请求限流，请稍后重试。",
+        }
+        return messages.get(code, f"供应商模型目录返回 HTTP {code}，请稍后重试。")
+    if isinstance(exc, httpx.RequestError):
+        return "无法连接模型目录，请检查基础地址和网络。"
+    return "模型目录响应无效，请稍后重试。"
+
+
+def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
+    if not isinstance(payload, dict):
+        raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
+    envelope = payload.get("output", payload)
+    if not isinstance(envelope, dict):
+        raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
+    rows = envelope.get("data", envelope.get("models"))
+    if not isinstance(rows, list):
+        raise LLMProviderCatalogError("供应商响应中没有模型列表。")
+    if any(not isinstance(row, dict) for row in rows):
+        raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
+    return rows, envelope
+
+
+def _model_id(row: dict[str, object]) -> str:
+    value = row.get("id") or row.get("model") or row.get("name")
+    if not isinstance(value, str) or not value.strip():
+        raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
+    return value.strip()
+
+
+def _model_date(row: dict[str, object]) -> float:
+    value = row.get("created_at") or row.get("published_time") or row.get("created")
+    if isinstance(value, (int, float)):
+        return float(value)
+    if isinstance(value, str):
+        try:
+            return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
+        except ValueError:
+            pass
+    return 0
+
+
+async def _get_catalog_page(
+    client: httpx.AsyncClient,
+    url: str,
+    headers: dict[str, str],
+    params: dict[str, str | int],
+) -> object:
+    for attempt in range(CATALOG_REQUEST_ATTEMPTS):
+        try:
+            response = await client.get(url, headers=headers, params=params)
+            response.raise_for_status()
+            return response.json()
+        except httpx.HTTPStatusError as exc:
+            if attempt or exc.response.status_code not in {502, 503, 504}:
+                raise
+        except httpx.TransportError:
+            if attempt:
+                raise
+        await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
+    raise LLMProviderCatalogError("模型目录请求失败。")
+
+
+async def fetch_model_catalog(
+    provider: str,
+    base_url: str,
+    provider_api: str,
+    api_key: str = "",
+    anthropic_version: str = "2023-06-01",
+    timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
+) -> ModelCatalog:
+    url = model_catalog_url(provider, base_url, provider_api)
+    public_catalog = provider in {"opencode-go", "openrouter"}
+    if not api_key and provider_api != "ollama-generate" and not public_catalog:
+        raise LLMProviderCatalogError("请先配置当前供应商的 API Key，再刷新模型列表。")
+    headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
+    if provider_api == "anthropic-messages" and provider not in {
+        "opencode-go",
+        "openrouter",
+        "alibaba",
+        "moonshotai",
+    }:
+        headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
+    elif api_key:
+        headers["Authorization"] = f"Bearer {api_key}"
+    native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
+    params: dict[str, str | int] = {}
+    if native_dashscope:
+        params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
+    rows_by_id: dict[str, dict[str, object]] = {}
+    timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
+    # Bound the complete pagination/retry cycle, not just each individual request.
+    async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
+        for page in range(CATALOG_MAX_PAGES):
+            payload = await _get_catalog_page(client, url, headers, params)
+            rows, envelope = _model_rows(payload)
+            previous_count = len(rows_by_id)
+            for row in rows:
+                rows_by_id[_model_id(row)] = row
+            has_more = envelope.get("has_more") is True
+            if native_dashscope:
+                total = envelope.get("total")
+                if not isinstance(total, int) or total < 0:
+                    raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
+                has_more = len(rows_by_id) < total
+                params["page_no"] = page + 2
+            elif has_more:
+                cursor = envelope.get("last_id")
+                if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
+                    raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
+                params["after_id"] = cursor
+            if not has_more:
+                models = sorted(
+                    rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
+                )
+                # An empty Ollama catalog is valid: no models have been installed yet.
+                if not models and provider_api != "ollama-generate":
+                    raise LLMProviderCatalogError("供应商返回了空模型目录，已保留上次模型列表。")
+                return ModelCatalog(url=url, models=models)
+            if len(rows_by_id) == previous_count:
+                raise LLMProviderCatalogError("供应商模型目录分页没有进展，已保留上次模型列表。")
+    raise LLMProviderCatalogError("供应商模型目录分页超过限制，已保留上次模型列表。")

backend/app/services/llm_provider_catalog.py

@@ -3,17 +3,30 @@
 from __future__ import annotations
 
 from typing import Any
+from urllib.parse import urlsplit
 
-import httpx
-
-MODELS_DEV_URL = "https://models.dev/api.json"
-OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
+from app.services.llm_model_catalog import fetch_model_catalog
 
 OPENCODE_GO_MODEL_PROVIDER_APIS = {
+    "minimax-m3": "anthropic-messages",
+    "qwen3.8-max": "anthropic-messages",
+    "qwen3.8-flash": "anthropic-messages",
+    "qwen3.7-max": "anthropic-messages",
+    "qwen3.7-plus": "anthropic-messages",
+    "qwen3.6-plus": "anthropic-messages",
+    "grok-4.6": "openai-responses",
+    "gpt-5.6-luna": "openai-responses",
+    "muse-spark-1.3-contributor": "openai-responses",
+    "muse-spark-1.2-contributor": "openai-responses",
     "minimax-m2.7": "anthropic-messages",
     "minimax-m2.5": "anthropic-messages",
 }
 OPENCODE_GO_FALLBACK_MODELS = [
+    "minimax-m3",
+    "kimi-k3",
+    "glm-5.3",
+    "qwen3.8-max",
+    "gpt-5.6-luna",
     "minimax-m2.7",
     "minimax-m2.5",
     "kimi-k2.6",
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
 ]
 
 
+OPENAI_MODEL_PROVIDER_APIS = {
+    model: "openai-responses"
+    for model in [
+        "gpt-6-astra",
+        "gpt-5.6-sol",
+        "gpt-5.6-terra",
+        "gpt-5.6-luna",
+        "gpt-5.1",
+        "gpt-5.1-codex",
+    ]
+}
+
+
 FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
     "minimax": {
         "provider": "minimax",
         "label": "MiniMax",
         "provider_api": "anthropic-messages",
         "base_url": "https://api.minimaxi.com/anthropic",
-        "model": "MiniMax-M2.7",
-        "models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
+        "model": "MiniMax-M3",
+        "models": [
+            "MiniMax-M3",
+            "MiniMax-M2.7",
+            "MiniMax-M2.7-highspeed",
+            "MiniMax-M2.5",
+            "MiniMax-M2.5-highspeed",
+            "MiniMax-M2.1",
+            "MiniMax-M2.1-highspeed",
+            "MiniMax-M2",
+        ],
         "api_key_env": "MINIMAX_API_KEY",
         "source": "fallback",
     },
     "openai": {
         "provider": "openai",
         "label": "OpenAI",
-        "provider_api": "openai-completions",
+        "provider_api": "openai-responses",
         "base_url": "https://api.openai.com/v1",
-        "model": "gpt-5.1",
-        "models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
+        "model": "gpt-6-astra",
+        "models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
+        "model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
         "api_key_env": "OPENAI_API_KEY",
         "source": "fallback",
     },
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
         "label": "Anthropic",
         "provider_api": "anthropic-messages",
         "base_url": "https://api.anthropic.com/v1",
-        "model": "claude-sonnet-4-6",
-        "models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
+        "model": "claude-opus-5",
+        "models": ["claude-opus-5", "claude-sonnet-4-6"],
         "api_key_env": "ANTHROPIC_API_KEY",
         "source": "fallback",
     },
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
         "label": "DeepSeek",
         "provider_api": "openai-completions",
         "base_url": "https://api.deepseek.com/v1",
-        "model": "deepseek-chat",
-        "models": ["deepseek-chat", "deepseek-reasoner"],
+        "model": "deepseek-flash",
+        "models": ["deepseek-flash", "deepseek-v4-pro"],
         "api_key_env": "DEEPSEEK_API_KEY",
         "source": "fallback",
     },
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
         "label": "Alibaba Qwen / DashScope",
         "provider_api": "openai-completions",
         "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
-        "model": "qwen3-max",
-        "models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
+        "model": "qwen3.8-max",
+        "models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
         "api_key_env": "DASHSCOPE_API_KEY",
         "source": "fallback",
     },
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
         "label": "Moonshot AI / Kimi",
         "provider_api": "openai-completions",
         "base_url": "https://api.moonshot.ai/v1",
-        "model": "kimi-k2.5",
-        "models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
+        "model": "kimi-k3",
+        "models": ["kimi-k3", "kimi-k2.5"],
         "api_key_env": "MOONSHOT_API_KEY",
         "source": "fallback",
     },
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
     },
 }
 
-MODELS_DEV_PROVIDER_KEYS = {
-    "minimax": "minimax",
-    "openai": "openai",
-    "anthropic": "anthropic",
-    "deepseek": "deepseek",
-    "alibaba": "alibaba",
-    "moonshotai": "moonshotai",
-    "openrouter": "openrouter",
-}
-
 
 def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
     return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
     }
 
 
-async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
+async def refresh_llm_provider_preset(
+    provider: str,
+    api_key: str | None = None,
+    *,
+    base_url: str | None = None,
+    provider_api: str | None = None,
+    anthropic_version: str = "2023-06-01",
+) -> dict[str, Any]:
     fallback = get_fallback_llm_provider_preset(provider)
-    if fallback["provider"] == "opencode-go":
-        headers = {"User-Agent": "Planet/1.0"}
-        if api_key:
-            headers["Authorization"] = f"Bearer {api_key}"
-        async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
-            response = await client.get(
-                OPENCODE_GO_MODELS_URL,
-                headers=headers,
-            )
-            response.raise_for_status()
-            payload = response.json()
-
-        data = payload.get("data") if isinstance(payload, dict) else []
-        model_ids = [
-            str(item.get("id"))
-            for item in data
-            if isinstance(item, dict) and item.get("id")
-        ][:120]
-        if not model_ids:
-            model_ids = fallback["models"]
-        return {
-            **fallback,
-            "model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
-            "models": model_ids,
-            "model_provider_apis": _opencode_go_model_provider_apis(model_ids),
-            "source": OPENCODE_GO_MODELS_URL,
-        }
-
-    models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
-    if not models_dev_key:
-        return fallback
-
-    async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
-        response = await client.get(
-            MODELS_DEV_URL,
-            headers={"User-Agent": "Planet/1.0"},
-        )
-        response.raise_for_status()
-        catalog = response.json()
-
-    upstream = catalog.get(models_dev_key)
-    if not isinstance(upstream, dict):
-        return fallback
-
-    upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
-    model_ids = list(upstream_models.keys())[:80]
-    base_url = upstream.get("api") or fallback["base_url"]
-    if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
-        base_url = "https://api.deepseek.com/v1"
-
-    refreshed = {
+    resolved_base_url = base_url or fallback["base_url"]
+    resolved_api = provider_api or fallback["provider_api"]
+    catalog = await fetch_model_catalog(
+        fallback["provider"],
+        resolved_base_url,
+        resolved_api,
+        api_key or "",
+        anthropic_version,
+    )
+    model_provider_apis = (
+        _opencode_go_model_provider_apis(catalog.models)
+        if fallback["provider"] == "opencode-go"
+        else fallback.get("model_provider_apis", {})
+    )
+    if (
+        fallback["provider"] == "openai"
+        and urlsplit(resolved_base_url).hostname != "api.openai.com"
+    ):
+        model_provider_apis = {}
+    return {
         **fallback,
-        "label": upstream.get("name") or fallback["label"],
-        "base_url": base_url,
-        "model": model_ids[0] if model_ids else fallback["model"],
-        "models": model_ids or fallback["models"],
-        "api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
-        "source": MODELS_DEV_URL,
+        "base_url": resolved_base_url,
+        "provider_api": resolved_api,
+        "model": catalog.models[0] if catalog.models else "",
+        "models": catalog.models,
+        "model_provider_apis": model_provider_apis,
+        "source": catalog.url,
     }
-    return refreshed

backend/app/services/playground_chat_service.py

@@ -635,6 +635,7 @@ async def _run_assistant_message(
                 constraints=_collect_constraints(payload.constraints),
                 context={
                     "source": "playground",
+                    "session_id": session_id,
                     "preset": payload.selected_preset_key,
                     "conversation_history": conversation_history,
                     "history_size": len(conversation_history),

backend/app/services/scheduler.py

@@ -2,9 +2,11 @@
 
 import asyncio
 from datetime import UTC, datetime, timedelta
+import time
 from typing import Any, Dict, Optional
 
 from apscheduler.schedulers.asyncio import AsyncIOScheduler
+from apscheduler.triggers.date import DateTrigger
 from apscheduler.triggers.interval import IntervalTrigger
 from sqlalchemy import select
 
@@ -14,19 +16,44 @@ from app.db.session import async_session_factory
 from app.core.time import to_iso8601_utc
 from app.models.datasource import DataSource
 from app.models.task import CollectionTask
-from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
+from app.services.business_logs import (
+    emit_business_log,
+    emit_business_log_background,
+    exception_context,
+)
 from app.services.collectors.registry import collector_registry
 from app.services.datasource_connectivity import (
     build_builtin_connectivity_checksum,
     get_builtin_effective_candidate,
     save_connectivity_success,
 )
+from app.services.earth_news_refresh import (
+    REFRESH_INTERVAL_SECONDS as EARTH_NEWS_REFRESH_INTERVAL_SECONDS,
+    create_earth_news_refresh_status,
+    refresh_earth_news_all_regions,
+    refresh_earth_news_region,
+    set_earth_news_refresh_status,
+    touch_earth_news_refresh_status,
+    REFRESH_STATUS_HEARTBEAT_SECONDS as EARTH_NEWS_STATUS_HEARTBEAT_SECONDS,
+)
+from app.services.earth_news import REGION_ANCHORS
 
 logger = get_logger(__name__)
 
 scheduler = AsyncIOScheduler()
 RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
 RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
+EARTH_NEWS_REFRESH_JOB_ID = "earth_news_rss_refresh"
+_pending_earth_news_refreshes: dict[str, dict[str, Any]] = {}
+_running_earth_news_refreshes: set[str] = set()
+_rerun_earth_news_refreshes: dict[str, dict[str, Any]] = {}
+EARTH_NEWS_REFRESH_WAIT_SECONDS = 300
+EARTH_NEWS_REFRESH_POLL_SECONDS = 1
+EARTH_NEWS_REFRESH_PUBLIC_ERRORS = {
+    "source_unavailable": "No news source could be refreshed for this region. Please retry.",
+    "timeout": "News sources timed out. Please try again.",
+    "refresh_failed": "Could not refresh news. Please try again.",
+}
 
 
 def _collector_task_name(collector_name: str) -> str:
@@ -86,7 +113,10 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
         logger.info_event(
             "Scheduled collector",
             event="collector.schedule.updated",
-            context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
+            context={
+                "collector_name": datasource.source,
+                "frequency_minutes": datasource.frequency_minutes,
+            },
         )
     else:
         logger.info_event(
@@ -133,7 +163,11 @@ async def run_collector_task(collector_name: str):
                 category="collector",
                 service="scheduler",
                 module=__name__,
-                context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource.id,
+                    "status": "skipped",
+                },
             )
             return
 
@@ -153,9 +187,8 @@ async def run_collector_task(collector_name: str):
             if started_at is not None and started_at.tzinfo is None:
                 started_at = started_at.replace(tzinfo=UTC)
 
-            is_stale = (
-                started_at is not None
-                and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
+            is_stale = started_at is not None and (now - started_at) > timedelta(
+                minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES
             )
             if not is_stale:
                 logger.warning_event(
@@ -189,9 +222,7 @@ async def run_collector_task(collector_name: str):
             existing_running.phase = "failed"
             existing_running.completed_at = now
             existing_running.error_message = (
-                f"{existing_error}\n{stale_reason}".strip()
-                if existing_error
-                else stale_reason
+                f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
             )
             await db.commit()
             logger.warning_event(
@@ -231,7 +262,11 @@ async def run_collector_task(collector_name: str):
                 category="collector",
                 service="scheduler",
                 module=__name__,
-                context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource_id,
+                    "status": "running",
+                },
             )
             task_result = await collector.run(db)
             datasource = await db.get(DataSource, datasource_id)
@@ -265,7 +300,11 @@ async def run_collector_task(collector_name: str):
             logger.info_event(
                 "Collector completed",
                 event="collector.run.completed",
-                context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource_id,
+                    "result": task_result,
+                },
             )
             await emit_business_log(
                 logger,
@@ -300,7 +339,11 @@ async def run_collector_task(collector_name: str):
                 level="warning",
                 service="scheduler",
                 module=__name__,
-                context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource.id,
+                    "status": "cancelled",
+                },
             )
             raise
         except Exception as exc:
@@ -312,7 +355,11 @@ async def run_collector_task(collector_name: str):
             logger.exception_event(
                 "Collector failed",
                 event="collector.run.failed",
-                context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
+                context={
+                    "collector_name": collector_name,
+                    "datasource_id": datasource.id,
+                    "error": str(exc),
+                },
             )
             await emit_business_log(
                 logger,
@@ -324,7 +371,11 @@ async def run_collector_task(collector_name: str):
                 module=__name__,
                 context=exception_context(
                     exc,
-                    {"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
+                    {
+                        "collector_name": collector_name,
+                        "datasource_id": datasource.id,
+                        "status": "failed",
+                    },
                 ),
             )
 
@@ -349,7 +400,9 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
             task.completed_at = datetime.now(UTC)
             existing_error = (task.error_message or "").strip()
             cleanup_error = "Marked failed automatically after stale running task cleanup"
-            task.error_message = f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
+            task.error_message = (
+                f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
+            )
 
         if stale_tasks:
             await db.commit()
@@ -365,6 +418,17 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
 def start_scheduler() -> None:
     """Start the scheduler."""
     if not scheduler.running:
+        scheduler.add_job(
+            refresh_earth_news_all_regions,
+            trigger=IntervalTrigger(seconds=EARTH_NEWS_REFRESH_INTERVAL_SECONDS),
+            id=EARTH_NEWS_REFRESH_JOB_ID,
+            name="Earth news RSS refresh",
+            replace_existing=True,
+            next_run_time=datetime.now(UTC),
+            max_instances=1,
+            coalesce=True,
+            misfire_grace_time=60,
+        )
         scheduler.start()
         logger.info_event("Scheduler started", event="scheduler.started")
 
@@ -376,6 +440,205 @@ def stop_scheduler() -> None:
         logger.info_event("Scheduler stopped", event="scheduler.stopped")
 
 
+def schedule_earth_news_region_refresh(
+    region: str, *, force: bool = False, refresh_id: str | None = None
+) -> bool:
+    """Queue one immediate, per-region RSS refresh and merge duplicate requests."""
+    normalized_region = str(region or "").strip().lower()
+    if normalized_region not in REGION_ANCHORS:
+        raise ValueError(f"Unsupported Earth news region: {region}")
+
+    if normalized_region in _running_earth_news_refreshes:
+        rerun = _rerun_earth_news_refreshes.setdefault(
+            normalized_region, {"force": False, "refresh_id": None}
+        )
+        rerun["force"] = rerun["force"] or force
+        rerun["refresh_id"] = refresh_id or rerun["refresh_id"]
+        return False
+    if normalized_region in _pending_earth_news_refreshes:
+        pending = _pending_earth_news_refreshes[normalized_region]
+        pending["force"] = pending["force"] or force
+        pending["refresh_id"] = refresh_id or pending["refresh_id"]
+        return False
+
+    _pending_earth_news_refreshes[normalized_region] = {
+        "force": force,
+        "refresh_id": refresh_id,
+    }
+    job_id = _earth_news_region_refresh_job_id(normalized_region)
+    try:
+        scheduler.add_job(
+            _run_earth_news_region_refresh,
+            trigger=DateTrigger(run_date=datetime.now(UTC)),
+            id=job_id,
+            name=f"Earth news RSS refresh: {normalized_region}",
+            kwargs={"region": normalized_region},
+            replace_existing=False,
+            max_instances=1,
+            coalesce=True,
+            misfire_grace_time=60,
+        )
+    except Exception:
+        _pending_earth_news_refreshes.pop(normalized_region, None)
+        raise
+    return True
+
+
+def request_earth_news_region_refresh(region: str) -> dict[str, Any]:
+    normalized_region = str(region or "").strip().lower()
+    if normalized_region not in REGION_ANCHORS:
+        raise ValueError(f"Unsupported Earth news region: {region}")
+    status, created = create_earth_news_refresh_status(normalized_region)
+    if created:
+        try:
+            schedule_earth_news_region_refresh(
+                normalized_region,
+                force=True,
+                refresh_id=status["refresh_id"],
+            )
+        except Exception:
+            return set_earth_news_refresh_status(
+                status["refresh_id"],
+                normalized_region,
+                "failed",
+                error={
+                    "code": "refresh_failed",
+                    "message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
+                },
+            )
+    return status
+
+
+def schedule_earth_news_refresh_for_all_regions(*, force: bool = False) -> None:
+    for region in REGION_ANCHORS:
+        schedule_earth_news_region_refresh(region, force=force)
+
+
+def _earth_news_region_refresh_job_id(region: str) -> str:
+    return f"{EARTH_NEWS_REFRESH_JOB_ID}:{region}"
+
+
+async def _run_earth_news_region_refresh(region: str) -> None:
+    request = _pending_earth_news_refreshes.pop(region, {"force": False, "refresh_id": None})
+    force = bool(request["force"])
+    refresh_id = request["refresh_id"]
+    _running_earth_news_refreshes.add(region)
+    status_heartbeat: asyncio.Task[None] | None = None
+    try:
+        if refresh_id:
+            set_earth_news_refresh_status(refresh_id, region, "running")
+            if not touch_earth_news_refresh_status(refresh_id, region):
+                return
+            status_heartbeat = asyncio.create_task(
+                _earth_news_refresh_status_heartbeat(refresh_id, region)
+            )
+            deadline = time.monotonic() + EARTH_NEWS_REFRESH_WAIT_SECONDS
+            while True:
+                result = await refresh_earth_news_region(region, force=force)
+                if result["state"] != "busy":
+                    break
+                if time.monotonic() >= deadline:
+                    set_earth_news_refresh_status(
+                        refresh_id,
+                        region,
+                        "failed",
+                        error={
+                            "code": "refresh_failed",
+                            "message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
+                        },
+                    )
+                    return
+                await asyncio.sleep(EARTH_NEWS_REFRESH_POLL_SECONDS)
+            if result["state"] == "coordination_unavailable":
+                error_code = "refresh_failed"
+            elif result["state"] == "failed":
+                error_code = result.get("error_code") or "source_unavailable"
+            elif result["state"] == "completed" and result.get("successful_sources", 0) > 0:
+                set_earth_news_refresh_status(refresh_id, region, "succeeded")
+                return
+            else:
+                error_code = "source_unavailable"
+            set_earth_news_refresh_status(
+                refresh_id,
+                region,
+                "failed",
+                error={
+                    "code": error_code,
+                    "message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS.get(
+                        error_code, EARTH_NEWS_REFRESH_PUBLIC_ERRORS["source_unavailable"]
+                    ),
+                },
+            )
+        else:
+            await refresh_earth_news_region(region, force=force)
+    except asyncio.CancelledError:
+        if refresh_id:
+            set_earth_news_refresh_status(
+                refresh_id,
+                region,
+                "failed",
+                error={
+                    "code": "refresh_failed",
+                    "message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
+                },
+            )
+        raise
+    except Exception as exc:
+        logger.warning_event(
+            "Earth news tracked refresh failed",
+            event="earth_news.rss.tracked_refresh_failed",
+            context={"region": region, "error": str(exc)},
+        )
+        if refresh_id:
+            try:
+                set_earth_news_refresh_status(
+                    refresh_id,
+                    region,
+                    "failed",
+                    error={
+                        "code": "refresh_failed",
+                        "message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
+                    },
+                )
+            except Exception as status_exc:
+                logger.warning_event(
+                    "Earth news refresh status update failed",
+                    event="earth_news.rss.refresh_status_update_failed",
+                    context={"region": region, "error": str(status_exc)},
+                )
+    finally:
+        if status_heartbeat is not None:
+            status_heartbeat.cancel()
+            try:
+                await status_heartbeat
+            except asyncio.CancelledError:
+                pass
+        _running_earth_news_refreshes.discard(region)
+        rerun = _rerun_earth_news_refreshes.pop(region, None)
+        if rerun is not None and scheduler.running:
+            schedule_earth_news_region_refresh(
+                region,
+                force=rerun["force"],
+                refresh_id=rerun["refresh_id"],
+            )
+
+
+async def _earth_news_refresh_status_heartbeat(refresh_id: str, region: str) -> None:
+    try:
+        while True:
+            await asyncio.sleep(EARTH_NEWS_STATUS_HEARTBEAT_SECONDS)
+            if not touch_earth_news_refresh_status(refresh_id, region):
+                return
+    except asyncio.CancelledError:
+        raise
+    except Exception as exc:
+        logger.warning_event(
+            "Earth news refresh status heartbeat failed",
+            event="earth_news.rss.refresh_status_heartbeat_failed",
+            context={"region": region, "error": str(exc)},
+        )
+
+
 async def sync_scheduler_with_datasources() -> None:
     """Synchronize scheduler jobs with datasource table."""
     async with async_session_factory() as db:
@@ -384,7 +647,10 @@ async def sync_scheduler_with_datasources() -> None:
 
         configured_sources = {datasource.source for datasource in datasources}
         for job in list(scheduler.get_jobs()):
-            if job.id not in configured_sources:
+            is_earth_news_refresh = job.id == EARTH_NEWS_REFRESH_JOB_ID or job.id.startswith(
+                f"{EARTH_NEWS_REFRESH_JOB_ID}:"
+            )
+            if job.id not in configured_sources and not is_earth_news_refresh:
                 scheduler.remove_job(job.id)
 
         for datasource in datasources:
@@ -461,7 +727,9 @@ def run_collector_now(collector_name: str) -> bool:
         return False
 
     try:
-        task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
+        task = asyncio.create_task(
+            run_collector_task(collector_name), name=_collector_task_name(collector_name)
+        )
         RUNNING_COLLECTOR_TASKS[collector_name] = task
 
         def _cleanup_task(done_task: asyncio.Task[Any]) -> None:

backend/app/services/tv_catalog.py

@@ -0,0 +1,148 @@
+"""Search and paginate the public live TV catalog at the database boundary."""
+
+from typing import Any
+
+from sqlalchemy import Select, func, select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.core.time import to_iso8601_utc
+from app.models.collected_data import CollectedData
+from app.services.tv_streams import (
+    TV_LIVE_SOURCE_COLLECTOR,
+    TV_LIVE_SOURCE_DATA_TYPE,
+    _build_collected_tv_source,
+    build_public_tv_payload,
+    get_tv_settings_payload,
+)
+
+
+def _source_key():
+    return func.coalesce(
+        func.nullif(CollectedData.extra_data["id"].as_string(), ""),
+        func.nullif(CollectedData.source_id, ""),
+        CollectedData.entity_key,
+    )
+
+
+def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
+    metadata = CollectedData.extra_data
+    source_id = _source_key()
+    enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
+    ranked = (
+        select(
+            CollectedData.id,
+            func.row_number()
+            .over(
+                partition_by=source_id,
+                order_by=CollectedData.id.desc(),
+            )
+            .label("source_rank"),
+        )
+        .where(
+            CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
+            CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
+            CollectedData.is_current.is_(True),
+            CollectedData.is_valid == 1,
+            CollectedData.deleted_at.is_(None),
+            enabled.notin_(("false", "0", "no", "off")),
+            source_id.notin_(configured_ids),
+        )
+        .subquery()
+    )
+    return (
+        select(CollectedData)
+        .join(ranked, ranked.c.id == CollectedData.id)
+        .where(ranked.c.source_rank == 1)
+    )
+
+
+def _filter_catalog_query(
+    query: Select[tuple[CollectedData]], terms: list[str]
+) -> Select[tuple[CollectedData]]:
+    metadata = CollectedData.extra_data
+    searchable = func.lower(
+        func.concat_ws(
+            " ",
+            CollectedData.name,
+            CollectedData.title,
+            CollectedData.source_id,
+            metadata["name"].as_string(),
+            metadata["provider"].as_string(),
+            metadata["region"].as_string(),
+            metadata["country"].as_string(),
+            metadata["language"].as_string(),
+        )
+    )
+    for term in terms:
+        query = query.where(searchable.contains(term, autoescape=True))
+    return query
+
+
+def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
+    searchable = " ".join(
+        str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
+    ).lower()
+    return all(term in searchable for term in terms)
+
+
+async def get_tv_catalog_page(
+    db: AsyncSession,
+    *,
+    offset: int = 0,
+    limit: int = 50,
+    q: str = "",
+    selected_id: str | None = None,
+) -> dict[str, Any]:
+    settings = await get_tv_settings_payload(db)
+    payload = build_public_tv_payload(settings, [])
+    configured = payload["sources"]
+    query = _collected_catalog_query([source["id"] for source in settings["sources"]])
+    if selected_id:
+        selected = next((source for source in configured if source["id"] == selected_id), None)
+        if selected is None:
+            record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
+            selected = _build_collected_tv_source(record, 0) if record else None
+        if selected:
+            payload["selected_source"] = selected
+    summary = (
+        await db.execute(
+            select(func.count(), func.max(CollectedData.collected_at))
+            .select_from(CollectedData)
+            .where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
+        )
+    ).one()
+    total_collected, latest_update = summary
+    terms = q.lower().split()
+    matched_configured = [source for source in configured if _matches_source(source, terms)]
+    filtered_query = _filter_catalog_query(query, terms)
+    matched_collected = (
+        await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
+        if terms
+        else total_collected
+    )
+    sources = matched_configured[offset : offset + limit]
+    remaining = limit - len(sources)
+    if remaining:
+        rows = await db.scalars(
+            filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
+            .offset(max(0, offset - len(matched_configured)))
+            .limit(remaining)
+        )
+        sources.extend(
+            _build_collected_tv_source(record, index) for index, record in enumerate(rows)
+        )
+    total = len(matched_configured) + matched_collected
+    next_offset = offset + len(sources)
+    return {
+        **payload,
+        "sources": sources,
+        "source_count": len(configured) + total_collected,
+        "latest_updated_at": (
+            to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
+        ),
+        "total": total,
+        "offset": offset,
+        "limit": limit,
+        "has_more": next_offset < total,
+        "next_offset": next_offset if next_offset < total else None,
+    }

backend/app/services/tv_streams.py

@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
 from app.models.collected_data import CollectedData
 from app.models.system_setting import SystemSetting
 
-DEFAULT_TV_SOURCE_ID = "cgtn-en"
+DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
 TV_SETTINGS_CATEGORY = "tv"
 TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
 TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
+CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
+CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
+UNSUPPORTED_CCTV4_STREAM = (
+    "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
+)
+CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版，使用 WCETV HLS 直播源；节目编排可能与亚洲版不同。"
 
 DEFAULT_TV_SETTINGS = {
     "default_source_id": DEFAULT_TV_SOURCE_ID,
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
     "sources": [
         {
             "id": "cctv4",
-            "name": "CCTV-4 中文国际",
+            "name": "CCTV-4 中文国际（美洲版）",
             "provider": "CCTV",
             "region": "China",
             "language": "zh-CN",
             "source_type": "hls",
-            "embed_url": "https://tv.cctv.com/live/cctv4/",
-            "stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
-            "homepage_url": "https://tv.cctv.com/live/cctv4/",
+            "embed_url": "",
+            "stream_url": CCTV4_STREAM,
+            "use_proxy": False,
+            "homepage_url": CCTV4_HOMEPAGE,
             "poster_url": "",
             "is_enabled": True,
             "is_fallback": True,
             "sort_order": 10,
             "collector_source": None,
-            "notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流，若直播放失败则回退到央视官网直播页。",
+            "notes": CCTV4_PLAYBACK_NOTE,
         },
         {
             "id": "reuters-tv",
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
 
 def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
     payload = dict(source or {})
+    # Retire only the formerly bundled CDRM URL, including saved copies of that default.
+    # HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
+    if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
+        payload.get("stream_url")
+    ) == UNSUPPORTED_CCTV4_STREAM:
+        payload.update(
+            name="CCTV-4 中文国际（美洲版）",
+            source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
+            use_proxy=False,
+            homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
+        )
     source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
     source_type = _clean_text(payload.get("source_type")).lower()
     youtube_video_id = _clean_text(payload.get("youtube_video_id"))
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
         "source_type": source_type,
         "embed_url": _clean_url(payload.get("embed_url")),
         "stream_url": _clean_url(payload.get("stream_url")),
+        "use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
         "homepage_url": _clean_url(payload.get("homepage_url")),
         "poster_url": _clean_url(payload.get("poster_url")),
         "youtube_video_id": youtube_video_id,
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
     ]
 
     if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
+        default_source = next(
+            source for source in DEFAULT_TV_SETTINGS["sources"]
+            if source["id"] == DEFAULT_TV_SOURCE_ID
+        )
         normalized_sources.append(
-            normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
+            normalize_tv_source(default_source, index=len(normalized_sources))
         )
 
     default_source_exists = any(

backend/app/services/vessel_ais_aggregation.py

@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
     return [item.to_dict() for item in result.scalars().all()]
 
 
+async def get_current_vessels_by_mmsi(
+    db: AsyncSession, mmsis: list[int]
+) -> list[dict[str, Any]]:
+    """Read canonical render state for the vessels changed by a stream flush."""
+    if not mmsis:
+        return []
+    result = await db.execute(
+        select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
+    )
+    return [_jsonable(item.to_dict()) for item in result.scalars().all()]
+
+
 async def aggregate_vessel_observations(
     db: AsyncSession,
     observations: Iterable[AISRawObservation],

backend/tests/conftest.py

@@ -8,6 +8,37 @@ from unittest.mock import AsyncMock, MagicMock
 from sqlalchemy.ext.asyncio import AsyncSession
 
 
+@pytest.fixture(autouse=True)
+def persisted_test_logs(monkeypatch):
+    """Keep application log persistence in memory, including business-log calls."""
+    from app.models.system_log import ObservabilityEventGroup
+    from app.services import persistent_logs
+
+    entries = []
+    groups = {}
+
+    class LogSession:
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, exc_type, exc, traceback):
+            return False
+
+        def add(self, item):
+            entries.append(item)
+            if isinstance(item, ObservabilityEventGroup):
+                groups[item.fingerprint] = item
+
+        async def get(self, model, key):
+            return groups.get(key) if model is ObservabilityEventGroup else None
+
+        async def commit(self):
+            return None
+
+    monkeypatch.setattr(persistent_logs, "async_session_factory", LogSession)
+    return entries
+
+
 @pytest.fixture(autouse=True)
 def bgp_collector_location_cache():
     """Mirror app startup seeding for tests that call sync BGP helpers."""

backend/tests/test_auth_identity.py

@@ -0,0 +1,72 @@
+from datetime import UTC, datetime
+from unittest.mock import AsyncMock, MagicMock
+
+import httpx
+import pytest
+from fastapi import FastAPI
+from pydantic import ValidationError
+
+from app.api.v1.auth import router
+from app.core import security
+from app.db.session import DEFAULT_LOGIN_USERS, get_db
+from app.models.user import User
+from app.schemas.user import UserRegister
+
+
+async def identity_request(monkeypatch, user, token=None):
+    result = MagicMock()
+    result.scalar_one_or_none.return_value = user
+    db = AsyncMock()
+    db.execute.return_value = result
+    monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
+    app = FastAPI()
+    app.include_router(router, prefix="/auth")
+
+    async def override_db():
+        yield db
+
+    app.dependency_overrides[get_db] = override_db
+    async with httpx.AsyncClient(
+        transport=httpx.ASGITransport(app=app), base_url="http://test"
+    ) as client:
+        return await client.get(
+            "/auth/me",
+            headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
+        )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
+async def test_default_identity_preserves_profile_fields(monkeypatch, default):
+    created = datetime(2026, 9, 21, tzinfo=UTC)
+    user = User(
+        id=1, username=default["username"], email=default["email"],
+        role=default["role"], password_hash="not-exposed", is_active=True,
+        email_verified=True, gatekeeper_groups=["ops"], created_at=created,
+    )
+    response = await identity_request(monkeypatch, user)
+    assert response.status_code == 200
+    payload = response.json()
+    assert payload["email"] == default["email"]
+    assert payload["email_verified"] is True
+    assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
+    assert payload["gatekeeper_groups"] == ["ops"]
+    assert "password_hash" not in payload
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
+async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
+    response = await identity_request(monkeypatch, user)
+    assert response.status_code == 401
+
+
+@pytest.mark.asyncio
+async def test_refresh_token_is_not_an_access_identity(monkeypatch):
+    response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
+    assert response.status_code == 401
+
+
+def test_registration_still_validates_email():
+    with pytest.raises(ValidationError):
+        UserRegister(username="newuser", email="newuser@planet.local", password="example-password")

backend/tests/test_bgp.py

@@ -1,5 +1,6 @@
 """Tests for BGP observability helpers."""
 
+import ipaddress
 from datetime import UTC, datetime, timedelta
 from types import SimpleNamespace
 
@@ -21,13 +22,20 @@ from app.services.collectors.bgp_common import (
     create_bgp_anomalies_for_batch,
     save_bgp_observations_for_batch,
 )
-from app.services.bgp_enrichment import enrich_bgp_events_for_batch, extract_bgp_network_fields
+from app.services.bgp_enrichment import (
+    enrich_bgp_events_for_batch,
+    extract_bgp_network_fields,
+    lookup_prefix_geography,
+)
 from app.services.bgp_incidents import (
     create_bgp_incidents_for_anomalies,
     infer_related_infrastructure,
 )
 from app.services.bgp_collectors import build_bgp_collector_coverage
-from app.api.v1.visualization import convert_bgp_incidents_to_geojson, build_incident_geography_hints
+from app.api.v1.visualization import (
+    convert_bgp_incidents_to_geojson,
+    build_incident_geography_hints,
+)
 from app.models.bgp_anomaly import BGPAnomaly
 from app.models.collected_data import CollectedData
 from app.models.bgp_incident import BGPIncident
@@ -58,8 +66,7 @@ class _FakeResult:
     def all(self):
         if self._rows and all(isinstance(row, BGPObservation) for row in self._rows):
             return [
-                (row.prefix, row.origin_asn, row.collector, row.collector_geo)
-                for row in self._rows
+                (row.prefix, row.origin_asn, row.collector, row.collector_geo) for row in self._rows
             ]
         return self._rows
 
@@ -93,6 +100,8 @@ class _FakeAsyncSession:
         self.commits = 0
 
     async def execute(self, _stmt, _params=None):
+        if str(_stmt).lstrip().startswith("SET LOCAL jit"):
+            return _FakeResult([])
         if not self._results:
             return _FakeResult([])
         return _FakeResult(self._results.pop(0))
@@ -107,6 +116,49 @@ class _FakeAsyncSession:
         self.commits += 1
 
 
+class _StrictFakeAsyncSession(_FakeAsyncSession):
+    async def execute(self, stmt, params=None):
+        if not str(stmt).lstrip().startswith("SET LOCAL jit") and not self._results:
+            raise AssertionError(f"Unexpected BGP query: {stmt}")
+        return await super().execute(stmt, params)
+
+
+@pytest.mark.asyncio
+async def test_prefix_geography_keeps_source_priority_and_ipv6_range_parameters():
+    calls = []
+
+    class GeoSession:
+        async def execute(self, _stmt, params=None):
+            if str(_stmt).lstrip().startswith("SET LOCAL jit"):
+                return _FakeResult([])
+            calls.append(dict(params))
+            if params["source"] == "nro_delegated_prefix_geo":
+                return _FakeResult(
+                    [
+                        {
+                            "metadata": {
+                                "country_code": "GB",
+                                "prefix": "2001:db8::/32",
+                            }
+                        }
+                    ]
+                )
+            return _FakeResult([])
+
+    network = ipaddress.ip_network("2001:db8:1::/48")
+    result = await lookup_prefix_geography(GeoSession(), [str(network), "invalid-prefix"])
+
+    assert result[str(network)]["source"] == "nro_delegated_stats"
+    assert [call["source"] for call in calls] == [
+        "opengeofeed_prefix_geo",
+        "iptoasn_prefix_geo",
+        "nro_delegated_prefix_geo",
+    ]
+    assert all(call["family"] == "ipv6" for call in calls)
+    assert all(call["range_start"] == str(network.network_address) for call in calls)
+    assert all(call["range_end"] == str(network.broadcast_address) for call in calls)
+
+
 def test_normalize_bgp_event_from_live_payload():
     event = normalize_bgp_event(
         {
@@ -449,7 +501,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
                 "prefix": "203.0.113.0/24",
                 "origin_asn": 64496,
                 "as_path": [64500, 64496],
-                "collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
+                "collector_location": {
+                    "country": "NL",
+                    "city": "Amsterdam",
+                    "latitude": 52.3,
+                    "longitude": 4.9,
+                },
                 "enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
             }
         },
@@ -460,7 +517,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
                 "prefix": "203.0.113.0/24",
                 "origin_asn": 64496,
                 "as_path": [64510, 64520, 64530, 64540, 64496],
-                "collector_location": {"country": "GB", "city": "London", "latitude": 51.5, "longitude": -0.1},
+                "collector_location": {
+                    "country": "GB",
+                    "city": "London",
+                    "latitude": 51.5,
+                    "longitude": -0.1,
+                },
                 "enrichment": {"prefix_scope": {"regions": [{"country": "GB"}]}},
             }
         },
@@ -481,7 +543,9 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
 def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
     base_timestamp = datetime(2026, 3, 27, 0, 0, tzinfo=UTC)
     events = []
-    for index, event_type in enumerate(["announcement", "withdrawal", "announcement", "withdrawal"]):
+    for index, event_type in enumerate(
+        ["announcement", "withdrawal", "announcement", "withdrawal"]
+    ):
         events.append(
             {
                 "metadata": {
@@ -491,7 +555,12 @@ def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
                     "prefix": "198.51.100.0/24",
                     "origin_asn": 64512,
                     "as_path": [64500 + index, 64512] if event_type == "announcement" else [],
-                    "collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
+                    "collector_location": {
+                        "country": "NL",
+                        "city": "Amsterdam",
+                        "latitude": 52.3,
+                        "longitude": 4.9,
+                    },
                     "enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
                 }
             }
@@ -920,7 +989,9 @@ async def test_create_bgp_incidents_for_anomalies_refreshes_existing_incident():
 
     with patch(
         "app.services.bgp_incidents.infer_related_infrastructure",
-        new=AsyncMock(return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}),
+        new=AsyncMock(
+            return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}
+        ),
     ):
         created = await create_bgp_incidents_for_anomalies(
             db,
@@ -1132,13 +1203,17 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
         source="ris_live_bgp",
         extra_data={"prefix": "203.0.113.0/24", "origin_asn": 64496},
     )
-    db = _FakeAsyncSession([
-        [],
-        [],
-        [],
-        [previous_record],
-        [],
-    ])
+    db = _StrictFakeAsyncSession(
+        [
+            [],  # Historical BGP observations
+            [],  # OpenGeoFeed prefix geography
+            [],  # IPtoASN prefix geography
+            [],  # NRO delegated prefix geography
+            [],  # PeeringDB ASN profiles
+            [previous_record],  # Prior collected snapshot for origin-change detection
+            [],  # Existing anomaly entity keys
+        ]
+    )
     events = [
         {
             "reference_date": "2026-03-30T10:00:00Z",
@@ -1178,6 +1253,7 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
     assert anomaly.anomaly_type == "origin_change"
     assert anomaly.prefix == "203.0.113.0/24"
     incident_mock.assert_awaited_once()
+    assert not db._results
 
 
 @pytest.mark.asyncio
@@ -1197,14 +1273,18 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
         origin_asn=64496,
         new_origin_asn=64497,
     )
-    db = _FakeAsyncSession([
-        [],
-        [],
-        [],
-        [previous_record],
-        [existing_key],
-        [existing_anomaly],
-    ])
+    db = _StrictFakeAsyncSession(
+        [
+            [],  # Historical BGP observations
+            [],  # OpenGeoFeed prefix geography
+            [],  # IPtoASN prefix geography
+            [],  # NRO delegated prefix geography
+            [],  # PeeringDB ASN profiles
+            [previous_record],  # Prior collected snapshot for origin-change detection
+            [existing_key],
+            [existing_anomaly],
+        ]
+    )
     events = [
         {
             "reference_date": "2026-03-30T10:00:00Z",
@@ -1235,6 +1315,7 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
     assert created == 0
     assert len(db.added) == 0
     incident_mock.assert_awaited_once()
+    assert not db._results
 
 
 async def _bgp_test_client(db_session):
@@ -1242,7 +1323,9 @@ async def _bgp_test_client(db_session):
         yield db_session
 
     def override_get_current_user():
-        return User(id=1, username="testuser", email="test@example.com", password_hash="x", role="admin")
+        return User(
+            id=1, username="testuser", email="test@example.com", password_hash="x", role="admin"
+        )
 
     app.dependency_overrides[get_db] = override_get_db
     app.dependency_overrides[get_current_user] = override_get_current_user
@@ -1264,7 +1347,7 @@ async def test_bgp_events_api_lists_observations():
         origin_asn=64496,
         observed_at=datetime(2026, 3, 30, 10, 0, tzinfo=UTC),
     )
-    db = _FakeAsyncSession([[observation]])
+    db = _StrictFakeAsyncSession([[1], [observation]])
     client = await _bgp_test_client(db)
 
     try:
@@ -1278,6 +1361,7 @@ async def test_bgp_events_api_lists_observations():
     assert payload["total"] == 1
     assert payload["data"][0]["collector"] == "rrc00"
     assert payload["data"][0]["prefix"] == "203.0.113.0/24"
+    assert not db._results
 
 
 @pytest.mark.asyncio
@@ -1376,7 +1460,18 @@ async def test_bgp_event_summary_api_returns_aggregates():
         event_type="withdrawal",
         observed_at=datetime(2026, 3, 30, 10, 5, tzinfo=UTC),
     )
-    db = _FakeAsyncSession([[observation_one, observation_two]])
+    observations = [observation_one, observation_two]
+    db = _StrictFakeAsyncSession(
+        [
+            [len(observations)],
+            [len({record.collector for record in observations})],
+            [len({record.prefix for record in observations})],
+            [
+                (event_type, sum(record.event_type == event_type for record in observations))
+                for event_type in sorted({record.event_type for record in observations})
+            ],
+        ]
+    )
     client = await _bgp_test_client(db)
 
     try:
@@ -1392,6 +1487,7 @@ async def test_bgp_event_summary_api_returns_aggregates():
     assert payload["prefix_count"] == 2
     assert payload["by_type"]["announcement"] == 1
     assert payload["by_type"]["withdrawal"] == 1
+    assert not db._results
 
 
 @pytest.mark.asyncio

backend/tests/test_bgp_prefix_geo.py

@@ -0,0 +1,22 @@
+from unittest.mock import AsyncMock
+
+import pytest
+
+from app.db.bgp_prefix_geo import (
+    BGP_PREFIX_GEO_FUNCTION_SQL,
+    BGP_PREFIX_GEO_INDEX_SQL,
+    install_bgp_prefix_geo_functions,
+)
+
+
+@pytest.mark.asyncio
+async def test_fresh_database_installs_lookup_functions_without_large_index():
+    connection = AsyncMock()
+
+    await install_bgp_prefix_geo_functions(connection)
+
+    statements = [str(call.args[0]) for call in connection.execute.call_args_list]
+    assert len(statements) == len(BGP_PREFIX_GEO_FUNCTION_SQL) == 2
+    assert all("CREATE OR REPLACE FUNCTION" in statement for statement in statements)
+    assert all("CREATE INDEX" not in statement for statement in statements)
+    assert "CREATE INDEX CONCURRENTLY" in BGP_PREFIX_GEO_INDEX_SQL

backend/tests/test_card_images.py

@@ -0,0 +1,333 @@
+import io
+from copy import deepcopy
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from fastapi import FastAPI, HTTPException
+from httpx import ASGITransport, AsyncClient
+from PIL import Image
+from pydantic import ValidationError
+
+from app.api.v1.card_images import router
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
+from app.services.card_image_configuration import (
+    effective_envelope,
+    publish_configuration,
+    resolve_image,
+)
+from app.services.card_image_defaults import bundled_configuration
+from app.services import card_image_files
+
+
+def configuration():
+    assets = [
+        CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
+        for key in ["exact", "model", "group", "category"]
+    ]
+    rules = [
+        CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
+        for key, scope, value in [
+            ("exact", "object", "123"),
+            ("model", "model", "m1"),
+            ("group", "group", "starlink"),
+            ("category", "category", "satellite"),
+        ]
+    ]
+    return CardImageConfiguration(assets=assets, rules=rules)
+
+
+def test_precedence_disabled_fallback_and_explicit_no_image():
+    config = configuration()
+    context = dict(
+        entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
+    )
+    for expected in ["exact", "model", "group", "category"]:
+        assert resolve_image(config, context)["image"]["id"] == expected
+        next(asset for asset in config.assets if asset.id == expected).enabled = False
+    assert resolve_image(config, context)["image"] is None
+    config = configuration()
+    config.rules[0].asset_id = None
+    config.rules[0].suppress = True
+    result = resolve_image(config, context)
+    assert result["suppressed"] and result["image"] is None
+
+
+def test_bundled_series_images_retain_truthful_source_metadata():
+    config, _ = bundled_configuration()
+    result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
+    assert result["image"]["id"] == "starlink-solar-array"
+    pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
+    assert pending.rights_status == "permission_pending"
+    assert (
+        resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
+        == pending.id
+    )
+    assert pending.enabled
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        "javascript:alert(1)",
+        "//evil.example/image.jpg",
+        "/earth/assets/cards/../secret",
+        "https://user:pass@example.org/a.jpg",
+    ],
+)
+def test_unsafe_urls_rejected(url):
+    with pytest.raises(ValidationError):
+        CardImageAsset(id="test", title="test", url=url)
+
+
+def test_duplicate_targets_and_dangling_assets_rejected():
+    data = configuration().model_dump()
+    duplicate = {**data["rules"][0], "id": "another"}
+    data["rules"].append(duplicate)
+    with pytest.raises(ValidationError):
+        CardImageConfiguration.model_validate(data)
+    data = configuration().model_dump()
+    data["assets"] = []
+    with pytest.raises(ValidationError):
+        CardImageConfiguration.model_validate(data)
+
+
+def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
+    defaults, _ = bundled_configuration()
+    payload = {
+        "revision": "custom",
+        "config": {"assets": [], "rules": []},
+        "known_assets": [asset.id for asset in defaults.assets],
+        "known_rules": [rule.id for rule in defaults.rules],
+    }
+    assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
+    payload["known_assets"] = []
+    assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
+
+
+@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
+def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
+    monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
+    output = io.BytesIO()
+    Image.new("RGB", (1600, 900), "blue").save(output, format)
+    asset = card_image_files.store_image(output.getvalue(), "Facility")
+    assert (asset.width, asset.height) == (1600, 900)
+    assert len(list(tmp_path.iterdir())) == 3
+    for size, url in asset.variants.items():
+        with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
+            assert max(image.size) == int(size)
+    assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
+    with pytest.raises(HTTPException):
+        card_image_files.store_image(b"<svg></svg>", "bad")
+    with pytest.raises(HTTPException):
+        card_image_files.uploaded_path("../private")
+
+
+def fake_db(payload):
+    db = AsyncMock()
+    db.add = MagicMock()
+    record = SimpleNamespace(payload=payload)
+    result = MagicMock()
+    result.scalar_one.return_value = record
+    result.scalar_one_or_none.return_value = record
+    db.execute.return_value = result
+    return db, record
+
+
+@pytest.mark.asyncio
+async def test_publish_revision_and_audit():
+    defaults, _ = bundled_configuration()
+    payload = {
+        "revision": "default",
+        "config": defaults.model_dump(mode="json"),
+        "known_assets": [],
+        "known_rules": [],
+    }
+    db, record = fake_db(payload)
+    revision = effective_envelope(payload)["revision"]
+    result = await publish_configuration(
+        db, revision, defaults, SimpleNamespace(id=1, username="admin")
+    )
+    assert result["revision"] != revision
+    assert db.add.call_args.args[0].action == "card_images.update"
+    before = deepcopy(record.payload)
+    with pytest.raises(HTTPException) as error:
+        await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
+    assert error.value.status_code == 409 and record.payload == before
+
+
+@pytest.mark.asyncio
+async def test_configuration_requires_admin_and_validates_http_payload():
+    app = FastAPI()
+    app.include_router(router)
+    defaults, _ = bundled_configuration()
+    db, _ = fake_db(
+        {
+            "revision": "default",
+            "config": defaults.model_dump(mode="json"),
+            "known_assets": [],
+            "known_rules": [],
+        }
+    )
+    app.dependency_overrides[get_db] = lambda: db
+    async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
+        assert (await client.get("/configuration")).status_code == 401
+        app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
+        assert (await client.get("/configuration")).status_code == 403
+        app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
+            role="admin", id=1, username="admin"
+        )
+        assert (await client.get("/configuration")).status_code == 200
+        assert (
+            await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
+        ).status_code == 422
+
+
+@pytest.mark.asyncio
+async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
+    monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
+    app = FastAPI()
+    app.include_router(router)
+    app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
+        role="admin", id=1, username="admin"
+    )
+    buffer = io.BytesIO()
+    Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
+    async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
+        response = await client.post(
+            "/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
+        )
+        assert response.status_code == 200
+        asset = response.json()
+        filename = asset["variants"]["640"].rsplit("/", 1)[1]
+        image = await client.get(f"/files/{filename}")
+        assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
+        assert "immutable" in image.headers["cache-control"]
+        assert (
+            await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
+        ).status_code == 422
+        app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
+        assert (
+            await client.post(
+                "/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
+            )
+        ).status_code == 403
+
+
+def test_all_object_types_have_a_default_without_overriding_exact_suppression():
+    from typing import get_args
+    from app.schemas.card_images import EntityType
+
+    config, _ = bundled_configuration()
+    for entity_type in get_args(EntityType):
+        result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
+        assert result["image"] and result["match"]["value"] == "*"
+    config = config.model_copy(deep=True)
+    config.rules.append(
+        CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
+    )
+    assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
+
+
+def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
+    from app.services.card_image_defaults import previous_bundled_assets
+
+    old = previous_bundled_assets()["starlink-solar-array"][0]
+    payload = {
+        "revision": "old",
+        "config": {"assets": [old.model_dump()], "rules": []},
+        "known_assets": [old.id],
+        "known_rules": [],
+    }
+
+    def starlink():
+        return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
+
+    assert starlink()["url"].endswith("starlink-orbit.webp")
+    payload["config"]["assets"][0]["focus_x"] = 20
+    assert starlink()["focus_x"] == 20
+    assert starlink()["url"].endswith("starlink-orbit.webp")
+    payload["config"]["assets"][0]["enabled"] = False
+    assert starlink()["enabled"] is False
+    payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
+    assert starlink()["url"] == "https://example.org/custom.jpg"
+
+
+def test_compute_categories_and_bgp_get_separate_user_images():
+    config, _ = bundled_configuration()
+    for kind, filename in [
+        ("supercomputer", "supercomputer.png"),
+        ("gpu_cluster", "gpu-cluster.jpg"),
+        ("data_center", "data-center.png"),
+        ("cloud_center", "cloud-center.png"),
+        ("edge_node", "edge-node.png"),
+    ]:
+        result = resolve_image(config, dict(entity_type="compute_center", category=kind))
+        assert result["image"]["url"].endswith(filename)
+    for kind in ("bgp", "bgp_collector"):
+        result = resolve_image(config, dict(entity_type=kind, category=kind))
+        assert result["image"]["url"].endswith(
+            "bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
+        )
+
+
+def test_saved_collector_default_migrates_without_changing_user_bindings():
+    from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
+
+    old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
+    old_asset = previous_bundled_assets()["network-switches"][0]
+    payload = {
+        "revision": "old",
+        "config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
+        "known_assets": [old_asset.id],
+        "known_rules": [old_rule.id],
+    }
+    result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
+    assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
+        "image"
+    ]["url"].endswith("bgp-collector.png")
+    payload["config"]["rules"][0]["enabled"] = False
+    result = effective_envelope(payload)["config"]
+    assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
+
+
+@pytest.mark.asyncio
+async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
+    from app.services.card_image_targets import get_target
+    from sqlalchemy.dialects import postgresql
+
+    db = AsyncMock()
+    result = MagicMock()
+    result.mappings.return_value.first.return_value = {
+        "entity_id": "epoch_ai_gpu:site",
+        "name": "Site",
+        "category": "cloud_center",
+        "group": "Operator",
+        "model": "Model",
+    }
+    db.execute.return_value = result
+    context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
+    query = db.execute.call_args.args[0]
+    sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
+    assert "collected_data.source = 'epoch_ai_gpu'" in sql
+    assert "collected_data.source_id = 'site'" in sql
+    assert context["category"] == "cloud_center" and context["model"] == "Model"
+
+
+def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
+    from app.services.card_image_defaults import previous_bundled_assets
+
+    old = previous_bundled_assets()["compute-gpu-cluster"][0]
+    payload = {
+        "revision": "old",
+        "config": {"assets": [old.model_dump()], "rules": []},
+        "known_assets": [old.id],
+        "known_rules": [],
+    }
+    result = effective_envelope(payload)["config"]
+    upgraded = next(a for a in result["assets"] if a["id"] == old.id)
+    latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
+    assert upgraded["variants"] == latest.variants
+    assert upgraded["url"] == latest.url

backend/tests/test_collector_downloads.py

@@ -0,0 +1,218 @@
+"""Request-level regressions for quota-limited satellite downloads."""
+
+import gzip
+import json
+import os
+import time
+from unittest.mock import AsyncMock
+
+import httpx
+import pytest
+
+from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
+from app.services.collectors.downloads import ResumableFileDownloader
+
+URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
+
+
+def setup_collector(monkeypatch, tmp_path, handler):
+    collector = CelesTrakTLECollector()
+    collector._resolved_url = "https://celestrak.example/gp.php"
+    monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
+    monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
+    monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
+    monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
+    original_client = httpx.AsyncClient
+    monkeypatch.setattr(
+        "app.services.collectors.celestrak.httpx.AsyncClient",
+        lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
+    )
+    return collector
+
+
+def expire(path):
+    old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
+    os.utime(path, (old, old))
+
+
+@pytest.mark.asyncio
+async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
+    requests = []
+
+    def handler(request):
+        requests.append(request)
+        assert request.method == "GET", "HEAD consumes this upstream's download allowance"
+        return httpx.Response(
+            200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
+        )
+
+    collector = setup_collector(monkeypatch, tmp_path, handler)
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
+    assert len(requests) == 1
+    path = collector._downloader.cached_file_path(URL, ".json")
+    expire(path)
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
+    assert requests[1].headers["if-none-match"] == '"v1"'
+
+
+@pytest.mark.asyncio
+async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
+    requests = []
+
+    def handler(request):
+        requests.append(request.method)
+        return httpx.Response(
+            403, text="GP data has not updated since your last successful download"
+        )
+
+    collector = setup_collector(monkeypatch, tmp_path, handler)
+    path = collector._downloader.cached_file_path(URL, ".json")
+    path.write_text('[{"NORAD_CAT_ID":42}]')
+    expire(path)
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
+    assert requests == ["GET"]
+
+
+@pytest.mark.asyncio
+async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
+    collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
+    path = collector._downloader.cached_file_path(URL, ".json")
+    path.write_text('[{"NORAD_CAT_ID":42}]')
+    expire(path)
+    with pytest.raises(RuntimeError, match="validation failed"):
+        await collector.fetch()
+    assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
+
+
+@pytest.mark.asyncio
+async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
+    requests = []
+
+    def handler(request):
+        requests.append(request)
+        return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
+
+    collector = setup_collector(monkeypatch, tmp_path, handler)
+    path, _, meta = collector._downloader._cache_paths(URL, ".json")
+    path.write_text("[]")
+    meta.write_text('{"etag":"old"}')
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
+    assert "if-none-match" not in requests[0].headers
+
+
+@pytest.mark.asyncio
+async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
+    downloader = ResumableFileDownloader(
+        cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
+    )
+    path, _, meta = downloader._cache_paths(URL, ".json")
+    path.parent.mkdir(parents=True)
+    path.write_text('[{"NORAD_CAT_ID":42}]')
+    meta.write_text('{"etag":"v1"}')
+    expire(path)
+    requests = []
+
+    def handler(request):
+        requests.append(request)
+        return httpx.Response(304)
+
+    async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+        for _ in range(2):
+            await downloader.download_file(
+                client,
+                URL,
+                extension=".json",
+                validate_existing=CelesTrakTLECollector._validate_json_file,
+            )
+    assert len(requests) == 1
+    assert requests[0].headers["if-none-match"] == "v1"
+    assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
+
+
+@pytest.mark.asyncio
+async def test_compressed_get_length_uses_decoded_payload(tmp_path):
+    raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
+    body = gzip.compress(raw)
+    downloader = ResumableFileDownloader(
+        cache_namespace="test", cache_root=tmp_path, preflight_head=False
+    )
+    async with httpx.AsyncClient(
+        transport=httpx.MockTransport(
+            lambda request: httpx.Response(
+                200,
+                content=body,
+                headers={"content-encoding": "gzip", "content-length": str(len(body))},
+            )
+        )
+    ) as client:
+        path = await downloader.download_file(
+            client,
+            URL,
+            extension=".json",
+            validate_existing=CelesTrakTLECollector._validate_json_file,
+        )
+    assert path.read_bytes() == raw
+
+
+@pytest.mark.asyncio
+async def test_other_collectors_keep_head_and_range_resume(tmp_path):
+    downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
+    final, part, meta = downloader._cache_paths(URL, ".bin")
+    part.parent.mkdir(parents=True)
+    part.write_bytes(b"abc")
+    meta.write_text('{"etag":"v1"}')
+    requests = []
+
+    def handler(request):
+        requests.append(request)
+        if request.method == "HEAD":
+            return httpx.Response(
+                200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
+            )
+        assert request.headers["range"] == "bytes=3-"
+        return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
+
+    async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+        await downloader.download_file(client, URL, extension=".bin")
+    assert [r.method for r in requests] == ["HEAD", "GET"]
+    assert final.read_bytes() == b"abcdef"
+
+
+@pytest.mark.asyncio
+async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
+    requests = []
+
+    def handler(request):
+        requests.append(request)
+        if request.url.params["GROUP"] == "active":
+            return httpx.Response(
+                403, text="GP data has not updated since your last successful download"
+            )
+        return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
+
+    collector = setup_collector(monkeypatch, tmp_path, handler)
+    monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
+    path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
+    path.write_text('[{"NORAD_CAT_ID":42}]')
+    expire(path)
+    assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
+    assert [r.method for r in requests] == ["GET", "GET"]
+
+
+@pytest.mark.asyncio
+async def test_generic_cache_without_validators_must_refresh(tmp_path):
+    downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
+    path = downloader.cached_file_path(URL, ".json")
+    path.parent.mkdir(parents=True)
+    path.write_text("[]")
+    requests = []
+
+    def handler(request):
+        requests.append(request.method)
+        return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
+
+    async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
+        await downloader.download_file(client, URL, extension=".json")
+    assert requests == ["HEAD", "GET"]
+    assert path.read_text() == "[1]"

backend/tests/test_collectors.py

@@ -79,7 +79,8 @@ class TestTOP500Collector:
         """Test parsing empty response"""
         collector = TOP500Collector()
         result = collector.parse_response("<html><body><table></table></body></html>")
-        assert len(result) > 0
+        assert result == []
+        assert collector.fail_on_empty
 
     def test_parse_response_single_item(self):
         """Test parsing single item response"""
@@ -97,9 +98,9 @@ class TestTOP500Collector:
           </tr>
         </table>
         """
-        result = collector.parse_response(response)
+        result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
         assert len(result) == 1
-        assert result[0]["source_id"] == "top500_1"
+        assert result[0]["source_id"] == "top500_system_1"
         assert result[0]["name"] == "Test Supercomputer"
         assert result[0]["country"] == "USA"
         assert result[0]["metadata"]["rank"] == 1
@@ -113,7 +114,7 @@ class TestTOP500Collector:
           <tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
           <tr>
             <td>1</td>
-            <td>Valid\nVendor\nSite\nUSA</td>
+            <td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
             <td>1000</td>
             <td>10 PFLOP/s</td>
             <td>12 PFLOP/s</td>
@@ -129,7 +130,7 @@ class TestTOP500Collector:
           </tr>
         </table>
         """
-        result = collector.parse_response(response)
+        result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
         assert len(result) == 1
         assert result[0]["name"] == "Valid"
 

backend/tests/test_compute_configuration.py

@@ -0,0 +1,222 @@
+from copy import deepcopy
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from app.api.v1.compute_intelligence import configuration_admin, get_series, router
+from app.core.security import get_current_user
+from app.db.session import get_db
+from app.models.system_log import AuditLog
+from app.schemas.compute_configuration import ComputeConfiguration
+from app.services.compute_configuration import (
+    DEFAULT_CONFIGURATION,
+    default_envelope,
+    publish_configuration,
+    read_configuration,
+)
+from app.services.compute_history import compute_history
+from fastapi import FastAPI, HTTPException, Response
+from httpx import ASGITransport, AsyncClient
+from pydantic import ValidationError
+
+
+def default_config():
+    return DEFAULT_CONFIGURATION.model_dump(mode="json")
+
+
+@pytest.mark.parametrize(
+    "mutation",
+    [
+        lambda value: value["observations"][0]["values"].update(CN=101),
+        lambda value: value["observations"][0]["values"].update(CN=-1),
+        lambda value: value["observations"][0]["values"].update(CN=float("nan")),
+        lambda value: value["observations"][0]["values"].update(CN=True),
+        lambda value: value["observations"][0]["values"].update(EUROPE=10),
+        lambda value: value["observations"][0].update(source_id="unknown"),
+        lambda value: value["observations"][0].update(year=2026),
+        lambda value: value["observations"].append(value["observations"][0]),
+        lambda value: value["sources"].append(value["sources"][0]),
+        lambda value: value["innovation"].append(value["innovation"][0]),
+        lambda value: value["sources"][0].update(url="javascript:alert(1)"),
+        lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
+    ],
+)
+def test_rejects_invalid_or_ambiguous_evidence(mutation):
+    payload = default_config()
+    mutation(payload)
+    with pytest.raises(ValidationError):
+        ComputeConfiguration.model_validate(payload)
+
+
+def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
+    payload = default_config()
+    payload["sources"].append(
+        {
+            "id": "new-report",
+            "title": "New report",
+            "url": "https://example.com/report",
+            "publisher": "Research group",
+            "report_year": 2026,
+            "locator": "Figure 2",
+        }
+    )
+    payload["observations"].append(
+        {
+            "year": 2025,
+            "metric": "global_share",
+            "source_id": "new-report",
+            "values": {"CN": 25, "US": 50, "JP": 0},
+        }
+    )
+    config = ComputeConfiguration.model_validate(payload)
+    result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
+    row = next(row for row in result["compute"] if row["year"] == 2025)
+    assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
+    assert result["evidence"][row["evidence_id"]]["title"] == "New report"
+    assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
+
+
+def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
+    payload = default_config()
+    payload["observations"][0]["values"] = {"CN": 0, "US": 0}
+    result = compute_history(
+        ["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
+    )
+    assert result["compute"][0]["values"] == {"CN": None, "US": None}
+    payload.update(sources=[], observations=[], innovation=[])
+    result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
+    assert all(row["values"]["CN"] is None for row in result["compute"])
+    assert result["evidence"] == {}
+
+
+def fake_db(record=None):
+    db = AsyncMock()
+    db.add = MagicMock()
+    result = MagicMock()
+    result.scalar_one_or_none.return_value = record
+    result.scalar_one.return_value = record
+    db.execute.return_value = result
+    return db
+
+
+@pytest.mark.asyncio
+async def test_fresh_install_uses_defaults_without_writing_database():
+    db = fake_db()
+    assert await read_configuration(db) == default_envelope()
+    db.add.assert_not_called()
+    db.commit.assert_not_awaited()
+
+
+@pytest.mark.asyncio
+async def test_series_reads_persisted_override_and_disables_http_cache():
+    envelope = default_envelope()
+    envelope["is_default"] = False
+    envelope["revision"] = "custom"
+    envelope["config"]["observations"][3]["values"]["CN"] = 30
+    response = Response()
+    data = await get_series(
+        response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
+    )
+    assert data["compute"][3]["values"]["CN"] == 30
+    assert data["version"] == "custom"
+    assert data["provenance"] == "admin_curated_reports"
+    assert response.headers["Cache-Control"] == "no-store"
+
+
+@pytest.mark.asyncio
+async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
+    record = SimpleNamespace(payload=default_envelope())
+    db = fake_db(record)
+    actor = SimpleNamespace(id=1, username="admin")
+    old = deepcopy(record.payload)
+    result = await publish_configuration(
+        db,
+        revision=old["revision"],
+        config=DEFAULT_CONFIGURATION,
+        actor=actor,
+    )
+    assert result["revision"] != old["revision"]
+    audit = db.add.call_args.args[0]
+    assert isinstance(audit, AuditLog)
+    assert audit.details == {"before": old, "after": result}
+    assert audit.actor_id == actor.id
+    db.commit.assert_awaited_once()
+    with pytest.raises(HTTPException) as failure:
+        await publish_configuration(
+            db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
+        )
+    assert failure.value.status_code == 409
+    assert record.payload == result
+    db.rollback.assert_awaited_once()
+    db.commit.assert_awaited_once()
+
+
+@pytest.mark.asyncio
+async def test_reset_changes_revision_and_restores_bundled_data():
+    record = SimpleNamespace(
+        payload={**default_envelope(), "revision": "custom", "is_default": False}
+    )
+    result = await publish_configuration(
+        fake_db(record),
+        revision="custom",
+        config=DEFAULT_CONFIGURATION,
+        actor=SimpleNamespace(id=1, username="admin"),
+        reset=True,
+    )
+    assert result["is_default"] is True
+    assert result["config"] == default_config()
+    assert result["revision"] != "custom"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
+async def test_non_admin_cannot_configure(role):
+    with pytest.raises(HTTPException) as failure:
+        await configuration_admin(SimpleNamespace(role=role))
+    assert failure.value.status_code == 403
+
+
+@pytest.mark.asyncio
+async def test_http_configuration_auth_validation_and_public_series():
+    app = FastAPI()
+    app.include_router(router)
+    record = SimpleNamespace(payload=default_envelope())
+    db = fake_db(record)
+    app.dependency_overrides[get_db] = lambda: db
+    async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
+        assert (await client.get("/configuration")).status_code == 401
+        assert (await client.get("/series")).status_code == 200
+        app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
+            id=1,
+            username="admin",
+            role="admin",
+        )
+        loaded = (await client.get("/configuration")).json()
+        invalid = deepcopy(loaded["config"])
+        invalid["observations"][0]["values"]["CN"] = 200
+        response = await client.put(
+            "/configuration",
+            json={
+                "revision": loaded["revision"],
+                "config": invalid,
+            },
+        )
+        assert response.status_code == 422
+        db.commit.assert_not_awaited()
+        response = await client.put(
+            "/configuration",
+            json={
+                "revision": loaded["revision"],
+                "config": loaded["config"],
+            },
+        )
+        assert response.status_code == 200
+        assert response.json()["revision"] != loaded["revision"]
+        assert (
+            await client.post(
+                "/configuration/reset",
+                json={
+                    "revision": loaded["revision"],
+                },
+            )
+        ).status_code == 409

backend/tests/test_compute_intelligence.py

@@ -0,0 +1,268 @@
+import csv
+import io
+from datetime import UTC, datetime
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from fastapi import HTTPException
+from sqlalchemy.dialects import postgresql
+
+from app.api.v1.compute_intelligence import sample_query, selected_countries
+from app.services.collectors.epoch_ai import EpochAIGPUCollector
+from app.services.collectors.top500 import TOP500Collector
+from app.services.compute_history import compute_history
+
+
+def epoch_csv(**overrides):
+    row = {
+        "Name": "Cluster Alpha",
+        "Country": "United States of America",
+        "Status": "Existing",
+        "First Operational Date": "2023-05-02",
+        "16-bit OP/s (log)": "18",
+        "Location": "3231 Riverport Rd, Memphis, TN 38109",
+        "latitude": "35",
+        "longitude": "-90",
+        **overrides,
+    }
+    output = io.StringIO()
+    writer = csv.DictWriter(output, fieldnames=list(row))
+    writer.writeheader()
+    writer.writerow(row)
+    return output.getvalue()
+
+
+def test_epoch_csv_preserves_precision_date_and_missing_values():
+    collector = EpochAIGPUCollector()
+    row = collector.parse_response(epoch_csv())[0]
+    assert row["value"] == 1e18
+    assert row["unit"] == "FLOP/s"
+    assert row["metadata"]["precision"] == "FP16"
+    assert row["metadata"]["location_precision"] == "precise"
+    assert row["metadata"]["raw_country"] == "United States of America"
+    assert row["reference_date"] == "2023-05-02"
+    missing = collector.parse_response(
+        epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
+    )[0]
+    assert missing["value"] is None
+    assert missing["reference_date"] is None
+
+
+@pytest.mark.parametrize(
+    ("location", "country", "expected"),
+    [
+        ("United States of America", "United States of America", "country"),
+        ("Japan", "Japan", "country"),
+        ("Korea (Republic of)", "South Korea", "country"),
+        ("Texas", "United States of America", "region"),
+        ("Iowa, USA", "United States of America", "region"),
+        ("Seoul", "South Korea", "city"),
+        ("Paris, France", "France", "city"),
+        ("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
+        ("Cluster 2, Texas", "United States of America", "site"),
+        ("Campus 7, Tokyo", "Japan", "site"),
+        ("Building 240, Argonne", "United States of America", "site"),
+        ("7", "United States of America", "unknown"),
+        ("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
+        ("35.380924,139.625008", "Japan", "precise"),
+        ('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
+        ("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
+        ('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
+        ("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
+    ],
+)
+def test_epoch_location_precision_follows_location_field(location, country, expected):
+    row = EpochAIGPUCollector().parse_response(
+        epoch_csv(Location=location, Country=country)
+    )[0]
+
+    assert row["metadata"]["location_precision"] == expected
+
+
+@pytest.mark.parametrize(
+    ("location", "expected"),
+    [
+        ({"lat": 35.0, "lng": 139.0}, "precise"),
+        ({"latitude": 35.0, "longitude": 181.0}, "unknown"),
+        ({"lat": 35.0}, "unknown"),
+        ({"lng": 139.0}, "unknown"),
+        ({"lat": "invalid", "lng": 139.0}, "unknown"),
+    ],
+)
+def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
+    from app.services.epoch_location import assess_epoch_location
+
+    assert assess_epoch_location(location).precision == expected
+
+
+def test_epoch_anonymous_rows_are_not_geocoded():
+    row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
+    assert row["latitude"] is None and row["longitude"] is None
+    assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
+
+
+@pytest.mark.parametrize(
+    "payload",
+    ["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
+)
+def test_epoch_invalid_payload_fails_closed(payload):
+    with pytest.raises(ValueError):
+        EpochAIGPUCollector().parse_response(payload)
+
+
+def test_epoch_duplicate_identity_rejected():
+    payload = epoch_csv()
+    with pytest.raises(ValueError, match="identity collision"):
+        EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
+
+
+def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
+    payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
+    rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
+    assert len(rows) == 2
+    assert rows[0]["source_id"] != rows[1]["source_id"]
+    assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
+
+
+def test_top500_units_and_stable_identity():
+    collector = TOP500Collector()
+    html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
+    first = collector.parse_response(
+        html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
+    )[0]
+    second = collector.parse_response(
+        html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
+    )[0]
+    assert first["source_id"] == second["source_id"] == "top500_system_123"
+    assert float(first["value"]) == 1.5e15
+    assert first["metadata"]["rpeak_flops"] == 2e15
+    assert first["reference_date"] == "2025-11-01"
+    assert second["reference_date"] == "2026-06-01"
+    with pytest.raises(ValueError, match="date missing"):
+        collector.parse_response(html.format(rank=1))
+
+
+@pytest.mark.asyncio
+async def test_large_source_loss_does_not_create_snapshot():
+    db = AsyncMock()
+    db.scalar.return_value = 500
+    db.add = MagicMock()
+    with pytest.raises(ValueError, match="40%"):
+        await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
+    db.add.assert_not_called()
+    db.commit.assert_not_awaited()
+
+
+def test_history_missing_years_and_denominators():
+    global_data = compute_history(["CN", "US"])
+    assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
+    assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
+    bilateral = compute_history(["CN", "US"], "bilateral_share")
+    assert sum(bilateral["compute"][3]["values"].values()) == 100
+    assert bilateral["denominator"] == "US+CN"
+    absolute = compute_history(["US", "CN"], "absolute")
+    assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
+    assert global_data["innovation"][-2]["value"] == 17.95
+    assert global_data["innovation"][-1]["value"] is None
+    assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
+
+
+@pytest.mark.asyncio
+async def test_snapshot_query_uses_observation_cutoff_not_location():
+    query = await sample_query(AsyncMock(), ["CN"], 2023)
+    sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
+    assert "2024-01-01" in sql
+    assert "row_number() OVER (PARTITION BY" in sql
+    assert "compute_contract" in sql and "中国" in sql
+    assert "latitude" not in sql and "longitude" not in sql
+
+
+def test_country_input_is_bounded_and_deduplicated():
+    assert selected_countries("CN,CN,US") == ["CN", "US"]
+    assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
+    with pytest.raises(HTTPException):
+        selected_countries("invalid")
+    with pytest.raises(HTTPException, match="overlapping"):
+        selected_countries("EUROPE,DE")
+
+
+def test_catalog_supports_global_participants_without_inventing_history():
+    from app.services.compute_catalog import (
+        CATALOG,
+        COMPARISON_GROUPS,
+        COUNTRIES,
+        source_country_names,
+        source_country_participants,
+    )
+
+    assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
+        {entry["code"] for entry in CATALOG}
+    )
+    assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
+    owners = source_country_names(["CN"])
+    assert {"中国", "中国（台湾）", "中国（香港）", "中国（澳门）"}.issubset(owners)
+    assert source_country_participants(["CN"])["中国（台湾）"] == "CN"
+    assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
+    assert [entry["kind"] for entry in CATALOG[:5]] == [
+        "country",
+        "country",
+        "region",
+        "region",
+        "region",
+    ]
+    group_members = [
+        member
+        for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
+        for member in members
+    ]
+    assert len(group_members) == len(set(group_members))
+    assert set(group_members) == set(COUNTRIES)
+    data = compute_history(["JP", "KR"])
+    assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
+    assert all(row["values"]["KR"] is None for row in data["compute"])
+    assert all(row["value"] is None for row in data["innovation"])
+    assert len(data["catalog"]) > 30
+
+
+def test_region_history_requires_complete_same_year_member_coverage():
+    data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
+    row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
+    assert row_2023["values"]["CN"] == 31
+    assert row_2023["values"]["US"] == 41
+    assert row_2023["values"]["EUROPE"] is None
+    assert row_2023["values"]["APAC"] is None
+    assert row_2023["values"]["OTHER"] is None
+    assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
+    assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
+    assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
+
+
+def test_anonymous_records_never_enter_the_location_repair_queue():
+    from app.api.v1.visualization import convert_compute_centers_to_geojson
+    from app.models.collected_data import CollectedData
+
+    record = CollectedData(
+        source="epoch_ai_gpu",
+        source_id="anonymous",
+        name="Anonymous",
+        extra_data={"compute_contract": "v1", "anonymous": True},
+    )
+    result = convert_compute_centers_to_geojson([record])
+    assert result["features"] == []
+    assert result["unresolved"] == []
+    assert result["withheld_count"] == 1
+
+
+def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
+    absolute = compute_history(["CN"], "absolute")
+    observed = {row["year"]: row for row in absolute["compute"]}
+    assert observed[2022]["values"]["CN"] == 302
+    assert observed[2022]["evidence_id"] == "caict-2023"
+    assert observed[2023]["evidence_id"] == "caict-2024"
+    bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
+    values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
+    assert values == {"CN": 43.06, "US": 56.94, "JP": None}
+    for row in absolute["compute"]:
+        if row["evidence_id"]:
+            report = absolute["evidence"][row["evidence_id"]]
+            assert report["url"].startswith("https://www.caict.ac.cn/")

backend/tests/test_compute_verified_sites.py

@@ -0,0 +1,18 @@
+from app.services.compute_center_verified_sites import get_verified_compute_site
+
+
+def test_registry_requires_exact_facility_identity_not_operator_name():
+    assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
+    assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
+
+
+def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
+    site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
+    assert site["city"] == "Yokohama"
+    assert site["precision"] == "site"
+    assert site["coordinate_basis"] == "official_access_map_center"
+    assert site["sources"]
+    site["latitude"] = 0
+    assert get_verified_compute_site(
+        "epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
+    )["latitude"] != 0

backend/tests/test_earth_boundaries.py

@@ -1,4 +1,5 @@
 import json
+from pathlib import Path
 
 import pytest
 
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
 
 
 def write_geojson(path, name="Test"):
+    names = [name]
+    if name == "earth_admin0_boundaries":
+        names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
+    codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
     path.write_text(
         json.dumps(
             {
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
                 "features": [
                     {
                         "type": "Feature",
-                        "properties": {"name": name},
+                        "properties": {
+                            "NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
+                            "ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
+                            "ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
+                        },
                         "geometry": {
                             "type": "Polygon",
                             "coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
                         },
                     }
+                    for item in names
                 ],
             }
         ),
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
     config = repo / "config/earth-boundary-sources.local.json"
     example = repo / "config/earth-boundary-sources.example.json"
     policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
+    geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
     for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
         path.mkdir(parents=True, exist_ok=True)
-    policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
+    source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
+    policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
+    geometry.parent.mkdir(parents=True, exist_ok=True)
+    source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
+    geometry.write_bytes(source_geometry.read_bytes())
     example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
     monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
     monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
     monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
     monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
     monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
-    monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
+    monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
     monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
     monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
     monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
+    monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
     return {
         "repo": repo,
         "config": config,
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
     }
 
 
+def write_fallback(path, policy_path, geometry_path):
+    policy = earth_boundaries._read_json(policy_path)
+    geometry = earth_boundaries._read_json(geometry_path)
+    coverage = {
+        "profile": "china-pov-v1",
+        "sha256": earth_boundaries._stable_json_hash(policy),
+        "geometrySha256": earth_boundaries._stable_json_hash(geometry),
+        "reviewedGeometrySourceCommit": geometry["sourceCommit"],
+        "applied": True,
+        "appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
+    }
+    path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
+
+
 def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
+    paths = patch_paths(monkeypatch, tmp_path)
+    write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
+
+    status = earth_boundaries.get_boundary_status()
+
+    assert status["provider"] == "policy-geojson-fallback"
+    assert status["fallback_available"] is True
+    assert status["high_precision_ready"] is False
+
+
+def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
     paths = patch_paths(monkeypatch, tmp_path)
     write_geojson(paths["legacy"])
 
     status = earth_boundaries.get_boundary_status()
 
-    assert status["provider"] == "legacy-geojson"
-    assert status["fallback_available"] is True
-    assert status["high_precision_ready"] is False
+    assert status["provider"] == "missing"
+    assert status["fallback_available"] is False
 
 
 def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
     paths = patch_paths(monkeypatch, tmp_path)
-    write_geojson(paths["legacy"])
+    write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
     paths["pmtiles"].write_bytes(b"pmtiles")
-    paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
+    (paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
+    coverage = {
+        "profile": "china-pov-v1",
+        "sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
+        "applied": True,
+        "appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
+        "reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
+        "geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
+    }
+    paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
 
     status = earth_boundaries.get_boundary_status()
 
     assert status["provider"] == "pmtiles-mvt"
     assert status["high_precision_ready"] is True
+    assert status["policy_publishable"] is False
+
+
+def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
+    paths = patch_paths(monkeypatch, tmp_path)
+    write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
+    paths["pmtiles"].write_bytes(b"pmtiles")
+    paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
+
+    status = earth_boundaries.get_boundary_status()
+
+    assert status["provider"] == "policy-geojson-fallback"
+    assert status["high_precision_ready"] is False
 
 
 def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
     assert status["high_precision_ready"] is True
     assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
     assert paths["manifest"].exists()
+    manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
+    assert manifest["povPolicy"]["applied"] is True
+    assert manifest["povPolicy"]["coverage"] == "partial"
+    assert manifest["povPolicy"]["publicationReady"] is False
+    assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
+    hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
+    by_rule = {
+        feature["properties"].get("PLANET_POLICY_RULE"): feature
+        for feature in hover["features"]
+    }
+    assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
+    assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
+    assert all(
+        feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
+        for feature in hover["features"]
+    )
     assert any(phase == "download" for _progress, phase, _message in progress_events)
 
 
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
     assert removed.isdisjoint(DEFAULT_DATASOURCES)
     for source in removed:
         assert collector_registry.get(source) is None
+
+
+def test_static_policy_fallback_contains_reviewed_jeju_polygon():
+    repo = Path(__file__).resolve().parents[2]
+    path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
+    payload = json.loads(path.read_text(encoding="utf-8"))
+    matches = [
+        feature for feature in payload["features"]
+        if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
+    ]
+
+    assert len(matches) == 1
+    jeju = matches[0]
+    assert jeju["properties"]["ISO_A3"] == "KOR"
+    assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
+    assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
+    ring = jeju["geometry"]["coordinates"][0]
+    assert len(ring) == 61
+    assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
+    assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)

backend/tests/test_earth_db_change_listener.py

@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
     assert vessel_update is not None
     assert vessel_update["source"] == "vessel_position"
     assert vessel_update["layers"] == ["vessels"]
+    assert vessel_update["refresh_strategy"] == "reload"
+
+
+def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
+    from app.services.earth_db_change_listener import PendingEarthDbChange
+
+    for source in ("aisstream_vessels", "barentswatch_vessels"):
+        update = build_earth_update_from_db_payload({
+            "source": source, "table": "vessel_current_state", "operation": "UPDATE",
+        })
+        assert update["refresh_strategy"] == "delta"
+        pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
+        pending.add({"operation": "UPDATE"})
+        pending.add({"operation": "DELETE"})
+        assert pending.refresh_strategy == "reload"
 
 
 def test_build_earth_update_maps_interactable_delete_to_delta():

backend/tests/test_earth_geography_policy.py

@@ -0,0 +1,41 @@
+import json
+from pathlib import Path
+
+import pytest
+
+from app.core.earth_geography_policy import display_country, display_iso_a3
+
+
+def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
+    from app.core import earth_geography_policy as policy
+
+    policy._policy_country_by_code.cache_clear()
+    monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
+    try:
+        with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
+            policy.display_country("China")
+    finally:
+        policy._policy_country_by_code.cache_clear()
+
+
+def test_display_country_groups_china_regions_and_kosovo() -> None:
+    for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国（台湾）"):
+        assert display_country(source) == "中国"
+        assert display_iso_a3(source) == "CHN"
+    for source in ("Kosovo", "XK", "SRB"):
+        assert display_country(source) == "塞尔维亚"
+        assert display_iso_a3(source) == "SRB"
+
+
+def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
+    assert display_country("Japan") == "日本"
+    assert display_iso_a3("Japan") == "JPN"
+    assert display_country("unexpected non-country") is None
+
+
+def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
+    path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
+    policy = json.loads(path.read_text(encoding="utf-8"))
+    for rule in policy["rules"]:
+        for code in rule.get("sourceIsoA3", []):
+            assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))

backend/tests/test_earth_layer_adapters.py

@@ -0,0 +1,15 @@
+from app.services.earth_layer_adapters import (
+    get_earth_cache_patterns_for_source,
+    get_earth_update_layers_for_change,
+)
+
+
+def test_news_changes_do_not_invalidate_earth_summary_cache():
+    assert get_earth_update_layers_for_change("earth_news_items", "earth_news_items") == ["news"]
+    assert get_earth_cache_patterns_for_source("earth_news_items") == []
+    assert get_earth_cache_patterns_for_source("media_news_archive") == []
+
+
+def test_counted_layers_still_invalidate_earth_summary_cache():
+    for source in ("aisstream_vessels", "arcgis_cables", "celestrak_tle", "top500", "ris_live_bgp"):
+        assert "summary*" in get_earth_cache_patterns_for_source(source)

backend/tests/test_earth_layer_cache.py

@@ -1,9 +1,11 @@
 from __future__ import annotations
 
+import asyncio
 import json
 
+import httpx
 import pytest
-from fastapi import Response
+from fastapi import FastAPI, HTTPException, Response
 
 from app.api.v1 import visualization
 from app.services.earth_layer_cache import (
@@ -49,6 +51,19 @@ class FakeRedis:
             deleted += 1 if self.store.pop(key, None) is not None else 0
         return deleted
 
+    def eval(self, script: str, numkeys: int, *args):
+        self._maybe_fail()
+        keys, values = args[:numkeys], args[numkeys:]
+        if self.store.get(keys[0]) != values[0]:
+            return 0
+        if numkeys == 3:
+            self.setex(keys[1], values[1], values[3])
+            self.setex(keys[2], values[2], values[3])
+            return 1
+        if "'expire'" in script:
+            return 1
+        return self.delete(keys[0])
+
     def scan_iter(self, match: str):
         self._maybe_fail()
         prefix = match.rstrip("*")
@@ -120,7 +135,9 @@ async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
 @pytest.mark.asyncio
 async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
     key = earth_layer_cache.key("bgp-incidents", status="active")
-    fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
+    fake_cache_client.store[f"{key}:stale"] = json.dumps(
+        {"type": "FeatureCollection", "features": []}
+    )
 
     async def builder():
         raise RuntimeError("db exploded")
@@ -165,6 +182,7 @@ async def test_resolve_layer_payload_bypasses_redis_failure():
     previous = earth_layer_cache._client
     earth_layer_cache._client = FakeRedis(fail=True)
     try:
+
         async def builder():
             return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
 
@@ -192,9 +210,13 @@ async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monk
     monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
 
     first_response = Response()
-    first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
+    first = await visualization.get_satellites_geojson(
+        limit=25, db=object(), response=first_response
+    )
     second_response = Response()
-    second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
+    second = await visualization.get_satellites_geojson(
+        limit=25, db=object(), response=second_response
+    )
 
     assert first == second
     assert calls == 1
@@ -252,3 +274,284 @@ async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
     assert calls == 1
     assert first_response.headers["X-Planet-Cache"] == "refresh"
     assert second_response.headers["X-Planet-Cache"] == "hit"
+
+
+@pytest.mark.asyncio
+async def test_concurrent_summary_requests_share_one_slow_build(monkeypatch):
+    started = asyncio.Event()
+    finish = asyncio.Event()
+    calls = 0
+
+    async def build_summary(db):
+        nonlocal calls
+        calls += 1
+        started.set()
+        await finish.wait()
+        return {"stats": {"vessel_count": 42}}
+
+    monkeypatch.setattr(visualization, "_build_visualization_geo_summary", build_summary)
+    app = FastAPI()
+    app.include_router(visualization.router, prefix="/api/v1/visualization")
+    app.dependency_overrides[visualization.get_db] = lambda: object()
+    async with httpx.AsyncClient(
+        transport=httpx.ASGITransport(app=app), base_url="http://test"
+    ) as client:
+        requests = [
+            asyncio.create_task(client.get("/api/v1/visualization/geo/summary")) for _ in range(6)
+        ]
+        await asyncio.wait_for(started.wait(), 1)
+        # Keep the build open beyond the old 200 ms fallback to duplicate SQL.
+        await asyncio.sleep(0.25)
+        assert calls == 1
+        finish.set()
+        results = await asyncio.gather(*requests)
+    assert all(response.status_code == 200 for response in results)
+    assert all(response.json() == {"stats": {"vessel_count": 42}} for response in results)
+    assert sorted(response.headers["X-Planet-Cache"] for response in results) == [
+        "hit",
+        "hit",
+        "hit",
+        "hit",
+        "hit",
+        "refresh",
+    ]
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("error_type", [RuntimeError, TimeoutError])
+async def test_shared_builder_error_is_not_retried(error_type):
+    started = asyncio.Event()
+    finish = asyncio.Event()
+    calls = 0
+
+    async def builder():
+        nonlocal calls
+        calls += 1
+        started.set()
+        await finish.wait()
+        raise error_type("db failed")
+
+    requests = [
+        asyncio.create_task(
+            resolve_layer_payload(
+                key="shared-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
+            )
+        )
+        for _ in range(4)
+    ]
+    await started.wait()
+    await asyncio.sleep(0)
+    finish.set()
+    results = await asyncio.gather(*requests, return_exceptions=True)
+    assert calls == 1
+    assert all(isinstance(result, error_type) for result in results)
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+async def test_concurrent_redis_failure_shares_bypass_result(fake_cache_client):
+    fake_cache_client.fail = True
+    started = asyncio.Event()
+    finish = asyncio.Event()
+    calls = 0
+
+    async def builder():
+        nonlocal calls
+        calls += 1
+        started.set()
+        await finish.wait()
+        return {"count": 42}
+
+    requests = [
+        asyncio.create_task(
+            resolve_layer_payload(
+                key="shared-bypass", policy=EarthLayerCachePolicy(60, 120), builder=builder
+            )
+        )
+        for _ in range(4)
+    ]
+    await started.wait()
+    await asyncio.sleep(0)
+    finish.set()
+    results = await asyncio.gather(*requests)
+    assert calls == 1
+    assert all(result.state == "bypass" and result.payload == {"count": 42} for result in results)
+
+
+@pytest.mark.asyncio
+async def test_cancelled_waiter_does_not_cancel_owner():
+    started = asyncio.Event()
+    finish = asyncio.Event()
+    policy = EarthLayerCachePolicy(60, 120)
+
+    async def builder():
+        started.set()
+        await finish.wait()
+        return {"count": 1}
+
+    owner = asyncio.create_task(
+        resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
+    )
+    await started.wait()
+    waiter = asyncio.create_task(
+        resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
+    )
+    await asyncio.sleep(0)
+    waiter.cancel()
+    with pytest.raises(asyncio.CancelledError):
+        await waiter
+    finish.set()
+    assert (await owner).payload == {"count": 1}
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+async def test_cancelled_owner_allows_waiter_to_use_its_own_builder():
+    started = asyncio.Event()
+    policy = EarthLayerCachePolicy(60, 120)
+
+    async def owner_builder():
+        started.set()
+        await asyncio.Event().wait()
+
+    async def waiter_builder():
+        return {"count": 2}
+
+    owner = asyncio.create_task(
+        resolve_layer_payload(key="cancel-owner", policy=policy, builder=owner_builder)
+    )
+    await started.wait()
+    waiter = asyncio.create_task(
+        resolve_layer_payload(key="cancel-owner", policy=policy, builder=waiter_builder)
+    )
+    await asyncio.sleep(0)
+    owner.cancel()
+    with pytest.raises(asyncio.CancelledError):
+        await owner
+    assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 2}
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+async def test_cross_process_lock_waits_for_payload_instead_of_building(fake_cache_client):
+    key = "remote-owner"
+    fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
+    policy = EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005)
+    calls = 0
+
+    async def builder():
+        nonlocal calls
+        calls += 1
+        return {"count": -1}
+
+    waiter = asyncio.create_task(resolve_layer_payload(key=key, policy=policy, builder=builder))
+    await asyncio.sleep(0.03)
+    assert calls == 0
+    fake_cache_client.store[key] = json.dumps({"count": 3})
+    assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 3}
+    assert calls == 0
+
+
+@pytest.mark.asyncio
+async def test_wait_timeout_does_not_bypass_other_owner(fake_cache_client):
+    key = "remote-timeout"
+    fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
+    calls = 0
+
+    async def builder():
+        nonlocal calls
+        calls += 1
+        return {}
+
+    with pytest.raises(HTTPException) as error:
+        await resolve_layer_payload(
+            key=key,
+            policy=EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005, build_wait_seconds=0.02),
+            builder=builder,
+        )
+    assert error.value.status_code == 503
+    assert calls == 0
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+async def test_lost_owner_cannot_publish_or_release_new_lock(fake_cache_client):
+    key = "lock-replaced"
+    policy = EarthLayerCachePolicy(60, 120)
+
+    async def builder():
+        fake_cache_client.store[earth_layer_cache.lock_key(key)] = "new-owner"
+        fake_cache_client.store[key] = json.dumps({"count": 99})
+        return {"count": 1}
+
+    result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
+    assert result.payload == {"count": 1}
+    assert result.state == "bypass"
+    assert fake_cache_client.store[earth_layer_cache.lock_key(key)] == "new-owner"
+    assert json.loads(fake_cache_client.store[key]) == {"count": 99}
+
+
+@pytest.mark.asyncio
+async def test_redis_publish_failure_returns_built_payload_without_retry(fake_cache_client):
+    calls = 0
+
+    async def builder():
+        nonlocal calls
+        calls += 1
+        fake_cache_client.fail = True
+        return {"count": 7}
+
+    result = await resolve_layer_payload(
+        key="publish-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
+    )
+    assert result.state == "bypass"
+    assert result.payload == {"count": 7}
+    assert calls == 1
+
+
+@pytest.mark.asyncio
+async def test_local_wait_timeout_leaves_owner_running():
+    started = asyncio.Event()
+    finish = asyncio.Event()
+    policy = EarthLayerCachePolicy(60, 120, build_wait_seconds=0.02)
+
+    async def builder():
+        started.set()
+        await finish.wait()
+        return {"count": 1}
+
+    owner = asyncio.create_task(
+        resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
+    )
+    await started.wait()
+    with pytest.raises(HTTPException) as error:
+        await resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
+    assert error.value.status_code == 503
+    assert not owner.done()
+    finish.set()
+    assert (await owner).payload == {"count": 1}
+    assert not earth_layer_cache._inflight
+
+
+@pytest.mark.asyncio
+async def test_different_keys_can_build_concurrently():
+    started = set()
+    both_started = asyncio.Event()
+
+    async def build(key):
+        started.add(key)
+        if len(started) == 2:
+            both_started.set()
+        await asyncio.wait_for(both_started.wait(), 1)
+        return {"key": key}
+
+    results = await asyncio.gather(
+        *(
+            resolve_layer_payload(
+                key=key, policy=EarthLayerCachePolicy(60, 120), builder=lambda key=key: build(key)
+            )
+            for key in ("one", "two")
+        )
+    )
+    assert {result.payload["key"] for result in results} == {"one", "two"}

backend/tests/test_earth_news.py

@@ -3,6 +3,7 @@ from types import SimpleNamespace
 
 import pytest
 
+from app.api.v1.news import get_earth_feed
 from app.services.earth_news import (
     NewsFeedEndpoint,
     NewsFeedSource,
@@ -1201,7 +1202,6 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
         db=FakeDb(),
     )
 
-    assert captured["freshness_region"] == "europe"
     assert captured["items_region"] == "europe"
     assert captured["items_categories"] == {"business", "ecommerce"}
     assert captured["items_source_ids"][0] is None
@@ -1237,71 +1237,41 @@ async def test_news_source_test_treats_type_reference_as_non_fetching():
 
 
 @pytest.mark.asyncio
-async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
+async def test_earth_news_feed_get_does_not_fetch_write_or_queue_rss(monkeypatch):
     db = object()
-    source = NewsFeedSource(
-        id="test-feed",
-        name="Test Feed",
-        region="global",
-        homepage_url="https://example.com",
-        feed_url="https://example.com/rss.xml",
+    read_calls = 0
+    async def fail_side_effect(*_args, **_kwargs):
+        raise AssertionError("RSS feed reads must not fetch, persist, or enqueue work")
+
+    async def empty_news_items(*_args, **_kwargs):
+        nonlocal read_calls
+        read_calls += 1
+        return []
+
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_side_effect)
+    monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fail_side_effect)
+    monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fail_side_effect)
+    monkeypatch.setattr(
+        "app.services.earth_news_store.list_earth_news_items", empty_news_items
     )
-    item = ParsedNewsItem(
-        id="test-feed:init",
-        title="Initial RSS story",
-        summary="Initial summary",
-        url="https://example.com/init",
-        source="Test Feed",
-        feed_name="Test Feed",
-        feed_region="global",
-        homepage_url="https://example.com",
-        published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
+
+    payload = await get_earth_feed(
+        lat=None,
+        lon=None,
+        region=None,
+        categories=None,
+        sources=None,
+        limit=None,
+        locale=None,
+        db=db,
     )
-    item.location_patch = {
-        "latitude": 20.0,
-        "longitude": 0.0,
-        "location_label": "全球",
-        "location_source": "region_anchor",
-        "verified": False,
-        "location_meta": {"target": None, "anchor": {"region": "global"}},
-    }
-    upserted = []
-    enqueued = []
 
-    async def fake_get_earth_news_freshness(_db, *, active_region):
-        return 0, None
-
-    async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
-        return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
-
-    async def fake_upsert_earth_news_items(_db, items):
-        upserted.extend(items)
-        return len(items)
-
-    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
-        return [item]
-
-    async def fake_enqueue_target_location_job(payload, **_kwargs):
-        enqueued.append(payload)
-        return True
-
-    monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
-    monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
-    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
-    monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
-    monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
-    monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
-
-    payload = await get_earth_news_payload(db=db)
-
-    assert upserted[0].id == "test-feed:init"
-    assert payload["items"][0]["id"] == "test-feed:init"
-    assert payload["items"][0]["verified"] is False
-    assert enqueued[0]["id"] == "test-feed:init"
+    assert payload["items"] == []
+    assert read_calls == 1
 
 
 @pytest.mark.asyncio
-async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
+async def test_earth_news_payload_does_not_supplement_stale_database_items(monkeypatch):
     db = object()
     old_item = ParsedNewsItem(
         id="db:old",
@@ -1314,41 +1284,17 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
         homepage_url="https://example.com",
         published_at=datetime(2026, 5, 14, 3, 0, tzinfo=UTC),
     )
-    old_item.location_patch = {
-        "latitude": 20.0,
-        "longitude": 0.0,
-        "location_label": "全球",
-        "location_source": "region_anchor",
-        "verified": False,
-        "location_meta": {"target": None, "anchor": {"region": "global"}},
-    }
-    fetched = []
+    async def fail_fetch(*_args, **_kwargs):
+        raise AssertionError("stale RSS supplementation belongs to the background worker")
 
-    async def fake_get_earth_news_freshness(_db, *, active_region):
-        return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
-
-    async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
-        fetched.append(True)
-        return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
-
-    async def fake_upsert_earth_news_items(_db, items):
-        return len(items)
-
-    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
+    async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
         return [old_item]
 
-    async def fake_enqueue_target_location_job(_payload, **_kwargs):
-        return True
-
-    monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
-    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
-    monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
     monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
-    monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
 
-    payload = await get_earth_news_payload(db=db)
+    payload = await get_earth_news_payload(db=db, source_ids={"stored"})
 
-    assert fetched == [True]
     assert payload["items"][0]["id"] == "db:old"
 
 

backend/tests/test_earth_news_encoding.py

@@ -0,0 +1,86 @@
+from typing import Any
+
+import httpx
+import pytest
+
+from app.services.earth_news import (
+    NewsFeedEndpoint,
+    NewsFeedSource,
+    ParsedNewsItem,
+    _fetch_single_feed_url,
+    _serialize_item,
+)
+
+
+async def fetch_feed(
+    content: bytes, content_type: str = "text/xml"
+) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
+    source = NewsFeedSource(
+        id="ebrun",
+        name="亿邦动力",
+        region="asia-pacific",
+        feed_url="https://example.com/rss.xml",
+        homepage_url="https://example.com/",
+    )
+    feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
+    transport = httpx.MockTransport(
+        lambda request: httpx.Response(
+            200, content=content, headers={"content-type": content_type}
+        )
+    )
+    async with httpx.AsyncClient(transport=transport) as client:
+        return await _fetch_single_feed_url(client, source, feed)
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    ("encoding", "declaration", "content_type"),
+    [
+        ("gbk", 'encoding="GBK"', "text/xml"),
+        ("gb2312", "encoding='GB2312'", "application/rss+xml"),
+        ("gb18030", 'encoding="GB18030"', "text/xml"),
+        ("gbk", "", "text/xml; charset=gbk"),
+        ("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
+        ("utf-8", "", "text/xml"),
+        ("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
+        ("utf-16", 'encoding="UTF-16"', "text/xml"),
+        ("utf-32", 'encoding="UTF-32"', "text/xml"),
+    ],
+)
+async def test_feed_encoding_preserves_chinese_in_display(
+    encoding: str, declaration: str, content_type: str
+):
+    title = "618电商战报：零售额增长266%"
+    summary = "亿邦动力：跨境电商订单增长，物流效率提升。"
+    xml = f'''<?xml version="1.0" {declaration}?>
+    <rss version="2.0"><channel><item>
+      <title><![CDATA[{title}]]></title>
+      <description><![CDATA[<p>{summary}</p>]]></description>
+      <link>https://example.com/story</link>
+    </item></channel></rss>'''
+
+    _, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
+
+    assert error is None
+    assert health["ok"] is True
+    assert len(items) == 1
+    assert items[0].title == title
+    assert items[0].summary == summary
+    payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
+    assert payload["display_title"] == title
+    assert payload["display_summary"] == summary
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
+async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
+    xml = f'''<?xml version="1.0" encoding="{encoding}"?>
+    <rss><channel><item><title>电商新闻</title>
+    <link>https://example.com/story</link></item></channel></rss>'''
+
+    _, items, error, health = await fetch_feed(xml.encode("gbk"))
+
+    assert items == []
+    assert error
+    assert health["ok"] is False
+    assert health["status"] == "format_error"

backend/tests/test_earth_news_refresh.py

@@ -0,0 +1,525 @@
+from contextlib import asynccontextmanager
+from datetime import UTC, datetime
+import json
+
+import pytest
+
+from app.services.earth_news import NewsFeedEndpoint, NewsFeedSource
+from app.services import earth_news_refresh
+from app.services.scheduler import (
+    EARTH_NEWS_REFRESH_JOB_ID,
+    _earth_news_region_refresh_job_id,
+    schedule_earth_news_region_refresh,
+    start_scheduler,
+    sync_scheduler_with_datasources,
+)
+
+
+class FakeRedis:
+    def __init__(self) -> None:
+        self.values: dict[str, str] = {}
+
+    def set(self, key: str, value: str, *, nx: bool = False, ex: int | None = None) -> bool:
+        del ex
+        if nx and key in self.values:
+            return False
+        self.values[key] = value
+        return True
+
+    def exists(self, key: str) -> bool:
+        return key in self.values
+
+    def eval(self, script: str, _key_count: int, key: str, token: str, *_args: object) -> int:
+        if self.values.get(key) != token:
+            return 0
+        if "redis.call('del'" in script:
+            del self.values[key]
+        return 1
+
+
+class FakeStatusRedis:
+    def __init__(self) -> None:
+        self.values: dict[str, str] = {}
+        self.expirations: dict[str, int] = {}
+
+    def get(self, key: str):
+        return self.values.get(key)
+
+    def eval(self, script: str, key_count: int, *args):
+        keys = args[:key_count]
+        values = args[key_count:]
+        if "local active_id" in script:
+            active_key, status_key = keys
+            refresh_id, encoded, active_ttl, status_ttl, status_prefix = values
+            active_id = self.values.get(active_key)
+            if active_id:
+                active_status = self.values.get(f"{status_prefix}{active_id}")
+                if active_status:
+                    status = json.loads(active_status)["status"]
+                    if status in {"pending", "running"}:
+                        return [active_id, active_status, 0]
+                self.values.pop(active_key, None)
+            self.values[active_key] = refresh_id
+            self.values[status_key] = encoded
+            self.expirations[active_key] = int(active_ttl)
+            self.expirations[status_key] = int(status_ttl)
+            return [refresh_id, encoded, 1]
+
+        active_key, status_key = keys
+        encoded, refresh_id, status_ttl, status, active_ttl = values
+        if self.values.get(active_key) != refresh_id:
+            return 0
+        self.values[status_key] = encoded
+        self.expirations[status_key] = int(status_ttl)
+        if status in {"succeeded", "failed"}:
+            self.values.pop(active_key, None)
+        else:
+            self.expirations[active_key] = int(active_ttl)
+        return 1
+
+
+class FakeSession:
+    def __init__(self) -> None:
+        self.commits = 0
+        self.rollbacks = 0
+
+    async def commit(self) -> None:
+        self.commits += 1
+
+    async def rollback(self) -> None:
+        self.rollbacks += 1
+
+
+@pytest.mark.asyncio
+async def test_region_refresh_fetches_only_missing_feed_coverage(monkeypatch):
+    redis_client = FakeRedis()
+    db = FakeSession()
+    feeds = (
+        NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global"),
+        NewsFeedEndpoint("feed-b", "Feed B", "https://example.com/b.xml", region="global"),
+    )
+    source = NewsFeedSource(
+        id="source-a",
+        name="Source A",
+        region="global",
+        feed_url=feeds[0].url,
+        homepage_url="https://example.com",
+        feeds=feeds,
+    )
+    fetched_source_feeds: list[list[str]] = []
+
+    @asynccontextmanager
+    async def fake_session_factory():
+        yield db
+
+    async def fake_get_sources_payload(_db):
+        return {"sources": []}
+
+    async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
+        assert config_payload == {"sources": []}
+        return [source]
+
+    async def fake_freshness(_db, *, active_region):
+        assert active_region == "global"
+        return 12, datetime.now(UTC)
+
+    async def fake_coverage(_db, *, active_region, recent_after):
+        assert active_region == "global"
+        assert recent_after is not None
+        return {("source-a", "feed-a")}
+
+    async def fake_fetch(sources, **_kwargs):
+        fetched_source_feeds.extend([[feed.id for feed in item.feeds] for item in sources])
+        return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
+
+    async def fake_record_health(_db, _health):
+        return None
+
+    async def fake_enqueue(_items):
+        return None
+
+    async def fake_upsert(_db, items):
+        assert items == []
+        return 0
+
+    monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
+    monkeypatch.setattr(
+        "app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
+    )
+    monkeypatch.setattr(
+        "app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
+    )
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
+    monkeypatch.setattr(
+        "app.services.earth_news.record_earth_news_sources_health", fake_record_health
+    )
+    monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_enqueue)
+    monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
+
+    result = await earth_news_refresh.refresh_earth_news_region("global")
+
+    assert result["state"] == "completed"
+    assert result["successful_sources"] == 1
+    assert fetched_source_feeds == [["feed-b"]]
+    assert db.commits == 1
+    assert earth_news_refresh._lock_key("global") not in redis_client.values
+
+
+@pytest.mark.asyncio
+async def test_region_refresh_reports_failure_when_all_sources_time_out(monkeypatch):
+    redis_client = FakeRedis()
+    db = FakeSession()
+    feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global")
+    source = NewsFeedSource(
+        id="source-a",
+        name="Source A",
+        region="global",
+        feed_url=feed.url,
+        homepage_url="https://example.com",
+        feeds=(feed,),
+    )
+
+    @asynccontextmanager
+    async def fake_session_factory():
+        yield db
+
+    async def fake_get_sources_payload(_db):
+        return {"sources": []}
+
+    async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
+        return [source]
+
+    async def fake_freshness(_db, *, active_region):
+        return 12, datetime.now(UTC)
+
+    async def fake_coverage(_db, *, active_region, recent_after):
+        return set()
+
+    async def fake_fetch(*_args, **_kwargs):
+        return (
+            [],
+            ["private upstream details"],
+            {
+                "source-a": {
+                    "source_id": "source-a",
+                    "ok": False,
+                    "status": "timeout",
+                    "feed_results": [{"feed_id": "feed-a", "ok": False, "status": "timeout"}],
+                }
+            },
+        )
+
+    async def fake_noop(*_args, **_kwargs):
+        return None
+
+    async def fake_upsert(_db, _items):
+        return 0
+
+    monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
+    monkeypatch.setattr(
+        "app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
+    )
+    monkeypatch.setattr(
+        "app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
+    )
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
+    monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
+    monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
+    monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
+
+    result = await earth_news_refresh.refresh_earth_news_region("global", force=True)
+
+    assert result["state"] == "failed"
+    assert result["error_code"] == "timeout"
+    assert db.commits == 1
+
+
+@pytest.mark.asyncio
+async def test_database_commit_failure_sets_refresh_failed_status(monkeypatch):
+    class CommitFailureSession(FakeSession):
+        async def commit(self) -> None:
+            raise RuntimeError("database commit failed with sensitive context")
+
+    redis_client = FakeRedis()
+    db = CommitFailureSession()
+    feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="europe")
+    source = NewsFeedSource(
+        id="source-a",
+        name="Source A",
+        region="europe",
+        feed_url=feed.url,
+        homepage_url="https://example.com",
+        feeds=(feed,),
+    )
+    scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
+    statuses = []
+
+    @asynccontextmanager
+    async def fake_session_factory():
+        yield db
+
+    async def fake_get_sources_payload(_db):
+        return {"sources": []}
+
+    async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
+        return [source]
+
+    async def fake_freshness(_db, *, active_region):
+        return 12, datetime.now(UTC)
+
+    async def fake_coverage(_db, *, active_region, recent_after):
+        return set()
+
+    async def fake_fetch(*_args, **_kwargs):
+        return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
+
+    async def fake_noop(*_args, **_kwargs):
+        return None
+
+    async def fake_upsert(_db, _items):
+        return 1
+
+    monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
+    monkeypatch.setattr(
+        "app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
+    )
+    monkeypatch.setattr(
+        "app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
+    )
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
+    monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
+    monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
+    monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
+    monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
+    monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
+    monkeypatch.setattr(scheduler_module, "touch_earth_news_refresh_status", lambda *_args: True)
+    monkeypatch.setattr(
+        scheduler_module,
+        "set_earth_news_refresh_status",
+        lambda refresh_id, region, status, *, error=None: statuses.append(
+            {"refresh_id": refresh_id, "region": region, "status": status, "error": error}
+        ),
+    )
+    scheduler_module._pending_earth_news_refreshes["europe"] = {
+        "force": True,
+        "refresh_id": "refresh-commit-failure",
+    }
+
+    await scheduler_module._run_earth_news_region_refresh("europe")
+
+    assert [entry["status"] for entry in statuses] == ["running", "failed"]
+    assert statuses[-1]["error"]["code"] == "refresh_failed"
+    assert "sensitive context" not in repr(statuses)
+    assert not redis_client.values
+
+
+@pytest.mark.asyncio
+async def test_region_refresh_skips_when_cross_process_lock_is_held(monkeypatch):
+    redis_client = FakeRedis()
+    redis_client.values[earth_news_refresh._lock_key("global")] = "other-owner"
+
+    def fail_session_factory():
+        raise AssertionError("a locked refresh must not open a database session")
+
+    monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
+
+    result = await earth_news_refresh.refresh_earth_news_region("global")
+
+    assert result["state"] == "busy"
+
+
+def test_failed_feed_enters_cooldown_without_blocking_other_sources():
+    redis_client = FakeRedis()
+    source = NewsFeedSource(
+        id="source-a",
+        name="Source A",
+        region="global",
+        feed_url="https://example.com/a.xml",
+        homepage_url="https://example.com",
+        feeds=(NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml"),),
+    )
+    health = {
+        "source-a": {
+            "ok": False,
+            "feed_results": [{"feed_id": "feed-a", "ok": False}],
+        }
+    }
+
+    earth_news_refresh._record_failed_feed_cooldowns(
+        redis_client,
+        active_region="global",
+        health_by_source=health,
+        sources=[source],
+    )
+    selected, cooling = earth_news_refresh._select_feeds(
+        [source],
+        active_region="global",
+        required_keys={("source-a", "feed-a")},
+        redis_client=redis_client,
+    )
+
+    assert selected == []
+    assert cooling == {("source-a", "feed-a")}
+
+
+@pytest.mark.asyncio
+async def test_region_refresh_skips_when_redis_coordination_is_unavailable(monkeypatch):
+    def fail_redis_client():
+        raise RuntimeError("redis unavailable")
+
+    def fail_session_factory():
+        raise AssertionError("uncoordinated processes must not start duplicate RSS fetches")
+
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", fail_redis_client)
+    monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
+
+    result = await earth_news_refresh.refresh_earth_news_region("global")
+
+    assert result["state"] == "coordination_unavailable"
+
+
+def test_refresh_status_deduplicates_active_region_and_bounds_completed_results(monkeypatch):
+    redis_client = FakeStatusRedis()
+    monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
+
+    first, created = earth_news_refresh.create_earth_news_refresh_status("global")
+    duplicate, duplicate_created = earth_news_refresh.create_earth_news_refresh_status("global")
+    assert created is True
+    assert duplicate_created is False
+    assert duplicate == first
+
+    earth_news_refresh.set_earth_news_refresh_status(first["refresh_id"], "global", "running")
+    running, running_created = earth_news_refresh.create_earth_news_refresh_status("global")
+    assert running_created is False
+    assert running["status"] == "running"
+
+    completed = earth_news_refresh.set_earth_news_refresh_status(
+        first["refresh_id"], "global", "succeeded"
+    )
+    assert completed["status"] == "succeeded"
+    assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
+
+    second, second_created = earth_news_refresh.create_earth_news_refresh_status("global")
+    assert second_created is True
+    assert second["refresh_id"] != first["refresh_id"]
+    assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
+    assert redis_client.expirations[earth_news_refresh._refresh_active_key("global")] == 120
+    assert (
+        redis_client.expirations[earth_news_refresh._refresh_status_key(second["refresh_id"])]
+        == 600
+    )
+    status_keys = [
+        key
+        for key in redis_client.values
+        if key.startswith(earth_news_refresh.REFRESH_STATUS_PREFIX)
+    ]
+    assert len(status_keys) == 2
+
+    pending, pending_created = earth_news_refresh.create_earth_news_refresh_status("europe")
+    assert pending_created is True
+    redis_client.values.pop(earth_news_refresh._refresh_active_key("europe"), None)
+    expired = earth_news_refresh.get_earth_news_refresh_status(pending["refresh_id"])
+    assert expired["status"] == "failed"
+    assert expired["error"]["code"] == "expired"
+
+
+@pytest.mark.asyncio
+async def test_earth_news_refresh_scheduler_job_survives_datasource_sync(monkeypatch):
+    class FakeJob:
+        def __init__(self, job_id: str) -> None:
+            self.id = job_id
+
+    class FakeScheduler:
+        def __init__(self) -> None:
+            self.running = False
+            self.jobs = {
+                "obsolete": FakeJob("obsolete"),
+                _earth_news_region_refresh_job_id("global"): FakeJob(
+                    _earth_news_region_refresh_job_id("global")
+                ),
+            }
+            self.added: dict[str, object] = {}
+
+        def add_job(self, _func, **kwargs):
+            self.added.update(kwargs)
+            self.jobs[kwargs["id"]] = FakeJob(kwargs["id"])
+
+        def start(self) -> None:
+            self.running = True
+
+        def get_jobs(self):
+            return list(self.jobs.values())
+
+        def remove_job(self, job_id: str) -> None:
+            self.jobs.pop(job_id, None)
+
+    class FakeResult:
+        def scalars(self):
+            return self
+
+        def all(self):
+            return []
+
+    class FakeDb:
+        async def execute(self, _query):
+            return FakeResult()
+
+    fake_scheduler = FakeScheduler()
+
+    @asynccontextmanager
+    async def fake_session_factory():
+        yield FakeDb()
+
+    monkeypatch.setattr("app.services.scheduler.scheduler", fake_scheduler)
+    monkeypatch.setattr("app.services.scheduler.async_session_factory", fake_session_factory)
+
+    start_scheduler()
+    await sync_scheduler_with_datasources()
+
+    assert fake_scheduler.added["id"] == EARTH_NEWS_REFRESH_JOB_ID
+    assert fake_scheduler.added["next_run_time"] <= datetime.now(UTC)
+    assert EARTH_NEWS_REFRESH_JOB_ID in fake_scheduler.jobs
+    assert _earth_news_region_refresh_job_id("global") in fake_scheduler.jobs
+    assert "obsolete" not in fake_scheduler.jobs
+
+
+@pytest.mark.asyncio
+async def test_region_refresh_requests_are_merged_and_force_is_preserved(monkeypatch):
+    class FakeScheduler:
+        running = True
+
+        def __init__(self) -> None:
+            self.jobs: dict[str, tuple[object, dict[str, object]]] = {}
+
+        def add_job(self, func, **kwargs):
+            self.jobs[kwargs["id"]] = (func, kwargs)
+
+    scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
+    fake_scheduler = FakeScheduler()
+    monkeypatch.setattr(scheduler_module, "scheduler", fake_scheduler)
+    captured: list[tuple[str, bool]] = []
+
+    async def fake_refresh(region: str, *, force: bool = False):
+        captured.append((region, force))
+        return {"locked": False, "fetched": 0, "upserted": 0}
+
+    monkeypatch.setattr(scheduler_module, "refresh_earth_news_region", fake_refresh)
+
+    assert schedule_earth_news_region_refresh("global") is True
+    assert schedule_earth_news_region_refresh("global", force=True) is False
+    assert len(fake_scheduler.jobs) == 1
+
+    job_id = _earth_news_region_refresh_job_id("global")
+    func, kwargs = fake_scheduler.jobs.pop(job_id)
+    await func(**kwargs["kwargs"])
+
+    assert captured == [("global", True)]
+    assert job_id not in fake_scheduler.jobs

backend/tests/test_earth_outbox_maintenance.py

@@ -0,0 +1,140 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+import pytest
+
+from app.services import earth_db_change_listener as listener_module
+from app.services.earth_db_change_listener import EarthDbChangeDispatcher, EarthDbChangeListener
+from app.services.earth_outbox_maintenance import (
+    DELETE_BATCH_SQL,
+    compact_consumed_events,
+    ensure_retention_index,
+    purge_consumed_events,
+)
+
+
+class FakeConnection:
+    def __init__(self, *, index_state: str = "ready", batch_counts: list[int] | None = None):
+        self.index_state = index_state
+        self.batch_counts = list(batch_counts or [])
+        self.calls: list[tuple[str, tuple[object, ...]]] = []
+        self.retained = {"total": 13, "pending": 3}
+        self.restored = dict(self.retained)
+        self.rolled_back = False
+
+    async def fetchrow(self, query: str, *args: object):
+        self.calls.append((query, args))
+        if "pg_index" in query:
+            if self.index_state == "missing":
+                return None
+            return {"indisvalid": self.index_state == "ready", "indisready": True}
+        if "FROM earth_outbox_retained" in query:
+            return self.retained
+        return self.restored
+
+    async def fetchval(self, query: str, *args: object):
+        self.calls.append((query, args))
+        if query == DELETE_BATCH_SQL:
+            return self.batch_counts.pop(0)
+        return True
+
+    async def execute(self, query: str, *args: object):
+        self.calls.append((query, args))
+        if "CREATE INDEX CONCURRENTLY" in query:
+            self.index_state = "ready"
+        return "OK"
+
+    def transaction(self):
+        connection = self
+
+        class Transaction:
+            async def __aenter__(self):
+                return self
+
+            async def __aexit__(self, exc_type, _exc, _traceback):
+                connection.rolled_back = exc_type is not None
+
+        return Transaction()
+
+
+@pytest.mark.asyncio
+async def test_retention_skips_without_index():
+    conn = FakeConnection(index_state="missing")
+    assert await purge_consumed_events(conn, retention_days=1) == 0
+    assert not any(query == DELETE_BATCH_SQL for query, _ in conn.calls)
+
+
+@pytest.mark.asyncio
+async def test_retention_batches_only_consumed_old_rows():
+    conn = FakeConnection(batch_counts=[1000, 19])
+    deleted = await purge_consumed_events(
+        conn, retention_days=1, batch_size=1000, max_batches=10, pause_seconds=0
+    )
+    assert deleted == 1019
+    calls = [args for query, args in conn.calls if query == DELETE_BATCH_SQL]
+    assert len(calls) == 2
+    assert all(isinstance(args[0], datetime) and args[0].tzinfo is UTC for args in calls)
+    assert "consumed_at IS NOT NULL AND consumed_at < $1" in DELETE_BATCH_SQL
+    assert "FOR UPDATE SKIP LOCKED" in DELETE_BATCH_SQL
+
+
+@pytest.mark.asyncio
+async def test_index_creation_is_concurrent_and_repairs_invalid_index():
+    conn = FakeConnection(index_state="invalid")
+    await ensure_retention_index(conn)
+    statements = [query for query, _ in conn.calls]
+    assert any("DROP INDEX CONCURRENTLY" in query for query in statements)
+    assert any("CREATE INDEX CONCURRENTLY" in query for query in statements)
+
+
+@pytest.mark.asyncio
+async def test_compact_retains_pending_and_recent_consumed_in_one_transaction():
+    conn = FakeConnection()
+    assert await compact_consumed_events(conn, retention_days=1) == (13, 3)
+    statements = [query for query, _ in conn.calls]
+    assert next(i for i, query in enumerate(statements) if query.startswith("LOCK TABLE")) < next(
+        i for i, query in enumerate(statements) if query.startswith("TRUNCATE TABLE")
+    )
+    assert any("consumed_at IS NULL OR consumed_at >= $1" in query for query in statements)
+    assert "TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT" in statements
+    assert not conn.rolled_back
+
+
+@pytest.mark.asyncio
+async def test_compact_verification_failure_rolls_back():
+    conn = FakeConnection()
+    conn.restored = {"total": 12, "pending": 2}
+    with pytest.raises(RuntimeError, match="verification failed"):
+        await compact_consumed_events(conn, retention_days=1)
+    assert conn.rolled_back
+
+
+@pytest.mark.asyncio
+async def test_listener_runs_retention_at_most_once_per_interval(monkeypatch):
+    calls: list[str] = []
+
+    async def acquire(_conn):
+        calls.append("acquire")
+        return True
+
+    async def purge(_conn, *, retention_days):
+        assert retention_days == 1
+        calls.append("purge")
+
+    async def release(_conn):
+        calls.append("release")
+
+    monkeypatch.setattr(listener_module, "monotonic", lambda: 100.0)
+    monkeypatch.setattr(listener_module, "try_maintenance_lock", acquire)
+    monkeypatch.setattr(listener_module, "purge_consumed_events", purge)
+    monkeypatch.setattr(listener_module, "release_maintenance_lock", release)
+    monkeypatch.setattr(listener_module, "configured_retention_days", lambda: 1)
+    dispatcher = EarthDbChangeDispatcher(broadcast_earth_update=lambda _: None)
+    listener = EarthDbChangeListener(dsn="postgresql://example/db", dispatcher=dispatcher)
+    listener._connection = FakeConnection()
+
+    await listener._maybe_purge_outbox()
+    await listener._maybe_purge_outbox()
+
+    assert calls == ["acquire", "purge", "release"]

backend/tests/test_llm_provider_catalog.py

@@ -0,0 +1,219 @@
+from copy import deepcopy
+
+import httpx
+import pytest
+
+from app.services import llm_model_catalog as discovery
+from app.services import llm_provider_catalog as catalog
+
+REAL_CLIENT = httpx.AsyncClient
+
+
+def mock_http(monkeypatch, handler):
+    transport = httpx.MockTransport(handler)
+    monkeypatch.setattr(
+        discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
+    )
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "provider,path,auth",
+    [
+        ("minimax", "/anthropic/v1/models", "x-api-key"),
+        ("anthropic", "/v1/models", "x-api-key"),
+        ("openai", "/v1/models", "authorization"),
+        ("deepseek", "/v1/models", "authorization"),
+        ("alibaba", "/api/v1/models", "authorization"),
+        ("moonshotai", "/v1/models", "authorization"),
+        ("openrouter", "/api/v1/models", "authorization"),
+        ("opencode-go", "/zen/go/v1/models", "authorization"),
+        ("ollama", "/api/tags", "authorization"),
+    ],
+)
+async def test_official_catalog_requests(monkeypatch, provider, path, auth):
+    preset = catalog.get_fallback_llm_provider_preset(provider)
+
+    def handle(request):
+        assert request.url.path == path
+        assert request.url.host == httpx.URL(preset["base_url"]).host
+        assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
+        if provider == "alibaba":
+            assert request.url.params["capabilities"] == "TG"
+            return httpx.Response(
+                200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
+            )
+        if provider == "ollama":
+            return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
+        return httpx.Response(200, json={"data": [{"id": "latest"}]})
+
+    mock_http(monkeypatch, handle)
+    result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
+    assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
+    assert result["base_url"] == preset["base_url"]
+    assert "test-key" not in str(result)
+
+
+@pytest.mark.parametrize(
+    "provider,base,api,path",
+    [
+        (
+            "minimax",
+            "https://api.minimax.io/anthropic/v1/",
+            "anthropic-messages",
+            "/anthropic/v1/models",
+        ),
+        ("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
+        (
+            "alibaba",
+            "https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
+            "openai-completions",
+            "/api/v1/models",
+        ),
+        (
+            "alibaba",
+            "https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
+            "openai-completions",
+            "/api/v1/models",
+        ),
+        ("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
+        ("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
+        ("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
+    ],
+)
+def test_urls_preserve_region_and_gateway(provider, base, api, path):
+    url = httpx.URL(discovery.model_catalog_url(provider, base, api))
+    assert url.host == httpx.URL(base).host
+    assert url.path == path
+
+
+@pytest.mark.asyncio
+async def test_anthropic_pagination_and_release_order(monkeypatch):
+    seen = []
+
+    def handle(request):
+        seen.append(request.url.params.get("after_id"))
+        if len(seen) == 1:
+            return httpx.Response(
+                200,
+                json={
+                    "data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
+                    "has_more": True,
+                    "last_id": "old",
+                },
+            )
+        return httpx.Response(
+            200,
+            json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
+        )
+
+    mock_http(monkeypatch, handle)
+    result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
+    assert seen == [None, "old"]
+    assert result["models"] == ["new", "old"]
+
+
+@pytest.mark.asyncio
+async def test_dashscope_pagination(monkeypatch):
+    def handle(request):
+        page = int(request.url.params["page_no"])
+        return httpx.Response(
+            200,
+            json={
+                "output": {
+                    "total": 2,
+                    "models": [
+                        {"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
+                    ],
+                }
+            },
+        )
+
+    mock_http(monkeypatch, handle)
+    result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
+    assert result["models"] == ["page-2", "page-1"]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "payload",
+    [
+        {},
+        {"data": []},
+        {"data": [None]},
+        {"data": [{}]},
+        {"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
+    ],
+)
+async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
+    mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
+    with pytest.raises(discovery.LLMProviderCatalogError):
+        await catalog.refresh_llm_provider_preset("minimax", "test-key")
+
+
+@pytest.mark.asyncio
+async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
+    before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
+    rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
+    mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
+    result = await catalog.refresh_llm_provider_preset("openai", "test-key")
+    assert len(result["models"]) == 140
+    assert result["models"][0] == "model-139"
+    assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
+
+
+@pytest.mark.asyncio
+async def test_empty_ollama_catalog_is_valid(monkeypatch):
+    mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
+    result = await catalog.refresh_llm_provider_preset("ollama")
+    assert result["models"] == []
+    assert result["model"] == ""
+
+
+@pytest.mark.asyncio
+async def test_opencode_documented_protocols(monkeypatch):
+    models = [
+        "minimax-m3",
+        "qwen3.8-max",
+        "gpt-5.6-luna",
+        "grok-4.6",
+        "muse-spark-1.3-contributor",
+        "kimi-k3",
+    ]
+    mock_http(
+        monkeypatch,
+        lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
+    )
+    result = await catalog.refresh_llm_provider_preset("opencode-go")
+    assert list(result["model_provider_apis"].values()) == [
+        "anthropic-messages",
+        "anthropic-messages",
+        "openai-responses",
+        "openai-responses",
+        "openai-responses",
+        "openai-completions",
+    ]
+
+
+@pytest.mark.asyncio
+async def test_retry_transient_error_only(monkeypatch):
+    calls = []
+
+    def handle(request):
+        calls.append(request)
+        return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
+
+    mock_http(monkeypatch, handle)
+    await catalog.refresh_llm_provider_preset("minimax", "test-key")
+    assert len(calls) == 2
+    calls.clear()
+
+    def unauthorized(request):
+        calls.append(request)
+        return httpx.Response(401, text="secret-upstream-text")
+
+    mock_http(monkeypatch, unauthorized)
+    with pytest.raises(httpx.HTTPStatusError) as error:
+        await catalog.refresh_llm_provider_preset("minimax", "test-key")
+    assert len(calls) == 1
+    assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)

backend/tests/test_llm_provider_settings.py

@@ -0,0 +1,165 @@
+from copy import deepcopy
+from types import SimpleNamespace
+from unittest.mock import AsyncMock
+
+import httpx
+import pytest
+from sqlalchemy import create_engine, select
+from sqlalchemy.orm import Session
+
+from app.api.v1 import settings as api
+from app.models.system_setting import SystemSetting
+from app.services import llm_model_catalog as discovery
+from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
+
+REAL_CLIENT = httpx.AsyncClient
+
+
+@pytest.fixture
+def settings_db(monkeypatch, tmp_path):
+    # Real SQL reads, commits and reloads; do not mock the settings helpers.
+    engine = create_engine("sqlite://")
+    SystemSetting.__table__.create(engine)
+    session = Session(engine)
+    db = SimpleNamespace(
+        execute=AsyncMock(side_effect=session.execute),
+        add=session.add,
+        commit=AsyncMock(side_effect=session.commit),
+        refresh=AsyncMock(side_effect=session.refresh),
+    )
+    monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
+    monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
+    yield db
+    session.close()
+    engine.dispose()
+
+
+def use_upstream(monkeypatch, handler):
+    transport = httpx.MockTransport(handler)
+    monkeypatch.setattr(
+        discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
+    )
+
+
+@pytest.mark.asyncio
+async def test_fresh_database_lists_all_builtin_presets(settings_db):
+    result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
+    assert len(result["data"]) == 9
+    assert all(row["models"] for row in result["data"])
+    assert "MiniMax-M3" in result["data"][0]["models"]
+
+
+@pytest.mark.asyncio
+async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
+    runtime = {
+        "ai_provider": {
+            "default_provider": "minimax",
+            "service_token": "private-service-token",
+            "providers": {
+                "minimax": {
+                    "base_url": "https://saved.test/anthropic",
+                    "model": "old",
+                    "api_key": "saved-secret",
+                }
+            },
+        }
+    }
+    await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
+    before = await api.get_setting_payload(settings_db, "external_integrations")
+    seen = []
+
+    def handle(request):
+        seen.append(request)
+        assert str(request.url) == "https://draft.test/anthropic/v1/models"
+        assert request.headers["x-api-key"] == "draft-secret"
+        return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
+
+    use_upstream(monkeypatch, handle)
+    draft = api.AIProviderIntegrationUpdate(
+        provider="minimax",
+        base_url="https://draft.test/anthropic",
+        model="unsaved-model",
+        api_key="draft-secret",
+        service_token="unsaved-service-token",
+    )
+    refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
+    result = await api.get_ai_provider_presets(None, settings_db)
+    preset = next(row for row in result["data"] if row["provider"] == "minimax")
+    assert len(seen) == 1
+    assert preset["models"] == ["MiniMax-M3", "old"]
+    assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
+    assert await api.get_setting_payload(settings_db, "external_integrations") == before
+    assert "secret" not in str(result)
+    assert "private-service-token" not in str(result)
+    assert draft.model == "unsaved-model"
+    stored = await settings_db.execute(
+        select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
+    )
+    assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
+
+
+@pytest.mark.asyncio
+async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
+    settings_db, monkeypatch
+):
+    await api.save_setting_payload(
+        settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
+    )
+    use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
+    commits = settings_db.commit.await_count
+    with pytest.raises(api.HTTPException) as error:
+        await api.refresh_ai_provider_preset("openrouter", None, settings_db)
+    assert error.value.status_code == 502
+    assert "private-upstream-secret" not in error.value.detail
+    assert settings_db.commit.await_count == commits
+    assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
+        "models"
+    ] == ["saved-model"]
+
+
+@pytest.mark.asyncio
+async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
+    def fail_request(request):
+        pytest.fail("must not query a private catalog without credentials")
+
+    use_upstream(monkeypatch, fail_request)
+    with pytest.raises(api.HTTPException) as error:
+        await api.refresh_ai_provider_preset("minimax", None, settings_db)
+    assert error.value.status_code == 400
+    assert "API Key" in error.value.detail
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
+)
+async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
+    use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
+    result = await api._check_ai_provider_lightweight(
+        {**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
+    )
+    assert result["success"] is False
+    assert result["connected"] is False
+    assert "private-upstream-secret" not in result["message"]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
+async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
+    use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
+    preset = list_fallback_llm_provider_presets()[0]
+    result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
+    assert result["success"] is False
+
+
+@pytest.mark.asyncio
+async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
+    use_upstream(
+        monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
+    )
+    preset = list_fallback_llm_provider_presets()[0]
+    result = await api._check_ai_provider_lightweight(
+        {**preset, "model": "future-model", "api_key": "test-key"}, 5
+    )
+    assert result["success"] is True
+    assert result["url"].endswith("/anthropic/v1/models")

backend/tests/test_logging.py

@@ -10,7 +10,7 @@ from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
 from app.core.request_context import set_request_id
 from app.services import business_logs
 from app.services import persistent_logs
-from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup
+from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup, SystemLog
 
 
 def _capture_output(callback):
@@ -100,6 +100,23 @@ def test_business_context_redacts_nested_sensitive_values():
     assert context["nested"]["safe"] == "visible"
 
 
+@pytest.mark.asyncio
+async def test_business_log_preserves_events_in_isolated_store(persisted_test_logs):
+    await business_logs.emit_business_log(
+        get_logger("tests.persistence"),
+        event="test.business.completed",
+        message="Business action completed",
+        category="test",
+        context={"api_key": "test-secret", "record_count": 2},
+    )
+
+    logs = [entry for entry in persisted_test_logs if isinstance(entry, SystemLog)]
+    assert len(logs) == 1
+    assert logs[0].event == "test.business.completed"
+    assert logs[0].context == {"api_key": "[REDACTED]", "record_count": 2}
+    assert any(isinstance(entry, ObservabilityEventGroup) for entry in persisted_test_logs)
+
+
 def test_observability_fingerprint_normalizes_hls_fragments():
     first = persistent_logs.build_observability_fingerprint(
         source="earth-client",

backend/tests/test_news_refresh_api.py

@@ -0,0 +1,142 @@
+from types import SimpleNamespace
+from uuid import UUID
+import importlib
+
+import pytest
+from fastapi import HTTPException
+
+from app.api.v1 import earth, news
+
+
+@pytest.mark.asyncio
+async def test_post_news_refresh_only_schedules_background_region_check(monkeypatch):
+    scheduled = []
+
+    def request(region: str):
+        scheduled.append(region)
+        return {"refresh_id": "refresh-europe", "region": region, "status": "pending"}
+
+    async def fail_network_refresh(*_args, **_kwargs):
+        raise AssertionError("the POST handler must not perform RSS requests")
+
+    monkeypatch.setattr("app.services.scheduler.request_earth_news_region_refresh", request)
+    monkeypatch.setattr("app.services.scheduler.refresh_earth_news_region", fail_network_refresh)
+
+    response = await news.request_earth_news_refresh(
+        news.EarthNewsRefreshRequest(region="Europe", lat=48.8, lon=2.3)
+    )
+
+    assert response == {"refresh_id": "refresh-europe", "region": "europe", "status": "pending"}
+    assert scheduled == ["europe"]
+    refresh_route = next(route for route in news.router.routes if route.path == "/refresh")
+    assert refresh_route.status_code == 202
+
+
+@pytest.mark.asyncio
+async def test_post_news_refresh_infers_region_and_rejects_unknown_region(monkeypatch):
+    scheduled = []
+    monkeypatch.setattr(
+        "app.services.scheduler.request_earth_news_region_refresh",
+        lambda region: scheduled.append(region)
+        or {"refresh_id": "refresh-americas", "region": region, "status": "pending"},
+    )
+
+    response = await news.request_earth_news_refresh(
+        news.EarthNewsRefreshRequest(lat=40.7, lon=-74.0)
+    )
+
+    assert response["region"] == "americas"
+    assert response["status"] == "pending"
+    assert scheduled == ["americas"]
+    with pytest.raises(HTTPException) as error:
+        await news.request_earth_news_refresh(news.EarthNewsRefreshRequest(region="unknown"))
+    assert error.value.status_code == 422
+
+
+@pytest.mark.asyncio
+async def test_get_news_refresh_returns_status_and_expired_shape(monkeypatch):
+    current = {"refresh_id": "abc", "region": "global", "status": "running"}
+    monkeypatch.setattr(
+        "app.services.earth_news_refresh.get_earth_news_refresh_status",
+        lambda _refresh_id: current,
+    )
+
+    response = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
+
+    assert response is current
+    assert response["status"] == "running"
+    monkeypatch.setattr(
+        "app.services.earth_news_refresh.get_earth_news_refresh_status", lambda _id: None
+    )
+    expired = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
+    assert expired == {
+        "refresh_id": UUID("00000000-0000-0000-0000-000000000abc").hex,
+        "region": None,
+        "status": "failed",
+        "error": {
+            "code": "expired",
+            "message": "Refresh status expired. Submit a new refresh.",
+        },
+    }
+
+
+@pytest.mark.asyncio
+async def test_database_failure_sets_terminal_failed_status_without_leaking_error(monkeypatch):
+    scheduler = importlib.import_module("app.services.scheduler")
+    statuses = []
+
+    async def fail_refresh(_region, *, force=False):
+        raise RuntimeError("postgres://private-token@host/database: commit failed")
+
+    monkeypatch.setattr(scheduler, "refresh_earth_news_region", fail_refresh)
+    monkeypatch.setattr(scheduler, "touch_earth_news_refresh_status", lambda *_args: True)
+    monkeypatch.setattr(
+        scheduler,
+        "set_earth_news_refresh_status",
+        lambda refresh_id, region, state, *, error=None: statuses.append(
+            {"refresh_id": refresh_id, "region": region, "status": state, "error": error}
+        ),
+    )
+    scheduler._pending_earth_news_refreshes["europe"] = {
+        "force": True,
+        "refresh_id": "refresh-db-failure",
+    }
+
+    await scheduler._run_earth_news_region_refresh("europe")
+
+    assert [item["status"] for item in statuses] == ["running", "failed"]
+    assert statuses[-1]["error"] == {
+        "code": "refresh_failed",
+        "message": "Could not refresh news. Please try again.",
+    }
+    assert "private-token" not in repr(statuses)
+
+
+@pytest.mark.asyncio
+async def test_news_source_save_and_reset_schedule_immediate_refresh(monkeypatch):
+    scheduled = []
+    monkeypatch.setattr(
+        earth,
+        "save_earth_news_sources_payload",
+        lambda _db, _payload: _async_result({"status": "updated"}),
+    )
+    monkeypatch.setattr(
+        earth,
+        "reset_earth_news_sources_payload",
+        lambda _db: _async_result({"status": "reset"}),
+    )
+    monkeypatch.setattr(
+        "app.services.scheduler.schedule_earth_news_refresh_for_all_regions",
+        lambda *, force=False: scheduled.append(force),
+    )
+
+    await earth.update_earth_news_sources(
+        SimpleNamespace(model_dump=lambda: {}), _current_user=None, db=object()
+    )
+    await earth.reset_earth_news_sources(_current_user=None, db=object())
+
+    assert scheduled == [True, True]
+
+
+async def _async_result(value):
+    return value

backend/tests/test_provider_protocols.py

@@ -0,0 +1,141 @@
+from pathlib import Path
+import sys
+from unittest.mock import patch
+
+import httpx
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+with patch(
+    "pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
+    return_value={},
+):
+    from aiprovider.provider_service import ProviderService
+from aiprovider.schemas import SituationalAnalysisRequest
+from app.api.v1.settings import _runtime_config_from_ai_payload
+
+
+def test_custom_openai_gateway_keeps_its_configured_protocol():
+    config = _runtime_config_from_ai_payload(
+        {
+            "default_provider": "openai",
+            "providers": {
+                "openai": {
+                    "base_url": "https://gateway.test/v1",
+                    "provider_api": "openai-completions",
+                    "model": "gpt-6-astra",
+                },
+            },
+        }
+    )["llm_config"]
+    service = ProviderService({**config, "api_key": "test-key"})
+    assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "model,path",
+    [("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
+)
+async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
+    llm = _runtime_config_from_ai_payload(
+        {
+            "default_provider": "opencode-go",
+            "providers": {
+                "opencode-go": {
+                    "model": model,
+                    "api_key": "test-key",
+                    "model_provider_apis": {model: "openai-completions"},
+                }
+            },
+        }
+    )["llm_config"]
+    service = ProviderService(llm)
+    seen = []
+
+    async def post(**request):
+        seen.append(request)
+        assert request["path"] == path
+        assert request["request_body"]["model"] == model
+        if path == "/responses":
+            assert request["request_body"]["store"] is False
+            assert "max_tokens" not in request["request_body"]
+            return {
+                "output": [
+                    {"type": "reasoning", "summary": [{"text": "reason"}]},
+                    {"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
+                ]
+            }
+        return {"content": [{"type": "text", "text": "OK"}]}
+
+    monkeypatch.setattr(service, "_post", post)
+    result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
+    assert len(seen) == 1
+    assert result.content == "OK"
+
+
+@pytest.mark.asyncio
+async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
+    seen = []
+    client_type = httpx.AsyncClient
+
+    def handle(request):
+        seen.append(str(request.url))
+        return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
+
+    transport = httpx.MockTransport(handle)
+    monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
+    for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
+        service = ProviderService(
+            {
+                "provider": "minimax",
+                "provider_api": "anthropic-messages",
+                "base_url": base,
+                "api_key": "test-key",
+                "model": "MiniMax-M3",
+            }
+        )
+        result = await service.analyze(
+            SituationalAnalysisRequest(title="test", objective="reply OK")
+        )
+        assert result.content == "OK"
+    assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
+
+
+@pytest.mark.asyncio
+async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
+    seen = []
+    client_type = httpx.AsyncClient
+
+    def handle(request):
+        import json
+
+        seen.append(request)
+        assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
+        assert request.headers["user-agent"] == "Planet/1.0"
+        return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
+
+    transport = httpx.MockTransport(handle)
+    monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
+    for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
+        service = ProviderService(
+            {
+                "provider": "opencode-go",
+                "provider_api": "anthropic-messages",
+                "base_url": "https://opencode.ai/zen/go/v1",
+                "api_key": "test-key",
+                "model": "minimax-m3",
+            }
+        )
+        await service.analyze(
+            SituationalAnalysisRequest(
+                title="test",
+                objective="reply OK",
+                thinking={"type": "enabled"},
+                context={"session_id": session_id},
+            )
+        )
+    session_headers = [request.headers["x-opencode-session"] for request in seen]
+    assert session_headers[0] == session_headers[1]
+    assert session_headers[0] != session_headers[2]

backend/tests/test_tv_catalog.py

@@ -0,0 +1,160 @@
+from datetime import UTC, datetime
+from unittest.mock import AsyncMock
+
+import pytest
+from sqlalchemy import create_engine, event
+from sqlalchemy.orm import Session
+
+from app.db.session import Base
+from app.models.collected_data import CollectedData
+from app.models.data_snapshot import DataSnapshot
+from app.models.system_setting import SystemSetting
+from app.models.task import CollectionTask
+from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
+from app.services.tv_catalog import get_tv_catalog_page
+from app.services.tv_streams import normalize_tv_settings
+
+
+class CatalogSession:
+    """Execute the real catalog queries against an isolated SQLite database."""
+
+    def __init__(self, session):
+        self.session = session
+
+    async def execute(self, query):
+        return self.session.execute(query)
+
+    async def scalar(self, query):
+        return self.session.scalar(query)
+
+    async def scalars(self, query):
+        return self.session.scalars(query)
+
+
+@pytest.fixture
+def catalog_db():
+    engine = create_engine("sqlite:///:memory:")
+
+    @event.listens_for(engine, "connect")
+    def register_functions(connection, _record):
+        connection.create_function(
+            "concat_ws",
+            -1,
+            lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
+        )
+
+    Base.metadata.create_all(
+        engine,
+        tables=[
+            CollectionTask.__table__,
+            DataSnapshot.__table__,
+            CollectedData.__table__,
+            SystemSetting.__table__,
+        ],
+    )
+    with Session(engine) as session:
+        for index in range(135):
+            session.add(
+                CollectedData(
+                    source="news_live_streams",
+                    source_id=f"channel-{index:03}",
+                    data_type="news_live_stream",
+                    name=f"Channel {index:03}",
+                    collected_at=datetime(2026, 9, 13, tzinfo=UTC),
+                    is_current=True,
+                    is_valid=1,
+                    extra_data={
+                        "stream_url": "https://example.invalid/live.m3u8",
+                        "region": "Canada",
+                    },
+                )
+            )
+        session.flush()
+        yield CatalogSession(session)
+    engine.dispose()
+
+
+@pytest.mark.asyncio
+async def test_pages_include_entire_catalog_without_overlap(catalog_db):
+    first = await get_tv_catalog_page(catalog_db, limit=50)
+    second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
+    third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
+    ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
+    assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
+    assert len(set(ids)) == first["source_count"] == 145
+    assert ids[-1] == "channel-134"
+    assert third["next_offset"] is None and not third["has_more"]
+    beyond = await get_tv_catalog_page(catalog_db, offset=200)
+    assert beyond["sources"] == [] and not beyond["has_more"]
+
+
+@pytest.mark.asyncio
+async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
+    payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
+    assert [source["id"] for source in payload["sources"]] == ["channel-134"]
+    assert payload["total"] == 1 and payload["source_count"] == 145
+    assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
+
+
+@pytest.mark.asyncio
+async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
+    payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
+    assert payload["selected_source"]["id"] == "channel-134"
+    assert "channel-134" not in [source["id"] for source in payload["sources"]]
+    assert payload["default_source_id"] == "aljazeera-mubasher"
+    default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
+    assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
+
+
+@pytest.mark.asyncio
+async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
+    for name, values in [
+        ("Disabled", {"extra_data": {"is_enabled": False}}),
+        ("Historical", {"is_current": False}),
+        ("Invalid", {"is_valid": 0}),
+        ("Deleted", {"deleted_at": datetime.now(UTC)}),
+        ("Replacement", {"source_id": "channel-134"}),
+    ]:
+        record = dict(
+            source="news_live_streams",
+            source_id=name,
+            name=name,
+            data_type="news_live_stream",
+            is_current=True,
+            is_valid=1,
+        )
+        catalog_db.session.add(CollectedData(**{**record, **values}))
+    catalog_db.session.flush()
+    payload = await get_tv_catalog_page(catalog_db, q="replacement")
+    assert payload["source_count"] == 145
+    assert [source["id"] for source in payload["sources"]] == ["channel-134"]
+
+
+def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
+    settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
+    assert settings["default_source_id"] == "aljazeera-mubasher"
+    assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
+)
+async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
+    monkeypatch, config, expected
+):
+    collector = NewsLiveStreamsCollector()
+    channels = [
+        {"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
+    ]
+    channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
+    streams = [
+        {"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
+        for channel in channels
+    ]
+    monkeypatch.setattr(
+        collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
+    )
+    records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
+    assert len(records) == expected
+    assert all(record["source_id"] != "sport" for record in records)

backend/tests/test_tv_playback.py

@@ -0,0 +1,37 @@
+from app.services.tv_streams import (
+    CCTV4_HOMEPAGE,
+    CCTV4_STREAM,
+    UNSUPPORTED_CCTV4_STREAM,
+    normalize_tv_settings,
+    normalize_tv_source,
+)
+
+
+def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
+    settings = normalize_tv_settings(None)
+    assert settings["default_source_id"] == "aljazeera-mubasher"
+    cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
+    assert cctv["source_type"] == "hls"
+    assert cctv["stream_url"] == CCTV4_STREAM
+    assert "美洲版" in cctv["name"]
+    assert cctv["use_proxy"] is False
+    assert cctv["homepage_url"] == CCTV4_HOMEPAGE
+
+
+def test_saved_legacy_default_is_corrected_without_mutating_input():
+    saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
+    source = normalize_tv_source(saved)
+    assert source["source_type"] == "hls"
+    assert source["stream_url"] == CCTV4_STREAM
+    assert source["use_proxy"] is False
+    assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
+
+
+def test_custom_cctv_stream_and_fallback_preference_are_preserved():
+    custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
+    settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
+    source = next(source for source in settings["sources"] if source["id"] == "cctv4")
+    assert source["stream_url"] == custom["stream_url"]
+    assert source["source_type"] == "hls"
+    assert source["use_proxy"] is True
+    assert settings["auto_fallback"] is False

backend/tests/test_visualization_compute_centers.py

@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
         longitude=-90.05,
         metadata={
             "organization": "xAI",
+            "location": "3231 Riverport Rd, Memphis, TN 38109",
             "gpu_type": "H100",
             "gpu_count": 100000,
             "value": "20000",
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
     compute_center_locations.set_compute_center_location_cache({})
 
 
+def test_epoch_country_level_coordinates_are_unresolved():
+    record = _build_record(
+        record_id=33,
+        source="epoch_ai_gpu",
+        data_type="gpu_cluster",
+        name="Meta 100k",
+        country="United States",
+        city="",
+        latitude=39.381266,
+        longitude=-97.922211,
+        metadata={
+            "location": "United States of America",
+            "raw_country": "United States of America",
+        },
+    )
+
+    payload = convert_compute_centers_to_geojson([record])
+
+    assert payload["features"] == []
+    assert len(payload["unresolved"]) == 1
+    assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
+    assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
+
+
+def test_epoch_city_coordinates_are_not_labeled_precise():
+    record = _build_record(
+        record_id=34,
+        source="epoch_ai_gpu",
+        data_type="gpu_cluster",
+        name="Seoul Cluster",
+        country="South Korea",
+        city="",
+        latitude=37.5665,
+        longitude=126.978,
+        metadata={"location": "Seoul", "raw_country": "South Korea"},
+    )
+
+    result = compute_center_locations.resolve_compute_center_location_full(
+        record, record.extra_data
+    )
+
+    assert result.is_resolved
+    assert result.location.location_precision == "city"
+    assert result.location.needs_confirmation is True
+    assert result.location.is_estimated is True
+    assert result.location.city is None
+    assert result.location.matched_location_name == "Seoul"
+    assert "Epoch Location: Seoul" in result.location.location_source_note
+
+
+def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
+    record = _build_record(
+        record_id=37,
+        source="epoch_ai_gpu",
+        data_type="gpu_cluster",
+        name="Petrobras Fênix",
+        country="Brazil",
+        city="",
+        latitude=-22.909534,
+        longitude=-43.209934,
+        metadata={
+            "location": "Rio de Janeiro, Brazil",
+            "raw_country": "Brazil",
+        },
+    )
+
+    payload = convert_compute_centers_to_geojson([record])
+
+    feature = payload["features"][0]
+    assert feature["properties"]["location_precision"] == "city"
+    assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
+    assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
+        "location_source_note"
+    ]
+
+
+def test_confirmed_stored_location_beats_epoch_precise_coordinates():
+    compute_center_locations.set_compute_center_location_cache({
+        "epoch_ai_gpu:epoch_ai_gpu-35": {
+            "source": "epoch_ai_gpu",
+            "source_id": "epoch_ai_gpu-35",
+            "name": "Confirmed Site",
+            "latitude": 35.0,
+            "longitude": 139.0,
+            "precision": "site",
+            "confidence": 1.0,
+            "location_source": "official_facility_registry",
+            "needs_confirmation": False,
+        }
+    })
+    try:
+        record = _build_record(
+            record_id=35,
+            source="epoch_ai_gpu",
+            data_type="gpu_cluster",
+            name="Japan Cluster",
+            country="Japan",
+            city="Tokyo",
+            latitude=35.68882,
+            longitude=139.692526,
+            metadata={
+                "location": "1-1 Verified Street, Tokyo 100-0001, Japan",
+                "raw_country": "Japan",
+            },
+        )
+
+        result = compute_center_locations.resolve_compute_center_location_full(
+            record, record.extra_data
+        )
+
+        assert result.location.location_source == "stored_compute_center_location"
+        assert result.location.location_precision == "site"
+        assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
+    finally:
+        compute_center_locations.set_compute_center_location_cache({})
+
+
+def test_legacy_seeded_country_coordinate_is_ignored_on_read():
+    compute_center_locations.set_compute_center_location_cache({
+        "epoch_ai_gpu:epoch_ai_gpu-36": {
+            "source": "epoch_ai_gpu",
+            "source_id": "epoch_ai_gpu-36",
+            "name": "Legacy Country Point",
+            "latitude": 36.386493,
+            "longitude": 138.59223,
+            "precision": "precise",
+            "confidence": 1.0,
+            "location_source": "source_coordinates",
+            "needs_confirmation": False,
+        }
+    })
+    try:
+        record = _build_record(
+            record_id=36,
+            source="epoch_ai_gpu",
+            data_type="gpu_cluster",
+            name="Legacy Japan Cluster",
+            country="Japan",
+            city="",
+            latitude=36.386493,
+            longitude=138.59223,
+            metadata={"location": "Japan", "raw_country": "Japan"},
+        )
+
+        result = compute_center_locations.resolve_compute_center_location_full(
+            record, record.extra_data
+        )
+
+        assert result.location is None
+    finally:
+        compute_center_locations.set_compute_center_location_cache({})
+
+
+def test_verified_sites_replace_epoch_country_center_coordinates():
+    specifications = [
+        (
+            "epoch_jamstec_zettascaler_2_0_gyoukou",
+            "Gyoukou 2.0",
+            "Japan",
+            36.386493,
+            138.59223,
+            [139.625008, 35.380924],
+        ),
+        (
+            "epoch_jamstec_zettascaler_2_2_gyoukou",
+            "Gyoukou 2.2",
+            "Japan",
+            36.386493,
+            138.59223,
+            [139.625008, 35.380924],
+        ),
+        (
+            "epoch_argonne_nl_sophia__formerly_theta_",
+            "Sophia",
+            "United States of America",
+            39.381266,
+            -97.922211,
+            [-87.984498648496, 41.718109383287235],
+        ),
+    ]
+    records = []
+    for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
+        specifications, start=40
+    ):
+        record = _build_record(
+            record_id=index,
+            source="epoch_ai_gpu",
+            data_type="gpu_cluster",
+            name=name,
+            country=country,
+            city="",
+            latitude=latitude,
+            longitude=longitude,
+            metadata={"location": country, "raw_country": country},
+        )
+        record.source_id = source_id
+        records.append(record)
+
+    payload = convert_compute_centers_to_geojson(records)
+
+    assert payload["unresolved"] == []
+    assert len(payload["features"]) == 3
+    for feature, specification in zip(payload["features"], specifications):
+        assert feature["geometry"]["coordinates"] == specification[-1]
+        assert feature["properties"]["location_precision"] == "site"
+        assert feature["properties"]["location_source"] == "official_facility_registry"
+        assert "https://" in feature["properties"]["location_source_note"]
+
+
+def test_verified_site_beats_epoch_city_level_source_coordinates():
+    record = _build_record(
+        record_id=44,
+        source="epoch_ai_gpu",
+        data_type="gpu_cluster",
+        name="Gyoukou 2.2",
+        country="Japan",
+        city="Tokyo",
+        latitude=35.68882,
+        longitude=139.692526,
+        metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
+    )
+    record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
+
+    result = compute_center_locations.resolve_compute_center_location_full(
+        record, record.extra_data
+    )
+
+    assert result.location.location_source == "official_facility_registry"
+    assert result.location.location_precision == "site"
+    assert (result.location.latitude, result.location.longitude) == (
+        35.380924,
+        139.625008,
+    )
+
+
 def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
     compute_center_locations.set_compute_center_location_cache({
         "epoch_ai_gpu:epoch_ai_gpu-32": {
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
 
 @pytest.mark.asyncio
 async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
+    monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
     candidate = compute_center_locations.LocationCandidate(
         latitude=45.764,
         longitude=4.8357,
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
 
 @pytest.mark.asyncio
 async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
+    monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
     llm_candidate = compute_center_locations.LocationCandidate(
         latitude=45.764,
         longitude=4.8357,
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
 
 
 @pytest.mark.asyncio
-async def test_compute_centers_geojson_endpoint_returns_stats():
+async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
+    async def build_without_shared_cache(**kwargs):
+        return await kwargs["builder"]()
+
+    monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
     records = [
         _build_record(
             record_id=1,
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
             city="Memphis",
             latitude=35.15,
             longitude=-90.05,
-            metadata={"value": "20000", "unit": "TFlop/s"},
+            metadata={
+                "location": "3231 Riverport Rd, Memphis, TN 38109",
+                "value": "20000",
+                "unit": "TFlop/s",
+            },
         ),
     ]
 
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
 
 @pytest.mark.asyncio
 async def test_visualization_geo_summary_returns_counts(monkeypatch):
+    async def build_without_shared_cache(**kwargs):
+        return await kwargs["builder"]()
+
+    # Exercise the fixture DB, never the running development server's Redis payload.
+    monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
     records = [
         _build_record(
             record_id=1,
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
             city="Memphis",
             latitude=35.15,
             longitude=-90.05,
-            metadata={"value": "20000", "unit": "TFlop/s"},
+            metadata={
+                "location": "3231 Riverport Rd, Memphis, TN 38109",
+                "value": "20000",
+                "unit": "TFlop/s",
+            },
         ),
     ]
 
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
     class _FakeSession:
         async def execute(self, query):
             query_text = str(query).lower()
+            if "group by collected_data.source" in query_text:
+                return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
             if "bgp_incidents" in query_text:
                 return _ScalarResult(scalar_value=2)
             if "bgp_anomalies" in query_text:
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
 
 @pytest.mark.asyncio
 async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
+    monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
     def _fake_ror(query):
         assert query == "Oak Ridge National Laboratory"
         return {
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
 
 @pytest.mark.asyncio
 async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
+    monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
     monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
     monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
 
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
                 assert "FORBIDDEN_PRECISIONS" in text, (
                     f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
                 )
+
+
+@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
+def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
+    from app.core.compute_types import compute_site_type
+    record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
+        name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
+        metadata={"location_precision": "site"})
+    result = convert_compute_centers_to_geojson([record])
+    assert compute_site_type(record) == kind
+    assert result["features"][0]["properties"]["site_type"] == kind
+    record.source = "epoch_ai_gpu"
+    record.data_type = "gpu_cluster"
+    record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
+    assert compute_site_type(record) == kind
+    assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind

backend/tests/test_websocket_manager.py

@@ -1,5 +1,9 @@
-import pytest
 import importlib
+import json
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
 
 from app.core.websocket.manager import ConnectionManager
 from app.core.websocket.broadcaster import DataBroadcaster
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
     broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
     monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
     broadcaster = DataBroadcaster()
+    async def load_current(keys):
+        assert keys == ["1"]
+        return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
+
+    monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
     broadcaster.enqueue_vessel_update(
         {
             "source": "aisstream_vessels",
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
     assert sent[0]["vessels"] == [
         {
             "mmsi": 1,
-            "lat": 59.1,
-            "lon": 10.1,
-            "source": "aisstream_vessels",
+            "lat": 60.0,
+            "lon": 11.0,
+            "source": "barentswatch_vessels",
             "action": "upsert",
             "created": None,
         }
     ]
+
+
+@pytest.mark.asyncio
+async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
+    manager = ConnectionManager()
+    socket = FakeWebSocket()
+    config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
+    json.dumps(config)
+    vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
+    await manager.broadcast_vessels({"vessels": vessels})
+
+    assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
+    assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
+    manager.unsubscribe(socket, ["vessels"])
+    await manager.broadcast_vessels({"vessels": vessels})
+    assert len(socket.sent) == 3
+
+
+@pytest.mark.asyncio
+async def test_global_vessel_removal_does_not_require_coordinates():
+    manager = ConnectionManager()
+    socket = FakeWebSocket()
+    manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
+    await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
+    assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
+
+
+def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
+    websocket_module = importlib.import_module("app.api.v1.websocket")
+    monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
+    app = FastAPI()
+    app.include_router(websocket_module.router)
+    with TestClient(app) as client, client.websocket_connect("/ws") as socket:
+        assert socket.receive_json()["type"] == "connection_established"
+        socket.send_json({
+            "type": "subscribe",
+            "data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
+        })
+        response = socket.receive_json()
+        assert response["type"] == "subscription_confirmed"
+        assert response["data"]["vessels"]["scope"] == "global"
+        assert response["data"]["vessels"]["type"] == []
+
+
+@pytest.mark.asyncio
+async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
+    broadcaster = DataBroadcaster()
+    broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
+
+    async def fail_read(_keys):
+        broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
+        raise RuntimeError("database unavailable")
+
+    monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
+    with pytest.raises(RuntimeError, match="database unavailable"):
+        await broadcaster.flush_vessel_updates()
+    assert broadcaster._pending_vessel_updates["1"]["lat"] == 61

config/earth-boundary-pov-policy.china-v1.json

@@ -25,6 +25,25 @@
     {
       "id": "china-taiwan-penghu",
       "name": "Taiwan and Penghu",
+      "sourceIsoA3": ["TWN"],
+      "action": "union_to_country",
+      "targetIsoA3": "CHN",
+      "hoverIsoA3": "CHN",
+      "labelPolicy": "show_country_only"
+    },
+    {
+      "id": "china-hong-kong",
+      "name": "Hong Kong",
+      "sourceIsoA3": ["HKG"],
+      "action": "union_to_country",
+      "targetIsoA3": "CHN",
+      "hoverIsoA3": "CHN",
+      "labelPolicy": "show_country_only"
+    },
+    {
+      "id": "china-macao",
+      "name": "Macao",
+      "sourceIsoA3": ["MAC"],
       "action": "union_to_country",
       "targetIsoA3": "CHN",
       "hoverIsoA3": "CHN",
@@ -57,6 +76,7 @@
     {
       "id": "kosovo",
       "name": "Kosovo",
+      "sourceIsoA3": ["KOS", "XKS"],
       "action": "render_as_disputed_with_parent",
       "parentIsoA3": "SRB",
       "hoverIsoA3": "SRB",

docker-compose.yml

@@ -1,5 +1,3 @@
-version: '3.8'
-
 services:
   aiprovider:
     image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
@@ -32,7 +30,7 @@ services:
       POSTGRES_PASSWORD: postgres
       POSTGRES_DB: planet_db
     ports:
-      - "5432:5432"
+      - "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
     volumes:
       - postgres_data:/var/lib/postgresql/data
     healthcheck:
@@ -45,7 +43,7 @@ services:
     image: redis:7-alpine
     container_name: planet_redis
     ports:
-      - "6379:6379"
+      - "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
     volumes:
       - redis_data:/data
     healthcheck: