Compare commits

..

32 Commits
v0.74.6 ... dev

Author SHA1 Message Date
linkong
1d153c2667 release: bump version to 0.76.20
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (push) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-11-10 17:52:24 +08:00
linkong
c1d651589a release: bump version to 0.76.19
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-11-09 15:12:50 +08:00
rayd1o
36be9e3da6 release: bump version to 0.76.18
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-10-02 05:46:33 +08:00
rayd1o
7d8822d34d release: bump version to 0.76.17
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:38:52 +08:00
rayd1o
ae1a4b3942 release: bump version to 0.76.16
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:22:11 +08:00
rayd1o
35e478642b release: bump version to 0.76.15
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:11:41 +08:00
rayd1o
33734b08b8 release: bump version to 0.76.14
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:56:14 +08:00
rayd1o
bbe47c5759 release: bump version to 0.76.13
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:43:55 +08:00
rayd1o
361bf258e4 release: bump version to 0.76.12
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:33:36 +08:00
rayd1o
83c552f9b2 release: bump version to 0.76.11
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:21:20 +08:00
rayd1o
15fed1e121 release: bump version to 0.76.10
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:10:26 +08:00
rayd1o
f4514b588e release: bump version to 0.76.9
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:56:05 +08:00
rayd1o
9523f2bb83 release: bump version to 0.76.8
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:41:46 +08:00
rayd1o
9dc3b61fa5 release: bump version to 0.76.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:09:28 +08:00
rayd1o
698863da68 release: bump version to 0.76.6
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 00:33:55 +08:00
rayd1o
4c0b49962c release: bump version to 0.76.5
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 00:17:38 +08:00
rayd1o
ffb3134d93 docs: document breaking news state machine
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-10-01 21:46:59 +08:00
rayd1o
e8742b5d3c release: bump version to 0.76.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 21:11:00 +08:00
rayd1o
28a7132eec release: bump version to 0.76.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 17:48:45 +08:00
rayd1o
dd41588527 release: bump version to 0.76.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 02:29:49 +08:00
rayd1o
45604a1ebb release: bump version to 0.76.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 01:25:24 +08:00
rayd1o
06c122bade fix: use whole-anchor occlusion for BGP icons
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:59:21 +08:00
rayd1o
96e9b15fda fix: keep satellite selection glow whole at limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:21:22 +08:00
rayd1o
a8713e58fd fix: preserve BGP event halo at globe limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 18:23:49 +08:00
rayd1o
05377bf5c6 release: bump version to 0.76.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-30 17:49:29 +08:00
rayd1o
67548d8278 release: bump version to 0.75.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-29 15:40:32 +08:00
rayd1o
4b8feeb87e release: bump version to 0.75.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-27 09:30:06 +08:00
rayd1o
a2025dbfe0 release: bump version to 0.75.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-25 02:03:53 +08:00
rayd1o
003b316d42 release: bump version to 0.75.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-25 00:16:30 +08:00
rayd1o
0c40764c58 feat: configure compute comparisons and restore adaptive globe dragging
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-24 22:59:55 +08:00
rayd1o
5bfc790dec release: bump version to 0.75.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-23 21:20:12 +08:00
linkong
60eefb19c9 release: bump version to 0.74.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-17 17:30:54 +08:00
478 changed files with 35055 additions and 7952 deletions

View File

@@ -19,9 +19,9 @@ Do not refactor architecture, add features, or change behavior.
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
Only report and fix issues present in **newly added or modified** lines of this diff — do not audit unchanged code, and do not rewrite unrelated files during cleanup.
## Token-Saving Rule
## Token-Saving Rule & Harness Checks
Prefer deterministic CLI checks before reading files into model context:
@@ -29,27 +29,36 @@ Prefer deterministic CLI checks before reading files into model context:
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
Use the repository's focused harness checks for the touched surfaces:
- Backend Python changes: `scripts/harness/backend-rules-check.sh` (catches direct `print()`, `breakpoint()`, and debugger calls)
- Frontend changes: `scripts/harness/frontend-rules-check.sh` (catches console logs, missing button types, Card nesting, layout height-chain issues)
- Secrets / credentials: `scripts/harness/security-check.sh`
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
## Checklist
### 1. Duplicate Logic
- Identical or near-identical code blocks appearing in multiple places
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
- Repeated DOM queries, regex literals, or template strings within the same file
- Identical or near-identical code blocks appearing in multiple places within the modified scope
- A helper that already exists but is re-implemented elsewhere instead of being reused
- Repeated complex queries or template strings that cause actual maintenance hazard
- **Anti-Over-Defensiveness**: Keep one-off logic at its call site. Do not prematurely abstract single-use or 2-use trivial helpers/templates that are clearer inline.
### 2. Magic Numbers / Magic Strings
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) where the intent is completely opaque and naming adds clear value
- Hardcoded domain strings (IDs, status values, protocol fragments) scattered through logic
- **Anti-Over-Defensiveness (AGENTS.md Quality Bar)**:
- Idiomatically clear values such as `0`, `1`, `-1`, `100`, `""` are fine inline.
- **Do NOT over-extract local single-use layout offsets, Three.js coordinates, padding, or math coefficients into module-level globals.**
- Keep one-off logic at its call site unless extraction clarifies a complex operation or lets callers share the same behavior.
### 3. Naming Issues
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
- Names that do not match actual behavior
- The same concept referred to by different names in different places
- The same concept referred to by different names in different places within the changed scope
### 4. Dead Code
- Commented-out code blocks (3+ lines)
@@ -58,12 +67,12 @@ Read full files only when the focused diff does not provide enough surrounding c
### 5. Style Inconsistencies
- Trailing whitespace
- Mixed quote styles or indentation within the same file
- Mixed quote styles or indentation within the same file (Python: 4 spaces Black style; TypeScript: 2 spaces Prettier style)
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
### 6. Other
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
- Overly verbose conditions that can be simplified without changing logic
- Keep fixes strictly scoped to the modified files; do not rewrite external callers or expand cleanup beyond the changed scope.
## Steps
@@ -107,30 +116,39 @@ If no issues are found, output "No code smells detected. Code quality looks good
Use the Edit tool for **minimal, targeted changes**:
- **Duplicate logic**: extract to a shared constant or function; update all call sites
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
- **Naming**: rename the variable/function; update all references
- **Duplicate logic**: extract to a shared constant or helper within the owned file or component scope; update call sites strictly within scope
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope only when the value is reused across places or its meaning is genuinely opaque; replace usages. Keep one-off UI/math offsets inline per `AGENTS.md`
- **Naming**: rename the variable/function; update all references within the modified scope
- **Dead code**: delete it
- **Trailing whitespace / style**: fix in place
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
- **Unexported helper**: add `export` only when requested; do not modify unrelated callers
Principles:
- Only fix issues identified in the checklist — no extra improvements
- Only fix issues identified in the checklist — no extra improvements or speculative refactorings
- Keep each Edit as small as possible
- After fixing, verify the old bad pattern is gone with grep
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
- After fixing, verify the old bad pattern is gone with grep or deterministic harness checks
- Prefer `apply_patch` or targeted edit tools; use formatters only when the repository already uses them for the touched file type
### Step 5 — Summary
### Step 5 — Verify and summarize
Run the smallest relevant harness checks for the modified surfaces:
- Python files: `uv run python -m py_compile <changed_py_files>` and `scripts/harness/backend-rules-check.sh`
- Frontend files: `scripts/harness/frontend-rules-check.sh` (and `bun run --cwd frontend build` if imports/types were adjusted)
- Formatting / whitespace: `git diff --check`
```
Cleanup complete:
Fixed N issues:
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
✓ main.js — clarified variable naming in news event payload handler
✓ controls.js — removed unused dead import
...
Validation:
✓ git diff --check passed
✓ scripts/harness/frontend-rules-check.sh passed
Skipped (needs manual review):
! foo.js L45 — large commented-out block; confirm it is safe to delete
```
@@ -141,4 +159,6 @@ Skipped (needs manual review):
- **Do not** add new features, abstractions, or parameters
- **Do not** rewrite comments (only delete commented-out dead code)
- **Do not** touch test file logic
- **Do not** rewrite unrelated files during cleanup (per `AGENTS.md` Prohibited)
- **Do not** over-extract local single-use coordinates, layout numbers, or inline math coefficients into global constants (preventing defensive over-engineering)
- If a magic number's intent is uncertain, skip it and flag it in the summary

View File

@@ -48,18 +48,22 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
3. Write the doc:
- Explain background/problem, design decisions, constraints, and operational impact.
- Include the context, rationale, or operational impact the reader needs for this change.
- Keep code snippets short and directly relevant.
- List related files only when they help future maintainers navigate.
- Use the repository’s existing language, heading style, and naming conventions.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. Only update the glossary when introducing new, ambiguous, or cross-language public concepts; avoid unnecessary glossary churn for ordinary bug fixes or internal refactors.
4. Verify:
- Read the completed doc once for clarity and stale statements.
- Verify important referenced paths exist with `test -e` or `rg --files`.
- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
- Run the repository's centralized documentation harness gate:
```bash
scripts/harness/docs-consistency-check.sh
```
This automatically verifies bilingual pairs, public registry whitelist, link anchors, language-scoped links, section deep-links, and stale admin route terms in a single pass.
- Run any additional domain-specific checks from `docs/documentation-coverage-rules.md` when applicable.
## Hard Constraints

View File

@@ -19,9 +19,11 @@ Do not use this skill for ordinary commits that are not being released.
## Versioning Rules
- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
- `bugfix` -> bump `+0.0.1`
- `improvement` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
- `bugfix` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
Mixed bugfix and small feature/UI work defaults to `improvement` unless the user explicitly chooses another release type.
When intent is mixed, prefer the user's stated release intent.
## Required Files
@@ -53,28 +55,32 @@ Do not inspect full diffs unless deciding whether changed code belongs in the re
### Step 1 — Environment check
```bash
git branch --show-current # must be on dev
git branch --show-current # check current branch
git status --short # check for unrelated uncommitted changes
cat VERSION # read current version
```
If not on `dev`, stop and tell the user. Do not proceed.
Default release branch is `dev`. If not on `dev` (e.g. on a dedicated maintenance or release branch), report the active branch and confirm before proceeding, rather than hard-failing.
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
### Step 2 — Determine release type and next version
- If the user provided an explicit type (`feature` / `bugfix`), use it
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
- If the user provided an explicit type (`feature` / `improvement` / `bugfix`), use it
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`:
- New features or major capabilities -> `feature`
- Mixed bugfix and small feature/UI work -> default to `improvement` (per `rules.md`)
- Bugfixes only -> `bugfix`
- Compute the next version:
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
- `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
- `improvement`: increment patch only (e.g. `0.76.18` → `0.76.19`)
- `bugfix`: increment patch only (e.g. `0.76.18` → `0.76.19`)
- **Show the release plan before making any changes:**
```
Release plan:
Type: bugfix
Version: 0.26.2 → 0.26.3
Type: improvement
Version: 0.76.18 → 0.76.19
Branch: dev
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
@@ -117,17 +123,25 @@ Get today's date with `date +%Y-%m-%d`.
### Step 6 — Validate
Run the smallest relevant validation for the changes in scope:
Run the smallest relevant harness validation for the changes in scope:
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
- **Version consistency**: confirm `VERSION`, `frontend/package.json`, `pyproject.toml`, and `uv.lock` all show the identical target version:
```bash
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
- **Security scan**: run `scripts/harness/security-check.sh`
- **Python files changed**:
- Compile: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `uv run python -m py_compile <changed_files>`
- Service rules: run `scripts/harness/backend-rules-check.sh`
- **Frontend files changed**:
- Rules: run `scripts/harness/frontend-rules-check.sh`
- Build: run `bun run --cwd frontend build` (Bun only; never skip)
- **Docs changed**: run `scripts/harness/docs-consistency-check.sh`
- **Broad / cross-subsystem changes**: run `scripts/harness/quick-check.sh`
### Step 7 — Pre-commit preview
Show what will be committed:
@@ -141,29 +155,36 @@ Confirm all required files are present and no unexpected files (debug files, `.e
### Step 8 — Commit, tag, and push
```bash
branch=$(git branch --show-current)
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# also stage any code changes included in this release
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
Commit message format is fixed: `release: bump version to x.x.x`
Push to remote:
- When remote publication is requested:
```bash
git push origin "$branch"
git push origin vx.x.x
```
- If remote publication was not requested or remote access is restricted, report the commit and tag as complete and output the push commands for manual execution.
### Step 9 — Completion summary
```
✓ Version bumped: 0.26.2 → 0.26.3
✓ Version bumped: 0.76.18 → 0.76.19
✓ CHANGELOG updated
✓ version-history updated
✓ uv.lock regenerated
✓ Validation passed
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ Pushed to origin/dev
✓ commit: release: bump version to 0.76.19
✓ tag: v0.76.19
✓ Pushed to origin/dev (or reported pending manual push)
```
## Notes

View File

@@ -3,8 +3,23 @@
!pyproject.toml
!uv.lock
!VERSION
!frontend/
!frontend/public/
!frontend/public/earth/
!frontend/public/earth/assets/
!frontend/public/earth/assets/cards/
!frontend/public/earth/assets/cards/manifest.json
!backend/
!backend/**
!config/
!config/earth-boundary-pov-policy.china-v1.json
!scripts/
!scripts/build_earth_boundary_china_pov_geojson.py
!scripts/build_earth_boundary_pmtiles.py
!scripts/earth_boundary_policy.py
!resources/
!resources/earth/
!resources/earth/boundary-pov-geometry.china-v1.json
!aiprovider/
!aiprovider/**

5
.gitignore vendored
View File

@@ -160,3 +160,8 @@ data/earth-boundary-sources/
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
# committing thousands of generated tile files.
frontend/public/earth/data/boundaries/
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
resources/earth/candidates/nasa-blue-marble/*.jpg
resources/earth/candidates/natural-earth-3/*.jpg
*:Zone.Identifier

View File

@@ -55,13 +55,19 @@ Preserve user changes already present in the worktree.
### Validation
Fast local harness validation:
For an isolated change, run the smallest check that exercises the affected
behavior, along with any gates required for that area. For example, run the
docs consistency check for a link change, or reproduce a bug at the entry
point where it occurred.
Use the quick suite when a change affects shared harness wiring or needs
regression checks across areas:
```bash
scripts/harness/quick-check.sh
```
Full local validation:
Use full validation for larger cross-subsystem changes:
```bash
scripts/harness/validate.sh
@@ -113,6 +119,9 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
## Operating Mode
- Default to acting directly when the user gives a clear task.
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
and Astra for complex reasoning; the main agent owns final verification.
- Ask before acting only when the missing decision is risky, cannot be
discovered from repository context, and no conservative assumption is safe.
- Read relevant files before editing.
@@ -120,6 +129,8 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
focused file reads, tests, builds, linters, and harness scripts.
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
release work, and documentation unless the task requires it.
- When the user corrects a detail, apply it to the original request. Carry the
correction into the result without turning discarded ideas into new requirements.
---
@@ -141,6 +152,12 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
visual-dependent part of the work.
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
diff, direct file check, or harness result.
- Verify the outcome through the entry point the user relies on. A source check
can confirm a code convention. A UI test with a mocked API can confirm what
the user sees for that response.
- When a test fails, identify the behavior or invariant it protects. Fix the
implementation if that regressed. If the behavior is correct and the assertion
pins an obsolete detail, update the test to exercise the behavior.
- For UI and rendering changes, verify the rendered result when local tooling
allows it.
@@ -159,7 +176,14 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
## Quality Bar
- Prefer existing project patterns over new abstractions.
- Prefer existing project patterns and the smallest change that satisfies the
request. Keep one-off logic at its call site unless extraction clarifies a
complex operation or lets callers share the same behavior.
- Validate and normalize external data at the boundary that owns it. Recheck a
guarantee when data crosses a new trust boundary or an operation can invalidate it.
- Before adding a retry, lock, or fallback, identify the failure it addresses
from a requirement, an existing contract, or observed behavior. Catch errors
where the caller can recover, translate them, or add context and propagate them.
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
a stable path exists.
- Centralize prompts, constants, defaults, and shared request/response handling.

View File

@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
## Runtime Entry Points

View File

@@ -1 +1 @@
0.74.6
0.76.20

View File

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
COPY VERSION /app/VERSION
EXPOSE 8000

View File

@@ -1,3 +1,4 @@
from app.api.v1 import card_images
from fastapi import APIRouter
from app.api.v1 import (
ai,
@@ -12,6 +13,7 @@ from app.api.v1 import (
alerts,
settings,
collected_data,
compute_intelligence,
data_products,
layers,
visualization,
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])

View File

@@ -0,0 +1,108 @@
"""Card-image configuration, upload, catalog selection and public resolution."""
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
from fastapi.responses import FileResponse
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.concurrency import run_in_threadpool
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
from app.schemas.card_images import (
CardImageConfiguration,
CardImagePreview,
CardImagePublish,
EntityType,
MatchScope,
)
from app.services.card_image_configuration import (
publish_configuration,
read_configuration,
resolve_image,
)
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
from app.services.card_image_targets import get_target, search_targets
router = APIRouter()
async def image_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
):
response.headers["Cache-Control"] = "no-store"
return await read_configuration(db)
@router.put("/configuration")
async def save_configuration(
payload: CardImagePublish,
actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return await publish_configuration(db, payload.revision, payload.config, actor)
@router.post("/upload")
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
content = await file.read(MAX_UPLOAD_BYTES + 1)
await file.close()
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
@router.get("/files/{filename}")
async def image_file(filename: str):
path = uploaded_path(filename)
if not path.is_file():
raise HTTPException(404, "Image not found")
return FileResponse(
path,
headers={
"Cache-Control": "public, max-age=31536000, immutable",
"X-Content-Type-Options": "nosniff",
},
)
@router.get("/targets")
async def targets(
entity_type: EntityType,
scope: MatchScope = "object",
q: str = Query(default="", max_length=100),
limit: int = Query(default=25, ge=1, le=50),
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
@router.get("/resolve")
async def resolve(
response: Response,
entity_type: EntityType,
entity_id: str = Query(min_length=1, max_length=300),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
envelope = await read_configuration(db)
config = CardImageConfiguration.model_validate(envelope["config"])
context = await get_target(db, entity_type, entity_id)
result = resolve_image(config, context)
return {**result, "revision": envelope["revision"], "entity": context}
@router.post("/preview")
async def preview(
payload: CardImagePreview,
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
context = await get_target(db, payload.entity_type, payload.entity_id)
return {**resolve_image(payload.config, context), "entity": context}

View File

@@ -0,0 +1,235 @@
"""Compute observations, administrator configuration and independent facility samples."""
from datetime import UTC, datetime
from typing import Literal
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.core.earth_geography_policy import display_country
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.user import User
from app.schemas.compute_configuration import (
ComputeConfiguration,
ComputeConfigurationReset,
ComputeConfigurationWrite,
)
from app.services.compute_catalog import (
PARTICIPANTS,
participant_members,
source_country_names,
source_country_participants,
)
from app.services.compute_center_locations import refresh_compute_center_location_cache
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
configuration_response,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
router = APIRouter()
def selected_countries(countries: str) -> list[str]:
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
members = participant_members(codes)
flattened = [country for participant in members.values() for country in participant]
if len(flattened) != len(set(flattened)):
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
return codes
@router.get("/series")
async def get_series(
response: Response,
countries: str = "CN,US",
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
envelope = await read_configuration(db)
response.headers["Cache-Control"] = "no-store"
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
result.update(
version=envelope["revision"],
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
)
return result
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response,
_actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
return configuration_response(await read_configuration(db))
@router.put("/configuration")
async def update_configuration(
payload: ComputeConfigurationWrite,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=payload.config, actor=actor,
)
return configuration_response(envelope)
@router.post("/configuration/reset")
async def reset_configuration(
payload: ComputeConfigurationReset,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
)
return configuration_response(envelope)
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
now = datetime.now(UTC)
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
ranked = (
select(
DataSnapshot.id,
func.row_number()
.over(
partition_by=DataSnapshot.source,
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
)
.label("rank"),
)
.where(
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
DataSnapshot.status == "success",
DataSnapshot.completed_at < cutoff,
)
.subquery()
)
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
# Old collector rows have ambiguous units/identity and must not become
# national evidence merely because a new UI can display them.
query = select(CollectedData).where(
compute_record_filter(CollectedData),
CollectedData.deleted_at.is_(None),
CollectedData.snapshot_id.in_(snapshots),
CollectedData.is_valid == 1,
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
or_(
infrastructure,
CollectedData.source == "top500",
and_(
CollectedData.extra_data["status"].as_string() == "Existing",
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
),
),
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
)
return query
@router.get("/facilities")
async def get_facilities(
countries: str = "CN,US",
year: int | None = Query(None, ge=2000, le=2100),
limit: int = Query(1000, ge=1, le=2000),
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
query = await sample_query(db, codes, year)
country = CollectedData.extra_data["country"].as_string()
source_participants = source_country_participants(codes)
# Canonical source subdivisions retain their raw label on each record but
# share one owner in the comparison, using the same policy as the globe.
owner_cases = [
(country == name, display_country(name))
for name in source_country_names(codes)
if display_country(name) != name
]
display = case(*owner_cases, else_=country) if owner_cases else country
participant_cases = [(country == name, code) for name, code in source_participants.items()]
participant = case(*participant_cases, else_=None)
counts = (
(
await db.execute(
query.with_only_columns(
display.label("country"),
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(display, participant, CollectedData.source)
)
)
.mappings()
.all()
)
participant_counts = (
(
await db.execute(
query.with_only_columns(
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(participant, CollectedData.source)
)
)
.mappings()
.all()
)
records = (
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
.scalars()
.all()
)
if year:
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
else:
await refresh_compute_center_location_cache(db)
payload = convert_compute_centers_to_geojson(records)
for feature in payload["features"]:
feature["properties"]["historical_snapshot"] = year is not None
properties = feature["properties"]
properties["source_country"] = properties.get("country")
properties["country"] = display_country(properties.get("country"))
total = sum(row["count"] for row in counts)
payload.update(
{
"counts": [dict(row) for row in counts],
"participant_counts": [dict(row) for row in participant_counts],
"participants": [PARTICIPANTS[code] for code in codes],
"total_count": total,
"returned_count": len(records),
"truncated": total > limit,
"reference_year": year,
"scope": "public_source_records",
"history_basis": (
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
),
"coverage": "available" if total else "no_verified_snapshot",
"cross_source_deduplicated": False,
}
)
return payload

View File

@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
"name": "算力",
"sources": ["top500", "epoch_ai_gpu"],
"primary_stat_key": "compute_center_count",
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
},
}

View File

@@ -13,6 +13,7 @@ from sqlalchemy import delete, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings as app_settings
from app.core.logging import get_logger
from app.core.security import decode_token, get_current_user, redis_client
from app.db.session import get_db
from app.models.collected_data import CollectedData
@@ -51,6 +52,7 @@ from app.services.earth_boundaries import (
router = APIRouter()
logger = get_logger(__name__)
optional_bearer = HTTPBearer(auto_error=False)
REPO_ROOT = Path(__file__).resolve().parents[4]
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
@@ -382,13 +384,28 @@ async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
return await get_earth_news_sources_payload(db)
def _schedule_earth_news_source_refresh() -> None:
try:
from app.services.scheduler import schedule_earth_news_refresh_for_all_regions
schedule_earth_news_refresh_for_all_regions(force=True)
except Exception as exc:
logger.warning_event(
"Failed to schedule Earth news refresh after source configuration change",
event="earth_news.rss.config_refresh_schedule_failed",
context={"error": str(exc)},
)
@router.put("/news-sources")
async def update_earth_news_sources(
payload: EarthNewsSourcesPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await save_earth_news_sources_payload(db, payload.model_dump())
result = await save_earth_news_sources_payload(db, payload.model_dump())
_schedule_earth_news_source_refresh()
return result
@router.delete("/news-sources")
@@ -397,7 +414,9 @@ async def reset_earth_news_sources(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await reset_earth_news_sources_payload(db)
result = await reset_earth_news_sources_payload(db)
_schedule_earth_news_source_refresh()
return result
@router.post("/news-sources/test")

View File

@@ -1,17 +1,26 @@
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, status
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from uuid import UUID
from app.db.session import get_db
from app.services.earth_news import (
ALLOWED_NEWS_CATEGORY_KEYS,
SUPPORTED_NEWS_LOCALES,
REGION_ANCHORS,
determine_focus_region,
get_earth_news_payload,
)
router = APIRouter()
class EarthNewsRefreshRequest(BaseModel):
lat: float | None = Field(default=None, ge=-90, le=90)
lon: float | None = Field(default=None, ge=-180, le=180)
region: str | None = Field(default=None, max_length=64)
def _parse_categories(raw: str | None) -> set[str] | None:
if raw is None or not raw.strip():
return None
@@ -39,7 +48,9 @@ def _parse_limit(raw: int | None) -> int:
if raw is None:
return 12
if raw < 1:
raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
raise HTTPException(
status_code=422, detail={"message": "News limit must be greater than 0."}
)
return min(raw, 100)
@@ -59,18 +70,76 @@ def _parse_locale(raw: str | None) -> str:
return requested
@router.post("/refresh", status_code=status.HTTP_202_ACCEPTED)
async def request_earth_news_refresh(payload: EarthNewsRefreshRequest):
requested_region = payload.region.strip().lower() if payload.region else None
if requested_region and requested_region not in REGION_ANCHORS:
raise HTTPException(
status_code=422,
detail={
"message": "Unsupported news region.",
"invalid_region": requested_region,
"allowed_regions": list(REGION_ANCHORS.keys()),
},
)
region = requested_region or determine_focus_region(payload.lat, payload.lon)
from app.services.scheduler import request_earth_news_region_refresh
try:
return request_earth_news_region_refresh(region)
except Exception as exc:
raise HTTPException(
status_code=503,
detail={
"code": "refresh_failed",
"message": "Could not start a news refresh. Please try again.",
},
) from exc
@router.get("/refresh/{refresh_id}")
async def get_earth_news_refresh(refresh_id: UUID):
from app.services.earth_news_refresh import get_earth_news_refresh_status
try:
result = get_earth_news_refresh_status(refresh_id.hex)
except Exception as exc:
raise HTTPException(
status_code=503,
detail={
"code": "refresh_failed",
"message": "Refresh status is temporarily unavailable. Please retry.",
},
) from exc
if result is not None:
return result
return {
"refresh_id": refresh_id.hex,
"region": None,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
@router.get("/earth-feed")
async def get_earth_feed(
lat: float | None = Query(None, description="Current Earth view center latitude"),
lon: float | None = Query(None, description="Current Earth view center longitude"),
region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
region: str | None = Query(
None, description="Explicit Earth news region for UE/client integrations"
),
categories: str | None = Query(None, description="Comma-separated news category keys"),
sources: str | None = Query(None, description="Comma-separated news source ids"),
limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
db: AsyncSession = Depends(get_db),
):
normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
normalized_region = (
region.strip().lower() if isinstance(region, str) and region.strip() else None
)
if normalized_region is not None and normalized_region not in REGION_ANCHORS:
raise HTTPException(
status_code=422,

View File

@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import select, func
from typing import List, Dict, Any, Optional
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
from app.core.collected_data_fields import get_record_field
from app.core.enums import BGPStatus
from app.core.satellite_tle import build_tle_lines_from_elements
from app.core.time import to_iso8601_utc
from app.db.session import get_db
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
@@ -107,6 +109,7 @@ SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
max_features=25000,
max_bytes=32 * BYTES_PER_MIB,
)
SATELLITE_QUERY_BATCH_SIZE = 500
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
COMPUTE_CENTER_CACHE_STALE_SECONDS,
@@ -337,22 +340,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
return {"type": "FeatureCollection", "features": features}
def _normalize_satellite_constellation_group(
raw_group: Any,
name: Optional[str],
) -> Optional[str]:
normalized_group = str(raw_group or "").strip().lower()
if normalized_group:
return normalized_group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
if constellation_group == "starlink":
@@ -505,16 +492,19 @@ async def _count_current_or_latest_task_data(
async def _load_current_collected_data_by_sources(
db: AsyncSession,
sources: List[str],
*, include_compute_types: bool = False,
) -> Dict[str, List[CollectedData]]:
if not sources:
return {}
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(sources))
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
)
if include_compute_types:
stmt = stmt.where(CollectedData.deleted_at.is_(None))
result = await db.execute(stmt)
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
@@ -644,7 +634,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
return "unknown"
unit = str(capacity_unit or "").strip().lower()
if unit in {"pflop/s", "pflops", "pflop"}:
if unit in {"flop/s", "flops"}:
normalized_tflops = capacity_value / 1e12
elif unit in {"pflop/s", "pflops", "pflop"}:
normalized_tflops = capacity_value * 1000
elif unit in {"gflop/s", "gflops", "gflop"}:
normalized_tflops = capacity_value
@@ -672,15 +664,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
"""
features: List[Dict[str, Any]] = []
unresolved: List[Dict[str, Any]] = []
withheld_count = 0
for record in records:
metadata = record.extra_data or {}
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
withheld_count += 1
continue
result = resolve_compute_center_location_full(record, metadata)
site_type = (
"supercomputer"
if record.source == "top500" or record.data_type == "supercomputer"
else "gpu_cluster"
)
site_type = compute_site_type(record)
if not result.is_resolved:
diagnostic = result.diagnostic or ResolutionDiagnostic(
@@ -706,7 +698,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
latitude = location.latitude
longitude = location.longitude
if site_type == "supercomputer":
if metadata.get("compute_contract") == "v1":
capacity_value = _parse_float(get_record_field(record, "value"))
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
elif site_type == "supercomputer":
capacity_value = _parse_float(get_record_field(record, "rmax"))
capacity_unit = "GFlops"
else:
@@ -767,7 +762,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
}
)
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
VESSEL_TYPE_FILTERS = {
@@ -1866,21 +1861,39 @@ async def _build_satellites_geojson(
limit: int | None,
db: AsyncSession,
) -> dict[str, Any]:
records = await _load_current_or_latest_task_data(
db,
"celestrak_tle",
exclude_unknown_name=True,
limit=limit,
)
features: list[dict[str, Any]] = []
source = "celestrak_tle"
for current_only in (True, False):
if current_only:
stmt = _current_collected_data_stmt(source)
else:
task_id = await _latest_task_id_for_source(db, source, exclude_unknown_name=True)
if task_id is None:
break
stmt = (
select(CollectedData)
.where(CollectedData.source == source, CollectedData.task_id == task_id)
.order_by(CollectedData.id.desc())
)
stmt = stmt.where(CollectedData.name != "Unknown")
if limit is not None:
stmt = stmt.limit(limit)
if not records:
return {"type": "FeatureCollection", "features": [], "count": 0}
# Release each ORM batch while retaining the complete, ordered display payload.
records = await db.stream_scalars(
stmt, execution_options={"yield_per": SATELLITE_QUERY_BATCH_SIZE}
)
found_records = False
try:
async for batch in records.partitions(SATELLITE_QUERY_BATCH_SIZE):
found_records = True
features.extend(convert_satellite_to_geojson(batch)["features"])
finally:
await records.close()
if found_records:
break
geojson = convert_satellite_to_geojson(list(records))
return {
**geojson,
"count": len(geojson.get("features", [])),
}
return {"type": "FeatureCollection", "features": features, "count": len(features)}
@router.get("/geo/supercomputers")
@@ -1955,9 +1968,10 @@ async def _build_compute_centers_geojson(
records_by_source = await _load_current_collected_data_by_sources(
db,
["top500", "epoch_ai_gpu"],
include_compute_types=True,
)
records = _filter_known_records(
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
[record for group in records_by_source.values() for record in group],
)
if limit is not None:
records = records[:limit]
@@ -1972,6 +1986,7 @@ async def _build_compute_centers_geojson(
"total": 0,
"supercomputers": 0,
"gpu_clusters": 0,
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
"unresolved": 0,
},
}
@@ -2014,6 +2029,7 @@ async def _build_compute_centers_geojson(
"count": len(sanitized_features),
"stats": {
"total": len(sanitized_features),
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
"supercomputers": sum(
1 for feature in sanitized_features
if feature.get("properties", {}).get("site_type") == "supercomputer"
@@ -2330,7 +2346,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
stmt = (
select(CollectedData)
.where(CollectedData.source_id == source_id)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.deleted_at.is_(None))
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
.limit(1)
)
@@ -2729,6 +2746,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
)
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
kind = compute_site_type_expression(CollectedData)
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
CollectedData.deleted_at.is_(None))
.group_by(CollectedData.source, kind))).all()
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
present_sources = set()
for source, site_type, count in rows:
present_sources.add(source)
counts[site_type] += int(count or 0)
# Retain the existing last-task fallback for the two collected catalogs.
for source in COMPUTE_SOURCES:
if source not in present_sources:
key = "supercomputer" if source == "top500" else "gpu_cluster"
counts[key] += await _count_current_or_latest_task_data(db, source)
return counts
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
@@ -2737,9 +2773,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"celestrak_tle",
exclude_unknown_name=True,
)
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
compute_center_count = supercomputer_count + gpu_cluster_count
compute_counts = await _compute_facility_counts(db)
supercomputer_count = compute_counts["supercomputer"]
gpu_cluster_count = compute_counts["gpu_cluster"]
compute_center_count = sum(compute_counts.values())
active_incident_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
@@ -2794,6 +2831,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
"supercomputer_count": supercomputer_count,
"gpu_cluster_count": gpu_cluster_count,
"data_center_count": compute_counts["data_center"],
"cloud_center_count": compute_counts["cloud_center"],
"edge_node_count": compute_counts["edge_node"],
"bgp_event_count": active_incident_count or active_anomaly_count,
"bgp_incident_count": active_incident_count,
"bgp_anomaly_count": active_anomaly_count,

View File

@@ -0,0 +1,35 @@
"""Compute facility classification shared by the map and image resolver."""
from sqlalchemy import case, or_
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
def compute_site_type(record) -> str:
metadata = record.extra_data or {}
for value in (metadata.get("site_type"), record.data_type):
if value in COMPUTE_SITE_TYPES:
return value
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
record.source, "data_center"
)
def compute_site_type_expression(record):
declared = record.extra_data["site_type"].as_string()
return case(
(declared.in_(COMPUTE_SITE_TYPES), declared),
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
(record.source == "top500", "supercomputer"),
(record.source == "epoch_ai_gpu", "gpu_cluster"),
else_="data_center",
)
def compute_record_filter(record):
return or_(
record.source.in_(COMPUTE_SOURCES),
record.data_type.in_(COMPUTE_SITE_TYPES),
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
)

View File

@@ -0,0 +1,57 @@
"""Country display ownership for Earth comparisons and cards.
Keep source-country evidence separate; use this projection only for display
grouping so regional source records cannot become competing national keys.
"""
from __future__ import annotations
import json
from functools import lru_cache
from pathlib import Path
from app.core.countries import COUNTRY_ENTRIES, normalize_country
_DISPLAY_ISO_A3 = {
canonical: next(
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
None,
)
for canonical, aliases in COUNTRY_ENTRIES
}
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
@lru_cache(maxsize=1)
def _policy_country_by_code() -> dict[str, str]:
try:
content = _POLICY_PATH.read_text(encoding="utf-8")
except FileNotFoundError as error:
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
policy = json.loads(content)
if policy.get("profile") != "china-pov-v1":
raise ValueError("unsupported Earth display-policy profile")
mappings = {}
for rule in policy["rules"]:
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
if target not in _COUNTRY_BY_ISO_A3:
continue
for code in rule.get("sourceIsoA3", []):
mappings[code] = _COUNTRY_BY_ISO_A3[target]
return mappings
def display_country(value: str | None) -> str | None:
country = normalize_country(value)
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
owner = _policy_country_by_code().get(code)
if owner:
return owner
return country
def display_iso_a3(value: str | None) -> str | None:
country = display_country(value)
return _DISPLAY_ISO_A3.get(country) if country else None

View File

@@ -0,0 +1,13 @@
"""Shared source-group normalization; never infer hardware generations."""
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
group = str(raw_group or "").strip().lower()
if group:
return group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None

View File

@@ -6,7 +6,7 @@ import redis
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jose import JWTError, jwt
from sqlalchemy import text
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
return None
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
try:
parsed_id = int(user_id)
except (TypeError, ValueError):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
result = await db.execute(select(User).where(User.id == parsed_id))
user = result.scalar_one_or_none()
if user is None or not user.is_active:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
return user
async def get_current_user(
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
def blacklist_token(token: str) -> None:

View File

@@ -0,0 +1,51 @@
"""SQL objects supporting indexed BGP prefix geography lookup."""
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncConnection
BGP_PREFIX_GEO_FUNCTION_SQL = (
"""
CREATE OR REPLACE FUNCTION planet_bgp_try_inet(value text)
RETURNS inet
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
AS $$
BEGIN
RETURN value::inet;
EXCEPTION WHEN data_exception THEN
RETURN NULL;
END;
$$
""",
"""
CREATE OR REPLACE FUNCTION planet_bgp_try_masklen(value text)
RETURNS integer
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
AS $$
BEGIN
RETURN masklen(value::cidr);
EXCEPTION WHEN data_exception THEN
RETURN NULL;
END;
$$
""",
)
BGP_PREFIX_GEO_INDEX_NAME = "idx_bgp_prefix_geo_range_current"
BGP_PREFIX_GEO_INDEX_SQL = f"""
CREATE INDEX CONCURRENTLY IF NOT EXISTS {BGP_PREFIX_GEO_INDEX_NAME}
ON collected_data USING gist (
planet_bgp_try_inet(metadata->>'range_start') inet_ops,
planet_bgp_try_inet(metadata->>'range_end') inet_ops
)
WHERE source IN (
'opengeofeed_prefix_geo',
'iptoasn_prefix_geo',
'nro_delegated_prefix_geo'
)
AND COALESCE(is_current, TRUE) = TRUE
"""
async def install_bgp_prefix_geo_functions(connection: AsyncConnection) -> None:
for statement in BGP_PREFIX_GEO_FUNCTION_SQL:
await connection.execute(text(statement))

View File

@@ -6,6 +6,7 @@ from sqlalchemy.orm import declarative_base
from app.core.config import settings
from app.core.logging import get_logger
from app.db.bgp_prefix_geo import install_bgp_prefix_geo_functions
logger = get_logger(__name__)
@@ -219,6 +220,7 @@ async def init_db():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
await install_bgp_prefix_geo_functions(conn)
users_email_verified_existed = (
await conn.execute(
text(
@@ -299,8 +301,12 @@ async def init_db():
change_event_id BIGINT;
change_payload JSONB;
BEGIN
change_event_id := nextval(
pg_get_serial_sequence('earth_data_change_events', 'id')::regclass
);
change_payload := jsonb_build_object(
'event', 'earth.layer.changed',
'event_id', change_event_id,
'table', change_table,
'operation', change_operation,
'source', change_source,
@@ -311,6 +317,7 @@ async def init_db():
);
INSERT INTO earth_data_change_events (
id,
table_name,
operation,
source,
@@ -318,23 +325,15 @@ async def init_db():
payload,
occurred_at
) VALUES (
change_event_id,
change_table,
change_operation,
change_source,
NULL,
change_payload,
NOW()
)
RETURNING id INTO change_event_id;
change_payload := change_payload || jsonb_build_object(
'event_id', change_event_id
);
UPDATE earth_data_change_events
SET payload = change_payload
WHERE id = change_event_id;
PERFORM pg_notify(
'planet_earth_data_changes',
change_payload::text
@@ -440,6 +439,16 @@ async def init_db():
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
IF TG_TABLE_NAME = 'earth_news_items' AND NOT EXISTS (
SELECT 1
FROM new_rows AS current_row
FULL JOIN old_rows AS previous_row USING (id)
WHERE (to_jsonb(current_row) - 'last_seen_at' - 'updated_at')
IS DISTINCT FROM
(to_jsonb(previous_row) - 'last_seen_at' - 'updated_at')
) THEN
RETURN NULL;
END IF;
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (

View File

@@ -0,0 +1,128 @@
"""Typed, validated configuration for optional object-card images."""
from typing import Literal, Self
from urllib.parse import urlsplit
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
MatchScope = Literal["object", "model", "group", "category"]
def safe_url(value: str, *, local: bool = False) -> str:
if not value:
return value
if any(ord(char) < 32 for char in value) or "\\" in value:
raise ValueError("Invalid URL")
parts = urlsplit(value)
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
if (
".." not in parts.path
and "%" not in parts.path
and not parts.netloc
and not parts.query
and not parts.fragment
):
return value
if (
parts.scheme in {"http", "https"}
and parts.hostname
and not parts.username
and not parts.password
):
return value
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
class CardImageAsset(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
title: str = Field(min_length=1, max_length=200)
url: str = Field(min_length=1, max_length=2000)
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
width: int = Field(default=0, ge=0, le=20000)
height: int = Field(default=0, ge=0, le=20000)
credit: str = Field(default="", max_length=500)
source_url: str = Field(default="", max_length=2000)
license: str = Field(default="", max_length=300)
license_url: str = Field(default="", max_length=2000)
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
rights_status: Literal["available", "permission_pending"] = "available"
enabled: bool = True
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
@field_validator("url")
@classmethod
def validate_image_url(cls, value: str) -> str:
return safe_url(value, local=True)
@field_validator("source_url", "license_url")
@classmethod
def validate_source_url(cls, value: str) -> str:
return safe_url(value)
@field_validator("variants")
@classmethod
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
for width, url in value.items():
if width not in {"640", "1280"}:
raise ValueError("Unsupported image variant")
safe_url(url, local=True)
return value
class CardImageRule(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
entity_type: EntityType
scope: MatchScope
value: str = Field(min_length=1, max_length=300)
label: str = Field(default="", max_length=500)
asset_id: str | None = Field(default=None, max_length=100)
suppress: bool = False
enabled: bool = True
@model_validator(mode="after")
def validate_action(self) -> Self:
if self.scope != "object":
self.value = self.value.casefold()
if self.suppress and (self.scope != "object" or self.asset_id):
raise ValueError("Only an object rule may suppress its image, without an asset")
if not self.suppress and not self.asset_id:
raise ValueError("Choose an image or suppress this object")
return self
def target(self) -> tuple[str, str, str]:
return self.entity_type, self.scope, self.value
class CardImageConfiguration(BaseModel):
model_config = ConfigDict(extra="forbid")
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
@model_validator(mode="after")
def validate_relations(self) -> Self:
assets = {asset.id for asset in self.assets}
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
self.rules
):
raise ValueError("Image and rule IDs must be unique")
targets = [rule.target() for rule in self.rules]
if len(set(targets)) != len(targets):
raise ValueError("Each target may have only one rule")
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
raise ValueError("Remove image associations before deleting the image")
return self
class CardImagePublish(BaseModel):
revision: str = Field(max_length=200)
config: CardImageConfiguration
class CardImagePreview(BaseModel):
entity_type: EntityType
entity_id: str = Field(min_length=1, max_length=300)
config: CardImageConfiguration

View File

@@ -0,0 +1,94 @@
"""Validated, source-backed observations for the compute comparison cards."""
from datetime import date
from typing import Annotated, Literal, Self
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
Text = Annotated[str, Field(min_length=1, max_length=500)]
class ConfigurationModel(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
class ComputeSource(ConfigurationModel):
id: Identifier
title: Text
url: HttpUrl
publisher: Text
report_year: Year
published_at: date | None = None
locator: Text
@field_validator("url")
@classmethod
def public_reference(cls, value: HttpUrl) -> HttpUrl:
if value.username or value.password or len(str(value)) > 2000:
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
return value
class ComputeObservation(ConfigurationModel):
year: Year
metric: Literal["global_share", "absolute"]
source_id: Identifier
values: dict[str, Value] = Field(min_length=1, max_length=250)
@model_validator(mode="after")
def comparable_values(self) -> Self:
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
raise ValueError(
"Use countries from the canonical compute catalog, not regional totals"
)
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
raise ValueError("Global shares for one year must total at most 100 percent")
return self
class InnovationObservation(ConfigurationModel):
year: Year
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
source_id: Identifier
class ComputeConfiguration(ConfigurationModel):
reviewed_at: date
sources: list[ComputeSource] = Field(max_length=200)
observations: list[ComputeObservation] = Field(max_length=202)
innovation: list[InnovationObservation] = Field(max_length=101)
@model_validator(mode="after")
def consistent_evidence(self) -> Self:
ids = [source.id for source in self.sources]
if len(ids) != len(set(ids)):
raise ValueError("Report source IDs must be unique")
keys = [(row.metric, row.year) for row in self.observations]
if len(keys) != len(set(keys)):
raise ValueError("Each metric/year may have only one observation")
years = [row.year for row in self.innovation]
if len(years) != len(set(years)):
raise ValueError("Innovation observation years must be unique")
sources = {source.id: source for source in self.sources}
for row in [*self.observations, *self.innovation]:
source = sources.get(row.source_id)
if source is None:
raise ValueError(f"Missing report source: {row.source_id}")
if row.year > source.report_year:
raise ValueError("Observation year must not be after its report year")
if self.reviewed_at > date.today():
raise ValueError("Review date must not be in the future")
return self
class ComputeConfigurationWrite(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)
config: ComputeConfiguration
class ComputeConfigurationReset(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)

View File

@@ -36,6 +36,9 @@ class UserInDB(UserBase):
class UserResponse(UserBase):
# Bootstrap accounts have reserved .local email addresses. Validate email
# on writes, while allowing existing local identities to be serialized.
email: str
id: int
role: str
gatekeeper_groups: list[str] = Field(default_factory=list)

View File

@@ -111,15 +111,24 @@ async def _lookup_prefix_geography(
result = await db.execute(
text(
"""
WITH candidates AS MATERIALIZED (
SELECT source, id, metadata
FROM collected_data
WHERE source IN (
'opengeofeed_prefix_geo',
'iptoasn_prefix_geo',
'nro_delegated_prefix_geo'
)
AND COALESCE(is_current, TRUE) = TRUE
AND planet_bgp_try_inet(metadata->>'range_start') <= CAST(:range_start AS inet)
AND planet_bgp_try_inet(metadata->>'range_end') >= CAST(:range_end AS inet)
)
SELECT metadata
FROM collected_data
FROM candidates
WHERE source = :source
AND COALESCE(is_current, TRUE) = TRUE
AND metadata->>'family' = :family
AND CAST(metadata->>'range_start' AS inet) <= CAST(:range_start AS inet)
AND CAST(metadata->>'range_end' AS inet) >= CAST(:range_end AS inet)
ORDER BY
masklen(CAST(metadata->>'prefix' AS cidr)) DESC NULLS LAST,
planet_bgp_try_masklen(metadata->>'prefix') DESC NULLS LAST,
id DESC
LIMIT 1
"""
@@ -146,6 +155,10 @@ async def _lookup_prefix_geography(
results: dict[str, dict[str, Any]] = {}
# The GiST range predicate is selective, but PostgreSQL 15 greatly overestimates its cost.
# Avoid spending more time compiling JIT code than fetching the small candidate set.
await db.execute(text("SET LOCAL jit = off"))
for prefix in prefix_values:
try:
network = ipaddress.ip_network(prefix, strict=False)

View File

@@ -0,0 +1,174 @@
"""Revision-checked image settings; new bundled assets never replace user edits."""
from uuid import uuid4
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.card_images import CardImageConfiguration
from app.services.card_image_defaults import (
bundled_configuration,
previous_bundled_assets,
previous_bundled_rules,
)
CATEGORY = "earth_card_images"
def effective_envelope(payload: dict | None) -> dict:
defaults, version = bundled_configuration()
config = (
CardImageConfiguration.model_validate(payload["config"])
if payload
else defaults.model_copy(deep=True)
)
if payload:
previous = previous_bundled_assets()
latest = {asset.id: asset for asset in defaults.assets}
for index, asset in enumerate(config.assets):
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
# Replace the old bundled source while preserving fields the user
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
if old is not None and asset.url == old.url and asset.id in latest:
overrides = {
key: value
for key, value in asset.model_dump().items()
if value != getattr(old, key)
}
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
known_assets = set(payload.get("known_assets", []))
asset_ids = {asset.id for asset in config.assets}
config.assets.extend(
asset.model_copy(deep=True)
for asset in defaults.assets
if asset.id not in known_assets and asset.id not in asset_ids
)
prior_rules = previous_bundled_rules()
latest_rules = {rule.id: rule for rule in defaults.rules}
for index, rule in enumerate(config.rules):
old_rule = prior_rules.get(rule.id)
if (
old_rule
and rule.id in latest_rules
and rule.target() == old_rule.target()
and rule.asset_id == old_rule.asset_id
):
overrides = {
key: value
for key, value in rule.model_dump().items()
if value != getattr(old_rule, key)
}
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
known_rules = set(payload.get("known_rules", []))
rule_ids = {rule.id for rule in config.rules}
targets = {rule.target() for rule in config.rules}
available = {asset.id for asset in config.assets}
config.rules.extend(
rule.model_copy(deep=True)
for rule in defaults.rules
if rule.id not in known_rules
and rule.id not in rule_ids
and rule.target() not in targets
and rule.asset_id in available
)
return {
"config": config.model_dump(mode="json"),
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
"is_default": payload is None,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return effective_envelope(record.payload if record else None)
async def publish_configuration(
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
) -> dict:
defaults, _ = bundled_configuration()
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
if any(
asset.id in pending and asset.rights_status != "permission_pending"
for asset in config.assets
):
raise HTTPException(422, "Bundled image permission is still unverified")
initial = {
"config": defaults.model_dump(mode="json"),
"revision": "default",
"known_assets": [],
"known_rules": [],
}
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=initial)
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if effective_envelope(record.payload)["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
before = record.payload
record.payload = {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="card_images.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": record.payload},
)
)
await db.commit()
return effective_envelope(record.payload)
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
assets = {asset.id: asset for asset in config.assets}
for scope in ("object", "model", "group", "category"):
value = context.get("entity_id" if scope == "object" else scope)
values = [value] if value else []
if scope == "category" and value != "*":
values.append("*")
for value in values:
value = value if scope == "object" else value.casefold()
rule = next(
(
rule
for rule in config.rules
if rule.enabled
and rule.entity_type == context["entity_type"]
and rule.scope == scope
and rule.value == value
),
None,
)
if not rule:
continue
match = {"scope": scope, "value": value, "rule_id": rule.id}
if rule.suppress:
return {"image": None, "match": match, "suppressed": True}
asset = assets.get(rule.asset_id)
if asset and asset.enabled:
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
return {"image": None, "match": None, "suppressed": False}

View File

@@ -0,0 +1,108 @@
"""Read immutable bundled assets without inserting machine-specific database rows."""
import hashlib
import json
from functools import lru_cache
from pathlib import Path
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
REPO_ROOT = Path(__file__).resolve().parents[3]
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
def _asset_from_manifest(item: dict) -> CardImageAsset:
pending = item.get("redistribution_status") == "permission_not_verified"
return CardImageAsset(
id=item["id"],
title=item["label"],
url=item["url"],
width=item["width"],
height=item["height"],
credit=item["credit"],
source_url=item.get("source_page") or "",
license=item["license"],
license_url=item.get("license_url") or "",
kind=item["kind"],
rights_status="permission_pending" if pending else "available",
enabled=item.get("enabled", not pending),
focus_x=item.get("focus_x", 50),
focus_y=item.get("focus_y", 50),
variants=item.get("variants", {}),
)
@lru_cache(maxsize=1)
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: [CardImageAsset.model_validate(item) for item in value]
for key, value in manifest.get("previous_defaults", {}).items()
}
@lru_cache(maxsize=1)
def previous_bundled_rules() -> dict[str, CardImageRule]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: CardImageRule.model_validate(value)
for key, value in manifest.get("previous_rules", {}).items()
}
@lru_cache(maxsize=1)
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
content = MANIFEST_PATH.read_bytes()
manifest = json.loads(content)
assets, rules = [], []
for item in manifest["assets"]:
asset = _asset_from_manifest(item)
assets.append(asset)
if not asset.enabled:
continue
hints = item.get("binding_hints", {})
for index, target in enumerate(hints.get("defaults", [])):
rules.append(
CardImageRule(
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
entity_type=target["entity_type"],
scope=target["scope"],
value=target["value"],
label=asset.title,
asset_id=asset.id,
)
)
for index, entity_id in enumerate(hints.get("entity_ids", [])):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-entity-{index}",
entity_type=item["entity_type"],
scope="object",
value=entity_id,
label=asset.title,
asset_id=asset.id,
)
)
for norad_id in hints.get("norad_ids", []):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-{norad_id}",
entity_type="satellite",
scope="object",
value=str(norad_id),
label=f"{asset.title} · NORAD {norad_id}",
asset_id=asset.id,
)
)
if group := hints.get("constellation_group"):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-group",
entity_type="satellite",
scope="group",
value=group,
label=asset.title,
asset_id=asset.id,
)
)
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()

View File

@@ -0,0 +1,95 @@
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
import hashlib
import io
import re
import warnings
from uuid import uuid4
from pathlib import Path
import os
from fastapi import HTTPException
from PIL import Image, ImageOps, UnidentifiedImageError
from app.schemas.card_images import CardImageAsset
from app.services.card_image_defaults import REPO_ROOT
ASSET_DIR = Path(
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
).resolve()
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
MAX_PIXELS = 32_000_000
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
def uploaded_path(filename: str) -> Path:
if not FILE_PATTERN.fullmatch(filename):
raise HTTPException(404, "Image not found")
return ASSET_DIR / filename
def store_image(content: bytes, title: str) -> CardImageAsset:
if len(content) > MAX_UPLOAD_BYTES:
raise HTTPException(413, "Image must be 12 MiB or smaller")
try:
with warnings.catch_warnings():
warnings.simplefilter("error", Image.DecompressionBombWarning)
image = Image.open(io.BytesIO(content))
extension = FORMATS.get(image.format)
if (
not extension
or image.width * image.height > MAX_PIXELS
or getattr(image, "is_animated", False)
):
raise ValueError("Unsupported or oversized raster")
image.load()
image = ImageOps.exif_transpose(image).convert(
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
)
except (
UnidentifiedImageError,
OSError,
ValueError,
Image.DecompressionBombError,
Image.DecompressionBombWarning,
) as error:
raise HTTPException(
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
) from error
digest = hashlib.sha256(content).hexdigest()
ASSET_DIR.mkdir(parents=True, exist_ok=True)
original = f"{digest}-original.{extension}"
original_path = uploaded_path(original)
if not original_path.exists():
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(original_path)
variants = {}
for size, content in render_variants(image).items():
filename = f"{digest}-{size}.webp"
path = uploaded_path(filename)
if not path.exists():
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(path)
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
return CardImageAsset(
id=f"upload-{digest[:24]}",
title=title or "Image",
url=f"/api/v1/earth/card-images/files/{original}",
variants=variants,
width=image.width,
height=image.height,
)
def render_variants(image: Image.Image) -> dict[str, bytes]:
result = {}
for size in (640, 1280):
resized = image.copy()
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
buffer = io.BytesIO()
resized.save(buffer, format="WEBP", quality=93, method=4)
result[str(size)] = buffer.getvalue()
return result

View File

@@ -0,0 +1,180 @@
"""Bounded catalog queries and type defaults for every Earth object card."""
from sqlalchemy import String, case, cast, func, literal, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import (
COMPUTE_SITE_TYPES,
compute_site_type_expression,
compute_record_filter,
)
from app.models.bgp_incident import BGPIncident
from app.models.bgp_collector_location import BGPCollectorLocation
from app.models.collected_data import CollectedData
from app.models.earth_interactable import EarthInteractable
from app.models.vessel import VesselCurrentState
from app.schemas.card_images import EntityType, MatchScope
SOURCES = {
"satellite": ("celestrak_tle", "spacetrack_tle"),
"compute_center": ("top500", "epoch_ai_gpu"),
"cable": ("arcgis_cables",),
"landing_point": ("arcgis_landing_points",),
}
# Named constellation families, not inferred hardware models or generations.
SATELLITE_GROUP_PATTERNS = (
("STARLINK%", "starlink"),
("IRIDIUM%", "iridium-next"),
("BEIDOU-3%", "beidou-3"),
("BEIDOU%", "beidou"),
("QIANFAN%", "qianfan"),
("ONEWEB%", "oneweb"),
("KUIPER%", "kuiper"),
)
def collected_query(entity_type: EntityType):
record, metadata = CollectedData, CollectedData.extra_data
model = func.coalesce(
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
)
category = (
compute_site_type_expression(record)
if entity_type == "compute_center"
else literal(entity_type)
)
if entity_type == "satellite":
group = func.coalesce(
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
case(
*[
(func.upper(record.name).like(pattern), value)
for pattern, value in SATELLITE_GROUP_PATTERNS
],
else_="",
),
)
entity_id = metadata["norad_cat_id"].as_string()
else:
group = func.coalesce(
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
)
entity_id = record.source + ":" + record.source_id
return select(
record.name.label("name"),
cast(entity_id, String).label("entity_id"),
model.label("model"),
group.label("group"),
category.label("category"),
).where(
(
compute_record_filter(record)
if entity_type == "compute_center"
else record.source.in_(SOURCES[entity_type])
),
record.is_current.is_(True),
record.deleted_at.is_(None),
metadata["anonymous"].as_boolean().is_not(True),
)
def target_query(entity_type: EntityType):
if entity_type in SOURCES:
return collected_query(entity_type)
if entity_type == "vessel":
model = VesselCurrentState
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
conditions = []
elif entity_type == "bgp":
model = BGPIncident
identity, name, group, category = (
model.incident_key,
model.title,
model.source,
model.incident_type,
)
conditions = []
elif entity_type == "bgp_collector":
model = BGPCollectorLocation
identity, name, group, category = (
model.collector_id,
model.collector_id,
model.operator,
literal("bgp_collector"),
)
conditions = []
else:
model = EarthInteractable
identity, name, group, category = model.id, model.label, model.layer, model.kind
conditions = [model.is_deleted.is_(False)]
return select(
name.label("name"),
cast(identity, String).label("entity_id"),
literal("").label("model"),
func.coalesce(group, "").label("group"),
func.coalesce(category, entity_type).label("category"),
).where(*conditions)
def category_context(entity_type: EntityType) -> dict[str, str]:
return {
"entity_type": entity_type,
"entity_id": "",
"name": "",
"model": "",
"group": "",
"category": entity_type,
}
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
category = entity_id.partition(":")[2]
if category in COMPUTE_SITE_TYPES:
return {**category_context(entity_type), "category": category}
if entity_id == "__category__":
return category_context(entity_type)
if entity_type in SOURCES and entity_type != "satellite":
source, separator, source_id = entity_id.partition(":")
if not separator or not source or not source_id:
return category_context(entity_type)
# Match the indexed identity columns before projecting JSON metadata.
statement = collected_query(entity_type).where(
CollectedData.source == source, CollectedData.source_id == source_id
)
else:
catalog = target_query(entity_type).subquery()
statement = select(catalog).where(catalog.c.entity_id == entity_id)
row = (await db.execute(statement.limit(1))).mappings().first()
if not row:
# Old map snapshots can outlive a catalog refresh. They may use the type
# illustration, but must not claim an unverified exact-object identity.
return category_context(entity_type)
context = {key: str(value or "") for key, value in row.items()}
return {**context, "entity_type": entity_type}
async def search_targets(
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
) -> list[dict[str, str]]:
catalog = target_query(entity_type).subquery()
value = catalog.c["entity_id" if scope == "object" else scope]
label = catalog.c.name if scope == "object" else value
statement = select(value.label("value"), label.label("label")).where(
value.is_not(None), value != ""
)
if query:
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
statement = statement.where(
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
)
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
if scope == "category" and not query:
known = {item["value"] for item in items}
if entity_type == "compute_center":
items.extend(
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
)
items.insert(0, {"value": "*", "label": "*"})
return items

View File

@@ -31,6 +31,7 @@ class BaseCollector(ABC):
frequency_hours: int = 4
data_type: str = "generic"
fail_on_empty: bool = False
preserve_snapshot_on_failure: bool = False
def __init__(self):
self._current_task = None
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
if self.fail_on_empty and not data:
raise RuntimeError(f"Collector {self.name} produced no valid records")
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
raise
except Exception as e:
await db.rollback()
if snapshot_id is not None and self.preserve_snapshot_on_failure:
await self._rollback_incomplete_run(
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
)
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.phase_message = str(e)
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.status = SnapshotStatus.FAILED.value
if self.preserve_snapshot_on_failure:
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {"error": str(e)}
await db.commit()

View File

@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
from app.services.collectors.base import BaseCollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
logger = get_logger(__name__, service="collector")
ACTIVE_GROUP = "active"
FALLBACK_GROUPS = (
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
)
FETCH_RETRY_ATTEMPTS = 3
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
_downloader = ResumableFileDownloader(
cache_namespace="celestrak",
default_accept="application/json",
preflight_head=False,
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
)
@property
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
if not is_final_attempt:
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
raise RuntimeError(
f"CelesTrak active satellite download failed after retries: {last_error}"
)
async def _fetch_fallback_groups(
self,
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
try:
for group in FALLBACK_GROUPS:
group_url = self._group_url(group)
cached_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is not None:
body_path = cached_path
else:
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(
f"CelesTrak fallback group '{group}' download failed: {exc}"
) from exc
body_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if body_path is None:
raise RuntimeError(
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
) from exc
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
commit=True,
)
@staticmethod
def _validate_json_file(path: Path) -> bool:
@classmethod
def _validate_json_file(cls, path: Path) -> bool:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
cls._load_active_payload(path)
except RuntimeError:
return False
return isinstance(data, list)
return True
async def _log_parse_failure(self, exc: Exception) -> None:
await emit_business_log(
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
),
)
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
@staticmethod
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
else:
invalid_count += 1
if invalid_count:
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
raise RuntimeError(
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
)
if not records:
raise RuntimeError("CelesTrak active payload contains no satellite records")
return records

View File

@@ -0,0 +1,82 @@
"""Quality gates shared by the two compute sources."""
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import normalize_country
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.models.data_snapshot import DataSnapshot
from app.services.collectors.base import BaseCollector
class ComputeCollector(BaseCollector):
fail_on_empty = True
preserve_snapshot_on_failure = True
async def _create_snapshot(
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
) -> int:
previous_count = await db.scalar(
select(DataSnapshot.record_count)
.where(
DataSnapshot.source == self.name,
DataSnapshot.is_current.is_(True),
DataSnapshot.status == "success",
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
if previous_count and len(data) < previous_count * 0.6:
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
return await super()._create_snapshot(db, task_id, data, started_at)
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
if self.name == "top500":
await self._copy_verified_locations(db, data)
return await super()._save_data(db, data, **kwargs)
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
"""Keep historical location keys; copy only unambiguous exact identities.
Never migrate by rank: the machine occupying rank 1 changes over time.
"""
names = [item["name"] for item in data]
locations = (
(
await db.execute(
select(ComputeCenterLocationRecord).where(
ComputeCenterLocationRecord.source == self.name,
ComputeCenterLocationRecord.name.in_(names),
)
)
)
.scalars()
.all()
)
existing = {location.source_id for location in locations}
for item in data:
if item["source_id"] in existing:
continue
country = normalize_country(item["country"])
if not country:
continue
matches = [
location
for location in locations
if location.name == item["name"]
and normalize_country(location.country) == country
and (not location.site or location.site == item.get("metadata", {}).get("site"))
and not location.source_id.startswith("top500_system_")
]
if len(matches) != 1:
continue
old = matches[0]
values = {
column.name: getattr(old, column.name)
for column in old.__table__.columns
if column.name not in {"id", "source_id", "created_at", "updated_at"}
}
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
existing.add(item["source_id"])

View File

@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
import httpx
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
ValidateCallback = Callable[[Path], bool]
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
default_accept: str = "*/*",
cache_root: Path | None = None,
preflight_head: bool = True,
cache_ttl_seconds: float = 0,
) -> None:
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
self._user_agent = user_agent
self._default_accept = default_accept
self._preflight_head = preflight_head
self._cache_ttl_seconds = cache_ttl_seconds
@staticmethod
def _cache_key(url: str) -> str:
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
if not meta_path.exists():
return {}
try:
return json.loads(meta_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
data = json.loads(meta_path.read_text(encoding="utf-8"))
return data if isinstance(data, dict) else {}
except json.JSONDecodeError, OSError:
return {}
@staticmethod
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
return last_modified == str(meta.get("last_modified") or "").strip()
return True
@staticmethod
def _response_info(response: httpx.Response) -> dict[str, Any]:
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
raw_length = response.headers.get("content-length")
encoded = response.headers.get("content-encoding", "identity") != "identity"
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": int(raw_length) if raw_length and not encoded else None,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
try:
response = await client.head(url)
if response.status_code >= 400:
return {}
content_length_raw = response.headers.get("content-length")
content_length = int(content_length_raw) if content_length_raw else None
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": content_length,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
except (httpx.HTTPError, ValueError):
return self._response_info(response)
except httpx.HTTPError, ValueError:
return {}
async def download_file(
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
) -> Path:
self._cache_dir.mkdir(parents=True, exist_ok=True)
final_path, part_path, meta_path = self._cache_paths(url, extension)
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
size = cached_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return cached_path
meta = self._load_meta(meta_path)
remote = await self.fetch_remote_info(client, url)
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
expected_size = remote.get("content_length")
if final_path.exists():
local_size = final_path.stat().st_size
size_match = expected_size is None or local_size == expected_size
if self._validators_match(meta, remote) and size_match:
if (
(remote.get("etag") or remote.get("last_modified"))
and self._validators_match(meta, remote)
and size_match
):
if validate_existing and not validate_existing(final_path):
final_path.unlink(missing_ok=True)
else:
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
headers["If-Range"] = str(remote.get("last_modified"))
async with client.stream("GET", url, headers=headers) as response:
if response.status_code == 304 and final_path.exists():
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)
if response.status_code == 304:
if cached_path is None:
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
final_path.touch()
size = final_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return final_path
if response.status_code >= 400:
body = (await response.aread()).decode("utf-8", errors="replace")
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
else:
mode = "wb"
resume_from = 0
remote = self._response_info(response)
expected_size = remote.get("content_length")
downloaded = resume_from
last_emit_bytes = 0
last_emit_time = time.monotonic()
min_emit_bytes = (
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
max(expected_size // 150, 512 * 1024)
if expected_size and expected_size > 0
else 1024 * 1024
)
with part_path.open(mode) as f:
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
)
# Never replace the last good payload with an HTML error or invalid JSON.
if validate_existing and not validate_existing(part_path):
part_path.unlink(missing_ok=True)
raise RuntimeError(f"Downloaded file validation failed for {url}")
part_path.replace(final_path)
self._save_meta(
meta_path,
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
},
)
if validate_existing and not validate_existing(final_path):
raise RuntimeError(f"Downloaded file validation failed for {url}")
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)

View File

@@ -1,119 +1,135 @@
"""Epoch AI GPU Clusters Collector
Collects data from Epoch AI GPU clusters tracking.
https://epoch.ai/data/gpu-clusters
"""
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
import csv
import hashlib
import io
import json
import math
import re
from typing import Dict, Any, List
from datetime import UTC, datetime
from bs4 import BeautifulSoup
from urllib.parse import urlparse
import httpx
from app.services.collectors.base import BaseCollector
from app.core.countries import normalize_country
from app.services.collectors.compute_base import ComputeCollector
from app.services.epoch_location import assess_epoch_location
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
class EpochAIGPUCollector(BaseCollector):
class EpochAIGPUCollector(ComputeCollector):
name = "epoch_ai_gpu"
priority = "P0"
module = "L1"
frequency_hours = 6
data_type = "gpu_cluster"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch Epoch AI GPU clusters data from webpage"""
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
async def fetch(self) -> list[dict]:
configured = self._resolved_url or EPOCH_CSV_URL
parsed = urlparse(configured)
url = (
EPOCH_CSV_URL
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
else configured
)
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.text)
return self.parse_response(response.text)
def parse_response(self, html: str) -> List[Dict[str, Any]]:
"""Parse Epoch AI webpage to extract GPU cluster data"""
data = []
soup = BeautifulSoup(html, "html.parser")
def parse_response(self, text: str) -> list[dict]:
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
if not required.issubset(reader.fieldnames or []):
raise ValueError(
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
)
entries = []
seen = set()
anonymous_counts = {}
for row in reader:
name = (row.get("Name") or "").strip()
country = normalize_country(row.get("Country"))
if not name:
raise ValueError("Epoch row has missing identity")
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
anonymous = "anonym" in name.lower()
if anonymous:
# Public anonymous rows have no facility identity. Preserve
# multiplicity within snapshots, without guessing names.
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
if source_id in seen:
raise ValueError("Epoch identity collision; previous snapshot retained")
seen.add(source_id)
raw_date = row.get("First Operational Date", "")
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
value = None
raw_value = row.get("16-bit OP/s (log)", "")
if raw_value:
exponent = float(raw_value)
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
raise ValueError("Epoch performance exponent out of bounds")
value = 10**exponent
anonymous = anonymous or (country == "中国" and not row.get("Location"))
location = row.get("Location")
location_assessment = assess_epoch_location(
location,
country=country,
raw_country=row.get("Country"),
)
entries.append(
{
"source_id": source_id,
"name": name,
"country": country,
"value": value,
"unit": "FLOP/s",
"reference_date": reference_date,
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
"metadata": {
"compute_contract": "v1",
"precision": "FP16",
"measurement": "theoretical",
"source_url": EPOCH_CSV_URL,
"raw_reference_date": raw_date,
"status": row.get("Status"),
"certainty": row.get("Certainty"),
"operator": row.get("Owner"),
"gpu_type": row.get("Chip type (primary)"),
"gpu_count": row.get("Chip quantity (primary)"),
"location": location,
"location_precision": location_assessment.precision,
"location_precision_reason": location_assessment.reason,
"raw_country": row.get("Country"),
"raw_performance": raw_value,
"performance_encoding": "log10",
"builds_upon": row.get("Builds Upon"),
"superseded_by": row.get("Superseded by"),
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
"include_in_standard_analysis": row.get(
"Include in Standard Analysis", ""
).lower()
== "true",
"rounded": country == "中国",
"anonymous": anonymous,
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
"country_unresolved": country is None,
"sources": [
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
],
},
}
)
return entries
# Try to find data table on the page
tables = soup.find_all("table")
for table in tables:
rows = table.find_all("tr")
for row in rows[1:]: # Skip header
cells = row.find_all(["td", "th"])
if len(cells) >= 5:
try:
cluster_name = cells[0].get_text(strip=True)
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
continue
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
country, city = self._parse_location(location_cell)
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
entry = {
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
"name": cluster_name,
"country": country,
"city": city,
"latitude": "",
"longitude": "",
"value": self._parse_performance(perf_cell),
"unit": "TFlop/s",
"metadata": {
"raw_data": perf_cell,
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}
data.append(entry)
except (ValueError, IndexError, AttributeError):
continue
# If no table found, return sample data
if not data:
data = self._get_sample_data()
return data
def _parse_location(self, location: str) -> tuple:
"""Parse location string into country and city"""
if not location:
return "", ""
if "," in location:
parts = location.rsplit(",", 1)
city = parts[0].strip()
country = parts[1].strip() if len(parts) > 1 else ""
return country, city
return location, ""
def _parse_performance(self, perf: str) -> str:
"""Parse performance string to extract value"""
if not perf:
return "0"
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
if match:
return match.group(1).replace(",", "")
match = re.search(r"([\d,.]+)", perf)
if match:
return match.group(1).replace(",", "")
return "0"
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "epoch_sample_1",
"name": "Sample GPU Cluster",
"country": "United States",
"city": "San Francisco, CA",
"latitude": "",
"longitude": "",
"value": "1000",
"unit": "TFlop/s",
"metadata": {
"note": "Sample data - Epoch AI page structure may vary",
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
},
]
@staticmethod
def coordinate(value: str | None, bound: int) -> float | None:
if not value:
return None
number = float(value)
if not math.isfinite(number) or abs(number) > bound:
raise ValueError("Epoch coordinate out of bounds")
return number

View File

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
"""
import asyncio
import math
import re
from urllib.parse import urljoin
from typing import Dict, Any, List
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
from app.services.collectors.compute_base import ComputeCollector
class TOP500Collector(BaseCollector):
class TOP500Collector(ComputeCollector):
name = "top500"
priority = "P0"
module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
entries = self.parse_response(response.text)
entries = self.parse_response(response.text, str(response.url))
semaphore = asyncio.Semaphore(8)
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
detail_url = ""
if link and link.get("href"):
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
detail_url = urljoin(str(top500_base_url), link.get("href"))
manufacturer = ""
if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
return detail_map
def parse_response(self, html: str) -> List[Dict[str, Any]]:
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
"""Parse TOP500 HTML response"""
data = []
soup = BeautifulSoup(html, "html.parser")
canonical = soup.find("link", rel="canonical")
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
# Find the table with TOP500 data
table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
if table:
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
if not reference_date:
raise ValueError("TOP500 list date missing; previous snapshot retained")
rows = table.select("tr")
for row in rows[1:]: # Skip header row
cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
cores = cells[2].get_text(strip=True).replace(",", "")
rmax_text = cells[3].get_text(strip=True)
rmax = self._parse_performance(rmax_text)
rmax = self._parse_performance(rmax_text, headers[3])
rpeak_text = cells[4].get_text(strip=True)
rpeak = self._parse_performance(rpeak_text)
rpeak = self._parse_performance(rpeak_text, headers[4])
power = ""
if len(cells) >= 6:
power = cells[5].get_text(strip=True).replace(",", "")
identity = re.search(r"/system/(\d+)", detail_url)
if not identity:
raise ValueError("TOP500 stable system ID missing")
entry = {
"source_id": f"top500_{rank}",
"source_id": f"top500_system_{identity[1]}",
"name": system_name,
"country": country,
"city": city,
"latitude": 0.0,
"longitude": 0.0,
"latitude": None,
"longitude": None,
"value": str(rmax),
"unit": "PFlop/s",
"unit": "FLOP/s",
"metadata": {
"compute_contract": "v1",
"precision": "FP64", "measurement": "LINPACK Rmax",
"source_url": detail_url, "list_url": date_url,
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
"rank": rank,
"cores": cores,
"rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
"site": site,
},
"_detail_url": detail_url,
"reference_date": "2025-11-01",
"reference_date": reference_date,
}
data.append(entry)
except (ValueError, IndexError, AttributeError) as e:
continue
# If scraping failed, return sample data for testing
if not data:
data = self._get_sample_data()
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
return data
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
return 0.0
return 0.0
def _parse_performance(self, text: str) -> float:
def _parse_performance(self, text: str, header: str = "") -> float:
"""Parse performance value from text (handles E, P, T suffixes)"""
text = text.strip().upper()
text = text.strip().upper().replace(",", "")
if not re.search(r"[EPTGMK]F", text):
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
if not unit:
raise ValueError("TOP500 performance unit missing")
text = f"{text} {unit[1]}FLOP/S"
multipliers = {
"E": 1e18,
"P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
# Try simple float parsing
try:
return float(text.replace(",", ""))
value = float(text.replace(",", ""))
if not math.isfinite(value) or value < 0:
raise ValueError("Invalid TOP500 performance")
return value
except ValueError:
return 0.0
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "top500_1",
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
"country": "United States",
"city": "Livermore, CA",
"latitude": 37.6819,
"longitude": -121.7681,
"value": "1742.00",
"unit": "PFlop/s",
"metadata": {
"rank": 1,
"cores": "11039616",
"rmax": "1742.00",
"rpeak": "2746.38",
"power": "29581",
"manufacturer": "HPE",
"site": "DOE/NNSA/LLNL",
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
"interconnect": "Slingshot-11",
"installation_year": "2025",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_2",
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
"country": "United States",
"city": "Oak Ridge, TN",
"latitude": 36.0107,
"longitude": -84.2663,
"value": "1353.00",
"unit": "PFlop/s",
"metadata": {
"rank": 2,
"cores": "9066176",
"rmax": "1353.00",
"rpeak": "2055.72",
"power": "24607",
"manufacturer": "HPE",
"site": "DOE/SC/Oak Ridge National Laboratory",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_3",
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
"country": "United States",
"city": "Argonne, IL",
"latitude": 41.3784,
"longitude": -87.8600,
"value": "1012.00",
"unit": "PFlop/s",
"metadata": {
"rank": 3,
"cores": "9264128",
"rmax": "1012.00",
"rpeak": "1980.01",
"power": "38698",
"manufacturer": "Intel",
},
"reference_date": "2025-11-01",
},
]
raise ValueError("Invalid TOP500 performance") from None

View File

@@ -0,0 +1,200 @@
"""Canonical country and mutually exclusive regional compute participants."""
import re
from collections.abc import Iterable
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
from app.core.earth_geography_policy import display_country
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
# Explicit membership prevents a regional total from double-counting a country.
# APAC deliberately excludes China.
COMPARISON_GROUPS = (
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
(
"EUROPE",
"欧洲",
"Europe",
(
"GB",
"DE",
"FR",
"CH",
"ES",
"FI",
"HU",
"IE",
"IT",
"LU",
"MD",
"NL",
"PL",
"RO",
"RU",
"UA",
),
{"latitude": 54.0, "longitude": 20.0},
),
(
"APAC",
"亚太(不含中国)",
"Asia Pacific (excluding China)",
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
{"latitude": 36.0, "longitude": 130.5},
),
(
"OTHER",
"其他地区",
"Other",
(
"CA",
"AO",
"EG",
"KE",
"MZ",
"NG",
"UG",
"ZA",
"BR",
"CO",
"CR",
"DO",
"MX",
"AE",
"SY",
"TR",
),
{"latitude": 12.0, "longitude": 10.0},
),
)
def _country_code(canonical: str, aliases: list[str]) -> str | None:
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
return "GB" if canonical == "英国" else code
def build_country_catalog() -> list[dict]:
"""Build the policy-projected canonical country drilldown catalog."""
countries = []
for canonical, aliases in COUNTRY_ENTRIES:
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
continue
code = _country_code(canonical, aliases)
if code:
countries.append(
{
"code": code,
"name": canonical,
"name_en": aliases[0],
"kind": "country",
"members": [code],
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
"anchor_kind": "regional_label",
}
)
return countries
COUNTRY_CATALOG = build_country_catalog()
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
def _validate_group_members() -> None:
members = [
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
]
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
raise ValueError("compute comparison groups must cover country catalog exactly once")
_validate_group_members()
GROUP_BY_MEMBER = {
member: code
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
}
def build_catalog() -> list[dict]:
"""Return the five default participants before country drilldown entries."""
catalog: list[dict] = []
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
entry = (
dict(COUNTRY_PARTICIPANTS[code])
if code in COUNTRY_PARTICIPANTS
else {
"code": code,
"name": name,
"name_en": name_en,
"kind": "region",
"members": list(members),
"anchor": dict(anchor),
"anchor_kind": "regional_label",
}
)
entry.update(
{
"members": list(members),
"parent_group": name,
"parent_group_code": code,
"order": index,
"default_selected": True,
"color": PALETTE[index % len(PALETTE)],
}
)
catalog.append(entry)
for entry in COUNTRY_CATALOG:
if entry["code"] in DEFAULT_PARTICIPANTS:
continue
country = dict(entry)
group_code = GROUP_BY_MEMBER[country["code"]]
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
country.update(
{
"parent_group": group_name,
"parent_group_code": group_code,
"order": len(catalog),
"default_selected": False,
"color": PALETTE[len(catalog) % len(PALETTE)],
}
)
catalog.append(country)
return catalog
CATALOG = build_catalog()
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
"""Resolve country or region participant codes to canonical country codes."""
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
def source_country_names(codes: Iterable[str]) -> list[str]:
"""Return unique canonical source labels, respecting China and Serbia ownership."""
selected_members = {
member for members in participant_members(codes).values() for member in members
}
selected_names = {COUNTRIES[code] for code in selected_members}
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
"""Map stored source-country labels to their requested participant code."""
owner_by_country = {
member: participant
for participant, members in participant_members(codes).items()
for member in members
}
code_by_name = {name: code for code, name in COUNTRIES.items()}
return {
name: owner_by_country[code_by_name[display_country(name)]]
for name in source_country_names(codes)
}

View File

@@ -3,7 +3,12 @@
This module is a thin domain wrapper that wires up
:mod:`app.services.location` for compute centers:
SourceCoordinates
Confirmed stored location -> trustworthy source coordinates -> exact
official facility registration -> source city/site coordinates.
Epoch AI coordinates are admitted only at the grain asserted by its official
``Location`` column. Country/region/empty locations remain unresolved even
when the CSV happens to contain a latitude/longitude pair.
The online Nominatim step is intentionally reserved for the user-triggered
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -28,9 +33,12 @@ import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import compute_record_filter
from app.core.collected_data_fields import get_record_field
from app.models.collected_data import CollectedData
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.services.compute_center_verified_sites import get_verified_compute_site
from app.services.epoch_location import assess_epoch_location
from app.services.location import (
LocationCandidate,
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
async def refresh_compute_center_location_cache(
session: AsyncSession,
*,
as_of: datetime | None = None,
) -> dict[str, dict[str, Any]]:
result = await session.execute(select(ComputeCenterLocationRecord))
query = select(ComputeCenterLocationRecord)
if as_of is not None:
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
result = await session.execute(query)
records = result.scalars().all()
cache = {}
for record in records:
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
)
class ComputeCenterSourceCoordinatesResolver:
"""Use source coordinates only at the grain asserted by the source text."""
name = "source_coordinates"
def __init__(self) -> None:
self._default = SourceCoordinatesResolver()
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
if extra.get("source") != "epoch_ai_gpu":
return self._default.resolve(query)
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
return ResolverOutput()
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput(
attempted_queries=(f"source_coordinates:{assessment.precision}",)
)
output = self._default.resolve(query)
if not output.candidates:
return output
candidate = output.candidates[0]
source_location = extra.get("source_location")
source_location_text = (
source_location.strip() if isinstance(source_location, str) else ""
)
source_note = assessment.reason
if source_location_text:
source_note = f"{source_note}; Epoch Location: {source_location_text}"
return ResolverOutput(
candidates=(
LocationCandidate(
**{
**candidate.__dict__,
"precision": assessment.precision,
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
"source_note": source_note,
"needs_confirmation": assessment.precision != "precise",
"matched_location_name": source_location_text
or candidate.matched_location_name,
}
),
),
attempted_queries=output.attempted_queries,
)
class StoredComputeCenterLocationResolver:
"""Resolve a compute center through the DB-backed current-location cache."""
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
longitude = parse_float(stored.get("longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
return ResolverOutput()
precision = stored.get("precision") or "city"
if (
extra.get("source") == "epoch_ai_gpu"
and stored.get("location_source") == "source_coordinates"
):
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput()
precision = assessment.precision
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=latitude,
longitude=longitude,
display_name=stored.get("name") or query.name or "Compute center",
precision=stored.get("precision") or "city",
precision=precision,
confidence=float(stored.get("confidence") or 0.85),
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
source=self.name,
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
matched_location_name=stored.get("site") or stored.get("name") or query.name,
location_verified_at=stored.get("verified_at"),
suggested_registry_entry=None,
raw_payload={
"stored_location_source": stored.get("location_source"),
"verification_status": stored.get("verification_status"),
},
),
)
)
class VerifiedComputeSiteResolver:
"""Resolve an exact source identity to an evidence-backed facility site."""
name = "official_facility_registry"
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
source = coerce_str(extra.get("source"))
source_id = coerce_str(extra.get("source_id"))
if not source or not source_id:
return ResolverOutput()
site = get_verified_compute_site(source, source_id)
if not site:
return ResolverOutput()
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
source_note = coerce_str(site.get("source_note"))
if source_urls:
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=float(site["latitude"]),
longitude=float(site["longitude"]),
display_name=coerce_str(site.get("matched_location_name"))
or query.name
or "Compute center",
precision=coerce_str(site.get("precision")) or "site",
confidence=1.0,
query=f"official_facility_registry::{source}:{source_id}",
source=self.name,
source_note=source_note or None,
matched_fields=("source", "source_id"),
needs_confirmation=False,
city=coerce_str(site.get("city")) or None,
region=None,
country=coerce_str(site.get("country")) or query.country,
matched_location_name=coerce_str(site.get("matched_location_name"))
or None,
location_verified_at=coerce_str(site.get("verified_at")) or None,
suggested_registry_entry=None,
raw_payload={
"address": site.get("address"),
"coordinate_basis": site.get("coordinate_basis"),
"sources": source_urls,
},
),
)
)
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
return head or text
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
name = coerce_str(getattr(record, "name", None))
return {
"source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
or metadata.get("manufacturer")
),
"organization": coerce_str(metadata.get("organization")),
"source_location": metadata.get("location"),
"source_raw_country": metadata.get("raw_country"),
}
def _context_to_query(
context: dict[str, str],
context: dict[str, Any],
*,
source_lat: float | None = None,
source_lon: float | None = None,
@@ -444,6 +578,8 @@ def _context_to_query(
"site": context.get("site") or "",
"operator": context.get("operator") or "",
"organization": context.get("organization") or "",
"source_location": context.get("source_location"),
"source_raw_country": context.get("source_raw_country"),
},
)
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
return plan
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
stored_confirmed = (
candidate.source == "stored_compute_center_location"
and not candidate.needs_confirmation
and stored_origin != "source_coordinates"
)
if stored_confirmed:
source_rank = 0
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
source_rank = 1
elif candidate.source == "official_facility_registry":
source_rank = 2
elif candidate.source == "source_coordinates":
source_rank = 3
elif candidate.source == "stored_compute_center_location":
source_rank = 4
else:
source_rank = 5
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
return (source_rank, precision_rank, -float(candidate.confidence or 0))
COMPUTE_CENTER_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
StoredComputeCenterLocationResolver(),
VerifiedComputeSiteResolver(),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
" or stored compute-center location."
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
VerifiedComputeSiteResolver(),
ROROrganizationResolver(),
NominatimResolver(
query_plan_builder=_compute_center_query_plan,
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
geocoder=lambda q: _geocode_online(q),
),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
", ROR organization lookup, or online geocoding."
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
_GEOGRAPHY_MODE_BY_SOURCE = {
"source_coordinates": "source_coordinates",
"stored_compute_center_location": "stored_compute_center_location",
"official_facility_registry": "official_facility_registry",
"ror_organization_registry": "ror_organization",
"nominatim_online_geocode": "online_geocode",
}
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
def _candidate_to_location(
candidate: LocationCandidate,
*,
context: dict[str, str],
context: dict[str, Any],
) -> ComputeCenterLocation:
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +758,7 @@ def _candidate_to_location(
def _diagnostic_for(
record: Any,
context: dict[str, str],
context: dict[str, Any],
*,
failure_reason: str,
attempted_queries: tuple[str, ...] = (),
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
organization: str | None = None,
) -> LocationQuery:
name_value = coerce_str(name)
context: dict[str, str] = {
context: dict[str, Any] = {
"source": coerce_str(source),
"source_id": coerce_str(source_id),
"name": name_value,
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
"""Seed stored compute-center locations only from real source coordinates."""
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.is_current.is_(True))
)
result = await session.execute(stmt)
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
longitude = parse_float(get_record_field(record, "longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
continue
metadata = record.extra_data or {}
precision = "precise"
confidence = 1.0
needs_confirmation = False
source_note = "Seeded from source-provided compute-center coordinates"
if source_value == "epoch_ai_gpu":
assessment = assess_epoch_location(
metadata.get("location"),
country=get_record_field(record, "country"),
raw_country=metadata.get("raw_country"),
)
if not assessment.is_renderable:
continue
precision = assessment.precision
confidence = 1.0 if precision == "precise" else 0.9
needs_confirmation = precision != "precise"
source_note = assessment.reason
existing = await session.scalar(
select(ComputeCenterLocationRecord)
.where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
)
if existing:
continue
metadata = record.extra_data or {}
session.add(
ComputeCenterLocationRecord(
source=source_value,
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
country=coerce_str(get_record_field(record, "country")) or None,
latitude=latitude,
longitude=longitude,
precision="precise",
confidence=1.0,
precision=precision,
confidence=confidence,
location_source="source_coordinates",
source_note="Seeded from source-provided compute-center coordinates",
source_note=source_note,
raw_payload={
"record_id": getattr(record, "id", None),
"source": source_value,
"source_id": source_id,
},
needs_confirmation=False,
needs_confirmation=needs_confirmation,
verification_status="source_provided",
verified_at=None,
)

View File

@@ -0,0 +1,44 @@
{
"version": 1,
"sites": [
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
"latitude": 35.380924,
"longitude": 139.625008,
"precision": "site",
"location_source": "official_facility_registry",
"country": "日本",
"city": "Yokohama",
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
]
},
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
"latitude": 41.718109383287235,
"longitude": -87.984498648496,
"precision": "site",
"location_source": "official_facility_registry",
"country": "美国",
"city": "Argonne",
"matched_location_name": "Argonne Leadership Computing Facility",
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.alcf.anl.gov/systems/sophia",
"https://docs.alcf.anl.gov/sophia/",
"https://www.alcf.anl.gov/about/visiting-alcf"
]
}
]
}

View File

@@ -0,0 +1,26 @@
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
import json
from copy import deepcopy
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=1)
def _site_index() -> dict[tuple[str, str], dict]:
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
index = {}
for site in payload["sites"]:
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
raise ValueError("Invalid verified compute facility coordinate")
for source_id in site["source_ids"]:
key = (site["source"], source_id)
if key in index:
raise ValueError("Duplicate verified compute facility identity")
index[key] = site
return index
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
site = _site_index().get((source, source_id))
return deepcopy(site) if site else None

View File

@@ -0,0 +1,99 @@
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
import hashlib
from pathlib import Path
from uuid import uuid4
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import COUNTRY_CATALOG
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
CATEGORY = "compute_comparison"
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
def default_envelope() -> dict:
return {
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
"revision": DEFAULT_REVISION,
"is_default": True,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return dict(record.payload) if record else default_envelope()
def configuration_response(envelope: dict) -> dict:
return {
**envelope,
"countries": [
{key: country[key] for key in ("code", "name", "name_en")}
for country in COUNTRY_CATALOG
],
}
async def publish_configuration(
db: AsyncSession,
*,
revision: str,
config: ComputeConfiguration,
actor: User,
reset: bool = False,
) -> dict:
# The unique category plus row lock also serializes concurrent first writes.
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=default_envelope())
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if record.payload["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
envelope = (
default_envelope()
if reset
else {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"is_default": reset,
}
)
before = record.payload
if reset:
await db.delete(record)
else:
record.payload = envelope
# Audit and configuration commit together: a published edit must be attributable.
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="compute_comparison.reset" if reset else "compute_comparison.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": envelope},
)
)
await db.commit()
return envelope

View File

@@ -0,0 +1,122 @@
"""Versioned, source-backed annual observations for compute comparisons."""
from copy import deepcopy
from datetime import date
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import (
CATALOG,
DEFAULT_PARTICIPANTS,
PARTICIPANTS,
participant_members,
)
from app.services.compute_configuration import DEFAULT_CONFIGURATION
def _participant_value(
members: tuple[str, ...], observations: dict[str, float | int], mode: str
) -> tuple[float | int | None, dict]:
covered = [member for member in members if member in observations]
missing = [member for member in members if member not in observations]
coverage = {
"status": "complete" if not missing else "incomplete",
"required_members": list(members),
"covered_members": covered,
"missing_members": missing,
}
if missing:
return None, coverage
if mode == "bilateral_share":
if (
not set(members).issubset({"CN", "US"})
or not {"CN", "US"}.issubset(observations)
or observations["US"] + observations["CN"] <= 0
):
coverage.update(
{
"status": "incompatible",
"reason": "bilateral_share_requires_CN_and_US_observations",
}
)
return None, coverage
return (
round(
sum(observations[member] for member in members)
/ (observations["US"] + observations["CN"])
* 100,
2,
),
coverage,
)
return sum(observations[member] for member in members), coverage
def compute_history(
countries: list[str],
mode: str = "global_share",
config: ComputeConfiguration | None = None,
) -> dict:
"""Aggregate only fully covered, same-year compatible observations."""
config = config if config is not None else DEFAULT_CONFIGURATION
metric = "global_share" if mode == "bilateral_share" else mode
annual = {row.year: row for row in config.observations if row.metric == metric}
gaps = {row.year: row for row in config.innovation}
observed_years = [row.year for row in config.observations] + list(gaps)
years = tuple(
range(
min(observed_years, default=date.today().year),
max([date.today().year, *observed_years]) + 1,
)
)
members_by_participant = participant_members(countries)
rows = []
for year in years:
record = annual.get(year)
observations = record.values if record else {}
values, coverage = {}, {}
for code, members in members_by_participant.items():
values[code], coverage[code] = _participant_value(members, observations, mode)
evidence_id = record.source_id if record else None
rows.append(
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
)
selected_members = {member for members in members_by_participant.values() for member in members}
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
return {
"version": f"compute-history-{config.reviewed_at}",
"provenance": "curated_reports",
"reviewed_at": config.reviewed_at.isoformat(),
"time_basis": "retrospective",
"years": years,
"mode": mode,
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
mode
],
"catalog": deepcopy(CATALOG),
"default_participants": list(DEFAULT_PARTICIPANTS),
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
"compute": rows,
"innovation": [
{
"year": year,
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
"evidence_id": gaps[year].source_id
if year in gaps and has_innovation_scope
else None,
}
for year in years
],
"evidence": {
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
},
"innovation_scope": ["US", "CN"],
"limitations": [
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
"全球份额不从中美或其他局部成员反推剩余地区。",
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
],
}

View File

@@ -0,0 +1,125 @@
{
"reviewed_at": "2026-09-22",
"sources": [
{
"id": "caict-2021",
"title": "中国算力发展指数白皮书(2021)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2021,
"published_at": null,
"locator": "正文 11 页图 4(2020 年全球份额)"
},
{
"id": "caict-2022",
"title": "中国算力发展指数白皮书(2022)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2022,
"published_at": null,
"locator": "正文 13 页图 5(2021 年全球份额)"
},
{
"id": "caict-2023",
"title": "中国算力发展指数白皮书(2023)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2023,
"published_at": null,
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
},
{
"id": "caict-2024",
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2024,
"published_at": null,
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
},
{
"id": "innovation-2026",
"title": "全球人工智能创新指数报告 2026:科技日报报道",
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
"publisher": "科技日报(新浪转载)",
"report_year": 2026,
"published_at": "2026-07-17",
"locator": "2023–2025 年美国减中国的指数分差"
}
],
"observations": [
{
"year": 2020,
"metric": "global_share",
"source_id": "caict-2021",
"values": {
"US": 36,
"CN": 31,
"JP": 6
}
},
{
"year": 2021,
"metric": "global_share",
"source_id": "caict-2022",
"values": {
"US": 34,
"CN": 33,
"JP": 5
}
},
{
"year": 2022,
"metric": "global_share",
"source_id": "caict-2023",
"values": {
"US": 34,
"CN": 33,
"JP": 4
}
},
{
"year": 2023,
"metric": "global_share",
"source_id": "caict-2024",
"values": {
"US": 41,
"CN": 31,
"JP": 4
}
},
{
"year": 2022,
"metric": "absolute",
"source_id": "caict-2023",
"values": {
"CN": 302
}
},
{
"year": 2023,
"metric": "absolute",
"source_id": "caict-2024",
"values": {
"CN": 435
}
}
],
"innovation": [
{
"year": 2023,
"value": 22.02,
"source_id": "innovation-2026"
},
{
"year": 2024,
"value": 19.96,
"source_id": "innovation-2026"
},
{
"year": 2025,
"value": 17.95,
"source_id": "innovation-2026"
}
]
}

View File

@@ -6,8 +6,10 @@ import asyncio
import hashlib
import json
import shutil
import subprocess
import sys
from datetime import UTC, datetime
from functools import lru_cache
from pathlib import Path
from typing import Any
from uuid import uuid4
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = (
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
)
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
POLICY_FALLBACK_PATH = (
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
)
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
@lru_cache(maxsize=16)
def _read_versioned_asset_cached(
path: Path, mtime_ns: int, size_bytes: int
) -> tuple[dict[str, Any], str]:
payload = _read_json(path)
return payload, _stable_json_hash(payload)
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
if not path.exists():
return {}, ""
stat = path.stat()
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
path = REPO_ROOT / source["path"]
payload = _read_json(path)
features = payload.get("features") if isinstance(payload, dict) else None
return {
"type": "FeatureCollection",
"features": features if isinstance(features, list) else [],
}
def _write_high_precision_geojson_manifest(
sources: list[dict[str, Any]],
build_input_hash: str,
missing_tools: list[str],
) -> dict[str, Any]:
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
admin0_payload = _load_source_feature_collection(admin0)
coastline_payload = _load_source_feature_collection(coastline)
claim_payload = _load_source_feature_collection(claim_lines)
for feature in coastline_payload["features"]:
props = feature.setdefault("properties", {})
if isinstance(props, dict):
props["PLANET_LAYER"] = "coastline"
base_payload = {
"type": "FeatureCollection",
"features": [*admin0_payload["features"], *coastline_payload["features"]],
}
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
_write_json(base_path, base_payload)
_write_json(hover_path, admin0_payload)
_write_json(claim_path, claim_payload)
manifest = {
"version": "natural-earth-v1",
"builtAt": _utc_now_iso(),
"tileProvider": "geojson-high-precision",
"format": "geojson-directory",
"buildInputHash": build_input_hash,
"base": base_path.name,
"hoverIndex": hover_path.name,
"claimLine": claim_path.name,
"sourceFeatureCount": {
"admin0": len(admin0_payload["features"]),
"coastline": len(coastline_payload["features"]),
"claimLines": len(claim_payload["features"]),
},
"pmtiles": None,
"missingTools": missing_tools,
}
result = subprocess.run(
[
sys.executable,
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
"--admin0-source", admin0["path"],
"--coastline-source", coastline["path"],
"--claims-source", claim_lines["path"],
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
"--build-input-hash", build_input_hash,
"--pov-policy", _relative(POV_POLICY_PATH),
"--pov-geometry", str(POV_GEOMETRY_PATH),
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
)
if result.returncode:
raise EarthBoundaryBuildError(
result.stderr.strip() or "Earth boundary policy build failed",
code="policy_build_failed",
)
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
manifest["missingTools"] = missing_tools
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
return manifest
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
"build_config": BUILD_CONFIG,
}
)
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
bool(boundary_manifest)
and boundary_manifest.get("buildInputHash") == build_input_hash
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and _policy_verified(boundary_manifest)
and PMTILES_ARTIFACT_PATH.exists()
and _pmtiles_hover_exists(boundary_manifest)
)
def _policy_verified(manifest: dict[str, Any]) -> bool:
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
coverage = manifest.get("povPolicy") or {}
return (
policy.get("profile") == "china-pov-v1"
and coverage.get("profile") == policy["profile"]
and coverage.get("sha256") == policy_hash
and coverage.get("reviewedGeometrySourceCommit")
== geometry.get("sourceCommit")
and coverage.get("geometrySha256") == geometry_hash
and coverage.get("applied") is True
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
<= set(coverage.get("appliedRuleIds") or [])
)
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
paths = [manifest.get("base"), manifest.get("hoverIndex")]
if manifest.get("chinaClaims", {}).get("available"):
paths.append(manifest.get("claimLine"))
for asset in paths:
if not isinstance(asset, str) or Path(asset).name != asset:
return False
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
return False
return True
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
hover = manifest.get("hoverIndex")
return (
isinstance(hover, str)
and Path(hover).name == hover
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
)
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
high_precision_ready = (
manifest_exists
and _policy_verified(boundary_manifest)
and (
(
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and pmtiles_exists
and _pmtiles_hover_exists(boundary_manifest)
)
or (
boundary_manifest.get("tileProvider") == "geojson-high-precision"
and _geojson_artifacts_exist(boundary_manifest)
)
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
)
)
legacy_exists = LEGACY_GEOJSON_PATH.exists()
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
_versioned_asset(POLICY_FALLBACK_PATH)[0]
)
provider = (
boundary_manifest.get("tileProvider")
if high_precision_ready
else "legacy-geojson" if legacy_exists else "missing"
else "policy-geojson-fallback" if legacy_exists else "missing"
)
return {
"provider": provider,
"high_precision_ready": high_precision_ready,
"fallback_available": legacy_exists,
"policy_publishable": bool(
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
),
"config_source": config_source,
"config_path": _relative(LOCAL_CONFIG_PATH),
"config_exists": LOCAL_CONFIG_PATH.exists(),
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
"tileProvider": boundary_manifest.get("tileProvider"),
"buildInputHash": boundary_manifest.get("buildInputHash"),
"builtAt": boundary_manifest.get("builtAt"),
"povPolicy": boundary_manifest.get("povPolicy"),
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
},
"pmtiles": {
"path": _relative(PMTILES_ARTIFACT_PATH),
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
},
"legacy": {
"path": _relative(LEGACY_GEOJSON_PATH),
"path": _relative(POLICY_FALLBACK_PATH),
"exists": legacy_exists,
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
},
"source_manifest": {
"path": _relative(SOURCE_MANIFEST_PATH),
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
build_input_hash,
"--pov-policy",
_relative(POV_POLICY_PATH),
"--pov-geometry",
str(POV_GEOMETRY_PATH),
]
)
if progress_callback:

View File

@@ -22,6 +22,12 @@ from app.services.earth_layer_adapters import (
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.earth_outbox_maintenance import (
configured_retention_days,
purge_consumed_events,
release_maintenance_lock,
try_maintenance_lock,
)
logger = get_logger(__name__)
@@ -30,6 +36,7 @@ DEFAULT_DEBOUNCE_SECONDS = 0.25
DEFAULT_MAX_WAIT_SECONDS = 1.5
DELETE_FAST_FLUSH_SECONDS = 0.05
LISTEN_KEEPALIVE_SECONDS = 5.0
OUTBOX_RETENTION_CHECK_SECONDS = 60.0
OUTBOX_POLL_LIMIT = 5000
MAX_ENTITY_KEY_SAMPLES = 20
MAX_SEEN_EVENT_IDS = 20000
@@ -346,6 +353,7 @@ class EarthDbChangeListener:
self._stop_event: asyncio.Event | None = None
self._connection: asyncpg.Connection | None = None
self._loop: asyncio.AbstractEventLoop | None = None
self._next_retention_check = 0.0
def start(self) -> None:
if self._task and not self._task.done():
@@ -385,6 +393,7 @@ class EarthDbChangeListener:
)
except asyncio.TimeoutError:
await self._poll_outbox()
await self._maybe_purge_outbox()
except asyncio.CancelledError:
raise
except Exception as exc:
@@ -459,6 +468,27 @@ class EarthDbChangeListener:
context={"events": len(rows), "accepted": accepted_count},
)
async def _maybe_purge_outbox(self) -> None:
if self._connection is None or monotonic() < self._next_retention_check:
return
self._next_retention_check = monotonic() + OUTBOX_RETENTION_CHECK_SECONDS
try:
if not await try_maintenance_lock(self._connection):
return
try:
await purge_consumed_events(
self._connection,
retention_days=configured_retention_days(),
)
finally:
await release_maintenance_lock(self._connection)
except Exception as exc:
logger.exception_event(
"Earth outbox retention failed",
event="earth.outbox.retention_failed",
context={"error": str(exc)},
)
def _on_notification(
self,
_connection: asyncpg.Connection,

View File

@@ -99,7 +99,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
sources=frozenset({"media_news_archive", "earth_news_items"}),
tables=frozenset({"earth_news_items"}),
layers=("news",),
cache_patterns=("summary*",),
cache_patterns=(),
refresh_strategy="reload",
),
EarthLayerAdapter(

View File

@@ -2,25 +2,27 @@ from __future__ import annotations
import asyncio
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from contextlib import suppress
from dataclasses import dataclass, replace
import hashlib
import json
import time
from typing import Any
from uuid import uuid4
from fastapi import Response
from fastapi import HTTPException, Response
from app.core.cache import _RedisClient
from app.core.config import settings
from app.core.logging import get_logger
logger = get_logger(__name__, service="earth_layer_cache")
EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
DEFAULT_LOCK_TTL_SECONDS = 10
DEFAULT_LOCK_WAIT_SECONDS = 0.2
DEFAULT_BUILD_WAIT_SECONDS = 30
DEFAULT_MAX_FEATURES = 5000
DEFAULT_MAX_BYTES = 5 * 1024 * 1024
DEFAULT_BBOX_PRECISION_DEGREES = 0.1
@@ -35,6 +37,7 @@ class EarthLayerCachePolicy:
max_bytes: int = DEFAULT_MAX_BYTES
lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
build_wait_seconds: float = DEFAULT_BUILD_WAIT_SECONDS
@dataclass(frozen=True)
@@ -49,6 +52,9 @@ class EarthLayerCacheResult:
class EarthLayerCache:
def __init__(self) -> None:
self._client = None
self._inflight: dict[
tuple[asyncio.AbstractEventLoop, str], asyncio.Future[EarthLayerCacheResult | None]
] = {}
@property
def client(self):
@@ -82,17 +88,62 @@ class EarthLayerCache:
value = json.loads(raw)
return value if isinstance(value, dict) else None
def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
def acquire_lock(self, key: str, ttl_seconds: int) -> str | None:
token = uuid4().hex
return (
token if self.client.set(self.lock_key(key), token, nx=True, ex=ttl_seconds) else None
)
def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
def renew_lock(self, key: str, token: str, ttl_seconds: int) -> bool:
return bool(
self.client.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('expire', KEYS[1], ARGV[2])
end
return 0
""",
1,
self.lock_key(key),
token,
ttl_seconds,
)
)
def release_lock(self, key: str) -> None:
try:
self.client.delete(self.lock_key(key))
except Exception:
pass
def release_lock(self, key: str, token: str) -> None:
self.client.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
end
return 0
""",
1,
self.lock_key(key),
token,
)
def publish(
self, key: str, token: str, payload: dict[str, Any], policy: EarthLayerCachePolicy
) -> bool:
return bool(
self.client.eval(
"""
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
redis.call('setex', KEYS[2], ARGV[2], ARGV[4])
redis.call('setex', KEYS[3], ARGV[3], ARGV[4])
return 1
""",
3,
self.lock_key(key),
key,
self.stale_key(key),
token,
policy.fresh_ttl_seconds,
policy.stale_ttl_seconds,
json.dumps(payload, ensure_ascii=False, default=str),
)
)
def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
keys = list(self.client.scan_iter(match=pattern))
@@ -169,56 +220,122 @@ async def resolve_layer_payload(
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
) -> EarthLayerCacheResult:
started = time.perf_counter()
try:
cached = earth_layer_cache.get_json(key)
if cached is not None:
return _result(cached, state="hit", key=key)
lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
if lock_acquired:
try:
payload = await _build_budgeted_payload(builder, policy)
_write_fresh_and_stale(key, payload, policy)
_log_cache_event("refresh", key, payload, started)
return _result(payload, state="refresh", key=key)
except Exception as exc:
stale = _read_stale(key)
if stale is not None:
logger.warning_event(
"Earth layer cache builder failed; returning stale payload",
event="earth_layer_cache.stale_after_builder_error",
context={"key": key, "error": str(exc)},
)
return _result(stale, state="stale", key=key)
raise
finally:
earth_layer_cache.release_lock(key)
flight_key = (asyncio.get_running_loop(), key)
while flight_key in earth_layer_cache._inflight:
stale = _read_stale(key)
if stale is not None:
return _result(stale, state="stale", key=key)
future = earth_layer_cache._inflight[flight_key]
done, _ = await asyncio.wait({future}, timeout=policy.build_wait_seconds)
if not done:
raise HTTPException(503, "Earth layer cache rebuild is still in progress")
result = future.result()
if result is not None:
state = "hit" if result.state == "refresh" else result.state
return replace(result, state=state)
await asyncio.sleep(policy.lock_wait_seconds)
cached_after_wait = earth_layer_cache.get_json(key)
if cached_after_wait is not None:
return _result(cached_after_wait, state="hit", key=key)
payload = await _build_budgeted_payload(builder, policy)
_log_cache_event("miss", key, payload, started)
return _result(payload, state="miss", key=key)
future = asyncio.get_running_loop().create_future()
earth_layer_cache._inflight[flight_key] = future
try:
result = await _resolve_layer_payload(key=key, policy=policy, builder=builder)
future.set_result(result)
return result
except asyncio.CancelledError:
# A waiting request can take over using its own still-open DB session.
future.set_result(None)
raise
except Exception as exc:
future.set_exception(exc)
future.exception()
raise
finally:
earth_layer_cache._inflight.pop(flight_key)
async def _resolve_layer_payload(
*,
key: str,
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
) -> EarthLayerCacheResult:
started = time.perf_counter()
try:
async with asyncio.timeout(policy.build_wait_seconds):
while True:
cached = earth_layer_cache.get_json(key)
if cached is not None:
return _result(cached, state="hit", key=key)
token = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
if token is not None:
break
stale = _read_stale(key)
if stale is not None:
return _result(stale, state="stale", key=key)
await asyncio.sleep(policy.lock_wait_seconds)
except TimeoutError:
raise HTTPException(503, "Earth layer cache rebuild is still in progress") from None
except Exception as exc:
try:
payload = await _build_budgeted_payload(builder, policy)
except Exception:
raise exc
logger.warning_event(
"Earth layer cache bypassed",
event="earth_layer_cache.bypass",
context={"key": key, "error": str(exc)},
)
payload = await _build_budgeted_payload(builder, policy)
return _result(payload, state="bypass", key=key)
renewal = asyncio.create_task(_renew_cache_lock(key, token, policy))
try:
try:
payload = await _build_budgeted_payload(builder, policy)
except Exception as exc:
stale = _read_stale(key)
if stale is None:
raise
logger.warning_event(
"Earth layer cache builder failed; returning stale payload",
event="earth_layer_cache.stale_after_builder_error",
context={"key": key, "error": str(exc)},
)
return _result(stale, state="stale", key=key)
try:
published = earth_layer_cache.publish(key, token, payload, policy)
except Exception as exc:
logger.warning_event(
"Earth layer cache publish failed",
event="earth_layer_cache.bypass",
context={"key": key, "error": str(exc)},
)
published = False
state = "refresh" if published else "bypass"
_log_cache_event(state, key, payload, started)
return _result(payload, state=state, key=key)
finally:
renewal.cancel()
with suppress(asyncio.CancelledError):
await renewal
try:
earth_layer_cache.release_lock(key, token)
except Exception as exc:
logger.warning_event(
"Earth layer cache lock release failed",
event="earth_layer_cache.lock_release_failed",
context={"key": key, "error": str(exc)},
)
async def _renew_cache_lock(key: str, token: str, policy: EarthLayerCachePolicy) -> None:
try:
while True:
await asyncio.sleep(policy.lock_ttl_seconds / 3)
if not earth_layer_cache.renew_lock(key, token, policy.lock_ttl_seconds):
raise RuntimeError("Earth layer cache lock ownership lost")
except Exception as exc:
logger.warning_event(
"Earth layer cache lock renewal stopped",
event="earth_layer_cache.lock_renewal_failed",
context={"key": key, "error": str(exc)},
)
def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
@@ -286,11 +403,6 @@ async def _build_budgeted_payload(
return apply_payload_budget(payload, policy)
def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
def _read_stale(key: str) -> dict[str, Any] | None:
try:
return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
@@ -359,14 +471,16 @@ def _payload_size(payload: dict[str, Any]) -> int:
def _safe_key_part(value: Any) -> str:
raw = str(value).strip().lower()
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[
:160
]
def _layer_from_key(key: str) -> str:
prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
if not key.startswith(prefix):
return "unknown"
remainder = key[len(prefix):]
remainder = key[len(prefix) :]
return remainder.split(":", 1)[0]

View File

@@ -1,8 +1,9 @@
from __future__ import annotations
import asyncio
import codecs
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
import hashlib
import html
@@ -1638,10 +1639,15 @@ async def record_earth_news_sources_health(
await db.commit()
async def get_configured_sources_for_region(db: AsyncSession | None, region: str) -> list[NewsFeedSource]:
async def get_configured_sources_for_region(
db: AsyncSession | None,
region: str,
*,
config_payload: dict[str, Any] | None = None,
) -> list[NewsFeedSource]:
if db is None:
return get_sources_for_region(region)
payload = await get_earth_news_sources_payload(db)
payload = config_payload or await get_earth_news_sources_payload(db)
sources = [
source
for raw in payload.get("sources", [])
@@ -1800,6 +1806,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
return ""
def _decode_feed_response(response: httpx.Response) -> str:
content = response.content
for marker, encoding in (
(codecs.BOM_UTF32_LE, "utf-32"),
(codecs.BOM_UTF32_BE, "utf-32"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
(codecs.BOM_UTF8, "utf-8-sig"),
):
if content.startswith(marker):
return content.decode(encoding)
declaration = re.match(
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
content,
re.IGNORECASE,
)
encoding = response.charset_encoding or (
declaration.group(1).decode("ascii") if declaration else "utf-8"
)
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
return content.decode(encoding)
def _parse_feed_entries(
xml_text: str,
source: NewsFeedSource,
@@ -2453,7 +2483,9 @@ async def _fetch_single_feed_url(
feed=feed,
)
try:
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
items = _parse_feed_entries(
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
)
except Exception as exc:
error, status = _format_source_fetch_error(
status_code=status_code,
@@ -2461,7 +2493,7 @@ async def _fetch_single_feed_url(
body=response.text,
parsed_count=0,
)
if error is None:
if status == "empty":
error = str(exc)
status = "format_error"
return source, [], error, _source_health_result(
@@ -2704,7 +2736,6 @@ async def get_earth_news_payload(
del provider_client
active_region = region if region in REGION_ANCHORS else determine_focus_region(lat, lon)
has_settings_db = db is not None and callable(getattr(db, "execute", None))
source_config_payload = await get_earth_news_sources_payload(db) if has_settings_db else None
sources = (
await get_configured_sources_for_region(db, active_region)
if has_settings_db
@@ -2723,39 +2754,7 @@ async def get_earth_news_payload(
locale=locale,
)
from app.services.earth_news_store import (
get_earth_news_feed_coverage,
get_earth_news_freshness,
list_earth_news_cruise_items,
list_earth_news_items,
upsert_earth_news_items,
)
errors: list[str] = []
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at)
if not should_supplement and callable(getattr(db, "execute", None)):
expected_feed_keys = _expected_feed_keys(sources, active_region=active_region)
if expected_feed_keys:
covered_feed_keys = await get_earth_news_feed_coverage(
db,
active_region=active_region,
recent_after=datetime.now(UTC) - timedelta(seconds=RSS_SUPPLEMENT_MAX_AGE_SECONDS),
)
should_supplement = bool(expected_feed_keys - covered_feed_keys)
if should_supplement:
if source_config_payload is None:
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region)
else:
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(
sources,
active_region=active_region,
config_payload=source_config_payload,
)
await record_earth_news_sources_health(db, health_by_source)
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
await _enqueue_unverified_locations(ranked_fetched_items)
await upsert_earth_news_items(db, ranked_fetched_items)
from app.services.earth_news_store import list_earth_news_cruise_items, list_earth_news_items
items = await _call_store_list_items(
list_earth_news_items,
@@ -2765,7 +2764,7 @@ async def get_earth_news_payload(
categories=categories,
source_ids=source_ids,
)
if not source_ids:
if items and not source_ids:
ready_sources = {
_news_item_source_id(item)
for item in items
@@ -2806,10 +2805,6 @@ async def get_earth_news_payload(
)
else:
cruise_items = items
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
stale = bool(errors and items)
return _build_payload(
lat=lat,
lon=lon,
@@ -2817,8 +2812,8 @@ async def get_earth_news_payload(
items=items,
cruise_items=cruise_items,
sources=sources,
errors=errors,
stale=stale,
errors=[],
stale=False,
categories=categories,
source_ids=source_ids,
limit=limit,

View File

@@ -0,0 +1,480 @@
"""Periodic RSS supplementation for the read-only Earth news feed."""
from __future__ import annotations
import asyncio
from contextlib import suppress
from dataclasses import replace
from datetime import UTC, datetime, timedelta
from hashlib import sha256
import json
from typing import Any
from uuid import uuid4
from app.core.cache import _RedisClient
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.services import earth_news
from app.services.earth_news_store import (
get_earth_news_feed_coverage,
get_earth_news_freshness,
upsert_earth_news_items,
)
logger = get_logger(__name__, service="earth_news")
REFRESH_INTERVAL_SECONDS = 10 * 60
REFRESH_LOCK_TTL_SECONDS = 120
REFRESH_LOCK_RENEW_SECONDS = 30
SOURCE_FAILURE_COOLDOWN_SECONDS = 30 * 60
REFRESH_LOCK_PREFIX = "earth:news:rss-refresh:v1"
SOURCE_COOLDOWN_PREFIX = "earth:news:rss-cooldown:v1"
REFRESH_STATUS_PREFIX = "earth:news:rss-refresh-status:v1"
REFRESH_ACTIVE_PREFIX = "earth:news:rss-refresh-active:v1"
REFRESH_STATUS_TTL_SECONDS = 10 * 60
REFRESH_ACTIVE_TTL_SECONDS = 120
REFRESH_STATUS_HEARTBEAT_SECONDS = 30
def _refresh_status_key(refresh_id: str) -> str:
return f"{REFRESH_STATUS_PREFIX}:{refresh_id}"
def _refresh_active_key(region: str) -> str:
return f"{REFRESH_ACTIVE_PREFIX}:{region}"
def create_earth_news_refresh_status(region: str) -> tuple[dict[str, Any], bool]:
"""Create one per-region active refresh status, or return the current one."""
refresh_id = uuid4().hex
status = {"refresh_id": refresh_id, "region": region, "status": "pending"}
script = """
local active_id = redis.call('get', KEYS[1])
if active_id then
local active_status = redis.call('get', ARGV[5] .. active_id)
if active_status then
local decoded = cjson.decode(active_status)
if decoded.status == 'pending' or decoded.status == 'running' then
return {active_id, active_status, 0}
end
end
redis.call('del', KEYS[1])
end
redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[3])
redis.call('set', KEYS[2], ARGV[2], 'EX', ARGV[4])
return {ARGV[1], ARGV[2], 1}
"""
raw = _RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
refresh_id,
json.dumps(status, separators=(",", ":")),
REFRESH_ACTIVE_TTL_SECONDS,
REFRESH_STATUS_TTL_SECONDS,
f"{REFRESH_STATUS_PREFIX}:",
)
_, encoded, created = raw
payload = json.loads(encoded)
return payload, bool(int(created))
def set_earth_news_refresh_status(
refresh_id: str,
region: str,
status: str,
*,
error: dict[str, str] | None = None,
) -> dict[str, Any]:
payload: dict[str, Any] = {
"refresh_id": refresh_id,
"region": region,
"status": status,
}
if error:
payload["error"] = error
script = """
if redis.call('get', KEYS[1]) ~= ARGV[2] then return 0 end
redis.call('set', KEYS[2], ARGV[1], 'EX', ARGV[3])
if ARGV[4] == 'succeeded' or ARGV[4] == 'failed' then
redis.call('del', KEYS[1])
else
redis.call('expire', KEYS[1], ARGV[5])
end
return 1
"""
_RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
json.dumps(payload, separators=(",", ":")),
refresh_id,
REFRESH_STATUS_TTL_SECONDS,
status,
REFRESH_ACTIVE_TTL_SECONDS,
)
return payload
def touch_earth_news_refresh_status(refresh_id: str, region: str) -> bool:
script = """
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
redis.call('expire', KEYS[1], ARGV[2])
redis.call('expire', KEYS[2], ARGV[3])
return 1
"""
return bool(
_RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
refresh_id,
REFRESH_ACTIVE_TTL_SECONDS,
REFRESH_STATUS_TTL_SECONDS,
)
)
def get_earth_news_refresh_status(refresh_id: str) -> dict[str, Any] | None:
raw = _RedisClient.get_client().get(_refresh_status_key(refresh_id))
if not raw:
return None
value = json.loads(raw)
if not isinstance(value, dict):
return None
if value.get("status") in {"pending", "running"}:
region = str(value.get("region") or "")
active_id = _RedisClient.get_client().get(_refresh_active_key(region)) if region else None
if active_id != refresh_id:
return {
**value,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
return value
def _lock_key(region: str) -> str:
return f"{REFRESH_LOCK_PREFIX}:{region}"
def _cooldown_key(region: str, source_id: str, feed_id: str) -> str:
identity = f"{region}\0{source_id}\0{feed_id}".encode("utf-8")
return f"{SOURCE_COOLDOWN_PREFIX}:{sha256(identity).hexdigest()}"
def _source_failure_cooldown_seconds(source: earth_news.NewsFeedSource | None) -> int:
if source is None:
return SOURCE_FAILURE_COOLDOWN_SECONDS
configured_minutes = source.health_policy.get("cooldown_minutes")
try:
return max(60, int(float(configured_minutes) * 60))
except TypeError, ValueError:
return SOURCE_FAILURE_COOLDOWN_SECONDS
def _select_feeds(
sources: list[earth_news.NewsFeedSource],
*,
active_region: str,
required_keys: set[tuple[str, str]],
redis_client: Any | None,
) -> tuple[list[earth_news.NewsFeedSource], set[tuple[str, str]]]:
selected_sources: list[earth_news.NewsFeedSource] = []
cooling_keys: set[tuple[str, str]] = set()
for source in sources:
selected_feeds = []
source_cooling = False
if redis_client is not None:
try:
source_cooling = bool(
redis_client.exists(_cooldown_key(active_region, source.id, "__source__"))
)
except Exception as exc:
logger.warning_event(
"Earth news source cooldown lookup failed",
event="earth_news.rss.cooldown_lookup_failed",
context={"region": active_region, "source_id": source.id, "error": str(exc)},
)
for feed in earth_news._source_feeds(source):
feed_key = (source.id, feed.id)
if feed_key not in required_keys:
continue
if source_cooling:
cooling_keys.add(feed_key)
continue
if redis_client is not None:
try:
if redis_client.exists(_cooldown_key(active_region, *feed_key)):
cooling_keys.add(feed_key)
continue
except Exception as exc:
logger.warning_event(
"Earth news source cooldown lookup failed",
event="earth_news.rss.cooldown_lookup_failed",
context={
"region": active_region,
"source_id": source.id,
"error": str(exc),
},
)
selected_feeds.append(feed)
if selected_feeds:
selected_sources.append(
replace(
source,
feed_url=selected_feeds[0].url,
feed_urls=tuple(feed.url for feed in selected_feeds),
feeds=tuple(selected_feeds),
)
)
return selected_sources, cooling_keys
def _record_failed_feed_cooldowns(
redis_client: Any | None,
*,
active_region: str,
health_by_source: dict[str, dict[str, Any]],
sources: list[earth_news.NewsFeedSource],
) -> None:
if redis_client is None:
return
cooldown_by_source = {source.id: source for source in sources}
for source_id, health in health_by_source.items():
feed_results = health.get("feed_results")
if isinstance(feed_results, list) and feed_results:
failed_feed_ids = [
str(result.get("feed_id") or "")
for result in feed_results
if isinstance(result, dict) and not result.get("ok") and result.get("feed_id")
]
else:
failed_feed_ids = [str(health.get("feed_id") or "__source__")]
if health.get("ok"):
failed_feed_ids = []
for feed_id in failed_feed_ids:
try:
redis_client.set(
_cooldown_key(active_region, source_id, feed_id),
"1",
ex=_source_failure_cooldown_seconds(cooldown_by_source.get(source_id)),
)
except Exception as exc:
logger.warning_event(
"Earth news source failure cooldown write failed",
event="earth_news.rss.cooldown_write_failed",
context={"region": active_region, "source_id": source_id, "error": str(exc)},
)
async def _renew_lock(redis_client: Any, key: str, token: str, lease_lost: asyncio.Event) -> None:
renew_script = """
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('expire', KEYS[1], ARGV[2])
end
return 0
"""
try:
while True:
await asyncio.sleep(REFRESH_LOCK_RENEW_SECONDS)
if not redis_client.eval(renew_script, 1, key, token, REFRESH_LOCK_TTL_SECONDS):
lease_lost.set()
return
except asyncio.CancelledError:
raise
except Exception as exc:
lease_lost.set()
logger.warning_event(
"Earth news refresh lock renewal failed",
event="earth_news.rss.refresh_lock_renewal_failed",
context={"error": str(exc)},
)
async def refresh_earth_news_region(active_region: str, *, force: bool = False) -> dict[str, Any]:
"""Fetch only stale or uncovered feeds for one region, outside the request path."""
if active_region not in earth_news.REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {active_region}")
lock_key = _lock_key(active_region)
token = uuid4().hex
redis_client = None
lock_acquired = False
lock_available = False
try:
redis_client = _RedisClient.get_client()
lock_acquired = bool(
redis_client.set(lock_key, token, nx=True, ex=REFRESH_LOCK_TTL_SECONDS)
)
lock_available = True
except Exception as exc:
logger.warning_event(
"Earth news refresh coordination unavailable",
event="earth_news.rss.refresh_lock_unavailable",
context={"region": active_region, "error": str(exc)},
)
if not lock_available or not lock_acquired:
state = "busy" if lock_available else "coordination_unavailable"
return {"state": state, "fetched": 0, "upserted": 0}
lease_lost = asyncio.Event()
renewal: asyncio.Task[None] | None = None
if lock_acquired and redis_client is not None:
renewal = asyncio.create_task(_renew_lock(redis_client, lock_key, token, lease_lost))
try:
async with async_session_factory() as db:
source_config = await earth_news.get_earth_news_sources_payload(db)
sources = await earth_news.get_configured_sources_for_region(
db, active_region, config_payload=source_config
)
if not sources:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
expected_keys = earth_news._expected_feed_keys(sources, active_region=active_region)
if not expected_keys:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
now = datetime.now(UTC)
needs_full_refresh = force or earth_news._needs_rss_supplement(
item_count=item_count, newest_at=newest_at
)
covered_keys = await get_earth_news_feed_coverage(
db,
active_region=active_region,
recent_after=now - timedelta(seconds=earth_news.RSS_SUPPLEMENT_MAX_AGE_SECONDS),
)
required_keys = expected_keys if needs_full_refresh else expected_keys - covered_keys
if not required_keys:
return {
"state": "completed",
"successful_sources": 0,
"failed_sources": 0,
"fetched": 0,
"upserted": 0,
}
selected_sources, cooling_keys = _select_feeds(
sources,
active_region=active_region,
required_keys=required_keys,
redis_client=redis_client,
)
if cooling_keys:
logger.info_event(
"Skipping failed Earth news feeds during cooldown",
event="earth_news.rss.cooldown_active",
context={"region": active_region, "feed_count": len(cooling_keys)},
)
if not selected_sources:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
fetched_items, _errors, health_by_source = (
await earth_news._fetch_rss_items_for_sources(
selected_sources,
active_region=active_region,
config_payload=source_config,
)
)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": 0, "upserted": 0}
_record_failed_feed_cooldowns(
redis_client,
active_region=active_region,
health_by_source=health_by_source,
sources=selected_sources,
)
await earth_news.record_earth_news_sources_health(db, health_by_source)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": 0, "upserted": 0}
ranked_items = earth_news._rank_and_trim_items(
fetched_items, active_region=active_region
)
await earth_news._enqueue_unverified_locations(ranked_items)
upserted = await upsert_earth_news_items(db, ranked_items)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": len(fetched_items), "upserted": 0}
await db.commit()
successful_sources = sum(bool(health.get("ok")) for health in health_by_source.values())
failed_sources = max(0, len(selected_sources) - successful_sources)
state = "completed" if successful_sources else "failed"
failed_statuses = {
str(health.get("status") or "")
for health in health_by_source.values()
if not health.get("ok")
}
error_code = None
if not successful_sources:
error_code = "timeout" if failed_statuses == {"timeout"} else "source_unavailable"
return {
"state": state,
"error_code": error_code,
"successful_sources": successful_sources,
"failed_sources": failed_sources,
"fetched": len(fetched_items),
"upserted": upserted,
}
except Exception as exc:
logger.warning_event(
"Earth news periodic RSS refresh failed",
event="earth_news.rss.refresh_failed",
context={"region": active_region, "error": str(exc)},
)
raise
finally:
if renewal is not None:
renewal.cancel()
with suppress(asyncio.CancelledError):
await renewal
if lock_acquired and redis_client is not None:
release_script = """
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
end
return 0
"""
try:
redis_client.eval(release_script, 1, lock_key, token)
except Exception as exc:
logger.warning_event(
"Earth news refresh lock release failed",
event="earth_news.rss.refresh_lock_release_failed",
context={"region": active_region, "error": str(exc)},
)
async def refresh_earth_news_all_regions() -> None:
for active_region in earth_news.REGION_ANCHORS:
try:
await refresh_earth_news_region(active_region)
except asyncio.CancelledError:
raise
except Exception:
continue

View File

@@ -278,22 +278,25 @@ async def get_earth_news_feed_coverage(
active_region: str,
recent_after: datetime | None = None,
) -> set[tuple[str, str]]:
query = select(
EarthNewsItem.id,
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
source_id = func.coalesce(
func.nullif(news_meta.op("->>")("source_id"), ""),
func.split_part(EarthNewsItem.id, ":", 1),
)
feed_id = func.nullif(news_meta.op("->>")("feed_id"), "")
query = select(
source_id,
feed_id,
).where(source_id != "", feed_id.is_not(None)).distinct()
if active_region != "global":
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
if recent_after is not None:
query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
result = await db.execute(query)
coverage: set[tuple[str, str]] = set()
for item_id, source_id, feed_id in result.all():
for source_id, feed_id in result.all():
normalized_source_id = str(source_id or "").strip()
normalized_feed_id = str(feed_id or "").strip()
if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
normalized_source_id = item_id.split(":", 1)[0]
if normalized_source_id and normalized_feed_id:
coverage.add((normalized_source_id, normalized_feed_id))
return coverage
@@ -340,30 +343,49 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
changed += 1
continue
record.title = item.title
record.summary = item.summary
record.url = item.url
record.source = item.source
record.feed_name = item.feed_name
record.region = item.feed_region
record.homepage_url = item.homepage_url
record.published_at = item.published_at
visible_changed = False
for field_name, value in (
("title", item.title),
("summary", item.summary),
("url", item.url),
("source", item.source),
("feed_name", item.feed_name),
("region", item.feed_region),
("homepage_url", item.homepage_url),
("published_at", item.published_at),
):
if getattr(record, field_name) != value:
setattr(record, field_name, value)
visible_changed = True
record.last_seen_at = now
location_meta = dict(record.location_meta or {})
location_meta["news_meta"] = _news_meta_patch(item)
record.location_meta = location_meta
news_meta = _news_meta_patch(item)
if location_meta.get("news_meta") != news_meta:
location_meta["news_meta"] = news_meta
record.location_meta = location_meta
visible_changed = True
if item.localizations:
merged_localizations = {
**dict(record.localizations or {}),
**dict(item.localizations or {}),
}
record.content_language = item.content_language
record.localizations = merged_localizations
if record.content_language != item.content_language:
record.content_language = item.content_language
visible_changed = True
if dict(record.localizations or {}) != merged_localizations:
record.localizations = merged_localizations
visible_changed = True
if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
record.enrichment_status = item.enrichment_status
record.enrichment_error = item.enrichment_error
record.enriched_at = item.enriched_at
changed += 1
for field_name, value in (
("enrichment_status", item.enrichment_status),
("enrichment_error", item.enrichment_error),
("enriched_at", item.enriched_at),
):
if getattr(record, field_name) != value:
setattr(record, field_name, value)
visible_changed = True
if visible_changed:
changed += 1
await db.flush()
return changed

View File

@@ -0,0 +1,184 @@
"""Bounded retention for consumed Earth change events."""
from __future__ import annotations
import asyncio
import os
from datetime import UTC, datetime, timedelta
from typing import AsyncContextManager, Protocol
from app.core.logging import get_logger
logger = get_logger(__name__)
INDEX_NAME = "idx_earth_data_change_events_consumed_retention"
DEFAULT_RETENTION_DAYS = 1
DEFAULT_BATCH_SIZE = 1000
DEFAULT_PAUSE_SECONDS = 0.1
MAINTENANCE_LOCK_KEY = (17001613, 1)
DELETE_BATCH_SQL = """
WITH doomed AS (
SELECT id
FROM earth_data_change_events
WHERE consumed_at IS NOT NULL AND consumed_at < $1
ORDER BY consumed_at, id
LIMIT $2
FOR UPDATE SKIP LOCKED
), deleted AS (
DELETE FROM earth_data_change_events AS events
USING doomed
WHERE events.id = doomed.id
RETURNING events.id
)
SELECT count(*) FROM deleted
"""
class OutboxConnection(Protocol):
async def fetchval(self, query: str, *args: object) -> object: ...
async def fetchrow(self, query: str, *args: object) -> object: ...
async def execute(self, query: str, *args: object) -> object: ...
def transaction(self) -> AsyncContextManager[object]: ...
def configured_retention_days() -> int:
raw = os.getenv("PLANET_EARTH_OUTBOX_RETENTION_DAYS", str(DEFAULT_RETENTION_DAYS))
try:
days = int(raw)
if 1 <= days <= 3650:
return days
except ValueError:
pass
logger.warning_event(
"Invalid Earth outbox retention; using default",
event="earth.outbox.retention_invalid",
context={"value": raw},
)
return DEFAULT_RETENTION_DAYS
async def try_maintenance_lock(conn: OutboxConnection) -> bool:
return bool(await conn.fetchval("SELECT pg_try_advisory_lock($1, $2)", *MAINTENANCE_LOCK_KEY))
async def release_maintenance_lock(conn: OutboxConnection) -> None:
await conn.fetchval("SELECT pg_advisory_unlock($1, $2)", *MAINTENANCE_LOCK_KEY)
async def retention_index_state(conn: OutboxConnection) -> str:
row = await conn.fetchrow(
"""
SELECT indexes.indisvalid, indexes.indisready
FROM pg_index AS indexes
JOIN pg_class AS index_table ON index_table.oid = indexes.indexrelid
WHERE indexes.indrelid = 'earth_data_change_events'::regclass
AND index_table.relname = $1
""",
INDEX_NAME,
)
if row is None:
return "missing"
return "ready" if row["indisvalid"] and row["indisready"] else "invalid"
async def ensure_retention_index(conn: OutboxConnection) -> None:
"""Call on an autocommit connection, never inside app init_db's transaction."""
state = await retention_index_state(conn)
if state == "invalid":
await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}")
if state != "ready":
await conn.execute(
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {INDEX_NAME} "
"ON earth_data_change_events (consumed_at, id) WHERE consumed_at IS NOT NULL"
)
if await retention_index_state(conn) != "ready":
raise RuntimeError("Earth outbox retention index is not valid and ready")
async def purge_consumed_events(
conn: OutboxConnection,
*,
retention_days: int,
batch_size: int = DEFAULT_BATCH_SIZE,
max_batches: int = 1,
pause_seconds: float = DEFAULT_PAUSE_SECONDS,
) -> int:
if not 1 <= retention_days <= 3650:
raise ValueError("retention_days must be between 1 and 3650")
if batch_size < 1 or max_batches < 1 or pause_seconds < 0:
raise ValueError("batch_size/max_batches must be positive and pause_seconds non-negative")
if await retention_index_state(conn) != "ready":
logger.warning_event(
"Skipping Earth outbox retention because its index is unavailable",
event="earth.outbox.retention_index_unavailable",
)
return 0
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
deleted_total = 0
for batch_number in range(max_batches):
deleted = int(await conn.fetchval(DELETE_BATCH_SQL, cutoff, batch_size) or 0)
deleted_total += deleted
if deleted < batch_size:
break
if batch_number + 1 < max_batches and pause_seconds:
await asyncio.sleep(pause_seconds)
if deleted_total:
logger.info_event(
"Purged consumed Earth outbox events",
event="earth.outbox.retention_purged",
context={"deleted": deleted_total, "retention_days": retention_days},
)
return deleted_total
async def compact_consumed_events(
conn: OutboxConnection,
*,
retention_days: int,
lock_timeout_seconds: int = 10,
statement_timeout_seconds: int = 900,
) -> tuple[int, int]:
"""One-time atomic rewrite; blocks table access until the transaction commits."""
if not 1 <= retention_days <= 3650:
raise ValueError("retention_days must be between 1 and 3650")
if lock_timeout_seconds < 1 or statement_timeout_seconds < 1:
raise ValueError("timeouts must be positive")
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
async with conn.transaction():
await conn.fetchval(
"SELECT set_config('lock_timeout', $1, true)", f"{lock_timeout_seconds}s"
)
await conn.fetchval(
"SELECT set_config('statement_timeout', $1, true)", f"{statement_timeout_seconds}s"
)
await conn.execute("LOCK TABLE earth_data_change_events IN ACCESS EXCLUSIVE MODE")
await conn.execute(
"""
CREATE TEMP TABLE earth_outbox_retained ON COMMIT DROP AS
SELECT * FROM earth_data_change_events
WHERE consumed_at IS NULL OR consumed_at >= $1
""",
cutoff,
)
retained = await conn.fetchrow("""
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
FROM earth_outbox_retained
""")
if retained is None:
raise RuntimeError("Unable to verify retained Earth outbox rows")
await conn.execute("TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT")
await conn.execute(
"INSERT INTO earth_data_change_events SELECT * FROM earth_outbox_retained"
)
restored = await conn.fetchrow("""
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
FROM earth_data_change_events
""")
if restored is None or dict(retained) != dict(restored):
raise RuntimeError("Earth outbox compact verification failed; transaction rolled back")
return int(retained["total"]), int(retained["pending"])

View File

@@ -0,0 +1,198 @@
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
from __future__ import annotations
from dataclasses import dataclass
import math
import re
from typing import Any
from app.core.countries import normalize_country
_COUNTRY_LABEL_OVERRIDES = {
"korea republic of": "韩国",
}
_REGION_ONLY_LABELS = {
"europe",
"iowa",
"maryland",
"new mexico",
"texas",
"the nordics",
"virginia",
"washington",
}
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
re.IGNORECASE,
)
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
re.IGNORECASE | re.DOTALL,
)
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
)
_NUMBERED_STREET_PATTERN = re.compile(
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
r"highway|hwy|way|parkway|pkwy|chome)\b|"
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
re.IGNORECASE,
)
_POSTAL_CODE_PATTERN = re.compile(
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
re.IGNORECASE,
)
_SITE_CONTEXT_PATTERN = re.compile(
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
re.IGNORECASE,
)
_COORDINATE_CUE_PATTERN = re.compile(
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
re.IGNORECASE | re.DOTALL,
)
@dataclass(frozen=True)
class EpochLocationAssessment:
precision: str
reason: str
@property
def is_renderable(self) -> bool:
return self.precision in {"precise", "site", "city"}
def _normalized_label(value: Any) -> str:
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
def _country_from_label(value: Any) -> str | None:
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
_normalized_label(value)
)
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
if isinstance(raw, bool):
return None
try:
number = float(raw)
except (TypeError, ValueError):
return None
return number if math.isfinite(number) and abs(number) <= bound else None
def _has_explicit_coordinate_pair(text: str) -> bool:
hemispheres = {}
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
hemisphere = match.group("hemisphere").upper()
degrees = int(match.group("degrees"))
minutes = int(match.group("minutes"))
seconds = float(match.group("seconds") or 0)
bound = 90 if hemisphere in {"N", "S"} else 180
if degrees <= bound and minutes < 60 and seconds < 60:
hemispheres[hemisphere] = degrees
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
return True
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
if labeled_pair:
latitude = float(labeled_pair.group("latitude"))
longitude = float(labeled_pair.group("longitude"))
return abs(latitude) <= 90 and abs(longitude) <= 180
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
if not decimal_pair:
return False
latitude, longitude = map(float, decimal_pair.groups())
return abs(latitude) <= 90 and abs(longitude) <= 180
def _has_address_evidence(text: str) -> bool:
if _NUMBERED_STREET_PATTERN.search(text):
return True
return bool(
_POSTAL_CODE_PATTERN.search(text)
and ("\n" in text or text.count(",") >= 2)
and _SITE_CONTEXT_PATTERN.search(text)
)
def assess_epoch_location(
location: Any,
*,
country: Any = None,
raw_country: Any = None,
) -> EpochLocationAssessment:
"""Classify source coordinates without upgrading their source text.
Epoch's CSV supplies coordinates even when ``Location`` contains only a
country. Those coordinates are useful source evidence, but are not a
facility point and must not enter the renderable location pipeline.
"""
if isinstance(location, dict):
latitude = _coordinate(location, ("lat", "latitude"), 90)
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
has_coordinates = latitude is not None and longitude is not None
return EpochLocationAssessment(
"precise" if has_coordinates else "unknown",
"complete explicit coordinate object"
if has_coordinates
else "coordinate object lacks a complete valid latitude/longitude pair",
)
text = str(location or "").strip()
if not text:
return EpochLocationAssessment("unknown", "Epoch Location is empty")
normalized = _normalized_label(text)
normalized_country = normalize_country(country)
source_country = normalize_country(raw_country)
location_country = _country_from_label(text)
if location_country or (
raw_country and normalized == _normalized_label(raw_country)
):
matched = location_country or source_country or normalized_country
return EpochLocationAssessment(
"country",
f"Epoch Location identifies country only: {matched or text}",
)
if normalized in _REGION_ONLY_LABELS:
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
if (
len(comma_parts) == 2
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
and _country_from_label(comma_parts[1])
):
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
if normalized in _UNLOCATABLE_LABELS:
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
if normalized.isdigit():
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
if _has_explicit_coordinate_pair(text):
return EpochLocationAssessment(
"precise", "Epoch Location contains a complete valid coordinate pair"
)
if _COORDINATE_CUE_PATTERN.search(text):
return EpochLocationAssessment(
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
)
if _has_address_evidence(text):
return EpochLocationAssessment("precise", "Epoch Location contains an address")
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
return EpochLocationAssessment("city", "Epoch Location identifies a locality")

View File

@@ -2,9 +2,11 @@
import asyncio
from datetime import UTC, datetime, timedelta
import time
from typing import Any, Dict, Optional
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.date import DateTrigger
from apscheduler.triggers.interval import IntervalTrigger
from sqlalchemy import select
@@ -14,19 +16,44 @@ from app.db.session import async_session_factory
from app.core.time import to_iso8601_utc
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
from app.services.business_logs import (
emit_business_log,
emit_business_log_background,
exception_context,
)
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
get_builtin_effective_candidate,
save_connectivity_success,
)
from app.services.earth_news_refresh import (
REFRESH_INTERVAL_SECONDS as EARTH_NEWS_REFRESH_INTERVAL_SECONDS,
create_earth_news_refresh_status,
refresh_earth_news_all_regions,
refresh_earth_news_region,
set_earth_news_refresh_status,
touch_earth_news_refresh_status,
REFRESH_STATUS_HEARTBEAT_SECONDS as EARTH_NEWS_STATUS_HEARTBEAT_SECONDS,
)
from app.services.earth_news import REGION_ANCHORS
logger = get_logger(__name__)
scheduler = AsyncIOScheduler()
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
EARTH_NEWS_REFRESH_JOB_ID = "earth_news_rss_refresh"
_pending_earth_news_refreshes: dict[str, dict[str, Any]] = {}
_running_earth_news_refreshes: set[str] = set()
_rerun_earth_news_refreshes: dict[str, dict[str, Any]] = {}
EARTH_NEWS_REFRESH_WAIT_SECONDS = 300
EARTH_NEWS_REFRESH_POLL_SECONDS = 1
EARTH_NEWS_REFRESH_PUBLIC_ERRORS = {
"source_unavailable": "No news source could be refreshed for this region. Please retry.",
"timeout": "News sources timed out. Please try again.",
"refresh_failed": "Could not refresh news. Please try again.",
}
def _collector_task_name(collector_name: str) -> str:
@@ -86,7 +113,10 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
logger.info_event(
"Scheduled collector",
event="collector.schedule.updated",
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
context={
"collector_name": datasource.source,
"frequency_minutes": datasource.frequency_minutes,
},
)
else:
logger.info_event(
@@ -133,7 +163,11 @@ async def run_collector_task(collector_name: str):
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "skipped",
},
)
return
@@ -153,9 +187,8 @@ async def run_collector_task(collector_name: str):
if started_at is not None and started_at.tzinfo is None:
started_at = started_at.replace(tzinfo=UTC)
is_stale = (
started_at is not None
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
is_stale = started_at is not None and (now - started_at) > timedelta(
minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES
)
if not is_stale:
logger.warning_event(
@@ -189,9 +222,7 @@ async def run_collector_task(collector_name: str):
existing_running.phase = "failed"
existing_running.completed_at = now
existing_running.error_message = (
f"{existing_error}\n{stale_reason}".strip()
if existing_error
else stale_reason
f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
)
await db.commit()
logger.warning_event(
@@ -231,7 +262,11 @@ async def run_collector_task(collector_name: str):
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"status": "running",
},
)
task_result = await collector.run(db)
datasource = await db.get(DataSource, datasource_id)
@@ -265,7 +300,11 @@ async def run_collector_task(collector_name: str):
logger.info_event(
"Collector completed",
event="collector.run.completed",
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"result": task_result,
},
)
await emit_business_log(
logger,
@@ -300,7 +339,11 @@ async def run_collector_task(collector_name: str):
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "cancelled",
},
)
raise
except Exception as exc:
@@ -312,7 +355,11 @@ async def run_collector_task(collector_name: str):
logger.exception_event(
"Collector failed",
event="collector.run.failed",
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"error": str(exc),
},
)
await emit_business_log(
logger,
@@ -324,7 +371,11 @@ async def run_collector_task(collector_name: str):
module=__name__,
context=exception_context(
exc,
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
{
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "failed",
},
),
)
@@ -349,7 +400,9 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
task.completed_at = datetime.now(UTC)
existing_error = (task.error_message or "").strip()
cleanup_error = "Marked failed automatically after stale running task cleanup"
task.error_message = f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
task.error_message = (
f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
)
if stale_tasks:
await db.commit()
@@ -365,6 +418,17 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
def start_scheduler() -> None:
"""Start the scheduler."""
if not scheduler.running:
scheduler.add_job(
refresh_earth_news_all_regions,
trigger=IntervalTrigger(seconds=EARTH_NEWS_REFRESH_INTERVAL_SECONDS),
id=EARTH_NEWS_REFRESH_JOB_ID,
name="Earth news RSS refresh",
replace_existing=True,
next_run_time=datetime.now(UTC),
max_instances=1,
coalesce=True,
misfire_grace_time=60,
)
scheduler.start()
logger.info_event("Scheduler started", event="scheduler.started")
@@ -376,6 +440,205 @@ def stop_scheduler() -> None:
logger.info_event("Scheduler stopped", event="scheduler.stopped")
def schedule_earth_news_region_refresh(
region: str, *, force: bool = False, refresh_id: str | None = None
) -> bool:
"""Queue one immediate, per-region RSS refresh and merge duplicate requests."""
normalized_region = str(region or "").strip().lower()
if normalized_region not in REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {region}")
if normalized_region in _running_earth_news_refreshes:
rerun = _rerun_earth_news_refreshes.setdefault(
normalized_region, {"force": False, "refresh_id": None}
)
rerun["force"] = rerun["force"] or force
rerun["refresh_id"] = refresh_id or rerun["refresh_id"]
return False
if normalized_region in _pending_earth_news_refreshes:
pending = _pending_earth_news_refreshes[normalized_region]
pending["force"] = pending["force"] or force
pending["refresh_id"] = refresh_id or pending["refresh_id"]
return False
_pending_earth_news_refreshes[normalized_region] = {
"force": force,
"refresh_id": refresh_id,
}
job_id = _earth_news_region_refresh_job_id(normalized_region)
try:
scheduler.add_job(
_run_earth_news_region_refresh,
trigger=DateTrigger(run_date=datetime.now(UTC)),
id=job_id,
name=f"Earth news RSS refresh: {normalized_region}",
kwargs={"region": normalized_region},
replace_existing=False,
max_instances=1,
coalesce=True,
misfire_grace_time=60,
)
except Exception:
_pending_earth_news_refreshes.pop(normalized_region, None)
raise
return True
def request_earth_news_region_refresh(region: str) -> dict[str, Any]:
normalized_region = str(region or "").strip().lower()
if normalized_region not in REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {region}")
status, created = create_earth_news_refresh_status(normalized_region)
if created:
try:
schedule_earth_news_region_refresh(
normalized_region,
force=True,
refresh_id=status["refresh_id"],
)
except Exception:
return set_earth_news_refresh_status(
status["refresh_id"],
normalized_region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
return status
def schedule_earth_news_refresh_for_all_regions(*, force: bool = False) -> None:
for region in REGION_ANCHORS:
schedule_earth_news_region_refresh(region, force=force)
def _earth_news_region_refresh_job_id(region: str) -> str:
return f"{EARTH_NEWS_REFRESH_JOB_ID}:{region}"
async def _run_earth_news_region_refresh(region: str) -> None:
request = _pending_earth_news_refreshes.pop(region, {"force": False, "refresh_id": None})
force = bool(request["force"])
refresh_id = request["refresh_id"]
_running_earth_news_refreshes.add(region)
status_heartbeat: asyncio.Task[None] | None = None
try:
if refresh_id:
set_earth_news_refresh_status(refresh_id, region, "running")
if not touch_earth_news_refresh_status(refresh_id, region):
return
status_heartbeat = asyncio.create_task(
_earth_news_refresh_status_heartbeat(refresh_id, region)
)
deadline = time.monotonic() + EARTH_NEWS_REFRESH_WAIT_SECONDS
while True:
result = await refresh_earth_news_region(region, force=force)
if result["state"] != "busy":
break
if time.monotonic() >= deadline:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
return
await asyncio.sleep(EARTH_NEWS_REFRESH_POLL_SECONDS)
if result["state"] == "coordination_unavailable":
error_code = "refresh_failed"
elif result["state"] == "failed":
error_code = result.get("error_code") or "source_unavailable"
elif result["state"] == "completed" and result.get("successful_sources", 0) > 0:
set_earth_news_refresh_status(refresh_id, region, "succeeded")
return
else:
error_code = "source_unavailable"
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": error_code,
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS.get(
error_code, EARTH_NEWS_REFRESH_PUBLIC_ERRORS["source_unavailable"]
),
},
)
else:
await refresh_earth_news_region(region, force=force)
except asyncio.CancelledError:
if refresh_id:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
raise
except Exception as exc:
logger.warning_event(
"Earth news tracked refresh failed",
event="earth_news.rss.tracked_refresh_failed",
context={"region": region, "error": str(exc)},
)
if refresh_id:
try:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
except Exception as status_exc:
logger.warning_event(
"Earth news refresh status update failed",
event="earth_news.rss.refresh_status_update_failed",
context={"region": region, "error": str(status_exc)},
)
finally:
if status_heartbeat is not None:
status_heartbeat.cancel()
try:
await status_heartbeat
except asyncio.CancelledError:
pass
_running_earth_news_refreshes.discard(region)
rerun = _rerun_earth_news_refreshes.pop(region, None)
if rerun is not None and scheduler.running:
schedule_earth_news_region_refresh(
region,
force=rerun["force"],
refresh_id=rerun["refresh_id"],
)
async def _earth_news_refresh_status_heartbeat(refresh_id: str, region: str) -> None:
try:
while True:
await asyncio.sleep(EARTH_NEWS_STATUS_HEARTBEAT_SECONDS)
if not touch_earth_news_refresh_status(refresh_id, region):
return
except asyncio.CancelledError:
raise
except Exception as exc:
logger.warning_event(
"Earth news refresh status heartbeat failed",
event="earth_news.rss.refresh_status_heartbeat_failed",
context={"region": region, "error": str(exc)},
)
async def sync_scheduler_with_datasources() -> None:
"""Synchronize scheduler jobs with datasource table."""
async with async_session_factory() as db:
@@ -384,7 +647,10 @@ async def sync_scheduler_with_datasources() -> None:
configured_sources = {datasource.source for datasource in datasources}
for job in list(scheduler.get_jobs()):
if job.id not in configured_sources:
is_earth_news_refresh = job.id == EARTH_NEWS_REFRESH_JOB_ID or job.id.startswith(
f"{EARTH_NEWS_REFRESH_JOB_ID}:"
)
if job.id not in configured_sources and not is_earth_news_refresh:
scheduler.remove_job(job.id)
for datasource in datasources:
@@ -461,7 +727,9 @@ def run_collector_now(collector_name: str) -> bool:
return False
try:
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
task = asyncio.create_task(
run_collector_task(collector_name), name=_collector_task_name(collector_name)
)
RUNNING_COLLECTOR_TASKS[collector_name] = task
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:

View File

@@ -15,6 +15,12 @@ DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
TV_SETTINGS_CATEGORY = "tv"
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
UNSUPPORTED_CCTV4_STREAM = (
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
)
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
DEFAULT_TV_SETTINGS = {
"default_source_id": DEFAULT_TV_SOURCE_ID,
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
"sources": [
{
"id": "cctv4",
"name": "CCTV-4 中文国际",
"name": "CCTV-4 中文国际(美洲版)",
"provider": "CCTV",
"region": "China",
"language": "zh-CN",
"source_type": "hls",
"embed_url": "https://tv.cctv.com/live/cctv4/",
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
"homepage_url": "https://tv.cctv.com/live/cctv4/",
"embed_url": "",
"stream_url": CCTV4_STREAM,
"use_proxy": False,
"homepage_url": CCTV4_HOMEPAGE,
"poster_url": "",
"is_enabled": True,
"is_fallback": True,
"sort_order": 10,
"collector_source": None,
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
"notes": CCTV4_PLAYBACK_NOTE,
},
{
"id": "reuters-tv",
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
payload = dict(source or {})
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
payload.get("stream_url")
) == UNSUPPORTED_CCTV4_STREAM:
payload.update(
name="CCTV-4 中文国际(美洲版)",
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
use_proxy=False,
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
)
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
source_type = _clean_text(payload.get("source_type")).lower()
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
"source_type": source_type,
"embed_url": _clean_url(payload.get("embed_url")),
"stream_url": _clean_url(payload.get("stream_url")),
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
"homepage_url": _clean_url(payload.get("homepage_url")),
"poster_url": _clean_url(payload.get("poster_url")),
"youtube_video_id": youtube_video_id,

View File

@@ -8,6 +8,37 @@ from unittest.mock import AsyncMock, MagicMock
from sqlalchemy.ext.asyncio import AsyncSession
@pytest.fixture(autouse=True)
def persisted_test_logs(monkeypatch):
"""Keep application log persistence in memory, including business-log calls."""
from app.models.system_log import ObservabilityEventGroup
from app.services import persistent_logs
entries = []
groups = {}
class LogSession:
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, traceback):
return False
def add(self, item):
entries.append(item)
if isinstance(item, ObservabilityEventGroup):
groups[item.fingerprint] = item
async def get(self, model, key):
return groups.get(key) if model is ObservabilityEventGroup else None
async def commit(self):
return None
monkeypatch.setattr(persistent_logs, "async_session_factory", LogSession)
return entries
@pytest.fixture(autouse=True)
def bgp_collector_location_cache():
"""Mirror app startup seeding for tests that call sync BGP helpers."""

View File

@@ -0,0 +1,72 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from fastapi import FastAPI
from pydantic import ValidationError
from app.api.v1.auth import router
from app.core import security
from app.db.session import DEFAULT_LOGIN_USERS, get_db
from app.models.user import User
from app.schemas.user import UserRegister
async def identity_request(monkeypatch, user, token=None):
result = MagicMock()
result.scalar_one_or_none.return_value = user
db = AsyncMock()
db.execute.return_value = result
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
app = FastAPI()
app.include_router(router, prefix="/auth")
async def override_db():
yield db
app.dependency_overrides[get_db] = override_db
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
return await client.get(
"/auth/me",
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
created = datetime(2026, 9, 21, tzinfo=UTC)
user = User(
id=1, username=default["username"], email=default["email"],
role=default["role"], password_hash="not-exposed", is_active=True,
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
)
response = await identity_request(monkeypatch, user)
assert response.status_code == 200
payload = response.json()
assert payload["email"] == default["email"]
assert payload["email_verified"] is True
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
assert payload["gatekeeper_groups"] == ["ops"]
assert "password_hash" not in payload
@pytest.mark.asyncio
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
response = await identity_request(monkeypatch, user)
assert response.status_code == 401
@pytest.mark.asyncio
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
assert response.status_code == 401
def test_registration_still_validates_email():
with pytest.raises(ValidationError):
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")

View File

@@ -1,5 +1,6 @@
"""Tests for BGP observability helpers."""
import ipaddress
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
@@ -21,13 +22,20 @@ from app.services.collectors.bgp_common import (
create_bgp_anomalies_for_batch,
save_bgp_observations_for_batch,
)
from app.services.bgp_enrichment import enrich_bgp_events_for_batch, extract_bgp_network_fields
from app.services.bgp_enrichment import (
enrich_bgp_events_for_batch,
extract_bgp_network_fields,
lookup_prefix_geography,
)
from app.services.bgp_incidents import (
create_bgp_incidents_for_anomalies,
infer_related_infrastructure,
)
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.api.v1.visualization import convert_bgp_incidents_to_geojson, build_incident_geography_hints
from app.api.v1.visualization import (
convert_bgp_incidents_to_geojson,
build_incident_geography_hints,
)
from app.models.bgp_anomaly import BGPAnomaly
from app.models.collected_data import CollectedData
from app.models.bgp_incident import BGPIncident
@@ -58,8 +66,7 @@ class _FakeResult:
def all(self):
if self._rows and all(isinstance(row, BGPObservation) for row in self._rows):
return [
(row.prefix, row.origin_asn, row.collector, row.collector_geo)
for row in self._rows
(row.prefix, row.origin_asn, row.collector, row.collector_geo) for row in self._rows
]
return self._rows
@@ -93,6 +100,8 @@ class _FakeAsyncSession:
self.commits = 0
async def execute(self, _stmt, _params=None):
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
return _FakeResult([])
if not self._results:
return _FakeResult([])
return _FakeResult(self._results.pop(0))
@@ -107,6 +116,49 @@ class _FakeAsyncSession:
self.commits += 1
class _StrictFakeAsyncSession(_FakeAsyncSession):
async def execute(self, stmt, params=None):
if not str(stmt).lstrip().startswith("SET LOCAL jit") and not self._results:
raise AssertionError(f"Unexpected BGP query: {stmt}")
return await super().execute(stmt, params)
@pytest.mark.asyncio
async def test_prefix_geography_keeps_source_priority_and_ipv6_range_parameters():
calls = []
class GeoSession:
async def execute(self, _stmt, params=None):
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
return _FakeResult([])
calls.append(dict(params))
if params["source"] == "nro_delegated_prefix_geo":
return _FakeResult(
[
{
"metadata": {
"country_code": "GB",
"prefix": "2001:db8::/32",
}
}
]
)
return _FakeResult([])
network = ipaddress.ip_network("2001:db8:1::/48")
result = await lookup_prefix_geography(GeoSession(), [str(network), "invalid-prefix"])
assert result[str(network)]["source"] == "nro_delegated_stats"
assert [call["source"] for call in calls] == [
"opengeofeed_prefix_geo",
"iptoasn_prefix_geo",
"nro_delegated_prefix_geo",
]
assert all(call["family"] == "ipv6" for call in calls)
assert all(call["range_start"] == str(network.network_address) for call in calls)
assert all(call["range_end"] == str(network.broadcast_address) for call in calls)
def test_normalize_bgp_event_from_live_payload():
event = normalize_bgp_event(
{
@@ -449,7 +501,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
"prefix": "203.0.113.0/24",
"origin_asn": 64496,
"as_path": [64500, 64496],
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
"collector_location": {
"country": "NL",
"city": "Amsterdam",
"latitude": 52.3,
"longitude": 4.9,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
}
},
@@ -460,7 +517,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
"prefix": "203.0.113.0/24",
"origin_asn": 64496,
"as_path": [64510, 64520, 64530, 64540, 64496],
"collector_location": {"country": "GB", "city": "London", "latitude": 51.5, "longitude": -0.1},
"collector_location": {
"country": "GB",
"city": "London",
"latitude": 51.5,
"longitude": -0.1,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "GB"}]}},
}
},
@@ -481,7 +543,9 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
base_timestamp = datetime(2026, 3, 27, 0, 0, tzinfo=UTC)
events = []
for index, event_type in enumerate(["announcement", "withdrawal", "announcement", "withdrawal"]):
for index, event_type in enumerate(
["announcement", "withdrawal", "announcement", "withdrawal"]
):
events.append(
{
"metadata": {
@@ -491,7 +555,12 @@ def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
"prefix": "198.51.100.0/24",
"origin_asn": 64512,
"as_path": [64500 + index, 64512] if event_type == "announcement" else [],
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
"collector_location": {
"country": "NL",
"city": "Amsterdam",
"latitude": 52.3,
"longitude": 4.9,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
}
}
@@ -920,7 +989,9 @@ async def test_create_bgp_incidents_for_anomalies_refreshes_existing_incident():
with patch(
"app.services.bgp_incidents.infer_related_infrastructure",
new=AsyncMock(return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}),
new=AsyncMock(
return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}
),
):
created = await create_bgp_incidents_for_anomalies(
db,
@@ -1132,13 +1203,17 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
source="ris_live_bgp",
extra_data={"prefix": "203.0.113.0/24", "origin_asn": 64496},
)
db = _FakeAsyncSession([
[],
[],
[],
[previous_record],
[],
])
db = _StrictFakeAsyncSession(
[
[], # Historical BGP observations
[], # OpenGeoFeed prefix geography
[], # IPtoASN prefix geography
[], # NRO delegated prefix geography
[], # PeeringDB ASN profiles
[previous_record], # Prior collected snapshot for origin-change detection
[], # Existing anomaly entity keys
]
)
events = [
{
"reference_date": "2026-03-30T10:00:00Z",
@@ -1178,6 +1253,7 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
assert anomaly.anomaly_type == "origin_change"
assert anomaly.prefix == "203.0.113.0/24"
incident_mock.assert_awaited_once()
assert not db._results
@pytest.mark.asyncio
@@ -1197,14 +1273,18 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
origin_asn=64496,
new_origin_asn=64497,
)
db = _FakeAsyncSession([
[],
[],
[],
[previous_record],
[existing_key],
[existing_anomaly],
])
db = _StrictFakeAsyncSession(
[
[], # Historical BGP observations
[], # OpenGeoFeed prefix geography
[], # IPtoASN prefix geography
[], # NRO delegated prefix geography
[], # PeeringDB ASN profiles
[previous_record], # Prior collected snapshot for origin-change detection
[existing_key],
[existing_anomaly],
]
)
events = [
{
"reference_date": "2026-03-30T10:00:00Z",
@@ -1235,6 +1315,7 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
assert created == 0
assert len(db.added) == 0
incident_mock.assert_awaited_once()
assert not db._results
async def _bgp_test_client(db_session):
@@ -1242,7 +1323,9 @@ async def _bgp_test_client(db_session):
yield db_session
def override_get_current_user():
return User(id=1, username="testuser", email="test@example.com", password_hash="x", role="admin")
return User(
id=1, username="testuser", email="test@example.com", password_hash="x", role="admin"
)
app.dependency_overrides[get_db] = override_get_db
app.dependency_overrides[get_current_user] = override_get_current_user
@@ -1264,7 +1347,7 @@ async def test_bgp_events_api_lists_observations():
origin_asn=64496,
observed_at=datetime(2026, 3, 30, 10, 0, tzinfo=UTC),
)
db = _FakeAsyncSession([[observation]])
db = _StrictFakeAsyncSession([[1], [observation]])
client = await _bgp_test_client(db)
try:
@@ -1278,6 +1361,7 @@ async def test_bgp_events_api_lists_observations():
assert payload["total"] == 1
assert payload["data"][0]["collector"] == "rrc00"
assert payload["data"][0]["prefix"] == "203.0.113.0/24"
assert not db._results
@pytest.mark.asyncio
@@ -1376,7 +1460,18 @@ async def test_bgp_event_summary_api_returns_aggregates():
event_type="withdrawal",
observed_at=datetime(2026, 3, 30, 10, 5, tzinfo=UTC),
)
db = _FakeAsyncSession([[observation_one, observation_two]])
observations = [observation_one, observation_two]
db = _StrictFakeAsyncSession(
[
[len(observations)],
[len({record.collector for record in observations})],
[len({record.prefix for record in observations})],
[
(event_type, sum(record.event_type == event_type for record in observations))
for event_type in sorted({record.event_type for record in observations})
],
]
)
client = await _bgp_test_client(db)
try:
@@ -1392,6 +1487,7 @@ async def test_bgp_event_summary_api_returns_aggregates():
assert payload["prefix_count"] == 2
assert payload["by_type"]["announcement"] == 1
assert payload["by_type"]["withdrawal"] == 1
assert not db._results
@pytest.mark.asyncio

View File

@@ -0,0 +1,22 @@
from unittest.mock import AsyncMock
import pytest
from app.db.bgp_prefix_geo import (
BGP_PREFIX_GEO_FUNCTION_SQL,
BGP_PREFIX_GEO_INDEX_SQL,
install_bgp_prefix_geo_functions,
)
@pytest.mark.asyncio
async def test_fresh_database_installs_lookup_functions_without_large_index():
connection = AsyncMock()
await install_bgp_prefix_geo_functions(connection)
statements = [str(call.args[0]) for call in connection.execute.call_args_list]
assert len(statements) == len(BGP_PREFIX_GEO_FUNCTION_SQL) == 2
assert all("CREATE OR REPLACE FUNCTION" in statement for statement in statements)
assert all("CREATE INDEX" not in statement for statement in statements)
assert "CREATE INDEX CONCURRENTLY" in BGP_PREFIX_GEO_INDEX_SQL

View File

@@ -0,0 +1,333 @@
import io
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import FastAPI, HTTPException
from httpx import ASGITransport, AsyncClient
from PIL import Image
from pydantic import ValidationError
from app.api.v1.card_images import router
from app.core.security import get_current_user
from app.db.session import get_db
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
from app.services.card_image_configuration import (
effective_envelope,
publish_configuration,
resolve_image,
)
from app.services.card_image_defaults import bundled_configuration
from app.services import card_image_files
def configuration():
assets = [
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
for key in ["exact", "model", "group", "category"]
]
rules = [
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
for key, scope, value in [
("exact", "object", "123"),
("model", "model", "m1"),
("group", "group", "starlink"),
("category", "category", "satellite"),
]
]
return CardImageConfiguration(assets=assets, rules=rules)
def test_precedence_disabled_fallback_and_explicit_no_image():
config = configuration()
context = dict(
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
)
for expected in ["exact", "model", "group", "category"]:
assert resolve_image(config, context)["image"]["id"] == expected
next(asset for asset in config.assets if asset.id == expected).enabled = False
assert resolve_image(config, context)["image"] is None
config = configuration()
config.rules[0].asset_id = None
config.rules[0].suppress = True
result = resolve_image(config, context)
assert result["suppressed"] and result["image"] is None
def test_bundled_series_images_retain_truthful_source_metadata():
config, _ = bundled_configuration()
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
assert result["image"]["id"] == "starlink-solar-array"
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
assert pending.rights_status == "permission_pending"
assert (
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
== pending.id
)
assert pending.enabled
@pytest.mark.parametrize(
"url",
[
"javascript:alert(1)",
"//evil.example/image.jpg",
"/earth/assets/cards/../secret",
"https://user:pass@example.org/a.jpg",
],
)
def test_unsafe_urls_rejected(url):
with pytest.raises(ValidationError):
CardImageAsset(id="test", title="test", url=url)
def test_duplicate_targets_and_dangling_assets_rejected():
data = configuration().model_dump()
duplicate = {**data["rules"][0], "id": "another"}
data["rules"].append(duplicate)
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
data = configuration().model_dump()
data["assets"] = []
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
defaults, _ = bundled_configuration()
payload = {
"revision": "custom",
"config": {"assets": [], "rules": []},
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
payload["known_assets"] = []
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
output = io.BytesIO()
Image.new("RGB", (1600, 900), "blue").save(output, format)
asset = card_image_files.store_image(output.getvalue(), "Facility")
assert (asset.width, asset.height) == (1600, 900)
assert len(list(tmp_path.iterdir())) == 3
for size, url in asset.variants.items():
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
assert max(image.size) == int(size)
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
with pytest.raises(HTTPException):
card_image_files.store_image(b"<svg></svg>", "bad")
with pytest.raises(HTTPException):
card_image_files.uploaded_path("../private")
def fake_db(payload):
db = AsyncMock()
db.add = MagicMock()
record = SimpleNamespace(payload=payload)
result = MagicMock()
result.scalar_one.return_value = record
result.scalar_one_or_none.return_value = record
db.execute.return_value = result
return db, record
@pytest.mark.asyncio
async def test_publish_revision_and_audit():
defaults, _ = bundled_configuration()
payload = {
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
db, record = fake_db(payload)
revision = effective_envelope(payload)["revision"]
result = await publish_configuration(
db, revision, defaults, SimpleNamespace(id=1, username="admin")
)
assert result["revision"] != revision
assert db.add.call_args.args[0].action == "card_images.update"
before = deepcopy(record.payload)
with pytest.raises(HTTPException) as error:
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
assert error.value.status_code == 409 and record.payload == before
@pytest.mark.asyncio
async def test_configuration_requires_admin_and_validates_http_payload():
app = FastAPI()
app.include_router(router)
defaults, _ = bundled_configuration()
db, _ = fake_db(
{
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (await client.get("/configuration")).status_code == 403
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
assert (await client.get("/configuration")).status_code == 200
assert (
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
).status_code == 422
@pytest.mark.asyncio
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
app = FastAPI()
app.include_router(router)
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
buffer = io.BytesIO()
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
response = await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
assert response.status_code == 200
asset = response.json()
filename = asset["variants"]["640"].rsplit("/", 1)[1]
image = await client.get(f"/files/{filename}")
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
assert "immutable" in image.headers["cache-control"]
assert (
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
).status_code == 422
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (
await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
).status_code == 403
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
from typing import get_args
from app.schemas.card_images import EntityType
config, _ = bundled_configuration()
for entity_type in get_args(EntityType):
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
assert result["image"] and result["match"]["value"] == "*"
config = config.model_copy(deep=True)
config.rules.append(
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
)
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["starlink-solar-array"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
def starlink():
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["focus_x"] = 20
assert starlink()["focus_x"] == 20
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["enabled"] = False
assert starlink()["enabled"] is False
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
assert starlink()["url"] == "https://example.org/custom.jpg"
def test_compute_categories_and_bgp_get_separate_user_images():
config, _ = bundled_configuration()
for kind, filename in [
("supercomputer", "supercomputer.png"),
("gpu_cluster", "gpu-cluster.jpg"),
("data_center", "data-center.png"),
("cloud_center", "cloud-center.png"),
("edge_node", "edge-node.png"),
]:
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
assert result["image"]["url"].endswith(filename)
for kind in ("bgp", "bgp_collector"):
result = resolve_image(config, dict(entity_type=kind, category=kind))
assert result["image"]["url"].endswith(
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
)
def test_saved_collector_default_migrates_without_changing_user_bindings():
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
old_asset = previous_bundled_assets()["network-switches"][0]
payload = {
"revision": "old",
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
"known_assets": [old_asset.id],
"known_rules": [old_rule.id],
}
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
"image"
]["url"].endswith("bgp-collector.png")
payload["config"]["rules"][0]["enabled"] = False
result = effective_envelope(payload)["config"]
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
@pytest.mark.asyncio
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
from app.services.card_image_targets import get_target
from sqlalchemy.dialects import postgresql
db = AsyncMock()
result = MagicMock()
result.mappings.return_value.first.return_value = {
"entity_id": "epoch_ai_gpu:site",
"name": "Site",
"category": "cloud_center",
"group": "Operator",
"model": "Model",
}
db.execute.return_value = result
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
query = db.execute.call_args.args[0]
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "collected_data.source = 'epoch_ai_gpu'" in sql
assert "collected_data.source_id = 'site'" in sql
assert context["category"] == "cloud_center" and context["model"] == "Model"
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["compute-gpu-cluster"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
result = effective_envelope(payload)["config"]
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
assert upgraded["variants"] == latest.variants
assert upgraded["url"] == latest.url

View File

@@ -0,0 +1,218 @@
"""Request-level regressions for quota-limited satellite downloads."""
import gzip
import json
import os
import time
from unittest.mock import AsyncMock
import httpx
import pytest
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
from app.services.collectors.downloads import ResumableFileDownloader
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
def setup_collector(monkeypatch, tmp_path, handler):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/gp.php"
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
original_client = httpx.AsyncClient
monkeypatch.setattr(
"app.services.collectors.celestrak.httpx.AsyncClient",
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
)
return collector
def expire(path):
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
os.utime(path, (old, old))
@pytest.mark.asyncio
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
return httpx.Response(
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
)
collector = setup_collector(monkeypatch, tmp_path, handler)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert len(requests) == 1
path = collector._downloader.cached_file_path(URL, ".json")
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
assert requests[1].headers["if-none-match"] == '"v1"'
@pytest.mark.asyncio
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
collector = setup_collector(monkeypatch, tmp_path, handler)
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
assert requests == ["GET"]
@pytest.mark.asyncio
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
with pytest.raises(RuntimeError, match="validation failed"):
await collector.fetch()
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
@pytest.mark.asyncio
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
collector = setup_collector(monkeypatch, tmp_path, handler)
path, _, meta = collector._downloader._cache_paths(URL, ".json")
path.write_text("[]")
meta.write_text('{"etag":"old"}')
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
assert "if-none-match" not in requests[0].headers
@pytest.mark.asyncio
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
)
path, _, meta = downloader._cache_paths(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text('[{"NORAD_CAT_ID":42}]')
meta.write_text('{"etag":"v1"}')
expire(path)
requests = []
def handler(request):
requests.append(request)
return httpx.Response(304)
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
for _ in range(2):
await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert len(requests) == 1
assert requests[0].headers["if-none-match"] == "v1"
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
@pytest.mark.asyncio
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
body = gzip.compress(raw)
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False
)
async with httpx.AsyncClient(
transport=httpx.MockTransport(
lambda request: httpx.Response(
200,
content=body,
headers={"content-encoding": "gzip", "content-length": str(len(body))},
)
)
) as client:
path = await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert path.read_bytes() == raw
@pytest.mark.asyncio
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
final, part, meta = downloader._cache_paths(URL, ".bin")
part.parent.mkdir(parents=True)
part.write_bytes(b"abc")
meta.write_text('{"etag":"v1"}')
requests = []
def handler(request):
requests.append(request)
if request.method == "HEAD":
return httpx.Response(
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
)
assert request.headers["range"] == "bytes=3-"
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".bin")
assert [r.method for r in requests] == ["HEAD", "GET"]
assert final.read_bytes() == b"abcdef"
@pytest.mark.asyncio
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
if request.url.params["GROUP"] == "active":
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
collector = setup_collector(monkeypatch, tmp_path, handler)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
assert [r.method for r in requests] == ["GET", "GET"]
@pytest.mark.asyncio
async def test_generic_cache_without_validators_must_refresh(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
path = downloader.cached_file_path(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text("[]")
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".json")
assert requests == ["HEAD", "GET"]
assert path.read_text() == "[1]"

View File

@@ -79,7 +79,8 @@ class TestTOP500Collector:
"""Test parsing empty response"""
collector = TOP500Collector()
result = collector.parse_response("<html><body><table></table></body></html>")
assert len(result) > 0
assert result == []
assert collector.fail_on_empty
def test_parse_response_single_item(self):
"""Test parsing single item response"""
@@ -97,9 +98,9 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["source_id"] == "top500_1"
assert result[0]["source_id"] == "top500_system_1"
assert result[0]["name"] == "Test Supercomputer"
assert result[0]["country"] == "USA"
assert result[0]["metadata"]["rank"] == 1
@@ -113,7 +114,7 @@ class TestTOP500Collector:
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
<tr>
<td>1</td>
<td>Valid\nVendor\nSite\nUSA</td>
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
<td>1000</td>
<td>10 PFLOP/s</td>
<td>12 PFLOP/s</td>
@@ -129,7 +130,7 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["name"] == "Valid"

View File

@@ -0,0 +1,222 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.system_log import AuditLog
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
default_envelope,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import FastAPI, HTTPException, Response
from httpx import ASGITransport, AsyncClient
from pydantic import ValidationError
def default_config():
return DEFAULT_CONFIGURATION.model_dump(mode="json")
@pytest.mark.parametrize(
"mutation",
[
lambda value: value["observations"][0]["values"].update(CN=101),
lambda value: value["observations"][0]["values"].update(CN=-1),
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
lambda value: value["observations"][0]["values"].update(CN=True),
lambda value: value["observations"][0]["values"].update(EUROPE=10),
lambda value: value["observations"][0].update(source_id="unknown"),
lambda value: value["observations"][0].update(year=2026),
lambda value: value["observations"].append(value["observations"][0]),
lambda value: value["sources"].append(value["sources"][0]),
lambda value: value["innovation"].append(value["innovation"][0]),
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
],
)
def test_rejects_invalid_or_ambiguous_evidence(mutation):
payload = default_config()
mutation(payload)
with pytest.raises(ValidationError):
ComputeConfiguration.model_validate(payload)
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
payload = default_config()
payload["sources"].append(
{
"id": "new-report",
"title": "New report",
"url": "https://example.com/report",
"publisher": "Research group",
"report_year": 2026,
"locator": "Figure 2",
}
)
payload["observations"].append(
{
"year": 2025,
"metric": "global_share",
"source_id": "new-report",
"values": {"CN": 25, "US": 50, "JP": 0},
}
)
config = ComputeConfiguration.model_validate(payload)
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
row = next(row for row in result["compute"] if row["year"] == 2025)
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
payload = default_config()
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
result = compute_history(
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
)
assert result["compute"][0]["values"] == {"CN": None, "US": None}
payload.update(sources=[], observations=[], innovation=[])
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
assert all(row["values"]["CN"] is None for row in result["compute"])
assert result["evidence"] == {}
def fake_db(record=None):
db = AsyncMock()
db.add = MagicMock()
result = MagicMock()
result.scalar_one_or_none.return_value = record
result.scalar_one.return_value = record
db.execute.return_value = result
return db
@pytest.mark.asyncio
async def test_fresh_install_uses_defaults_without_writing_database():
db = fake_db()
assert await read_configuration(db) == default_envelope()
db.add.assert_not_called()
db.commit.assert_not_awaited()
@pytest.mark.asyncio
async def test_series_reads_persisted_override_and_disables_http_cache():
envelope = default_envelope()
envelope["is_default"] = False
envelope["revision"] = "custom"
envelope["config"]["observations"][3]["values"]["CN"] = 30
response = Response()
data = await get_series(
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
)
assert data["compute"][3]["values"]["CN"] == 30
assert data["version"] == "custom"
assert data["provenance"] == "admin_curated_reports"
assert response.headers["Cache-Control"] == "no-store"
@pytest.mark.asyncio
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
actor = SimpleNamespace(id=1, username="admin")
old = deepcopy(record.payload)
result = await publish_configuration(
db,
revision=old["revision"],
config=DEFAULT_CONFIGURATION,
actor=actor,
)
assert result["revision"] != old["revision"]
audit = db.add.call_args.args[0]
assert isinstance(audit, AuditLog)
assert audit.details == {"before": old, "after": result}
assert audit.actor_id == actor.id
db.commit.assert_awaited_once()
with pytest.raises(HTTPException) as failure:
await publish_configuration(
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
)
assert failure.value.status_code == 409
assert record.payload == result
db.rollback.assert_awaited_once()
db.commit.assert_awaited_once()
@pytest.mark.asyncio
async def test_reset_changes_revision_and_restores_bundled_data():
record = SimpleNamespace(
payload={**default_envelope(), "revision": "custom", "is_default": False}
)
result = await publish_configuration(
fake_db(record),
revision="custom",
config=DEFAULT_CONFIGURATION,
actor=SimpleNamespace(id=1, username="admin"),
reset=True,
)
assert result["is_default"] is True
assert result["config"] == default_config()
assert result["revision"] != "custom"
@pytest.mark.asyncio
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
async def test_non_admin_cannot_configure(role):
with pytest.raises(HTTPException) as failure:
await configuration_admin(SimpleNamespace(role=role))
assert failure.value.status_code == 403
@pytest.mark.asyncio
async def test_http_configuration_auth_validation_and_public_series():
app = FastAPI()
app.include_router(router)
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
assert (await client.get("/series")).status_code == 200
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=1,
username="admin",
role="admin",
)
loaded = (await client.get("/configuration")).json()
invalid = deepcopy(loaded["config"])
invalid["observations"][0]["values"]["CN"] = 200
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": invalid,
},
)
assert response.status_code == 422
db.commit.assert_not_awaited()
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": loaded["config"],
},
)
assert response.status_code == 200
assert response.json()["revision"] != loaded["revision"]
assert (
await client.post(
"/configuration/reset",
json={
"revision": loaded["revision"],
},
)
).status_code == 409

View File

@@ -0,0 +1,268 @@
import csv
import io
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import HTTPException
from sqlalchemy.dialects import postgresql
from app.api.v1.compute_intelligence import sample_query, selected_countries
from app.services.collectors.epoch_ai import EpochAIGPUCollector
from app.services.collectors.top500 import TOP500Collector
from app.services.compute_history import compute_history
def epoch_csv(**overrides):
row = {
"Name": "Cluster Alpha",
"Country": "United States of America",
"Status": "Existing",
"First Operational Date": "2023-05-02",
"16-bit OP/s (log)": "18",
"Location": "3231 Riverport Rd, Memphis, TN 38109",
"latitude": "35",
"longitude": "-90",
**overrides,
}
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
return output.getvalue()
def test_epoch_csv_preserves_precision_date_and_missing_values():
collector = EpochAIGPUCollector()
row = collector.parse_response(epoch_csv())[0]
assert row["value"] == 1e18
assert row["unit"] == "FLOP/s"
assert row["metadata"]["precision"] == "FP16"
assert row["metadata"]["location_precision"] == "precise"
assert row["metadata"]["raw_country"] == "United States of America"
assert row["reference_date"] == "2023-05-02"
missing = collector.parse_response(
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
)[0]
assert missing["value"] is None
assert missing["reference_date"] is None
@pytest.mark.parametrize(
("location", "country", "expected"),
[
("United States of America", "United States of America", "country"),
("Japan", "Japan", "country"),
("Korea (Republic of)", "South Korea", "country"),
("Texas", "United States of America", "region"),
("Iowa, USA", "United States of America", "region"),
("Seoul", "South Korea", "city"),
("Paris, France", "France", "city"),
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
("Cluster 2, Texas", "United States of America", "site"),
("Campus 7, Tokyo", "Japan", "site"),
("Building 240, Argonne", "United States of America", "site"),
("7", "United States of America", "unknown"),
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
("35.380924,139.625008", "Japan", "precise"),
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
],
)
def test_epoch_location_precision_follows_location_field(location, country, expected):
row = EpochAIGPUCollector().parse_response(
epoch_csv(Location=location, Country=country)
)[0]
assert row["metadata"]["location_precision"] == expected
@pytest.mark.parametrize(
("location", "expected"),
[
({"lat": 35.0, "lng": 139.0}, "precise"),
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
({"lat": 35.0}, "unknown"),
({"lng": 139.0}, "unknown"),
({"lat": "invalid", "lng": 139.0}, "unknown"),
],
)
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
from app.services.epoch_location import assess_epoch_location
assert assess_epoch_location(location).precision == expected
def test_epoch_anonymous_rows_are_not_geocoded():
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
assert row["latitude"] is None and row["longitude"] is None
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
@pytest.mark.parametrize(
"payload",
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
)
def test_epoch_invalid_payload_fails_closed(payload):
with pytest.raises(ValueError):
EpochAIGPUCollector().parse_response(payload)
def test_epoch_duplicate_identity_rejected():
payload = epoch_csv()
with pytest.raises(ValueError, match="identity collision"):
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
assert len(rows) == 2
assert rows[0]["source_id"] != rows[1]["source_id"]
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
def test_top500_units_and_stable_identity():
collector = TOP500Collector()
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
first = collector.parse_response(
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
)[0]
second = collector.parse_response(
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
)[0]
assert first["source_id"] == second["source_id"] == "top500_system_123"
assert float(first["value"]) == 1.5e15
assert first["metadata"]["rpeak_flops"] == 2e15
assert first["reference_date"] == "2025-11-01"
assert second["reference_date"] == "2026-06-01"
with pytest.raises(ValueError, match="date missing"):
collector.parse_response(html.format(rank=1))
@pytest.mark.asyncio
async def test_large_source_loss_does_not_create_snapshot():
db = AsyncMock()
db.scalar.return_value = 500
db.add = MagicMock()
with pytest.raises(ValueError, match="40%"):
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
db.add.assert_not_called()
db.commit.assert_not_awaited()
def test_history_missing_years_and_denominators():
global_data = compute_history(["CN", "US"])
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
bilateral = compute_history(["CN", "US"], "bilateral_share")
assert sum(bilateral["compute"][3]["values"].values()) == 100
assert bilateral["denominator"] == "US+CN"
absolute = compute_history(["US", "CN"], "absolute")
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
assert global_data["innovation"][-2]["value"] == 17.95
assert global_data["innovation"][-1]["value"] is None
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
@pytest.mark.asyncio
async def test_snapshot_query_uses_observation_cutoff_not_location():
query = await sample_query(AsyncMock(), ["CN"], 2023)
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "2024-01-01" in sql
assert "row_number() OVER (PARTITION BY" in sql
assert "compute_contract" in sql and "中国" in sql
assert "latitude" not in sql and "longitude" not in sql
def test_country_input_is_bounded_and_deduplicated():
assert selected_countries("CN,CN,US") == ["CN", "US"]
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
with pytest.raises(HTTPException):
selected_countries("invalid")
with pytest.raises(HTTPException, match="overlapping"):
selected_countries("EUROPE,DE")
def test_catalog_supports_global_participants_without_inventing_history():
from app.services.compute_catalog import (
CATALOG,
COMPARISON_GROUPS,
COUNTRIES,
source_country_names,
source_country_participants,
)
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
{entry["code"] for entry in CATALOG}
)
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
owners = source_country_names(["CN"])
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
assert [entry["kind"] for entry in CATALOG[:5]] == [
"country",
"country",
"region",
"region",
"region",
]
group_members = [
member
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
]
assert len(group_members) == len(set(group_members))
assert set(group_members) == set(COUNTRIES)
data = compute_history(["JP", "KR"])
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
assert all(row["values"]["KR"] is None for row in data["compute"])
assert all(row["value"] is None for row in data["innovation"])
assert len(data["catalog"]) > 30
def test_region_history_requires_complete_same_year_member_coverage():
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
assert row_2023["values"]["CN"] == 31
assert row_2023["values"]["US"] == 41
assert row_2023["values"]["EUROPE"] is None
assert row_2023["values"]["APAC"] is None
assert row_2023["values"]["OTHER"] is None
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
def test_anonymous_records_never_enter_the_location_repair_queue():
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.models.collected_data import CollectedData
record = CollectedData(
source="epoch_ai_gpu",
source_id="anonymous",
name="Anonymous",
extra_data={"compute_contract": "v1", "anonymous": True},
)
result = convert_compute_centers_to_geojson([record])
assert result["features"] == []
assert result["unresolved"] == []
assert result["withheld_count"] == 1
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
absolute = compute_history(["CN"], "absolute")
observed = {row["year"]: row for row in absolute["compute"]}
assert observed[2022]["values"]["CN"] == 302
assert observed[2022]["evidence_id"] == "caict-2023"
assert observed[2023]["evidence_id"] == "caict-2024"
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
for row in absolute["compute"]:
if row["evidence_id"]:
report = absolute["evidence"][row["evidence_id"]]
assert report["url"].startswith("https://www.caict.ac.cn/")

View File

@@ -0,0 +1,18 @@
from app.services.compute_center_verified_sites import get_verified_compute_site
def test_registry_requires_exact_facility_identity_not_operator_name():
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
assert site["city"] == "Yokohama"
assert site["precision"] == "site"
assert site["coordinate_basis"] == "official_access_map_center"
assert site["sources"]
site["latitude"] = 0
assert get_verified_compute_site(
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
)["latitude"] != 0

View File

@@ -1,4 +1,5 @@
import json
from pathlib import Path
import pytest
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
def write_geojson(path, name="Test"):
names = [name]
if name == "earth_admin0_boundaries":
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
path.write_text(
json.dumps(
{
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
"features": [
{
"type": "Feature",
"properties": {"name": name},
"properties": {
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
},
}
for item in names
],
}
),
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
config = repo / "config/earth-boundary-sources.local.json"
example = repo / "config/earth-boundary-sources.example.json"
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
path.mkdir(parents=True, exist_ok=True)
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
geometry.parent.mkdir(parents=True, exist_ok=True)
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
geometry.write_bytes(source_geometry.read_bytes())
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
return {
"repo": repo,
"config": config,
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
}
def write_fallback(path, policy_path, geometry_path):
policy = earth_boundaries._read_json(policy_path)
geometry = earth_boundaries._read_json(geometry_path)
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(policy),
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
}
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "legacy-geojson"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
assert status["provider"] == "missing"
assert status["fallback_available"] is False
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
}
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "pmtiles-mvt"
assert status["high_precision_ready"] is True
assert status["policy_publishable"] is False
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["high_precision_ready"] is False
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
assert status["high_precision_ready"] is True
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
assert paths["manifest"].exists()
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
assert manifest["povPolicy"]["applied"] is True
assert manifest["povPolicy"]["coverage"] == "partial"
assert manifest["povPolicy"]["publicationReady"] is False
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
by_rule = {
feature["properties"].get("PLANET_POLICY_RULE"): feature
for feature in hover["features"]
}
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
assert all(
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
for feature in hover["features"]
)
assert any(phase == "download" for _progress, phase, _message in progress_events)
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
assert removed.isdisjoint(DEFAULT_DATASOURCES)
for source in removed:
assert collector_registry.get(source) is None
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
repo = Path(__file__).resolve().parents[2]
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
payload = json.loads(path.read_text(encoding="utf-8"))
matches = [
feature for feature in payload["features"]
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
]
assert len(matches) == 1
jeju = matches[0]
assert jeju["properties"]["ISO_A3"] == "KOR"
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
ring = jeju["geometry"]["coordinates"][0]
assert len(ring) == 61
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)

View File

@@ -0,0 +1,41 @@
import json
from pathlib import Path
import pytest
from app.core.earth_geography_policy import display_country, display_iso_a3
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
from app.core import earth_geography_policy as policy
policy._policy_country_by_code.cache_clear()
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
try:
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
policy.display_country("China")
finally:
policy._policy_country_by_code.cache_clear()
def test_display_country_groups_china_regions_and_kosovo() -> None:
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
assert display_country(source) == "中国"
assert display_iso_a3(source) == "CHN"
for source in ("Kosovo", "XK", "SRB"):
assert display_country(source) == "塞尔维亚"
assert display_iso_a3(source) == "SRB"
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
assert display_country("Japan") == "日本"
assert display_iso_a3("Japan") == "JPN"
assert display_country("unexpected non-country") is None
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy = json.loads(path.read_text(encoding="utf-8"))
for rule in policy["rules"]:
for code in rule.get("sourceIsoA3", []):
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))

View File

@@ -0,0 +1,15 @@
from app.services.earth_layer_adapters import (
get_earth_cache_patterns_for_source,
get_earth_update_layers_for_change,
)
def test_news_changes_do_not_invalidate_earth_summary_cache():
assert get_earth_update_layers_for_change("earth_news_items", "earth_news_items") == ["news"]
assert get_earth_cache_patterns_for_source("earth_news_items") == []
assert get_earth_cache_patterns_for_source("media_news_archive") == []
def test_counted_layers_still_invalidate_earth_summary_cache():
for source in ("aisstream_vessels", "arcgis_cables", "celestrak_tle", "top500", "ris_live_bgp"):
assert "summary*" in get_earth_cache_patterns_for_source(source)

View File

@@ -1,9 +1,11 @@
from __future__ import annotations
import asyncio
import json
import httpx
import pytest
from fastapi import Response
from fastapi import FastAPI, HTTPException, Response
from app.api.v1 import visualization
from app.services.earth_layer_cache import (
@@ -49,6 +51,19 @@ class FakeRedis:
deleted += 1 if self.store.pop(key, None) is not None else 0
return deleted
def eval(self, script: str, numkeys: int, *args):
self._maybe_fail()
keys, values = args[:numkeys], args[numkeys:]
if self.store.get(keys[0]) != values[0]:
return 0
if numkeys == 3:
self.setex(keys[1], values[1], values[3])
self.setex(keys[2], values[2], values[3])
return 1
if "'expire'" in script:
return 1
return self.delete(keys[0])
def scan_iter(self, match: str):
self._maybe_fail()
prefix = match.rstrip("*")
@@ -120,7 +135,9 @@ async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
@pytest.mark.asyncio
async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
key = earth_layer_cache.key("bgp-incidents", status="active")
fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
fake_cache_client.store[f"{key}:stale"] = json.dumps(
{"type": "FeatureCollection", "features": []}
)
async def builder():
raise RuntimeError("db exploded")
@@ -165,6 +182,7 @@ async def test_resolve_layer_payload_bypasses_redis_failure():
previous = earth_layer_cache._client
earth_layer_cache._client = FakeRedis(fail=True)
try:
async def builder():
return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
@@ -192,9 +210,13 @@ async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monk
monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
first_response = Response()
first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
first = await visualization.get_satellites_geojson(
limit=25, db=object(), response=first_response
)
second_response = Response()
second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
second = await visualization.get_satellites_geojson(
limit=25, db=object(), response=second_response
)
assert first == second
assert calls == 1
@@ -252,3 +274,284 @@ async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
assert calls == 1
assert first_response.headers["X-Planet-Cache"] == "refresh"
assert second_response.headers["X-Planet-Cache"] == "hit"
@pytest.mark.asyncio
async def test_concurrent_summary_requests_share_one_slow_build(monkeypatch):
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def build_summary(db):
nonlocal calls
calls += 1
started.set()
await finish.wait()
return {"stats": {"vessel_count": 42}}
monkeypatch.setattr(visualization, "_build_visualization_geo_summary", build_summary)
app = FastAPI()
app.include_router(visualization.router, prefix="/api/v1/visualization")
app.dependency_overrides[visualization.get_db] = lambda: object()
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
requests = [
asyncio.create_task(client.get("/api/v1/visualization/geo/summary")) for _ in range(6)
]
await asyncio.wait_for(started.wait(), 1)
# Keep the build open beyond the old 200 ms fallback to duplicate SQL.
await asyncio.sleep(0.25)
assert calls == 1
finish.set()
results = await asyncio.gather(*requests)
assert all(response.status_code == 200 for response in results)
assert all(response.json() == {"stats": {"vessel_count": 42}} for response in results)
assert sorted(response.headers["X-Planet-Cache"] for response in results) == [
"hit",
"hit",
"hit",
"hit",
"hit",
"refresh",
]
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
@pytest.mark.parametrize("error_type", [RuntimeError, TimeoutError])
async def test_shared_builder_error_is_not_retried(error_type):
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def builder():
nonlocal calls
calls += 1
started.set()
await finish.wait()
raise error_type("db failed")
requests = [
asyncio.create_task(
resolve_layer_payload(
key="shared-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
)
for _ in range(4)
]
await started.wait()
await asyncio.sleep(0)
finish.set()
results = await asyncio.gather(*requests, return_exceptions=True)
assert calls == 1
assert all(isinstance(result, error_type) for result in results)
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_concurrent_redis_failure_shares_bypass_result(fake_cache_client):
fake_cache_client.fail = True
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def builder():
nonlocal calls
calls += 1
started.set()
await finish.wait()
return {"count": 42}
requests = [
asyncio.create_task(
resolve_layer_payload(
key="shared-bypass", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
)
for _ in range(4)
]
await started.wait()
await asyncio.sleep(0)
finish.set()
results = await asyncio.gather(*requests)
assert calls == 1
assert all(result.state == "bypass" and result.payload == {"count": 42} for result in results)
@pytest.mark.asyncio
async def test_cancelled_waiter_does_not_cancel_owner():
started = asyncio.Event()
finish = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120)
async def builder():
started.set()
await finish.wait()
return {"count": 1}
owner = asyncio.create_task(
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
)
await started.wait()
waiter = asyncio.create_task(
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
)
await asyncio.sleep(0)
waiter.cancel()
with pytest.raises(asyncio.CancelledError):
await waiter
finish.set()
assert (await owner).payload == {"count": 1}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_cancelled_owner_allows_waiter_to_use_its_own_builder():
started = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120)
async def owner_builder():
started.set()
await asyncio.Event().wait()
async def waiter_builder():
return {"count": 2}
owner = asyncio.create_task(
resolve_layer_payload(key="cancel-owner", policy=policy, builder=owner_builder)
)
await started.wait()
waiter = asyncio.create_task(
resolve_layer_payload(key="cancel-owner", policy=policy, builder=waiter_builder)
)
await asyncio.sleep(0)
owner.cancel()
with pytest.raises(asyncio.CancelledError):
await owner
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 2}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_cross_process_lock_waits_for_payload_instead_of_building(fake_cache_client):
key = "remote-owner"
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
policy = EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005)
calls = 0
async def builder():
nonlocal calls
calls += 1
return {"count": -1}
waiter = asyncio.create_task(resolve_layer_payload(key=key, policy=policy, builder=builder))
await asyncio.sleep(0.03)
assert calls == 0
fake_cache_client.store[key] = json.dumps({"count": 3})
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 3}
assert calls == 0
@pytest.mark.asyncio
async def test_wait_timeout_does_not_bypass_other_owner(fake_cache_client):
key = "remote-timeout"
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
calls = 0
async def builder():
nonlocal calls
calls += 1
return {}
with pytest.raises(HTTPException) as error:
await resolve_layer_payload(
key=key,
policy=EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005, build_wait_seconds=0.02),
builder=builder,
)
assert error.value.status_code == 503
assert calls == 0
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_lost_owner_cannot_publish_or_release_new_lock(fake_cache_client):
key = "lock-replaced"
policy = EarthLayerCachePolicy(60, 120)
async def builder():
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "new-owner"
fake_cache_client.store[key] = json.dumps({"count": 99})
return {"count": 1}
result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
assert result.payload == {"count": 1}
assert result.state == "bypass"
assert fake_cache_client.store[earth_layer_cache.lock_key(key)] == "new-owner"
assert json.loads(fake_cache_client.store[key]) == {"count": 99}
@pytest.mark.asyncio
async def test_redis_publish_failure_returns_built_payload_without_retry(fake_cache_client):
calls = 0
async def builder():
nonlocal calls
calls += 1
fake_cache_client.fail = True
return {"count": 7}
result = await resolve_layer_payload(
key="publish-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
assert result.state == "bypass"
assert result.payload == {"count": 7}
assert calls == 1
@pytest.mark.asyncio
async def test_local_wait_timeout_leaves_owner_running():
started = asyncio.Event()
finish = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120, build_wait_seconds=0.02)
async def builder():
started.set()
await finish.wait()
return {"count": 1}
owner = asyncio.create_task(
resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
)
await started.wait()
with pytest.raises(HTTPException) as error:
await resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
assert error.value.status_code == 503
assert not owner.done()
finish.set()
assert (await owner).payload == {"count": 1}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_different_keys_can_build_concurrently():
started = set()
both_started = asyncio.Event()
async def build(key):
started.add(key)
if len(started) == 2:
both_started.set()
await asyncio.wait_for(both_started.wait(), 1)
return {"key": key}
results = await asyncio.gather(
*(
resolve_layer_payload(
key=key, policy=EarthLayerCachePolicy(60, 120), builder=lambda key=key: build(key)
)
for key in ("one", "two")
)
)
assert {result.payload["key"] for result in results} == {"one", "two"}

View File

@@ -3,6 +3,7 @@ from types import SimpleNamespace
import pytest
from app.api.v1.news import get_earth_feed
from app.services.earth_news import (
NewsFeedEndpoint,
NewsFeedSource,
@@ -1201,7 +1202,6 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
db=FakeDb(),
)
assert captured["freshness_region"] == "europe"
assert captured["items_region"] == "europe"
assert captured["items_categories"] == {"business", "ecommerce"}
assert captured["items_source_ids"][0] is None
@@ -1237,71 +1237,41 @@ async def test_news_source_test_treats_type_reference_as_non_fetching():
@pytest.mark.asyncio
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
async def test_earth_news_feed_get_does_not_fetch_write_or_queue_rss(monkeypatch):
db = object()
source = NewsFeedSource(
id="test-feed",
name="Test Feed",
region="global",
homepage_url="https://example.com",
feed_url="https://example.com/rss.xml",
read_calls = 0
async def fail_side_effect(*_args, **_kwargs):
raise AssertionError("RSS feed reads must not fetch, persist, or enqueue work")
async def empty_news_items(*_args, **_kwargs):
nonlocal read_calls
read_calls += 1
return []
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_side_effect)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fail_side_effect)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fail_side_effect)
monkeypatch.setattr(
"app.services.earth_news_store.list_earth_news_items", empty_news_items
)
item = ParsedNewsItem(
id="test-feed:init",
title="Initial RSS story",
summary="Initial summary",
url="https://example.com/init",
source="Test Feed",
feed_name="Test Feed",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
payload = await get_earth_feed(
lat=None,
lon=None,
region=None,
categories=None,
sources=None,
limit=None,
locale=None,
db=db,
)
item.location_patch = {
"latitude": 20.0,
"longitude": 0.0,
"location_label": "全球",
"location_source": "region_anchor",
"verified": False,
"location_meta": {"target": None, "anchor": {"region": "global"}},
}
upserted = []
enqueued = []
async def fake_get_earth_news_freshness(_db, *, active_region):
return 0, None
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
async def fake_upsert_earth_news_items(_db, items):
upserted.extend(items)
return len(items)
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
return [item]
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued.append(payload)
return True
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
payload = await get_earth_news_payload(db=db)
assert upserted[0].id == "test-feed:init"
assert payload["items"][0]["id"] == "test-feed:init"
assert payload["items"][0]["verified"] is False
assert enqueued[0]["id"] == "test-feed:init"
assert payload["items"] == []
assert read_calls == 1
@pytest.mark.asyncio
async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
async def test_earth_news_payload_does_not_supplement_stale_database_items(monkeypatch):
db = object()
old_item = ParsedNewsItem(
id="db:old",
@@ -1314,41 +1284,17 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
homepage_url="https://example.com",
published_at=datetime(2026, 5, 14, 3, 0, tzinfo=UTC),
)
old_item.location_patch = {
"latitude": 20.0,
"longitude": 0.0,
"location_label": "全球",
"location_source": "region_anchor",
"verified": False,
"location_meta": {"target": None, "anchor": {"region": "global"}},
}
fetched = []
async def fail_fetch(*_args, **_kwargs):
raise AssertionError("stale RSS supplementation belongs to the background worker")
async def fake_get_earth_news_freshness(_db, *, active_region):
return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
fetched.append(True)
return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
async def fake_upsert_earth_news_items(_db, items):
return len(items)
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
return [old_item]
async def fake_enqueue_target_location_job(_payload, **_kwargs):
return True
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
payload = await get_earth_news_payload(db=db)
payload = await get_earth_news_payload(db=db, source_ids={"stored"})
assert fetched == [True]
assert payload["items"][0]["id"] == "db:old"

View File

@@ -0,0 +1,86 @@
from typing import Any
import httpx
import pytest
from app.services.earth_news import (
NewsFeedEndpoint,
NewsFeedSource,
ParsedNewsItem,
_fetch_single_feed_url,
_serialize_item,
)
async def fetch_feed(
content: bytes, content_type: str = "text/xml"
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
source = NewsFeedSource(
id="ebrun",
name="亿邦动力",
region="asia-pacific",
feed_url="https://example.com/rss.xml",
homepage_url="https://example.com/",
)
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
transport = httpx.MockTransport(
lambda request: httpx.Response(
200, content=content, headers={"content-type": content_type}
)
)
async with httpx.AsyncClient(transport=transport) as client:
return await _fetch_single_feed_url(client, source, feed)
@pytest.mark.asyncio
@pytest.mark.parametrize(
("encoding", "declaration", "content_type"),
[
("gbk", 'encoding="GBK"', "text/xml"),
("gb2312", "encoding='GB2312'", "application/rss+xml"),
("gb18030", 'encoding="GB18030"', "text/xml"),
("gbk", "", "text/xml; charset=gbk"),
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
("utf-8", "", "text/xml"),
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
("utf-16", 'encoding="UTF-16"', "text/xml"),
("utf-32", 'encoding="UTF-32"', "text/xml"),
],
)
async def test_feed_encoding_preserves_chinese_in_display(
encoding: str, declaration: str, content_type: str
):
title = "618电商战报:零售额增长266%"
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
xml = f'''<?xml version="1.0" {declaration}?>
<rss version="2.0"><channel><item>
<title><![CDATA[{title}]]></title>
<description><![CDATA[<p>{summary}</p>]]></description>
<link>https://example.com/story</link>
</item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
assert error is None
assert health["ok"] is True
assert len(items) == 1
assert items[0].title == title
assert items[0].summary == summary
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
assert payload["display_title"] == title
assert payload["display_summary"] == summary
@pytest.mark.asyncio
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
<rss><channel><item><title>电商新闻</title>
<link>https://example.com/story</link></item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode("gbk"))
assert items == []
assert error
assert health["ok"] is False
assert health["status"] == "format_error"

View File

@@ -0,0 +1,525 @@
from contextlib import asynccontextmanager
from datetime import UTC, datetime
import json
import pytest
from app.services.earth_news import NewsFeedEndpoint, NewsFeedSource
from app.services import earth_news_refresh
from app.services.scheduler import (
EARTH_NEWS_REFRESH_JOB_ID,
_earth_news_region_refresh_job_id,
schedule_earth_news_region_refresh,
start_scheduler,
sync_scheduler_with_datasources,
)
class FakeRedis:
def __init__(self) -> None:
self.values: dict[str, str] = {}
def set(self, key: str, value: str, *, nx: bool = False, ex: int | None = None) -> bool:
del ex
if nx and key in self.values:
return False
self.values[key] = value
return True
def exists(self, key: str) -> bool:
return key in self.values
def eval(self, script: str, _key_count: int, key: str, token: str, *_args: object) -> int:
if self.values.get(key) != token:
return 0
if "redis.call('del'" in script:
del self.values[key]
return 1
class FakeStatusRedis:
def __init__(self) -> None:
self.values: dict[str, str] = {}
self.expirations: dict[str, int] = {}
def get(self, key: str):
return self.values.get(key)
def eval(self, script: str, key_count: int, *args):
keys = args[:key_count]
values = args[key_count:]
if "local active_id" in script:
active_key, status_key = keys
refresh_id, encoded, active_ttl, status_ttl, status_prefix = values
active_id = self.values.get(active_key)
if active_id:
active_status = self.values.get(f"{status_prefix}{active_id}")
if active_status:
status = json.loads(active_status)["status"]
if status in {"pending", "running"}:
return [active_id, active_status, 0]
self.values.pop(active_key, None)
self.values[active_key] = refresh_id
self.values[status_key] = encoded
self.expirations[active_key] = int(active_ttl)
self.expirations[status_key] = int(status_ttl)
return [refresh_id, encoded, 1]
active_key, status_key = keys
encoded, refresh_id, status_ttl, status, active_ttl = values
if self.values.get(active_key) != refresh_id:
return 0
self.values[status_key] = encoded
self.expirations[status_key] = int(status_ttl)
if status in {"succeeded", "failed"}:
self.values.pop(active_key, None)
else:
self.expirations[active_key] = int(active_ttl)
return 1
class FakeSession:
def __init__(self) -> None:
self.commits = 0
self.rollbacks = 0
async def commit(self) -> None:
self.commits += 1
async def rollback(self) -> None:
self.rollbacks += 1
@pytest.mark.asyncio
async def test_region_refresh_fetches_only_missing_feed_coverage(monkeypatch):
redis_client = FakeRedis()
db = FakeSession()
feeds = (
NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global"),
NewsFeedEndpoint("feed-b", "Feed B", "https://example.com/b.xml", region="global"),
)
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url=feeds[0].url,
homepage_url="https://example.com",
feeds=feeds,
)
fetched_source_feeds: list[list[str]] = []
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
assert config_payload == {"sources": []}
return [source]
async def fake_freshness(_db, *, active_region):
assert active_region == "global"
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
assert active_region == "global"
assert recent_after is not None
return {("source-a", "feed-a")}
async def fake_fetch(sources, **_kwargs):
fetched_source_feeds.extend([[feed.id for feed in item.feeds] for item in sources])
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
async def fake_record_health(_db, _health):
return None
async def fake_enqueue(_items):
return None
async def fake_upsert(_db, items):
assert items == []
return 0
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr(
"app.services.earth_news.record_earth_news_sources_health", fake_record_health
)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_enqueue)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "completed"
assert result["successful_sources"] == 1
assert fetched_source_feeds == [["feed-b"]]
assert db.commits == 1
assert earth_news_refresh._lock_key("global") not in redis_client.values
@pytest.mark.asyncio
async def test_region_refresh_reports_failure_when_all_sources_time_out(monkeypatch):
redis_client = FakeRedis()
db = FakeSession()
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global")
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url=feed.url,
homepage_url="https://example.com",
feeds=(feed,),
)
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
return [source]
async def fake_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
return set()
async def fake_fetch(*_args, **_kwargs):
return (
[],
["private upstream details"],
{
"source-a": {
"source_id": "source-a",
"ok": False,
"status": "timeout",
"feed_results": [{"feed_id": "feed-a", "ok": False, "status": "timeout"}],
}
},
)
async def fake_noop(*_args, **_kwargs):
return None
async def fake_upsert(_db, _items):
return 0
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
result = await earth_news_refresh.refresh_earth_news_region("global", force=True)
assert result["state"] == "failed"
assert result["error_code"] == "timeout"
assert db.commits == 1
@pytest.mark.asyncio
async def test_database_commit_failure_sets_refresh_failed_status(monkeypatch):
class CommitFailureSession(FakeSession):
async def commit(self) -> None:
raise RuntimeError("database commit failed with sensitive context")
redis_client = FakeRedis()
db = CommitFailureSession()
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="europe")
source = NewsFeedSource(
id="source-a",
name="Source A",
region="europe",
feed_url=feed.url,
homepage_url="https://example.com",
feeds=(feed,),
)
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
statuses = []
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
return [source]
async def fake_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
return set()
async def fake_fetch(*_args, **_kwargs):
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
async def fake_noop(*_args, **_kwargs):
return None
async def fake_upsert(_db, _items):
return 1
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
monkeypatch.setattr(scheduler_module, "touch_earth_news_refresh_status", lambda *_args: True)
monkeypatch.setattr(
scheduler_module,
"set_earth_news_refresh_status",
lambda refresh_id, region, status, *, error=None: statuses.append(
{"refresh_id": refresh_id, "region": region, "status": status, "error": error}
),
)
scheduler_module._pending_earth_news_refreshes["europe"] = {
"force": True,
"refresh_id": "refresh-commit-failure",
}
await scheduler_module._run_earth_news_region_refresh("europe")
assert [entry["status"] for entry in statuses] == ["running", "failed"]
assert statuses[-1]["error"]["code"] == "refresh_failed"
assert "sensitive context" not in repr(statuses)
assert not redis_client.values
@pytest.mark.asyncio
async def test_region_refresh_skips_when_cross_process_lock_is_held(monkeypatch):
redis_client = FakeRedis()
redis_client.values[earth_news_refresh._lock_key("global")] = "other-owner"
def fail_session_factory():
raise AssertionError("a locked refresh must not open a database session")
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "busy"
def test_failed_feed_enters_cooldown_without_blocking_other_sources():
redis_client = FakeRedis()
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url="https://example.com/a.xml",
homepage_url="https://example.com",
feeds=(NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml"),),
)
health = {
"source-a": {
"ok": False,
"feed_results": [{"feed_id": "feed-a", "ok": False}],
}
}
earth_news_refresh._record_failed_feed_cooldowns(
redis_client,
active_region="global",
health_by_source=health,
sources=[source],
)
selected, cooling = earth_news_refresh._select_feeds(
[source],
active_region="global",
required_keys={("source-a", "feed-a")},
redis_client=redis_client,
)
assert selected == []
assert cooling == {("source-a", "feed-a")}
@pytest.mark.asyncio
async def test_region_refresh_skips_when_redis_coordination_is_unavailable(monkeypatch):
def fail_redis_client():
raise RuntimeError("redis unavailable")
def fail_session_factory():
raise AssertionError("uncoordinated processes must not start duplicate RSS fetches")
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", fail_redis_client)
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "coordination_unavailable"
def test_refresh_status_deduplicates_active_region_and_bounds_completed_results(monkeypatch):
redis_client = FakeStatusRedis()
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
first, created = earth_news_refresh.create_earth_news_refresh_status("global")
duplicate, duplicate_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert created is True
assert duplicate_created is False
assert duplicate == first
earth_news_refresh.set_earth_news_refresh_status(first["refresh_id"], "global", "running")
running, running_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert running_created is False
assert running["status"] == "running"
completed = earth_news_refresh.set_earth_news_refresh_status(
first["refresh_id"], "global", "succeeded"
)
assert completed["status"] == "succeeded"
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
second, second_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert second_created is True
assert second["refresh_id"] != first["refresh_id"]
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
assert redis_client.expirations[earth_news_refresh._refresh_active_key("global")] == 120
assert (
redis_client.expirations[earth_news_refresh._refresh_status_key(second["refresh_id"])]
== 600
)
status_keys = [
key
for key in redis_client.values
if key.startswith(earth_news_refresh.REFRESH_STATUS_PREFIX)
]
assert len(status_keys) == 2
pending, pending_created = earth_news_refresh.create_earth_news_refresh_status("europe")
assert pending_created is True
redis_client.values.pop(earth_news_refresh._refresh_active_key("europe"), None)
expired = earth_news_refresh.get_earth_news_refresh_status(pending["refresh_id"])
assert expired["status"] == "failed"
assert expired["error"]["code"] == "expired"
@pytest.mark.asyncio
async def test_earth_news_refresh_scheduler_job_survives_datasource_sync(monkeypatch):
class FakeJob:
def __init__(self, job_id: str) -> None:
self.id = job_id
class FakeScheduler:
def __init__(self) -> None:
self.running = False
self.jobs = {
"obsolete": FakeJob("obsolete"),
_earth_news_region_refresh_job_id("global"): FakeJob(
_earth_news_region_refresh_job_id("global")
),
}
self.added: dict[str, object] = {}
def add_job(self, _func, **kwargs):
self.added.update(kwargs)
self.jobs[kwargs["id"]] = FakeJob(kwargs["id"])
def start(self) -> None:
self.running = True
def get_jobs(self):
return list(self.jobs.values())
def remove_job(self, job_id: str) -> None:
self.jobs.pop(job_id, None)
class FakeResult:
def scalars(self):
return self
def all(self):
return []
class FakeDb:
async def execute(self, _query):
return FakeResult()
fake_scheduler = FakeScheduler()
@asynccontextmanager
async def fake_session_factory():
yield FakeDb()
monkeypatch.setattr("app.services.scheduler.scheduler", fake_scheduler)
monkeypatch.setattr("app.services.scheduler.async_session_factory", fake_session_factory)
start_scheduler()
await sync_scheduler_with_datasources()
assert fake_scheduler.added["id"] == EARTH_NEWS_REFRESH_JOB_ID
assert fake_scheduler.added["next_run_time"] <= datetime.now(UTC)
assert EARTH_NEWS_REFRESH_JOB_ID in fake_scheduler.jobs
assert _earth_news_region_refresh_job_id("global") in fake_scheduler.jobs
assert "obsolete" not in fake_scheduler.jobs
@pytest.mark.asyncio
async def test_region_refresh_requests_are_merged_and_force_is_preserved(monkeypatch):
class FakeScheduler:
running = True
def __init__(self) -> None:
self.jobs: dict[str, tuple[object, dict[str, object]]] = {}
def add_job(self, func, **kwargs):
self.jobs[kwargs["id"]] = (func, kwargs)
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
fake_scheduler = FakeScheduler()
monkeypatch.setattr(scheduler_module, "scheduler", fake_scheduler)
captured: list[tuple[str, bool]] = []
async def fake_refresh(region: str, *, force: bool = False):
captured.append((region, force))
return {"locked": False, "fetched": 0, "upserted": 0}
monkeypatch.setattr(scheduler_module, "refresh_earth_news_region", fake_refresh)
assert schedule_earth_news_region_refresh("global") is True
assert schedule_earth_news_region_refresh("global", force=True) is False
assert len(fake_scheduler.jobs) == 1
job_id = _earth_news_region_refresh_job_id("global")
func, kwargs = fake_scheduler.jobs.pop(job_id)
await func(**kwargs["kwargs"])
assert captured == [("global", True)]
assert job_id not in fake_scheduler.jobs

View File

@@ -0,0 +1,140 @@
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from app.services import earth_db_change_listener as listener_module
from app.services.earth_db_change_listener import EarthDbChangeDispatcher, EarthDbChangeListener
from app.services.earth_outbox_maintenance import (
DELETE_BATCH_SQL,
compact_consumed_events,
ensure_retention_index,
purge_consumed_events,
)
class FakeConnection:
def __init__(self, *, index_state: str = "ready", batch_counts: list[int] | None = None):
self.index_state = index_state
self.batch_counts = list(batch_counts or [])
self.calls: list[tuple[str, tuple[object, ...]]] = []
self.retained = {"total": 13, "pending": 3}
self.restored = dict(self.retained)
self.rolled_back = False
async def fetchrow(self, query: str, *args: object):
self.calls.append((query, args))
if "pg_index" in query:
if self.index_state == "missing":
return None
return {"indisvalid": self.index_state == "ready", "indisready": True}
if "FROM earth_outbox_retained" in query:
return self.retained
return self.restored
async def fetchval(self, query: str, *args: object):
self.calls.append((query, args))
if query == DELETE_BATCH_SQL:
return self.batch_counts.pop(0)
return True
async def execute(self, query: str, *args: object):
self.calls.append((query, args))
if "CREATE INDEX CONCURRENTLY" in query:
self.index_state = "ready"
return "OK"
def transaction(self):
connection = self
class Transaction:
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, _exc, _traceback):
connection.rolled_back = exc_type is not None
return Transaction()
@pytest.mark.asyncio
async def test_retention_skips_without_index():
conn = FakeConnection(index_state="missing")
assert await purge_consumed_events(conn, retention_days=1) == 0
assert not any(query == DELETE_BATCH_SQL for query, _ in conn.calls)
@pytest.mark.asyncio
async def test_retention_batches_only_consumed_old_rows():
conn = FakeConnection(batch_counts=[1000, 19])
deleted = await purge_consumed_events(
conn, retention_days=1, batch_size=1000, max_batches=10, pause_seconds=0
)
assert deleted == 1019
calls = [args for query, args in conn.calls if query == DELETE_BATCH_SQL]
assert len(calls) == 2
assert all(isinstance(args[0], datetime) and args[0].tzinfo is UTC for args in calls)
assert "consumed_at IS NOT NULL AND consumed_at < $1" in DELETE_BATCH_SQL
assert "FOR UPDATE SKIP LOCKED" in DELETE_BATCH_SQL
@pytest.mark.asyncio
async def test_index_creation_is_concurrent_and_repairs_invalid_index():
conn = FakeConnection(index_state="invalid")
await ensure_retention_index(conn)
statements = [query for query, _ in conn.calls]
assert any("DROP INDEX CONCURRENTLY" in query for query in statements)
assert any("CREATE INDEX CONCURRENTLY" in query for query in statements)
@pytest.mark.asyncio
async def test_compact_retains_pending_and_recent_consumed_in_one_transaction():
conn = FakeConnection()
assert await compact_consumed_events(conn, retention_days=1) == (13, 3)
statements = [query for query, _ in conn.calls]
assert next(i for i, query in enumerate(statements) if query.startswith("LOCK TABLE")) < next(
i for i, query in enumerate(statements) if query.startswith("TRUNCATE TABLE")
)
assert any("consumed_at IS NULL OR consumed_at >= $1" in query for query in statements)
assert "TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT" in statements
assert not conn.rolled_back
@pytest.mark.asyncio
async def test_compact_verification_failure_rolls_back():
conn = FakeConnection()
conn.restored = {"total": 12, "pending": 2}
with pytest.raises(RuntimeError, match="verification failed"):
await compact_consumed_events(conn, retention_days=1)
assert conn.rolled_back
@pytest.mark.asyncio
async def test_listener_runs_retention_at_most_once_per_interval(monkeypatch):
calls: list[str] = []
async def acquire(_conn):
calls.append("acquire")
return True
async def purge(_conn, *, retention_days):
assert retention_days == 1
calls.append("purge")
async def release(_conn):
calls.append("release")
monkeypatch.setattr(listener_module, "monotonic", lambda: 100.0)
monkeypatch.setattr(listener_module, "try_maintenance_lock", acquire)
monkeypatch.setattr(listener_module, "purge_consumed_events", purge)
monkeypatch.setattr(listener_module, "release_maintenance_lock", release)
monkeypatch.setattr(listener_module, "configured_retention_days", lambda: 1)
dispatcher = EarthDbChangeDispatcher(broadcast_earth_update=lambda _: None)
listener = EarthDbChangeListener(dsn="postgresql://example/db", dispatcher=dispatcher)
listener._connection = FakeConnection()
await listener._maybe_purge_outbox()
await listener._maybe_purge_outbox()
assert calls == ["acquire", "purge", "release"]

View File

@@ -10,7 +10,7 @@ from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
from app.core.request_context import set_request_id
from app.services import business_logs
from app.services import persistent_logs
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup, SystemLog
def _capture_output(callback):
@@ -100,6 +100,23 @@ def test_business_context_redacts_nested_sensitive_values():
assert context["nested"]["safe"] == "visible"
@pytest.mark.asyncio
async def test_business_log_preserves_events_in_isolated_store(persisted_test_logs):
await business_logs.emit_business_log(
get_logger("tests.persistence"),
event="test.business.completed",
message="Business action completed",
category="test",
context={"api_key": "test-secret", "record_count": 2},
)
logs = [entry for entry in persisted_test_logs if isinstance(entry, SystemLog)]
assert len(logs) == 1
assert logs[0].event == "test.business.completed"
assert logs[0].context == {"api_key": "[REDACTED]", "record_count": 2}
assert any(isinstance(entry, ObservabilityEventGroup) for entry in persisted_test_logs)
def test_observability_fingerprint_normalizes_hls_fragments():
first = persistent_logs.build_observability_fingerprint(
source="earth-client",

View File

@@ -0,0 +1,142 @@
from types import SimpleNamespace
from uuid import UUID
import importlib
import pytest
from fastapi import HTTPException
from app.api.v1 import earth, news
@pytest.mark.asyncio
async def test_post_news_refresh_only_schedules_background_region_check(monkeypatch):
scheduled = []
def request(region: str):
scheduled.append(region)
return {"refresh_id": "refresh-europe", "region": region, "status": "pending"}
async def fail_network_refresh(*_args, **_kwargs):
raise AssertionError("the POST handler must not perform RSS requests")
monkeypatch.setattr("app.services.scheduler.request_earth_news_region_refresh", request)
monkeypatch.setattr("app.services.scheduler.refresh_earth_news_region", fail_network_refresh)
response = await news.request_earth_news_refresh(
news.EarthNewsRefreshRequest(region="Europe", lat=48.8, lon=2.3)
)
assert response == {"refresh_id": "refresh-europe", "region": "europe", "status": "pending"}
assert scheduled == ["europe"]
refresh_route = next(route for route in news.router.routes if route.path == "/refresh")
assert refresh_route.status_code == 202
@pytest.mark.asyncio
async def test_post_news_refresh_infers_region_and_rejects_unknown_region(monkeypatch):
scheduled = []
monkeypatch.setattr(
"app.services.scheduler.request_earth_news_region_refresh",
lambda region: scheduled.append(region)
or {"refresh_id": "refresh-americas", "region": region, "status": "pending"},
)
response = await news.request_earth_news_refresh(
news.EarthNewsRefreshRequest(lat=40.7, lon=-74.0)
)
assert response["region"] == "americas"
assert response["status"] == "pending"
assert scheduled == ["americas"]
with pytest.raises(HTTPException) as error:
await news.request_earth_news_refresh(news.EarthNewsRefreshRequest(region="unknown"))
assert error.value.status_code == 422
@pytest.mark.asyncio
async def test_get_news_refresh_returns_status_and_expired_shape(monkeypatch):
current = {"refresh_id": "abc", "region": "global", "status": "running"}
monkeypatch.setattr(
"app.services.earth_news_refresh.get_earth_news_refresh_status",
lambda _refresh_id: current,
)
response = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
assert response is current
assert response["status"] == "running"
monkeypatch.setattr(
"app.services.earth_news_refresh.get_earth_news_refresh_status", lambda _id: None
)
expired = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
assert expired == {
"refresh_id": UUID("00000000-0000-0000-0000-000000000abc").hex,
"region": None,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
@pytest.mark.asyncio
async def test_database_failure_sets_terminal_failed_status_without_leaking_error(monkeypatch):
scheduler = importlib.import_module("app.services.scheduler")
statuses = []
async def fail_refresh(_region, *, force=False):
raise RuntimeError("postgres://private-token@host/database: commit failed")
monkeypatch.setattr(scheduler, "refresh_earth_news_region", fail_refresh)
monkeypatch.setattr(scheduler, "touch_earth_news_refresh_status", lambda *_args: True)
monkeypatch.setattr(
scheduler,
"set_earth_news_refresh_status",
lambda refresh_id, region, state, *, error=None: statuses.append(
{"refresh_id": refresh_id, "region": region, "status": state, "error": error}
),
)
scheduler._pending_earth_news_refreshes["europe"] = {
"force": True,
"refresh_id": "refresh-db-failure",
}
await scheduler._run_earth_news_region_refresh("europe")
assert [item["status"] for item in statuses] == ["running", "failed"]
assert statuses[-1]["error"] == {
"code": "refresh_failed",
"message": "Could not refresh news. Please try again.",
}
assert "private-token" not in repr(statuses)
@pytest.mark.asyncio
async def test_news_source_save_and_reset_schedule_immediate_refresh(monkeypatch):
scheduled = []
monkeypatch.setattr(
earth,
"save_earth_news_sources_payload",
lambda _db, _payload: _async_result({"status": "updated"}),
)
monkeypatch.setattr(
earth,
"reset_earth_news_sources_payload",
lambda _db: _async_result({"status": "reset"}),
)
monkeypatch.setattr(
"app.services.scheduler.schedule_earth_news_refresh_for_all_regions",
lambda *, force=False: scheduled.append(force),
)
await earth.update_earth_news_sources(
SimpleNamespace(model_dump=lambda: {}), _current_user=None, db=object()
)
await earth.reset_earth_news_sources(_current_user=None, db=object())
assert scheduled == [True, True]
async def _async_result(value):
return value

View File

@@ -0,0 +1,37 @@
from app.services.tv_streams import (
CCTV4_HOMEPAGE,
CCTV4_STREAM,
UNSUPPORTED_CCTV4_STREAM,
normalize_tv_settings,
normalize_tv_source,
)
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
settings = normalize_tv_settings(None)
assert settings["default_source_id"] == "aljazeera-mubasher"
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert cctv["source_type"] == "hls"
assert cctv["stream_url"] == CCTV4_STREAM
assert "美洲版" in cctv["name"]
assert cctv["use_proxy"] is False
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
def test_saved_legacy_default_is_corrected_without_mutating_input():
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
source = normalize_tv_source(saved)
assert source["source_type"] == "hls"
assert source["stream_url"] == CCTV4_STREAM
assert source["use_proxy"] is False
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert source["stream_url"] == custom["stream_url"]
assert source["source_type"] == "hls"
assert source["use_proxy"] is True
assert settings["auto_fallback"] is False

View File

@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
longitude=-90.05,
metadata={
"organization": "xAI",
"location": "3231 Riverport Rd, Memphis, TN 38109",
"gpu_type": "H100",
"gpu_count": 100000,
"value": "20000",
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
compute_center_locations.set_compute_center_location_cache({})
def test_epoch_country_level_coordinates_are_unresolved():
record = _build_record(
record_id=33,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Meta 100k",
country="United States",
city="",
latitude=39.381266,
longitude=-97.922211,
metadata={
"location": "United States of America",
"raw_country": "United States of America",
},
)
payload = convert_compute_centers_to_geojson([record])
assert payload["features"] == []
assert len(payload["unresolved"]) == 1
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
def test_epoch_city_coordinates_are_not_labeled_precise():
record = _build_record(
record_id=34,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Seoul Cluster",
country="South Korea",
city="",
latitude=37.5665,
longitude=126.978,
metadata={"location": "Seoul", "raw_country": "South Korea"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.is_resolved
assert result.location.location_precision == "city"
assert result.location.needs_confirmation is True
assert result.location.is_estimated is True
assert result.location.city is None
assert result.location.matched_location_name == "Seoul"
assert "Epoch Location: Seoul" in result.location.location_source_note
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
record = _build_record(
record_id=37,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Petrobras Fênix",
country="Brazil",
city="",
latitude=-22.909534,
longitude=-43.209934,
metadata={
"location": "Rio de Janeiro, Brazil",
"raw_country": "Brazil",
},
)
payload = convert_compute_centers_to_geojson([record])
feature = payload["features"][0]
assert feature["properties"]["location_precision"] == "city"
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
"location_source_note"
]
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-35": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-35",
"name": "Confirmed Site",
"latitude": 35.0,
"longitude": 139.0,
"precision": "site",
"confidence": 1.0,
"location_source": "official_facility_registry",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=35,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Japan Cluster",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
"raw_country": "Japan",
},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "stored_compute_center_location"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-36": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-36",
"name": "Legacy Country Point",
"latitude": 36.386493,
"longitude": 138.59223,
"precision": "precise",
"confidence": 1.0,
"location_source": "source_coordinates",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=36,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Legacy Japan Cluster",
country="Japan",
city="",
latitude=36.386493,
longitude=138.59223,
metadata={"location": "Japan", "raw_country": "Japan"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location is None
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_verified_sites_replace_epoch_country_center_coordinates():
specifications = [
(
"epoch_jamstec_zettascaler_2_0_gyoukou",
"Gyoukou 2.0",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_jamstec_zettascaler_2_2_gyoukou",
"Gyoukou 2.2",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_argonne_nl_sophia__formerly_theta_",
"Sophia",
"United States of America",
39.381266,
-97.922211,
[-87.984498648496, 41.718109383287235],
),
]
records = []
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
specifications, start=40
):
record = _build_record(
record_id=index,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name=name,
country=country,
city="",
latitude=latitude,
longitude=longitude,
metadata={"location": country, "raw_country": country},
)
record.source_id = source_id
records.append(record)
payload = convert_compute_centers_to_geojson(records)
assert payload["unresolved"] == []
assert len(payload["features"]) == 3
for feature, specification in zip(payload["features"], specifications):
assert feature["geometry"]["coordinates"] == specification[-1]
assert feature["properties"]["location_precision"] == "site"
assert feature["properties"]["location_source"] == "official_facility_registry"
assert "https://" in feature["properties"]["location_source_note"]
def test_verified_site_beats_epoch_city_level_source_coordinates():
record = _build_record(
record_id=44,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Gyoukou 2.2",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
)
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "official_facility_registry"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (
35.380924,
139.625008,
)
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-32": {
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
@pytest.mark.asyncio
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
@pytest.mark.asyncio
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
llm_candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
@pytest.mark.asyncio
async def test_compute_centers_geojson_endpoint_returns_stats():
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
@pytest.mark.asyncio
async def test_visualization_geo_summary_returns_counts(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
# Exercise the fixture DB, never the running development server's Redis payload.
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
class _FakeSession:
async def execute(self, query):
query_text = str(query).lower()
if "group by collected_data.source" in query_text:
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
if "bgp_incidents" in query_text:
return _ScalarResult(scalar_value=2)
if "bgp_anomalies" in query_text:
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
def _fake_ror(query):
assert query == "Oak Ridge National Laboratory"
return {
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
assert "FORBIDDEN_PRECISIONS" in text, (
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
)
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
from app.core.compute_types import compute_site_type
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
metadata={"location_precision": "site"})
result = convert_compute_centers_to_geojson([record])
assert compute_site_type(record) == kind
assert result["features"][0]["properties"]["site_type"] == kind
record.source = "epoch_ai_gpu"
record.data_type = "gpu_cluster"
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
assert compute_site_type(record) == kind
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind

View File

@@ -0,0 +1,156 @@
from collections.abc import AsyncIterator, Iterator, Sequence
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import Select, create_engine
from sqlalchemy.engine import Result, ScalarResult
from sqlalchemy.orm import Session
from app.api.v1 import visualization
from app.models.collected_data import CollectedData
class SatelliteRows:
def __init__(self, result: ScalarResult[CollectedData]) -> None:
self.result = result
self.closed = False
async def partitions(self, size: int) -> AsyncIterator[Sequence[CollectedData]]:
for batch in self.result.partitions(size):
yield batch
async def close(self) -> None:
self.result.close()
self.closed = True
class SatelliteSession:
def __init__(self, session: Session) -> None:
self.session = session
self.streams: list[SatelliteRows] = []
async def stream_scalars(
self, statement: Select, *, execution_options: dict[str, int]
) -> SatelliteRows:
rows = SatelliteRows(self.session.scalars(statement, execution_options=execution_options))
self.streams.append(rows)
return rows
async def execute(self, statement: Select) -> Result:
return self.session.execute(statement)
@pytest.fixture
def satellite_db() -> Iterator[SatelliteSession]:
engine = create_engine("sqlite://")
CollectedData.__table__.create(engine)
with Session(engine) as session:
yield SatelliteSession(session)
engine.dispose()
def satellite(
record_id: int, *, current: bool = True, task_id: int = 1,
name: str | None = None, source: str = "celestrak_tle",
) -> CollectedData:
return CollectedData(
id=record_id,
source=source,
source_id=str(record_id),
data_type="satellite_tle",
name=name or f"SAT-{record_id}",
is_current=current,
task_id=task_id,
collected_at=datetime(2026, 1, 1, tzinfo=UTC) + timedelta(days=task_id),
extra_data={
"norad_cat_id": record_id + 40000,
"tle_line1": f"line-1-{record_id}",
"tle_line2": f"line-2-{record_id}",
},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("limit", [None, 1, 501])
async def test_streamed_satellites_preserve_all_rows_order_tle_and_limit(
satellite_db: SatelliteSession, limit: int | None,
) -> None:
satellite_db.session.add_all([satellite(record_id) for record_id in range(1, 1004)])
satellite_db.session.add_all([
satellite(2000, name="Unknown"),
satellite(3000, current=False),
satellite(4000, source="other"),
])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=limit, db=satellite_db)
expected_ids = list(range(1003, 0, -1))[:limit]
assert payload["count"] == len(expected_ids)
assert [feature["properties"]["id"] for feature in payload["features"]] == expected_ids
assert [feature["properties"]["tle_line1"] for feature in payload["features"]] == [
f"line-1-{record_id}" for record_id in expected_ids
]
assert len(satellite_db.streams) == 1
assert satellite_db.streams[0].closed
@pytest.mark.asyncio
async def test_streamed_satellites_fall_back_to_latest_non_unknown_task(
satellite_db: SatelliteSession,
) -> None:
satellite_db.session.add_all([
satellite(1, current=False, task_id=1),
satellite(2, current=False, task_id=2),
satellite(3, current=False, task_id=2),
satellite(4, current=False, task_id=3, name="Unknown"),
])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=1, db=satellite_db)
assert payload["count"] == 1
assert payload["features"][0]["properties"]["id"] == 3
assert all(stream.closed for stream in satellite_db.streams)
@pytest.mark.asyncio
async def test_unrenderable_current_rows_do_not_restore_historical_satellites(
satellite_db: SatelliteSession,
) -> None:
current = satellite(2)
current.extra_data = {}
satellite_db.session.add_all([satellite(1, current=False), current])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert payload == {"type": "FeatureCollection", "features": [], "count": 0}
assert len(satellite_db.streams) == 1
@pytest.mark.asyncio
async def test_empty_satellite_catalog_returns_empty_collection(
satellite_db: SatelliteSession,
) -> None:
payload = await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert payload == {"type": "FeatureCollection", "features": [], "count": 0}
assert satellite_db.streams[0].closed
@pytest.mark.asyncio
async def test_satellite_cursor_closes_if_conversion_fails(
satellite_db: SatelliteSession, monkeypatch: pytest.MonkeyPatch,
) -> None:
satellite_db.session.add(satellite(1))
satellite_db.session.commit()
def fail_conversion(_records: Sequence[CollectedData]) -> None:
raise ValueError("invalid orbital data")
monkeypatch.setattr(visualization, "convert_satellite_to_geojson", fail_conversion)
with pytest.raises(ValueError, match="invalid orbital data"):
await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert satellite_db.streams[0].closed

View File

@@ -25,6 +25,25 @@
{
"id": "china-taiwan-penghu",
"name": "Taiwan and Penghu",
"sourceIsoA3": ["TWN"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-hong-kong",
"name": "Hong Kong",
"sourceIsoA3": ["HKG"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-macao",
"name": "Macao",
"sourceIsoA3": ["MAC"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
@@ -57,6 +76,7 @@
{
"id": "kosovo",
"name": "Kosovo",
"sourceIsoA3": ["KOS", "XKS"],
"action": "render_as_disputed_with_parent",
"parentIsoA3": "SRB",
"hoverIsoA3": "SRB",

View File

@@ -1,5 +1,3 @@
version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
@@ -32,7 +30,7 @@ services:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: planet_db
ports:
- "5432:5432"
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
@@ -45,7 +43,7 @@ services:
image: redis:7-alpine
container_name: planet_redis
ports:
- "6379:6379"
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
volumes:
- redis_data:/data
healthcheck:

View File

@@ -8,6 +8,391 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.76.20] — 2026-11-10
Released: 2026-11-10
### Highlights
- 统一 Earth 对象详情、区域新闻与算力卡片的两侧布局,改善连接线、悬停展开和拖放稳定性,并降低卫星数据查询的 ORM 内存开销。
### Added / Fixed / Improved
- 共享布局仲裁器分配左右两侧与纵向走廊,使用换侧、换序滞后减少卡片争位;手动固定优先,空间受限时保留完整卡片。
- 地理原始投影与平滑连线锚点分离,固定状态依据真实背面深度解除;连接线选择合适的入卡边并平衡折线段,减少倒钩与抖动。
- 修复新闻及算力胶囊悬停展开后的尺寸、正文和操作区显示;新闻打字在悬停展开时继续,统一选择框与标签移除样式。
- 连接测试图标补充明暗主题的悬停和按下反馈;卫星查询以 500 条 ORM 批次转换,保留当前/最新任务回退、完整 TLE、顺序与 limit 语义。
- 清理旧布局引用和无效计算,同步中英文布局、渲染及使用文档,增加布局仲裁与卫星分批查询回归。
---
## [0.76.19] — 2026-11-09
Released: 2026-11-09
### Highlights
- 改善 Earth 桌面与移动端卡片、新闻刷新和数据更新链路,降低重复抓取、缓存重建与历史事件维护的开销。
### Added / Fixed / Improved
- 统一区域卡片的展开与胶囊展示、悬停和拖拽跟随,分别应用算力/新闻卡片展开上限;提供细线/粗线引导线及移动端交互改进,修复语言切换后状态提示标题残留。
- 新闻 GET 只读取已保存内容,RSS 补抓改为独立调度;手动刷新提供可追踪状态、超时和重试,完成展示后才提示成功。
- 图层缓存按 key 合并并发构建,续租与发布校验锁持有者;纯新闻变更不再触发 summary 缓存重建。
- BGP 前缀归属使用容错转换和可独立部署的 GiST 索引;Outbox 单次写入事件,跳过纯时间戳变更,并提供有界保留与维护脚本。
- 卫星粒子以球体遮挡判断替代点精灵深度裁切;同步中英文文档,补充回归并隔离前端验证构建产物。
---
## [0.76.18] — 2026-10-02
Released: 2026-10-02
### Highlights
- 收敛 Earth 新闻 breaking 交互、卡片 hover、面板稳定性与 ticker 玻璃扫光,并修复船只地表锚点显隐。
### Added / Fixed / Improved
- breaking 数据、ticker 当前项、面板生命周期和选中态分离;展开面板时冻结卡片 DOM,后台更新在关闭时应用。
- ticker 保留原有容器与滚动,breaking 扫光独立于静态上下边线,连续 breaking 切换不重启;卡片 hover/扫光统一复用等级色相。
- 新闻卡片内容高度、首卡间距和 cruise 选中态统一;船只 marker 复用地表锚点遮挡,保持前后显隐与 BGP/算力对象一致。
- 同步 Earth 前端状态机与渲染层中英文开发文档。
---
## [0.76.17] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 ticker 连续 breaking 新闻切换时扫光动画重启导致的闪烁、回跳和断帧。
### Added / Fixed / Improved
- breaking 等级 class 只负责颜色,稳定的 active 状态负责动画生命周期。
- ticker 与卡片扫光方向统一为左到右,连续 breaking 切换不重置时间轴。
---
## [0.76.16] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 ticker breaking class 每 120ms 重置导致的边框闪烁、扫光停滞和方向反转。
### Added / Fixed / Improved
- watcher 仅在当前 breaking 等级变化时更新 class,动画可以完整播放。
- ticker 与卡片扫光方向统一为左到右,ticker 扫光覆盖完整 ticker 区域。
---
## [0.76.15] — 2026-10-02
Released: 2026-10-02
### Highlights
- 将 ticker 扫光限制在上下边线以内,消除 breaking 边框被扫光伪元素带动闪烁。
### Added / Fixed / Improved
- ticker 扫光与卡片扫光保持独立横向移动,边线保持稳定,只响应 hover/等级颜色。
---
## [0.76.14] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复卡片扫光从中途出现的问题,让反射从卡片左边界完整扫到右边界。
### Added / Fixed / Improved
- 卡片反射改用内部 `background-position` 扫描,不再使用卡片外部 transform 起始。
- ticker 上下边线保持静态,横向扫光独立运行。
---
## [0.76.13] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修正 breaking hover 强度基准,hover 会在自身 breaking 高亮上继续增强。
### Added / Fixed / Improved
- ticker hover 从 breaking 原始色相和强度继续增强,不再回落到普通新闻强度。
- breaking 卡片边框 hover 同样相对自身等级增强,保持红/橙/黄颜色连续。
---
## [0.76.12] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 breaking ticker 和黄卡 hover 被白色混合漂淡的问题。
### Added / Fixed / Improved
- hover 使用共享强度参数与等级色相,玻璃底复用深色 HUD surface。
- 红色 ticker 保持红色边线,黄色卡片保持黄色 tint,不再被白色边框稀释。
---
## [0.76.11] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修正 breaking 卡片整卡扫光层级,并将 ticker hover 亮度对齐共享 HUD 面板。
### Added / Fixed / Improved
- 卡片扫光覆层位于内容之上,整张卡片 hover/focus 时才扫过。
- ticker 上下边线 hover 使用共享 HUD 边框 hover 强度,breaking 只替换色相。
---
## [0.76.10] — 2026-10-02
Released: 2026-10-02
### Highlights
- 将 breaking 卡片扫光限制为当前 hover/focus 卡片,避免所有卡片同时动态闪烁。
### Added / Fixed / Improved
- 卡片反射层默认关闭,只在当前交互卡片上启动对应等级颜色扫光。
- ticker 的全宽 breaking 扫光保持独立,不影响卡片浏览。
---
## [0.76.9] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 breaking 卡片黄色 hover 变淡,并补齐 ticker 上下边线的 hover 高亮反馈。
### Added / Fixed / Improved
- 卡片 hover 使用不透明等级色相和统一混合强度,红/橙/黄表现一致。
- ticker hover 只强化上下两条边,普通新闻使用蓝色,breaking 使用对应等级颜色。
---
## [0.76.8] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复展开新闻面板时后台 ticker / realtime 更新导致卡片重建和 hover 抖动的问题。
### Added / Fixed / Improved
- 面板浏览期间冻结卡片 DOM,后台更新只写入最新 payload。
- 关闭面板时一次性应用待处理新闻,保持滚动位置和 hover 稳定。
- 展开面板时暂停隐藏 ticker,手动刷新仍可立即更新面板内容。
---
## [0.76.7] — 2026-10-02
Released: 2026-10-02
### Highlights
- 统一 breaking 新闻卡片的 hover 交互,让通用 hover 行为叠加卡片自身等级颜色。
### Added / Fixed / Improved
- 红、橙、黄 breaking 卡片共用同一套 hover 提亮、边框增强和浮起逻辑。
- 等级颜色通过卡片变量注入,避免为每种颜色复制 hover 样式。
---
## [0.76.6] — 2026-10-02
Released: 2026-10-02
### Highlights
- 扩展 breaking ticker 的全域扫光效果,并拉开新闻卡片与上方状态内容的间距。
### Added / Fixed / Improved
- ticker 反射从局部光斑改为覆盖整条横向区域的扫光/闪烁带。
- 桌面与移动新闻列表首卡预留顶部呼吸空间,保持内容区层次。
---
## [0.76.5] — 2026-10-02
Released: 2026-10-02
### Highlights
- 增加 breaking 新闻的玻璃反射式警灯脉冲,并修复新闻卡片内容被 flex 压缩的问题。
### Added / Fixed / Improved
- ticker、卡片边框和卡片反射统一使用同一 breaking 等级颜色。
- breaking ticker 使用局部红/橙高光脉冲,卡片使用反射带扫过;面板底色保持蓝色。
- 新闻卡片固定非收缩布局并预留最小内容高度,避免只剩媒体名和时间。
- 同步中英文 Earth 前端开发文档,记录视觉状态与 reduced-motion 行为。
---
## [0.76.4] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修正 breaking 新闻状态显示:ticker 仅在当前滚动到 breaking 新闻时变红,展开面板始终保持蓝色。
### Added / Fixed / Improved
- 选中、焦点、关闭动画与 breaking 数据状态彻底分离,关闭新闻栏不再留下橙色边线。
- breaking 新闻按过期时间重新计算,单条新闻卡片继续保留自身 breaking 红色样式。
- 增加 ticker breaking 状态单测和浏览器关闭/重开验证。
---
## [0.76.3] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修复新闻栏关闭后残留橙色边线的问题,明确 breaking、展开面板和新闻选择的状态边界。
### Added / Fixed / Improved
- ticker 保持中性,breaking 等级只作用于展开新闻面板整体背景和阴影。
- 关闭新闻面板时清理选中故事和过期焦点状态,breaking 按新闻过期时间重新计算。
- 增加 breaking 状态单测,保持新闻滚动、选择和面板动画行为不变。
---
## [0.76.2] — 2026-10-01
Released: 2026-10-01
### Highlights
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
### Added / Fixed / Improved
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
---
## [0.76.1] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
### Added / Fixed / Improved
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
---
## [0.76.0] — 2026-09-30
Released: 2026-09-30
### Highlights
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
### Added / Fixed / Improved
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
---
## [0.75.4] — 2026-09-29
Released: 2026-09-29
### Highlights
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
### Improved / Fixed
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
---
## [0.75.3] — 2026-09-27
Released: 2026-09-27
### Highlights
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
### Improved / Fixed
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
---
## [0.75.2] — 2026-09-25
Released: 2026-09-25
### Highlights
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
### Fixed
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
---
## [0.75.1] — 2026-09-25
Released: 2026-09-25
### Highlights
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
### Added / Fixed / Improved
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
---
## [0.75.0] — 2026-09-23
Released: 2026-09-23
### Highlights
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
### Added / Fixed / Improved
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
---
## [0.74.7] — 2026-09-17
Released: 2026-09-17
### Highlights
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
### Added / Fixed / Improved
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
---
## [0.74.6] — 2026-09-16
Released: 2026-09-16

View File

@@ -79,13 +79,23 @@ git diff --unified=0 HEAD -- <path>
## Harness Commands
Windows desktop launcher tests run separately on Windows with
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
They use temporary desktop folders and mocked restart commands; no UAC prompt
or real service restart is triggered. The WSL bridge regression suite,
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
disposable command substitutes. For a real entry-point check without restarting,
run the repository `planet.cmd --install-only`, then the generated desktop
launcher with the same flag. Installation and mocked checks do not prove a live
elevated restart; report that separately when it has not been exercised.
| Tier | Command | What It Does |
| --- | --- | --- |
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, and documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
@@ -151,11 +161,6 @@ PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
controls should reuse the shared component, shared token, or established CSS
owner for that area instead of introducing a page-local lookalike.
- `scripts/harness/frontend-rules-check.sh` warns when semantic
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
approved React and Earth CSS owner files. A warning means the reviewer should
either move the style into the shared owner or document why this is a genuinely
new visual family.
### Earth I18n Harness Rules
@@ -209,6 +214,12 @@ required tool is not visible there, `scripts/harness/lib.sh` asks the user's
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
path. This avoids hardcoding a dotfile while still covering agent environments
that do not inherit the user's normal shell setup.
Validation adds the resolved Bun directory to its process `PATH` so package
scripts can invoke Bun as well. `test_tool_resolution.py` exercises this through
the validation entry point with an isolated login-shell tool path.
The validation build and preview share a temporary output directory that is
removed after the run. This gives the smoke a consistent build while local
development tools may rebuild `frontend/dist`.
Required for full local stack operation:
@@ -257,8 +268,7 @@ modify the host Docker environment.
No automatic hooks are installed in this phase. Manual reminders:
- Run `scripts/harness/quick-check.sh` before handing off small changes.
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
- Choose the verification scope using the [validation guidance in the agent guide](../AGENTS.md#validation).
- Run `scripts/harness/security-check.sh` after touching config, auth,
credentials, docs examples, or generated fixtures.
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
@@ -280,7 +290,7 @@ No automatic hooks are installed in this phase. Manual reminders:
2. Check `CODEMAP.md` for entry points and ownership boundaries.
3. Inspect existing tests and docs before editing.
4. Make the smallest behavior-preserving or feature-scoped change.
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
5. Run the checks for the affected behavior and any gates required for that area.
6. Update relevant docs when behavior, workflow, or operations change.
7. For rendered frontend changes, verify the affected route with Playwright or
the full harness smoke, because `bun run build` alone does not prove page
@@ -291,8 +301,6 @@ No automatic hooks are installed in this phase. Manual reminders:
1. Reproduce with a focused test or command.
2. Patch the owning module, not a caller-side workaround.
3. Run the focused regression test.
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
locally.
### Documentation Change

View File

@@ -14,6 +14,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
## Scope Rules
- Focus documentation changes on the primary document that owns the affected behavior. Do not trigger speculative updates across secondary or architectural docs unless the underlying contract or operator-facing workflow actually changed.
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
@@ -23,6 +24,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
- Internal bug fixes or optimizations that preserve existing APIs and workflows do not require updating user manuals or creating new documentation.
## Public Docs Rules
@@ -42,86 +44,17 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
## Recommended Checks
Run the checks that match the affected docs.
### Duplicate Bilingual Docs
Always run the centralized harness documentation gate:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
scripts/harness/docs-consistency-check.sh
```
### Language-Less Technical Links
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This should return no matches.
### Public Docs Registry
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
### Public Bilingual Pairs
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
### Raw Filename Link Titles
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
This should return no matches for polished public docs.
This single command automatically covers:
1. Public docs whitelist and `DOCS_METADATA` synchronization (`frontend/src/pages/Docs/docs-content.ts` vs backend Gatekeeper)
2. Technical doc bilingual file pairs and identical duplicate-content detection
3. Public doc link validity and human-readable link titles (no raw `.md` filenames)
4. Language-scoped internal links (`docs/technical/zh/` links pointing to `docs/technical/zh/`)
5. Supported credential collector contracts and UI guide registration
6. Manual console route coverage against actual admin manifests
7. Documented `?section=` deep-link validity and UI route drift in active docs

View File

@@ -28,8 +28,7 @@ compatibility note, not a replacement for existing rules or architecture docs.
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
| `.codex/skills/*/SKILL.md` | Present | Local workflows for cleanup, docs, goal-driven work, and release. |
## Existing CI Gates
@@ -57,8 +56,7 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
## Release And Deploy Process
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
`.claude/commands/release.md`.
- Release workflow is documented in `.codex/skills/release/SKILL.md`.
- Version-bearing files include `VERSION`, `frontend/package.json`,
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
`docs/version-history.md`.
@@ -102,7 +100,6 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
@@ -137,7 +134,7 @@ index is the routing layer; this table is the coverage/evidence layer.
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
| `uiux` | Related visual surfaces should use one style system per product area. | Manual visual review. | Compare changed controls with the shared component on the rendered page. |
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
@@ -154,8 +151,8 @@ index is the routing layer; this table is the coverage/evidence layer.
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, and admin-config-derived section deep-link consistency. |
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |

View File

@@ -22,13 +22,14 @@
当前重点入口:
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)

View File

@@ -1,9 +1,11 @@
# 算力与资源态势:展示、采集与 AI 迭代实施计划
状态:待实施。2026-09-14 已完成当前代码边界、主要公开数据入口和历史指标口径核查;本计划不代表采集器、评分服务或 AI 迭代已经上线。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
状态:首版空间卡片原型实现中(2026-09-21)。已接入算力专题入口、45 个参与方目录、年份/口径选择、区域数据卡与来源、成功快照设施查询,以及 Epoch/TOP500 的首批质量修复。旧固定双图布局已移除。历史年度数据使用受控版本导入;各国完整指标覆盖、卡片图表、全榜获取、跨源设施合并、持久化指标管理、变化归因与 A4 AI 解读尚未完成,不视为全部 A1—A3 已验收。后续章节保留完整设计目标,当前实现以技术文档为准。2026-09-14 完成的口径核查仍是内容依据。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
## 0. 当前执行方向与范围
2026-09-21 展示方向修订:UI、地球材质、政治展示规则、地理资源对齐及性能门槛以[Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)为准。下文固定右侧国家面板、底部横贯双图等布局不再作为实施目标。采用从区域锚点引出连接线、卡片围绕地球自适应浮动的方向,先验证空间交互再完善卡片内容。数据来源、口径与证据要求继续有效。
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚:地图显示资源在哪里,图表说明规模和相对位置怎样变化,证据说明数字从哪里来,AI 帮助阅读变化。
采用“专题先交付,公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线,不是首版展示的前置条件;第 6 节总分保持实验性,不能为了首屏有大数字而生成不完整排名。
@@ -99,6 +101,14 @@
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读,不进入计算;不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度,不让刷新网页造成“国家实力实时跳动”。
### 3.3a 当前数据合同(2026-09-22)
默认主指标是全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。CAICT 报告核查目前使用官方 PDF 索引文本;本地 PDF 下载未成功,因此不宣称已肉眼核验 PDF 页面。绝对量合同记录 2022 年 302(证据版本 `caict2023`)和 2023 年 435(证据版本 `caict2024`);2024 年报告正式书名为《先进计算暨算力发展指数蓝皮书(2024年)》。官方 URL 已替换为四个镜像来源。
日本全球份额观测为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 `APAC` 与 `OTHER` 不能直接套用报告中的 Japan 或“其他”;`EUROPE` 成员集合尚未与报告口径对齐,仍返回缺失。双边份额分母固定为 US + China,新加入日本不改变分母。主要指标缺失时可显示最近有报告的年份(当前可到 2023),而不是留空为 2026;用户明确选择 2026 时仍显示缺失,不回填最近值。不可比性尚未确认,不展示未经确认的同比变化;覆盖数据保留在辅助区并带 count。
数据 API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现,用户对该问法的选择待后续确认;当前合同不提前新增该模式或声称完成 dataset 计算。
默认“历史回顾”允许查看最新证据重述后的过去;“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时,必须显示“最新观测:2023”,或在严格同年模式显示缺失。
### 3.4 地球编码与交互

View File

@@ -0,0 +1,154 @@
# Planet 开发 Agent 模型分配与 Token 控制计划
日期:2026-09-21。状态:已用于本轮地理与空间卡片开发;未改变全局或项目默认模型配置。适用于已授权的开发任务,不代替用户对任务范围的决定。
目标:在固定政治展示规则、现有 UI 逻辑和性能门槛下,降低完成一个合格任务的总成本与返工次数。不是让所有工作都使用低价模型,也不是让最强模型从头包办。
当前重点任务见 [Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)。该计划的功能与验收要求优先;本文件只决定如何组织开发。
## 1. 模型角色
模型定位参考 OpenAI Docs;下表的具体分工是针对 Planet 的工程建议,需要结合实际一次通过率和返工记录调整,不是模型能力保证。
| 模型 | 精确模型 ID | 默认推理强度 | 分配的工作 | 不作为默认用途 |
| --- | --- | --- | --- | --- |
| Astra | `gpt-6-astra` | `high` | 跨系统难题、空间展示方案、复杂坐标/深度/性能诊断、关键设计与独立审查 | 日常文件搜索、机械编辑、反复运行已确定命令 |
| Sol | `gpt-5.6-sol` | `medium`;复杂实现用 `high` | 执行主控、核心代码、共享组件整合、Three.js 状态与生命周期、最终集成 | 逐条整理大批重复记录、重复复述整个项目背景 |
| Terra | `gpt-5.6-terra` | `medium` | 接口已明确的常规实现、局部 UI 修复、数据处理脚本、测试编写、浏览器回归 | 独自决定空间交互方向、坐标模型或跨模块架构 |
| Luna | `gpt-5.6-luna` | `low`;明确的小修改可用 `medium` | 定向检索、清单比对、日志归类、说明文档初稿、固定格式整理 | 核心渲染、数据迁移、几何归属修改、最终审美或性能放行 |
日常推荐组合是 Sol 主控加一个 Terra worker;Luna 和 Astra 按需进入,不要求每个阶段凑齐四种模型。已经完成的资源准备不因采用这份方案重新做一遍。
当前会话的主模型不会因为写入计划自动变成 Sol。真正执行时,主会话模型由实际任务设置决定,子 Agent 在派发时明确指定模型和推理强度,并核对返回的运行配置;不能把角色名称当作已经切换模型的证据。
## 2. 本项目任务分配表
| 工作包 | 主执行 | 独立验证/审查 | 交付和升级条件 |
| --- | --- | --- | --- |
| 现有窗口、连线、巡航、滚动条入口盘点 | Terra | Sol 检查引用与遗漏 | 文件/符号清单、现有行为证据;Luna 可辅助定向搜索,无需读取全仓库 |
| 贴图归档、尺寸、哈希、配准元数据检查 | 直接运行脚本;脚本需新增时 Terra | Sol 抽查实际输出 | S0 已完成,后续只做有变化的核对,不委托模型重复计算哈希 |
| 固定政治规则的技术合同与几何覆盖边界 | Sol | Astra 审查规则是否贯穿各数据路径 | 权属口径已由项目确定,任何模型均不得重新决定;标识不明确先核对,不能猜测几何 |
| 贴图、海岸线、拾取错位定位 | Sol `high` | Astra 在根因不清或涉及多个坐标层时介入 | 用明确经纬点复现,区分错误数据、回退、投影、UV 与半像素偏移 |
| 规则已明确的映射、来源清单、局部转换脚本 | Terra | Sol;关键几何结果由 Astra 抽查 | 只落实批准合同;发现新的几何语义不得自行扩大规则 |
| 多区域空间卡片的第一版运动设计 | Astra `high` | Sol 检查能否复用当前实现 | 明确锚点、投影、避让、换侧、背面及阅读稳定性;产出小合同,不产出整套 UI 大改 |
| 2 张卡片的连接、弹性移动、生命周期原型 | Sol `high` | Terra 实测;Astra 看关键动效证据 | 优先实现最小完整循环;使用参考图和实际运动证据,不用文字自评代替 |
| 扩至 4/6 卡、增删区域和事件整合 | Sol | Terra 跑交互/竞态测试 | 复用原型合同;`main.js` 等共享入口只由指定 owner 写入 |
| 共享窗口行为、标准字体、移动抽屉及深色滚动条 | Terra | Sol 看渲染与旧面板回归 | 修改公共滚动所有者,验证图层等所有相关页;涉及共享状态重构时升级给 Sol |
| 地球柔光、材质层次、节点与背景关系 | Sol | Astra 对照 UE/PPT/海报审查 | 第一轮只改少量确定参数并给实际截图;性能退化交 Sol 诊断,难题交 Astra |
| 卡片内指标、普通图表、加载/缺失/错误状态 | Terra | Sol | 数据合同先冻结;不复活固定全屏大面板,不编造缺失年份 |
| 国家/区域接口和前后端联动 | Terra;跨模块变更由 Sol | Sol | 不硬编码中美为唯一对象;范围筛选、比较对象和组合去重必须一致 |
| 性能基准工具和回归测试 | Terra | Sol 检查方法;Astra 审查疑难退化 | 固定设备、数据与相机操作;日志含帧时间和内存证据,不能只报平均 FPS |
| 单位、日期、缓存、旧请求覆盖等复杂数据故障 | Sol | Astra 仅处理跨系统未解问题 | 先给最小复现和失败测试,避免多个模型同时试不同修复 |
| 中英文手册、资源清单、计划状态同步 | Luna 初稿;技术性较强时 Terra | Sol 对照实际 diff | 已实现与待实施分开;Luna 不能自行宣告功能完成 |
| 最终集成与验收 | Sol | Astra 对关键风险做一次集中审查 | 相关测试、渲染、政治规则及性能证据齐全;普通修订不逐项再次走 Astra |
## 3. 顺序与并行安排
### 第一轮:证据与边界
Sol 读取现有计划、当前工作区差异和必要规则,整理短任务清单。Terra 盘点实际运行的数据提供者、组件和性能基线;独立且批量的文档/引用整理才交 Luna。先读已有检查结果,避免重新下载贴图、重新研究 World Monitor 或重复跑已经通过的检查。
Sol 将确定的组件接口、规则约束、实际截图与未解问题整理成一个短包,Astra 只审查高返工风险的空间方案和坐标问题。Astra 返回决定、理由和验证点,不再从零扫描整个仓库。
### 第二轮:最小空间原型与独立 UI 修复
Sol 实现 2 张卡片的锚点、连线和有阻尼移动;Terra 可以同时处理公共滚动条或写回归测试,前提是文件所有权不冲突。先完成最小可演示运动,再扩展内容和卡片数量。若两者都要改 `controls.js`、`hud.css` 等共享文件,则按顺序交接,不并行写入。
Astra 对原型的实际截图/录屏做一次聚焦审查,检查是否符合“从地球四周延伸”的方向。空间原型尚未确认时,不展开完整指标卡片和全量国家 UI,以免重演方向错误后大规模返工。
### 第三轮:内容、全球扩展与性能
合同稳定后,Terra 接入指标与状态,Sol 负责与渲染/生命周期的整合。性能测试由脚本执行,Terra 整理差异;可复现的退化先交 Sol。只有不能局部解释的 GPU/CPU、坐标、异步问题才交 Astra。
### 第四轮:一次集成验收
Sol 整合并运行所需检查;Luna 根据最终 diff 更新文档初稿。Astra 集中审查关键几何、政治规则完整性、卡片运动与性能结论。已通过范围不反复重测,除非新修改影响该范围。
## 4. 并发上限与文件所有权
当前环境总并发槽位为 4,含主 Agent;执行时仍以实际可用额度为准。默认使用主 Agent 加 1 个 worker。只有存在两项以上真正独立的工作,且主 Agent 同时有有用工作可做时,才扩大到主 Agent 加 2 个 worker;第 4 个槽位只给临时审查或独立批处理。
每项任务指定可写文件范围。`main.js`、`controls.js`、共享 HUD/InfoCard 与跨层状态接口同一时间只有一个写入 owner。其他 Agent 只读检查或提交补丁建议,由 owner 集成。不得并行启动多个浏览器重测相同场景,也不得共享有冲突的登录/本地存储测试状态。
浏览器验证优先使用当前可用的 Browser 控制能力,包括统一 CUA 入口。不能仅因技能目录中缺少单独的 Browser 条目就判定工具不存在;应继续检查已暴露工具并多次重试连接。只有多次确认 Browser 不可用后,才考虑独立 Playwright,并说明具体阻塞。
Agent 完成有边界的任务后停止。只有同一任务需要补充验证时才续用;不能让 worker 在空闲时自动寻找新功能或自行扩大范围。创建用户侧新任务不作为内部拆工方式。
## 5. 节省 Token 的具体做法
1. 派发窄任务,不默认继承整个历史对话。使用简短任务卡加源文件路径;必要历史按最小范围提供。短上下文不能省略政治规则、性能要求和相关视觉证据。
2. 主控保存一份简洁任务账本:当前合同、文件 owner、已完成证据、失败原因、下一步。各 Agent 引用同一事实,不各写一份长计划。
3. 先 `rg` 定位,再读函数和局部 diff;不重复读取巨大 JSON、整仓日志或全部历史文档。大数据留在文件里,由脚本给出数量、哈希、差异和异常样例。
4. 图片、截图和录屏用实际文件查看。不能为了省 token 把视觉任务变成只读文件名或文字概括。
5. 相同来源只研究一次,保存来源、日期和关键结论;只有数据变化、结论不足或任务扩大时重查。
6. 子任务首次任务卡建议控制在约 500–1500 token,结果摘要约 300–800 token;这是上下文组织目标,不是截断必要证据的硬上限。
7. 返回“改了什么、哪些文件、验证证据、剩余阻塞”即可。完整日志和截图放到文件,用路径引用;失败时返回相关片段,不整段倒入主控上下文。
8. 哈希、解压、下载、构建、测试和统计交给确定性工具。不要为了运行一条已知命令另开一个 Agent。
9. 不重复安排多个模型完整实现同一功能。关键风险做一次独立审查,普通修改由测试与 owner 检查。
10. 默认不开 `max` / `ultra` 推理。提高模型或推理强度只针对具体难题,并记录升级理由。
## 6. 升级与停止条件
| 情况 | 处理 |
| --- | --- |
| Luna 遇到不明确需求、需要理解跨文件状态或开始猜测行为 | 交 Terra;若涉及核心状态/几何,直接交 Sol |
| Terra 的任务超出明确合同,或同一实质失败在一次有针对性的修复后仍未解决 | 给 Sol 最小复现、尝试记录和失败证据,不继续盲试 |
| Sol 遇到跨坐标层、性能来源不明或与现有交互语义冲突的问题 | 给 Astra 聚焦诊断任务;可以由 Astra 完成必要关键补丁,不限于口头审查 |
| 已知一开始就是复杂的空间布局或高风险几何设计 | 直接用 Sol/Astra,不为了经过低模型梯级而增加一次无效尝试 |
| 网络、凭证、工具缺失、原始资料不存在 | 不通过升模型解决环境问题;检查环境或记录缺口,继续独立工作 |
| 失败来自缺失用户决定或无法识别的地名/视觉目标 | 提出具体问题,不让多个模型猜不同版本 |
默认允许 worker 一次有证据的修复;是否继续由主控根据新信息决定。超时或 token 消耗增加但没有新增证据时停止扩展,整理已知结果后重新分配,不能无限自递归。
## 7. 审查门槛
Astra 重点进入三个位置:最初高风险空间/几何合同、首次运动原型、最终关键风险验收;中间疑难问题按需增加。它不需要审阅每一处 CSS、每一段翻译或每个测试日志。
Sol 对各 worker 的交付负责。不能以“另一模型说通过”代替运行结果。任何模型都无权改变项目政治口径、降低性能门槛、把未知数据填成确定值,或自行部署和发版。
涉及政治几何和性能结论,使用与实现过程分离的检查证据;普通文档格式或固定映射使用针对性检查即可。视觉方向仍由已确认的用户要求和参考图决定,Astra 的审美判断不替代用户目标。
## 8. 单任务派发模板
```text
任务:一个可独立验收的目标
模型与推理:明确 model ID + reasoning effort
必要约束:相关 rules 模块、政治口径、性能、共享组件、视觉参考
输入:最少文件/符号、当前合同、已有验证结果和图片路径
可写范围:明确文件;共享文件由指定 owner 修改
验收:必须出现的行为、必要测试/渲染/性能证据
停止/升级:哪些条件不能继续猜测或扩展
返回:结果摘要、改动文件、检查结果、证据路径、遗留问题
```
派发有意使用较小上下文时,必须把不可省略的约束写进任务卡。执行任务的 Agent 自行按规则加载相应模块,不因主控已读过就假定子 Agent 也知道。
## 9. 如何验证分配确实更高效
每个工作包记录模型、推理强度、输入/输出 token(环境可提供时)、耗时、工具耗时、修复轮数、一次验收是否通过和最终证据。若读不到准确 token,就标记未知,不用消息长度伪造 token 统计。
比较每个已验收任务的总消耗,包含主控、worker、审查和返工,而不是仅比较某次调用。先完成 3–5 个有代表性的工作包,再调整默认路由;这个数量是试点取样建议,不代表统计显著性或保证节省比例。
API 单价、订阅额度和任务级成本不是同一个指标。本计划不预先承诺固定节省百分比;模型可用性和实际费用以执行时配置为准。低成本模型导致反复修复时应提前升级;高能力模型能明显减少整体返工时可以直接采用。
## 10. 官方依据与执行边界
OpenAI Docs 将 Astra 定位为最具能力、适合困难端到端任务的模型,Sol 面向复杂专业工作,Terra 平衡能力与成本,Luna 面向成本敏感任务;它们并未规定 Planet 各文件应由哪个模型处理。本计划按任务风险给出项目级分工。
- [OpenAI 模型说明](https://developers.openai.com/api/docs/models)
- [Codex 模型选择](https://learn.chatgpt.com/docs/models)
- [子 Agent 配置与模型选择](https://learn.chatgpt.com/docs/agent-configuration/subagents)
只在用户已授权相应开发与委派、且运行环境支持时执行这套拆工。仅制定本计划不会切换当前会话模型、修改默认配置或自动启动 Agent。执行时若某个精确模型不可用,先说明并选择可用替代,不把替代模型冒充原分配。
## 11. 本轮实际执行账本
| 工作包 | 实际分配 | 结果与升级 |
| --- | --- | --- |
| 地理政策、源证据、几何构建与回退 | Sol `high`,`geography_policy` | 版本化规则与校验已实现;如实保留未核验几何,未用模型猜测填补 |
| 空间连接线、卡片生命周期与主入口 | Sol `high`,`spatial_callouts` | 完成原型;六卡避让多次渲染仍有冲突后,交主控接手约束求解,worker 停止同文件写入 |
| 同机性能基线、对照工具、公共滚动条 | Terra `medium`,`performance_baseline` | 保存桌面/模拟手机各 5 次基线;共享滚动容器修复和对照工具已完成 |
| 复杂避让、全球目录/API、视觉参数、集成验证 | 主会话 | 主控整合并审查;性能脚本的场景同步多次失败后也由主控接手修复。独占布局文件,浏览器/构建按顺序执行,避免测试相互干扰 |
| Luna 辅助 | 未启用 | 没有值得额外传递上下文的独立机械任务,不为凑齐模型而派工 |
本轮精确 token 与费用不可得,记为未知;不报告未经测量的节省比例。空间避让存在返工,不能把功能实现或子任务数等同于效率提升。性能问题应在首次稳定复现后采集 CPU/布局证据,再修改热点;本轮最终定位到共享 SVG 长度读取造成的同步布局,单纯调整避让参数不足以解释该成本。性能、运动观感及地理缺项以功能计划的验收状态为准。

View File

@@ -1,715 +0,0 @@
# Earth 天球背景与日月位置实施方案
## 目标
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
1. 用真实天球背景替换当前随机星点
2. 在当前时间下显示太阳与月亮的相对位置
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
本方案优先追求:
- 与当前 Three.js Earth 架构兼容
- 风险可控
- 先落地一版真实感明显提升的 V1
- 为后续更严格的天文参考系升级预留余地
## 当前现状
当前 Earth 的基础条件已经具备:
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
这意味着:
- 不需要重写 Earth
- 可以在现有 scene/world 层新增一个 celestial layer
- 第一阶段不必拆 Earth / satellite / cable 的参考系
## 总体策略
采用“两层现实”设计:
### 1. 世界层(world-space celestial layer)
用于放置:
- 天球背景
- 太阳
- 月亮
- 太阳光方向
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
### 2. 地球层(earth-fixed layer)
继续保持当前结构:
- 海缆
- 登陆点
- 卫星点与轨迹
- BGP 覆盖
- 地球纹理、云层、地形
这些对象继续挂在 `earthObj` 下,不打断现有交互。
## 为什么先这样做
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
如果现在直接做严格惯性参考系改造,会同时影响:
- `earthObj.rotation`
- 卫星轨迹与锁定逻辑
- 海缆与登陆点附着关系
- resetView / autoRotate / hover / click 等交互链路
所以第一阶段只做:
- 真正的天空
- 真正的日月方向
- 不碰现有 Earth 附着对象的语义
## 推荐技术选型
### 天文计算库
推荐:
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
原因:
- 有 JavaScript 版本
- 支持 Sun / Moon 的矢量与坐标变换
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
不作为主选的库:
- [SunCalc](https://github.com/mourner/suncalc)
原因:
- 更偏本地观察者视角的太阳/月亮高度角
- 用于“地面日出日落”很好
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
### Three.js 表现层
推荐组合:
- 天球:内翻球壳 + 星图纹理
- 太阳:`THREE.Sprite`
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
- 太阳光:`THREE.DirectionalLight`
参考:
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
## 天球背景资源与星体数据来源
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
### 1. 背景资源:全天星图贴图
用于 Phase 1 的“真实天空背景”。
推荐优先来源:
- NASA SVS 的 Tycho 全天星图
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
- NASA Deep Star Maps 2020
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议要求:
- 使用官方来源或官方衍生可复用资源
- 等距矩形投影(equirectangular)
- 坐标定义尽量明确为赤道坐标展开
- 分辨率建议至少 `4k`
- 颜色不要过亮,避免压过 Earth HUD 前景
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
建议本地资源目录:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
### 2. 位置数据:星表与天体计算
用于 Phase 2+ 的“位置正确的星体”。
推荐来源分两层:
- 太阳、月亮位置
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
- 恒星位置
- 第一优先:Hipparcos / Tycho
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
- 第二优先:Gaia
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议策略:
- V1:背景球壳只用全天星图,不立即生成全量恒星点
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
这样做的原因:
- 背景球壳负责“天球真实感”
- 亮星点负责“位置正确、可后续标注和高亮”
- 不需要一开始就处理数十万甚至数百万颗星
### 3. 对外部成熟实现的参考结论
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
- 图形渲染使用 `TWGL.js`
- 天文计算使用 `Skyfield` 与 `Astronomia`
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
这给本项目的启发是:
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
- “位置正确的动态天体”则应依赖单独的天文计算链路
- 没有必要在第一版就直接渲染完整星表
因此本项目推荐继续坚持两层拆分:
- 背景层:官方深空图 / 全天星图
- 计算层:太阳、月亮与后续亮星点
## 如何保证星体位置正确
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
### 1. 统一坐标基准
本方案推荐统一使用:
- `J2000` 赤道坐标系作为恒星位置基准
原因:
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
### 2. 背景贴图与点位必须使用同一展开逻辑
如果背景球壳使用赤道坐标全天图,那么:
- 亮星点也必须按赤道坐标贴到同一球面方向
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
否则会出现:
- 背景银河带是对的
- 但太阳/月亮或亮星点飘到不匹配的位置
### 3. RA / Dec 到 Three.js 坐标的落点方式
亮星点和日月方向最终都要转成单位球面向量。
概念步骤:
1. 读取赤经 `RA`
2. 读取赤纬 `Dec`
3. 转成弧度
4. 映射到单位球面向量
5. 再根据 Three.js 当前世界坐标定义做轴向映射
参考公式:
```text
x = cos(dec) * cos(ra)
y = sin(dec)
z = cos(dec) * sin(ra)
```
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
- 验证 `RA = 0h`
- 验证 `RA = 6h`
- 验证北天极
- 验证银河带主方向
然后确定最终的:
- `x/y/z` 对应 Three.js 哪个轴
- 是否需要 `z` 取反
- 是否需要整体再做一个固定 `rotation`
建议把这层显式封装在:
```js
function equatorialToWorldVector(raRad, decRad)
```
不要把轴映射散落在不同模块里。
### 4. 背景球壳与恒星点的关系
推荐最终组合:
- 背景层:全天星图球壳
- 点位层:亮星点
- 动态层:太阳 / 月亮
这样有三个好处:
- 背景层提供密集真实的天空纹理
- 亮星点提供位置正确、可扩展的标注基础
- 太阳/月亮提供与时间相关的真实动态对象
## 数据与资源建议清单
### 推荐首批引入资源
1. 全天星图
- 来源:NASA Tycho all-sky map
- 用途:背景球壳纹理
2. 月亮纹理
- 用途:Phase 4 月相表现
- 路径建议:
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
3. 太阳 glow 贴图
- 用途:太阳 sprite halo
- 路径建议:
- `frontend/public/earth/assets/celestial/sun_glow.png`
### 推荐首批数据文件
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
- `frontend/public/earth/assets/celestial/bright-stars.json`
建议字段:
```json
[
{
"id": 32349,
"name": "Sirius",
"raDeg": 101.2875,
"decDeg": -16.7161,
"mag": -1.46,
"colorIndex": 0.00
}
]
```
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
- 只保留亮星
- 只保留渲染必需字段
- JSON 或二进制轻量格式
## 资源与数据实施路线
### 路线 A:先做可用版本(推荐)
1. 引入 NASA Tycho 全天图
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
2. 实现背景球壳
3. 用 Astronomy Engine 计算太阳/月亮方向
4. 暂不做亮星点
优点:
- 最快见效
- 风险最低
- 就能明显提升天球真实感
### 路线 B:在 A 基础上增强
1. 离线生成 `bright-stars.json`
2. 浏览器端渲染亮星点
3. 后续可加:
- 星座线
- 亮星名称
- 特定星体高亮
优点:
- 背景真实感和“位置正确的可交互星体”同时兼顾
## 代码模块建议细化
### 新增模块
- `frontend/public/earth/js/celestial.js`
- 管理天球背景
- 管理太阳/月亮
- 管理亮星层(后续)
- `frontend/public/earth/js/celestial-data.js`
- 资源路径
- 星图方向配置
- 亮星数据加载(后续)
### 建议函数设计
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date)
export function setCelestialVisibility(visible)
export function disposeCelestialLayer()
function loadStarMapTexture()
function createSkySphere(texture)
function createSunSprite()
function createMoonSprite()
function getSunEquatorialPosition(date)
function getMoonEquatorialPosition(date)
function equatorialToWorldVector(raRad, decRad)
```
### 推荐后续预处理脚本
如要引入亮星层,建议单独做离线脚本:
- `scripts/build_bright_stars.py`
职责:
- 从 Hipparcos / Tycho 源数据读取
- 过滤亮星
- 生成 `bright-stars.json`
这样浏览器端只消费轻量结果,不承担大表解析成本。
## 分阶段实施
## Phase 1:真实天球背景
### 目标
用真实全天星图替换当前随机星点背景。
### 做法
1. 新增一张全天星图纹理
建议路径:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
纹理要求:
- 等距矩形投影
- 赤经/赤纬坐标展开
- 无地平线、无地景遮挡
- 尽量深色、弱干扰,适合大屏 HUD 叠加
2. 新增天球球壳
新增模块:
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
建议接口:
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date, camera, earth)
export function disposeCelestialLayer()
```
3. 实现一个大半径内翻球体
建议参数:
- 半径:`600 ~ 900`
- 材质:`MeshBasicMaterial`
- `side: THREE.BackSide`
- 不受场景光照影响
- 始终围绕场景中心
### 验收标准
- 初始加载后背景不再是随机星点
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
- 不明显干扰海缆/卫星/BGP 的前景识别
## Phase 2:太阳与月亮真实位置
### 目标
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
### 做法
1. 在 `celestial.js` 内封装天体位置计算
建议函数:
```js
function getSunDirection(date)
function getMoonDirection(date)
```
输出统一为 world-space `THREE.Vector3`
2. 太阳显示
- 一个暖色发光 sprite
- 比月亮更大、更亮
- 可选添加柔和 halo
3. 月亮显示
- 一个较小 sprite 或 sphere
- 灰白偏冷色
- 后续 Phase 3 再做月相
4. 更新频率
不要每帧重新做完整天文计算,建议:
- 每 30 秒或 60 秒重算一次真实位置
- 渲染帧内做平滑过渡
### 验收标准
- 页面可见太阳与月亮两个对象
- 时间变化时位置会更新
- 日月不会跟随地球局部旋转而错误附着
## Phase 3:太阳驱动地球受光
### 目标
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
### 做法
1. 替换或接管当前主定向光
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
建议改为:
- 保留环境补光
- 主太阳光方向由 `sunDirection` 决定
2. 太阳光参数建议
- `DirectionalLight` 颜色偏暖白
- 强度略高于当前主光
- 保留一个弱背光作为氛围补偿,避免背面过死黑
3. 先不做物理级大气散射
第一版只要求:
- 亮面与暗面方向真实
- 云层和大气仍保持当前风格
### 验收标准
- 地球明暗面会随太阳方向改变
- 太阳 sprite 和地球亮面方向一致
- 不破坏现有海缆、卫星、BGP 的可见性
## Phase 4:月相与天文细节增强
### 目标
在日月真实位置基础上增加更强的“天文可信度”。
### 可选项
1. 月相
- 根据日月夹角计算 illuminated fraction
- 用月相纹理或 shader 表达盈亏
2. 赤道/黄道辅助线
- 可作为开发调试层,不默认显示
3. 太阳 terminator 增强
- 给地球夜面加入更自然的 night tint
- 未来可叠加城市夜光纹理
4. 天文时间入口
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
### 验收标准
- 月亮不再只是一个静态圆点
- 后续扩展行星或观测模式时无需推倒重来
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
### 目标
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
### 需要处理的问题
- 地球自转角与 UTC 的一致性
- 赤道坐标系、地固坐标系、相机交互层分离
- 卫星轨道显示与 Earth 旋转同步关系
- resetView 和 autoRotate 的语义重定
### 风险
这一步会影响:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
因此不建议与 V1 同时推进。
## 代码改造清单
## 1. 新增文件
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
职责:
- 管理天球背景、太阳、月亮
- 对外暴露 init/update/dispose
## 2. 修改 `constants.js`
文件:
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
新增:
```js
export const CELESTIAL_CONFIG = {
sphereRadius: 800,
updateIntervalMs: 60000,
sunSpriteScale: 28,
moonSpriteScale: 16,
sunLightIntensity: 1.25,
ambientIntensity: 0.28,
backLightIntensity: 0.18,
};
```
## 3. 修改 `main.js`
文件:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
主要改动:
1. `init()` 中:
- 初始化 celestial layer
2. `addLights()` 中:
- 把固定太阳光改成可更新的 celestial sun light
3. `animate()` 中:
- 每帧调 `updateCelestialLayer()`
4. `destroy()` 中:
- 清理 celestial 资源
## 4. 修改 `earth.js`
文件:
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
主要改动:
- `createStars()` 逐步退役
- 第一阶段可先保留作为 fallback
- 当真实星图加载成功后,不再显示随机星点
## 5. 新增资源
目录建议:
- `frontend/public/earth/assets/celestial/`
建议至少包含:
- `starmap_equatorial_4k.jpg`
- `sun_glow.png`
- `moon_albedo_2k.jpg`
## 数据流设计
```mermaid
flowchart TD
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
B --> C["创建天球球壳"]
B --> D["创建太阳 sprite + 主定向光"]
B --> E["创建月亮 sprite"]
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
H --> I["更新 sun sprite / moon sprite 位置"]
H --> J["更新太阳 DirectionalLight 方向"]
J --> K["地球昼夜方向变化"]
```
## 风险与注意事项
### 1. 星图投影方向容易反
这会表现为:
- 星图左右镜像
- 赤经方向颠倒
- 日月位置和背景对不上
建议:
- 先做一个开发调试模式
- 显示赤经/赤纬参考点,快速校正纹理朝向
### 2. 不要让天球跟随 Earth 旋转
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
### 3. 不要每帧做重型天文计算
真实位置更新应节流,否则会浪费 CPU。
### 4. 月亮先求“方向正确”,再求“月相精致”
月相属于第二步优化,不应阻塞 V1 上线。
## 推荐实施顺序
1. 新建 `celestial.js`
2. 用星图球壳替换随机星点
3. 接入 Astronomy Engine
4. 加太阳/月亮 sprite
5. 用太阳方向驱动主光
6. 再决定要不要做月相和更严格参考系
## 最终建议
对于当前 Planet Earth,最稳妥的方案是:
- 先做真实天球背景
- 再做真实太阳/月亮方向
- 再让太阳驱动地球受光
- 暂时不做 Earth 参考系重构
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。

View File

@@ -1,106 +0,0 @@
# Earth Predicted Orbit Plan
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
## Goal
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
- 从当前时刻开始
- 绕地球一圈
- 当前点最亮
- 向后沿轨道逐步衰减
## Current State
当前已经有:
- 卫星历史轨迹
- 锁定卫星
- 轨道高亮与相关联动
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
## Why It Is Valuable
预测轨道可以明显提升:
- 锁定卫星后的空间可读性
- 轨道类型辨识
- 演示解释力
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
## Scope
### Phase 1
- 锁定卫星时显示一整圈预测轨道
- 解锁时隐藏
- 不替代现有普通轨迹系统
### Phase 2
- 根据轨道类型调整采样率
- GEO / MEO / LEO 不同密度
- 进一步减少 fallback 轨迹的比例
## Implementation Direction
### 1. Orbit period
基于 `meanMotion` 估算轨道周期。
### 2. Predicted samples
以固定采样步长从 `now -> now + period` 推算轨迹点。
### 3. Render object lifecycle
预测轨道应是一个独立渲染对象:
- show
- update
- hide
- dispose
### 4. Visual semantics
预测轨道不应与普通尾迹混淆:
- 更稳定
- 更完整
- 透明度沿轨道衰减
- 当前点附近更亮
## Known Risks
### 1. TLE propagation gaps
部分卫星可能出现 SGP4 计算不足,需要 fallback。
### 2. Multiple orbit lines
必须确保:
- 锁定切换前先清旧轨道
- 页面隐藏/销毁时清理
### 3. Performance
GEO 轨道点数高,采样率需要按轨道类型分层。
## Acceptance
1. 锁定单颗卫星时只显示一条预测轨道
2. 解锁后轨道立即清除
3. 不同轨道类型下点数可控
4. 页面切换回来不会闪出旧轨道残留
## Satellite Footprint Follow-Up Items
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog:
1. 为 `iridium-next` 新建独立 footprint adapter。
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint。

View File

@@ -0,0 +1,237 @@
# Earth 地理资源与空间卡片展示实施计划
日期:2026-09-21。状态:本地可运行原型。贴图已替换并归档旧图;固定规则、部分核验几何、全球参与方目录、空间卡片、共享移动滚动条与轻量材质调整已实现。完整政治几何、逐段海岸对齐、视觉认可及性能门槛尚未全部验收,不代表可发布。
本计划以已确认的展示方向为准,替代算力专题旧方案中的固定右侧栏、横贯底部双图和隐藏其他面板的布局。空间卡片与算力目录已有本地可运行原型,但仍需完成最终 QA;算力数据采集、指标证据和长期模型设计继续参见[算力与资源态势计划](compute-resource-intelligence-plan.md)。
## 1. 目标与工作边界
地球是空间信息的主体。打开区域对比后,多个区域分别引出连接线,牵引周围的浮动信息卡。地球转动时,地理锚点跟随球体,卡片保持正向可读,通过平滑、有阻尼的运动调整位置并互相避让,形成信息从地球四周延伸出来的感觉。
地球质感参考 UE 3D 截图在易拉宝与 PPT 中的表现:球面有层次、边缘与明暗交界柔和,背景退后,数据与球体构成同一空间。保持当前操作流畅度是硬验收条件。
已授权实施范围包括资源、空间卡片、共享 UI 与性能验证。继续使用现有 Three.js 引擎和资源加载机制。资源版本与实际转换参数以[地球资源说明](../../resources/earth/README.md)和清单为准。
### 当前实现与未完成项
本轮已实现卡片运动、配置控制、内容承载及中性地球材质调整。配置窗口与卡片显示开关已分离,关闭配置不再停用卡片。卡片默认常显基础信息:36px 主数值、年份/指标/口径说明、设施样本与 Epoch 来源、趋势首末年份及单位、证据入口和区域成员数;长列表仅在需要时展开完整图表。桌面卡片约宽 268px,高度随内容自适应,国家约 214px、区域约 223px;展开卡片支持用户调整至最高 480px,位置/尺寸可保存并恢复自动布局。手机保留同等摘要信息。高清 overlay 使用白色 MeshBasicMaterial、opacity 1、toneMapped:false;r128/r160 按 active API 使用对应 sRGB 读写路径。没有新增 GPU draw pass 或高分辨率纹理,但新增并预加载 45 个本地 SVG 旗帜资源。
旧版底边换侧的连续跳动已定位并由路径规划与稳定换侧策略处理。新实现由 `spatial-callout-controller.js` 管理,布局采用事件驱动并保留有效旧矩形;投影连线持续跟随地球,另一侧候选需持续 1800ms 才换侧,12 秒阅读间隔用于抑制不必要的替换。悬停、焦点和详情阅读会锁定卡片,用户拖动复用 `hud-drag.js`,现有 controls 也消费该共享模块。多卡布局通过实际空闲矩形安排最终位置,优先避开已有功能窗;局部 UI QA 已完成,全球性能与最终发布回归仍按后续门槛验证。下方性能数字和旧尺寸的回归记录属于此前原型,不能直接作为当前大卡片的性能验收结论。
- 底图默认“原始底图”(菜单首项),显示菜单可切换并保存;Natural Earth I 保留,NASA 21K 仅作源归档。运行时统一 8K,按需只保留一张 GPU 纹理;成功后释放旧图,失败保留旧图,过期响应不能覆盖当前图源,不新增渲染层。星空逐字节恢复 commit `83a10a6c`,不再校准密度和颜色。高清 overlay 使用白色 MeshBasicMaterial;r128/r160 按 active API 选择对应 sRGB 读写。没有新增长驻高分辨率纹理、后处理或 GPU draw pass;算力卡另新增并预加载 45 个本地 SVG 旗帜资源。Natural Earth 原图偏浅,不把不存在的细节写成恢复结果。
- 地理政策由版本化清单贯穿规范编码、后台聚合、高精几何及低精回退。中国、台湾、港澳、科索沃等已有回归;已核验几何及济州岛补全见资源说明。高精技术就绪与政策可发布分开:`high_precision_ready=true`,`policy_publishable=false`。东沙、中沙、曾母暗沙点注记及经核验海上主张线仍有缺项。
- 规范目录覆盖 45 个国家编码,界面分组为中国、美国、欧洲、不含中国的亚太和其他;分组不当作统计合计。当前空间原型最多同时显示 6 张,后端最多接受 12 个参与方。没有历史观测的地区返回 `null` 历史与覆盖信息;设施样本数永远不作为计算出的国家总量。
- 桌面卡片复用共享 HUD 和连接线,保留稳定 DOM 身份;选择器和来源窗使用共享拖动。背面卡片保留内容并隐去连接线。旋转通过原动画循环更新,投影连线持续更新;布局按事件分帧处理并保留有效旧矩形。`main.js` 的 `isDragging`、`activeTouchPoints`、`hasActiveGlobeInertia` 接入运动模式:手动/惯性快速模式以 72ms/12px 为响应预算,在当前帧完成受限布局,停止后一次 settle;slow auto 保留 1.8 秒换侧和 12 秒阅读保持,用户拖动放置优先。
- 手机复用公共抽屉与深色滚动所有者,使用适合手机阅读的字号;目前是抽屉内紧凑卡片列表,尚未实现手机环绕空间卡片。桌面原型没有强制隐藏新闻、媒体或统计窗口。
- 2/4/6 卡片、背面投影、共享窗口避让、饱和纵列与共享连线动画已有 11 项单元测试。浏览器已检查 1366×768、1920×1080、390×844 及 125%/150% 对应的有效 CSS 视口;单元测试不替代运动观感验收。
### 本轮空间卡片 UI 定向验收(2026-09-21)
本计划保留此前的历史验证记录;它们不代表当前地图源切换或全球 Earth 验收已经完成。当前实现说明只记录所需资源生命周期与 UI 约束,Browser/全球性能结论仍按后续门槛单独复核。
以上是本轮卡片 UI 与局部资产的定向证据。全球地理政策、逐段海岸对齐、完整 Earth 渲染性能、真实设备性能及发布级回归仍按本计划后续门槛单独验收。
## 2. 优先级与不可退让的要求
| 优先级 | 要求 | 执行原则 |
| --- | --- | --- |
| P0 | 固定政治展示规则 | 外部数据、更新、回退及 AI 均不能覆盖项目规则 |
| P0 | 性能不退化 | 在同设备、同数据、同场景下验证;可复现的退化不能用视觉提升抵消 |
| P1 | 地理对应一致 | 贴图定位、海岸线、区域归属、拾取和卡片锚点一致 |
| P1 | 地球关联的空间卡片 | 使用现有连接线与展示机制扩展,不恢复固定大屏布局 |
| P1 | 共享 UI 与移动端一致 | 共用字体、标题栏、按钮、滚动容器及窗口行为 |
| P2 | 卡片内图表和内容完善 | 先验收空间关系与运动,再充实图表和解释 |
## 3. 固定政治展示规则
项目展示采用固定中国视角:中国相关领土及争议区域按项目既定中国视角处理,台湾按中国处理;科索沃按塞尔维亚处理。该口径先于数据源选择,不由数据源默认分类、模型输出或换源结果决定。
实施时建立单一、版本化的规则清单与几何修正包。名称映射只是其中一部分,必须覆盖国家编码、Polygon/MultiPolygon 归属、边界绘制、区域填色、鼠标拾取、高亮、统计及卡片归属。处理后的结果才可以成为运行资源;未经规则校验的新来源不得替换正式资源,旧资源回退也必须通过同一规则。
规则清单应包含:稳定规则 ID、目标区域的明确标识、目标归属、显示名称、几何来源、合并/裁切操作、版本及回归样例。不能只依赖模糊地名字符串替换。语音中未能确认的具体岛礁名称必须在修正清单中明确定位,不能猜测名字后直接移动多边形。
海岸线、陆地行政区域和海上主张线是不同几何层。项目口径在各层一致执行;海上主张线不能作为陆地面直接填充,也不能代替具体岛礁的几何和归属规则。
必须具备的失败测试:外部源恢复原始分类、别名变化、同区域不同编码、几何修改后拾取仍用旧数据、低精度回退覆盖规则。任何一项失败,都不发布该资源版本。
## 4. 已查明的现状
### 4.1 贴图与国界
- 原运行贴图为 `frontend/public/earth/assets/8k_earth_daymap.jpg`,8192×4096;代码直接加载 JPG,未读取该文件配套的地理定位元数据。
- `country-boundaries.js` 使用独立 GeoJSON/PMTiles 几何绘线与拾取;高精度是一个选择状态,存在退回低精度的路径,按钮名称不等于实际正在使用的文件。
- 本机此前检查到 `natural-earth-v1`、`geojson-high-precision` 构建清单,以及独立 base、hover、claim 文件;该构建不能直接等同于完整应用了项目政治几何规则。实施前需要重新读取实际运行版本。
- 已提供的两张问题截图中,一张出现海岸折线与影像陆地错位;另一张有可见岛屿但没有对应线框。截图只能证明现象,尚不能断言根因一定是 UV、投影、低精度回退或缺岛。
- 精度检查必须同时检查“看得到轮廓”和“能选中对应区域”,二者不能各用一套不一致的几何。
### 4.2 World Monitor 对照
本机 World Monitor `463aec0f` 使用独立地表图片与 `countries.geojson`;经纬度归属由点落入多边形的计算决定,二维底图和自带的交互区域也分开加载。
本次文件统计:World Monitor 的中国、日本、韩国分别为 200、60、14 个坐标点;Planet 的低精度分别为 240、65、19;本机高精度文件分别为 14,118、6,952、3,060。点数不等于测绘准确度,但表明不能靠直接换用 World Monitor 的简化国家文件解决近景海岸问题。它的原始归属处理也不能代替本项目规则。
### 4.3 已有共享机制
| 入口 | 当前作用 | 计划复用方式 |
| --- | --- | --- |
| `frontend/public/earth/js/callout-connector.js` | 连接线、元素锚点与路径 | 扩展为多个稳定区域锚点与卡片配对 |
| `frontend/public/earth/js/presentation-controller.js` | 展示生命周期与连接器更新 | 在单项展示的基础上设计受控的多项展示,不破坏现有模式 |
| `frontend/public/earth/js/cruise-sequencer.js` | 巡航时序 | 保留当前巡航行为,区域对比明确自己的生命周期 |
| `frontend/public/earth/js/info-card.js` | 公用详情与移动详情 | 抽出/复用内容渲染能力,不复制成另一套卡片组件 |
| `frontend/public/earth/js/hud-panels.js`、`controls.js` | 面板收起、显示、拖动与约束 | 工具窗继续遵循这些规则;明确 createHUDPanel 本身不包办所有拖动/缩放 |
| `frontend/public/earth/css/hud.css`、`info-panel.css` | 公用外观、字号、滚动与移动抽屉 | 去掉专题私设的大字号、独立控件和白色滚动条 |
## 5. 地理资源:先采用配套贴图的简单方案
### 5.1 选择与准备
默认运行底图为“原始底图”(菜单首项),显示菜单可切换并保存底图源;Natural Earth I 保留可用。NASA 21K 仅保留为源归档,运行包统一使用 8192×4096 的单张 GPU 贴图。官方源版本与许可信息仍以资源清单为准,不能把归档源误写成运行资源。
采用不带政治标注的物理地表图。按需只保留一张运行 GPU 贴图;新源成功加载后释放旧图,失败保留旧图,过期响应不得覆盖当前图源。源包保留 TIFF、世界文件和说明,运行资源统一为 8192×4096 的 Web 图。转换只做尺寸与格式适配,不用生成式图像重绘海岸、岛屿或国界,不随意裁切、旋转或拉伸经纬范围;底图切换不新增渲染层。
下载的原始大文件保存在本地 `downloads/earth/`,不纳入前端发布包。可追溯清单、配准信息与旧贴图归档放在 `resources/earth/`;运行图放在现有 `frontend/public/earth/assets/`。旧图必须先按字节完整归档并核对 SHA256,之后才能替换运行路径。
### 5.2 对齐验证
1. 读取 TIFF/TFW 的像元大小、原点、行列方向、地理范围和 CRS;同时记录 JPEG 派生后的像元中心定位,避免半像素偏移。
2. 明确运行中加载的贴图、边界文件、资源版本与是否发生回退。
3. 使用同一投影和球面经纬转换叠加成套矢量,检查赤道、零经线、日期变更线和高纬区域。
4. 在截图对应海岸、近海岛屿、台湾周边和其他不同经度的典型海岸逐点验证,区分整体偏移、局部概化和缺失几何。
5. 政治修正后的同一份区域几何用于标注、高亮和拾取。所有需要烘焙的线也从这份数据导出,不能在图片和矢量上分别手工描线。
6. 小岛、岛礁或海岸不满足要求时,先补充对应区域数据并重新配准;仅当这条简化路线确实不够用时,再考虑局部细节或分级资源。
成套资源声明配准不等于本项目已经逐像素验收。不能承诺卫星影像每一处潮线都与概化海岸线完全一致;验收需保存具体偏差和缺失清单,不能靠强行拉伸边界迎合图片。
### 5.3 常态渲染与交互
优先预合成常态显示所需的边界,以减少额外线段绘制。需要独立国界开关时,评估复用现有覆盖层或同定位的透明边界层;不默认再常驻一张 8K 纹理。矢量数据继续承担区域判定和选中高亮,不在每帧重新解释整份 GeoJSON。
本阶段不以替换 Three.js、引入新 GIS 平台或建设全球在线瓦片服务作为前置条件。
## 6. 整体视觉方向
参照资产:`Planet-Handoff-20260916` 中的完整易拉宝效果稿和卡片展示 PPT,尤其 PPT 内来自 UE 3D 的原始星球画面。海报中的线点含概念叠加,不能拿来充当精确地理数据。
- 地球保持正圆与正确地理位置,视觉上成为主体;不把纹理精细度等同于画面品质。
- 球面采用柔和、连续的明暗层次,边缘有薄而自然的大气过渡;地理细节仍然可读。
- 降低背景星点对比与密度造成的干扰,让背景退后;变化以参考图与实际渲染对照决定。
- 数据节点与连接线有轻重层级,避免强霓虹、油亮塑料感、浓重太空雾和粗硬轮廓。
- 优先在现有材质、光照与透明度范围内调整;新增后处理、辉光或高分辨率透明层须先证明成本。
- 用同一位置、距离、光照条件对比前后效果,近景与远景都检查边缘、深度冲突和细节。
## 7. 区域卡片:从地球四周延伸出来
### 7.1 行为模型
一个区域对应稳定的地理锚点、连接线和卡片 ID。锚点随球体旋转;卡片处在屏幕空间,保持正向、稳定大小和内容身份。转动时变化的是屏幕位置与线的路径,不重建卡片、不交换身份、不清空正在阅读的内容。
卡片根据区域投影分布到球体周围,保留疏密和空间关系,不固定平均分成八个方向,不横排占满屏幕。使用有限的避让和有阻尼的缓动,最终能够安定;不持续抖动或为了追踪锚点逐帧跳位。
### 7.2 布局与生命周期
- 沿用现有连线的起点/终点接口和动画语言,先验证 2 张、4 张、6 张卡片;6 张是原型测试档位,不是已确定的产品上限。
- 保持地域锚点与区域绑定;算法可以移动卡片,但不能移动真实地理坐标来换取好看的布局。
- 优先避让其他卡片和已有工具窗,不自动隐藏新闻、直播、统计、图层面板。
- 对卡片换侧设置稳定区间,避免锚点越过某条中心线就立即左右互换。
- 背面区域不画成穿过地球的前景锚点。建议保留卡片身份,用边缘提示表达背面方位;完整隐藏、收束或边缘提示的最终表现用短动画原型比较后决定。
- 用户正在操作的卡片短暂锁定其屏幕位置;是否允许长期固定/拖动偏移,先以现有交互能力做原型验证,不增加未经确认的复杂模式。
- 增删区域、切换年份、退出对比、隐藏相关图层和请求失败都要有明确的清理与恢复逻辑;旧响应不能复活已关闭卡片。
- 区域名称锚点用于对比展示;不能把区域中心点画成设施位置。
### 7.3 卡片内容与参与方
先验证空间展示,再完善图表。首轮卡片可以只包含名称、一个有来源的关键指标、参考期及来源入口;固定侧栏和横贯底部双图的旧设计不再继续扩展,普通浮窗加分页的方案也不是最终展示方向。
参与方必须支持全球扩展,不能在接口和 UI 中永久硬编码成中美两项。日本、韩国等可以作为独立对象;亚太、欧洲可用于选择菜单的分组或区域筛选。具体默认组合与区域汇总口径还需在完整 UI 原型中确定,不把早期讨论的菜单建议当成已确认设计。组合区域与成员国不能重复计入同一分母。
缺少同口径历史数据的参与方仍可展示公开设施覆盖及明确的缺失状态,不伪造总量、得分或排名。全国总算力、公开 AI 集群样本、TOP500 FP64 和创新指数继续分开。2026 年未核验值不得加入正式历史。A4 AI 解读不阻塞空间展示验收。
## 8. 共享窗口与移动端
设置、来源、图层等操作窗口继续复用已有标题栏、按钮、拖动和按能力启用的缩放逻辑。不得用单独字号和固定网格重新构建一套视觉体系。数据更新和语言切换应保留窗口位置、尺寸、选中对象和滚动进度。
移动端复用原底部抽屉与详情展示。区域卡片数量受可用空间控制,保证地球与主要交互可见;可收束为可切换的关联卡片,但具体排列需要实机原型确认。不能把桌面全部卡片原尺寸堆到手机上。
白色滚动条问题从 `.earth-mobile-drawer-content` 及实际滚动所有者核对,统一复用 Earth 的深色滚动条样式与滚动容器。覆盖图层、搜索、态势、算力、新闻和设置页;底部导航保持固定,避免嵌套重复滚动,不仅对算力页补一条局部 CSS。
## 9. 性能基线与硬验收
### 9.1 基线
资源替换前的画面可通过归档旧贴图复现。保存设备、浏览器、GPU、视口、DPR、图层开关、数据量、相机路径与资源版本。在相同条件下重复测试,至少记录 5 次稳定运行,并区分冷启动、热启动和稳定旋转。
观察:首个可操作画面时间、首个有效地球画面时间、帧时间 p50/p95/p99、长帧/长任务、交互延迟、JS 堆与纹理内存估算、draw calls、纹理/对象数、网络字节及请求次数。只比较平均 FPS 不足以证明没有卡顿。
### 9.2 实现预算
- 运行贴图先维持 8192×4096。相同格式下纹理像素量不增加;RGBA8 粗估基础约 128 MiB,完整 mipmap 约 171 MiB,不能把 JPEG 文件变小当成显存同比下降。
- 保持已有 mesh、图标实例与纹理缓存;旋转不重新解码图片或构造国家几何。
- 地理投影和轻量位置更新与渲染帧同步;碰撞布局只在锚点明显变化、卡片变化或窗口尺寸变化时重算,不让每张卡片各起一套动画循环。
- 卡片 DOM 保持稳定,图表仅在数据/尺寸变化时更新;避免每帧测量布局后又写布局造成同步回流。
- 网络、缓存、监听器、纹理及卡片数量均有边界,关闭/重开多轮不得持续增长。
- 默认不增加新的高分辨率常驻覆盖层、昂贵后处理或全部区域的最高精度常驻数据。
### 9.3 判定
同条件下不能出现可复现的流畅度、加载、交互或资源占用退化。测量噪声必须记录并重复验证,不能事先允许固定比例退化来降低要求。超出预算先调整效果、卡片密度或实现,再复测。桌面与移动分别验收;没有真实手机数据时只能说明模拟检查通过,不能宣布手机性能达标。
### 9.4 内容卡扩容前的原型验证记录(2026-09-21)
同机 Headless Edge、桌面 1440×900 / 手机尺寸模拟 390×844、DPR 1、自动旋转,每种尺寸 5 次;预热 1 秒、帧时间采样 2 秒。该短采样用于定位,不能代替真实设备长时交互验收。原始基线标识 `2026-09-20T18-30-56-184Z`,最新对照标识 `2026-09-20T21-07-40-495Z`(UTC);完整 JSON 留在本机临时验证目录 `planet-earth-performance-baseline`。
| 项目 | 原基线 | 本轮结果 | 判断 |
| --- | --- | --- | --- |
| 桌面无卡 p95 帧时间,5 次中位数 | 7.09 ms | 7.10 ms | 接近;不能据此宣称完整性能通过 |
| 手机尺寸模拟无卡 p95 | 7.00 ms | 7.20 ms | 未做真实手机测量 |
| 桌面 2 / 4 / 6 卡 p95,5 次中位数 | 无同场景旧实现对照 | 7.10 / 7.10 / 14.00 ms | 六卡比本轮无卡场景有明确成本,仍需优化 |
| 六卡采样窗口长任务 | — | 5 次均为 0 | 没有 50 ms 长任务不等于没有掉帧 |
| 桌面 DOMContentLoaded 中位数 | 1.262 s | 1.627 s | 已移除 Google Fonts 的约 10 秒阻塞;仍不宣称加载性能与基线完全持平 |
| 桌面基础 JS 堆中位数 | 约 50.2 MiB | 约 53.2 MiB | 不是显存;未替代纹理/绘制统计 |
每轮另做 5 次专题开关并强制 GC 检查,关闭后的保留堆约 25 MB,首末差约 0.28–0.33 MB;这不构成长期无泄漏证明。浏览器单独核对关闭后的卡片悬停/焦点监听器为 0,来源窗与拖出容器的选择器均能关闭。
最终 Bun 构建、前端规则检查、文档一致性检查与 `git diff --check` 通过。既有前端 smoke(使用其 API mocks)通过 321 项路由/交互检查,包括公共页面、鉴权、管理导航、移动布局和管理页缩放;Earth 空间卡片另用真实本地接口检查,不能混同两种证据。
布局求解已分帧执行、减少重复候选评分、优先前景区域,在合格布局找到后停止重复试排;工具栏按可见按钮而非透明容器避让。CPU 采样发现共享 SVG 的 `getTotalLength()` 同步布局热点,已改为按已知折线点计算长度,保留原绘线动画。11 项布局/投影/连线测试通过。1366×768 浏览器中 4/6 卡、旋转后、稳定后及工具栏展开时未发现与卡片、选择器或现有 HUD 的矩形交叠;1920×1080、1093×614、911×512 同样完成六卡边界检查,后两项是浏览器缩放的等效 CSS 视口,不冒充真实设备缩放测试。46 项相关后端测试通过,高精与回退的台湾、藏南、阿克赛钦、科索沃、济州岛浏览器拾取均符合规则。官方几何缺项、完整海岸对齐、六卡性能与真实手机仍未通过最终验收。
## 10. 分阶段实施与交付门槛
开发模型分工、文件所有权、并行限制和上下文控制按[开发 Agent 模型分配计划](development-agent-model-allocation-plan.md)执行:Sol 主控与核心实现,Terra 承担明确子任务,Luna 处理有边界的辅助工作,Astra 负责高风险设计及疑难审查。模型分工不改变本计划的政治规则、视觉方向和性能验收门槛。
| 阶段 | 工作 | 必须交付的证据 |
| --- | --- | --- |
| S0 资源准备 | 新贴图、配准信息、旧图归档、来源清单 | 新旧尺寸/哈希、解码与球面渲染检查;本轮范围 |
| S1 规则与配准 | 固定政治清单、实际提供者诊断、配套边界对齐 | 规则测试、截图问题点对照、回退测试;不能只报高精度开关已打开 |
| S2 空间运动原型 | 复用连接器,多区域锚点、卡片稳定身份与避让 | 转动/缩放/换侧/背面/退出短录屏,2/4/6 卡档位及性能差值 |
| S3 视觉与组件 | 柔光质感、背景层次、共享字体与移动滚动 | UE/PPT/海报参照对照、多个视角、桌面/手机及缩放截图 |
| S4 内容联动 | 多参与方、真实指标与来源、时间/设施联动 | 可操作路径、缺失/错误状态、来源与口径验证 |
| S5 回归与发布准备 | 性能复测、旧图层/巡航/工具窗回归、同步文档 | 完整验收记录;需要发布时另走仓库发布流程 |
S1 未通过,不能把新图片称为边界问题已解决。S2 的空间交互未确认,不能直接投入完整图表和全部国家卡片。性能或政治规则任何一项未通过,后续阶段不能据此宣布完成。
## 11. 验收清单
- 政治:标签、几何、填色、拾取、统计、卡片与回退源口径一致;科索沃等明确规则有专门用例。
- 地理:截图所指海岸和小岛、台湾周边、日期变更线、高纬度;记录偏差而非只肉眼说“差不多”。
- 视觉:球体不变形,边缘柔和但地理清楚,卡片从区域延伸且不压满地球;远距无闪烁和黑块。
- 交互:转动、缩放、选中、背面、卡片增删、触摸、键盘、语言切换、退出、窗口大小变化及旧响应竞态。
- 共存:新闻、媒体、统计、图层及原详情卡保持可用;不通过强制隐藏制造布局空间。
- 移动:公共抽屉无白色原生滚动条,底部导航固定,只有预期内容区滚动,文字和操作不被截断。
- 响应式:1366×768、1920×1080、约 390×844,以及 125%/150% 缩放;模拟与实机结果分别记录。
- 回归:公共页面、鉴权、管理路由、导航/搜索/标签页、既有 BGP/新闻/动捕巡航与旧图层。
- 性能:与归档基线同条件比较,保存帧时间、内存、网络和交互证据;循环打开关闭验证资源释放。
- 文档:更新实际变更的中英文手册、Earth context、图层规则和资源清单;计划与已实现行为不能混写。
## 12. 来源、恢复与完成定义
- [Natural Earth 栅格与矢量配准说明](https://www.naturalearthdata.com/downloads/10m-raster-data/)
- [Natural Earth I 资源页面](https://www.naturalearthdata.com/downloads/10m-raster-data/10m-natural-earth-1/)
- [Natural Earth 使用条款](https://www.naturalearthdata.com/about/terms-of-use/)
- [现有 Presentation 解耦计划](earth-presentation-decoupled-architecture-plan.md)
- [原高精度边界计划及历史说明](earth-high-precision-boundary-tiles-plan.md)
- [地球前端上下文](../technical/zh/earth-frontend-context.md)
- [渲染层次规范](../technical/zh/earth-render-layer-order.md)
旧贴图恢复路径和 SHA256 见资源清单。先核对归档哈希,再将归档副本复制回原运行资源路径并重建预览;不以回退图片为由撤销其他用户改动。新资源和旧归档均不得无记录覆盖。
最终完成意味着:展示规则固定且经验证,地图与区域选择对应,多个卡片确实由地球锚点牵引并稳定运动,操作组件和手机滚动保持一致,实际性能不低于基线。单张效果图、构建通过或“高精度”名称都不能单独证明完成。

View File

@@ -99,7 +99,7 @@ AIS vessel collectors use a different persistence path from regular `CollectedDa
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
TOP500 and Epoch AI compute sources do not always provide usable coordinates. Epoch's official CSV `Location` may contain only a country or be empty even when a row has latitude/longitude; it is never treated as a precise site without the required source semantics. The unified Earth compute-center endpoint uses valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without renderable coordinates remain in the complete `unresolved` list instead of being rendered from a local registry, country centroid, or guessed headquarters. `precise`/`site` keep source coordinates; city-level and confirmation-required points may receive only a deterministic display offset keyed by `source` + `source_id`, with the original coordinate untouched. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
@@ -209,6 +209,11 @@ def parse_response(self, html):
## VII. Scheduler
The `earth_news_rss_refresh` system job checks regional feeds on startup and
every 10 minutes. Homepage GET requests read saved news. Datasource schedule
synchronization preserves this system job. Freshness, locking, and cooldown
details are in the [news source guide](/home/ray/dev/linkong/planet/docs/technical/en/earth-news-sources.md).
```python
# Register all collectors into scheduled tasks at startup
def start_scheduler():
@@ -460,3 +465,27 @@ POST /api/v1/datasources/trigger-batch
The request body may pass `source_ids` for selected rows. Without `source_ids`, the backend filters by `product`, `module`, `is_active`, `run_status`, `collected`, `credential_status`, and `q`. The endpoint skips disabled sources, sources already running without `force`, and sources still inside their frequency window, then returns `triggered`, `skipped`, and `failed` groups.
**Core file**: `backend/app/api/v1/datasources.py`
## Compute source contracts and topic APIs
Epoch uses its official public CSV, retaining theoretical FP16 performance, original dates, status, anonymization/rounding flags, and upgrade relationships. Anonymous rows use content fingerprints plus equal-row ordinals to preserve snapshot multiplicity, without claiming persistent facility identity or assigning coordinates. Unknown countries remain flagged and are excluded from selected-country counts. TOP500 uses system-detail IDs, reads the edition from the list URL, and converts header/cell units into FLOP/s with an explicit FP64 LINPACK basis. Public pages may expose only part of an edition; this does not establish full TOP500 coverage.
Neither collector returns sample data on failure. Empty fetches or transformed results, schema errors, and losses exceeding 40% of the previous snapshot fail while retaining prior data. Incomplete writes are rolled back. TOP500 location migration copies only a unique exact name and canonical-country match to the new system ID, retaining the historical key; rank is never a migration key.
`compute_history_defaults.json` ships the original report data. `compute_configuration.py` reads a complete override from the `compute_comparison` category in `system_settings`; absent records use bundled defaults without migrations or manual initialization. `compute_history.py` performs same-year aggregation. Reference periods and publication dates remain distinct, and unknown publication days remain unspecified. `/api/v1/data-products/compute/series` reads configuration on each request with `Cache-Control: no-store`, offering global share, fixed US + China bilateral share and FP32 absolute observations. Innovation gaps remain independent. An empty override stays empty instead of silently restoring observations.
Under the same prefix, `GET/PUT /configuration` and `POST /configuration/reset` require admin/super_admin. Writes accept `{revision, config}` and reset accepts `revision`. Pydantic validates canonical countries, finite values, share totals, duplicate metric/year pairs and report references; each metric/year has one report basis. PostgreSQL category uniqueness and row locking serialize first writes and updates; stale revisions return 409. Configuration and before/after audit records commit in one transaction. Reset removes the override so subsequent reads use the current bundled defaults. `version` is a configuration revision, while `reviewed_at` is the supplied review date. Admin-curated data is not automatically labeled official. Source URLs are references, not scrape instructions.
`/api/v1/data-products/compute/facilities` reads successful snapshots with `compute_contract=v1`, applying country filters and per-source counts in SQL. Epoch excludes planned, decommissioned, superseded, and suspected duplicate records; TOP500 means list observations. Cross-source deduplication remains pending. Historical cutoffs use collection time, never pretending a recently fetched old source was available earlier. Geographic results are bounded and report truncation; total record counts are independent of this limit. Official site helpers are keyed by exact `source` + `source_id`; their access-map centers are site-level references only and retain their official URL in `source_note`.
### CelesTrak download windows and caching
Both active and fallback-group downloads skip HEAD preflight and send GET directly: a preflight may consume the upstream download allowance before the body request. Valid complete cache entries are reused for two hours, then refreshed with conditional GET when validators exist, or ordinary GET otherwise. A 304 reuses validated cache; only the explicit “GP data has not updated” 403 permits a valid-cache fallback. Other 403 errors remain failures. Fallback groups also respect cache age rather than reusing old files indefinitely.
A nonempty GP record list is validated in `.part` before atomically replacing the cache; failed downloads preserve the previous valid payload. Length comes from the GET response, and compressed wire length is not compared with decoded bytes. Other collectors retain HEAD metadata and Range resume support. Restoring a raw cache does not populate the database: trigger a new collection task afterward.
### Earth news encoding
Before parsing RSS/Atom, `earth_news.py` selects encoding from the BOM, HTTP charset, XML encoding declaration, then UTF-8, and decodes strictly. GBK/GB2312/GB18030 feeds no longer silently replace Chinese bytes with corrupted text. Unknown encodings or invalid bytes produce `format_error` instead of damaged titles. `test_earth_news_encoding.py` covers titles, summaries, display serialization and failure status.

View File

@@ -31,8 +31,16 @@ flowchart LR
5. The listener merges short-window same-layer events, invalidates cache, and broadcasts `earth_updates`.
6. The outbox row is marked consumed only after successful broadcast; failed rows stay retryable.
Cache rebuilds are coalesced by key. Concurrent requests in one process share the build result or error, while cross-process cold misses wait for the Redis lock holder. Long builds renew the lease, and publishing or releasing checks ownership. If the owning request is cancelled, a waiter can take over with its own database session. Available stale data is returned immediately. Without stale data, a waiter returns HTTP 503 after the default 30 seconds. Redis coordination failures return the current build result, with concurrent requests in that process still coalesced. A failed build returns stale data when available and otherwise propagates the error.
DB changes no longer create default `earth_refresh` jobs, so they are not blocked by long same-source collection or clearing jobs. `earth_refresh` remains for manual cache clearing and non-DB refresh hints.
### Outbox Writes and Retention
The trigger obtains `event_id` from the sequence, then writes the payload containing that ID to `earth_data_change_events` once and sends `NOTIFY`. It does not update the same event after insertion. An `earth_news_items` update that only changes `last_seen_at` or `updated_at` emits no Earth change event; visible content changes still emit one. Those timestamps continue to record news-source health and freshness.
Consumed events are retained for one day by default, configurable through `PLANET_EARTH_OUTBOX_RETENTION_DAYS`. Every 60 seconds, the listener removes at most 1,000 expired consumed rows, and only when the separate retention index is valid; pending rows never match the deletion condition. A PostgreSQL advisory lock prevents concurrent cleanup across instances. The [standalone Outbox maintenance script](/home/ray/dev/linkong/planet/scripts/maintain_earth_outbox.py) creates the index with `CREATE INDEX CONCURRENTLY`, outside the application startup transaction. The script is read-only by default and also supports batched deletion and a one-time compact mode. See the [operations runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md) for execution steps.
## Data Job Queue
`collection_tasks` is the unified job ledger. Workers claim `queued` jobs with PostgreSQL `FOR UPDATE SKIP LOCKED`; write jobs for the same `source` run serially, while different sources may run in parallel.

View File

@@ -321,7 +321,7 @@ The new vessel list entry point is no longer the legacy `/api/v1/visualization/g
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
```
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](backend-collectors.md).
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](/home/ray/dev/linkong/planet/docs/technical/en/backend-collectors.md).
## Custom REST / WebSocket Mapping Runtime

View File

@@ -1,5 +1,15 @@
# BGP Context
## Prefix geography lookup
Lookups preserve the OpenGeoFeed, IPtoASN, then NRO source order, address-family
checks, and full start/end range containment. Within each source, the original
prefix mask length and record ID determine ordering. `planet_bgp_try_inet` and
`planet_bgp_try_masklen` return NULL for invalid historical values. Database
initialization installs these lightweight functions. A separate maintenance
script installs the combined GiST range index, as described in the
[operations runbook](/home/ray/dev/linkong/planet/docs/technical/en/ops-runbook.md).
## Current Goal
The BGP module is being evolved from an anomaly-only demo into a layered observability pipeline:

View File

@@ -1,5 +1,21 @@
# Earth Frontend Context
## HD surface and atmosphere materials
The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled.
Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.075, 0.16]; night shading has its own geographic texture ghosting and cool ambient branch. A 0.55 power of the locally tone-mapped texture retains shadow variation, followed by separate muted land and deep-indigo ocean tints and low ambient floors. Day/night mixing remains linear with one final sRGB output. Day exposure and reflections are unchanged; no synthetic city lights are added. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
## Configurable compute observations
Compute cards read annual observations from `/api/v1/data-products/compute/series`; the admin Compute comparison section is the maintenance entry point. `compute-topic.js` fetches with no-store, listens for same-origin `planet-compute-configuration` notifications, and checks every 60 seconds while visible and enabled, plus when the page becomes visible. Observation refresh does not reload facility layers. `destroy()` releases the timer, channel and visibility listener. Country membership, full regional coverage and missing-value semantics remain backend-owned.
## Surface texture provenance
On 2026-09-21, `assets/8k_earth_daymap.jpg` was replaced with an 8192×4096 Web derivative of the paired Natural Earth I raster. Its loading path and rendering code remain unchanged. The source is a 16200×8100 EPSG:4326 image; the previous JPEG is preserved byte-for-byte under `resources/earth/archive/2026-09-21/`. Source/license records, the catalogue versus embedded-version discrepancy, SHA256 hashes, JPEG parameters, and recalculated world-file metadata live in `resources/earth/natural-earth-1/`. Large originals remain in Git-ignored `downloads/earth/`, outside the frontend release bundle.
Basemaps and TV channels share `hud-select-menu.js` for the HUD popover, list, selection mark, keyboard navigation, and positioning; `tv-source-menu.js` only adapts channel copy. Basemaps disable search, pagination, and counts. The trigger owner identifies popover clicks as internal to the mobile drawer.
This document describes the current real structure of the Earth display frontend. The focus is on helping future changes to the HUD, layers, media panel, real terrain, and BGP visualization avoid repeating past structural and state-sync pitfalls.
Related references:
@@ -56,7 +72,7 @@ Responsibilities:
- Layer module integration
- Earth-level state synchronization
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
Earth treats `/ws` `earth_updates` as refresh hints only; real data is fetched again from `/api/v1/visualization/...`. Backend database-driven refresh now has the listener clear cache and broadcast directly instead of going through the default `earth_refresh` job queue. After `database_changed`, the frontend applies the per-layer `clear_then_reload`, `reload`, or `delta` strategy. News-only changes leave the summary cache intact; a batch that also changes another layer still refreshes the summary. `clear_then_reload` must clear Three.js objects before a no-store refetch, and summary is only a consistency check, not a reason to skip a layer reload when the count is `0`. See [Data Jobs and Outbox Architecture](/home/ray/dev/linkong/planet/docs/technical/en/data-job-earth-sync-architecture.md) for the technical pipeline and [Business Architecture and Data Flows](/home/ray/dev/linkong/planet/docs/technical/en/platform-data-flows.md) for business data flows.
### 3. Earth Control Layer
@@ -161,6 +177,12 @@ Each module is responsible for its own:
`tv-source-menu.js` reuses HUD and legend-list styling in a popover with search, a scrolling list, and a fixed count footer. `tv.js` requests 50 entries from `/api/v1/tv/streams` and caches channel details; search runs against the complete backend catalog, and pagination uses `next_offset`. Cancellation and a request generation counter prevent stale results from replacing a newer search. Catalog refresh uses `selected_id` to restore a channel outside the first page.
`tv-playback-policy.js` owns live recovery decisions: all non-fatal HLS events remain with the player, while fatal network/media failures receive at most three recovery attempts. Switching requires automatic fallback to be enabled, no explicit user channel selection, and an embeddable backup. Async callbacks in `tv.js` verify player identity and channel; native video errors no longer bypass HLS recovery. `tv_streams.py` replaces the old bundled CCTV-4 CDRM URL, including saved copies, with WCETV's America-edition HLS stream, names the edition explicitly, and preserves custom streams. An HTTP success does not establish video decodability.
Sources with verified CORS support may explicitly set `use_proxy:false` for direct browser playback without segment forwarding. Other sources continue to use the TV proxy by default. The bundled WCETV source uses direct playback.
Source information reuses `createHUDPanel` and its existing 0.22-second CSS transition. The shared controller measures content into `--hud-body-content-height`, avoiding the non-interpolable change from `max-height: none` to zero. ResizeObserver updates content measurements and compensates the top position of upward-expanding dragged panels during animation, keeping their bottom edge fixed without disabling the body transition. The shell's top transition is paused during compensation to prevent drift from double interpolation.
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
@@ -231,7 +253,7 @@ All material, layer, satellite, BGP, cable, terrain, celestial, and other style
## Render hot paths
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object.
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object. In dense cable areas, screen-distance picking compares satellites, BGP, compute and other point objects before using cable lines as the fallback; hover and click share the same ordering so small objects threaded through a line bundle remain selectable.
Satellite breathing runs in vertex shaders. `satellite-position-worker.js` moves full SGP4 position and initial trail calculations off the main thread; the main thread applies snapshots to interaction coordinates and render buffers. `satellite-propagation.js` supplies the same mathematics to the Worker, predicted orbits, and synchronous fallback. Reload, clear, and altitude changes must terminate prior work so stale snapshots cannot resurrect cleared layers.
@@ -268,7 +290,7 @@ The surface hover tooltip preference is persisted by `controls.js` as `shared.su
The real satellite altitude preference is persisted by `controls.js`, while the rendering state lives in `satellites.js`. When enabled, the real radius from SGP4 is compressed logarithmically into the current Earth visual radius range. When disabled, satellite dots, trails, and predicted orbits all return to the legacy same-sphere display. Toggling this setting must refresh satellite positions and clear trail buffers so a trail never mixes both height models. `maxRealAltitudeOffset = 25` is a visual cap tuned for the current camera and `earthRadius = 100`: GEO / MEO remain clearly higher than LEO, but the highest orbits stay within about 25% beyond the globe radius so selection targets, red trails, and the globe do not feel disconnected.
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for the whole future orbit, projecting the inertial orbit plane onto the current globe pose; that keeps the line closed and keeps the visual orbit inclination aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
SGP4 propagation returns an inertial-frame position, so it must not be drawn directly as Earth-fixed longitude / latitude. `satellites.js` uses `gstime` to convert ECI/TEME positions to ECF, then maps that result into the same Three.js axes as `latLonToVector3()`. Satellite dots and short trails use Earth-fixed coordinates for each sample time, representing the object's current position relative to the globe surface. The locked predicted orbit uses the `gstime` from the lock moment for an open full-period prediction curve, projecting the inertial orbit plane onto the current globe pose. No first vertex or closing chord is appended; arc-length highlights travel toward increasing prediction time, and visual inclination stays aligned with the details card. Fallback predicted orbits must also use a real RAAN + inclination orbital-plane formula, not treat inclination as a constant latitude.
Boundary precision is stored separately by `country-boundaries.js` under `planet.earth.boundaries.highPrecisionEnabled`. When high precision is off, Earth keeps using the bundled low-precision `countries-admin0.min.geojson` fallback even if high-precision manifest/PMTiles files exist locally. When high precision is on but the artifact is missing, the Earth toolbar settings call `/api/v1/earth/boundaries/build` and poll progress. After success, `reloadCountryBoundaries()` hot-swaps the boundary layer without refreshing the page. Boundary hover is independent from interactable hover: a country polygon remains highlighted whenever the surface coordinate is inside it, while the tooltip can still prioritize a satellite, vessel, BGP marker, or other interactable.
@@ -349,7 +371,9 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps projection-based clustering, and `none` disables it. Vessels explicitly disable clustering and avoidance so incremental updates do not rebuild cluster topology. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
`globe-navigation.js` retains the safe zoom-distance mapping. Mouse and single-touch drags uniformly use legacy screen deltas, 0.18 smoothing and release inertia, including movement outside the globe silhouette; ray/sphere intersection no longer gates dragging. Normal zoom retains legacy sensitivity. Once the globe exceeds the viewport, a smooth surface-distance/viewport cap limits excessive rotation at extreme zoom. Horizontal motion uses geographic yaw, vertical input changes tilt, and quaternion math prevents multi-revolution reversal. Pinch transitions and pointer capture remain intact.
Surface markers retain category icons and fixed geographic cluster membership. Numbered solid circles expand through a 320ms tween above 250% zoom; zoom never regroups members, and 500% has no aggregate badges. Vessels retain independent heading arrows without clustering. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md).
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
@@ -385,3 +409,190 @@ Therefore:
For console structure, see:
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
## Spatial compute cards
`compute-topic.js` owns periods, measures, participants, and evidence. `compute_catalog.py` supplies canonical names, picker groups, and regional label anchors for 45 currently registered participants. Unknown historical values remain missing. APIs are no longer limited to CN/US. Group labels never become overlapping statistical denominators, and regional coordinates must not become facility locations.
Earth's existing Roboto Condensed face is now served locally from `assets/fonts/`, with its source and license in that directory and the shared declaration in `css/fonts.css`. The remote Google Fonts stylesheet no longer blocks the page; font family and weights are preserved.
`spatial-callouts.js` provides geographic projection and shared scopes; `spatial-callout-controller.js` selects expanded compute and news cards independently. `callout-limits.js` persists separate 1–3 card limits under `planet.earth.callout-limits.v1`, defaulting to one each. Object details, news and compute register with shared `callout-stage-arbitrator.js`. Real anchors determine the flank with 15px side-change hysteresis. Vertical anchor positions determine order; initial near-ties prefer selection, news, then compute, and later swaps require more than 60px separation. Pinned rectangles reserve their intervals before automatic cards receive vertical corridors. Limited capacity allows peripheral overlap while retaining full cards. Capsules and temporary hover expansions neither occupy stage slots nor become dynamic sibling obstacles. Hovering any card and the following 2.5 seconds pause automatic focus switching; existing cards continue smooth following inside their corridors. Dimensions and reading state remain separate from business coordinates.
Object presentation uses `PresentationController`, `object-callout-anchor.js` and `object-callout-layout.js`. Selecting BGP, compute, vessel, satellite or cable geometry draws a connector before showing details. `rawPoint` and `rawVisibility` retain the real projection while a separate smoothed source follows visible highlight geometry. Stage assignment uses the raw projection to prevent card-to-anchor feedback jitter. Rear objects retain a globe-rim direction cue, which is never treated as a real coordinate or depth. Object details share corridors with regional cards and follow with faster horizontal than vertical damping; hover does not freeze this motion. Manual drops take priority and release only when a rear object has confirmed raw visibility below -0.65; missing depth never implies release. Cable anchors stay on selected geometry and satellites track NORAD identity. Connectors choose an edge facing the source, balance the two segments and connect directly when nearly aligned; dragging adjusts the perimeter endpoint automatically. Mobile retains the compact popup and full-detail drawer. Stale asynchronous presentations cannot dismiss newer selections.
`configVisible` and `cardsEnabled` are separate. Closing the configuration window with X or Escape preserves card visibility. The desktop title bar uses the news panel’s card icon and selected style, with `aria-pressed` reflecting `cardsEnabled`. Turning off cards through this button or the mobile switch releases the connectors and restores the previous map query and layer state. Configuration and source windows bind HUD typography directly so reparenting cannot enlarge their text. The main card presents the selected compute metric as a large share percentage or an absolute value with unit, observation year, historical context, and source entry. Missing comparable data is explicit; no empty trend chart is rendered without observations. Facility records and coverage remain complete in the expanded auxiliary area, where sample counts are evidence context rather than a compute value or global coverage rate. Long lists open only when needed. Mobile preserves the same single metric meaning in its accordion. Local position/size persistence remains available. Stable DOM, dimensions, and reading state survive updates, and stale responses cannot re-enable cards.
Expanded cards continuously follow targets inside arbitrated vertical corridors; active dragging takes over placement and decorative floating remains paused. Capsules independently follow geographic projections. Hover expansion retains full dimensions, body and actions without displacing siblings. News typing can continue in either permanent or temporary hover expansion. Connector anchors update each frame. `shared.calloutLineStyle` persists thin/bold preferences through desktop/mobile controls and `body.dataset.calloutLineStyle`, without changing business anchors.
The shared `CalloutConnector` calculates polyline length from its points instead of reading `getTotalLength()` after each SVG write, avoiding repeated synchronous layout during motion. Initial draw animation is retained, so existing cruise presentations receive the same optimization.
On mobile, compute uses a compact regional overview in the existing drawer. Rows are an in-place accordion: tapping the same header collapses it, tapping another collapses the previous row, and at most one expanded card is shown. The expanded card keeps trends, region members, facilities, media, and sources; there is no region subpage, while Settings remains a separate on-demand page. The shared HUD collapsible panel animation is reused and respects reduced motion. Summary loading remains active when map cards are off; enabling cards shares the map loading event and cancels stale requests. Drawer handlers use `composedPath()` so inner HTML replacement cannot collapse the active row. Facility names remain in a scroll region capped at 240px, with left-aligned wrapping rows and at least 42px touch height.
`/series` retains source-backed annual observations; the default primary metric is global compute-device scale split into basic, intelligent, and supercomputer capacity for 2020–2023. Reviewed absolute observations are 302 for 2022 (`caict2023`) and 435 for 2023 (`caict2024`); Japan's global share is 6%/5%/4%/4% for 2020–2023. Product groups APAC excluding China and Other do not map directly to report Japan or “other”; Europe remains `null` until membership aligns with the report. Bilateral share keeps a fixed US + China denominator. Missing primary values may show the latest reported year (currently 2023), while an explicit 2026 selection stays missing. Unconfirmed comparability does not produce year-over-year change. `/facilities` filters and counts unique canonical country codes in successful collection snapshots; coverage counts remain auxiliary, and are not capacity or a global coverage rate. `compute_catalog.py` groups China, US, Europe, APAC excluding China, and Other, covering 45 canonical country codes; these display groups never become statistical denominators. `earth_geography_policy.py` shares display ownership with map policy, grouping Taiwan/Hong Kong/Macao source subdivisions under China while retaining `source_country`. API review metadata is `reviewed_at=2026-09-22` with the matching version date. An independent Epoch share mode is not implemented. Innovation differences are supplied only when both CN and US are selected.
### Boundary surface attachment
Normal borders, coastlines, claim lines, highlights and landing points use `earth-surface.js` to share the HD sphere UV-triangle surface. Borders draw after the texture but before landing points. Highlight lines are always transparent-queue objects with depth testing, preventing through-globe display or overpainting markers. Geometry is computed and cached at construction, not reprojected per frame. See the [render-layer specification](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
### News ticker edge fading and breaking state machine
A separate pseudo-element paints the ticker background and horizontal rules, fading to full transparency outside the text bounds. The shell has no rectangular backdrop filter, box shadow, or text mask. The region label sizes to its content and the scrolling text follows directly without fading.
The previous breaking rule was global shell promotion: if the payload contained any unexpired breaking story, the highest level (`critical > breaking > watch`) was applied to the ticker or expanded panel. That made a closed panel, or a ticker currently showing a normal story, retain an orange/red state. `breaking_expires_at` eventually cleared the state, while selection, opening and closing did not change the underlying breaking data.
The current state machine keeps data, ticker visibility, panel lifecycle and selection separate:
- `getNewsBreakingState()` derives the data state from `items` and `cruise_items`, ignores expired stories, and ranks `critical > breaking > watch`. The expiry timestamp only schedules the next recomputation.
- A ticker viewport watcher runs every 120 ms and chooses the item with the greatest viewport overlap, breaking ties by distance to the viewport center. Only when that current item has a non-`none` level does the ticker switch its horizontal rules to `has-breaking-*`; a brief no-overlap gap at the duplicated-track seam retains the active state so the sweep timeline and borders do not restart.
- The panel lifecycle is `closed → opening → open → closing → closed`. Opening and closing control morphing, visibility and ticker hiding; they do not mutate the breaking data. During `opening/open`, the current card DOM stays stable while incoming payloads are deferred and applied once on close, so realtime updates cannot rebuild a hovered card.
- `selectedCruiseStoryId` is independent of breaking severity. Selection, focus, hover and close only change the selected story and reading position; close also clears the selection and filter popover.
- The expanded panel always keeps its blue glass surface. Breaking colors remain on the matching story cards, so opening the panel never turns the entire glass panel red; the ticker can still show a red signal when the currently scrolling story is breaking.
The glass-reflection effect is implemented: when the current ticker story enters a breaking state, its `::after` carries a short, feathered light cloud across the ticker glass area; the matching breaking card gets a full-card reflection sweep through its own `::after`. The panel base remains blue and the static top/bottom rules do not participate in the sweep. `prefers-reduced-motion` disables the animation and keeps a low-intensity static cue. This visual sub-state stays attached to the current breaking story and matching card, independent of whether the panel is open.
Compute cards directly reuse the shared `.hud-panel` background, border, shadow, and glass treatment. Summary, essentials, and detail areas add no separate black surface. Their headers share inset spacing with the layer panel through `.hud-panel__header--inset` / `.layer-panel-header`; primary numbers remain prominent and essential information stays visible.
### First frame and spatial card entrance
`earth-runtime-plugin.mjs` bundles local Earth modules into content-hashed entries and shared chunks for both development and production. The existing import map still pins third-party versions; relative assets and workers retain the `/earth/js/` base. `earth-preload.js` requests the selected full-size map early. `earth-texture-image.js` decodes an ImageBitmap asynchronously and transfers that same image to Three.js without resampling or changing texture color/sampling settings. Replaced and destroyed textures release their bitmaps. The full-resolution surface precedes independent concurrent data layers; data refresh retains it. Versioned boundary geometry, picking indexes and claim lines use HTTP caching and parallel requests. The live player engine is imported on demand; media and realtime connections start after the surface frame.
The root attributes `data-earth-first-frame-ms` and `data-earth-surface-ready-ms` record navigation-to-render timings. Measure cold and warm caches separately; local warm results are not a guarantee for all first-visit networks.
Positioned cards stay hidden while the shared connector draws for 420 ms, then fade in using the HUD 180 ms opacity timing. Targets remain fixed during drawing; subsequent motion does not replay entrance. Reduced-motion preferences skip this animation.
Object clicks, search focus and gesture confirmation share `PresentationController`. Cable selection updates selection/legend state without independently opening the card. InfoCard reveal callbacks are versioned so staging or hiding invalidates older callbacks. Connector completion gates card reveal; cancelled cruises no longer hide a newer selection. Object details reuse the regional-card edge candidates and collision scoring, with dimensions measured while staged. Selecting a mobile search result collapses the search drawer so the focused object, connector and compact card remain visible.
### Satellite particles and smooth rendering
Satellites use one `THREE.Points` batch with `PointsMaterial`: a 32×32 white disc texture of radius 0.42, `alphaTest=0.5`, and `sizeAttenuation=false`. The dark backdrop batch and simulated sphere lighting are removed. `satellite-display.js` calibrates against satellitemap screenshot pixels: point squares are 2.4px at zooms up to 150% and 8px at 500%, with a smoothstep transition. The disc fills 84% of the square, yielding visible cores of about 2px and 6–7px respectively. Blue/orange/green/purple/rose use the reference palette directly without the prototype's extra exposure. Point sizes retain the DPR-cap-2 adjustment without reducing globe resolution. Ordinary breathing defaults off but remains an opt-in setting. Existing hover/selected rings, halos, open predicted orbits and forward-flow animation remain intact.
Satellite point sprites use `depthTest=false`. The vertex shader tests the camera-to-satellite segment against the Earth sphere at `CONFIG.earthRadius * 0.998`, moving truly occluded points outside clip space while retaining complete visible sprites near the limb. Orbits, coverage and business coordinates retain their existing paths.
Real GP/TLE propagation stays in the Worker at 250ms intervals. `satellite-motion.js` caches sample radii and interpolates radius-preserving positions each frame, shared by points, picking, trail tips, selection rings and coverage. `satellite-trail-geometry.js` merges redundant ribbon segments only when the projected error remains below 0.2 pixels across supported zooms. All satellites and 10 history samples are retained; curved or near-camera trails keep their original segments. Only active buffer ranges are uploaded. Trail width is capped at 60% of the point size and tapers toward the tail, avoiding a thick fragmented rim at distant views. `satellite-footprint-geometry.js` retains the full 260×170 coverage grid, created once. The vertex shader evaluates the same surface projection, while local basis and extent uniforms update each frame instead of rebuilding tens of thousands of vertices. Starlink/Iridium footprints share blue `#3399ff`; sunlight changes opacity only, with real coverage models preserved.
`SATELLITE_CATEGORIES` owns real inclination categories and the blue/orange/green/purple/rose/gray palette. Selecting a satellite automatically switches to its legend. Legend categories directly control points, trails, labels and picking; hiding a selected category clears selection and coverage. Layers and settings have no separate filter entry. Explicit search selection re-enables its category. `satellite-labels.js` shows actual names and models (NORAD identifiers when unavailable) at the configured zoom threshold (**400%** by default, adjustable from **250% to 500%**). The slider below Close-up labels sets a persisted desktop/mobile count limit from 5 to 36 (default 36), immediately rescanning candidates on change. Actual visible counts still depend on available positions and occlusion. It checks occlusion and overlap in the central 65% of the view, scans candidates every 160ms and follows active labels each frame.
Settings → Display → Show FPS persists through `shared.showFps`, defaulting off. `frame-stats.js` displays actual frame-interval statistics beside the brand, updating once per second; its native tooltip contains mean interval, P95, main-thread CPU time, draw calls and an approximately 30-second rolling mean FPS (total frames / total duration, not an average of per-second rates). Slow foreground frames remain in the sample; visibility changes reset the sampling window. A single high-FPS sample is not a sustained-performance guarantee. Performance acceptance uses an actual canvas of at least 1920×1080, samples at least 30 seconds and targets a mean of 60 FPS while allowing occasional low frames. Compare distant and 400% views with matching viewport, complete data, layers and playback state, and verify category hiding, search selection and FPS persistence after reload.
The boundary loading tint, land fill, base and HD layers share the `earth-surface.js` geometry factory. Tint and fill share R+0.32, fixing pre-texture diamond rings without changing boundary/HD controls. Intersection and far-zoom depth margins are covered by `earth-surface.test.js`; see [render layer order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
Cables and landing points share HD-surface coordinates through `cable-surface.js`. `cable-batches.js` controls related halos with per-instance emphasis cleared on deselection. `interactable.js` uses `surface-marker-occlusion.js` for whole-badge visibility and picking near the limb. BGP change events retain their original icons; collectors use activity-colored broadcast badges. Sky material uses the original brightness (multiplier 1.0).
`toolbar-hover.js` owns the zoom submenu and radial toolbar as one state machine. Magnifier hover opens zoom; trigger, transparent gap bridge and vertical controls form a continuous keep-open region. Leaving first runs the submenu's 220ms exit, then checks the latest pointer location and pinned state before collapsing the parent. Remaining in the radial wake region or pinning the parent closes only zoom; reentry cancels stale timers. CSS hover/focus-within no longer owns visibility independently. Zoom clicks retain parent pinning, with keyboard and touch entry preserved.
The radial menu inherits global `--hud-scale` (1920×1080 reference, clamped to 0.7–1) instead of fixing its scale at 1. Buttons, icons, fan radii, bottom spacing and the zoom submenu scale together. Base fan coordinates are scaled in CSS transforms; hover wake bounds use the same coordinate scale to prevent hit-region drift under DPI changes. Scaling changes geometry only, preserving animation timing and pinned/hover state transitions.
### Sky color output
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
### Direct legend filtering
The current interactive object continues to determine the legend mode; enabling a layer with its existing switch also selects its legend, without adding a filter entry. Cables filter by route name, compute facilities by type, vessels by type and anchored state, and BGP by severity and collector activity. Coverage and impacted-region entries control their respective effects. The geographic legend separately controls lines, land fill and ocean fill without changing the shared land mask data. The obsolete BGP connections/hubs entry is removed.
`legend-filters.js` owns independent per-mode session filters. `legend.js` renders shared desktop/mobile toggle buttons with `aria-pressed`, preserving keyboard focus when refreshed. `interactable.js` filters draw batches, cluster candidates and picking while retaining complete source data, counts and object identities. Live vessel type changes, data reload and layer toggles respect filters. Hiding a selected item clears its card and overlays; explicit search selection reveals its category. Vessel track requests are versioned so late responses cannot resurrect a hidden track.
### Stable cards and display preferences
`callout-rest.js` retains its 450ms gate for non-urgent layout commits. Decorative floating of expanded cards stays paused; shared stage corridors and continuous following own placement. Decorative offsets never enter persistence or corridor assignment.
Display settings add `shared.satelliteLabelsEnabled` (true by default), `shared.satelliteLabelMinZoom` (2.5–5, default 4) and `shared.skyBrightness` (0–2, default 1). Desktop/mobile controls, reload and reset share runtime state. Sky brightness multiplies the sky material in linear space with its existing local sRGB output, leaving Earth exposure unchanged. Compute legend counts reflect located types. Chip and stacked SVG glyphs remain mapped to `site_type`; a GPU-only dataset does not invent other types or coordinates.
Focus state and per-type limits decide expanded cards versus capsules. Shared stage arbitration owns flank and corridor assignment for details, news and compute, with manual pinning taking priority. Hover pauses automatic focus changes, not geographic following of existing expanded cards. The older `callout-dock-motion.js` / `callout-routing.js` utilities no longer decide this presentation path.
### Cluster badges and Spiderfy
Ordinary surface markers retain GPU batches. Each aggregate uses the same projected children in `cluster-presentation.js` through collapse, expansion and the final resting state. A 26px solid numbered circle represents the group, with an amber ring for mixed categories. Membership changes only with data or filters, never through intermediate groups while zooming.
`cluster-motion.js` drives a 320ms cubic tween after visible icons are prepared and decoded, with encoded textures cached. Children interpolate from the group center to their real projections; `marker-stack.js` owns final display offsets. Animation and resting endpoints retain the same DOM nodes, with continuous reversal.
The UI's `globeZoomForDistance()` expands groups above 250%, with return hysteresis and no aggregate badges at 500%. Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, with projected children at `z-index: 1000` and unclustered locked overlays at shared `renderOrder >= 100`, so focus stays above overlapping layers; clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged. Data refresh removes retired member instances and synchronizes GPU ownership. Satellites and AIS vessels stay separate.
### Regional news cards
`news-region-cards.js` reuses `renderNewsCardContent` from `info-card.js` and the cruise adapter's news presentation mapping. Cruise and regional cards share the news template, 📰 icon and gold header. Regional metadata is reduced to source, category and time. Each card owns an independent cancellable `typewriter.js` instance, while an invisible sizing layer reserves the full summary height. Summaries are not truncated. A fixed content area with internal scrolling keeps new headlines and typing from changing placement. Automatic stowing pauses typing and rotation; expanding resumes, and the seven-second reading interval begins after typing completes.
`news-region-model.js` owns four isolated queues and remaining reading time, using `CRUISE_CONFIG.dwellMs` (seven seconds). It supports per-card pause, navigation, hidden-page resume and current-ID preservation across refreshes. Existing `getCruiseNewsItems()` data updates through `earth:news-payload-updated`, retaining source/category/content-readiness filtering. No new collection requests or global/other-region backfill are introduced.
News and compute submit scopes through `createCalloutGroups` to one `createSpatialCallouts` controller, sharing destination planning, drag locks, size changes and obstacle handling. Resize state is looked up by element identity; news layout keys use `news:<region>` to avoid compute-key collisions. Existing regional connector styling and routing remain unchanged, as do normal object-detail connectors.
The opt-in switch persists under `planet.earth.news-region-cards.v1` and lives in the existing news panel. Mobile moves the same nodes into the news drawer and removes their map connectors. Destruction releases cards, animations, observers and subscriptions. Verification covers region isolation, pause/resume, current-story retention, scope removal, simultaneous 1080p layout, mobile and returning to desktop.
The news cruise adapter completes `presentItem` only after typing finishes, so `CruiseSequencer` dwell never cuts off long text. The wait checks sequence ownership every frame and exits on manual interruption. Regional-news toggles reflect `aria-pressed` with a highlighted background and border. Compute headers retain country flags and use small representative emoji for regional groups.
### Card DPI and frame-stage boundaries
Compute, regional news and normal/cruise detail cards share `--hud-detail-card-width` and the same `info-panel.css` typography, metadata and summary rules. Compute cards, charts, icons, spacing and controls use root `--hud-scale`; manually resized inline pixel dimensions are normalized to a scale expression. SVG viewBox units scale with the chart viewport rather than receiving a second multiplier. Europe uses a small EU monogram with the title's font metrics, avoiding Windows flag-emoji fallback alignment.
The configuration menu retains its independent 340px width and original height limit rather than inheriting detail-card geometry scaling. Compute headers keep the shared HUD specification; larger primary values, charts and content spacing support the common width. Regional motion is owned by the JS tween: CSS position transitions and forced transform compositing are removed. Final left/top values align to device pixels, expanded-card decorative floating remains paused, and connectors use the same displayed position. Tight space permits peripheral overlap without automatically stowing full stage cards.
Global selections are heterogeneous: satellites are GeoJSON objects, cables are lines, and only surface interaction markers carry `userData.icon_layer_id`. `markerPresentationId` generates stack identity only for that marker contract, returning null for other selections instead of dereferencing missing fields during the frame loop.
`frame-stages.js` isolates independent update stages so a failed card, label or layer cannot skip other stages and scene rendering. It reports the stage/error, retries at one-second intervals or immediately after selection changes, and resumes normally on success. Fixing the cause and observing no runtime errors in normal scenarios remain acceptance requirements; isolation does not replace the fix.
All object-selection confirmations use `showSelectionStatus(name)` and “Selected: name”, including direct clicks, cluster children, search and motion confirmation. `confirmInteractionCandidate` and its feedback option are input-neutral. The cable layer updates selection/legend state while the interaction entry emits one notification. Obsolete lock/location/motion-confirmation message keys are removed; actual motion-mode changes and gesture guidance retain their mode label.
### Detail-card information cleanup
Normal details retain identity, business properties, timestamps and useful reliability context. Satellite cards omit the three rendering diagnostics (coverage capability, current presentation and coverage model) without changing coverage controls. News details omit feed identity, source type and fetch channel, retaining publisher, category, region, publication time and summary. Facility cards omit geocoding provenance, confidence scores, reasoning, matched names and verification timestamps while retaining location precision and verification status. Identical GPU model/platform values appear once. Internal source status is omitted; compute precision/measurement and original-source access remain.
Collector cards omit the redundant activity summary while retaining observation counts, event times and coverage. BGP event confidence remains relevant to alert reliability. Cables omit a single-coordinate row and custom points omit internal record IDs. Empty vessel enrichment no longer displays a caching message or enrichment confidence scores; identity, navigation, available profile data and source attribution remain. Desktop and mobile share field definitions. Underlying data and location-correction operations remain intact.
### Performance with multiple cards and the full satellite set
Satellite GeoJSON queries use `stream_scalars` with `SATELLITE_QUERY_BATCH_SIZE=500`, converting batches and closing the result in finally instead of retaining a complete ORM row list. Empty current records still fall back to the latest non-Unknown task. Source filtering, descending order, limit, all satellites and TLE fields retain their semantics. Final features and the cached response remain complete collections; HTTP output is not streamed and total response memory is not limited to one ORM batch.
Regional news and compute cards retain their shared dark surfaces, borders, shadows and text while avoiding per-card backdrop filtering of the changing WebGL/video scene and SVG edge distortion. `contain: layout style` limits layout propagation. Pixel-aligned positioning, card count, content, typing and float timing remain intact.
Each `CalloutConnector` SVG covers its path plus node/shadow padding rather than the entire viewport; path coordinates remain in screen space. Geographic anchors cache unchanged local coordinates, invalidate on coordinate changes, and still update world projection and front/rear visibility each frame. Obstacle tests compare rectangle coordinates directly without allocating bounds for every candidate pair. HUD obstacle measurement precedes that frame's card-style writes.
Satellite interpolation copies exact endpoint samples and retains radius-preserving interpolation between them. Trail compaction only adds an early rejection for high-altitude trails that already fail the existing error bound. Repeated identical hover/selection indices avoid full alpha-buffer rewrites. Static cable and landing-point local matrices are cached; landing-point scale changes explicitly rebuild their matrix while parent-world transforms propagate normally. No satellites, trail samples or rendering pixels are removed.
For development diagnostics, append `?profileFrames=1` to the Earth URL. `data-earth-frame-profile` reports per-stage CPU mean, maximum and count every second; profiling is disabled normally. `data-earth-gpu-renderer` identifies the GPU. Performance evidence must include actual canvas pixel dimensions, fully visible card count, satellite visibility, zoom, video state and 30-second average FPS. Pending cards or smaller viewports do not establish a 1080p full-scene result. CPU stage durations are not GPU timings.
## Detail-card images
`card-images.js` integrates optional imagery into the existing `showInfoCard` path used by selection and cruise, with one presenter implementation for desktop and mobile. It requests `/api/v1/earth/card-images/resolve` on selection using NORAD IDs or source-qualified object IDs. The server derives models, groups and categories from current catalog data. A bounded 100-identity, 30-second cache avoids repeated lookups; a same-origin storage revision event refreshes open cards after an admin save. No image work runs in the animation loop.
The shared `card-images.css` places imagery behind the rightmost 68% of the existing body. Horizontal and bottom gradients intersect using standard/WebKit mask composition. The image uses 0.85 opacity with `saturate(1.2) contrast(1.1)`; its containing layer fades to 1, without an additional dark scrim or group opacity reduction. Text, headers, DPI sizing, scrolling, dragging and connectors retain their own layout. Decode precedes the fade; reduced motion skips animation. Missing/broken images produce a data-only card. Selection tokens guard both resolution and decoding, so late results cannot replace newer selections or reopen a closed card. The header source link exposes attribution and license information.
`card_image_configuration.py` stores assets and assignments in a separate system setting, using revision checks, row locks and audit transactions. Collected records are never edited. Resolution prefers object, model, group and category in that order; explicit per-object suppression stops fallback. Disabled assets cannot resolve; permission metadata remains truthful and separate from the enabled flag. Schemas reject duplicate targets and dangling references. Built-in assets come from the same manifest copied into the backend container, without seeding the database. Known built-in IDs preserve edits/removals while merging newly shipped entries; previous_defaults replaces old bundled URLs while preserving fields changed from the prior default and all custom image URLs.
Pillow validates still JPEG/PNG/WebP uploads and generates 640/1280-bound WebP derivatives, preserving original files. Content-addressed files are written atomically and served through a read-only public endpoint; management, upload, catalog search and preview require administrators. See the operations runbook for storage/backup. The Frontier photo is facility-specific; Starlink remains a series illustration. Qianfan series imagery is enabled as requested, while its permission metadata remains unverified.
Category `*` covers satellites, compute facilities, cables, landing stations, vessels, BGP incidents, BGP collectors and custom points, after specific category rules. Stable keys are MMSI, incident_key, collector_id and the custom-point ID. Missing identities in old snapshots allow only a type illustration. Rebind a collected object if its upstream source_id changes.
### Five compute facility types and default imagery
Supported types are `supercomputer`, `gpu_cluster`, `data_center`, `cloud_center` and `edge_node`. `compute-site-types.js` centralizes frontend titles, labels, counts and fallback; backend `compute_types.py` shares classification between GeoJSON and image lookup. An explicit supported metadata.site_type wins, followed by data_type and known source defaults. Unrecognized facilities use a generic data-center presentation instead of becoming GPU clusters. No facilities are fabricated or existing records reclassified by name.
Each type supports legend filtering, marker icons, desktop/mobile details and image identity. New infrastructure cards use identity, operator, capacity, power and location fields without GPU counts or TOP500 ranks. The map accepts known sources or records declaring a supported compute type. Regional facilities retain snapshot/country constraints; infrastructure records do not require a GPU measurement contract and do not imply a global compute share. Built-in collectors remain TOP500 and Epoch AI; adding types does not create records.
Seven user-supplied images provide defaults for the five facility types, BGP collectors and BGP events. All five categories are available in image management before data exists. Exact assignments retain priority. Manifest previous_defaults / previous_rules migrate old bundled sources and unchanged collector assignments while preserving custom crop, enabled state and specific bindings.
### Frame-work reductions with unchanged presentation
Cluster presentation reuses each parent's world matrix and local camera position within one update, refreshing on every frame. Fully collapsed members without child DOM still participate in hidden-state reconciliation, but skip invisible endpoint projections. Unchanged count and ARIA text no longer cause repeated DOM writes.
Static point batches refresh on data, selection, filtering, distance scaling or explicit invalidation; selected overlays continue pulsing every frame. Parsed colors cache immutable strings/numbers only, never mutable caller-owned Color objects. Satellite dot/halo overlays update only the required ancestor transform rather than traversing all Earth descendants twice.
Trail sampling combines the radius bound and initial error scan, preserving the existing threshold and subdivision indices. No satellites, trajectory samples, card content or resolution are removed. Deterministic comparison covered 10,000 histories with identical indices. Local Bun microbenchmarks (not whole-page FPS): 480 cluster members took about 1.19→0.90 ms collapsed and 4.47→2.86 ms expanded; scanning 16,621 trails took about 11.50→8.94 ms. End-to-end FPS requires a separate 30-second foreground comparison at identical resolution and scene/card/video settings.
Regional news collapse buttons explicitly invoke the shared HUD controller. Their bodies use hud-panel__body--collapsible and aria-expanded follows the actual state. Only user clicks collapse cards; layout and performance code never hide their contents automatically.
Floating object details and regional news share `--hud-detail-card-max-height`: 390px times HUD scale, capped at viewport height minus 24px. A min-height:0 flex chain keeps the header fixed and scrolls only the image-backed body. Short cards keep their natural height. BGP incidents prioritize type/severity/status/impact/time/summary; collectors prioritize identity/status/recent activity; facilities prioritize name/capacity/hardware, with location verification and provenance later. Desktop and mobile field order share CARD_CONFIG.
### Detail identity and connector consistency
BGP collector indexes use collector names. Shared marker indexes omit empty IDs and refuse cross-refresh replacement for ambiguous duplicates; a clicked marker still in the current layer retains its own reference. Stack presentation shares icon_item_id with the index. BGP incidents prefer incident_key, while raw anomalies use source plus ID to prevent numeric collisions across feeds.
Satellite detail connectors identify the selected NORAD and relocate its index when the catalog array changes. Selection highlights rebind to that NORAD after refresh; a removed target clears selection. Uninitialized zero positions are not valid anchors. Cable anchors remain confined to the selected cable geometry, landing points use the clicked marker, and regional news/compute cards retain their regional anchors.
When compute image configuration has no model/group rules, resolution uses a bounded fast path: the stable source prefix supplies the supercomputer/GPU category, then object and category rules are checked without scanning the collected catalog. Adding model or group rules automatically uses the full catalog resolver. Frontend identities, collected data and other object types are unchanged.

View File

@@ -1,84 +1,35 @@
# Intelligent Planet Interactable Clustering
# Earth interactable clustering
Earth interactable icons are managed by `createInteractableLayer`. Rendering still uses Three.js `Points`, and picking, hover, locked state, tooltips, and cruise focus still depend on marker `userData`; the clustering strategy only decides which markers are represented by a cluster dot.
`interactable.js` owns fixed geographic membership, `cluster-motion.js` owns timed tweening, and `cluster-presentation.js` owns circular badges, projected children, picking and Spiderfy. Independent markers remain GPU Points. Each clustered child uses the same DOM icon throughout its animation and at the final position.
## Strategies
## Membership and zoom
`cluster.strategy` supports three modes:
Spherical buckets use real `icon_base_position` anchors. Defaults are distance 15 in Earth units, at least two members and at most 14 per group. Data, filter and visibility changes invalidate membership; rotation and zoom do not. Badges stay at the spherical centroid without screen avoidance.
- `stable-spherical`: stable spherical clustering. It clusters by local 3D positions on the globe and caches topology by zoom band. Rotation and small zoom changes within the same band only update projection and size. Use it for semi-static layers such as BGP, compute centers, and Earth interactables.
- `dynamic-screen`: dynamic screen-space clustering. This keeps the previous projection-based behavior and evaluates visible relationships per frame. Use it for high-frequency realtime layers such as vessels, or as a fallback.
- `none`: no clustering. Every marker is shown independently. Use it for low-count layers or precision-first views.
Use `cluster: { enabled: true }` or `cluster: false`. Options are `distance`, `minCount`, `maxMarkersPerDot`, `disableAboveZoom` (2.5) and `expansionHysteresis` (0.08). Zoom uses the UI's `globeZoomForDistance()` mapping: expand above 250%, collapse below 230%, and show no aggregate badges at 500%. Satellites and AIS vessels remain separate.
`cluster: false` is equivalent to `strategy: "none"`. If a layer enables clustering without declaring a strategy, it keeps the compatible `dynamic-screen` behavior.
## Motion and Spiderfy
## Stable Spherical
Both directions use a 320ms easeInOutCubic tween, complete after wheel input stops, and reverse continuously. Children interpolate directly from the centroid to their real local positions and retain the same DOM nodes at the endpoint. Because this chord can temporarily enter the sphere, occlusion uses a radial surface anchor while projection retains the interpolated position; intermediate frames remain visible.
`stable-spherical` moves cluster identity from screen distance to globe distance:
Only co-located or extremely close geographic anchors (about 0.00015 degrees) share a compact cross-layer stack. Screen overlap and coarse city buckets never merge different locations. Each icon retains its own real projection as its baseline, with the entire stack bounded to 24px per axis and small groups exposing approximately 8–12px edges. Stable IDs define resting order; selecting a rear item brings it forward, and clearing focus restores its slot. Ordinary expansion ends exactly at real projected positions; coordinates and marker positions remain unchanged.
- Each marker uses `icon_base_position` as its geographic anchor.
- By default, zoom levels above `2.5` force clustering off and show every marker as its original icon.
- The current zoom maps to a discrete band; rotation and small zoom changes inside the same band do not recompute topology.
- Clusters are recomputed only when the band, data revision, visibility, or filter state changes.
- A cluster centroid is computed from member 3D positions and normalized back to the globe shell, so the cluster dot stays rigidly aligned to its geographic center.
- Cluster dots do not participate in 2D avoidance, so screen-space repulsion cannot push them away from their real geographic projection.
- Band changes use a small hysteresis margin so zooming at a boundary does not repeatedly bounce between two bands.
- Newly created marker and cluster dots run a short scale + opacity ease. This is only a rendering transition; it does not change marker coordinates, picking objects, or locked state.
## Interaction and BGP
The stable strategy uses spherical bucket/hash neighbor lookup and must not use an all-pairs loop. Most frames only pay projection and material-size cost; topology cost is paid only on band or data changes.
Clicking a badge expands it and pauses rotation. Legs have 14px transparent hit areas; hover highlights the matching leg and child, and either click selects the same business object. The canvas container accepts wheel input over the canvas and projected markers while excluding HUD scrolling.
## Configuration
Projected textures bake the original category tint. BGP events retain severity colors. Badges containing BGP and expanded BGP event children use an expanding/fading halo based on `BGP_CONFIG.ring`; replaced GPU effects are hidden to avoid duplicate rings at old anchors. Reduced-motion preference disables the pulse.
```js
const computeCenterIconLayer = createInteractableLayer({
id: "computeCenters",
// ...
avoidance: SURFACE_AVOIDANCE_PROFILES.city,
cluster: {
strategy: "stable-spherical",
minCount: 2,
maxMarkersPerDot: 14,
transitionMs: 220,
bandHysteresis: 0.08,
disableAboveZoom: 2.5,
bands: [
{ key: "far", maxZoom: 1.7, distance: 15 },
{ key: "mid", maxZoom: 2.6, distance: 8 },
{ key: "near", maxZoom: 3.5, distance: 4 },
{ key: "detail", maxZoom: Infinity, distance: 0 },
],
},
});
```
## Removed paths
Realtime layers can stay dynamic:
The low-zoom compact-dot desktop/mobile controls, persistence field, textures, rendering/picking branches and translations are removed. Dynamic screen clustering, discrete zoom bands, GPU aggregate dots and the old creation-time fade/scale transition are also removed. There is no legacy strategy fallback.
```js
const vesselIconLayer = createInteractableLayer({
id: "vessels",
// ...
cluster: {
strategy: "dynamic-screen",
enabled: true,
maxMarkersPerDot: 10,
},
});
```
## Regression coverage
## Tuning
Tests cover 320ms completion, reversal continuity, visible intermediate frames, retained endpoint nodes, 500% expansion, immutable source coordinates, filtering cleanup, wheel target routing, linked leg hover/click and projected BGP category colors: `cluster-motion.test.js`, `cluster-presentation.test.js`, `globe-navigation.test.js`, `marker-badges.test.js`.
- `distance` is the 3D globe-distance threshold and uses the same unit as `CONFIG.earthRadius`. Larger values cluster more aggressively.
- The farthest band usually uses a larger `distance`; the nearest detail band usually uses `0` to split clusters into original icons.
- `bandHysteresis` controls band-boundary stickiness. Too little can flicker near thresholds; too much can make band changes feel late.
- `transitionMs` controls cluster split/merge easing. Keep it around 160-260ms; longer durations can make realtime layers feel sluggish.
- `disableAboveZoom` controls the precision-view threshold. It defaults to `2.5`; above that zoom, no cluster dots are generated. Set it to `false` to disable the hard threshold.
- High-frequency realtime layers should prefer `dynamic-screen` so frequent data changes do not trigger stable topology recomputation.
- If a layer behaves poorly, switch it back to `dynamic-screen` or use `cluster: false`.
## Refresh and first-frame preparation
## Acceptance Checks
A reused cluster ID does not imply reused Object3D instances: stale children are reconciled against current instances. Ownership changes immediately rebuild the GPU batch, and topology invalidation clears its old signature.
- Rotating the globe inside one zoom band should not cause clusters to flicker or regroup.
- Cluster dots should stay aligned with the globe-surface centroid and should not be pushed by avoidance.
- Zooming into the detail band should restore the layer's original icon textures and click behavior.
- Zoom levels above 250% should not show cluster dots.
- Realtime layers such as vessels should reflect incoming updates immediately.
Expansion prepares visible icons and waits for image decoding before starting its 320ms clock. Encoded URLs are reused per texture, and invisible rear members are not instantiated. Regression tests cover repeated refresh/zoom, preparation slower than the tween duration, selection/restoration and connector anchors.

View File

@@ -1,5 +1,17 @@
# Earth Layer Style Property Index
`EARTH_MATERIAL_CONFIG.surface.night` centralizes night texture contrast (detailGamma=0.55) and separate land/ocean tint and ambient values. It only contributes to the night branch, keeps day exposure unchanged, and reuses the geographic mask without another texture or draw call.
## Modern object presentation
- Surface: `EARTH_MATERIAL_CONFIG.surface` owns exposure=0.9, landSaturation=0.9, oceanAbsorption=0.36, oceanDeepColor=[0.004,0.012,0.032], night retention and Fresnel parameters. `atmosphere` owns the single shell's radius, color and strength. Base and terrain specular highlights are disabled.
- Cables blend a cool normal tint with their category color, consistently with the legend. Selected pulses follow cumulative route length with 6 repetitions and speed 4.5 in the existing batch draw. Drawing no longer writes depth; original Line pick objects, cable IDs and state logic remain intact.
- Landing points use the 128×128 monochrome `marker-landing-point.svg` micro-core instead of a glossy procedural ball; Sprite picking and billboard size logic remain intact.
- Compute markers use canonical SVG inner glyphs over shared gradient plates, pale rims, top highlights and shadows. GPU/AI uses purple stacked layers, supercomputers purple chips, and infrastructure teal server racks. Estimated badges and picking remain intact.
- Satellites share vivid real-inclination category colors with the legend and predicted orbits. Normal markers are prototype hard discs without outlines or sphere shading; breathing defaults off. Coverage stays blue and sunlight only affects opacity.
This document records the material, color, opacity, line width, radius offset, and `renderOrder` style properties of all Earth frontend layers. For layer ordering relationships, see [Earth Render Layer Order](/home/ray/dev/linkong/planet/docs/technical/en/earth-render-layer-order.md).
## Naming Conventions
@@ -20,15 +32,11 @@ This document records the material, color, opacity, line width, radius offset, a
| Earth base radius | `CONFIG.earthRadius` | `100` | `earth.js:createEarth()` |
| Earth base color | `EARTH_MATERIAL_CONFIG.color` | `0x010609` | `MeshPhongMaterial.color` |
| Earth base emissive | `EARTH_MATERIAL_CONFIG.emissive` | `0x010609` | `MeshPhongMaterial.emissive` |
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x1a2d45` | `MeshPhongMaterial.specular` |
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `12` | `MeshPhongMaterial.shininess` |
| Earth base specular | `EARTH_MATERIAL_CONFIG.specular` | `0x000000` | `MeshPhongMaterial.specular` |
| Earth base shininess | `EARTH_MATERIAL_CONFIG.shininess` | `0` | `MeshPhongMaterial.shininess` |
| Earth base opacity | `EARTH_MATERIAL_CONFIG.opacity` | `1` | `MeshPhongMaterial.opacity` |
| HD texture radius offset | `EARTH_MATERIAL_CONFIG.textureOverlayAltitudeOffset` | `0.48` | Standalone HD texture sphere radius; must keep enough depth separation from the land/ocean base and Earth base sphere to avoid far-zoom z-fighting |
| HD texture opacity | `EARTH_MATERIAL_CONFIG.textureOverlayOpacity` | `0.88` | HD texture `MeshPhongMaterial.opacity` |
| HD texture renderOrder | `EARTH_MATERIAL_CONFIG.textureOverlayRenderOrder` | `0.96` | `_earthTextureOverlay.renderOrder` |
| HD texture specular | `EARTH_MATERIAL_CONFIG.textureOverlaySpecular` | `0x05080d` | Reduces specular highlight in direct-light areas to avoid blown-out texture |
| HD texture shininess | `EARTH_MATERIAL_CONFIG.textureOverlayShininess` | `4` | Reduces specular concentration |
| HD texture color multiplier | inline | `0xffffff` | `_earthTextureOverlayMaterial.color` |
## Earth Occluder and Day/Night
@@ -36,7 +44,7 @@ This document records the material, color, opacity, line width, radius offset, a
| --- | --- | --- | --- |
| Occluder radius factor | `EARTH_MATERIAL_CONFIG.occluderRadiusFactor` | `0.999` | Depth occluder sphere radius |
| Occluder segments | `EARTH_MATERIAL_CONFIG.occluderSegments` | `48` | Occluder geometry segments |
| Occluder renderOrder | inline | `-1` | `occluder.renderOrder` |
| Occluder renderOrder | inline | `0.5` | `occluder.renderOrder` |
| Day/night sun direction | `EARTH_MATERIAL_CONFIG.dayNight.sunDirection` | `{ x: 1, y: 0.2, z: 0.4 }` | Custom day/night shader |
| Night-side minimum brightness | `EARTH_MATERIAL_CONFIG.dayNight.nightFloor` | `0.24` | Shader uniform |
| Day-side boost | `EARTH_MATERIAL_CONFIG.dayNight.dayBoost` | `1.12` | Shader uniform |
@@ -50,27 +58,12 @@ This document records the material, color, opacity, line width, radius offset, a
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Inner atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosInnerRadiusFactor` | `1.01` | `atmosInnerGeo` |
| Inner atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosInnerSegments` | `64` | `atmosInnerGeo` |
| Inner atmosphere color | `EARTH_MATERIAL_CONFIG.atmosInnerColor` | `[0.25, 0.62, 1.0]` | Shader RGB |
| Inner atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosInnerRimPower` | `3.2` | Shader rim falloff |
| Inner atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosInnerIntensity` | `0.18` | Shader alpha multiplier |
| Outer atmosphere radius factor | `EARTH_MATERIAL_CONFIG.atmosOuterRadiusFactor` | `1.016` | `atmosOuterGeo` |
| Outer atmosphere segments | `EARTH_MATERIAL_CONFIG.atmosOuterSegments` | `48` | `atmosOuterGeo` |
| Outer atmosphere color | `EARTH_MATERIAL_CONFIG.atmosOuterColor` | `[0.18, 0.45, 0.9]` | Shader RGB |
| Outer atmosphere rim power | `EARTH_MATERIAL_CONFIG.atmosOuterRimPower` | `5.0` | Shader rim falloff |
| Outer atmosphere intensity | `EARTH_MATERIAL_CONFIG.atmosOuterIntensity` | `0.02` | Shader alpha multiplier |
| Atmosphere blending | inline | `THREE.AdditiveBlending` | `ShaderMaterial.blending` |
| Atmosphere renderOrder | inline | `1` | `atmosInner/Outer.renderOrder` |
| No-HD-texture rim glow color | `EARTH_MATERIAL_CONFIG.rimGlowColor` | `[0.35, 0.65, 1.0]` | Fresnel shell RGB when HD texture is hidden or unavailable |
| No-HD-texture rim glow power | `EARTH_MATERIAL_CONFIG.rimGlowPower` | `3.8` | Shader rim falloff; higher = narrower edge |
| No-HD-texture rim glow intensity | `EARTH_MATERIAL_CONFIG.rimGlowIntensity` | `0.28` | Shader alpha multiplier |
| No-HD-texture rim glow segments | `EARTH_MATERIAL_CONFIG.rimGlowSegments` | `64` | `earth-rim-glow` geometry segments |
| Cloud layer radius offset | `CLOUD_LAYER_CONFIG.radiusOffset` | `3` | Cloud sphere radius |
| Cloud layer segments | `CLOUD_LAYER_CONFIG.widthSegments / heightSegments` | `64 / 64` | Cloud sphere geometry |
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshPhongMaterial.opacity` |
| Cloud layer opacity | `CLOUD_LAYER_CONFIG.opacity` | `0.15` | `MeshBasicMaterial.opacity` |
| Cloud texture | `CLOUD_LAYER_CONFIG.textureUrl` | `"./assets/earth_clouds_1024.png"` | Cloud texture map |
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshPhongMaterial.blending` |
| Cloud blending | inline | `THREE.AdditiveBlending` | `MeshBasicMaterial.blending` |
## Land/Ocean Base and Country Borders
@@ -94,7 +87,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Land/ocean base renderOrder | `COUNTRY_BOUNDARY_CONFIG.landRenderOrder` | `0.86` | `country-land-ocean.renderOrder` |
| Land/ocean mask size | `landMaskWidth / landMaskHeight` | `2048 / 1024` | Canvas / DataTexture size |
| Country tint color | `COUNTRY_BOUNDARY_CONFIG.tintColor` | `0x0b1830` | Tint when HD texture is off |
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.04` | `country-tint` radius |
| Country tint radius offset | `COUNTRY_BOUNDARY_CONFIG.tintAltitudeOffset` | `0.32` | `country-tint` radius |
| Country tint renderOrder | `COUNTRY_BOUNDARY_CONFIG.tintRenderOrder` | `0.2` | `country-tint.renderOrder` |
| Border line color | `COUNTRY_BOUNDARY_CONFIG.lineColor` | `0x7fc7ff` | Normal border line |
| Border line opacity | `COUNTRY_BOUNDARY_CONFIG.lineOpacity` | `0.58` | Normal border line opacity |
@@ -123,8 +116,8 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Terrain opacity | `TERRAIN_CONFIG.opacity` | `0.68` | `MeshPhongMaterial.opacity` |
| Terrain color | `TERRAIN_CONFIG.color` | `0x8aa884` | `MeshPhongMaterial.color` |
| Terrain emissive | `TERRAIN_CONFIG.emissive` | `0x030704` | Reduces self-emission to preserve terrain shading |
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x344438` | Gives terrain local sheen without boosting HD texture brightness |
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `16` | Tightens terrain highlight |
| Terrain specular | `TERRAIN_CONFIG.specular` | `0x000000` | Matte terrain; no white specular lobe |
| Terrain shininess | `TERRAIN_CONFIG.shininess` | `0` | Specular disabled |
| Terrain renderOrder | inline | `1.2` | `terrain.renderOrder` |
| Terrain polygonOffset | inline | `factor -1`, `units -1` | Reduces z-fighting near sphere surface |
@@ -148,11 +141,11 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Default cable color | `CABLE_COLORS.default` | `0xffff44` | Used when no data color available |
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.2` | Cable line radius |
| Cable radius offset | `CABLE_CONFIG.line.altitudeOffset` | `0.48` | Cable line radius |
| Cable line width | `CABLE_CONFIG.line.lineWidth` | `1` | `LineBasicMaterial.linewidth` |
| Cable opacity | `CABLE_CONFIG.line.opacity` | `1.0` | Cable line opacity |
| Cable renderOrder | `CABLE_CONFIG.line.renderOrder` | `1` | Cable line level |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.2` | Same surface height as cable lines, avoiding a floating marker |
| Landing point radius offset | `CABLE_CONFIG.landingPoint.altitudeOffset` | `0.48` | Same surface height as cable lines, avoiding a floating marker |
| Landing point sprite height | local `LANDING_POINT_SPRITE_HEIGHT` | `3` | `THREE.Sprite` base height |
| Landing point reference FOV | local `LANDING_POINT_SIZE_REFERENCE_FOV` | `75` | Matches the current Earth camera FOV |
| Landing point scale minimum | local `LANDING_POINT_SIZE_SCALE_MIN` | `0.16` | Minimum multiplier after roughly 200% zoom, limiting high-zoom screen footprint; `3 * 0.16 = 0.48` |
@@ -162,7 +155,7 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Landing point opacity | `CABLE_CONFIG.landingPoint.opacity` | `1.0` | `SpriteMaterial.opacity` |
| Landing point renderOrder | `CABLE_CONFIG.landingPoint.renderOrder` | `1` | Same level as cable lines; `depthTest: false` keeps the ball whole, while camera-to-center globe occlusion hides back-side points |
| Landing point dim brightness | `landingPointVisual.dimBrightness` | `0.62` | Dim state color multiplier |
| Related landing point opacity | `landingPointVisual.related.opacityBase / opacityPulse` | `0.8 / 0.2` | Highlight pulse |
| Related landing halo extent | `landingPointVisual.related.haloScale / haloFalloff` | `1.65 / 8` | Expands only the halo canvas; core size and color stay fixed |
| Dimmed landing point color | `landingPointVisual.dimmed.colorRGB` | `{ r: 180, g: 116, b: 28 }` | Dim state color; avoids dark base showing through as a dark hole |
| Dimmed landing point opacity | `landingPointVisual.dimmed.opacity` | `0.78` | Dim state opacity; no longer uses low alpha blending with dark base |
@@ -174,14 +167,11 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
| Satellite altitude compression scale | `SATELLITE_CONFIG.altitudeCompressionKm` | `1200` | TLE/SGP4 altitude compression; larger values soften high-orbit separation |
| Satellite real-altitude cap | `SATELLITE_CONFIG.maxDisplayAltitudeKm` | `40000` | Altitude clamp for the compressed display mapping, covering GEO-range altitude |
| Satellite real-altitude display offset range | `minRealAltitudeOffset / maxRealAltitudeOffset` | `4 / 25` | Compressed offset added to `CONFIG.earthRadius`; `25` is about one quarter of the current globe radius `100`, keeping high-orbit separation visible without pushing GEO/MEO trails too far away |
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.8` | Point shader size |
| Satellite backdrop dot scale | `SATELLITE_CONFIG.dotBackdropScale` | `1.28` | Backdrop dot size |
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.42 / 1.0` | Breathing animation |
| Satellite dot breathing speed | `SATELLITE_CONFIG.dotBreathingSpeed` | `0.12` | Dot opacity animation |
| Satellite backdrop renderOrder | inline | `5` | `satelliteBackdropPoints.renderOrder` |
| Satellite dot base pixel size | `SATELLITE_CONFIG.dotBaseSize` | `2.2` | Legacy interaction hit-radius basis; not visual size |
| Satellite dot opacity range | `dotOpacityMin / dotOpacityMax` | `0.72 / 1.0` | Selected-point breathing; ordinary breathing defaults off |
| Satellite dot renderOrder | inline | `6` | `satellitePoints.renderOrder` |
| Satellite trail length | `SATELLITE_CONFIG.trailLength` | `10` | Trail buffer |
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` | Ribbon shader uniform |
| Satellite trail line width | `SATELLITE_CONFIG.trailLineWidth` | `3` (cap) | Also capped at 60% of point size with tapered short tails |
| Selected ring size | `SATELLITE_CONFIG.ringSize` | `0.07` | Hover / locked ring sprite |
| Satellite overlay renderOrder | `SATELLITE_CONFIG.overlayRenderOrder` | `12` | Locked ring / halo / orbit |
| Footprint renderOrder | local `GROUND_FOOTPRINT_RENDER_ORDER` | `3` | Starlink footprint fill and Iridium coverage ring; must stay above land / texture / terrain surface layers |
@@ -216,8 +206,8 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
| Hover scale | `COMPUTE_CENTER_CONFIG.marker.hoverScale` | `1.16` | Hover overlay size multiplier |
| Locked scale | `COMPUTE_CENTER_CONFIG.marker.lockedScale` | `1.22` | Locked overlay size multiplier, with pulse |
| Dimmed scale / opacity | `dimmedScale / dimmedOpacity` | `0.82 / 0.34` | Dim state |
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#38bdf8"` | Marker texture |
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#2dd4bf"` | Marker texture |
| Supercomputer color | `COMPUTE_CENTER_CONFIG.colors.supercomputer` | `"#a855f7"` | Marker texture |
| GPU cluster color | `COMPUTE_CENTER_CONFIG.colors.gpu_cluster` | `"#a855f7"` | Marker texture |
| Linked color | `COMPUTE_CENTER_CONFIG.colors.linked` | `"#f8fafc"` | Linked state |
| Compute center renderOrder | local `COMPUTE_CENTER_RENDER_ORDER` | `4.5` | Surface facility below satellites |
@@ -259,7 +249,54 @@ AIS vessel markers use batched `THREE.Points`, not one `THREE.Sprite` per vessel
| Moon halo scale | `CELESTIAL_CONFIG.moonHaloScale` | `62` | Moon halo |
| Sun light color / intensity | `sunLightColor / sunLightIntensity` | `0xfff4df / 1.02` | Scene light |
| Back light color / intensity | `backLightColor / backLightIntensity` | `0x2b4c78 / 0.3` | Scene light |
| Star count | `STARFIELD_CONFIG.count` | `8000` | `createStars()` |
| Star radius range | `minRadius + radiusJitter` | `800 + 200` | Random distribution |
| Star color | `STARFIELD_CONFIG.color` | `0xffffff` | `PointsMaterial.color` |
| Star size | `STARFIELD_CONFIG.size` | `0.5` | `PointsMaterial.size` |
## Badge taxonomy and palette
| Type | Primary color | Inner glyph |
| --- | --- | --- |
| Datacenter/infrastructure | Teal `#14b8a6` → `#0d9488`, rim `#5eead4` | Server rack |
| GPU/AI | Purple `#a855f7` → `#7e22ce`, rim `#d8b4fe` | Stacked compute layers |
| Supercomputer | Same purple family | Chip |
| Cable landing point | Amber `#f59e0b`, pale-gold `#fef3c7` rim | Glowing circular core |
| BGP collector | Blue and existing activity heat colors | Rounded badge containing the broadcast-tower glyph |
| BGP event | Low/medium/high/critical: blue/amber/orange/red | Irregular flat event symbols, no badge plate |
| Vessel | Cargo blue, tanker orange, passenger teal, fishing amber, military indigo, other silver | Heading arrow; anchored state retained |
| Custom marker | Place green, note yellow, alert red, default blue | Pin, note sheet, warning triangle |
BGP change events retain irregular flat symbols as the exception; collectors use rounded badges; other finite surface objects share cached badge textures without a new DOM node or draw object per marker. Vessels retain heading buckets and vertex colors rather than multiplying buckets by vessel type. Satellites use distinct category colors and hard discs without an outline; orbit and footprint calculations stay intact. The low-zoom dot control, persisted field and rendering/picking branches are removed.
### Stable satellite palette
Category colors follow [satellitemap.space](https://satellitemap.space/) while retaining this project's real inclination classes: 0–30° blue `#3399ff`, 30–60° orange `#ff801a`, 60–90° green `#00e680`, 90–120° purple `#cc33ff`, 120–180° rose `#ff3366`, unknown gray `#94a3b8`. A satellite keeps its hue across day/night; predicted orbits no longer use an unrelated yellow-purple gradient.
All coverage uses `#3399ff`, opacity 0.34 and day-side opacity multiplier 1.65. Normal points have no dark outline; one `PointsMaterial` batch uses alphaTest=0.5. `satellitePixelSize()` smoothly varies visual size from 2.4 to 8.0.
### Selected orbit and panel outlines
Prediction covers the full period without truncating or fading the endpoints and without an appended closing vertex. Highlights travel forward along cumulative arc length in the same Line draw. TV and shared HUD shells use square corners, including the player clip, without changing panel size or dragging. Temporary frame-statistics instrumentation and its QA-only DOM attributes are removed from release code.
### Landing points, limb badges and subtle sky brightness
Cables and landing points share R+0.48 and renderOrder=1. `cable-surface.js` generates both on the actual surface triangles, removing ideal-sphere versus surface endpoint parallax. Normal points are yellow discs with a pale ring and no persistent halo. Only related points get a subtle pulsing halo in the existing batch shader; unrelated points dim, and deselection restores the normal yellow state.
Rounded badges use anchor occlusion through `surface-marker-occlusion.js`: the GPU keeps a visible anchor's whole badge and hides it when Earth occludes the anchor. Screen-space picking shares the same check. Only badge layers replace per-pixel depth clipping; BGP change events keep their flat symbols and existing depth behavior. Collectors retain activity colors inside rounded plates. Normal, hover, locked and cluster badges share the path.
Only the sky texture material receives `CELESTIAL_CONFIG.skyBrightness = 1.0`; Earth exposure, celestial lighting and business-layer brightness stay unchanged.
Vessels are independent moving objects: their pre-badge heading arrows, anchored dots and hover/selected glow are restored. They do not use the fixed-facility rounded plates or whole-badge occlusion path.
Normal and related landing points retain the same orange core and yellow rim. Related state adds a wider soft pulsing halo without changing core color, opacity or size. Deselecting clears the halo; unrelated points retain the dim state.
### Sky color output
The star map is decoded as sRGB. `celestial.js` uses shared `preserveTextureOutput()` to encode only the sky material back to display sRGB. The Three.js r128 renderer retains its existing global linear output so Earth, satellites and other layers remain unchanged. `skyBrightness=1.0` retains source brightness; missing output encoding displays decoded dark values directly and cannot be corrected by raising the multiplier alone. Validate with the same map, camera and orientation, comparing rendered pixels and the 1.0/1.5 response instead of checking asset loading only.
Satellite hover rings use the hovered satellite inclination category color, matching the point and legend. Changing the hovered object resolves its color again; position updates preserve it.
All map cards use the root HUD DPI scale. Regional and cruise news share header icon, colors, typography and padding; regional cards simplify metadata and add independent rotation controls. Expanded cards retain full content while other regions use capsules that temporarily open with the same card styling on hover. Rear capsules dim and shrink without removing business objects. Thin/bold lines share `CalloutConnector`; bold styling widens the white line, adds dark outlines and enlarges endpoints. Compute metrics, trend charts and title icons follow HUD scaling. Europe uses a compact EU monogram; other region emoji follow the same scale. Connectors choose the entrance edge facing the source and balance their two segments, joining directly when nearly aligned. Hover-expanded capsules retain full dimensions, body and actions so compact-state rules cannot clip their contents.
Compute markers use five canonical SVGs: compute-supercomputer.svg, compute-gpu-cluster.svg, compute-hdd-network.svg, compute-cloud.svg and compute-edge.svg. They retain the compute layer purple badge and existing selection behavior.

View File

@@ -84,6 +84,20 @@ Importance levels are fixed: `low` 0–34, `medium` 35–59, `high` 60–79, and
## Configuration and Cache
`GET /api/v1/news/earth-feed` reads saved news. The scheduler's
`earth_news_rss_refresh` job supplements RSS on startup and every 10 minutes,
checking freshness and feed coverage by region. Redis locks coordinate refreshes
across processes. Failed feeds use the source's configured cooldown while other
sources continue. Health timestamp updates alone do not broadcast content changes.
`POST /api/v1/news/refresh` accepts a region or view coordinates and returns
HTTP 202 with a `refresh_id` shared by active manual requests for that region.
`GET /api/v1/news/refresh/{refresh_id}` reports pending, running, succeeded, or
failed state. On success, the frontend reloads news and updates the panel before
showing success. Existing content stays readable during the wait. Failure
re-enables the button for retry. Saving or resetting source configuration also
schedules an immediate background check.
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
## Manual News Content
@@ -160,13 +174,13 @@ The Web Earth category chips only store the current browser preference; changing
When `sources` is omitted, the service layer prefers stories that already have displayable title and summary content for the requested `locale`, supplements candidates from enabled sources, and rotates sources so one source's newest pending items cannot occupy all 12 default slots. An explicit `sources` filter remains precise and does not supplement other sources. The database query layer owns region, category, source, and ordering constraints only; it does not own locale presentation policy.
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not prove that items are stored or visible under the current region/category filters. Saving or resetting sources increments the configuration version, clears cache and immediately schedules background refreshes for all regions. Background work supplements feeds without recent stored items; `earth-feed` requests remain read-only.
## Regional Balancing and Async Enrichment
`earth_news_items` is the current-state table for EarthFeed. When no explicit `sources` filter is present, the endpoint sorts by Breaking state, region, and publish time, then applies a regional round-robin so Asia Pacific or any other high-volume source cannot fill the entire global view and cruise queue. The default region order is Americas, Europe, Middle East / Africa, Asia Pacific, then Global; unknown regions participate after the known regions. Regional views still follow the active-region-plus-global rule and do not mix unrelated regions into the regional panel.
Both `items` and `cruise_items` are enqueued for target-location and localization enrichment, but the frontend must not wait for AI before rendering. If the requested display locale is not ready yet, Web Earth falls back to the original `title / summary` so a card does not show a "translation pending" placeholder when readable source content already exists. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
RSS refresh and content-write paths enqueue location and localization enrichment; `earth-feed` reads no longer create those tasks. The frontend renders existing content without waiting for AI, subject to the active locale’s content-availability filter. English mode does not fall back to Chinese source text. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
Target-location enrichment uses two Redis Streams queues:
@@ -236,7 +250,11 @@ flowchart LR
ManualAPI --> Store
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
NewsAPI --> Resolver["Source Resolver"]
NewsAPI --> Config
Earth --> RefreshAPI["POST /api/v1/news/refresh"]
RefreshAPI --> Scheduler["RSS refresh scheduler"]
ConfigAPI --> Scheduler
Scheduler --> Resolver["Source Resolver"]
Resolver --> Config
Resolver --> Cache["Region Feed Cache"]
Resolver --> Fetcher["RSS / Atom Fetcher"]

View File

@@ -1,5 +1,7 @@
# Earth Render Layer Order
The compute topic reuses the compute-center Interactable icons, depth behavior, and layer order. Country/year changes replace that layer's data and clear prior selection. Its HUD is screen UI excluded from Earth picking in `main.js`. Regional cards are tethered in screen space; rear anchors do not draw connectors through the globe. Facility details reuse the existing card and mobile details drawer. The visible HD surface uses a single-draw ShaderMaterial in `earth-surface-material.js`, retaining the R+0.48 mesh, depth policy and surface pick target. The existing controller still switches all four 8K maps, updates the sampler and disposes the previous texture only after success. Normals, view direction and astronomical sunlight share world coordinates. Land is matte; water has grazing-angle cool reflection without a white specular lobe. The existing geographic land/ocean texture owns the optical mask independently of border visibility, with a conservative chroma fallback until loaded and reflection suppression over ice. Original JPEGs are neither repainted nor downsampled. Color textures are tagged sRGB with mipmaps. The r128 surface shader decodes sRGB; r160 uses hardware texture decoding. Both combine light in linear space and encode once. A local filmic curve affects only the surface instead of enabling global ACES exposure on the sky and data layers. The terminator spans normal dot products [-0.12, 0.28]; night attenuation retains 0.12–0.32 linear illumination after tone mapping to avoid crushing source shadows twice. One R×1.025 BackSide/Additive shell replaces the three rim shells, with depthWrite=false and depthTest=true; inner scattering is part of the surface pass. Independently switchable clouds use matte day/night shading at renderOrder 1.1. The sky uses the existing `starmap_deep_8k.jpg` without a tint overlay.
This document records the current Earth renderer's layer order and the intent of each layer. When adjusting `renderOrder`, radius offsets, depth strategy, or pointer interaction, update this document accordingly.
Note: the layer control panel order and the registration / startup load order are two separate semantics.
@@ -14,32 +16,31 @@ Note: the layer control panel order and the registration / startup load order ar
| Order | Layer | Source | Render / Radius Strategy | Depth / Interaction Strategy | Notes |
| --- | --- | --- | --- | --- | --- |
| -1000 | Celestial background mesh | `celestial.js` | Background sphere | Not part of surface picking | Behind all Earth content. |
| -1 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
| 0.5 | Earth occluder sphere | `earth.js` | Invisible inner sphere | Writes depth buffer | Occludes objects behind the Earth. |
| 0 | Earth base sphere | `earth.js` | `CONFIG.earthRadius` | Surface picking fallback target | Dark base; still visible when all optional map layers are off. |
| 0.2 | Country dark tint | `country-boundaries.js` | `tintAltitudeOffset` | Raycast disabled | Used when HD texture is off. |
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order `1` and altitude offset `0.2` are unchanged | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
| 1 / 1.1 | Atmosphere / clouds | `earth.js` | One atmosphere shell at R×1.025; clouds at R+3 | Atmosphere uses BackSide, depthWrite=false, depthTest=true; cloud toggle retained | Inner scattering shares the surface pass; no fullscreen post-processing. |
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order remains `1`; cables and landing-point anchors share the HD surface offset `0.48` and the same triangle-surface coordinates | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
| 0.97–0.99 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
| 0.98 | Country border hover glow | `country-boundaries.js` | Same surface mapping, no extra lift | `depthTest: true`, transparent render queue, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 0.99 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: true`, transparent render queue, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
| 3-4.5 | BGP collectors, event rings, and event markers | `bgp.js`, `interactable.js` | Collector markers use `BGP_COLLECTOR_RENDER_ORDER = 4.4`; event markers use `BGP_EVENT_RENDER_ORDER = 4.5`; overlays remain in BGP-owned groups | Screen-space Interactable picking; BGP collectors/events may use collision avoidance where appropriate | BGP collectors share the surface-facility band with vessels; BGP events share the compute-center band. |
| 4.3 | AIS vessel track lines | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`; `CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | Follows vessel selection, not independently picked | Recent track for the selected vessel, below vessel markers. |
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays | `depthTest: true`; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
| 4.4 | AIS vessel markers | `vessels.js`, `interactable.js` | `VESSEL_RENDER_ORDER`; business radius `CONFIG.earthRadius + VESSEL_CONFIG.altitudeOffset`; normal markers are bucketed `THREE.Points`, hover / locked states are single-point overlays; whole-anchor occlusion matches BGP and compute markers at the globe limb | `depthTest: false`, with visibility decided by anchor occlusion; `main.js` uses screen-space picking for front-facing markers; `cluster: false` and `avoidance: false` | Moving vessels use triangular point textures, anchored / slow vessels use dots. Dense waterways may overlap and do not participate in dynamic screen clustering, avoiding Points rebuilds during rotation. |
| 4.5 | Compute centers | `compute-centers.js`, `interactable.js` | `COMPUTE_CENTER_RENDER_ORDER` | Screen-space Interactable picking, with surface-icon collision handling | Surface facilities, below satellites. |
| 5 | Satellite background dot | `satellites.js` | Fixed renderOrder; by default TLE/SGP4 altitude is compressed to `CONFIG.earthRadius + 4..25`; with real altitude disabled or propagation failed, uses `fallbackAltitudeOffset = 8` | Screen-space satellite picking | Below satellite dots. |
| 6 | Satellite dots | `satellites.js` | Same compressed / fallback height as satellite backdrop dots | Screen-space satellite picking | Satellite dots above footprints and compute centers. |
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
| 6 | Satellite dots | `satellites.js` | SGP4 altitude compressed to R+4..25; fallback R+8 | `depthTest=false`; vertex shader tests camera-to-satellite segment occlusion against an R×0.998 sphere; screen-space picking | Visible sprites remain complete; truly occluded points move outside clip space instead of being partially depth-clipped near the limb. |
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw an open full-period inertial curve with forward-flow highlights; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
| 98-100 | Sun / moon halo and sprite | `celestial.js` | Fixed renderOrder | Celestial picking disabled | Foreground celestial sprites. |
## Full-set rendering and updates
- Batching changes GPU submission, not the number of satellites, cable segments, or landing points. It does not filter data by the visible hemisphere. Every adjacent cable vertex pair remains present, and picking still returns the original business object.
- Satellite foreground and backdrop remain two `Points` draws. Vertex shaders compute breathing from static per-point parameters and a shared frame-time uniform; existing hover/locked state still controls point masking.
- Normal satellites use one hard-disc `Points` batch without a backdrop batch. Optional breathing remains in the vertex shader, defaulting off; category filters and hover/locked masking share the same visibility state.
- `satellite-position-worker.js` uses the same Three.js / SGP4 versions as the main thread. `satellite-propagation.js` owns shared orbit, display-altitude, and fallback calculations. Full positions and initial trail samples use transferable arrays; only one calculation is in flight. Reload, clear, and altitude-mode changes terminate the previous Worker.
- Worker startup failure or `SATELLITE_CONFIG.workerStartupTimeoutMs` expiry falls back to the shared synchronous calculation. Counts, APIs, the existing update interval, and trail length are unchanged.
- Verify full draw ranges, per-item picking, locked overlays, trails, rear-side occlusion, visibility toggles, and resource release after clearing. Frame-time comparisons require identical data, view, and resolution.
@@ -73,7 +74,35 @@ Maintenance rules:
| --- | --- |
| Earth coordinate hover | When HD texture is visible, uses the HD texture overlay as the surface picking target; otherwise uses the Earth base sphere. |
| Country border hover | Converts surface pick coordinates to lat/lon, then uses GeoJSON point-in-polygon; the border hover line itself does not receive raycasts. |
| Country border hover visual | On hover, dims normal border lines and draws no-depth-test glow and solid lines. |
| Country border hover visual | On hover, dims normal border lines and draws depth-tested glow and solid lines below landing points. |
| China / Taiwan hover | `CHN` and `TWN` are grouped in the same hover highlight group; the tooltip still shows the actually-hit feature. |
| Terrain | Acts as a visual layer only; `terrain.raycast` is disabled. |
| Satellites | Uses screen-space satellite picking to prevent footprints or surface layers from blocking satellite clicks. |
### Surface attachment and landing point order
`earth-surface.js` projects geographic vertices onto the high-resolution SphereGeometry UV triangles at geometry construction time. Boundaries, coastlines, claim lines, hover outlines and landing points share this surface mapping. Long edges are subdivided using the surface grid spacing. Landing-point anchors now use the texture shell offset of 0.48; cable line offsets remain 0.2. The draw order is texture 0.96, boundaries/highlights 0.97–0.99, then landing points 1. Hover outlines always use the transparent queue with depth testing, so they do not draw through the globe or over landing markers. Geography is unchanged, and cached geometry is not rebuilt during rotation.
### Startup loading versus drawing order
The selected full-resolution surface loads before independent data requests; boundary construction no longer blocks its first frame. This does not change renderOrder, depth behavior, geometry precision or image resolution. Data refresh retains the surface, with texture disposal limited to successful replacement or scene teardown.
### Loading-state boundary diamond regression
`v0.58.0` moved land fill to R+0.32 and HD imagery to R+0.48, but left the tint placeholder at R+0.04 with a 64×32 mesh. Its chords intersected the 128×128 base, producing latitude-ring diamonds until loaded imagery covered them. The old geometry penetrated the base by approximately 0.153 at tested samples.
Base, tint, land fill and HD surface now use `createEarthSurfaceGeometry()` and one UV triangulation. Tint and land share R+0.32; neither writes depth, so their existing renderOrder lets loaded fill replace the tint. HD remains R+0.48. Occluder, terrain, clouds, picking and geographic display policy are unchanged. Do not mask intersecting geometry by changing renderOrder alone, disabling depth tests or hiding the loading globe.
`earth-surface.test.js` ray-tests the actual loading tint and verifies at least two 24-bit depth steps of separation at 50%, 100% and 500%. Browser checks cover delayed textures, boundary toggles, rotation/zoom, and land/HD completion.
### Surface marker alignment and limb visibility
Cables and landing points share R+0.48, the `earthSurfacePoint` triangle projection and renderOrder=1. Related landing halos use an emphasis attribute in the existing batch, adding no Sprite or draw call. Rounded badges and vessel heading arrows use GPU anchor ray/sphere occlusion: draw the entire billboard when its anchor is visible, otherwise clip it entirely. CPU picking uses the same test to reject rear anchors. The selected BGP event's static overlay halo uses the event icon's R+0.48 anchor, foreground order and whole-anchor front/rear visibility; the existing animated BGP event ring remains a separate effect with its original behavior.
The `.map-clusters` projection layer uses z-index 8 below HUD and whole-marker occlusion from real surface anchors. Ordinary points retain their Three.js layers; transitioning members render only once in the projection layer. Spider legs use widened transparent hit areas; hover highlights the matching line and child, and both clicks resolve the same business object. Wheel input over projected markers and legs continues to zoom the globe.
Co-located markers share a compact projection stack across categories; screen collisions never merge different locations. A selected child uses projection `z-index: 1000`, while the locked overlay for an unclustered marker uses the shared foreground `renderOrder >= 100`, so focus stays on top across layers; clearing focus restores stable ID order. The aggregate badge stays below a selected child during its opening tween. `icon_presentation_screen` lets detail connectors follow display offsets while markers retain their real anchors. Ownership changes immediately rebuild the corresponding GPU batch.
Regional news and compute retain the `#earth-region-callouts` root (z-index 20) and share flank/corridor assignment with object details through `callout-stage-arbitrator.js`. Expanded cards inside the regional root use 50, drag foreground 55, hover expansion 60–65, ordinary capsules 30 and dimmed/shrunk rear capsules 8. Hover capsules preserve full content and actions without becoming stage obstacles. Pinned rectangles reserve space first; compact viewports retain full stage cards. Smoothed anchors preserve raw visibility and manual pinning releases only at confirmed rear depth below -0.65. Geographic projections, connector sources and card targets remain independent; layout changes no Three.js coordinates, picking or render depth. News anchors still represent regions rather than precise events. Thin/bold styles come from `body.dataset.calloutLineStyle`; mobile cards move into the drawer without map connectors.

View File

@@ -1,5 +1,9 @@
# Admin Frontend Context
## Compute comparison configuration
The Earth content `compute_comparison` section mounts `ComputeComparisonSettings.tsx`. Structured forms maintain report sources, annual country observations and innovation gaps using the shared Panel, Input, Select, Button and confirmation dialog; operators do not edit JSON. Drafts survive section switches, full-page unload warns about unsaved work, and discarding a draft or restoring defaults requires confirmation. Management data loads on activation. Publishing sends the backend revision to prevent overwriting concurrent edits, and failures preserve the draft. The content area scrolls internally while the heading and publish controls stay in the first viewport. `compute-comparison-model.ts` owns types and same-origin update notifications; `compute-comparison.css` only owns this section's layout.
This document describes the current real structure of the console frontend. The goal is to help future page development, table refactoring, layout governance, and state consolidation quickly find the right entry points.
Related references:
@@ -71,6 +75,8 @@ This keeps Earth, AI, collection management, and other multi-section pages from
## Datasource Collection Queue
`datasource-task-queue.ts` reconciles attempts by task ID. Retrying a source adds a new attempt and preserves failed/completed/cancelled history; source matching is allowed only for active placeholders without an ID. Late events cannot reactivate terminal attempts, and older task polls cannot overwrite the latest datasource task. Task status uses `ListChecks`; refresh retains `RefreshCw`.
The Admin datasource page routes single-source trigger, table-selected trigger, and trigger-all into a browser-download-list style collection queue:
- Queue state is managed by [PlainResourcePages.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx). It is a current-session visibility layer and does not fake task history in `localStorage`.
@@ -389,7 +395,7 @@ Current boundary:
- `TV Livestream` owns the Earth media-panel source configuration.
- `Branding` owns Earth HUD brand assets. `Logo URL` and `Title Image URL` use inline upload controls inside the fields; the upload buttons keep the primary `TactileButton` style, and dropping an image onto the matching field shows a low-saturation drag reaction instead of the old global asset-picker toolbar.
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
- `Base Map`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
Do not add Earth experience resources or collection-lifecycle tabs back into `/settings`; collection belongs to `/collection-management`, and Earth display resources belong to `/earth-content`.
@@ -448,3 +454,11 @@ Therefore:
For Earth-related structure, see:
- [Earth Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/earth-frontend-context.md)
## Detail-card image workspace
`CardImageSettings.tsx` replaces the `layer_resources` placeholder in Earth Content, retaining the shared page frame, section navigation, controls and confirmation dialogs. Its persistent component preserves drafts across section changes. List and editor own their scrolling with a constrained height chain; narrow layouts stack these regions. `CardImagePreview.tsx` reuses the actual Earth mask stylesheet.
The image library supports upload/URL entry, attribution, licensing, enabled state and crop focus. Referenced assets cannot be deleted until assignments are removed. Assignments use database-backed object/model/group/category choices and support edit, disable, removal and explicit per-object suppression. Draft preview calls the same resolver used after publication. Only Save publishes; conflicts retain drafts, and the editing fieldset is disabled during save/upload. Uploading alone does not publish an asset.
Under `/earth/card-images`, configuration GET/PUT, upload POST, targets GET and preview POST require administrators. Resolve and content-addressed files are public read-only routes. Configuration writes use revision checks and audit transactions.

View File

@@ -16,6 +16,10 @@ The refactored rules:
- Earth renders only city-level or better locations.
- Local JSON registries are not runtime candidate sources for compute centers or BGP collectors; persisted location facts live in database dimension tables.
For compute-center display, `precise` and `site` candidates keep their source coordinates. `city`, `estimated`, and `needs_confirmation` records that share a coordinate are separated only in `displayLatLon`: a stable hash of `source` and `source_id` selects an irregular cloud direction and a radius in the 0.4–6 km range. The display offset is deterministic, is neither a circle, grid, nor spiral, and never uses a database row ID or list order. Source coordinates are never written back. Tooltips and detail cards label these points “City-level approximate location (dispersed display)”.
The resolver priority is confirmation cache, source precise coordinate, official registered site, source city/site, then unconfirmed cache. A city-level approximation must never be presented as `precise` or as a guessed headquarters. Official registration helpers are keyed by exact `source` + `source_id`; the current entries cover Gyoukou 2.0/2.2 at JAMSTEC Yokohama and Sophia ALCF (three records across two campuses). Their official access-map centers are site points only, not rack/building survey points, and each candidate keeps the official URL in `source_note`.
## Core Interfaces
```python
@@ -51,7 +55,7 @@ class LocationResolver(Protocol):
def resolve(self, query: LocationQuery) -> ResolverOutput: ...
```
`LocationPipeline.collect_candidates()` returns sorted candidates plus `attempted_queries`; `resolve_best()` returns the best candidate with diagnostics. The default sort key ranks source, precision, and confidence, then deduplicates candidates with the same source and rounded coordinates.
`LocationPipeline.collect_candidates()` returns sorted candidates plus `attempted_queries`; `resolve_best()` returns the best candidate with diagnostics. The default sort key ranks source, precision, and confidence, then deduplicates candidates with the same source and rounded coordinates. Legacy records are reclassified through the same path as `country`, `region`, or `unknown`; non-renderable records remain in the complete `unresolved` list.
## Built-In Resolvers

Some files were not shown because too many files have changed in this diff Show More