Compare commits

..

34 Commits
v0.74.4 ... dev

Author SHA1 Message Date
linkong
1d153c2667 release: bump version to 0.76.20
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (push) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-11-10 17:52:24 +08:00
linkong
c1d651589a release: bump version to 0.76.19
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-11-09 15:12:50 +08:00
rayd1o
36be9e3da6 release: bump version to 0.76.18
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-10-02 05:46:33 +08:00
rayd1o
7d8822d34d release: bump version to 0.76.17
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:38:52 +08:00
rayd1o
ae1a4b3942 release: bump version to 0.76.16
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:22:11 +08:00
rayd1o
35e478642b release: bump version to 0.76.15
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 03:11:41 +08:00
rayd1o
33734b08b8 release: bump version to 0.76.14
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:56:14 +08:00
rayd1o
bbe47c5759 release: bump version to 0.76.13
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:43:55 +08:00
rayd1o
361bf258e4 release: bump version to 0.76.12
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:33:36 +08:00
rayd1o
83c552f9b2 release: bump version to 0.76.11
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:21:20 +08:00
rayd1o
15fed1e121 release: bump version to 0.76.10
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 02:10:26 +08:00
rayd1o
f4514b588e release: bump version to 0.76.9
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:56:05 +08:00
rayd1o
9523f2bb83 release: bump version to 0.76.8
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:41:46 +08:00
rayd1o
9dc3b61fa5 release: bump version to 0.76.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 01:09:28 +08:00
rayd1o
698863da68 release: bump version to 0.76.6
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 00:33:55 +08:00
rayd1o
4c0b49962c release: bump version to 0.76.5
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-02 00:17:38 +08:00
rayd1o
ffb3134d93 docs: document breaking news state machine
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-10-01 21:46:59 +08:00
rayd1o
e8742b5d3c release: bump version to 0.76.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 21:11:00 +08:00
rayd1o
28a7132eec release: bump version to 0.76.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 17:48:45 +08:00
rayd1o
dd41588527 release: bump version to 0.76.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 02:29:49 +08:00
rayd1o
45604a1ebb release: bump version to 0.76.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 01:25:24 +08:00
rayd1o
06c122bade fix: use whole-anchor occlusion for BGP icons
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:59:21 +08:00
rayd1o
96e9b15fda fix: keep satellite selection glow whole at limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:21:22 +08:00
rayd1o
a8713e58fd fix: preserve BGP event halo at globe limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 18:23:49 +08:00
rayd1o
05377bf5c6 release: bump version to 0.76.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-30 17:49:29 +08:00
rayd1o
67548d8278 release: bump version to 0.75.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-29 15:40:32 +08:00
rayd1o
4b8feeb87e release: bump version to 0.75.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-27 09:30:06 +08:00
rayd1o
a2025dbfe0 release: bump version to 0.75.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-25 02:03:53 +08:00
rayd1o
003b316d42 release: bump version to 0.75.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-25 00:16:30 +08:00
rayd1o
0c40764c58 feat: configure compute comparisons and restore adaptive globe dragging
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-24 22:59:55 +08:00
rayd1o
5bfc790dec release: bump version to 0.75.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-23 21:20:12 +08:00
linkong
60eefb19c9 release: bump version to 0.74.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-17 17:30:54 +08:00
rayd1o
83a10a6c34 release: bump version to 0.74.6
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-16 21:05:40 +08:00
rayd1o
cee1996809 release: bump version to 0.74.5
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-13 14:04:34 +08:00
494 changed files with 37670 additions and 8375 deletions

View File

@@ -19,9 +19,9 @@ Do not refactor architecture, add features, or change behavior.
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
Only report and fix issues present in **newly added or modified** lines of this diff — do not audit unchanged code, and do not rewrite unrelated files during cleanup.
## Token-Saving Rule
## Token-Saving Rule & Harness Checks
Prefer deterministic CLI checks before reading files into model context:
@@ -29,27 +29,36 @@ Prefer deterministic CLI checks before reading files into model context:
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
Use the repository's focused harness checks for the touched surfaces:
- Backend Python changes: `scripts/harness/backend-rules-check.sh` (catches direct `print()`, `breakpoint()`, and debugger calls)
- Frontend changes: `scripts/harness/frontend-rules-check.sh` (catches console logs, missing button types, Card nesting, layout height-chain issues)
- Secrets / credentials: `scripts/harness/security-check.sh`
Read full files only when the focused diff does not provide enough surrounding context to make a safe edit.
## Checklist
### 1. Duplicate Logic
- Identical or near-identical code blocks appearing in multiple places
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
- Repeated DOM queries, regex literals, or template strings within the same file
- Identical or near-identical code blocks appearing in multiple places within the modified scope
- A helper that already exists but is re-implemented elsewhere instead of being reused
- Repeated complex queries or template strings that cause actual maintenance hazard
- **Anti-Over-Defensiveness**: Keep one-off logic at its call site. Do not prematurely abstract single-use or 2-use trivial helpers/templates that are clearer inline.
### 2. Magic Numbers / Magic Strings
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) where the intent is completely opaque and naming adds clear value
- Hardcoded domain strings (IDs, status values, protocol fragments) scattered through logic
- **Anti-Over-Defensiveness (AGENTS.md Quality Bar)**:
- Idiomatically clear values such as `0`, `1`, `-1`, `100`, `""` are fine inline.
- **Do NOT over-extract local single-use layout offsets, Three.js coordinates, padding, or math coefficients into module-level globals.**
- Keep one-off logic at its call site unless extraction clarifies a complex operation or lets callers share the same behavior.
### 3. Naming Issues
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
- Names that do not match actual behavior
- The same concept referred to by different names in different places
- The same concept referred to by different names in different places within the changed scope
### 4. Dead Code
- Commented-out code blocks (3+ lines)
@@ -58,12 +67,12 @@ Read full files only when the focused diff does not provide enough surrounding c
### 5. Style Inconsistencies
- Trailing whitespace
- Mixed quote styles or indentation within the same file
- Mixed quote styles or indentation within the same file (Python: 4 spaces Black style; TypeScript: 2 spaces Prettier style)
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
### 6. Other
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
- Overly verbose conditions that can be simplified without changing logic
- Keep fixes strictly scoped to the modified files; do not rewrite external callers or expand cleanup beyond the changed scope.
## Steps
@@ -107,30 +116,39 @@ If no issues are found, output "No code smells detected. Code quality looks good
Use the Edit tool for **minimal, targeted changes**:
- **Duplicate logic**: extract to a shared constant or function; update all call sites
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
- **Naming**: rename the variable/function; update all references
- **Duplicate logic**: extract to a shared constant or helper within the owned file or component scope; update call sites strictly within scope
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope only when the value is reused across places or its meaning is genuinely opaque; replace usages. Keep one-off UI/math offsets inline per `AGENTS.md`
- **Naming**: rename the variable/function; update all references within the modified scope
- **Dead code**: delete it
- **Trailing whitespace / style**: fix in place
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
- **Unexported helper**: add `export` only when requested; do not modify unrelated callers
Principles:
- Only fix issues identified in the checklist — no extra improvements
- Only fix issues identified in the checklist — no extra improvements or speculative refactorings
- Keep each Edit as small as possible
- After fixing, verify the old bad pattern is gone with grep
- Prefer `apply_patch` for targeted edits; use formatters only when the repository already uses them for the touched file type
- After fixing, verify the old bad pattern is gone with grep or deterministic harness checks
- Prefer `apply_patch` or targeted edit tools; use formatters only when the repository already uses them for the touched file type
### Step 5 — Summary
### Step 5 — Verify and summarize
Run the smallest relevant harness checks for the modified surfaces:
- Python files: `uv run python -m py_compile <changed_py_files>` and `scripts/harness/backend-rules-check.sh`
- Frontend files: `scripts/harness/frontend-rules-check.sh` (and `bun run --cwd frontend build` if imports/types were adjusted)
- Formatting / whitespace: `git diff --check`
```
Cleanup complete:
Fixed N issues:
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
✓ main.js — clarified variable naming in news event payload handler
✓ controls.js — removed unused dead import
...
Validation:
✓ git diff --check passed
✓ scripts/harness/frontend-rules-check.sh passed
Skipped (needs manual review):
! foo.js L45 — large commented-out block; confirm it is safe to delete
```
@@ -141,4 +159,6 @@ Skipped (needs manual review):
- **Do not** add new features, abstractions, or parameters
- **Do not** rewrite comments (only delete commented-out dead code)
- **Do not** touch test file logic
- **Do not** rewrite unrelated files during cleanup (per `AGENTS.md` Prohibited)
- **Do not** over-extract local single-use coordinates, layout numbers, or inline math coefficients into global constants (preventing defensive over-engineering)
- If a magic number's intent is uncertain, skip it and flag it in the summary

View File

@@ -48,18 +48,22 @@ rg -n "class |def |function |export |router|@router|interface |type " <path>
3. Write the doc:
- Explain background/problem, design decisions, constraints, and operational impact.
- Include the context, rationale, or operational impact the reader needs for this change.
- Keep code snippets short and directly relevant.
- List related files only when they help future maintainers navigate.
- Use the repository’s existing language, heading style, and naming conventions.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. Only update the glossary when introducing new, ambiguous, or cross-language public concepts; avoid unnecessary glossary churn for ordinary bug fixes or internal refactors.
4. Verify:
- Read the completed doc once for clarity and stale statements.
- Verify important referenced paths exist with `test -e` or `rg --files`.
- Run repository-specific doc checks from `docs/documentation-coverage-rules.md` when present.
- For Markdown links, check that user-facing titles are readable and not raw filenames unless the repository rules allow it.
- Run the repository's centralized documentation harness gate:
```bash
scripts/harness/docs-consistency-check.sh
```
This automatically verifies bilingual pairs, public registry whitelist, link anchors, language-scoped links, section deep-links, and stale admin route terms in a single pass.
- Run any additional domain-specific checks from `docs/documentation-coverage-rules.md` when applicable.
## Hard Constraints

View File

@@ -19,9 +19,11 @@ Do not use this skill for ordinary commits that are not being released.
## Versioning Rules
- `feature` -> bump minor and reset patch to `0` (`x.y.z` → `x.(y+1).0`; for example `0.41.2` → `0.42.0`)
- `bugfix` -> bump `+0.0.1`
- `improvement` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
- `bugfix` -> bump patch `+0.0.1` (`x.y.z` → `x.y.(z+1)`)
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
Mixed bugfix and small feature/UI work defaults to `improvement` unless the user explicitly chooses another release type.
When intent is mixed, prefer the user's stated release intent.
## Required Files
@@ -53,28 +55,32 @@ Do not inspect full diffs unless deciding whether changed code belongs in the re
### Step 1 — Environment check
```bash
git branch --show-current # must be on dev
git branch --show-current # check current branch
git status --short # check for unrelated uncommitted changes
cat VERSION # read current version
```
If not on `dev`, stop and tell the user. Do not proceed.
Default release branch is `dev`. If not on `dev` (e.g. on a dedicated maintenance or release branch), report the active branch and confirm before proceeding, rather than hard-failing.
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
### Step 2 — Determine release type and next version
- If the user provided an explicit type (`feature` / `bugfix`), use it
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`
- If the user provided an explicit type (`feature` / `improvement` / `bugfix`), use it
- Otherwise infer from `git diff --stat HEAD`, `git diff --name-only HEAD`, focused diffs for changed code, and recent `git log`:
- New features or major capabilities -> `feature`
- Mixed bugfix and small feature/UI work -> default to `improvement` (per `rules.md`)
- Bugfixes only -> `bugfix`
- Compute the next version:
- `feature`: increment minor and reset patch to `0` (e.g. `0.41.2` → `0.42.0`)
- `bugfix`: increment patch only (e.g. `0.26.2` → `0.26.3`)
- `improvement`: increment patch only (e.g. `0.76.18` → `0.76.19`)
- `bugfix`: increment patch only (e.g. `0.76.18` → `0.76.19`)
- **Show the release plan before making any changes:**
```
Release plan:
Type: bugfix
Version: 0.26.2 → 0.26.3
Type: improvement
Version: 0.76.18 → 0.76.19
Branch: dev
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
@@ -117,17 +123,25 @@ Get today's date with `date +%Y-%m-%d`.
### Step 6 — Validate
Run the smallest relevant validation for the changes in scope:
Run the smallest relevant harness validation for the changes in scope:
- Python files changed: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `python3 -m py_compile <changed_files>`
- Frontend files changed: list changed frontend files with `git diff --name-only HEAD -- frontend`, then run the project-standard check if available; otherwise skip and say so
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
- **Version consistency**: confirm `VERSION`, `frontend/package.json`, `pyproject.toml`, and `uv.lock` all show the identical target version:
```bash
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
- **Security scan**: run `scripts/harness/security-check.sh`
- **Python files changed**:
- Compile: list changed Python files with `git diff --name-only HEAD -- '*.py'`, then run `uv run python -m py_compile <changed_files>`
- Service rules: run `scripts/harness/backend-rules-check.sh`
- **Frontend files changed**:
- Rules: run `scripts/harness/frontend-rules-check.sh`
- Build: run `bun run --cwd frontend build` (Bun only; never skip)
- **Docs changed**: run `scripts/harness/docs-consistency-check.sh`
- **Broad / cross-subsystem changes**: run `scripts/harness/quick-check.sh`
### Step 7 — Pre-commit preview
Show what will be committed:
@@ -141,29 +155,36 @@ Confirm all required files are present and no unexpected files (debug files, `.e
### Step 8 — Commit, tag, and push
```bash
branch=$(git branch --show-current)
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# also stage any code changes included in this release
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
Commit message format is fixed: `release: bump version to x.x.x`
Push to remote:
- When remote publication is requested:
```bash
git push origin "$branch"
git push origin vx.x.x
```
- If remote publication was not requested or remote access is restricted, report the commit and tag as complete and output the push commands for manual execution.
### Step 9 — Completion summary
```
✓ Version bumped: 0.26.2 → 0.26.3
✓ Version bumped: 0.76.18 → 0.76.19
✓ CHANGELOG updated
✓ version-history updated
✓ uv.lock regenerated
✓ Validation passed
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ Pushed to origin/dev
✓ commit: release: bump version to 0.76.19
✓ tag: v0.76.19
✓ Pushed to origin/dev (or reported pending manual push)
```
## Notes

View File

@@ -3,8 +3,23 @@
!pyproject.toml
!uv.lock
!VERSION
!frontend/
!frontend/public/
!frontend/public/earth/
!frontend/public/earth/assets/
!frontend/public/earth/assets/cards/
!frontend/public/earth/assets/cards/manifest.json
!backend/
!backend/**
!config/
!config/earth-boundary-pov-policy.china-v1.json
!scripts/
!scripts/build_earth_boundary_china_pov_geojson.py
!scripts/build_earth_boundary_pmtiles.py
!scripts/earth_boundary_policy.py
!resources/
!resources/earth/
!resources/earth/boundary-pov-geometry.china-v1.json
!aiprovider/
!aiprovider/**

5
.gitignore vendored
View File

@@ -160,3 +160,8 @@ data/earth-boundary-sources/
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
# committing thousands of generated tile files.
frontend/public/earth/data/boundaries/
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
resources/earth/candidates/nasa-blue-marble/*.jpg
resources/earth/candidates/natural-earth-3/*.jpg
*:Zone.Identifier

View File

@@ -55,13 +55,19 @@ Preserve user changes already present in the worktree.
### Validation
Fast local harness validation:
For an isolated change, run the smallest check that exercises the affected
behavior, along with any gates required for that area. For example, run the
docs consistency check for a link change, or reproduce a bug at the entry
point where it occurred.
Use the quick suite when a change affects shared harness wiring or needs
regression checks across areas:
```bash
scripts/harness/quick-check.sh
```
Full local validation:
Use full validation for larger cross-subsystem changes:
```bash
scripts/harness/validate.sh
@@ -113,6 +119,9 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
## Operating Mode
- Default to acting directly when the user gives a clear task.
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
and Astra for complex reasoning; the main agent owns final verification.
- Ask before acting only when the missing decision is risky, cannot be
discovered from repository context, and no conservative assumption is safe.
- Read relevant files before editing.
@@ -120,6 +129,8 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
focused file reads, tests, builds, linters, and harness scripts.
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
release work, and documentation unless the task requires it.
- When the user corrects a detail, apply it to the original request. Carry the
correction into the result without turning discarded ideas into new requirements.
---
@@ -141,6 +152,12 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
visual-dependent part of the work.
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
diff, direct file check, or harness result.
- Verify the outcome through the entry point the user relies on. A source check
can confirm a code convention. A UI test with a mocked API can confirm what
the user sees for that response.
- When a test fails, identify the behavior or invariant it protects. Fix the
implementation if that regressed. If the behavior is correct and the assertion
pins an obsolete detail, update the test to exercise the behavior.
- For UI and rendering changes, verify the rendered result when local tooling
allows it.
@@ -159,7 +176,14 @@ behavior and document the compatibility note in `docs/harness-audit.md` or
## Quality Bar
- Prefer existing project patterns over new abstractions.
- Prefer existing project patterns and the smallest change that satisfies the
request. Keep one-off logic at its call site unless extraction clarifies a
complex operation or lets callers share the same behavior.
- Validate and normalize external data at the boundary that owns it. Recheck a
guarantee when data crosses a new trust boundary or an operation can invalidate it.
- Before adding a retry, lock, or fallback, identify the failure it addresses
from a requirement, an existing contract, or observed behavior. Catch errors
where the caller can recover, translate them, or add context and propagate them.
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
a stable path exists.
- Centralize prompts, constants, defaults, and shared request/response handling.

View File

@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
## Runtime Entry Points

View File

@@ -1 +1 @@
0.74.4
0.76.20

View File

@@ -1,4 +1,4 @@
# syntax=docker/dockerfile:1.7
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest

View File

@@ -4,7 +4,7 @@
完整使用说明见:
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
当前支持:
@@ -15,6 +15,7 @@
- `AI_PROVIDER=ollama`
- request adapter:
- `AI_PROVIDER_API=openai-completions`
- `AI_PROVIDER_API=openai-responses`
- `AI_PROVIDER_API=anthropic-messages`
- `AI_PROVIDER_API=ollama-generate`

View File

@@ -3,6 +3,7 @@ from __future__ import annotations
import asyncio
import json
from typing import Any
from uuid import NAMESPACE_URL, uuid4, uuid5
import httpx
from fastapi import HTTPException, status
@@ -65,6 +66,7 @@ class ProviderService:
self.model_provider_apis = self._parse_model_provider_apis(
overrides.get("model_provider_apis")
)
self.session_id = str(uuid4())
def get_status(self) -> AIProviderStatusResponse:
enabled = self.provider != "disabled"
@@ -95,6 +97,8 @@ class ProviderService:
)
prompt = self._build_prompt(payload)
if payload.context.get("session_id") is not None:
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
provider_api = self._resolve_model_provider_api(model)
@@ -102,6 +106,10 @@ class ProviderService:
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
content = self._extract_openai_content(data)
content_blocks = self._extract_openai_blocks(data)
elif provider_api == "openai-responses":
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
content_blocks = self._extract_responses_blocks(data)
content = "".join(block.text for block in content_blocks if block.text)
elif provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(
model,
@@ -200,6 +208,39 @@ class ProviderService:
request_body=request_body,
)
async def _request_openai_responses(
self, model: str, prompt: str, system_prompt: str | None = None,
) -> dict[str, Any]:
request_body: dict[str, Any] = {
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
}
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["instructions"] = resolved_system_prompt
return await self._post(
path="/responses",
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
request_body=request_body,
)
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
blocks: list[AIContentBlock] = []
for item in payload.get("output") or []:
if not isinstance(item, dict):
continue
if item.get("type") == "message":
for part in item.get("content") or []:
if not isinstance(part, dict):
continue
text = part.get("text") or part.get("refusal")
if isinstance(text, str) and text:
blocks.append(AIContentBlock(type="text", text=text))
elif item.get("type") == "reasoning":
for part in item.get("summary") or []:
if isinstance(part, dict) and isinstance(part.get("text"), str):
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
return blocks
async def _request_anthropic_messages(
self,
model: str,
@@ -226,7 +267,7 @@ class ProviderService:
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["system"] = resolved_system_prompt
resolved_thinking = self._resolve_anthropic_thinking(thinking)
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
if resolved_thinking:
request_body["thinking"] = resolved_thinking
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
@@ -243,8 +284,12 @@ class ProviderService:
request_body=request_body,
)
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
def _resolve_anthropic_thinking(
self, thinking: dict[str, Any] | None, model: str,
) -> dict[str, Any] | None:
if thinking:
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
return {"type": "adaptive"}
return thinking
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
@@ -295,6 +340,9 @@ class ProviderService:
headers: dict[str, str],
request_body: dict[str, Any],
) -> dict[str, Any]:
headers = {"User-Agent": "Planet/1.0", **headers}
if self.provider == "opencode-go":
headers["x-opencode-session"] = self.session_id
last_error: Exception | None = None
for attempt in range(1, self.http_retry_attempts + 1):
try:

View File

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
COPY VERSION /app/VERSION
EXPOSE 8000

View File

@@ -1,3 +1,4 @@
from app.api.v1 import card_images
from fastapi import APIRouter
from app.api.v1 import (
ai,
@@ -12,6 +13,7 @@ from app.api.v1 import (
alerts,
settings,
collected_data,
compute_intelligence,
data_products,
layers,
visualization,
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])

View File

@@ -0,0 +1,108 @@
"""Card-image configuration, upload, catalog selection and public resolution."""
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
from fastapi.responses import FileResponse
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.concurrency import run_in_threadpool
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
from app.schemas.card_images import (
CardImageConfiguration,
CardImagePreview,
CardImagePublish,
EntityType,
MatchScope,
)
from app.services.card_image_configuration import (
publish_configuration,
read_configuration,
resolve_image,
)
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
from app.services.card_image_targets import get_target, search_targets
router = APIRouter()
async def image_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
):
response.headers["Cache-Control"] = "no-store"
return await read_configuration(db)
@router.put("/configuration")
async def save_configuration(
payload: CardImagePublish,
actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return await publish_configuration(db, payload.revision, payload.config, actor)
@router.post("/upload")
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
content = await file.read(MAX_UPLOAD_BYTES + 1)
await file.close()
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
@router.get("/files/{filename}")
async def image_file(filename: str):
path = uploaded_path(filename)
if not path.is_file():
raise HTTPException(404, "Image not found")
return FileResponse(
path,
headers={
"Cache-Control": "public, max-age=31536000, immutable",
"X-Content-Type-Options": "nosniff",
},
)
@router.get("/targets")
async def targets(
entity_type: EntityType,
scope: MatchScope = "object",
q: str = Query(default="", max_length=100),
limit: int = Query(default=25, ge=1, le=50),
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
@router.get("/resolve")
async def resolve(
response: Response,
entity_type: EntityType,
entity_id: str = Query(min_length=1, max_length=300),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
envelope = await read_configuration(db)
config = CardImageConfiguration.model_validate(envelope["config"])
context = await get_target(db, entity_type, entity_id)
result = resolve_image(config, context)
return {**result, "revision": envelope["revision"], "entity": context}
@router.post("/preview")
async def preview(
payload: CardImagePreview,
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
context = await get_target(db, payload.entity_type, payload.entity_id)
return {**resolve_image(payload.config, context), "entity": context}

View File

@@ -0,0 +1,235 @@
"""Compute observations, administrator configuration and independent facility samples."""
from datetime import UTC, datetime
from typing import Literal
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.core.earth_geography_policy import display_country
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.user import User
from app.schemas.compute_configuration import (
ComputeConfiguration,
ComputeConfigurationReset,
ComputeConfigurationWrite,
)
from app.services.compute_catalog import (
PARTICIPANTS,
participant_members,
source_country_names,
source_country_participants,
)
from app.services.compute_center_locations import refresh_compute_center_location_cache
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
configuration_response,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
router = APIRouter()
def selected_countries(countries: str) -> list[str]:
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
members = participant_members(codes)
flattened = [country for participant in members.values() for country in participant]
if len(flattened) != len(set(flattened)):
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
return codes
@router.get("/series")
async def get_series(
response: Response,
countries: str = "CN,US",
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
envelope = await read_configuration(db)
response.headers["Cache-Control"] = "no-store"
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
result.update(
version=envelope["revision"],
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
)
return result
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response,
_actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
return configuration_response(await read_configuration(db))
@router.put("/configuration")
async def update_configuration(
payload: ComputeConfigurationWrite,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=payload.config, actor=actor,
)
return configuration_response(envelope)
@router.post("/configuration/reset")
async def reset_configuration(
payload: ComputeConfigurationReset,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
)
return configuration_response(envelope)
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
now = datetime.now(UTC)
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
ranked = (
select(
DataSnapshot.id,
func.row_number()
.over(
partition_by=DataSnapshot.source,
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
)
.label("rank"),
)
.where(
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
DataSnapshot.status == "success",
DataSnapshot.completed_at < cutoff,
)
.subquery()
)
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
# Old collector rows have ambiguous units/identity and must not become
# national evidence merely because a new UI can display them.
query = select(CollectedData).where(
compute_record_filter(CollectedData),
CollectedData.deleted_at.is_(None),
CollectedData.snapshot_id.in_(snapshots),
CollectedData.is_valid == 1,
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
or_(
infrastructure,
CollectedData.source == "top500",
and_(
CollectedData.extra_data["status"].as_string() == "Existing",
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
),
),
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
)
return query
@router.get("/facilities")
async def get_facilities(
countries: str = "CN,US",
year: int | None = Query(None, ge=2000, le=2100),
limit: int = Query(1000, ge=1, le=2000),
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
query = await sample_query(db, codes, year)
country = CollectedData.extra_data["country"].as_string()
source_participants = source_country_participants(codes)
# Canonical source subdivisions retain their raw label on each record but
# share one owner in the comparison, using the same policy as the globe.
owner_cases = [
(country == name, display_country(name))
for name in source_country_names(codes)
if display_country(name) != name
]
display = case(*owner_cases, else_=country) if owner_cases else country
participant_cases = [(country == name, code) for name, code in source_participants.items()]
participant = case(*participant_cases, else_=None)
counts = (
(
await db.execute(
query.with_only_columns(
display.label("country"),
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(display, participant, CollectedData.source)
)
)
.mappings()
.all()
)
participant_counts = (
(
await db.execute(
query.with_only_columns(
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(participant, CollectedData.source)
)
)
.mappings()
.all()
)
records = (
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
.scalars()
.all()
)
if year:
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
else:
await refresh_compute_center_location_cache(db)
payload = convert_compute_centers_to_geojson(records)
for feature in payload["features"]:
feature["properties"]["historical_snapshot"] = year is not None
properties = feature["properties"]
properties["source_country"] = properties.get("country")
properties["country"] = display_country(properties.get("country"))
total = sum(row["count"] for row in counts)
payload.update(
{
"counts": [dict(row) for row in counts],
"participant_counts": [dict(row) for row in participant_counts],
"participants": [PARTICIPANTS[code] for code in codes],
"total_count": total,
"returned_count": len(records),
"truncated": total > limit,
"reference_year": year,
"scope": "public_source_records",
"history_basis": (
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
),
"coverage": "available" if total else "no_verified_snapshot",
"cross_source_deduplicated": False,
}
)
return payload

View File

@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
"name": "算力",
"sources": ["top500", "epoch_ai_gpu"],
"primary_stat_key": "compute_center_count",
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
},
}

View File

@@ -13,6 +13,7 @@ from sqlalchemy import delete, func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings as app_settings
from app.core.logging import get_logger
from app.core.security import decode_token, get_current_user, redis_client
from app.db.session import get_db
from app.models.collected_data import CollectedData
@@ -51,6 +52,7 @@ from app.services.earth_boundaries import (
router = APIRouter()
logger = get_logger(__name__)
optional_bearer = HTTPBearer(auto_error=False)
REPO_ROOT = Path(__file__).resolve().parents[4]
EARTH_BRAND_ASSET_DIR = REPO_ROOT / "data" / "earth-brand"
@@ -382,13 +384,28 @@ async def get_earth_news_sources(db: AsyncSession = Depends(get_db)):
return await get_earth_news_sources_payload(db)
def _schedule_earth_news_source_refresh() -> None:
try:
from app.services.scheduler import schedule_earth_news_refresh_for_all_regions
schedule_earth_news_refresh_for_all_regions(force=True)
except Exception as exc:
logger.warning_event(
"Failed to schedule Earth news refresh after source configuration change",
event="earth_news.rss.config_refresh_schedule_failed",
context={"error": str(exc)},
)
@router.put("/news-sources")
async def update_earth_news_sources(
payload: EarthNewsSourcesPayload,
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await save_earth_news_sources_payload(db, payload.model_dump())
result = await save_earth_news_sources_payload(db, payload.model_dump())
_schedule_earth_news_source_refresh()
return result
@router.delete("/news-sources")
@@ -397,7 +414,9 @@ async def reset_earth_news_sources(
_current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await reset_earth_news_sources_payload(db)
result = await reset_earth_news_sources_payload(db)
_schedule_earth_news_source_refresh()
return result
@router.post("/news-sources/test")

View File

@@ -1,17 +1,26 @@
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, status
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from uuid import UUID
from app.db.session import get_db
from app.services.earth_news import (
ALLOWED_NEWS_CATEGORY_KEYS,
SUPPORTED_NEWS_LOCALES,
REGION_ANCHORS,
determine_focus_region,
get_earth_news_payload,
)
router = APIRouter()
class EarthNewsRefreshRequest(BaseModel):
lat: float | None = Field(default=None, ge=-90, le=90)
lon: float | None = Field(default=None, ge=-180, le=180)
region: str | None = Field(default=None, max_length=64)
def _parse_categories(raw: str | None) -> set[str] | None:
if raw is None or not raw.strip():
return None
@@ -39,7 +48,9 @@ def _parse_limit(raw: int | None) -> int:
if raw is None:
return 12
if raw < 1:
raise HTTPException(status_code=422, detail={"message": "News limit must be greater than 0."})
raise HTTPException(
status_code=422, detail={"message": "News limit must be greater than 0."}
)
return min(raw, 100)
@@ -59,18 +70,76 @@ def _parse_locale(raw: str | None) -> str:
return requested
@router.post("/refresh", status_code=status.HTTP_202_ACCEPTED)
async def request_earth_news_refresh(payload: EarthNewsRefreshRequest):
requested_region = payload.region.strip().lower() if payload.region else None
if requested_region and requested_region not in REGION_ANCHORS:
raise HTTPException(
status_code=422,
detail={
"message": "Unsupported news region.",
"invalid_region": requested_region,
"allowed_regions": list(REGION_ANCHORS.keys()),
},
)
region = requested_region or determine_focus_region(payload.lat, payload.lon)
from app.services.scheduler import request_earth_news_region_refresh
try:
return request_earth_news_region_refresh(region)
except Exception as exc:
raise HTTPException(
status_code=503,
detail={
"code": "refresh_failed",
"message": "Could not start a news refresh. Please try again.",
},
) from exc
@router.get("/refresh/{refresh_id}")
async def get_earth_news_refresh(refresh_id: UUID):
from app.services.earth_news_refresh import get_earth_news_refresh_status
try:
result = get_earth_news_refresh_status(refresh_id.hex)
except Exception as exc:
raise HTTPException(
status_code=503,
detail={
"code": "refresh_failed",
"message": "Refresh status is temporarily unavailable. Please retry.",
},
) from exc
if result is not None:
return result
return {
"refresh_id": refresh_id.hex,
"region": None,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
@router.get("/earth-feed")
async def get_earth_feed(
lat: float | None = Query(None, description="Current Earth view center latitude"),
lon: float | None = Query(None, description="Current Earth view center longitude"),
region: str | None = Query(None, description="Explicit Earth news region for UE/client integrations"),
region: str | None = Query(
None, description="Explicit Earth news region for UE/client integrations"
),
categories: str | None = Query(None, description="Comma-separated news category keys"),
sources: str | None = Query(None, description="Comma-separated news source ids"),
limit: int | None = Query(None, description="Maximum news items to return, capped at 100"),
locale: str | None = Query(None, description="Display locale, zh-CN or en-US"),
db: AsyncSession = Depends(get_db),
):
normalized_region = region.strip().lower() if isinstance(region, str) and region.strip() else None
normalized_region = (
region.strip().lower() if isinstance(region, str) and region.strip() else None
)
if normalized_region is not None and normalized_region not in REGION_ANCHORS:
raise HTTPException(
status_code=422,

View File

@@ -3,9 +3,9 @@ from datetime import UTC, datetime
import os
from pathlib import Path
from typing import Optional
from urllib.parse import urlsplit
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
import httpx
from pydantic import BaseModel, EmailStr, Field
from dotenv import dotenv_values
from sqlalchemy import select
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
list_fallback_llm_provider_presets,
refresh_llm_provider_preset,
)
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
@@ -346,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
merged = deepcopy(DEFAULT_SETTINGS[category])
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
defaults = get_fallback_llm_provider_preset(
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
)
else:
defaults = DEFAULT_SETTINGS[category]
merged = deepcopy(defaults)
if payload:
merged.update(payload)
return merged
@@ -383,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
merged = merge_with_defaults(category, payload)
record = await get_setting_record(db, category)
if record is None:
record = SystemSetting(category=category, payload=payload)
@@ -392,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
await db.commit()
await db.refresh(record)
return merge_with_defaults(category, record.payload)
return merged
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
@@ -656,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
)
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
model_provider_apis = {
**(provider_config.get("model_provider_apis") or {}),
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
}
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
model_provider_apis = {}
return {
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": _resolve_service_token(normalized_ai)[0],
@@ -673,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
"api_key": api_key,
"max_tokens": int(provider_config.get("max_tokens") or 1200),
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
"model_provider_apis": provider_config.get("model_provider_apis") or {},
"preset_models": _get_provider_preset(default_provider).get("models") or [],
"model_provider_apis": model_provider_apis,
},
}
@@ -754,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
}
def _join_provider_url(base_url: str, path: str) -> str:
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
def _extract_model_ids(payload: dict) -> list[str]:
data = payload.get("data") if isinstance(payload, dict) else None
if isinstance(data, list):
return [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
]
models = payload.get("models") if isinstance(payload, dict) else None
if isinstance(models, list):
return [
str(item.get("name") or item.get("model") or item.get("id") or item)
for item in models
if item
]
return []
def _contains_model(model_ids: list[str], model: str) -> bool:
normalized_model = model.strip().lower()
return any(str(item).strip().lower() == normalized_model for item in model_ids)
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
provider = _normalize_provider_id(llm_config.get("provider") or "")
configured_api = (
str(llm_config.get("provider_api") or "").strip()
or ProviderApi.OPENAI_COMPLETIONS.value
)
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
model = str(llm_config.get("model") or "").strip()
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
base_url = str(llm_config.get("base_url") or "").strip()
api_key = str(llm_config.get("api_key") or "").strip()
provider_api = configured_api
model_provider_apis = llm_config.get("model_provider_apis")
if isinstance(model_provider_apis, dict):
provider_api = str(model_provider_apis.get(model) or provider_api)
preset_models = [
str(item)
for item in (llm_config.get("preset_models") or [])
if str(item).strip()
]
if not provider or not base_url or not model:
return {
"success": False,
"connected": False,
"message": "当前 provider/base_url/model 未完整配置。",
"mode": "lightweight_config",
}
result = {
"success": False, "connected": False, "mode": "lightweight_models",
"provider": provider, "model": model,
}
if not base_url or not model:
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
return {
"success": False,
"connected": False,
"message": "当前 provider 未配置 API Key。",
"mode": "lightweight_config",
}
if provider == "opencode-go":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
url = _join_provider_url(base_url, "/api/tags")
headers: dict[str, str] = {}
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
url = _join_provider_url(base_url, "/models")
headers = {
"x-api-key": api_key,
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
}
else:
return {
"success": False,
"connected": False,
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
"mode": "lightweight_unsupported",
}
return {**result, "message": "当前 provider 未配置 API Key。"}
try:
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
response = await client.get(url, headers=headers)
response.raise_for_status()
payload = response.json()
except httpx.HTTPStatusError as exc:
detail = exc.response.text or exc.response.reason_phrase
if exc.response.status_code == 404 and _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
"mode": "lightweight_preset",
"provider": provider,
"provider_api": provider_api,
"model": model,
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
"mode": "lightweight_models",
"url": url,
}
catalog = await fetch_model_catalog(
provider, base_url, provider_api, api_key,
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
)
except Exception as exc:
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: {exc}",
"mode": "lightweight_models",
"url": url,
}
model_ids = _extract_model_ids(payload)
if model_ids and not _contains_model(model_ids, model):
if _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
"mode": "lightweight_models_with_preset_alias",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"连接可用,但模型目录中没有当前模型: {model}",
"mode": "lightweight_models",
"provider": provider,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {**result, "message": catalog_error_message(exc)}
result.update({"url": catalog.url, "models_count": len(catalog.models)})
if model.casefold() not in {item.casefold() for item in catalog.models}:
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
model_apis = llm_config.get("model_provider_apis") or {}
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过",
"mode": "lightweight_models",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
**result, "success": True, "connected": True,
"provider_api": model_apis.get(model) or provider_api,
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
}
@@ -1690,7 +1579,8 @@ async def connect_ai_provider_integration(
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.success",
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
else "settings.ai_provider.connect.failed"),
message="AI provider connection test completed",
category="ai",
service="api",
@@ -1700,7 +1590,7 @@ async def connect_ai_provider_integration(
"provider": payload.provider,
"model": payload.model,
"configured": True,
"lightweight_status": lightweight_result.get("status"),
"connected": lightweight_result["connected"],
},
)
return {
@@ -2064,6 +1954,7 @@ async def refresh_ai_provider_preset(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
payload: AIProviderIntegrationUpdate | None = None,
):
try:
provider_id = _normalize_provider_id(provider)
@@ -2071,16 +1962,25 @@ async def refresh_ai_provider_preset(
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
current_payload = await get_setting_payload(db, "external_integrations")
if payload is not None:
if _normalize_provider_id(payload.provider) != provider_id:
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
ai_payload = _build_ai_provider_payload(current_payload, payload)
else:
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
raise HTTPException(
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
)
try:
api_key = None
if provider_id == "opencode-go":
current_payload = await get_setting_payload(db, "external_integrations")
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(
provider_id
)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
refreshed = await refresh_llm_provider_preset(provider_id, api_key=api_key)
refreshed = await refresh_llm_provider_preset(
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
provider_api=provider_config.get("provider_api"),
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
)
except Exception as exc:
logger.warning(
"LLM provider catalog refresh failed",
@@ -2094,7 +1994,7 @@ async def refresh_ai_provider_preset(
)
raise HTTPException(
status_code=502,
detail="模型列表刷新失败,已保留上次模型列表。",
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
) from exc
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))

View File

@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import select, func
from typing import List, Dict, Any, Optional
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
from app.core.collected_data_fields import get_record_field
from app.core.enums import BGPStatus
from app.core.satellite_tle import build_tle_lines_from_elements
from app.core.time import to_iso8601_utc
from app.db.session import get_db
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
@@ -107,6 +109,7 @@ SATELLITE_CACHE_POLICY = EarthLayerCachePolicy(
max_features=25000,
max_bytes=32 * BYTES_PER_MIB,
)
SATELLITE_QUERY_BATCH_SIZE = 500
COMPUTE_CENTER_CACHE_POLICY = EarthLayerCachePolicy(
COMPUTE_CENTER_CACHE_FRESH_SECONDS,
COMPUTE_CENTER_CACHE_STALE_SECONDS,
@@ -337,22 +340,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
return {"type": "FeatureCollection", "features": features}
def _normalize_satellite_constellation_group(
raw_group: Any,
name: Optional[str],
) -> Optional[str]:
normalized_group = str(raw_group or "").strip().lower()
if normalized_group:
return normalized_group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
if constellation_group == "starlink":
@@ -505,16 +492,19 @@ async def _count_current_or_latest_task_data(
async def _load_current_collected_data_by_sources(
db: AsyncSession,
sources: List[str],
*, include_compute_types: bool = False,
) -> Dict[str, List[CollectedData]]:
if not sources:
return {}
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(sources))
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
)
if include_compute_types:
stmt = stmt.where(CollectedData.deleted_at.is_(None))
result = await db.execute(stmt)
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
@@ -644,7 +634,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
return "unknown"
unit = str(capacity_unit or "").strip().lower()
if unit in {"pflop/s", "pflops", "pflop"}:
if unit in {"flop/s", "flops"}:
normalized_tflops = capacity_value / 1e12
elif unit in {"pflop/s", "pflops", "pflop"}:
normalized_tflops = capacity_value * 1000
elif unit in {"gflop/s", "gflops", "gflop"}:
normalized_tflops = capacity_value
@@ -672,15 +664,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
"""
features: List[Dict[str, Any]] = []
unresolved: List[Dict[str, Any]] = []
withheld_count = 0
for record in records:
metadata = record.extra_data or {}
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
withheld_count += 1
continue
result = resolve_compute_center_location_full(record, metadata)
site_type = (
"supercomputer"
if record.source == "top500" or record.data_type == "supercomputer"
else "gpu_cluster"
)
site_type = compute_site_type(record)
if not result.is_resolved:
diagnostic = result.diagnostic or ResolutionDiagnostic(
@@ -706,7 +698,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
latitude = location.latitude
longitude = location.longitude
if site_type == "supercomputer":
if metadata.get("compute_contract") == "v1":
capacity_value = _parse_float(get_record_field(record, "value"))
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
elif site_type == "supercomputer":
capacity_value = _parse_float(get_record_field(record, "rmax"))
capacity_unit = "GFlops"
else:
@@ -767,7 +762,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
}
)
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
VESSEL_TYPE_FILTERS = {
@@ -1866,21 +1861,39 @@ async def _build_satellites_geojson(
limit: int | None,
db: AsyncSession,
) -> dict[str, Any]:
records = await _load_current_or_latest_task_data(
db,
"celestrak_tle",
exclude_unknown_name=True,
limit=limit,
)
features: list[dict[str, Any]] = []
source = "celestrak_tle"
for current_only in (True, False):
if current_only:
stmt = _current_collected_data_stmt(source)
else:
task_id = await _latest_task_id_for_source(db, source, exclude_unknown_name=True)
if task_id is None:
break
stmt = (
select(CollectedData)
.where(CollectedData.source == source, CollectedData.task_id == task_id)
.order_by(CollectedData.id.desc())
)
stmt = stmt.where(CollectedData.name != "Unknown")
if limit is not None:
stmt = stmt.limit(limit)
if not records:
return {"type": "FeatureCollection", "features": [], "count": 0}
# Release each ORM batch while retaining the complete, ordered display payload.
records = await db.stream_scalars(
stmt, execution_options={"yield_per": SATELLITE_QUERY_BATCH_SIZE}
)
found_records = False
try:
async for batch in records.partitions(SATELLITE_QUERY_BATCH_SIZE):
found_records = True
features.extend(convert_satellite_to_geojson(batch)["features"])
finally:
await records.close()
if found_records:
break
geojson = convert_satellite_to_geojson(list(records))
return {
**geojson,
"count": len(geojson.get("features", [])),
}
return {"type": "FeatureCollection", "features": features, "count": len(features)}
@router.get("/geo/supercomputers")
@@ -1955,9 +1968,10 @@ async def _build_compute_centers_geojson(
records_by_source = await _load_current_collected_data_by_sources(
db,
["top500", "epoch_ai_gpu"],
include_compute_types=True,
)
records = _filter_known_records(
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
[record for group in records_by_source.values() for record in group],
)
if limit is not None:
records = records[:limit]
@@ -1972,6 +1986,7 @@ async def _build_compute_centers_geojson(
"total": 0,
"supercomputers": 0,
"gpu_clusters": 0,
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
"unresolved": 0,
},
}
@@ -2014,6 +2029,7 @@ async def _build_compute_centers_geojson(
"count": len(sanitized_features),
"stats": {
"total": len(sanitized_features),
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
"supercomputers": sum(
1 for feature in sanitized_features
if feature.get("properties", {}).get("site_type") == "supercomputer"
@@ -2330,7 +2346,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
stmt = (
select(CollectedData)
.where(CollectedData.source_id == source_id)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.deleted_at.is_(None))
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
.limit(1)
)
@@ -2729,6 +2746,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
)
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
kind = compute_site_type_expression(CollectedData)
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
CollectedData.deleted_at.is_(None))
.group_by(CollectedData.source, kind))).all()
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
present_sources = set()
for source, site_type, count in rows:
present_sources.add(source)
counts[site_type] += int(count or 0)
# Retain the existing last-task fallback for the two collected catalogs.
for source in COMPUTE_SOURCES:
if source not in present_sources:
key = "supercomputer" if source == "top500" else "gpu_cluster"
counts[key] += await _count_current_or_latest_task_data(db, source)
return counts
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
@@ -2737,9 +2773,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"celestrak_tle",
exclude_unknown_name=True,
)
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
compute_center_count = supercomputer_count + gpu_cluster_count
compute_counts = await _compute_facility_counts(db)
supercomputer_count = compute_counts["supercomputer"]
gpu_cluster_count = compute_counts["gpu_cluster"]
compute_center_count = sum(compute_counts.values())
active_incident_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
@@ -2794,6 +2831,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
"supercomputer_count": supercomputer_count,
"gpu_cluster_count": gpu_cluster_count,
"data_center_count": compute_counts["data_center"],
"cloud_center_count": compute_counts["cloud_center"],
"edge_node_count": compute_counts["edge_node"],
"bgp_event_count": active_incident_count or active_anomaly_count,
"bgp_incident_count": active_incident_count,
"bgp_anomaly_count": active_anomaly_count,

View File

@@ -0,0 +1,35 @@
"""Compute facility classification shared by the map and image resolver."""
from sqlalchemy import case, or_
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
def compute_site_type(record) -> str:
metadata = record.extra_data or {}
for value in (metadata.get("site_type"), record.data_type):
if value in COMPUTE_SITE_TYPES:
return value
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
record.source, "data_center"
)
def compute_site_type_expression(record):
declared = record.extra_data["site_type"].as_string()
return case(
(declared.in_(COMPUTE_SITE_TYPES), declared),
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
(record.source == "top500", "supercomputer"),
(record.source == "epoch_ai_gpu", "gpu_cluster"),
else_="data_center",
)
def compute_record_filter(record):
return or_(
record.source.in_(COMPUTE_SOURCES),
record.data_type.in_(COMPUTE_SITE_TYPES),
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
)

View File

@@ -0,0 +1,57 @@
"""Country display ownership for Earth comparisons and cards.
Keep source-country evidence separate; use this projection only for display
grouping so regional source records cannot become competing national keys.
"""
from __future__ import annotations
import json
from functools import lru_cache
from pathlib import Path
from app.core.countries import COUNTRY_ENTRIES, normalize_country
_DISPLAY_ISO_A3 = {
canonical: next(
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
None,
)
for canonical, aliases in COUNTRY_ENTRIES
}
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
@lru_cache(maxsize=1)
def _policy_country_by_code() -> dict[str, str]:
try:
content = _POLICY_PATH.read_text(encoding="utf-8")
except FileNotFoundError as error:
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
policy = json.loads(content)
if policy.get("profile") != "china-pov-v1":
raise ValueError("unsupported Earth display-policy profile")
mappings = {}
for rule in policy["rules"]:
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
if target not in _COUNTRY_BY_ISO_A3:
continue
for code in rule.get("sourceIsoA3", []):
mappings[code] = _COUNTRY_BY_ISO_A3[target]
return mappings
def display_country(value: str | None) -> str | None:
country = normalize_country(value)
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
owner = _policy_country_by_code().get(code)
if owner:
return owner
return country
def display_iso_a3(value: str | None) -> str | None:
country = display_country(value)
return _DISPLAY_ISO_A3.get(country) if country else None

View File

@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
class ProviderApi(StrEnum):
ANTHROPIC_MESSAGES = "anthropic-messages"
OPENAI_COMPLETIONS = "openai-completions"
OPENAI_RESPONSES = "openai-responses"
OLLAMA_GENERATE = "ollama-generate"

View File

@@ -0,0 +1,13 @@
"""Shared source-group normalization; never infer hardware generations."""
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
group = str(raw_group or "").strip().lower()
if group:
return group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None

View File

@@ -6,7 +6,7 @@ import redis
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jose import JWTError, jwt
from sqlalchemy import text
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
return None
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
try:
parsed_id = int(user_id)
except (TypeError, ValueError):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
result = await db.execute(select(User).where(User.id == parsed_id))
user = result.scalar_one_or_none()
if user is None or not user.is_active:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
return user
async def get_current_user(
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
def blacklist_token(token: str) -> None:

View File

@@ -0,0 +1,51 @@
"""SQL objects supporting indexed BGP prefix geography lookup."""
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncConnection
BGP_PREFIX_GEO_FUNCTION_SQL = (
"""
CREATE OR REPLACE FUNCTION planet_bgp_try_inet(value text)
RETURNS inet
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
AS $$
BEGIN
RETURN value::inet;
EXCEPTION WHEN data_exception THEN
RETURN NULL;
END;
$$
""",
"""
CREATE OR REPLACE FUNCTION planet_bgp_try_masklen(value text)
RETURNS integer
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE
AS $$
BEGIN
RETURN masklen(value::cidr);
EXCEPTION WHEN data_exception THEN
RETURN NULL;
END;
$$
""",
)
BGP_PREFIX_GEO_INDEX_NAME = "idx_bgp_prefix_geo_range_current"
BGP_PREFIX_GEO_INDEX_SQL = f"""
CREATE INDEX CONCURRENTLY IF NOT EXISTS {BGP_PREFIX_GEO_INDEX_NAME}
ON collected_data USING gist (
planet_bgp_try_inet(metadata->>'range_start') inet_ops,
planet_bgp_try_inet(metadata->>'range_end') inet_ops
)
WHERE source IN (
'opengeofeed_prefix_geo',
'iptoasn_prefix_geo',
'nro_delegated_prefix_geo'
)
AND COALESCE(is_current, TRUE) = TRUE
"""
async def install_bgp_prefix_geo_functions(connection: AsyncConnection) -> None:
for statement in BGP_PREFIX_GEO_FUNCTION_SQL:
await connection.execute(text(statement))

View File

@@ -6,6 +6,7 @@ from sqlalchemy.orm import declarative_base
from app.core.config import settings
from app.core.logging import get_logger
from app.db.bgp_prefix_geo import install_bgp_prefix_geo_functions
logger = get_logger(__name__)
@@ -219,6 +220,7 @@ async def init_db():
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
await install_bgp_prefix_geo_functions(conn)
users_email_verified_existed = (
await conn.execute(
text(
@@ -299,8 +301,12 @@ async def init_db():
change_event_id BIGINT;
change_payload JSONB;
BEGIN
change_event_id := nextval(
pg_get_serial_sequence('earth_data_change_events', 'id')::regclass
);
change_payload := jsonb_build_object(
'event', 'earth.layer.changed',
'event_id', change_event_id,
'table', change_table,
'operation', change_operation,
'source', change_source,
@@ -311,6 +317,7 @@ async def init_db():
);
INSERT INTO earth_data_change_events (
id,
table_name,
operation,
source,
@@ -318,23 +325,15 @@ async def init_db():
payload,
occurred_at
) VALUES (
change_event_id,
change_table,
change_operation,
change_source,
NULL,
change_payload,
NOW()
)
RETURNING id INTO change_event_id;
change_payload := change_payload || jsonb_build_object(
'event_id', change_event_id
);
UPDATE earth_data_change_events
SET payload = change_payload
WHERE id = change_event_id;
PERFORM pg_notify(
'planet_earth_data_changes',
change_payload::text
@@ -440,6 +439,16 @@ async def init_db():
);
END LOOP;
ELSIF TG_OP = 'UPDATE' THEN
IF TG_TABLE_NAME = 'earth_news_items' AND NOT EXISTS (
SELECT 1
FROM new_rows AS current_row
FULL JOIN old_rows AS previous_row USING (id)
WHERE (to_jsonb(current_row) - 'last_seen_at' - 'updated_at')
IS DISTINCT FROM
(to_jsonb(previous_row) - 'last_seen_at' - 'updated_at')
) THEN
RETURN NULL;
END IF;
FOR change_source IN
SELECT DISTINCT COALESCE(NULLIF(row_data->>'source', ''), TG_TABLE_NAME)
FROM (

View File

@@ -0,0 +1,128 @@
"""Typed, validated configuration for optional object-card images."""
from typing import Literal, Self
from urllib.parse import urlsplit
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
MatchScope = Literal["object", "model", "group", "category"]
def safe_url(value: str, *, local: bool = False) -> str:
if not value:
return value
if any(ord(char) < 32 for char in value) or "\\" in value:
raise ValueError("Invalid URL")
parts = urlsplit(value)
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
if (
".." not in parts.path
and "%" not in parts.path
and not parts.netloc
and not parts.query
and not parts.fragment
):
return value
if (
parts.scheme in {"http", "https"}
and parts.hostname
and not parts.username
and not parts.password
):
return value
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
class CardImageAsset(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
title: str = Field(min_length=1, max_length=200)
url: str = Field(min_length=1, max_length=2000)
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
width: int = Field(default=0, ge=0, le=20000)
height: int = Field(default=0, ge=0, le=20000)
credit: str = Field(default="", max_length=500)
source_url: str = Field(default="", max_length=2000)
license: str = Field(default="", max_length=300)
license_url: str = Field(default="", max_length=2000)
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
rights_status: Literal["available", "permission_pending"] = "available"
enabled: bool = True
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
@field_validator("url")
@classmethod
def validate_image_url(cls, value: str) -> str:
return safe_url(value, local=True)
@field_validator("source_url", "license_url")
@classmethod
def validate_source_url(cls, value: str) -> str:
return safe_url(value)
@field_validator("variants")
@classmethod
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
for width, url in value.items():
if width not in {"640", "1280"}:
raise ValueError("Unsupported image variant")
safe_url(url, local=True)
return value
class CardImageRule(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
entity_type: EntityType
scope: MatchScope
value: str = Field(min_length=1, max_length=300)
label: str = Field(default="", max_length=500)
asset_id: str | None = Field(default=None, max_length=100)
suppress: bool = False
enabled: bool = True
@model_validator(mode="after")
def validate_action(self) -> Self:
if self.scope != "object":
self.value = self.value.casefold()
if self.suppress and (self.scope != "object" or self.asset_id):
raise ValueError("Only an object rule may suppress its image, without an asset")
if not self.suppress and not self.asset_id:
raise ValueError("Choose an image or suppress this object")
return self
def target(self) -> tuple[str, str, str]:
return self.entity_type, self.scope, self.value
class CardImageConfiguration(BaseModel):
model_config = ConfigDict(extra="forbid")
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
@model_validator(mode="after")
def validate_relations(self) -> Self:
assets = {asset.id for asset in self.assets}
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
self.rules
):
raise ValueError("Image and rule IDs must be unique")
targets = [rule.target() for rule in self.rules]
if len(set(targets)) != len(targets):
raise ValueError("Each target may have only one rule")
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
raise ValueError("Remove image associations before deleting the image")
return self
class CardImagePublish(BaseModel):
revision: str = Field(max_length=200)
config: CardImageConfiguration
class CardImagePreview(BaseModel):
entity_type: EntityType
entity_id: str = Field(min_length=1, max_length=300)
config: CardImageConfiguration

View File

@@ -0,0 +1,94 @@
"""Validated, source-backed observations for the compute comparison cards."""
from datetime import date
from typing import Annotated, Literal, Self
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
Text = Annotated[str, Field(min_length=1, max_length=500)]
class ConfigurationModel(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
class ComputeSource(ConfigurationModel):
id: Identifier
title: Text
url: HttpUrl
publisher: Text
report_year: Year
published_at: date | None = None
locator: Text
@field_validator("url")
@classmethod
def public_reference(cls, value: HttpUrl) -> HttpUrl:
if value.username or value.password or len(str(value)) > 2000:
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
return value
class ComputeObservation(ConfigurationModel):
year: Year
metric: Literal["global_share", "absolute"]
source_id: Identifier
values: dict[str, Value] = Field(min_length=1, max_length=250)
@model_validator(mode="after")
def comparable_values(self) -> Self:
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
raise ValueError(
"Use countries from the canonical compute catalog, not regional totals"
)
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
raise ValueError("Global shares for one year must total at most 100 percent")
return self
class InnovationObservation(ConfigurationModel):
year: Year
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
source_id: Identifier
class ComputeConfiguration(ConfigurationModel):
reviewed_at: date
sources: list[ComputeSource] = Field(max_length=200)
observations: list[ComputeObservation] = Field(max_length=202)
innovation: list[InnovationObservation] = Field(max_length=101)
@model_validator(mode="after")
def consistent_evidence(self) -> Self:
ids = [source.id for source in self.sources]
if len(ids) != len(set(ids)):
raise ValueError("Report source IDs must be unique")
keys = [(row.metric, row.year) for row in self.observations]
if len(keys) != len(set(keys)):
raise ValueError("Each metric/year may have only one observation")
years = [row.year for row in self.innovation]
if len(years) != len(set(years)):
raise ValueError("Innovation observation years must be unique")
sources = {source.id: source for source in self.sources}
for row in [*self.observations, *self.innovation]:
source = sources.get(row.source_id)
if source is None:
raise ValueError(f"Missing report source: {row.source_id}")
if row.year > source.report_year:
raise ValueError("Observation year must not be after its report year")
if self.reviewed_at > date.today():
raise ValueError("Review date must not be in the future")
return self
class ComputeConfigurationWrite(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)
config: ComputeConfiguration
class ComputeConfigurationReset(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)

View File

@@ -36,6 +36,9 @@ class UserInDB(UserBase):
class UserResponse(UserBase):
# Bootstrap accounts have reserved .local email addresses. Validate email
# on writes, while allowing existing local identities to be serialized.
email: str
id: int
role: str
gatekeeper_groups: list[str] = Field(default_factory=list)

View File

@@ -111,15 +111,24 @@ async def _lookup_prefix_geography(
result = await db.execute(
text(
"""
WITH candidates AS MATERIALIZED (
SELECT source, id, metadata
FROM collected_data
WHERE source IN (
'opengeofeed_prefix_geo',
'iptoasn_prefix_geo',
'nro_delegated_prefix_geo'
)
AND COALESCE(is_current, TRUE) = TRUE
AND planet_bgp_try_inet(metadata->>'range_start') <= CAST(:range_start AS inet)
AND planet_bgp_try_inet(metadata->>'range_end') >= CAST(:range_end AS inet)
)
SELECT metadata
FROM collected_data
FROM candidates
WHERE source = :source
AND COALESCE(is_current, TRUE) = TRUE
AND metadata->>'family' = :family
AND CAST(metadata->>'range_start' AS inet) <= CAST(:range_start AS inet)
AND CAST(metadata->>'range_end' AS inet) >= CAST(:range_end AS inet)
ORDER BY
masklen(CAST(metadata->>'prefix' AS cidr)) DESC NULLS LAST,
planet_bgp_try_masklen(metadata->>'prefix') DESC NULLS LAST,
id DESC
LIMIT 1
"""
@@ -146,6 +155,10 @@ async def _lookup_prefix_geography(
results: dict[str, dict[str, Any]] = {}
# The GiST range predicate is selective, but PostgreSQL 15 greatly overestimates its cost.
# Avoid spending more time compiling JIT code than fetching the small candidate set.
await db.execute(text("SET LOCAL jit = off"))
for prefix in prefix_values:
try:
network = ipaddress.ip_network(prefix, strict=False)

View File

@@ -0,0 +1,174 @@
"""Revision-checked image settings; new bundled assets never replace user edits."""
from uuid import uuid4
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.card_images import CardImageConfiguration
from app.services.card_image_defaults import (
bundled_configuration,
previous_bundled_assets,
previous_bundled_rules,
)
CATEGORY = "earth_card_images"
def effective_envelope(payload: dict | None) -> dict:
defaults, version = bundled_configuration()
config = (
CardImageConfiguration.model_validate(payload["config"])
if payload
else defaults.model_copy(deep=True)
)
if payload:
previous = previous_bundled_assets()
latest = {asset.id: asset for asset in defaults.assets}
for index, asset in enumerate(config.assets):
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
# Replace the old bundled source while preserving fields the user
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
if old is not None and asset.url == old.url and asset.id in latest:
overrides = {
key: value
for key, value in asset.model_dump().items()
if value != getattr(old, key)
}
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
known_assets = set(payload.get("known_assets", []))
asset_ids = {asset.id for asset in config.assets}
config.assets.extend(
asset.model_copy(deep=True)
for asset in defaults.assets
if asset.id not in known_assets and asset.id not in asset_ids
)
prior_rules = previous_bundled_rules()
latest_rules = {rule.id: rule for rule in defaults.rules}
for index, rule in enumerate(config.rules):
old_rule = prior_rules.get(rule.id)
if (
old_rule
and rule.id in latest_rules
and rule.target() == old_rule.target()
and rule.asset_id == old_rule.asset_id
):
overrides = {
key: value
for key, value in rule.model_dump().items()
if value != getattr(old_rule, key)
}
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
known_rules = set(payload.get("known_rules", []))
rule_ids = {rule.id for rule in config.rules}
targets = {rule.target() for rule in config.rules}
available = {asset.id for asset in config.assets}
config.rules.extend(
rule.model_copy(deep=True)
for rule in defaults.rules
if rule.id not in known_rules
and rule.id not in rule_ids
and rule.target() not in targets
and rule.asset_id in available
)
return {
"config": config.model_dump(mode="json"),
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
"is_default": payload is None,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return effective_envelope(record.payload if record else None)
async def publish_configuration(
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
) -> dict:
defaults, _ = bundled_configuration()
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
if any(
asset.id in pending and asset.rights_status != "permission_pending"
for asset in config.assets
):
raise HTTPException(422, "Bundled image permission is still unverified")
initial = {
"config": defaults.model_dump(mode="json"),
"revision": "default",
"known_assets": [],
"known_rules": [],
}
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=initial)
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if effective_envelope(record.payload)["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
before = record.payload
record.payload = {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="card_images.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": record.payload},
)
)
await db.commit()
return effective_envelope(record.payload)
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
assets = {asset.id: asset for asset in config.assets}
for scope in ("object", "model", "group", "category"):
value = context.get("entity_id" if scope == "object" else scope)
values = [value] if value else []
if scope == "category" and value != "*":
values.append("*")
for value in values:
value = value if scope == "object" else value.casefold()
rule = next(
(
rule
for rule in config.rules
if rule.enabled
and rule.entity_type == context["entity_type"]
and rule.scope == scope
and rule.value == value
),
None,
)
if not rule:
continue
match = {"scope": scope, "value": value, "rule_id": rule.id}
if rule.suppress:
return {"image": None, "match": match, "suppressed": True}
asset = assets.get(rule.asset_id)
if asset and asset.enabled:
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
return {"image": None, "match": None, "suppressed": False}

View File

@@ -0,0 +1,108 @@
"""Read immutable bundled assets without inserting machine-specific database rows."""
import hashlib
import json
from functools import lru_cache
from pathlib import Path
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
REPO_ROOT = Path(__file__).resolve().parents[3]
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
def _asset_from_manifest(item: dict) -> CardImageAsset:
pending = item.get("redistribution_status") == "permission_not_verified"
return CardImageAsset(
id=item["id"],
title=item["label"],
url=item["url"],
width=item["width"],
height=item["height"],
credit=item["credit"],
source_url=item.get("source_page") or "",
license=item["license"],
license_url=item.get("license_url") or "",
kind=item["kind"],
rights_status="permission_pending" if pending else "available",
enabled=item.get("enabled", not pending),
focus_x=item.get("focus_x", 50),
focus_y=item.get("focus_y", 50),
variants=item.get("variants", {}),
)
@lru_cache(maxsize=1)
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: [CardImageAsset.model_validate(item) for item in value]
for key, value in manifest.get("previous_defaults", {}).items()
}
@lru_cache(maxsize=1)
def previous_bundled_rules() -> dict[str, CardImageRule]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: CardImageRule.model_validate(value)
for key, value in manifest.get("previous_rules", {}).items()
}
@lru_cache(maxsize=1)
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
content = MANIFEST_PATH.read_bytes()
manifest = json.loads(content)
assets, rules = [], []
for item in manifest["assets"]:
asset = _asset_from_manifest(item)
assets.append(asset)
if not asset.enabled:
continue
hints = item.get("binding_hints", {})
for index, target in enumerate(hints.get("defaults", [])):
rules.append(
CardImageRule(
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
entity_type=target["entity_type"],
scope=target["scope"],
value=target["value"],
label=asset.title,
asset_id=asset.id,
)
)
for index, entity_id in enumerate(hints.get("entity_ids", [])):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-entity-{index}",
entity_type=item["entity_type"],
scope="object",
value=entity_id,
label=asset.title,
asset_id=asset.id,
)
)
for norad_id in hints.get("norad_ids", []):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-{norad_id}",
entity_type="satellite",
scope="object",
value=str(norad_id),
label=f"{asset.title} · NORAD {norad_id}",
asset_id=asset.id,
)
)
if group := hints.get("constellation_group"):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-group",
entity_type="satellite",
scope="group",
value=group,
label=asset.title,
asset_id=asset.id,
)
)
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()

View File

@@ -0,0 +1,95 @@
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
import hashlib
import io
import re
import warnings
from uuid import uuid4
from pathlib import Path
import os
from fastapi import HTTPException
from PIL import Image, ImageOps, UnidentifiedImageError
from app.schemas.card_images import CardImageAsset
from app.services.card_image_defaults import REPO_ROOT
ASSET_DIR = Path(
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
).resolve()
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
MAX_PIXELS = 32_000_000
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
def uploaded_path(filename: str) -> Path:
if not FILE_PATTERN.fullmatch(filename):
raise HTTPException(404, "Image not found")
return ASSET_DIR / filename
def store_image(content: bytes, title: str) -> CardImageAsset:
if len(content) > MAX_UPLOAD_BYTES:
raise HTTPException(413, "Image must be 12 MiB or smaller")
try:
with warnings.catch_warnings():
warnings.simplefilter("error", Image.DecompressionBombWarning)
image = Image.open(io.BytesIO(content))
extension = FORMATS.get(image.format)
if (
not extension
or image.width * image.height > MAX_PIXELS
or getattr(image, "is_animated", False)
):
raise ValueError("Unsupported or oversized raster")
image.load()
image = ImageOps.exif_transpose(image).convert(
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
)
except (
UnidentifiedImageError,
OSError,
ValueError,
Image.DecompressionBombError,
Image.DecompressionBombWarning,
) as error:
raise HTTPException(
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
) from error
digest = hashlib.sha256(content).hexdigest()
ASSET_DIR.mkdir(parents=True, exist_ok=True)
original = f"{digest}-original.{extension}"
original_path = uploaded_path(original)
if not original_path.exists():
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(original_path)
variants = {}
for size, content in render_variants(image).items():
filename = f"{digest}-{size}.webp"
path = uploaded_path(filename)
if not path.exists():
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(path)
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
return CardImageAsset(
id=f"upload-{digest[:24]}",
title=title or "Image",
url=f"/api/v1/earth/card-images/files/{original}",
variants=variants,
width=image.width,
height=image.height,
)
def render_variants(image: Image.Image) -> dict[str, bytes]:
result = {}
for size in (640, 1280):
resized = image.copy()
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
buffer = io.BytesIO()
resized.save(buffer, format="WEBP", quality=93, method=4)
result[str(size)] = buffer.getvalue()
return result

View File

@@ -0,0 +1,180 @@
"""Bounded catalog queries and type defaults for every Earth object card."""
from sqlalchemy import String, case, cast, func, literal, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import (
COMPUTE_SITE_TYPES,
compute_site_type_expression,
compute_record_filter,
)
from app.models.bgp_incident import BGPIncident
from app.models.bgp_collector_location import BGPCollectorLocation
from app.models.collected_data import CollectedData
from app.models.earth_interactable import EarthInteractable
from app.models.vessel import VesselCurrentState
from app.schemas.card_images import EntityType, MatchScope
SOURCES = {
"satellite": ("celestrak_tle", "spacetrack_tle"),
"compute_center": ("top500", "epoch_ai_gpu"),
"cable": ("arcgis_cables",),
"landing_point": ("arcgis_landing_points",),
}
# Named constellation families, not inferred hardware models or generations.
SATELLITE_GROUP_PATTERNS = (
("STARLINK%", "starlink"),
("IRIDIUM%", "iridium-next"),
("BEIDOU-3%", "beidou-3"),
("BEIDOU%", "beidou"),
("QIANFAN%", "qianfan"),
("ONEWEB%", "oneweb"),
("KUIPER%", "kuiper"),
)
def collected_query(entity_type: EntityType):
record, metadata = CollectedData, CollectedData.extra_data
model = func.coalesce(
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
)
category = (
compute_site_type_expression(record)
if entity_type == "compute_center"
else literal(entity_type)
)
if entity_type == "satellite":
group = func.coalesce(
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
case(
*[
(func.upper(record.name).like(pattern), value)
for pattern, value in SATELLITE_GROUP_PATTERNS
],
else_="",
),
)
entity_id = metadata["norad_cat_id"].as_string()
else:
group = func.coalesce(
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
)
entity_id = record.source + ":" + record.source_id
return select(
record.name.label("name"),
cast(entity_id, String).label("entity_id"),
model.label("model"),
group.label("group"),
category.label("category"),
).where(
(
compute_record_filter(record)
if entity_type == "compute_center"
else record.source.in_(SOURCES[entity_type])
),
record.is_current.is_(True),
record.deleted_at.is_(None),
metadata["anonymous"].as_boolean().is_not(True),
)
def target_query(entity_type: EntityType):
if entity_type in SOURCES:
return collected_query(entity_type)
if entity_type == "vessel":
model = VesselCurrentState
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
conditions = []
elif entity_type == "bgp":
model = BGPIncident
identity, name, group, category = (
model.incident_key,
model.title,
model.source,
model.incident_type,
)
conditions = []
elif entity_type == "bgp_collector":
model = BGPCollectorLocation
identity, name, group, category = (
model.collector_id,
model.collector_id,
model.operator,
literal("bgp_collector"),
)
conditions = []
else:
model = EarthInteractable
identity, name, group, category = model.id, model.label, model.layer, model.kind
conditions = [model.is_deleted.is_(False)]
return select(
name.label("name"),
cast(identity, String).label("entity_id"),
literal("").label("model"),
func.coalesce(group, "").label("group"),
func.coalesce(category, entity_type).label("category"),
).where(*conditions)
def category_context(entity_type: EntityType) -> dict[str, str]:
return {
"entity_type": entity_type,
"entity_id": "",
"name": "",
"model": "",
"group": "",
"category": entity_type,
}
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
category = entity_id.partition(":")[2]
if category in COMPUTE_SITE_TYPES:
return {**category_context(entity_type), "category": category}
if entity_id == "__category__":
return category_context(entity_type)
if entity_type in SOURCES and entity_type != "satellite":
source, separator, source_id = entity_id.partition(":")
if not separator or not source or not source_id:
return category_context(entity_type)
# Match the indexed identity columns before projecting JSON metadata.
statement = collected_query(entity_type).where(
CollectedData.source == source, CollectedData.source_id == source_id
)
else:
catalog = target_query(entity_type).subquery()
statement = select(catalog).where(catalog.c.entity_id == entity_id)
row = (await db.execute(statement.limit(1))).mappings().first()
if not row:
# Old map snapshots can outlive a catalog refresh. They may use the type
# illustration, but must not claim an unverified exact-object identity.
return category_context(entity_type)
context = {key: str(value or "") for key, value in row.items()}
return {**context, "entity_type": entity_type}
async def search_targets(
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
) -> list[dict[str, str]]:
catalog = target_query(entity_type).subquery()
value = catalog.c["entity_id" if scope == "object" else scope]
label = catalog.c.name if scope == "object" else value
statement = select(value.label("value"), label.label("label")).where(
value.is_not(None), value != ""
)
if query:
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
statement = statement.where(
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
)
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
if scope == "category" and not query:
known = {item["value"] for item in items}
if entity_type == "compute_center":
items.extend(
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
)
items.insert(0, {"value": "*", "label": "*"})
return items

View File

@@ -31,6 +31,7 @@ class BaseCollector(ABC):
frequency_hours: int = 4
data_type: str = "generic"
fail_on_empty: bool = False
preserve_snapshot_on_failure: bool = False
def __init__(self):
self._current_task = None
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
if self.fail_on_empty and not data:
raise RuntimeError(f"Collector {self.name} produced no valid records")
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
raise
except Exception as e:
await db.rollback()
if snapshot_id is not None and self.preserve_snapshot_on_failure:
await self._rollback_incomplete_run(
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
)
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.phase_message = str(e)
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.status = SnapshotStatus.FAILED.value
if self.preserve_snapshot_on_failure:
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {"error": str(e)}
await db.commit()

View File

@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
from app.services.collectors.base import BaseCollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
logger = get_logger(__name__, service="collector")
ACTIVE_GROUP = "active"
FALLBACK_GROUPS = (
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
)
FETCH_RETRY_ATTEMPTS = 3
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
_downloader = ResumableFileDownloader(
cache_namespace="celestrak",
default_accept="application/json",
preflight_head=False,
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
)
@property
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
if not is_final_attempt:
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
raise RuntimeError(
f"CelesTrak active satellite download failed after retries: {last_error}"
)
async def _fetch_fallback_groups(
self,
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
try:
for group in FALLBACK_GROUPS:
group_url = self._group_url(group)
cached_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is not None:
body_path = cached_path
else:
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(
f"CelesTrak fallback group '{group}' download failed: {exc}"
) from exc
body_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if body_path is None:
raise RuntimeError(
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
) from exc
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
commit=True,
)
@staticmethod
def _validate_json_file(path: Path) -> bool:
@classmethod
def _validate_json_file(cls, path: Path) -> bool:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
cls._load_active_payload(path)
except RuntimeError:
return False
return isinstance(data, list)
return True
async def _log_parse_failure(self, exc: Exception) -> None:
await emit_business_log(
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
),
)
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
@staticmethod
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
else:
invalid_count += 1
if invalid_count:
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
raise RuntimeError(
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
)
if not records:
raise RuntimeError("CelesTrak active payload contains no satellite records")
return records

View File

@@ -0,0 +1,82 @@
"""Quality gates shared by the two compute sources."""
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import normalize_country
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.models.data_snapshot import DataSnapshot
from app.services.collectors.base import BaseCollector
class ComputeCollector(BaseCollector):
fail_on_empty = True
preserve_snapshot_on_failure = True
async def _create_snapshot(
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
) -> int:
previous_count = await db.scalar(
select(DataSnapshot.record_count)
.where(
DataSnapshot.source == self.name,
DataSnapshot.is_current.is_(True),
DataSnapshot.status == "success",
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
if previous_count and len(data) < previous_count * 0.6:
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
return await super()._create_snapshot(db, task_id, data, started_at)
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
if self.name == "top500":
await self._copy_verified_locations(db, data)
return await super()._save_data(db, data, **kwargs)
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
"""Keep historical location keys; copy only unambiguous exact identities.
Never migrate by rank: the machine occupying rank 1 changes over time.
"""
names = [item["name"] for item in data]
locations = (
(
await db.execute(
select(ComputeCenterLocationRecord).where(
ComputeCenterLocationRecord.source == self.name,
ComputeCenterLocationRecord.name.in_(names),
)
)
)
.scalars()
.all()
)
existing = {location.source_id for location in locations}
for item in data:
if item["source_id"] in existing:
continue
country = normalize_country(item["country"])
if not country:
continue
matches = [
location
for location in locations
if location.name == item["name"]
and normalize_country(location.country) == country
and (not location.site or location.site == item.get("metadata", {}).get("site"))
and not location.source_id.startswith("top500_system_")
]
if len(matches) != 1:
continue
old = matches[0]
values = {
column.name: getattr(old, column.name)
for column in old.__table__.columns
if column.name not in {"id", "source_id", "created_at", "updated_at"}
}
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
existing.add(item["source_id"])

View File

@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
import httpx
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
ValidateCallback = Callable[[Path], bool]
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
default_accept: str = "*/*",
cache_root: Path | None = None,
preflight_head: bool = True,
cache_ttl_seconds: float = 0,
) -> None:
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
self._user_agent = user_agent
self._default_accept = default_accept
self._preflight_head = preflight_head
self._cache_ttl_seconds = cache_ttl_seconds
@staticmethod
def _cache_key(url: str) -> str:
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
if not meta_path.exists():
return {}
try:
return json.loads(meta_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
data = json.loads(meta_path.read_text(encoding="utf-8"))
return data if isinstance(data, dict) else {}
except json.JSONDecodeError, OSError:
return {}
@staticmethod
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
return last_modified == str(meta.get("last_modified") or "").strip()
return True
@staticmethod
def _response_info(response: httpx.Response) -> dict[str, Any]:
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
raw_length = response.headers.get("content-length")
encoded = response.headers.get("content-encoding", "identity") != "identity"
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": int(raw_length) if raw_length and not encoded else None,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
try:
response = await client.head(url)
if response.status_code >= 400:
return {}
content_length_raw = response.headers.get("content-length")
content_length = int(content_length_raw) if content_length_raw else None
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": content_length,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
except (httpx.HTTPError, ValueError):
return self._response_info(response)
except httpx.HTTPError, ValueError:
return {}
async def download_file(
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
) -> Path:
self._cache_dir.mkdir(parents=True, exist_ok=True)
final_path, part_path, meta_path = self._cache_paths(url, extension)
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
size = cached_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return cached_path
meta = self._load_meta(meta_path)
remote = await self.fetch_remote_info(client, url)
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
expected_size = remote.get("content_length")
if final_path.exists():
local_size = final_path.stat().st_size
size_match = expected_size is None or local_size == expected_size
if self._validators_match(meta, remote) and size_match:
if (
(remote.get("etag") or remote.get("last_modified"))
and self._validators_match(meta, remote)
and size_match
):
if validate_existing and not validate_existing(final_path):
final_path.unlink(missing_ok=True)
else:
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
headers["If-Range"] = str(remote.get("last_modified"))
async with client.stream("GET", url, headers=headers) as response:
if response.status_code == 304 and final_path.exists():
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)
if response.status_code == 304:
if cached_path is None:
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
final_path.touch()
size = final_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return final_path
if response.status_code >= 400:
body = (await response.aread()).decode("utf-8", errors="replace")
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
else:
mode = "wb"
resume_from = 0
remote = self._response_info(response)
expected_size = remote.get("content_length")
downloaded = resume_from
last_emit_bytes = 0
last_emit_time = time.monotonic()
min_emit_bytes = (
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
max(expected_size // 150, 512 * 1024)
if expected_size and expected_size > 0
else 1024 * 1024
)
with part_path.open(mode) as f:
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
)
# Never replace the last good payload with an HTML error or invalid JSON.
if validate_existing and not validate_existing(part_path):
part_path.unlink(missing_ok=True)
raise RuntimeError(f"Downloaded file validation failed for {url}")
part_path.replace(final_path)
self._save_meta(
meta_path,
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
},
)
if validate_existing and not validate_existing(final_path):
raise RuntimeError(f"Downloaded file validation failed for {url}")
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)

View File

@@ -1,119 +1,135 @@
"""Epoch AI GPU Clusters Collector
Collects data from Epoch AI GPU clusters tracking.
https://epoch.ai/data/gpu-clusters
"""
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
import csv
import hashlib
import io
import json
import math
import re
from typing import Dict, Any, List
from datetime import UTC, datetime
from bs4 import BeautifulSoup
from urllib.parse import urlparse
import httpx
from app.services.collectors.base import BaseCollector
from app.core.countries import normalize_country
from app.services.collectors.compute_base import ComputeCollector
from app.services.epoch_location import assess_epoch_location
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
class EpochAIGPUCollector(BaseCollector):
class EpochAIGPUCollector(ComputeCollector):
name = "epoch_ai_gpu"
priority = "P0"
module = "L1"
frequency_hours = 6
data_type = "gpu_cluster"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch Epoch AI GPU clusters data from webpage"""
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
async def fetch(self) -> list[dict]:
configured = self._resolved_url or EPOCH_CSV_URL
parsed = urlparse(configured)
url = (
EPOCH_CSV_URL
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
else configured
)
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.text)
return self.parse_response(response.text)
def parse_response(self, html: str) -> List[Dict[str, Any]]:
"""Parse Epoch AI webpage to extract GPU cluster data"""
data = []
soup = BeautifulSoup(html, "html.parser")
def parse_response(self, text: str) -> list[dict]:
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
if not required.issubset(reader.fieldnames or []):
raise ValueError(
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
)
entries = []
seen = set()
anonymous_counts = {}
for row in reader:
name = (row.get("Name") or "").strip()
country = normalize_country(row.get("Country"))
if not name:
raise ValueError("Epoch row has missing identity")
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
anonymous = "anonym" in name.lower()
if anonymous:
# Public anonymous rows have no facility identity. Preserve
# multiplicity within snapshots, without guessing names.
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
if source_id in seen:
raise ValueError("Epoch identity collision; previous snapshot retained")
seen.add(source_id)
raw_date = row.get("First Operational Date", "")
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
value = None
raw_value = row.get("16-bit OP/s (log)", "")
if raw_value:
exponent = float(raw_value)
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
raise ValueError("Epoch performance exponent out of bounds")
value = 10**exponent
anonymous = anonymous or (country == "中国" and not row.get("Location"))
location = row.get("Location")
location_assessment = assess_epoch_location(
location,
country=country,
raw_country=row.get("Country"),
)
entries.append(
{
"source_id": source_id,
"name": name,
"country": country,
"value": value,
"unit": "FLOP/s",
"reference_date": reference_date,
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
"metadata": {
"compute_contract": "v1",
"precision": "FP16",
"measurement": "theoretical",
"source_url": EPOCH_CSV_URL,
"raw_reference_date": raw_date,
"status": row.get("Status"),
"certainty": row.get("Certainty"),
"operator": row.get("Owner"),
"gpu_type": row.get("Chip type (primary)"),
"gpu_count": row.get("Chip quantity (primary)"),
"location": location,
"location_precision": location_assessment.precision,
"location_precision_reason": location_assessment.reason,
"raw_country": row.get("Country"),
"raw_performance": raw_value,
"performance_encoding": "log10",
"builds_upon": row.get("Builds Upon"),
"superseded_by": row.get("Superseded by"),
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
"include_in_standard_analysis": row.get(
"Include in Standard Analysis", ""
).lower()
== "true",
"rounded": country == "中国",
"anonymous": anonymous,
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
"country_unresolved": country is None,
"sources": [
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
],
},
}
)
return entries
# Try to find data table on the page
tables = soup.find_all("table")
for table in tables:
rows = table.find_all("tr")
for row in rows[1:]: # Skip header
cells = row.find_all(["td", "th"])
if len(cells) >= 5:
try:
cluster_name = cells[0].get_text(strip=True)
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
continue
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
country, city = self._parse_location(location_cell)
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
entry = {
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
"name": cluster_name,
"country": country,
"city": city,
"latitude": "",
"longitude": "",
"value": self._parse_performance(perf_cell),
"unit": "TFlop/s",
"metadata": {
"raw_data": perf_cell,
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}
data.append(entry)
except (ValueError, IndexError, AttributeError):
continue
# If no table found, return sample data
if not data:
data = self._get_sample_data()
return data
def _parse_location(self, location: str) -> tuple:
"""Parse location string into country and city"""
if not location:
return "", ""
if "," in location:
parts = location.rsplit(",", 1)
city = parts[0].strip()
country = parts[1].strip() if len(parts) > 1 else ""
return country, city
return location, ""
def _parse_performance(self, perf: str) -> str:
"""Parse performance string to extract value"""
if not perf:
return "0"
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
if match:
return match.group(1).replace(",", "")
match = re.search(r"([\d,.]+)", perf)
if match:
return match.group(1).replace(",", "")
return "0"
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "epoch_sample_1",
"name": "Sample GPU Cluster",
"country": "United States",
"city": "San Francisco, CA",
"latitude": "",
"longitude": "",
"value": "1000",
"unit": "TFlop/s",
"metadata": {
"note": "Sample data - Epoch AI page structure may vary",
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
},
]
@staticmethod
def coordinate(value: str | None, bound: int) -> float | None:
if not value:
return None
number = float(value)
if not math.isfinite(number) or abs(number) > bound:
raise ValueError("Epoch coordinate out of bounds")
return number

View File

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
"""
import asyncio
import math
import re
from urllib.parse import urljoin
from typing import Dict, Any, List
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
from app.services.collectors.compute_base import ComputeCollector
class TOP500Collector(BaseCollector):
class TOP500Collector(ComputeCollector):
name = "top500"
priority = "P0"
module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
entries = self.parse_response(response.text)
entries = self.parse_response(response.text, str(response.url))
semaphore = asyncio.Semaphore(8)
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
detail_url = ""
if link and link.get("href"):
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
detail_url = urljoin(str(top500_base_url), link.get("href"))
manufacturer = ""
if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
return detail_map
def parse_response(self, html: str) -> List[Dict[str, Any]]:
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
"""Parse TOP500 HTML response"""
data = []
soup = BeautifulSoup(html, "html.parser")
canonical = soup.find("link", rel="canonical")
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
# Find the table with TOP500 data
table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
if table:
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
if not reference_date:
raise ValueError("TOP500 list date missing; previous snapshot retained")
rows = table.select("tr")
for row in rows[1:]: # Skip header row
cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
cores = cells[2].get_text(strip=True).replace(",", "")
rmax_text = cells[3].get_text(strip=True)
rmax = self._parse_performance(rmax_text)
rmax = self._parse_performance(rmax_text, headers[3])
rpeak_text = cells[4].get_text(strip=True)
rpeak = self._parse_performance(rpeak_text)
rpeak = self._parse_performance(rpeak_text, headers[4])
power = ""
if len(cells) >= 6:
power = cells[5].get_text(strip=True).replace(",", "")
identity = re.search(r"/system/(\d+)", detail_url)
if not identity:
raise ValueError("TOP500 stable system ID missing")
entry = {
"source_id": f"top500_{rank}",
"source_id": f"top500_system_{identity[1]}",
"name": system_name,
"country": country,
"city": city,
"latitude": 0.0,
"longitude": 0.0,
"latitude": None,
"longitude": None,
"value": str(rmax),
"unit": "PFlop/s",
"unit": "FLOP/s",
"metadata": {
"compute_contract": "v1",
"precision": "FP64", "measurement": "LINPACK Rmax",
"source_url": detail_url, "list_url": date_url,
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
"rank": rank,
"cores": cores,
"rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
"site": site,
},
"_detail_url": detail_url,
"reference_date": "2025-11-01",
"reference_date": reference_date,
}
data.append(entry)
except (ValueError, IndexError, AttributeError) as e:
continue
# If scraping failed, return sample data for testing
if not data:
data = self._get_sample_data()
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
return data
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
return 0.0
return 0.0
def _parse_performance(self, text: str) -> float:
def _parse_performance(self, text: str, header: str = "") -> float:
"""Parse performance value from text (handles E, P, T suffixes)"""
text = text.strip().upper()
text = text.strip().upper().replace(",", "")
if not re.search(r"[EPTGMK]F", text):
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
if not unit:
raise ValueError("TOP500 performance unit missing")
text = f"{text} {unit[1]}FLOP/S"
multipliers = {
"E": 1e18,
"P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
# Try simple float parsing
try:
return float(text.replace(",", ""))
value = float(text.replace(",", ""))
if not math.isfinite(value) or value < 0:
raise ValueError("Invalid TOP500 performance")
return value
except ValueError:
return 0.0
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "top500_1",
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
"country": "United States",
"city": "Livermore, CA",
"latitude": 37.6819,
"longitude": -121.7681,
"value": "1742.00",
"unit": "PFlop/s",
"metadata": {
"rank": 1,
"cores": "11039616",
"rmax": "1742.00",
"rpeak": "2746.38",
"power": "29581",
"manufacturer": "HPE",
"site": "DOE/NNSA/LLNL",
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
"interconnect": "Slingshot-11",
"installation_year": "2025",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_2",
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
"country": "United States",
"city": "Oak Ridge, TN",
"latitude": 36.0107,
"longitude": -84.2663,
"value": "1353.00",
"unit": "PFlop/s",
"metadata": {
"rank": 2,
"cores": "9066176",
"rmax": "1353.00",
"rpeak": "2055.72",
"power": "24607",
"manufacturer": "HPE",
"site": "DOE/SC/Oak Ridge National Laboratory",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_3",
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
"country": "United States",
"city": "Argonne, IL",
"latitude": 41.3784,
"longitude": -87.8600,
"value": "1012.00",
"unit": "PFlop/s",
"metadata": {
"rank": 3,
"cores": "9264128",
"rmax": "1012.00",
"rpeak": "1980.01",
"power": "38698",
"manufacturer": "Intel",
},
"reference_date": "2025-11-01",
},
]
raise ValueError("Invalid TOP500 performance") from None

View File

@@ -0,0 +1,200 @@
"""Canonical country and mutually exclusive regional compute participants."""
import re
from collections.abc import Iterable
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
from app.core.earth_geography_policy import display_country
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
# Explicit membership prevents a regional total from double-counting a country.
# APAC deliberately excludes China.
COMPARISON_GROUPS = (
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
(
"EUROPE",
"欧洲",
"Europe",
(
"GB",
"DE",
"FR",
"CH",
"ES",
"FI",
"HU",
"IE",
"IT",
"LU",
"MD",
"NL",
"PL",
"RO",
"RU",
"UA",
),
{"latitude": 54.0, "longitude": 20.0},
),
(
"APAC",
"亚太(不含中国)",
"Asia Pacific (excluding China)",
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
{"latitude": 36.0, "longitude": 130.5},
),
(
"OTHER",
"其他地区",
"Other",
(
"CA",
"AO",
"EG",
"KE",
"MZ",
"NG",
"UG",
"ZA",
"BR",
"CO",
"CR",
"DO",
"MX",
"AE",
"SY",
"TR",
),
{"latitude": 12.0, "longitude": 10.0},
),
)
def _country_code(canonical: str, aliases: list[str]) -> str | None:
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
return "GB" if canonical == "英国" else code
def build_country_catalog() -> list[dict]:
"""Build the policy-projected canonical country drilldown catalog."""
countries = []
for canonical, aliases in COUNTRY_ENTRIES:
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
continue
code = _country_code(canonical, aliases)
if code:
countries.append(
{
"code": code,
"name": canonical,
"name_en": aliases[0],
"kind": "country",
"members": [code],
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
"anchor_kind": "regional_label",
}
)
return countries
COUNTRY_CATALOG = build_country_catalog()
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
def _validate_group_members() -> None:
members = [
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
]
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
raise ValueError("compute comparison groups must cover country catalog exactly once")
_validate_group_members()
GROUP_BY_MEMBER = {
member: code
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
}
def build_catalog() -> list[dict]:
"""Return the five default participants before country drilldown entries."""
catalog: list[dict] = []
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
entry = (
dict(COUNTRY_PARTICIPANTS[code])
if code in COUNTRY_PARTICIPANTS
else {
"code": code,
"name": name,
"name_en": name_en,
"kind": "region",
"members": list(members),
"anchor": dict(anchor),
"anchor_kind": "regional_label",
}
)
entry.update(
{
"members": list(members),
"parent_group": name,
"parent_group_code": code,
"order": index,
"default_selected": True,
"color": PALETTE[index % len(PALETTE)],
}
)
catalog.append(entry)
for entry in COUNTRY_CATALOG:
if entry["code"] in DEFAULT_PARTICIPANTS:
continue
country = dict(entry)
group_code = GROUP_BY_MEMBER[country["code"]]
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
country.update(
{
"parent_group": group_name,
"parent_group_code": group_code,
"order": len(catalog),
"default_selected": False,
"color": PALETTE[len(catalog) % len(PALETTE)],
}
)
catalog.append(country)
return catalog
CATALOG = build_catalog()
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
"""Resolve country or region participant codes to canonical country codes."""
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
def source_country_names(codes: Iterable[str]) -> list[str]:
"""Return unique canonical source labels, respecting China and Serbia ownership."""
selected_members = {
member for members in participant_members(codes).values() for member in members
}
selected_names = {COUNTRIES[code] for code in selected_members}
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
"""Map stored source-country labels to their requested participant code."""
owner_by_country = {
member: participant
for participant, members in participant_members(codes).items()
for member in members
}
code_by_name = {name: code for code, name in COUNTRIES.items()}
return {
name: owner_by_country[code_by_name[display_country(name)]]
for name in source_country_names(codes)
}

View File

@@ -3,7 +3,12 @@
This module is a thin domain wrapper that wires up
:mod:`app.services.location` for compute centers:
SourceCoordinates
Confirmed stored location -> trustworthy source coordinates -> exact
official facility registration -> source city/site coordinates.
Epoch AI coordinates are admitted only at the grain asserted by its official
``Location`` column. Country/region/empty locations remain unresolved even
when the CSV happens to contain a latitude/longitude pair.
The online Nominatim step is intentionally reserved for the user-triggered
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -28,9 +33,12 @@ import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import compute_record_filter
from app.core.collected_data_fields import get_record_field
from app.models.collected_data import CollectedData
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.services.compute_center_verified_sites import get_verified_compute_site
from app.services.epoch_location import assess_epoch_location
from app.services.location import (
LocationCandidate,
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
async def refresh_compute_center_location_cache(
session: AsyncSession,
*,
as_of: datetime | None = None,
) -> dict[str, dict[str, Any]]:
result = await session.execute(select(ComputeCenterLocationRecord))
query = select(ComputeCenterLocationRecord)
if as_of is not None:
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
result = await session.execute(query)
records = result.scalars().all()
cache = {}
for record in records:
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
)
class ComputeCenterSourceCoordinatesResolver:
"""Use source coordinates only at the grain asserted by the source text."""
name = "source_coordinates"
def __init__(self) -> None:
self._default = SourceCoordinatesResolver()
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
if extra.get("source") != "epoch_ai_gpu":
return self._default.resolve(query)
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
return ResolverOutput()
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput(
attempted_queries=(f"source_coordinates:{assessment.precision}",)
)
output = self._default.resolve(query)
if not output.candidates:
return output
candidate = output.candidates[0]
source_location = extra.get("source_location")
source_location_text = (
source_location.strip() if isinstance(source_location, str) else ""
)
source_note = assessment.reason
if source_location_text:
source_note = f"{source_note}; Epoch Location: {source_location_text}"
return ResolverOutput(
candidates=(
LocationCandidate(
**{
**candidate.__dict__,
"precision": assessment.precision,
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
"source_note": source_note,
"needs_confirmation": assessment.precision != "precise",
"matched_location_name": source_location_text
or candidate.matched_location_name,
}
),
),
attempted_queries=output.attempted_queries,
)
class StoredComputeCenterLocationResolver:
"""Resolve a compute center through the DB-backed current-location cache."""
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
longitude = parse_float(stored.get("longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
return ResolverOutput()
precision = stored.get("precision") or "city"
if (
extra.get("source") == "epoch_ai_gpu"
and stored.get("location_source") == "source_coordinates"
):
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput()
precision = assessment.precision
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=latitude,
longitude=longitude,
display_name=stored.get("name") or query.name or "Compute center",
precision=stored.get("precision") or "city",
precision=precision,
confidence=float(stored.get("confidence") or 0.85),
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
source=self.name,
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
matched_location_name=stored.get("site") or stored.get("name") or query.name,
location_verified_at=stored.get("verified_at"),
suggested_registry_entry=None,
raw_payload={
"stored_location_source": stored.get("location_source"),
"verification_status": stored.get("verification_status"),
},
),
)
)
class VerifiedComputeSiteResolver:
"""Resolve an exact source identity to an evidence-backed facility site."""
name = "official_facility_registry"
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
source = coerce_str(extra.get("source"))
source_id = coerce_str(extra.get("source_id"))
if not source or not source_id:
return ResolverOutput()
site = get_verified_compute_site(source, source_id)
if not site:
return ResolverOutput()
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
source_note = coerce_str(site.get("source_note"))
if source_urls:
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=float(site["latitude"]),
longitude=float(site["longitude"]),
display_name=coerce_str(site.get("matched_location_name"))
or query.name
or "Compute center",
precision=coerce_str(site.get("precision")) or "site",
confidence=1.0,
query=f"official_facility_registry::{source}:{source_id}",
source=self.name,
source_note=source_note or None,
matched_fields=("source", "source_id"),
needs_confirmation=False,
city=coerce_str(site.get("city")) or None,
region=None,
country=coerce_str(site.get("country")) or query.country,
matched_location_name=coerce_str(site.get("matched_location_name"))
or None,
location_verified_at=coerce_str(site.get("verified_at")) or None,
suggested_registry_entry=None,
raw_payload={
"address": site.get("address"),
"coordinate_basis": site.get("coordinate_basis"),
"sources": source_urls,
},
),
)
)
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
return head or text
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
name = coerce_str(getattr(record, "name", None))
return {
"source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
or metadata.get("manufacturer")
),
"organization": coerce_str(metadata.get("organization")),
"source_location": metadata.get("location"),
"source_raw_country": metadata.get("raw_country"),
}
def _context_to_query(
context: dict[str, str],
context: dict[str, Any],
*,
source_lat: float | None = None,
source_lon: float | None = None,
@@ -444,6 +578,8 @@ def _context_to_query(
"site": context.get("site") or "",
"operator": context.get("operator") or "",
"organization": context.get("organization") or "",
"source_location": context.get("source_location"),
"source_raw_country": context.get("source_raw_country"),
},
)
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
return plan
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
stored_confirmed = (
candidate.source == "stored_compute_center_location"
and not candidate.needs_confirmation
and stored_origin != "source_coordinates"
)
if stored_confirmed:
source_rank = 0
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
source_rank = 1
elif candidate.source == "official_facility_registry":
source_rank = 2
elif candidate.source == "source_coordinates":
source_rank = 3
elif candidate.source == "stored_compute_center_location":
source_rank = 4
else:
source_rank = 5
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
return (source_rank, precision_rank, -float(candidate.confidence or 0))
COMPUTE_CENTER_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
StoredComputeCenterLocationResolver(),
VerifiedComputeSiteResolver(),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
" or stored compute-center location."
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
VerifiedComputeSiteResolver(),
ROROrganizationResolver(),
NominatimResolver(
query_plan_builder=_compute_center_query_plan,
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
geocoder=lambda q: _geocode_online(q),
),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
", ROR organization lookup, or online geocoding."
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
_GEOGRAPHY_MODE_BY_SOURCE = {
"source_coordinates": "source_coordinates",
"stored_compute_center_location": "stored_compute_center_location",
"official_facility_registry": "official_facility_registry",
"ror_organization_registry": "ror_organization",
"nominatim_online_geocode": "online_geocode",
}
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
def _candidate_to_location(
candidate: LocationCandidate,
*,
context: dict[str, str],
context: dict[str, Any],
) -> ComputeCenterLocation:
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +758,7 @@ def _candidate_to_location(
def _diagnostic_for(
record: Any,
context: dict[str, str],
context: dict[str, Any],
*,
failure_reason: str,
attempted_queries: tuple[str, ...] = (),
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
organization: str | None = None,
) -> LocationQuery:
name_value = coerce_str(name)
context: dict[str, str] = {
context: dict[str, Any] = {
"source": coerce_str(source),
"source_id": coerce_str(source_id),
"name": name_value,
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
"""Seed stored compute-center locations only from real source coordinates."""
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.is_current.is_(True))
)
result = await session.execute(stmt)
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
longitude = parse_float(get_record_field(record, "longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
continue
metadata = record.extra_data or {}
precision = "precise"
confidence = 1.0
needs_confirmation = False
source_note = "Seeded from source-provided compute-center coordinates"
if source_value == "epoch_ai_gpu":
assessment = assess_epoch_location(
metadata.get("location"),
country=get_record_field(record, "country"),
raw_country=metadata.get("raw_country"),
)
if not assessment.is_renderable:
continue
precision = assessment.precision
confidence = 1.0 if precision == "precise" else 0.9
needs_confirmation = precision != "precise"
source_note = assessment.reason
existing = await session.scalar(
select(ComputeCenterLocationRecord)
.where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
)
if existing:
continue
metadata = record.extra_data or {}
session.add(
ComputeCenterLocationRecord(
source=source_value,
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
country=coerce_str(get_record_field(record, "country")) or None,
latitude=latitude,
longitude=longitude,
precision="precise",
confidence=1.0,
precision=precision,
confidence=confidence,
location_source="source_coordinates",
source_note="Seeded from source-provided compute-center coordinates",
source_note=source_note,
raw_payload={
"record_id": getattr(record, "id", None),
"source": source_value,
"source_id": source_id,
},
needs_confirmation=False,
needs_confirmation=needs_confirmation,
verification_status="source_provided",
verified_at=None,
)

View File

@@ -0,0 +1,44 @@
{
"version": 1,
"sites": [
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
"latitude": 35.380924,
"longitude": 139.625008,
"precision": "site",
"location_source": "official_facility_registry",
"country": "日本",
"city": "Yokohama",
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
]
},
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
"latitude": 41.718109383287235,
"longitude": -87.984498648496,
"precision": "site",
"location_source": "official_facility_registry",
"country": "美国",
"city": "Argonne",
"matched_location_name": "Argonne Leadership Computing Facility",
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.alcf.anl.gov/systems/sophia",
"https://docs.alcf.anl.gov/sophia/",
"https://www.alcf.anl.gov/about/visiting-alcf"
]
}
]
}

View File

@@ -0,0 +1,26 @@
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
import json
from copy import deepcopy
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=1)
def _site_index() -> dict[tuple[str, str], dict]:
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
index = {}
for site in payload["sites"]:
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
raise ValueError("Invalid verified compute facility coordinate")
for source_id in site["source_ids"]:
key = (site["source"], source_id)
if key in index:
raise ValueError("Duplicate verified compute facility identity")
index[key] = site
return index
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
site = _site_index().get((source, source_id))
return deepcopy(site) if site else None

View File

@@ -0,0 +1,99 @@
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
import hashlib
from pathlib import Path
from uuid import uuid4
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import COUNTRY_CATALOG
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
CATEGORY = "compute_comparison"
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
def default_envelope() -> dict:
return {
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
"revision": DEFAULT_REVISION,
"is_default": True,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return dict(record.payload) if record else default_envelope()
def configuration_response(envelope: dict) -> dict:
return {
**envelope,
"countries": [
{key: country[key] for key in ("code", "name", "name_en")}
for country in COUNTRY_CATALOG
],
}
async def publish_configuration(
db: AsyncSession,
*,
revision: str,
config: ComputeConfiguration,
actor: User,
reset: bool = False,
) -> dict:
# The unique category plus row lock also serializes concurrent first writes.
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=default_envelope())
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if record.payload["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
envelope = (
default_envelope()
if reset
else {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"is_default": reset,
}
)
before = record.payload
if reset:
await db.delete(record)
else:
record.payload = envelope
# Audit and configuration commit together: a published edit must be attributable.
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="compute_comparison.reset" if reset else "compute_comparison.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": envelope},
)
)
await db.commit()
return envelope

View File

@@ -0,0 +1,122 @@
"""Versioned, source-backed annual observations for compute comparisons."""
from copy import deepcopy
from datetime import date
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import (
CATALOG,
DEFAULT_PARTICIPANTS,
PARTICIPANTS,
participant_members,
)
from app.services.compute_configuration import DEFAULT_CONFIGURATION
def _participant_value(
members: tuple[str, ...], observations: dict[str, float | int], mode: str
) -> tuple[float | int | None, dict]:
covered = [member for member in members if member in observations]
missing = [member for member in members if member not in observations]
coverage = {
"status": "complete" if not missing else "incomplete",
"required_members": list(members),
"covered_members": covered,
"missing_members": missing,
}
if missing:
return None, coverage
if mode == "bilateral_share":
if (
not set(members).issubset({"CN", "US"})
or not {"CN", "US"}.issubset(observations)
or observations["US"] + observations["CN"] <= 0
):
coverage.update(
{
"status": "incompatible",
"reason": "bilateral_share_requires_CN_and_US_observations",
}
)
return None, coverage
return (
round(
sum(observations[member] for member in members)
/ (observations["US"] + observations["CN"])
* 100,
2,
),
coverage,
)
return sum(observations[member] for member in members), coverage
def compute_history(
countries: list[str],
mode: str = "global_share",
config: ComputeConfiguration | None = None,
) -> dict:
"""Aggregate only fully covered, same-year compatible observations."""
config = config if config is not None else DEFAULT_CONFIGURATION
metric = "global_share" if mode == "bilateral_share" else mode
annual = {row.year: row for row in config.observations if row.metric == metric}
gaps = {row.year: row for row in config.innovation}
observed_years = [row.year for row in config.observations] + list(gaps)
years = tuple(
range(
min(observed_years, default=date.today().year),
max([date.today().year, *observed_years]) + 1,
)
)
members_by_participant = participant_members(countries)
rows = []
for year in years:
record = annual.get(year)
observations = record.values if record else {}
values, coverage = {}, {}
for code, members in members_by_participant.items():
values[code], coverage[code] = _participant_value(members, observations, mode)
evidence_id = record.source_id if record else None
rows.append(
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
)
selected_members = {member for members in members_by_participant.values() for member in members}
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
return {
"version": f"compute-history-{config.reviewed_at}",
"provenance": "curated_reports",
"reviewed_at": config.reviewed_at.isoformat(),
"time_basis": "retrospective",
"years": years,
"mode": mode,
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
mode
],
"catalog": deepcopy(CATALOG),
"default_participants": list(DEFAULT_PARTICIPANTS),
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
"compute": rows,
"innovation": [
{
"year": year,
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
"evidence_id": gaps[year].source_id
if year in gaps and has_innovation_scope
else None,
}
for year in years
],
"evidence": {
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
},
"innovation_scope": ["US", "CN"],
"limitations": [
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
"全球份额不从中美或其他局部成员反推剩余地区。",
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
],
}

View File

@@ -0,0 +1,125 @@
{
"reviewed_at": "2026-09-22",
"sources": [
{
"id": "caict-2021",
"title": "中国算力发展指数白皮书(2021)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2021,
"published_at": null,
"locator": "正文 11 页图 4(2020 年全球份额)"
},
{
"id": "caict-2022",
"title": "中国算力发展指数白皮书(2022)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2022,
"published_at": null,
"locator": "正文 13 页图 5(2021 年全球份额)"
},
{
"id": "caict-2023",
"title": "中国算力发展指数白皮书(2023)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2023,
"published_at": null,
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
},
{
"id": "caict-2024",
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2024,
"published_at": null,
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
},
{
"id": "innovation-2026",
"title": "全球人工智能创新指数报告 2026:科技日报报道",
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
"publisher": "科技日报(新浪转载)",
"report_year": 2026,
"published_at": "2026-07-17",
"locator": "2023–2025 年美国减中国的指数分差"
}
],
"observations": [
{
"year": 2020,
"metric": "global_share",
"source_id": "caict-2021",
"values": {
"US": 36,
"CN": 31,
"JP": 6
}
},
{
"year": 2021,
"metric": "global_share",
"source_id": "caict-2022",
"values": {
"US": 34,
"CN": 33,
"JP": 5
}
},
{
"year": 2022,
"metric": "global_share",
"source_id": "caict-2023",
"values": {
"US": 34,
"CN": 33,
"JP": 4
}
},
{
"year": 2023,
"metric": "global_share",
"source_id": "caict-2024",
"values": {
"US": 41,
"CN": 31,
"JP": 4
}
},
{
"year": 2022,
"metric": "absolute",
"source_id": "caict-2023",
"values": {
"CN": 302
}
},
{
"year": 2023,
"metric": "absolute",
"source_id": "caict-2024",
"values": {
"CN": 435
}
}
],
"innovation": [
{
"year": 2023,
"value": 22.02,
"source_id": "innovation-2026"
},
{
"year": 2024,
"value": 19.96,
"source_id": "innovation-2026"
},
{
"year": 2025,
"value": 17.95,
"source_id": "innovation-2026"
}
]
}

View File

@@ -6,8 +6,10 @@ import asyncio
import hashlib
import json
import shutil
import subprocess
import sys
from datetime import UTC, datetime
from functools import lru_cache
from pathlib import Path
from typing import Any
from uuid import uuid4
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = (
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
)
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
POLICY_FALLBACK_PATH = (
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
)
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
@lru_cache(maxsize=16)
def _read_versioned_asset_cached(
path: Path, mtime_ns: int, size_bytes: int
) -> tuple[dict[str, Any], str]:
payload = _read_json(path)
return payload, _stable_json_hash(payload)
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
if not path.exists():
return {}, ""
stat = path.stat()
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
path = REPO_ROOT / source["path"]
payload = _read_json(path)
features = payload.get("features") if isinstance(payload, dict) else None
return {
"type": "FeatureCollection",
"features": features if isinstance(features, list) else [],
}
def _write_high_precision_geojson_manifest(
sources: list[dict[str, Any]],
build_input_hash: str,
missing_tools: list[str],
) -> dict[str, Any]:
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
admin0_payload = _load_source_feature_collection(admin0)
coastline_payload = _load_source_feature_collection(coastline)
claim_payload = _load_source_feature_collection(claim_lines)
for feature in coastline_payload["features"]:
props = feature.setdefault("properties", {})
if isinstance(props, dict):
props["PLANET_LAYER"] = "coastline"
base_payload = {
"type": "FeatureCollection",
"features": [*admin0_payload["features"], *coastline_payload["features"]],
}
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
_write_json(base_path, base_payload)
_write_json(hover_path, admin0_payload)
_write_json(claim_path, claim_payload)
manifest = {
"version": "natural-earth-v1",
"builtAt": _utc_now_iso(),
"tileProvider": "geojson-high-precision",
"format": "geojson-directory",
"buildInputHash": build_input_hash,
"base": base_path.name,
"hoverIndex": hover_path.name,
"claimLine": claim_path.name,
"sourceFeatureCount": {
"admin0": len(admin0_payload["features"]),
"coastline": len(coastline_payload["features"]),
"claimLines": len(claim_payload["features"]),
},
"pmtiles": None,
"missingTools": missing_tools,
}
result = subprocess.run(
[
sys.executable,
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
"--admin0-source", admin0["path"],
"--coastline-source", coastline["path"],
"--claims-source", claim_lines["path"],
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
"--build-input-hash", build_input_hash,
"--pov-policy", _relative(POV_POLICY_PATH),
"--pov-geometry", str(POV_GEOMETRY_PATH),
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
)
if result.returncode:
raise EarthBoundaryBuildError(
result.stderr.strip() or "Earth boundary policy build failed",
code="policy_build_failed",
)
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
manifest["missingTools"] = missing_tools
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
return manifest
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
"build_config": BUILD_CONFIG,
}
)
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
bool(boundary_manifest)
and boundary_manifest.get("buildInputHash") == build_input_hash
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and _policy_verified(boundary_manifest)
and PMTILES_ARTIFACT_PATH.exists()
and _pmtiles_hover_exists(boundary_manifest)
)
def _policy_verified(manifest: dict[str, Any]) -> bool:
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
coverage = manifest.get("povPolicy") or {}
return (
policy.get("profile") == "china-pov-v1"
and coverage.get("profile") == policy["profile"]
and coverage.get("sha256") == policy_hash
and coverage.get("reviewedGeometrySourceCommit")
== geometry.get("sourceCommit")
and coverage.get("geometrySha256") == geometry_hash
and coverage.get("applied") is True
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
<= set(coverage.get("appliedRuleIds") or [])
)
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
paths = [manifest.get("base"), manifest.get("hoverIndex")]
if manifest.get("chinaClaims", {}).get("available"):
paths.append(manifest.get("claimLine"))
for asset in paths:
if not isinstance(asset, str) or Path(asset).name != asset:
return False
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
return False
return True
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
hover = manifest.get("hoverIndex")
return (
isinstance(hover, str)
and Path(hover).name == hover
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
)
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
high_precision_ready = (
manifest_exists
and _policy_verified(boundary_manifest)
and (
(
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and pmtiles_exists
and _pmtiles_hover_exists(boundary_manifest)
)
or (
boundary_manifest.get("tileProvider") == "geojson-high-precision"
and _geojson_artifacts_exist(boundary_manifest)
)
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
)
)
legacy_exists = LEGACY_GEOJSON_PATH.exists()
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
_versioned_asset(POLICY_FALLBACK_PATH)[0]
)
provider = (
boundary_manifest.get("tileProvider")
if high_precision_ready
else "legacy-geojson" if legacy_exists else "missing"
else "policy-geojson-fallback" if legacy_exists else "missing"
)
return {
"provider": provider,
"high_precision_ready": high_precision_ready,
"fallback_available": legacy_exists,
"policy_publishable": bool(
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
),
"config_source": config_source,
"config_path": _relative(LOCAL_CONFIG_PATH),
"config_exists": LOCAL_CONFIG_PATH.exists(),
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
"tileProvider": boundary_manifest.get("tileProvider"),
"buildInputHash": boundary_manifest.get("buildInputHash"),
"builtAt": boundary_manifest.get("builtAt"),
"povPolicy": boundary_manifest.get("povPolicy"),
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
},
"pmtiles": {
"path": _relative(PMTILES_ARTIFACT_PATH),
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
},
"legacy": {
"path": _relative(LEGACY_GEOJSON_PATH),
"path": _relative(POLICY_FALLBACK_PATH),
"exists": legacy_exists,
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
},
"source_manifest": {
"path": _relative(SOURCE_MANIFEST_PATH),
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
build_input_hash,
"--pov-policy",
_relative(POV_POLICY_PATH),
"--pov-geometry",
str(POV_GEOMETRY_PATH),
]
)
if progress_callback:

View File

@@ -22,6 +22,12 @@ from app.services.earth_layer_adapters import (
get_earth_update_layers_for_source,
)
from app.services.earth_layer_cache import invalidate_earth_layer_cache_for_source
from app.services.earth_outbox_maintenance import (
configured_retention_days,
purge_consumed_events,
release_maintenance_lock,
try_maintenance_lock,
)
logger = get_logger(__name__)
@@ -30,6 +36,7 @@ DEFAULT_DEBOUNCE_SECONDS = 0.25
DEFAULT_MAX_WAIT_SECONDS = 1.5
DELETE_FAST_FLUSH_SECONDS = 0.05
LISTEN_KEEPALIVE_SECONDS = 5.0
OUTBOX_RETENTION_CHECK_SECONDS = 60.0
OUTBOX_POLL_LIMIT = 5000
MAX_ENTITY_KEY_SAMPLES = 20
MAX_SEEN_EVENT_IDS = 20000
@@ -346,6 +353,7 @@ class EarthDbChangeListener:
self._stop_event: asyncio.Event | None = None
self._connection: asyncpg.Connection | None = None
self._loop: asyncio.AbstractEventLoop | None = None
self._next_retention_check = 0.0
def start(self) -> None:
if self._task and not self._task.done():
@@ -385,6 +393,7 @@ class EarthDbChangeListener:
)
except asyncio.TimeoutError:
await self._poll_outbox()
await self._maybe_purge_outbox()
except asyncio.CancelledError:
raise
except Exception as exc:
@@ -459,6 +468,27 @@ class EarthDbChangeListener:
context={"events": len(rows), "accepted": accepted_count},
)
async def _maybe_purge_outbox(self) -> None:
if self._connection is None or monotonic() < self._next_retention_check:
return
self._next_retention_check = monotonic() + OUTBOX_RETENTION_CHECK_SECONDS
try:
if not await try_maintenance_lock(self._connection):
return
try:
await purge_consumed_events(
self._connection,
retention_days=configured_retention_days(),
)
finally:
await release_maintenance_lock(self._connection)
except Exception as exc:
logger.exception_event(
"Earth outbox retention failed",
event="earth.outbox.retention_failed",
context={"error": str(exc)},
)
def _on_notification(
self,
_connection: asyncpg.Connection,

View File

@@ -99,7 +99,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
sources=frozenset({"media_news_archive", "earth_news_items"}),
tables=frozenset({"earth_news_items"}),
layers=("news",),
cache_patterns=("summary*",),
cache_patterns=(),
refresh_strategy="reload",
),
EarthLayerAdapter(

View File

@@ -2,25 +2,27 @@ from __future__ import annotations
import asyncio
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from contextlib import suppress
from dataclasses import dataclass, replace
import hashlib
import json
import time
from typing import Any
from uuid import uuid4
from fastapi import Response
from fastapi import HTTPException, Response
from app.core.cache import _RedisClient
from app.core.config import settings
from app.core.logging import get_logger
logger = get_logger(__name__, service="earth_layer_cache")
EARTH_LAYER_CACHE_PREFIX = "earth:layer:v1"
EARTH_LAYER_LOCK_PREFIX = "earth:layer:lock:v1"
DEFAULT_LOCK_TTL_SECONDS = 10
DEFAULT_LOCK_WAIT_SECONDS = 0.2
DEFAULT_BUILD_WAIT_SECONDS = 30
DEFAULT_MAX_FEATURES = 5000
DEFAULT_MAX_BYTES = 5 * 1024 * 1024
DEFAULT_BBOX_PRECISION_DEGREES = 0.1
@@ -35,6 +37,7 @@ class EarthLayerCachePolicy:
max_bytes: int = DEFAULT_MAX_BYTES
lock_ttl_seconds: int = DEFAULT_LOCK_TTL_SECONDS
lock_wait_seconds: float = DEFAULT_LOCK_WAIT_SECONDS
build_wait_seconds: float = DEFAULT_BUILD_WAIT_SECONDS
@dataclass(frozen=True)
@@ -49,6 +52,9 @@ class EarthLayerCacheResult:
class EarthLayerCache:
def __init__(self) -> None:
self._client = None
self._inflight: dict[
tuple[asyncio.AbstractEventLoop, str], asyncio.Future[EarthLayerCacheResult | None]
] = {}
@property
def client(self):
@@ -82,17 +88,62 @@ class EarthLayerCache:
value = json.loads(raw)
return value if isinstance(value, dict) else None
def set_json(self, key: str, payload: dict[str, Any], ttl_seconds: int) -> None:
self.client.setex(key, ttl_seconds, json.dumps(payload, ensure_ascii=False, default=str))
def acquire_lock(self, key: str, ttl_seconds: int) -> str | None:
token = uuid4().hex
return (
token if self.client.set(self.lock_key(key), token, nx=True, ex=ttl_seconds) else None
)
def acquire_lock(self, key: str, ttl_seconds: int) -> bool:
return bool(self.client.set(self.lock_key(key), "1", nx=True, ex=ttl_seconds))
def renew_lock(self, key: str, token: str, ttl_seconds: int) -> bool:
return bool(
self.client.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('expire', KEYS[1], ARGV[2])
end
return 0
""",
1,
self.lock_key(key),
token,
ttl_seconds,
)
)
def release_lock(self, key: str) -> None:
try:
self.client.delete(self.lock_key(key))
except Exception:
pass
def release_lock(self, key: str, token: str) -> None:
self.client.eval(
"""
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
end
return 0
""",
1,
self.lock_key(key),
token,
)
def publish(
self, key: str, token: str, payload: dict[str, Any], policy: EarthLayerCachePolicy
) -> bool:
return bool(
self.client.eval(
"""
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
redis.call('setex', KEYS[2], ARGV[2], ARGV[4])
redis.call('setex', KEYS[3], ARGV[3], ARGV[4])
return 1
""",
3,
self.lock_key(key),
key,
self.stale_key(key),
token,
policy.fresh_ttl_seconds,
policy.stale_ttl_seconds,
json.dumps(payload, ensure_ascii=False, default=str),
)
)
def delete_pattern(self, pattern: str = f"{EARTH_LAYER_CACHE_PREFIX}:*") -> int:
keys = list(self.client.scan_iter(match=pattern))
@@ -169,56 +220,122 @@ async def resolve_layer_payload(
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
) -> EarthLayerCacheResult:
started = time.perf_counter()
try:
cached = earth_layer_cache.get_json(key)
if cached is not None:
return _result(cached, state="hit", key=key)
lock_acquired = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
if lock_acquired:
try:
payload = await _build_budgeted_payload(builder, policy)
_write_fresh_and_stale(key, payload, policy)
_log_cache_event("refresh", key, payload, started)
return _result(payload, state="refresh", key=key)
except Exception as exc:
stale = _read_stale(key)
if stale is not None:
logger.warning_event(
"Earth layer cache builder failed; returning stale payload",
event="earth_layer_cache.stale_after_builder_error",
context={"key": key, "error": str(exc)},
)
return _result(stale, state="stale", key=key)
raise
finally:
earth_layer_cache.release_lock(key)
flight_key = (asyncio.get_running_loop(), key)
while flight_key in earth_layer_cache._inflight:
stale = _read_stale(key)
if stale is not None:
return _result(stale, state="stale", key=key)
future = earth_layer_cache._inflight[flight_key]
done, _ = await asyncio.wait({future}, timeout=policy.build_wait_seconds)
if not done:
raise HTTPException(503, "Earth layer cache rebuild is still in progress")
result = future.result()
if result is not None:
state = "hit" if result.state == "refresh" else result.state
return replace(result, state=state)
await asyncio.sleep(policy.lock_wait_seconds)
cached_after_wait = earth_layer_cache.get_json(key)
if cached_after_wait is not None:
return _result(cached_after_wait, state="hit", key=key)
payload = await _build_budgeted_payload(builder, policy)
_log_cache_event("miss", key, payload, started)
return _result(payload, state="miss", key=key)
future = asyncio.get_running_loop().create_future()
earth_layer_cache._inflight[flight_key] = future
try:
result = await _resolve_layer_payload(key=key, policy=policy, builder=builder)
future.set_result(result)
return result
except asyncio.CancelledError:
# A waiting request can take over using its own still-open DB session.
future.set_result(None)
raise
except Exception as exc:
future.set_exception(exc)
future.exception()
raise
finally:
earth_layer_cache._inflight.pop(flight_key)
async def _resolve_layer_payload(
*,
key: str,
policy: EarthLayerCachePolicy,
builder: Callable[[], Awaitable[dict[str, Any]]],
) -> EarthLayerCacheResult:
started = time.perf_counter()
try:
async with asyncio.timeout(policy.build_wait_seconds):
while True:
cached = earth_layer_cache.get_json(key)
if cached is not None:
return _result(cached, state="hit", key=key)
token = earth_layer_cache.acquire_lock(key, policy.lock_ttl_seconds)
if token is not None:
break
stale = _read_stale(key)
if stale is not None:
return _result(stale, state="stale", key=key)
await asyncio.sleep(policy.lock_wait_seconds)
except TimeoutError:
raise HTTPException(503, "Earth layer cache rebuild is still in progress") from None
except Exception as exc:
try:
payload = await _build_budgeted_payload(builder, policy)
except Exception:
raise exc
logger.warning_event(
"Earth layer cache bypassed",
event="earth_layer_cache.bypass",
context={"key": key, "error": str(exc)},
)
payload = await _build_budgeted_payload(builder, policy)
return _result(payload, state="bypass", key=key)
renewal = asyncio.create_task(_renew_cache_lock(key, token, policy))
try:
try:
payload = await _build_budgeted_payload(builder, policy)
except Exception as exc:
stale = _read_stale(key)
if stale is None:
raise
logger.warning_event(
"Earth layer cache builder failed; returning stale payload",
event="earth_layer_cache.stale_after_builder_error",
context={"key": key, "error": str(exc)},
)
return _result(stale, state="stale", key=key)
try:
published = earth_layer_cache.publish(key, token, payload, policy)
except Exception as exc:
logger.warning_event(
"Earth layer cache publish failed",
event="earth_layer_cache.bypass",
context={"key": key, "error": str(exc)},
)
published = False
state = "refresh" if published else "bypass"
_log_cache_event(state, key, payload, started)
return _result(payload, state=state, key=key)
finally:
renewal.cancel()
with suppress(asyncio.CancelledError):
await renewal
try:
earth_layer_cache.release_lock(key, token)
except Exception as exc:
logger.warning_event(
"Earth layer cache lock release failed",
event="earth_layer_cache.lock_release_failed",
context={"key": key, "error": str(exc)},
)
async def _renew_cache_lock(key: str, token: str, policy: EarthLayerCachePolicy) -> None:
try:
while True:
await asyncio.sleep(policy.lock_ttl_seconds / 3)
if not earth_layer_cache.renew_lock(key, token, policy.lock_ttl_seconds):
raise RuntimeError("Earth layer cache lock ownership lost")
except Exception as exc:
logger.warning_event(
"Earth layer cache lock renewal stopped",
event="earth_layer_cache.lock_renewal_failed",
context={"key": key, "error": str(exc)},
)
def apply_payload_budget(payload: dict[str, Any], policy: EarthLayerCachePolicy) -> dict[str, Any]:
budgeted = _truncate_features(payload, policy.max_features, "feature_budget")
@@ -286,11 +403,6 @@ async def _build_budgeted_payload(
return apply_payload_budget(payload, policy)
def _write_fresh_and_stale(key: str, payload: dict[str, Any], policy: EarthLayerCachePolicy) -> None:
earth_layer_cache.set_json(key, payload, policy.fresh_ttl_seconds)
earth_layer_cache.set_json(earth_layer_cache.stale_key(key), payload, policy.stale_ttl_seconds)
def _read_stale(key: str) -> dict[str, Any] | None:
try:
return earth_layer_cache.get_json(earth_layer_cache.stale_key(key))
@@ -359,14 +471,16 @@ def _payload_size(payload: dict[str, Any]) -> int:
def _safe_key_part(value: Any) -> str:
raw = str(value).strip().lower()
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[:160]
return "".join(char if char.isalnum() or char in {"-", "_", ".", ","} else "_" for char in raw)[
:160
]
def _layer_from_key(key: str) -> str:
prefix = f"{EARTH_LAYER_CACHE_PREFIX}:"
if not key.startswith(prefix):
return "unknown"
remainder = key[len(prefix):]
remainder = key[len(prefix) :]
return remainder.split(":", 1)[0]

View File

@@ -1,8 +1,9 @@
from __future__ import annotations
import asyncio
import codecs
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
import hashlib
import html
@@ -1638,10 +1639,15 @@ async def record_earth_news_sources_health(
await db.commit()
async def get_configured_sources_for_region(db: AsyncSession | None, region: str) -> list[NewsFeedSource]:
async def get_configured_sources_for_region(
db: AsyncSession | None,
region: str,
*,
config_payload: dict[str, Any] | None = None,
) -> list[NewsFeedSource]:
if db is None:
return get_sources_for_region(region)
payload = await get_earth_news_sources_payload(db)
payload = config_payload or await get_earth_news_sources_payload(db)
sources = [
source
for raw in payload.get("sources", [])
@@ -1800,6 +1806,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
return ""
def _decode_feed_response(response: httpx.Response) -> str:
content = response.content
for marker, encoding in (
(codecs.BOM_UTF32_LE, "utf-32"),
(codecs.BOM_UTF32_BE, "utf-32"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
(codecs.BOM_UTF8, "utf-8-sig"),
):
if content.startswith(marker):
return content.decode(encoding)
declaration = re.match(
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
content,
re.IGNORECASE,
)
encoding = response.charset_encoding or (
declaration.group(1).decode("ascii") if declaration else "utf-8"
)
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
return content.decode(encoding)
def _parse_feed_entries(
xml_text: str,
source: NewsFeedSource,
@@ -2453,7 +2483,9 @@ async def _fetch_single_feed_url(
feed=feed,
)
try:
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
items = _parse_feed_entries(
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
)
except Exception as exc:
error, status = _format_source_fetch_error(
status_code=status_code,
@@ -2461,7 +2493,7 @@ async def _fetch_single_feed_url(
body=response.text,
parsed_count=0,
)
if error is None:
if status == "empty":
error = str(exc)
status = "format_error"
return source, [], error, _source_health_result(
@@ -2704,7 +2736,6 @@ async def get_earth_news_payload(
del provider_client
active_region = region if region in REGION_ANCHORS else determine_focus_region(lat, lon)
has_settings_db = db is not None and callable(getattr(db, "execute", None))
source_config_payload = await get_earth_news_sources_payload(db) if has_settings_db else None
sources = (
await get_configured_sources_for_region(db, active_region)
if has_settings_db
@@ -2723,39 +2754,7 @@ async def get_earth_news_payload(
locale=locale,
)
from app.services.earth_news_store import (
get_earth_news_feed_coverage,
get_earth_news_freshness,
list_earth_news_cruise_items,
list_earth_news_items,
upsert_earth_news_items,
)
errors: list[str] = []
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
should_supplement = _needs_rss_supplement(item_count=item_count, newest_at=newest_at)
if not should_supplement and callable(getattr(db, "execute", None)):
expected_feed_keys = _expected_feed_keys(sources, active_region=active_region)
if expected_feed_keys:
covered_feed_keys = await get_earth_news_feed_coverage(
db,
active_region=active_region,
recent_after=datetime.now(UTC) - timedelta(seconds=RSS_SUPPLEMENT_MAX_AGE_SECONDS),
)
should_supplement = bool(expected_feed_keys - covered_feed_keys)
if should_supplement:
if source_config_payload is None:
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(sources, active_region=active_region)
else:
fetched_items, errors, health_by_source = await _fetch_rss_items_for_sources(
sources,
active_region=active_region,
config_payload=source_config_payload,
)
await record_earth_news_sources_health(db, health_by_source)
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
await _enqueue_unverified_locations(ranked_fetched_items)
await upsert_earth_news_items(db, ranked_fetched_items)
from app.services.earth_news_store import list_earth_news_cruise_items, list_earth_news_items
items = await _call_store_list_items(
list_earth_news_items,
@@ -2765,7 +2764,7 @@ async def get_earth_news_payload(
categories=categories,
source_ids=source_ids,
)
if not source_ids:
if items and not source_ids:
ready_sources = {
_news_item_source_id(item)
for item in items
@@ -2806,10 +2805,6 @@ async def get_earth_news_payload(
)
else:
cruise_items = items
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
stale = bool(errors and items)
return _build_payload(
lat=lat,
lon=lon,
@@ -2817,8 +2812,8 @@ async def get_earth_news_payload(
items=items,
cruise_items=cruise_items,
sources=sources,
errors=errors,
stale=stale,
errors=[],
stale=False,
categories=categories,
source_ids=source_ids,
limit=limit,

View File

@@ -0,0 +1,480 @@
"""Periodic RSS supplementation for the read-only Earth news feed."""
from __future__ import annotations
import asyncio
from contextlib import suppress
from dataclasses import replace
from datetime import UTC, datetime, timedelta
from hashlib import sha256
import json
from typing import Any
from uuid import uuid4
from app.core.cache import _RedisClient
from app.core.logging import get_logger
from app.db.session import async_session_factory
from app.services import earth_news
from app.services.earth_news_store import (
get_earth_news_feed_coverage,
get_earth_news_freshness,
upsert_earth_news_items,
)
logger = get_logger(__name__, service="earth_news")
REFRESH_INTERVAL_SECONDS = 10 * 60
REFRESH_LOCK_TTL_SECONDS = 120
REFRESH_LOCK_RENEW_SECONDS = 30
SOURCE_FAILURE_COOLDOWN_SECONDS = 30 * 60
REFRESH_LOCK_PREFIX = "earth:news:rss-refresh:v1"
SOURCE_COOLDOWN_PREFIX = "earth:news:rss-cooldown:v1"
REFRESH_STATUS_PREFIX = "earth:news:rss-refresh-status:v1"
REFRESH_ACTIVE_PREFIX = "earth:news:rss-refresh-active:v1"
REFRESH_STATUS_TTL_SECONDS = 10 * 60
REFRESH_ACTIVE_TTL_SECONDS = 120
REFRESH_STATUS_HEARTBEAT_SECONDS = 30
def _refresh_status_key(refresh_id: str) -> str:
return f"{REFRESH_STATUS_PREFIX}:{refresh_id}"
def _refresh_active_key(region: str) -> str:
return f"{REFRESH_ACTIVE_PREFIX}:{region}"
def create_earth_news_refresh_status(region: str) -> tuple[dict[str, Any], bool]:
"""Create one per-region active refresh status, or return the current one."""
refresh_id = uuid4().hex
status = {"refresh_id": refresh_id, "region": region, "status": "pending"}
script = """
local active_id = redis.call('get', KEYS[1])
if active_id then
local active_status = redis.call('get', ARGV[5] .. active_id)
if active_status then
local decoded = cjson.decode(active_status)
if decoded.status == 'pending' or decoded.status == 'running' then
return {active_id, active_status, 0}
end
end
redis.call('del', KEYS[1])
end
redis.call('set', KEYS[1], ARGV[1], 'EX', ARGV[3])
redis.call('set', KEYS[2], ARGV[2], 'EX', ARGV[4])
return {ARGV[1], ARGV[2], 1}
"""
raw = _RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
refresh_id,
json.dumps(status, separators=(",", ":")),
REFRESH_ACTIVE_TTL_SECONDS,
REFRESH_STATUS_TTL_SECONDS,
f"{REFRESH_STATUS_PREFIX}:",
)
_, encoded, created = raw
payload = json.loads(encoded)
return payload, bool(int(created))
def set_earth_news_refresh_status(
refresh_id: str,
region: str,
status: str,
*,
error: dict[str, str] | None = None,
) -> dict[str, Any]:
payload: dict[str, Any] = {
"refresh_id": refresh_id,
"region": region,
"status": status,
}
if error:
payload["error"] = error
script = """
if redis.call('get', KEYS[1]) ~= ARGV[2] then return 0 end
redis.call('set', KEYS[2], ARGV[1], 'EX', ARGV[3])
if ARGV[4] == 'succeeded' or ARGV[4] == 'failed' then
redis.call('del', KEYS[1])
else
redis.call('expire', KEYS[1], ARGV[5])
end
return 1
"""
_RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
json.dumps(payload, separators=(",", ":")),
refresh_id,
REFRESH_STATUS_TTL_SECONDS,
status,
REFRESH_ACTIVE_TTL_SECONDS,
)
return payload
def touch_earth_news_refresh_status(refresh_id: str, region: str) -> bool:
script = """
if redis.call('get', KEYS[1]) ~= ARGV[1] then return 0 end
redis.call('expire', KEYS[1], ARGV[2])
redis.call('expire', KEYS[2], ARGV[3])
return 1
"""
return bool(
_RedisClient.get_client().eval(
script,
2,
_refresh_active_key(region),
_refresh_status_key(refresh_id),
refresh_id,
REFRESH_ACTIVE_TTL_SECONDS,
REFRESH_STATUS_TTL_SECONDS,
)
)
def get_earth_news_refresh_status(refresh_id: str) -> dict[str, Any] | None:
raw = _RedisClient.get_client().get(_refresh_status_key(refresh_id))
if not raw:
return None
value = json.loads(raw)
if not isinstance(value, dict):
return None
if value.get("status") in {"pending", "running"}:
region = str(value.get("region") or "")
active_id = _RedisClient.get_client().get(_refresh_active_key(region)) if region else None
if active_id != refresh_id:
return {
**value,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
return value
def _lock_key(region: str) -> str:
return f"{REFRESH_LOCK_PREFIX}:{region}"
def _cooldown_key(region: str, source_id: str, feed_id: str) -> str:
identity = f"{region}\0{source_id}\0{feed_id}".encode("utf-8")
return f"{SOURCE_COOLDOWN_PREFIX}:{sha256(identity).hexdigest()}"
def _source_failure_cooldown_seconds(source: earth_news.NewsFeedSource | None) -> int:
if source is None:
return SOURCE_FAILURE_COOLDOWN_SECONDS
configured_minutes = source.health_policy.get("cooldown_minutes")
try:
return max(60, int(float(configured_minutes) * 60))
except TypeError, ValueError:
return SOURCE_FAILURE_COOLDOWN_SECONDS
def _select_feeds(
sources: list[earth_news.NewsFeedSource],
*,
active_region: str,
required_keys: set[tuple[str, str]],
redis_client: Any | None,
) -> tuple[list[earth_news.NewsFeedSource], set[tuple[str, str]]]:
selected_sources: list[earth_news.NewsFeedSource] = []
cooling_keys: set[tuple[str, str]] = set()
for source in sources:
selected_feeds = []
source_cooling = False
if redis_client is not None:
try:
source_cooling = bool(
redis_client.exists(_cooldown_key(active_region, source.id, "__source__"))
)
except Exception as exc:
logger.warning_event(
"Earth news source cooldown lookup failed",
event="earth_news.rss.cooldown_lookup_failed",
context={"region": active_region, "source_id": source.id, "error": str(exc)},
)
for feed in earth_news._source_feeds(source):
feed_key = (source.id, feed.id)
if feed_key not in required_keys:
continue
if source_cooling:
cooling_keys.add(feed_key)
continue
if redis_client is not None:
try:
if redis_client.exists(_cooldown_key(active_region, *feed_key)):
cooling_keys.add(feed_key)
continue
except Exception as exc:
logger.warning_event(
"Earth news source cooldown lookup failed",
event="earth_news.rss.cooldown_lookup_failed",
context={
"region": active_region,
"source_id": source.id,
"error": str(exc),
},
)
selected_feeds.append(feed)
if selected_feeds:
selected_sources.append(
replace(
source,
feed_url=selected_feeds[0].url,
feed_urls=tuple(feed.url for feed in selected_feeds),
feeds=tuple(selected_feeds),
)
)
return selected_sources, cooling_keys
def _record_failed_feed_cooldowns(
redis_client: Any | None,
*,
active_region: str,
health_by_source: dict[str, dict[str, Any]],
sources: list[earth_news.NewsFeedSource],
) -> None:
if redis_client is None:
return
cooldown_by_source = {source.id: source for source in sources}
for source_id, health in health_by_source.items():
feed_results = health.get("feed_results")
if isinstance(feed_results, list) and feed_results:
failed_feed_ids = [
str(result.get("feed_id") or "")
for result in feed_results
if isinstance(result, dict) and not result.get("ok") and result.get("feed_id")
]
else:
failed_feed_ids = [str(health.get("feed_id") or "__source__")]
if health.get("ok"):
failed_feed_ids = []
for feed_id in failed_feed_ids:
try:
redis_client.set(
_cooldown_key(active_region, source_id, feed_id),
"1",
ex=_source_failure_cooldown_seconds(cooldown_by_source.get(source_id)),
)
except Exception as exc:
logger.warning_event(
"Earth news source failure cooldown write failed",
event="earth_news.rss.cooldown_write_failed",
context={"region": active_region, "source_id": source_id, "error": str(exc)},
)
async def _renew_lock(redis_client: Any, key: str, token: str, lease_lost: asyncio.Event) -> None:
renew_script = """
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('expire', KEYS[1], ARGV[2])
end
return 0
"""
try:
while True:
await asyncio.sleep(REFRESH_LOCK_RENEW_SECONDS)
if not redis_client.eval(renew_script, 1, key, token, REFRESH_LOCK_TTL_SECONDS):
lease_lost.set()
return
except asyncio.CancelledError:
raise
except Exception as exc:
lease_lost.set()
logger.warning_event(
"Earth news refresh lock renewal failed",
event="earth_news.rss.refresh_lock_renewal_failed",
context={"error": str(exc)},
)
async def refresh_earth_news_region(active_region: str, *, force: bool = False) -> dict[str, Any]:
"""Fetch only stale or uncovered feeds for one region, outside the request path."""
if active_region not in earth_news.REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {active_region}")
lock_key = _lock_key(active_region)
token = uuid4().hex
redis_client = None
lock_acquired = False
lock_available = False
try:
redis_client = _RedisClient.get_client()
lock_acquired = bool(
redis_client.set(lock_key, token, nx=True, ex=REFRESH_LOCK_TTL_SECONDS)
)
lock_available = True
except Exception as exc:
logger.warning_event(
"Earth news refresh coordination unavailable",
event="earth_news.rss.refresh_lock_unavailable",
context={"region": active_region, "error": str(exc)},
)
if not lock_available or not lock_acquired:
state = "busy" if lock_available else "coordination_unavailable"
return {"state": state, "fetched": 0, "upserted": 0}
lease_lost = asyncio.Event()
renewal: asyncio.Task[None] | None = None
if lock_acquired and redis_client is not None:
renewal = asyncio.create_task(_renew_lock(redis_client, lock_key, token, lease_lost))
try:
async with async_session_factory() as db:
source_config = await earth_news.get_earth_news_sources_payload(db)
sources = await earth_news.get_configured_sources_for_region(
db, active_region, config_payload=source_config
)
if not sources:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
item_count, newest_at = await get_earth_news_freshness(db, active_region=active_region)
expected_keys = earth_news._expected_feed_keys(sources, active_region=active_region)
if not expected_keys:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
now = datetime.now(UTC)
needs_full_refresh = force or earth_news._needs_rss_supplement(
item_count=item_count, newest_at=newest_at
)
covered_keys = await get_earth_news_feed_coverage(
db,
active_region=active_region,
recent_after=now - timedelta(seconds=earth_news.RSS_SUPPLEMENT_MAX_AGE_SECONDS),
)
required_keys = expected_keys if needs_full_refresh else expected_keys - covered_keys
if not required_keys:
return {
"state": "completed",
"successful_sources": 0,
"failed_sources": 0,
"fetched": 0,
"upserted": 0,
}
selected_sources, cooling_keys = _select_feeds(
sources,
active_region=active_region,
required_keys=required_keys,
redis_client=redis_client,
)
if cooling_keys:
logger.info_event(
"Skipping failed Earth news feeds during cooldown",
event="earth_news.rss.cooldown_active",
context={"region": active_region, "feed_count": len(cooling_keys)},
)
if not selected_sources:
return {
"state": "failed",
"error_code": "source_unavailable",
"fetched": 0,
"upserted": 0,
}
fetched_items, _errors, health_by_source = (
await earth_news._fetch_rss_items_for_sources(
selected_sources,
active_region=active_region,
config_payload=source_config,
)
)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": 0, "upserted": 0}
_record_failed_feed_cooldowns(
redis_client,
active_region=active_region,
health_by_source=health_by_source,
sources=selected_sources,
)
await earth_news.record_earth_news_sources_health(db, health_by_source)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": 0, "upserted": 0}
ranked_items = earth_news._rank_and_trim_items(
fetched_items, active_region=active_region
)
await earth_news._enqueue_unverified_locations(ranked_items)
upserted = await upsert_earth_news_items(db, ranked_items)
if lease_lost.is_set():
await db.rollback()
return {"state": "busy", "fetched": len(fetched_items), "upserted": 0}
await db.commit()
successful_sources = sum(bool(health.get("ok")) for health in health_by_source.values())
failed_sources = max(0, len(selected_sources) - successful_sources)
state = "completed" if successful_sources else "failed"
failed_statuses = {
str(health.get("status") or "")
for health in health_by_source.values()
if not health.get("ok")
}
error_code = None
if not successful_sources:
error_code = "timeout" if failed_statuses == {"timeout"} else "source_unavailable"
return {
"state": state,
"error_code": error_code,
"successful_sources": successful_sources,
"failed_sources": failed_sources,
"fetched": len(fetched_items),
"upserted": upserted,
}
except Exception as exc:
logger.warning_event(
"Earth news periodic RSS refresh failed",
event="earth_news.rss.refresh_failed",
context={"region": active_region, "error": str(exc)},
)
raise
finally:
if renewal is not None:
renewal.cancel()
with suppress(asyncio.CancelledError):
await renewal
if lock_acquired and redis_client is not None:
release_script = """
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
end
return 0
"""
try:
redis_client.eval(release_script, 1, lock_key, token)
except Exception as exc:
logger.warning_event(
"Earth news refresh lock release failed",
event="earth_news.rss.refresh_lock_release_failed",
context={"region": active_region, "error": str(exc)},
)
async def refresh_earth_news_all_regions() -> None:
for active_region in earth_news.REGION_ANCHORS:
try:
await refresh_earth_news_region(active_region)
except asyncio.CancelledError:
raise
except Exception:
continue

View File

@@ -278,22 +278,25 @@ async def get_earth_news_feed_coverage(
active_region: str,
recent_after: datetime | None = None,
) -> set[tuple[str, str]]:
query = select(
EarthNewsItem.id,
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id"),
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_id"),
news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
source_id = func.coalesce(
func.nullif(news_meta.op("->>")("source_id"), ""),
func.split_part(EarthNewsItem.id, ":", 1),
)
feed_id = func.nullif(news_meta.op("->>")("feed_id"), "")
query = select(
source_id,
feed_id,
).where(source_id != "", feed_id.is_not(None)).distinct()
if active_region != "global":
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
if recent_after is not None:
query = query.where(func.coalesce(EarthNewsItem.published_at, EarthNewsItem.last_seen_at) >= recent_after)
result = await db.execute(query)
coverage: set[tuple[str, str]] = set()
for item_id, source_id, feed_id in result.all():
for source_id, feed_id in result.all():
normalized_source_id = str(source_id or "").strip()
normalized_feed_id = str(feed_id or "").strip()
if not normalized_source_id and isinstance(item_id, str) and ":" in item_id:
normalized_source_id = item_id.split(":", 1)[0]
if normalized_source_id and normalized_feed_id:
coverage.add((normalized_source_id, normalized_feed_id))
return coverage
@@ -340,30 +343,49 @@ async def upsert_earth_news_items(db: AsyncSession, items: list[ParsedNewsItem])
changed += 1
continue
record.title = item.title
record.summary = item.summary
record.url = item.url
record.source = item.source
record.feed_name = item.feed_name
record.region = item.feed_region
record.homepage_url = item.homepage_url
record.published_at = item.published_at
visible_changed = False
for field_name, value in (
("title", item.title),
("summary", item.summary),
("url", item.url),
("source", item.source),
("feed_name", item.feed_name),
("region", item.feed_region),
("homepage_url", item.homepage_url),
("published_at", item.published_at),
):
if getattr(record, field_name) != value:
setattr(record, field_name, value)
visible_changed = True
record.last_seen_at = now
location_meta = dict(record.location_meta or {})
location_meta["news_meta"] = _news_meta_patch(item)
record.location_meta = location_meta
news_meta = _news_meta_patch(item)
if location_meta.get("news_meta") != news_meta:
location_meta["news_meta"] = news_meta
record.location_meta = location_meta
visible_changed = True
if item.localizations:
merged_localizations = {
**dict(record.localizations or {}),
**dict(item.localizations or {}),
}
record.content_language = item.content_language
record.localizations = merged_localizations
if record.content_language != item.content_language:
record.content_language = item.content_language
visible_changed = True
if dict(record.localizations or {}) != merged_localizations:
record.localizations = merged_localizations
visible_changed = True
if item.enrichment_status != "pending" or item.enrichment_error or item.enriched_at:
record.enrichment_status = item.enrichment_status
record.enrichment_error = item.enrichment_error
record.enriched_at = item.enriched_at
changed += 1
for field_name, value in (
("enrichment_status", item.enrichment_status),
("enrichment_error", item.enrichment_error),
("enriched_at", item.enriched_at),
):
if getattr(record, field_name) != value:
setattr(record, field_name, value)
visible_changed = True
if visible_changed:
changed += 1
await db.flush()
return changed

View File

@@ -0,0 +1,184 @@
"""Bounded retention for consumed Earth change events."""
from __future__ import annotations
import asyncio
import os
from datetime import UTC, datetime, timedelta
from typing import AsyncContextManager, Protocol
from app.core.logging import get_logger
logger = get_logger(__name__)
INDEX_NAME = "idx_earth_data_change_events_consumed_retention"
DEFAULT_RETENTION_DAYS = 1
DEFAULT_BATCH_SIZE = 1000
DEFAULT_PAUSE_SECONDS = 0.1
MAINTENANCE_LOCK_KEY = (17001613, 1)
DELETE_BATCH_SQL = """
WITH doomed AS (
SELECT id
FROM earth_data_change_events
WHERE consumed_at IS NOT NULL AND consumed_at < $1
ORDER BY consumed_at, id
LIMIT $2
FOR UPDATE SKIP LOCKED
), deleted AS (
DELETE FROM earth_data_change_events AS events
USING doomed
WHERE events.id = doomed.id
RETURNING events.id
)
SELECT count(*) FROM deleted
"""
class OutboxConnection(Protocol):
async def fetchval(self, query: str, *args: object) -> object: ...
async def fetchrow(self, query: str, *args: object) -> object: ...
async def execute(self, query: str, *args: object) -> object: ...
def transaction(self) -> AsyncContextManager[object]: ...
def configured_retention_days() -> int:
raw = os.getenv("PLANET_EARTH_OUTBOX_RETENTION_DAYS", str(DEFAULT_RETENTION_DAYS))
try:
days = int(raw)
if 1 <= days <= 3650:
return days
except ValueError:
pass
logger.warning_event(
"Invalid Earth outbox retention; using default",
event="earth.outbox.retention_invalid",
context={"value": raw},
)
return DEFAULT_RETENTION_DAYS
async def try_maintenance_lock(conn: OutboxConnection) -> bool:
return bool(await conn.fetchval("SELECT pg_try_advisory_lock($1, $2)", *MAINTENANCE_LOCK_KEY))
async def release_maintenance_lock(conn: OutboxConnection) -> None:
await conn.fetchval("SELECT pg_advisory_unlock($1, $2)", *MAINTENANCE_LOCK_KEY)
async def retention_index_state(conn: OutboxConnection) -> str:
row = await conn.fetchrow(
"""
SELECT indexes.indisvalid, indexes.indisready
FROM pg_index AS indexes
JOIN pg_class AS index_table ON index_table.oid = indexes.indexrelid
WHERE indexes.indrelid = 'earth_data_change_events'::regclass
AND index_table.relname = $1
""",
INDEX_NAME,
)
if row is None:
return "missing"
return "ready" if row["indisvalid"] and row["indisready"] else "invalid"
async def ensure_retention_index(conn: OutboxConnection) -> None:
"""Call on an autocommit connection, never inside app init_db's transaction."""
state = await retention_index_state(conn)
if state == "invalid":
await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}")
if state != "ready":
await conn.execute(
f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {INDEX_NAME} "
"ON earth_data_change_events (consumed_at, id) WHERE consumed_at IS NOT NULL"
)
if await retention_index_state(conn) != "ready":
raise RuntimeError("Earth outbox retention index is not valid and ready")
async def purge_consumed_events(
conn: OutboxConnection,
*,
retention_days: int,
batch_size: int = DEFAULT_BATCH_SIZE,
max_batches: int = 1,
pause_seconds: float = DEFAULT_PAUSE_SECONDS,
) -> int:
if not 1 <= retention_days <= 3650:
raise ValueError("retention_days must be between 1 and 3650")
if batch_size < 1 or max_batches < 1 or pause_seconds < 0:
raise ValueError("batch_size/max_batches must be positive and pause_seconds non-negative")
if await retention_index_state(conn) != "ready":
logger.warning_event(
"Skipping Earth outbox retention because its index is unavailable",
event="earth.outbox.retention_index_unavailable",
)
return 0
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
deleted_total = 0
for batch_number in range(max_batches):
deleted = int(await conn.fetchval(DELETE_BATCH_SQL, cutoff, batch_size) or 0)
deleted_total += deleted
if deleted < batch_size:
break
if batch_number + 1 < max_batches and pause_seconds:
await asyncio.sleep(pause_seconds)
if deleted_total:
logger.info_event(
"Purged consumed Earth outbox events",
event="earth.outbox.retention_purged",
context={"deleted": deleted_total, "retention_days": retention_days},
)
return deleted_total
async def compact_consumed_events(
conn: OutboxConnection,
*,
retention_days: int,
lock_timeout_seconds: int = 10,
statement_timeout_seconds: int = 900,
) -> tuple[int, int]:
"""One-time atomic rewrite; blocks table access until the transaction commits."""
if not 1 <= retention_days <= 3650:
raise ValueError("retention_days must be between 1 and 3650")
if lock_timeout_seconds < 1 or statement_timeout_seconds < 1:
raise ValueError("timeouts must be positive")
cutoff = datetime.now(UTC) - timedelta(days=retention_days)
async with conn.transaction():
await conn.fetchval(
"SELECT set_config('lock_timeout', $1, true)", f"{lock_timeout_seconds}s"
)
await conn.fetchval(
"SELECT set_config('statement_timeout', $1, true)", f"{statement_timeout_seconds}s"
)
await conn.execute("LOCK TABLE earth_data_change_events IN ACCESS EXCLUSIVE MODE")
await conn.execute(
"""
CREATE TEMP TABLE earth_outbox_retained ON COMMIT DROP AS
SELECT * FROM earth_data_change_events
WHERE consumed_at IS NULL OR consumed_at >= $1
""",
cutoff,
)
retained = await conn.fetchrow("""
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
FROM earth_outbox_retained
""")
if retained is None:
raise RuntimeError("Unable to verify retained Earth outbox rows")
await conn.execute("TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT")
await conn.execute(
"INSERT INTO earth_data_change_events SELECT * FROM earth_outbox_retained"
)
restored = await conn.fetchrow("""
SELECT count(*) AS total, count(*) FILTER (WHERE consumed_at IS NULL) AS pending
FROM earth_data_change_events
""")
if restored is None or dict(retained) != dict(restored):
raise RuntimeError("Earth outbox compact verification failed; transaction rolled back")
return int(retained["total"]), int(retained["pending"])

View File

@@ -0,0 +1,198 @@
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
from __future__ import annotations
from dataclasses import dataclass
import math
import re
from typing import Any
from app.core.countries import normalize_country
_COUNTRY_LABEL_OVERRIDES = {
"korea republic of": "韩国",
}
_REGION_ONLY_LABELS = {
"europe",
"iowa",
"maryland",
"new mexico",
"texas",
"the nordics",
"virginia",
"washington",
}
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
re.IGNORECASE,
)
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
re.IGNORECASE | re.DOTALL,
)
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
)
_NUMBERED_STREET_PATTERN = re.compile(
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
r"highway|hwy|way|parkway|pkwy|chome)\b|"
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
re.IGNORECASE,
)
_POSTAL_CODE_PATTERN = re.compile(
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
re.IGNORECASE,
)
_SITE_CONTEXT_PATTERN = re.compile(
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
re.IGNORECASE,
)
_COORDINATE_CUE_PATTERN = re.compile(
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
re.IGNORECASE | re.DOTALL,
)
@dataclass(frozen=True)
class EpochLocationAssessment:
precision: str
reason: str
@property
def is_renderable(self) -> bool:
return self.precision in {"precise", "site", "city"}
def _normalized_label(value: Any) -> str:
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
def _country_from_label(value: Any) -> str | None:
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
_normalized_label(value)
)
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
if isinstance(raw, bool):
return None
try:
number = float(raw)
except (TypeError, ValueError):
return None
return number if math.isfinite(number) and abs(number) <= bound else None
def _has_explicit_coordinate_pair(text: str) -> bool:
hemispheres = {}
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
hemisphere = match.group("hemisphere").upper()
degrees = int(match.group("degrees"))
minutes = int(match.group("minutes"))
seconds = float(match.group("seconds") or 0)
bound = 90 if hemisphere in {"N", "S"} else 180
if degrees <= bound and minutes < 60 and seconds < 60:
hemispheres[hemisphere] = degrees
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
return True
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
if labeled_pair:
latitude = float(labeled_pair.group("latitude"))
longitude = float(labeled_pair.group("longitude"))
return abs(latitude) <= 90 and abs(longitude) <= 180
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
if not decimal_pair:
return False
latitude, longitude = map(float, decimal_pair.groups())
return abs(latitude) <= 90 and abs(longitude) <= 180
def _has_address_evidence(text: str) -> bool:
if _NUMBERED_STREET_PATTERN.search(text):
return True
return bool(
_POSTAL_CODE_PATTERN.search(text)
and ("\n" in text or text.count(",") >= 2)
and _SITE_CONTEXT_PATTERN.search(text)
)
def assess_epoch_location(
location: Any,
*,
country: Any = None,
raw_country: Any = None,
) -> EpochLocationAssessment:
"""Classify source coordinates without upgrading their source text.
Epoch's CSV supplies coordinates even when ``Location`` contains only a
country. Those coordinates are useful source evidence, but are not a
facility point and must not enter the renderable location pipeline.
"""
if isinstance(location, dict):
latitude = _coordinate(location, ("lat", "latitude"), 90)
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
has_coordinates = latitude is not None and longitude is not None
return EpochLocationAssessment(
"precise" if has_coordinates else "unknown",
"complete explicit coordinate object"
if has_coordinates
else "coordinate object lacks a complete valid latitude/longitude pair",
)
text = str(location or "").strip()
if not text:
return EpochLocationAssessment("unknown", "Epoch Location is empty")
normalized = _normalized_label(text)
normalized_country = normalize_country(country)
source_country = normalize_country(raw_country)
location_country = _country_from_label(text)
if location_country or (
raw_country and normalized == _normalized_label(raw_country)
):
matched = location_country or source_country or normalized_country
return EpochLocationAssessment(
"country",
f"Epoch Location identifies country only: {matched or text}",
)
if normalized in _REGION_ONLY_LABELS:
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
if (
len(comma_parts) == 2
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
and _country_from_label(comma_parts[1])
):
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
if normalized in _UNLOCATABLE_LABELS:
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
if normalized.isdigit():
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
if _has_explicit_coordinate_pair(text):
return EpochLocationAssessment(
"precise", "Epoch Location contains a complete valid coordinate pair"
)
if _COORDINATE_CUE_PATTERN.search(text):
return EpochLocationAssessment(
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
)
if _has_address_evidence(text):
return EpochLocationAssessment("precise", "Epoch Location contains an address")
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
return EpochLocationAssessment("city", "Epoch Location identifies a locality")

View File

@@ -0,0 +1,186 @@
"""Authenticated model discovery shared by refresh and connection checks."""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime
from urllib.parse import urlsplit, urlunsplit
import httpx
CATALOG_TIMEOUT_SECONDS = 30
CATALOG_MAX_PAGES = 100
CATALOG_PAGE_SIZE = 100
CATALOG_REQUEST_ATTEMPTS = 2
CATALOG_RETRY_DELAY_SECONDS = 0.2
SUPPORTED_PROVIDER_APIS = {
"openai-completions",
"openai-responses",
"anthropic-messages",
"ollama-generate",
}
class LLMProviderCatalogError(RuntimeError):
"""A safe, user-facing catalog failure with no upstream response or credentials."""
@dataclass(frozen=True)
class ModelCatalog:
url: str
models: list[str]
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
parts = urlsplit(base_url.strip())
if parts.scheme not in {"http", "https"} or not parts.hostname:
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
if parts.username or parts.password or parts.query or parts.fragment:
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
if provider_api not in SUPPORTED_PROVIDER_APIS:
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
path = parts.path.rstrip("/")
if provider_api == "ollama-generate":
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
path = "/api/v1/models"
else:
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
path += "/v1"
path += "/models"
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
def catalog_error_message(exc: Exception) -> str:
if isinstance(exc, LLMProviderCatalogError):
return str(exc)
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
return "模型目录请求超时,请检查网络后重试。"
if isinstance(exc, httpx.HTTPStatusError):
code = exc.response.status_code
messages = {
401: "API Key 验证失败,请检查当前供应商的凭证。",
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
429: "供应商请求限流,请稍后重试。",
}
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
if isinstance(exc, httpx.RequestError):
return "无法连接模型目录,请检查基础地址和网络。"
return "模型目录响应无效,请稍后重试。"
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
if not isinstance(payload, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
envelope = payload.get("output", payload)
if not isinstance(envelope, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
rows = envelope.get("data", envelope.get("models"))
if not isinstance(rows, list):
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
if any(not isinstance(row, dict) for row in rows):
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
return rows, envelope
def _model_id(row: dict[str, object]) -> str:
value = row.get("id") or row.get("model") or row.get("name")
if not isinstance(value, str) or not value.strip():
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
return value.strip()
def _model_date(row: dict[str, object]) -> float:
value = row.get("created_at") or row.get("published_time") or row.get("created")
if isinstance(value, (int, float)):
return float(value)
if isinstance(value, str):
try:
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
except ValueError:
pass
return 0
async def _get_catalog_page(
client: httpx.AsyncClient,
url: str,
headers: dict[str, str],
params: dict[str, str | int],
) -> object:
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
try:
response = await client.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
except httpx.HTTPStatusError as exc:
if attempt or exc.response.status_code not in {502, 503, 504}:
raise
except httpx.TransportError:
if attempt:
raise
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
raise LLMProviderCatalogError("模型目录请求失败。")
async def fetch_model_catalog(
provider: str,
base_url: str,
provider_api: str,
api_key: str = "",
anthropic_version: str = "2023-06-01",
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
) -> ModelCatalog:
url = model_catalog_url(provider, base_url, provider_api)
public_catalog = provider in {"opencode-go", "openrouter"}
if not api_key and provider_api != "ollama-generate" and not public_catalog:
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
if provider_api == "anthropic-messages" and provider not in {
"opencode-go",
"openrouter",
"alibaba",
"moonshotai",
}:
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
elif api_key:
headers["Authorization"] = f"Bearer {api_key}"
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
params: dict[str, str | int] = {}
if native_dashscope:
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
rows_by_id: dict[str, dict[str, object]] = {}
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
# Bound the complete pagination/retry cycle, not just each individual request.
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
for page in range(CATALOG_MAX_PAGES):
payload = await _get_catalog_page(client, url, headers, params)
rows, envelope = _model_rows(payload)
previous_count = len(rows_by_id)
for row in rows:
rows_by_id[_model_id(row)] = row
has_more = envelope.get("has_more") is True
if native_dashscope:
total = envelope.get("total")
if not isinstance(total, int) or total < 0:
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
has_more = len(rows_by_id) < total
params["page_no"] = page + 2
elif has_more:
cursor = envelope.get("last_id")
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
params["after_id"] = cursor
if not has_more:
models = sorted(
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
)
# An empty Ollama catalog is valid: no models have been installed yet.
if not models and provider_api != "ollama-generate":
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
return ModelCatalog(url=url, models=models)
if len(rows_by_id) == previous_count:
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")

View File

@@ -3,22 +3,30 @@
from __future__ import annotations
from typing import Any
from urllib.parse import urlsplit
import httpx
MODELS_DEV_URL = "https://models.dev/api.json"
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
class LLMProviderCatalogError(RuntimeError):
"""The upstream catalog cannot supply a usable model list."""
from app.services.llm_model_catalog import fetch_model_catalog
OPENCODE_GO_MODEL_PROVIDER_APIS = {
"minimax-m3": "anthropic-messages",
"qwen3.8-max": "anthropic-messages",
"qwen3.8-flash": "anthropic-messages",
"qwen3.7-max": "anthropic-messages",
"qwen3.7-plus": "anthropic-messages",
"qwen3.6-plus": "anthropic-messages",
"grok-4.6": "openai-responses",
"gpt-5.6-luna": "openai-responses",
"muse-spark-1.3-contributor": "openai-responses",
"muse-spark-1.2-contributor": "openai-responses",
"minimax-m2.7": "anthropic-messages",
"minimax-m2.5": "anthropic-messages",
}
OPENCODE_GO_FALLBACK_MODELS = [
"minimax-m3",
"kimi-k3",
"glm-5.3",
"qwen3.8-max",
"gpt-5.6-luna",
"minimax-m2.7",
"minimax-m2.5",
"kimi-k2.6",
@@ -34,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
]
OPENAI_MODEL_PROVIDER_APIS = {
model: "openai-responses"
for model in [
"gpt-6-astra",
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
"gpt-5.1",
"gpt-5.1-codex",
]
}
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"minimax": {
"provider": "minimax",
"label": "MiniMax",
"provider_api": "anthropic-messages",
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-M2.7",
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
"model": "MiniMax-M3",
"models": [
"MiniMax-M3",
"MiniMax-M2.7",
"MiniMax-M2.7-highspeed",
"MiniMax-M2.5",
"MiniMax-M2.5-highspeed",
"MiniMax-M2.1",
"MiniMax-M2.1-highspeed",
"MiniMax-M2",
],
"api_key_env": "MINIMAX_API_KEY",
"source": "fallback",
},
"openai": {
"provider": "openai",
"label": "OpenAI",
"provider_api": "openai-completions",
"provider_api": "openai-responses",
"base_url": "https://api.openai.com/v1",
"model": "gpt-5.1",
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
"model": "gpt-6-astra",
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
"api_key_env": "OPENAI_API_KEY",
"source": "fallback",
},
@@ -60,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Anthropic",
"provider_api": "anthropic-messages",
"base_url": "https://api.anthropic.com/v1",
"model": "claude-sonnet-4-6",
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
"model": "claude-opus-5",
"models": ["claude-opus-5", "claude-sonnet-4-6"],
"api_key_env": "ANTHROPIC_API_KEY",
"source": "fallback",
},
@@ -70,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "DeepSeek",
"provider_api": "openai-completions",
"base_url": "https://api.deepseek.com/v1",
"model": "deepseek-chat",
"models": ["deepseek-chat", "deepseek-reasoner"],
"model": "deepseek-flash",
"models": ["deepseek-flash", "deepseek-v4-pro"],
"api_key_env": "DEEPSEEK_API_KEY",
"source": "fallback",
},
@@ -80,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Alibaba Qwen / DashScope",
"provider_api": "openai-completions",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"model": "qwen3-max",
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
"model": "qwen3.8-max",
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
"api_key_env": "DASHSCOPE_API_KEY",
"source": "fallback",
},
@@ -90,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Moonshot AI / Kimi",
"provider_api": "openai-completions",
"base_url": "https://api.moonshot.ai/v1",
"model": "kimi-k2.5",
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
"model": "kimi-k3",
"models": ["kimi-k3", "kimi-k2.5"],
"api_key_env": "MOONSHOT_API_KEY",
"source": "fallback",
},
@@ -128,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
},
}
MODELS_DEV_PROVIDER_KEYS = {
"minimax": "minimax",
"openai": "openai",
"anthropic": "anthropic",
"deepseek": "deepseek",
"alibaba": "alibaba",
"moonshotai": "moonshotai",
"openrouter": "openrouter",
}
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
@@ -157,76 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
}
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
async def refresh_llm_provider_preset(
provider: str,
api_key: str | None = None,
*,
base_url: str | None = None,
provider_api: str | None = None,
anthropic_version: str = "2023-06-01",
) -> dict[str, Any]:
fallback = get_fallback_llm_provider_preset(provider)
if fallback["provider"] == "opencode-go":
headers = {"User-Agent": "Planet/1.0"}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
OPENCODE_GO_MODELS_URL,
headers=headers,
)
response.raise_for_status()
payload = response.json()
data = payload.get("data") if isinstance(payload, dict) else []
model_ids = [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
]
if not model_ids:
raise LLMProviderCatalogError("The provider returned an empty model catalog")
return {
**fallback,
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
"models": model_ids,
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
"source": OPENCODE_GO_MODELS_URL,
}
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
if not models_dev_key:
raise LLMProviderCatalogError("Live catalog refresh is unavailable for this provider")
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
MODELS_DEV_URL,
headers={"User-Agent": "Planet/1.0"},
)
response.raise_for_status()
catalog = response.json()
upstream = catalog.get(models_dev_key) if isinstance(catalog, dict) else None
if not isinstance(upstream, dict):
raise LLMProviderCatalogError("The provider is missing from the model catalog")
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
# Catalog insertion order is not release order; old entries can appear first.
model_ids = sorted(
(
model_id
for model_id, model in upstream_models.items()
if model_id and isinstance(model, dict)
),
key=lambda model_id: (str(upstream_models[model_id].get("release_date") or ""), model_id),
reverse=True,
resolved_base_url = base_url or fallback["base_url"]
resolved_api = provider_api or fallback["provider_api"]
catalog = await fetch_model_catalog(
fallback["provider"],
resolved_base_url,
resolved_api,
api_key or "",
anthropic_version,
)
if not model_ids:
raise LLMProviderCatalogError("The provider returned an empty model catalog")
base_url = upstream.get("api") or fallback["base_url"]
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
base_url = "https://api.deepseek.com/v1"
refreshed = {
model_provider_apis = (
_opencode_go_model_provider_apis(catalog.models)
if fallback["provider"] == "opencode-go"
else fallback.get("model_provider_apis", {})
)
if (
fallback["provider"] == "openai"
and urlsplit(resolved_base_url).hostname != "api.openai.com"
):
model_provider_apis = {}
return {
**fallback,
"label": upstream.get("name") or fallback["label"],
"base_url": base_url,
"model": model_ids[0],
"models": model_ids,
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
"source": MODELS_DEV_URL,
"base_url": resolved_base_url,
"provider_api": resolved_api,
"model": catalog.models[0] if catalog.models else "",
"models": catalog.models,
"model_provider_apis": model_provider_apis,
"source": catalog.url,
}
return refreshed

View File

@@ -635,6 +635,7 @@ async def _run_assistant_message(
constraints=_collect_constraints(payload.constraints),
context={
"source": "playground",
"session_id": session_id,
"preset": payload.selected_preset_key,
"conversation_history": conversation_history,
"history_size": len(conversation_history),

View File

@@ -2,9 +2,11 @@
import asyncio
from datetime import UTC, datetime, timedelta
import time
from typing import Any, Dict, Optional
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.date import DateTrigger
from apscheduler.triggers.interval import IntervalTrigger
from sqlalchemy import select
@@ -14,19 +16,44 @@ from app.db.session import async_session_factory
from app.core.time import to_iso8601_utc
from app.models.datasource import DataSource
from app.models.task import CollectionTask
from app.services.business_logs import emit_business_log, emit_business_log_background, exception_context
from app.services.business_logs import (
emit_business_log,
emit_business_log_background,
exception_context,
)
from app.services.collectors.registry import collector_registry
from app.services.datasource_connectivity import (
build_builtin_connectivity_checksum,
get_builtin_effective_candidate,
save_connectivity_success,
)
from app.services.earth_news_refresh import (
REFRESH_INTERVAL_SECONDS as EARTH_NEWS_REFRESH_INTERVAL_SECONDS,
create_earth_news_refresh_status,
refresh_earth_news_all_regions,
refresh_earth_news_region,
set_earth_news_refresh_status,
touch_earth_news_refresh_status,
REFRESH_STATUS_HEARTBEAT_SECONDS as EARTH_NEWS_STATUS_HEARTBEAT_SECONDS,
)
from app.services.earth_news import REGION_ANCHORS
logger = get_logger(__name__)
scheduler = AsyncIOScheduler()
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
EARTH_NEWS_REFRESH_JOB_ID = "earth_news_rss_refresh"
_pending_earth_news_refreshes: dict[str, dict[str, Any]] = {}
_running_earth_news_refreshes: set[str] = set()
_rerun_earth_news_refreshes: dict[str, dict[str, Any]] = {}
EARTH_NEWS_REFRESH_WAIT_SECONDS = 300
EARTH_NEWS_REFRESH_POLL_SECONDS = 1
EARTH_NEWS_REFRESH_PUBLIC_ERRORS = {
"source_unavailable": "No news source could be refreshed for this region. Please retry.",
"timeout": "News sources timed out. Please try again.",
"refresh_failed": "Could not refresh news. Please try again.",
}
def _collector_task_name(collector_name: str) -> str:
@@ -86,7 +113,10 @@ async def _apply_datasource_schedule(datasource: DataSource, session) -> None:
logger.info_event(
"Scheduled collector",
event="collector.schedule.updated",
context={"collector_name": datasource.source, "frequency_minutes": datasource.frequency_minutes},
context={
"collector_name": datasource.source,
"frequency_minutes": datasource.frequency_minutes,
},
)
else:
logger.info_event(
@@ -133,7 +163,11 @@ async def run_collector_task(collector_name: str):
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "skipped"},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "skipped",
},
)
return
@@ -153,9 +187,8 @@ async def run_collector_task(collector_name: str):
if started_at is not None and started_at.tzinfo is None:
started_at = started_at.replace(tzinfo=UTC)
is_stale = (
started_at is not None
and (now - started_at) > timedelta(minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES)
is_stale = started_at is not None and (now - started_at) > timedelta(
minutes=RUNNING_TASK_GUARD_TIMEOUT_MINUTES
)
if not is_stale:
logger.warning_event(
@@ -189,9 +222,7 @@ async def run_collector_task(collector_name: str):
existing_running.phase = "failed"
existing_running.completed_at = now
existing_running.error_message = (
f"{existing_error}\n{stale_reason}".strip()
if existing_error
else stale_reason
f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
)
await db.commit()
logger.warning_event(
@@ -231,7 +262,11 @@ async def run_collector_task(collector_name: str):
category="collector",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource_id, "status": "running"},
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"status": "running",
},
)
task_result = await collector.run(db)
datasource = await db.get(DataSource, datasource_id)
@@ -265,7 +300,11 @@ async def run_collector_task(collector_name: str):
logger.info_event(
"Collector completed",
event="collector.run.completed",
context={"collector_name": collector_name, "datasource_id": datasource_id, "result": task_result},
context={
"collector_name": collector_name,
"datasource_id": datasource_id,
"result": task_result,
},
)
await emit_business_log(
logger,
@@ -300,7 +339,11 @@ async def run_collector_task(collector_name: str):
level="warning",
service="scheduler",
module=__name__,
context={"collector_name": collector_name, "datasource_id": datasource.id, "status": "cancelled"},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "cancelled",
},
)
raise
except Exception as exc:
@@ -312,7 +355,11 @@ async def run_collector_task(collector_name: str):
logger.exception_event(
"Collector failed",
event="collector.run.failed",
context={"collector_name": collector_name, "datasource_id": datasource.id, "error": str(exc)},
context={
"collector_name": collector_name,
"datasource_id": datasource.id,
"error": str(exc),
},
)
await emit_business_log(
logger,
@@ -324,7 +371,11 @@ async def run_collector_task(collector_name: str):
module=__name__,
context=exception_context(
exc,
{"collector_name": collector_name, "datasource_id": datasource.id, "status": "failed"},
{
"collector_name": collector_name,
"datasource_id": datasource.id,
"status": "failed",
},
),
)
@@ -349,7 +400,9 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
task.completed_at = datetime.now(UTC)
existing_error = (task.error_message or "").strip()
cleanup_error = "Marked failed automatically after stale running task cleanup"
task.error_message = f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
task.error_message = (
f"{existing_error}\n{cleanup_error}".strip() if existing_error else cleanup_error
)
if stale_tasks:
await db.commit()
@@ -365,6 +418,17 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
def start_scheduler() -> None:
"""Start the scheduler."""
if not scheduler.running:
scheduler.add_job(
refresh_earth_news_all_regions,
trigger=IntervalTrigger(seconds=EARTH_NEWS_REFRESH_INTERVAL_SECONDS),
id=EARTH_NEWS_REFRESH_JOB_ID,
name="Earth news RSS refresh",
replace_existing=True,
next_run_time=datetime.now(UTC),
max_instances=1,
coalesce=True,
misfire_grace_time=60,
)
scheduler.start()
logger.info_event("Scheduler started", event="scheduler.started")
@@ -376,6 +440,205 @@ def stop_scheduler() -> None:
logger.info_event("Scheduler stopped", event="scheduler.stopped")
def schedule_earth_news_region_refresh(
region: str, *, force: bool = False, refresh_id: str | None = None
) -> bool:
"""Queue one immediate, per-region RSS refresh and merge duplicate requests."""
normalized_region = str(region or "").strip().lower()
if normalized_region not in REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {region}")
if normalized_region in _running_earth_news_refreshes:
rerun = _rerun_earth_news_refreshes.setdefault(
normalized_region, {"force": False, "refresh_id": None}
)
rerun["force"] = rerun["force"] or force
rerun["refresh_id"] = refresh_id or rerun["refresh_id"]
return False
if normalized_region in _pending_earth_news_refreshes:
pending = _pending_earth_news_refreshes[normalized_region]
pending["force"] = pending["force"] or force
pending["refresh_id"] = refresh_id or pending["refresh_id"]
return False
_pending_earth_news_refreshes[normalized_region] = {
"force": force,
"refresh_id": refresh_id,
}
job_id = _earth_news_region_refresh_job_id(normalized_region)
try:
scheduler.add_job(
_run_earth_news_region_refresh,
trigger=DateTrigger(run_date=datetime.now(UTC)),
id=job_id,
name=f"Earth news RSS refresh: {normalized_region}",
kwargs={"region": normalized_region},
replace_existing=False,
max_instances=1,
coalesce=True,
misfire_grace_time=60,
)
except Exception:
_pending_earth_news_refreshes.pop(normalized_region, None)
raise
return True
def request_earth_news_region_refresh(region: str) -> dict[str, Any]:
normalized_region = str(region or "").strip().lower()
if normalized_region not in REGION_ANCHORS:
raise ValueError(f"Unsupported Earth news region: {region}")
status, created = create_earth_news_refresh_status(normalized_region)
if created:
try:
schedule_earth_news_region_refresh(
normalized_region,
force=True,
refresh_id=status["refresh_id"],
)
except Exception:
return set_earth_news_refresh_status(
status["refresh_id"],
normalized_region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
return status
def schedule_earth_news_refresh_for_all_regions(*, force: bool = False) -> None:
for region in REGION_ANCHORS:
schedule_earth_news_region_refresh(region, force=force)
def _earth_news_region_refresh_job_id(region: str) -> str:
return f"{EARTH_NEWS_REFRESH_JOB_ID}:{region}"
async def _run_earth_news_region_refresh(region: str) -> None:
request = _pending_earth_news_refreshes.pop(region, {"force": False, "refresh_id": None})
force = bool(request["force"])
refresh_id = request["refresh_id"]
_running_earth_news_refreshes.add(region)
status_heartbeat: asyncio.Task[None] | None = None
try:
if refresh_id:
set_earth_news_refresh_status(refresh_id, region, "running")
if not touch_earth_news_refresh_status(refresh_id, region):
return
status_heartbeat = asyncio.create_task(
_earth_news_refresh_status_heartbeat(refresh_id, region)
)
deadline = time.monotonic() + EARTH_NEWS_REFRESH_WAIT_SECONDS
while True:
result = await refresh_earth_news_region(region, force=force)
if result["state"] != "busy":
break
if time.monotonic() >= deadline:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
return
await asyncio.sleep(EARTH_NEWS_REFRESH_POLL_SECONDS)
if result["state"] == "coordination_unavailable":
error_code = "refresh_failed"
elif result["state"] == "failed":
error_code = result.get("error_code") or "source_unavailable"
elif result["state"] == "completed" and result.get("successful_sources", 0) > 0:
set_earth_news_refresh_status(refresh_id, region, "succeeded")
return
else:
error_code = "source_unavailable"
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": error_code,
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS.get(
error_code, EARTH_NEWS_REFRESH_PUBLIC_ERRORS["source_unavailable"]
),
},
)
else:
await refresh_earth_news_region(region, force=force)
except asyncio.CancelledError:
if refresh_id:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
raise
except Exception as exc:
logger.warning_event(
"Earth news tracked refresh failed",
event="earth_news.rss.tracked_refresh_failed",
context={"region": region, "error": str(exc)},
)
if refresh_id:
try:
set_earth_news_refresh_status(
refresh_id,
region,
"failed",
error={
"code": "refresh_failed",
"message": EARTH_NEWS_REFRESH_PUBLIC_ERRORS["refresh_failed"],
},
)
except Exception as status_exc:
logger.warning_event(
"Earth news refresh status update failed",
event="earth_news.rss.refresh_status_update_failed",
context={"region": region, "error": str(status_exc)},
)
finally:
if status_heartbeat is not None:
status_heartbeat.cancel()
try:
await status_heartbeat
except asyncio.CancelledError:
pass
_running_earth_news_refreshes.discard(region)
rerun = _rerun_earth_news_refreshes.pop(region, None)
if rerun is not None and scheduler.running:
schedule_earth_news_region_refresh(
region,
force=rerun["force"],
refresh_id=rerun["refresh_id"],
)
async def _earth_news_refresh_status_heartbeat(refresh_id: str, region: str) -> None:
try:
while True:
await asyncio.sleep(EARTH_NEWS_STATUS_HEARTBEAT_SECONDS)
if not touch_earth_news_refresh_status(refresh_id, region):
return
except asyncio.CancelledError:
raise
except Exception as exc:
logger.warning_event(
"Earth news refresh status heartbeat failed",
event="earth_news.rss.refresh_status_heartbeat_failed",
context={"region": region, "error": str(exc)},
)
async def sync_scheduler_with_datasources() -> None:
"""Synchronize scheduler jobs with datasource table."""
async with async_session_factory() as db:
@@ -384,7 +647,10 @@ async def sync_scheduler_with_datasources() -> None:
configured_sources = {datasource.source for datasource in datasources}
for job in list(scheduler.get_jobs()):
if job.id not in configured_sources:
is_earth_news_refresh = job.id == EARTH_NEWS_REFRESH_JOB_ID or job.id.startswith(
f"{EARTH_NEWS_REFRESH_JOB_ID}:"
)
if job.id not in configured_sources and not is_earth_news_refresh:
scheduler.remove_job(job.id)
for datasource in datasources:
@@ -461,7 +727,9 @@ def run_collector_now(collector_name: str) -> bool:
return False
try:
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
task = asyncio.create_task(
run_collector_task(collector_name), name=_collector_task_name(collector_name)
)
RUNNING_COLLECTOR_TASKS[collector_name] = task
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:

View File

@@ -15,6 +15,12 @@ DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
TV_SETTINGS_CATEGORY = "tv"
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
UNSUPPORTED_CCTV4_STREAM = (
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
)
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
DEFAULT_TV_SETTINGS = {
"default_source_id": DEFAULT_TV_SOURCE_ID,
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
"sources": [
{
"id": "cctv4",
"name": "CCTV-4 中文国际",
"name": "CCTV-4 中文国际(美洲版)",
"provider": "CCTV",
"region": "China",
"language": "zh-CN",
"source_type": "hls",
"embed_url": "https://tv.cctv.com/live/cctv4/",
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
"homepage_url": "https://tv.cctv.com/live/cctv4/",
"embed_url": "",
"stream_url": CCTV4_STREAM,
"use_proxy": False,
"homepage_url": CCTV4_HOMEPAGE,
"poster_url": "",
"is_enabled": True,
"is_fallback": True,
"sort_order": 10,
"collector_source": None,
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
"notes": CCTV4_PLAYBACK_NOTE,
},
{
"id": "reuters-tv",
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
payload = dict(source or {})
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
payload.get("stream_url")
) == UNSUPPORTED_CCTV4_STREAM:
payload.update(
name="CCTV-4 中文国际(美洲版)",
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
use_proxy=False,
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
)
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
source_type = _clean_text(payload.get("source_type")).lower()
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
"source_type": source_type,
"embed_url": _clean_url(payload.get("embed_url")),
"stream_url": _clean_url(payload.get("stream_url")),
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
"homepage_url": _clean_url(payload.get("homepage_url")),
"poster_url": _clean_url(payload.get("poster_url")),
"youtube_video_id": youtube_video_id,

View File

@@ -8,6 +8,37 @@ from unittest.mock import AsyncMock, MagicMock
from sqlalchemy.ext.asyncio import AsyncSession
@pytest.fixture(autouse=True)
def persisted_test_logs(monkeypatch):
"""Keep application log persistence in memory, including business-log calls."""
from app.models.system_log import ObservabilityEventGroup
from app.services import persistent_logs
entries = []
groups = {}
class LogSession:
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, traceback):
return False
def add(self, item):
entries.append(item)
if isinstance(item, ObservabilityEventGroup):
groups[item.fingerprint] = item
async def get(self, model, key):
return groups.get(key) if model is ObservabilityEventGroup else None
async def commit(self):
return None
monkeypatch.setattr(persistent_logs, "async_session_factory", LogSession)
return entries
@pytest.fixture(autouse=True)
def bgp_collector_location_cache():
"""Mirror app startup seeding for tests that call sync BGP helpers."""

View File

@@ -0,0 +1,72 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from fastapi import FastAPI
from pydantic import ValidationError
from app.api.v1.auth import router
from app.core import security
from app.db.session import DEFAULT_LOGIN_USERS, get_db
from app.models.user import User
from app.schemas.user import UserRegister
async def identity_request(monkeypatch, user, token=None):
result = MagicMock()
result.scalar_one_or_none.return_value = user
db = AsyncMock()
db.execute.return_value = result
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
app = FastAPI()
app.include_router(router, prefix="/auth")
async def override_db():
yield db
app.dependency_overrides[get_db] = override_db
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
return await client.get(
"/auth/me",
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
created = datetime(2026, 9, 21, tzinfo=UTC)
user = User(
id=1, username=default["username"], email=default["email"],
role=default["role"], password_hash="not-exposed", is_active=True,
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
)
response = await identity_request(monkeypatch, user)
assert response.status_code == 200
payload = response.json()
assert payload["email"] == default["email"]
assert payload["email_verified"] is True
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
assert payload["gatekeeper_groups"] == ["ops"]
assert "password_hash" not in payload
@pytest.mark.asyncio
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
response = await identity_request(monkeypatch, user)
assert response.status_code == 401
@pytest.mark.asyncio
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
assert response.status_code == 401
def test_registration_still_validates_email():
with pytest.raises(ValidationError):
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")

View File

@@ -1,5 +1,6 @@
"""Tests for BGP observability helpers."""
import ipaddress
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
@@ -21,13 +22,20 @@ from app.services.collectors.bgp_common import (
create_bgp_anomalies_for_batch,
save_bgp_observations_for_batch,
)
from app.services.bgp_enrichment import enrich_bgp_events_for_batch, extract_bgp_network_fields
from app.services.bgp_enrichment import (
enrich_bgp_events_for_batch,
extract_bgp_network_fields,
lookup_prefix_geography,
)
from app.services.bgp_incidents import (
create_bgp_incidents_for_anomalies,
infer_related_infrastructure,
)
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.api.v1.visualization import convert_bgp_incidents_to_geojson, build_incident_geography_hints
from app.api.v1.visualization import (
convert_bgp_incidents_to_geojson,
build_incident_geography_hints,
)
from app.models.bgp_anomaly import BGPAnomaly
from app.models.collected_data import CollectedData
from app.models.bgp_incident import BGPIncident
@@ -58,8 +66,7 @@ class _FakeResult:
def all(self):
if self._rows and all(isinstance(row, BGPObservation) for row in self._rows):
return [
(row.prefix, row.origin_asn, row.collector, row.collector_geo)
for row in self._rows
(row.prefix, row.origin_asn, row.collector, row.collector_geo) for row in self._rows
]
return self._rows
@@ -93,6 +100,8 @@ class _FakeAsyncSession:
self.commits = 0
async def execute(self, _stmt, _params=None):
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
return _FakeResult([])
if not self._results:
return _FakeResult([])
return _FakeResult(self._results.pop(0))
@@ -107,6 +116,49 @@ class _FakeAsyncSession:
self.commits += 1
class _StrictFakeAsyncSession(_FakeAsyncSession):
async def execute(self, stmt, params=None):
if not str(stmt).lstrip().startswith("SET LOCAL jit") and not self._results:
raise AssertionError(f"Unexpected BGP query: {stmt}")
return await super().execute(stmt, params)
@pytest.mark.asyncio
async def test_prefix_geography_keeps_source_priority_and_ipv6_range_parameters():
calls = []
class GeoSession:
async def execute(self, _stmt, params=None):
if str(_stmt).lstrip().startswith("SET LOCAL jit"):
return _FakeResult([])
calls.append(dict(params))
if params["source"] == "nro_delegated_prefix_geo":
return _FakeResult(
[
{
"metadata": {
"country_code": "GB",
"prefix": "2001:db8::/32",
}
}
]
)
return _FakeResult([])
network = ipaddress.ip_network("2001:db8:1::/48")
result = await lookup_prefix_geography(GeoSession(), [str(network), "invalid-prefix"])
assert result[str(network)]["source"] == "nro_delegated_stats"
assert [call["source"] for call in calls] == [
"opengeofeed_prefix_geo",
"iptoasn_prefix_geo",
"nro_delegated_prefix_geo",
]
assert all(call["family"] == "ipv6" for call in calls)
assert all(call["range_start"] == str(network.network_address) for call in calls)
assert all(call["range_end"] == str(network.broadcast_address) for call in calls)
def test_normalize_bgp_event_from_live_payload():
event = normalize_bgp_event(
{
@@ -449,7 +501,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
"prefix": "203.0.113.0/24",
"origin_asn": 64496,
"as_path": [64500, 64496],
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
"collector_location": {
"country": "NL",
"city": "Amsterdam",
"latitude": 52.3,
"longitude": 4.9,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
}
},
@@ -460,7 +517,12 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
"prefix": "203.0.113.0/24",
"origin_asn": 64496,
"as_path": [64510, 64520, 64530, 64540, 64496],
"collector_location": {"country": "GB", "city": "London", "latitude": 51.5, "longitude": -0.1},
"collector_location": {
"country": "GB",
"city": "London",
"latitude": 51.5,
"longitude": -0.1,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "GB"}]}},
}
},
@@ -481,7 +543,9 @@ def test_detect_route_leak_anomalies_creates_candidate_for_divergent_long_paths(
def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
base_timestamp = datetime(2026, 3, 27, 0, 0, tzinfo=UTC)
events = []
for index, event_type in enumerate(["announcement", "withdrawal", "announcement", "withdrawal"]):
for index, event_type in enumerate(
["announcement", "withdrawal", "announcement", "withdrawal"]
):
events.append(
{
"metadata": {
@@ -491,7 +555,12 @@ def test_detect_path_flap_anomalies_creates_signal_for_repeated_state_changes():
"prefix": "198.51.100.0/24",
"origin_asn": 64512,
"as_path": [64500 + index, 64512] if event_type == "announcement" else [],
"collector_location": {"country": "NL", "city": "Amsterdam", "latitude": 52.3, "longitude": 4.9},
"collector_location": {
"country": "NL",
"city": "Amsterdam",
"latitude": 52.3,
"longitude": 4.9,
},
"enrichment": {"prefix_scope": {"regions": [{"country": "NL"}]}},
}
}
@@ -920,7 +989,9 @@ async def test_create_bgp_incidents_for_anomalies_refreshes_existing_incident():
with patch(
"app.services.bgp_incidents.infer_related_infrastructure",
new=AsyncMock(return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}),
new=AsyncMock(
return_value={"related_cables": [{"landing_point": "NYC"}], "related_ixps": []}
),
):
created = await create_bgp_incidents_for_anomalies(
db,
@@ -1132,13 +1203,17 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
source="ris_live_bgp",
extra_data={"prefix": "203.0.113.0/24", "origin_asn": 64496},
)
db = _FakeAsyncSession([
[],
[],
[],
[previous_record],
[],
])
db = _StrictFakeAsyncSession(
[
[], # Historical BGP observations
[], # OpenGeoFeed prefix geography
[], # IPtoASN prefix geography
[], # NRO delegated prefix geography
[], # PeeringDB ASN profiles
[previous_record], # Prior collected snapshot for origin-change detection
[], # Existing anomaly entity keys
]
)
events = [
{
"reference_date": "2026-03-30T10:00:00Z",
@@ -1178,6 +1253,7 @@ async def test_create_bgp_anomalies_for_batch_calls_incident_aggregation():
assert anomaly.anomaly_type == "origin_change"
assert anomaly.prefix == "203.0.113.0/24"
incident_mock.assert_awaited_once()
assert not db._results
@pytest.mark.asyncio
@@ -1197,14 +1273,18 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
origin_asn=64496,
new_origin_asn=64497,
)
db = _FakeAsyncSession([
[],
[],
[],
[previous_record],
[existing_key],
[existing_anomaly],
])
db = _StrictFakeAsyncSession(
[
[], # Historical BGP observations
[], # OpenGeoFeed prefix geography
[], # IPtoASN prefix geography
[], # NRO delegated prefix geography
[], # PeeringDB ASN profiles
[previous_record], # Prior collected snapshot for origin-change detection
[existing_key],
[existing_anomaly],
]
)
events = [
{
"reference_date": "2026-03-30T10:00:00Z",
@@ -1235,6 +1315,7 @@ async def test_create_bgp_anomalies_for_batch_skips_existing_entity_keys():
assert created == 0
assert len(db.added) == 0
incident_mock.assert_awaited_once()
assert not db._results
async def _bgp_test_client(db_session):
@@ -1242,7 +1323,9 @@ async def _bgp_test_client(db_session):
yield db_session
def override_get_current_user():
return User(id=1, username="testuser", email="test@example.com", password_hash="x", role="admin")
return User(
id=1, username="testuser", email="test@example.com", password_hash="x", role="admin"
)
app.dependency_overrides[get_db] = override_get_db
app.dependency_overrides[get_current_user] = override_get_current_user
@@ -1264,7 +1347,7 @@ async def test_bgp_events_api_lists_observations():
origin_asn=64496,
observed_at=datetime(2026, 3, 30, 10, 0, tzinfo=UTC),
)
db = _FakeAsyncSession([[observation]])
db = _StrictFakeAsyncSession([[1], [observation]])
client = await _bgp_test_client(db)
try:
@@ -1278,6 +1361,7 @@ async def test_bgp_events_api_lists_observations():
assert payload["total"] == 1
assert payload["data"][0]["collector"] == "rrc00"
assert payload["data"][0]["prefix"] == "203.0.113.0/24"
assert not db._results
@pytest.mark.asyncio
@@ -1376,7 +1460,18 @@ async def test_bgp_event_summary_api_returns_aggregates():
event_type="withdrawal",
observed_at=datetime(2026, 3, 30, 10, 5, tzinfo=UTC),
)
db = _FakeAsyncSession([[observation_one, observation_two]])
observations = [observation_one, observation_two]
db = _StrictFakeAsyncSession(
[
[len(observations)],
[len({record.collector for record in observations})],
[len({record.prefix for record in observations})],
[
(event_type, sum(record.event_type == event_type for record in observations))
for event_type in sorted({record.event_type for record in observations})
],
]
)
client = await _bgp_test_client(db)
try:
@@ -1392,6 +1487,7 @@ async def test_bgp_event_summary_api_returns_aggregates():
assert payload["prefix_count"] == 2
assert payload["by_type"]["announcement"] == 1
assert payload["by_type"]["withdrawal"] == 1
assert not db._results
@pytest.mark.asyncio

View File

@@ -0,0 +1,22 @@
from unittest.mock import AsyncMock
import pytest
from app.db.bgp_prefix_geo import (
BGP_PREFIX_GEO_FUNCTION_SQL,
BGP_PREFIX_GEO_INDEX_SQL,
install_bgp_prefix_geo_functions,
)
@pytest.mark.asyncio
async def test_fresh_database_installs_lookup_functions_without_large_index():
connection = AsyncMock()
await install_bgp_prefix_geo_functions(connection)
statements = [str(call.args[0]) for call in connection.execute.call_args_list]
assert len(statements) == len(BGP_PREFIX_GEO_FUNCTION_SQL) == 2
assert all("CREATE OR REPLACE FUNCTION" in statement for statement in statements)
assert all("CREATE INDEX" not in statement for statement in statements)
assert "CREATE INDEX CONCURRENTLY" in BGP_PREFIX_GEO_INDEX_SQL

View File

@@ -0,0 +1,333 @@
import io
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import FastAPI, HTTPException
from httpx import ASGITransport, AsyncClient
from PIL import Image
from pydantic import ValidationError
from app.api.v1.card_images import router
from app.core.security import get_current_user
from app.db.session import get_db
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
from app.services.card_image_configuration import (
effective_envelope,
publish_configuration,
resolve_image,
)
from app.services.card_image_defaults import bundled_configuration
from app.services import card_image_files
def configuration():
assets = [
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
for key in ["exact", "model", "group", "category"]
]
rules = [
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
for key, scope, value in [
("exact", "object", "123"),
("model", "model", "m1"),
("group", "group", "starlink"),
("category", "category", "satellite"),
]
]
return CardImageConfiguration(assets=assets, rules=rules)
def test_precedence_disabled_fallback_and_explicit_no_image():
config = configuration()
context = dict(
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
)
for expected in ["exact", "model", "group", "category"]:
assert resolve_image(config, context)["image"]["id"] == expected
next(asset for asset in config.assets if asset.id == expected).enabled = False
assert resolve_image(config, context)["image"] is None
config = configuration()
config.rules[0].asset_id = None
config.rules[0].suppress = True
result = resolve_image(config, context)
assert result["suppressed"] and result["image"] is None
def test_bundled_series_images_retain_truthful_source_metadata():
config, _ = bundled_configuration()
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
assert result["image"]["id"] == "starlink-solar-array"
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
assert pending.rights_status == "permission_pending"
assert (
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
== pending.id
)
assert pending.enabled
@pytest.mark.parametrize(
"url",
[
"javascript:alert(1)",
"//evil.example/image.jpg",
"/earth/assets/cards/../secret",
"https://user:pass@example.org/a.jpg",
],
)
def test_unsafe_urls_rejected(url):
with pytest.raises(ValidationError):
CardImageAsset(id="test", title="test", url=url)
def test_duplicate_targets_and_dangling_assets_rejected():
data = configuration().model_dump()
duplicate = {**data["rules"][0], "id": "another"}
data["rules"].append(duplicate)
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
data = configuration().model_dump()
data["assets"] = []
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
defaults, _ = bundled_configuration()
payload = {
"revision": "custom",
"config": {"assets": [], "rules": []},
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
payload["known_assets"] = []
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
output = io.BytesIO()
Image.new("RGB", (1600, 900), "blue").save(output, format)
asset = card_image_files.store_image(output.getvalue(), "Facility")
assert (asset.width, asset.height) == (1600, 900)
assert len(list(tmp_path.iterdir())) == 3
for size, url in asset.variants.items():
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
assert max(image.size) == int(size)
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
with pytest.raises(HTTPException):
card_image_files.store_image(b"<svg></svg>", "bad")
with pytest.raises(HTTPException):
card_image_files.uploaded_path("../private")
def fake_db(payload):
db = AsyncMock()
db.add = MagicMock()
record = SimpleNamespace(payload=payload)
result = MagicMock()
result.scalar_one.return_value = record
result.scalar_one_or_none.return_value = record
db.execute.return_value = result
return db, record
@pytest.mark.asyncio
async def test_publish_revision_and_audit():
defaults, _ = bundled_configuration()
payload = {
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
db, record = fake_db(payload)
revision = effective_envelope(payload)["revision"]
result = await publish_configuration(
db, revision, defaults, SimpleNamespace(id=1, username="admin")
)
assert result["revision"] != revision
assert db.add.call_args.args[0].action == "card_images.update"
before = deepcopy(record.payload)
with pytest.raises(HTTPException) as error:
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
assert error.value.status_code == 409 and record.payload == before
@pytest.mark.asyncio
async def test_configuration_requires_admin_and_validates_http_payload():
app = FastAPI()
app.include_router(router)
defaults, _ = bundled_configuration()
db, _ = fake_db(
{
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (await client.get("/configuration")).status_code == 403
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
assert (await client.get("/configuration")).status_code == 200
assert (
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
).status_code == 422
@pytest.mark.asyncio
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
app = FastAPI()
app.include_router(router)
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
buffer = io.BytesIO()
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
response = await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
assert response.status_code == 200
asset = response.json()
filename = asset["variants"]["640"].rsplit("/", 1)[1]
image = await client.get(f"/files/{filename}")
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
assert "immutable" in image.headers["cache-control"]
assert (
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
).status_code == 422
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (
await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
).status_code == 403
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
from typing import get_args
from app.schemas.card_images import EntityType
config, _ = bundled_configuration()
for entity_type in get_args(EntityType):
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
assert result["image"] and result["match"]["value"] == "*"
config = config.model_copy(deep=True)
config.rules.append(
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
)
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["starlink-solar-array"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
def starlink():
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["focus_x"] = 20
assert starlink()["focus_x"] == 20
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["enabled"] = False
assert starlink()["enabled"] is False
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
assert starlink()["url"] == "https://example.org/custom.jpg"
def test_compute_categories_and_bgp_get_separate_user_images():
config, _ = bundled_configuration()
for kind, filename in [
("supercomputer", "supercomputer.png"),
("gpu_cluster", "gpu-cluster.jpg"),
("data_center", "data-center.png"),
("cloud_center", "cloud-center.png"),
("edge_node", "edge-node.png"),
]:
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
assert result["image"]["url"].endswith(filename)
for kind in ("bgp", "bgp_collector"):
result = resolve_image(config, dict(entity_type=kind, category=kind))
assert result["image"]["url"].endswith(
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
)
def test_saved_collector_default_migrates_without_changing_user_bindings():
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
old_asset = previous_bundled_assets()["network-switches"][0]
payload = {
"revision": "old",
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
"known_assets": [old_asset.id],
"known_rules": [old_rule.id],
}
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
"image"
]["url"].endswith("bgp-collector.png")
payload["config"]["rules"][0]["enabled"] = False
result = effective_envelope(payload)["config"]
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
@pytest.mark.asyncio
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
from app.services.card_image_targets import get_target
from sqlalchemy.dialects import postgresql
db = AsyncMock()
result = MagicMock()
result.mappings.return_value.first.return_value = {
"entity_id": "epoch_ai_gpu:site",
"name": "Site",
"category": "cloud_center",
"group": "Operator",
"model": "Model",
}
db.execute.return_value = result
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
query = db.execute.call_args.args[0]
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "collected_data.source = 'epoch_ai_gpu'" in sql
assert "collected_data.source_id = 'site'" in sql
assert context["category"] == "cloud_center" and context["model"] == "Model"
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["compute-gpu-cluster"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
result = effective_envelope(payload)["config"]
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
assert upgraded["variants"] == latest.variants
assert upgraded["url"] == latest.url

View File

@@ -0,0 +1,218 @@
"""Request-level regressions for quota-limited satellite downloads."""
import gzip
import json
import os
import time
from unittest.mock import AsyncMock
import httpx
import pytest
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
from app.services.collectors.downloads import ResumableFileDownloader
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
def setup_collector(monkeypatch, tmp_path, handler):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/gp.php"
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
original_client = httpx.AsyncClient
monkeypatch.setattr(
"app.services.collectors.celestrak.httpx.AsyncClient",
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
)
return collector
def expire(path):
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
os.utime(path, (old, old))
@pytest.mark.asyncio
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
return httpx.Response(
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
)
collector = setup_collector(monkeypatch, tmp_path, handler)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert len(requests) == 1
path = collector._downloader.cached_file_path(URL, ".json")
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
assert requests[1].headers["if-none-match"] == '"v1"'
@pytest.mark.asyncio
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
collector = setup_collector(monkeypatch, tmp_path, handler)
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
assert requests == ["GET"]
@pytest.mark.asyncio
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
with pytest.raises(RuntimeError, match="validation failed"):
await collector.fetch()
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
@pytest.mark.asyncio
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
collector = setup_collector(monkeypatch, tmp_path, handler)
path, _, meta = collector._downloader._cache_paths(URL, ".json")
path.write_text("[]")
meta.write_text('{"etag":"old"}')
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
assert "if-none-match" not in requests[0].headers
@pytest.mark.asyncio
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
)
path, _, meta = downloader._cache_paths(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text('[{"NORAD_CAT_ID":42}]')
meta.write_text('{"etag":"v1"}')
expire(path)
requests = []
def handler(request):
requests.append(request)
return httpx.Response(304)
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
for _ in range(2):
await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert len(requests) == 1
assert requests[0].headers["if-none-match"] == "v1"
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
@pytest.mark.asyncio
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
body = gzip.compress(raw)
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False
)
async with httpx.AsyncClient(
transport=httpx.MockTransport(
lambda request: httpx.Response(
200,
content=body,
headers={"content-encoding": "gzip", "content-length": str(len(body))},
)
)
) as client:
path = await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert path.read_bytes() == raw
@pytest.mark.asyncio
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
final, part, meta = downloader._cache_paths(URL, ".bin")
part.parent.mkdir(parents=True)
part.write_bytes(b"abc")
meta.write_text('{"etag":"v1"}')
requests = []
def handler(request):
requests.append(request)
if request.method == "HEAD":
return httpx.Response(
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
)
assert request.headers["range"] == "bytes=3-"
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".bin")
assert [r.method for r in requests] == ["HEAD", "GET"]
assert final.read_bytes() == b"abcdef"
@pytest.mark.asyncio
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
if request.url.params["GROUP"] == "active":
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
collector = setup_collector(monkeypatch, tmp_path, handler)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
assert [r.method for r in requests] == ["GET", "GET"]
@pytest.mark.asyncio
async def test_generic_cache_without_validators_must_refresh(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
path = downloader.cached_file_path(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text("[]")
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".json")
assert requests == ["HEAD", "GET"]
assert path.read_text() == "[1]"

View File

@@ -79,7 +79,8 @@ class TestTOP500Collector:
"""Test parsing empty response"""
collector = TOP500Collector()
result = collector.parse_response("<html><body><table></table></body></html>")
assert len(result) > 0
assert result == []
assert collector.fail_on_empty
def test_parse_response_single_item(self):
"""Test parsing single item response"""
@@ -97,9 +98,9 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["source_id"] == "top500_1"
assert result[0]["source_id"] == "top500_system_1"
assert result[0]["name"] == "Test Supercomputer"
assert result[0]["country"] == "USA"
assert result[0]["metadata"]["rank"] == 1
@@ -113,7 +114,7 @@ class TestTOP500Collector:
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
<tr>
<td>1</td>
<td>Valid\nVendor\nSite\nUSA</td>
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
<td>1000</td>
<td>10 PFLOP/s</td>
<td>12 PFLOP/s</td>
@@ -129,7 +130,7 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["name"] == "Valid"

View File

@@ -0,0 +1,222 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.system_log import AuditLog
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
default_envelope,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import FastAPI, HTTPException, Response
from httpx import ASGITransport, AsyncClient
from pydantic import ValidationError
def default_config():
return DEFAULT_CONFIGURATION.model_dump(mode="json")
@pytest.mark.parametrize(
"mutation",
[
lambda value: value["observations"][0]["values"].update(CN=101),
lambda value: value["observations"][0]["values"].update(CN=-1),
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
lambda value: value["observations"][0]["values"].update(CN=True),
lambda value: value["observations"][0]["values"].update(EUROPE=10),
lambda value: value["observations"][0].update(source_id="unknown"),
lambda value: value["observations"][0].update(year=2026),
lambda value: value["observations"].append(value["observations"][0]),
lambda value: value["sources"].append(value["sources"][0]),
lambda value: value["innovation"].append(value["innovation"][0]),
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
],
)
def test_rejects_invalid_or_ambiguous_evidence(mutation):
payload = default_config()
mutation(payload)
with pytest.raises(ValidationError):
ComputeConfiguration.model_validate(payload)
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
payload = default_config()
payload["sources"].append(
{
"id": "new-report",
"title": "New report",
"url": "https://example.com/report",
"publisher": "Research group",
"report_year": 2026,
"locator": "Figure 2",
}
)
payload["observations"].append(
{
"year": 2025,
"metric": "global_share",
"source_id": "new-report",
"values": {"CN": 25, "US": 50, "JP": 0},
}
)
config = ComputeConfiguration.model_validate(payload)
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
row = next(row for row in result["compute"] if row["year"] == 2025)
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
payload = default_config()
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
result = compute_history(
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
)
assert result["compute"][0]["values"] == {"CN": None, "US": None}
payload.update(sources=[], observations=[], innovation=[])
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
assert all(row["values"]["CN"] is None for row in result["compute"])
assert result["evidence"] == {}
def fake_db(record=None):
db = AsyncMock()
db.add = MagicMock()
result = MagicMock()
result.scalar_one_or_none.return_value = record
result.scalar_one.return_value = record
db.execute.return_value = result
return db
@pytest.mark.asyncio
async def test_fresh_install_uses_defaults_without_writing_database():
db = fake_db()
assert await read_configuration(db) == default_envelope()
db.add.assert_not_called()
db.commit.assert_not_awaited()
@pytest.mark.asyncio
async def test_series_reads_persisted_override_and_disables_http_cache():
envelope = default_envelope()
envelope["is_default"] = False
envelope["revision"] = "custom"
envelope["config"]["observations"][3]["values"]["CN"] = 30
response = Response()
data = await get_series(
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
)
assert data["compute"][3]["values"]["CN"] == 30
assert data["version"] == "custom"
assert data["provenance"] == "admin_curated_reports"
assert response.headers["Cache-Control"] == "no-store"
@pytest.mark.asyncio
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
actor = SimpleNamespace(id=1, username="admin")
old = deepcopy(record.payload)
result = await publish_configuration(
db,
revision=old["revision"],
config=DEFAULT_CONFIGURATION,
actor=actor,
)
assert result["revision"] != old["revision"]
audit = db.add.call_args.args[0]
assert isinstance(audit, AuditLog)
assert audit.details == {"before": old, "after": result}
assert audit.actor_id == actor.id
db.commit.assert_awaited_once()
with pytest.raises(HTTPException) as failure:
await publish_configuration(
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
)
assert failure.value.status_code == 409
assert record.payload == result
db.rollback.assert_awaited_once()
db.commit.assert_awaited_once()
@pytest.mark.asyncio
async def test_reset_changes_revision_and_restores_bundled_data():
record = SimpleNamespace(
payload={**default_envelope(), "revision": "custom", "is_default": False}
)
result = await publish_configuration(
fake_db(record),
revision="custom",
config=DEFAULT_CONFIGURATION,
actor=SimpleNamespace(id=1, username="admin"),
reset=True,
)
assert result["is_default"] is True
assert result["config"] == default_config()
assert result["revision"] != "custom"
@pytest.mark.asyncio
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
async def test_non_admin_cannot_configure(role):
with pytest.raises(HTTPException) as failure:
await configuration_admin(SimpleNamespace(role=role))
assert failure.value.status_code == 403
@pytest.mark.asyncio
async def test_http_configuration_auth_validation_and_public_series():
app = FastAPI()
app.include_router(router)
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
assert (await client.get("/series")).status_code == 200
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=1,
username="admin",
role="admin",
)
loaded = (await client.get("/configuration")).json()
invalid = deepcopy(loaded["config"])
invalid["observations"][0]["values"]["CN"] = 200
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": invalid,
},
)
assert response.status_code == 422
db.commit.assert_not_awaited()
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": loaded["config"],
},
)
assert response.status_code == 200
assert response.json()["revision"] != loaded["revision"]
assert (
await client.post(
"/configuration/reset",
json={
"revision": loaded["revision"],
},
)
).status_code == 409

View File

@@ -0,0 +1,268 @@
import csv
import io
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import HTTPException
from sqlalchemy.dialects import postgresql
from app.api.v1.compute_intelligence import sample_query, selected_countries
from app.services.collectors.epoch_ai import EpochAIGPUCollector
from app.services.collectors.top500 import TOP500Collector
from app.services.compute_history import compute_history
def epoch_csv(**overrides):
row = {
"Name": "Cluster Alpha",
"Country": "United States of America",
"Status": "Existing",
"First Operational Date": "2023-05-02",
"16-bit OP/s (log)": "18",
"Location": "3231 Riverport Rd, Memphis, TN 38109",
"latitude": "35",
"longitude": "-90",
**overrides,
}
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
return output.getvalue()
def test_epoch_csv_preserves_precision_date_and_missing_values():
collector = EpochAIGPUCollector()
row = collector.parse_response(epoch_csv())[0]
assert row["value"] == 1e18
assert row["unit"] == "FLOP/s"
assert row["metadata"]["precision"] == "FP16"
assert row["metadata"]["location_precision"] == "precise"
assert row["metadata"]["raw_country"] == "United States of America"
assert row["reference_date"] == "2023-05-02"
missing = collector.parse_response(
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
)[0]
assert missing["value"] is None
assert missing["reference_date"] is None
@pytest.mark.parametrize(
("location", "country", "expected"),
[
("United States of America", "United States of America", "country"),
("Japan", "Japan", "country"),
("Korea (Republic of)", "South Korea", "country"),
("Texas", "United States of America", "region"),
("Iowa, USA", "United States of America", "region"),
("Seoul", "South Korea", "city"),
("Paris, France", "France", "city"),
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
("Cluster 2, Texas", "United States of America", "site"),
("Campus 7, Tokyo", "Japan", "site"),
("Building 240, Argonne", "United States of America", "site"),
("7", "United States of America", "unknown"),
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
("35.380924,139.625008", "Japan", "precise"),
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
],
)
def test_epoch_location_precision_follows_location_field(location, country, expected):
row = EpochAIGPUCollector().parse_response(
epoch_csv(Location=location, Country=country)
)[0]
assert row["metadata"]["location_precision"] == expected
@pytest.mark.parametrize(
("location", "expected"),
[
({"lat": 35.0, "lng": 139.0}, "precise"),
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
({"lat": 35.0}, "unknown"),
({"lng": 139.0}, "unknown"),
({"lat": "invalid", "lng": 139.0}, "unknown"),
],
)
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
from app.services.epoch_location import assess_epoch_location
assert assess_epoch_location(location).precision == expected
def test_epoch_anonymous_rows_are_not_geocoded():
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
assert row["latitude"] is None and row["longitude"] is None
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
@pytest.mark.parametrize(
"payload",
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
)
def test_epoch_invalid_payload_fails_closed(payload):
with pytest.raises(ValueError):
EpochAIGPUCollector().parse_response(payload)
def test_epoch_duplicate_identity_rejected():
payload = epoch_csv()
with pytest.raises(ValueError, match="identity collision"):
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
assert len(rows) == 2
assert rows[0]["source_id"] != rows[1]["source_id"]
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
def test_top500_units_and_stable_identity():
collector = TOP500Collector()
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
first = collector.parse_response(
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
)[0]
second = collector.parse_response(
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
)[0]
assert first["source_id"] == second["source_id"] == "top500_system_123"
assert float(first["value"]) == 1.5e15
assert first["metadata"]["rpeak_flops"] == 2e15
assert first["reference_date"] == "2025-11-01"
assert second["reference_date"] == "2026-06-01"
with pytest.raises(ValueError, match="date missing"):
collector.parse_response(html.format(rank=1))
@pytest.mark.asyncio
async def test_large_source_loss_does_not_create_snapshot():
db = AsyncMock()
db.scalar.return_value = 500
db.add = MagicMock()
with pytest.raises(ValueError, match="40%"):
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
db.add.assert_not_called()
db.commit.assert_not_awaited()
def test_history_missing_years_and_denominators():
global_data = compute_history(["CN", "US"])
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
bilateral = compute_history(["CN", "US"], "bilateral_share")
assert sum(bilateral["compute"][3]["values"].values()) == 100
assert bilateral["denominator"] == "US+CN"
absolute = compute_history(["US", "CN"], "absolute")
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
assert global_data["innovation"][-2]["value"] == 17.95
assert global_data["innovation"][-1]["value"] is None
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
@pytest.mark.asyncio
async def test_snapshot_query_uses_observation_cutoff_not_location():
query = await sample_query(AsyncMock(), ["CN"], 2023)
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "2024-01-01" in sql
assert "row_number() OVER (PARTITION BY" in sql
assert "compute_contract" in sql and "中国" in sql
assert "latitude" not in sql and "longitude" not in sql
def test_country_input_is_bounded_and_deduplicated():
assert selected_countries("CN,CN,US") == ["CN", "US"]
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
with pytest.raises(HTTPException):
selected_countries("invalid")
with pytest.raises(HTTPException, match="overlapping"):
selected_countries("EUROPE,DE")
def test_catalog_supports_global_participants_without_inventing_history():
from app.services.compute_catalog import (
CATALOG,
COMPARISON_GROUPS,
COUNTRIES,
source_country_names,
source_country_participants,
)
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
{entry["code"] for entry in CATALOG}
)
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
owners = source_country_names(["CN"])
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
assert [entry["kind"] for entry in CATALOG[:5]] == [
"country",
"country",
"region",
"region",
"region",
]
group_members = [
member
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
]
assert len(group_members) == len(set(group_members))
assert set(group_members) == set(COUNTRIES)
data = compute_history(["JP", "KR"])
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
assert all(row["values"]["KR"] is None for row in data["compute"])
assert all(row["value"] is None for row in data["innovation"])
assert len(data["catalog"]) > 30
def test_region_history_requires_complete_same_year_member_coverage():
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
assert row_2023["values"]["CN"] == 31
assert row_2023["values"]["US"] == 41
assert row_2023["values"]["EUROPE"] is None
assert row_2023["values"]["APAC"] is None
assert row_2023["values"]["OTHER"] is None
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
def test_anonymous_records_never_enter_the_location_repair_queue():
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.models.collected_data import CollectedData
record = CollectedData(
source="epoch_ai_gpu",
source_id="anonymous",
name="Anonymous",
extra_data={"compute_contract": "v1", "anonymous": True},
)
result = convert_compute_centers_to_geojson([record])
assert result["features"] == []
assert result["unresolved"] == []
assert result["withheld_count"] == 1
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
absolute = compute_history(["CN"], "absolute")
observed = {row["year"]: row for row in absolute["compute"]}
assert observed[2022]["values"]["CN"] == 302
assert observed[2022]["evidence_id"] == "caict-2023"
assert observed[2023]["evidence_id"] == "caict-2024"
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
for row in absolute["compute"]:
if row["evidence_id"]:
report = absolute["evidence"][row["evidence_id"]]
assert report["url"].startswith("https://www.caict.ac.cn/")

View File

@@ -0,0 +1,18 @@
from app.services.compute_center_verified_sites import get_verified_compute_site
def test_registry_requires_exact_facility_identity_not_operator_name():
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
assert site["city"] == "Yokohama"
assert site["precision"] == "site"
assert site["coordinate_basis"] == "official_access_map_center"
assert site["sources"]
site["latitude"] = 0
assert get_verified_compute_site(
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
)["latitude"] != 0

View File

@@ -1,4 +1,5 @@
import json
from pathlib import Path
import pytest
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
def write_geojson(path, name="Test"):
names = [name]
if name == "earth_admin0_boundaries":
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
path.write_text(
json.dumps(
{
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
"features": [
{
"type": "Feature",
"properties": {"name": name},
"properties": {
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
},
}
for item in names
],
}
),
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
config = repo / "config/earth-boundary-sources.local.json"
example = repo / "config/earth-boundary-sources.example.json"
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
path.mkdir(parents=True, exist_ok=True)
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
geometry.parent.mkdir(parents=True, exist_ok=True)
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
geometry.write_bytes(source_geometry.read_bytes())
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
return {
"repo": repo,
"config": config,
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
}
def write_fallback(path, policy_path, geometry_path):
policy = earth_boundaries._read_json(policy_path)
geometry = earth_boundaries._read_json(geometry_path)
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(policy),
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
}
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "legacy-geojson"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
assert status["provider"] == "missing"
assert status["fallback_available"] is False
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
}
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "pmtiles-mvt"
assert status["high_precision_ready"] is True
assert status["policy_publishable"] is False
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["high_precision_ready"] is False
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
assert status["high_precision_ready"] is True
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
assert paths["manifest"].exists()
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
assert manifest["povPolicy"]["applied"] is True
assert manifest["povPolicy"]["coverage"] == "partial"
assert manifest["povPolicy"]["publicationReady"] is False
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
by_rule = {
feature["properties"].get("PLANET_POLICY_RULE"): feature
for feature in hover["features"]
}
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
assert all(
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
for feature in hover["features"]
)
assert any(phase == "download" for _progress, phase, _message in progress_events)
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
assert removed.isdisjoint(DEFAULT_DATASOURCES)
for source in removed:
assert collector_registry.get(source) is None
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
repo = Path(__file__).resolve().parents[2]
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
payload = json.loads(path.read_text(encoding="utf-8"))
matches = [
feature for feature in payload["features"]
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
]
assert len(matches) == 1
jeju = matches[0]
assert jeju["properties"]["ISO_A3"] == "KOR"
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
ring = jeju["geometry"]["coordinates"][0]
assert len(ring) == 61
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)

View File

@@ -0,0 +1,41 @@
import json
from pathlib import Path
import pytest
from app.core.earth_geography_policy import display_country, display_iso_a3
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
from app.core import earth_geography_policy as policy
policy._policy_country_by_code.cache_clear()
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
try:
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
policy.display_country("China")
finally:
policy._policy_country_by_code.cache_clear()
def test_display_country_groups_china_regions_and_kosovo() -> None:
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
assert display_country(source) == "中国"
assert display_iso_a3(source) == "CHN"
for source in ("Kosovo", "XK", "SRB"):
assert display_country(source) == "塞尔维亚"
assert display_iso_a3(source) == "SRB"
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
assert display_country("Japan") == "日本"
assert display_iso_a3("Japan") == "JPN"
assert display_country("unexpected non-country") is None
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy = json.loads(path.read_text(encoding="utf-8"))
for rule in policy["rules"]:
for code in rule.get("sourceIsoA3", []):
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))

View File

@@ -0,0 +1,15 @@
from app.services.earth_layer_adapters import (
get_earth_cache_patterns_for_source,
get_earth_update_layers_for_change,
)
def test_news_changes_do_not_invalidate_earth_summary_cache():
assert get_earth_update_layers_for_change("earth_news_items", "earth_news_items") == ["news"]
assert get_earth_cache_patterns_for_source("earth_news_items") == []
assert get_earth_cache_patterns_for_source("media_news_archive") == []
def test_counted_layers_still_invalidate_earth_summary_cache():
for source in ("aisstream_vessels", "arcgis_cables", "celestrak_tle", "top500", "ris_live_bgp"):
assert "summary*" in get_earth_cache_patterns_for_source(source)

View File

@@ -1,9 +1,11 @@
from __future__ import annotations
import asyncio
import json
import httpx
import pytest
from fastapi import Response
from fastapi import FastAPI, HTTPException, Response
from app.api.v1 import visualization
from app.services.earth_layer_cache import (
@@ -49,6 +51,19 @@ class FakeRedis:
deleted += 1 if self.store.pop(key, None) is not None else 0
return deleted
def eval(self, script: str, numkeys: int, *args):
self._maybe_fail()
keys, values = args[:numkeys], args[numkeys:]
if self.store.get(keys[0]) != values[0]:
return 0
if numkeys == 3:
self.setex(keys[1], values[1], values[3])
self.setex(keys[2], values[2], values[3])
return 1
if "'expire'" in script:
return 1
return self.delete(keys[0])
def scan_iter(self, match: str):
self._maybe_fail()
prefix = match.rstrip("*")
@@ -120,7 +135,9 @@ async def test_resolve_layer_payload_writes_fresh_and_stale(fake_cache_client):
@pytest.mark.asyncio
async def test_resolve_layer_payload_returns_stale_when_builder_fails(fake_cache_client):
key = earth_layer_cache.key("bgp-incidents", status="active")
fake_cache_client.store[f"{key}:stale"] = json.dumps({"type": "FeatureCollection", "features": []})
fake_cache_client.store[f"{key}:stale"] = json.dumps(
{"type": "FeatureCollection", "features": []}
)
async def builder():
raise RuntimeError("db exploded")
@@ -165,6 +182,7 @@ async def test_resolve_layer_payload_bypasses_redis_failure():
previous = earth_layer_cache._client
earth_layer_cache._client = FakeRedis(fail=True)
try:
async def builder():
return {"type": "FeatureCollection", "features": [{"id": "safe"}]}
@@ -192,9 +210,13 @@ async def test_visualization_endpoint_sets_cache_headers(fake_cache_client, monk
monkeypatch.setattr(visualization, "_build_satellites_geojson", fake_build_satellites_geojson)
first_response = Response()
first = await visualization.get_satellites_geojson(limit=25, db=object(), response=first_response)
first = await visualization.get_satellites_geojson(
limit=25, db=object(), response=first_response
)
second_response = Response()
second = await visualization.get_satellites_geojson(limit=25, db=object(), response=second_response)
second = await visualization.get_satellites_geojson(
limit=25, db=object(), response=second_response
)
assert first == second
assert calls == 1
@@ -252,3 +274,284 @@ async def test_vessel_snapshot_uses_short_cache(fake_cache_client, monkeypatch):
assert calls == 1
assert first_response.headers["X-Planet-Cache"] == "refresh"
assert second_response.headers["X-Planet-Cache"] == "hit"
@pytest.mark.asyncio
async def test_concurrent_summary_requests_share_one_slow_build(monkeypatch):
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def build_summary(db):
nonlocal calls
calls += 1
started.set()
await finish.wait()
return {"stats": {"vessel_count": 42}}
monkeypatch.setattr(visualization, "_build_visualization_geo_summary", build_summary)
app = FastAPI()
app.include_router(visualization.router, prefix="/api/v1/visualization")
app.dependency_overrides[visualization.get_db] = lambda: object()
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
requests = [
asyncio.create_task(client.get("/api/v1/visualization/geo/summary")) for _ in range(6)
]
await asyncio.wait_for(started.wait(), 1)
# Keep the build open beyond the old 200 ms fallback to duplicate SQL.
await asyncio.sleep(0.25)
assert calls == 1
finish.set()
results = await asyncio.gather(*requests)
assert all(response.status_code == 200 for response in results)
assert all(response.json() == {"stats": {"vessel_count": 42}} for response in results)
assert sorted(response.headers["X-Planet-Cache"] for response in results) == [
"hit",
"hit",
"hit",
"hit",
"hit",
"refresh",
]
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
@pytest.mark.parametrize("error_type", [RuntimeError, TimeoutError])
async def test_shared_builder_error_is_not_retried(error_type):
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def builder():
nonlocal calls
calls += 1
started.set()
await finish.wait()
raise error_type("db failed")
requests = [
asyncio.create_task(
resolve_layer_payload(
key="shared-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
)
for _ in range(4)
]
await started.wait()
await asyncio.sleep(0)
finish.set()
results = await asyncio.gather(*requests, return_exceptions=True)
assert calls == 1
assert all(isinstance(result, error_type) for result in results)
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_concurrent_redis_failure_shares_bypass_result(fake_cache_client):
fake_cache_client.fail = True
started = asyncio.Event()
finish = asyncio.Event()
calls = 0
async def builder():
nonlocal calls
calls += 1
started.set()
await finish.wait()
return {"count": 42}
requests = [
asyncio.create_task(
resolve_layer_payload(
key="shared-bypass", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
)
for _ in range(4)
]
await started.wait()
await asyncio.sleep(0)
finish.set()
results = await asyncio.gather(*requests)
assert calls == 1
assert all(result.state == "bypass" and result.payload == {"count": 42} for result in results)
@pytest.mark.asyncio
async def test_cancelled_waiter_does_not_cancel_owner():
started = asyncio.Event()
finish = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120)
async def builder():
started.set()
await finish.wait()
return {"count": 1}
owner = asyncio.create_task(
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
)
await started.wait()
waiter = asyncio.create_task(
resolve_layer_payload(key="cancel-waiter", policy=policy, builder=builder)
)
await asyncio.sleep(0)
waiter.cancel()
with pytest.raises(asyncio.CancelledError):
await waiter
finish.set()
assert (await owner).payload == {"count": 1}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_cancelled_owner_allows_waiter_to_use_its_own_builder():
started = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120)
async def owner_builder():
started.set()
await asyncio.Event().wait()
async def waiter_builder():
return {"count": 2}
owner = asyncio.create_task(
resolve_layer_payload(key="cancel-owner", policy=policy, builder=owner_builder)
)
await started.wait()
waiter = asyncio.create_task(
resolve_layer_payload(key="cancel-owner", policy=policy, builder=waiter_builder)
)
await asyncio.sleep(0)
owner.cancel()
with pytest.raises(asyncio.CancelledError):
await owner
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 2}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_cross_process_lock_waits_for_payload_instead_of_building(fake_cache_client):
key = "remote-owner"
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
policy = EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005)
calls = 0
async def builder():
nonlocal calls
calls += 1
return {"count": -1}
waiter = asyncio.create_task(resolve_layer_payload(key=key, policy=policy, builder=builder))
await asyncio.sleep(0.03)
assert calls == 0
fake_cache_client.store[key] = json.dumps({"count": 3})
assert (await asyncio.wait_for(waiter, 1)).payload == {"count": 3}
assert calls == 0
@pytest.mark.asyncio
async def test_wait_timeout_does_not_bypass_other_owner(fake_cache_client):
key = "remote-timeout"
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "another-owner"
calls = 0
async def builder():
nonlocal calls
calls += 1
return {}
with pytest.raises(HTTPException) as error:
await resolve_layer_payload(
key=key,
policy=EarthLayerCachePolicy(60, 120, lock_wait_seconds=0.005, build_wait_seconds=0.02),
builder=builder,
)
assert error.value.status_code == 503
assert calls == 0
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_lost_owner_cannot_publish_or_release_new_lock(fake_cache_client):
key = "lock-replaced"
policy = EarthLayerCachePolicy(60, 120)
async def builder():
fake_cache_client.store[earth_layer_cache.lock_key(key)] = "new-owner"
fake_cache_client.store[key] = json.dumps({"count": 99})
return {"count": 1}
result = await resolve_layer_payload(key=key, policy=policy, builder=builder)
assert result.payload == {"count": 1}
assert result.state == "bypass"
assert fake_cache_client.store[earth_layer_cache.lock_key(key)] == "new-owner"
assert json.loads(fake_cache_client.store[key]) == {"count": 99}
@pytest.mark.asyncio
async def test_redis_publish_failure_returns_built_payload_without_retry(fake_cache_client):
calls = 0
async def builder():
nonlocal calls
calls += 1
fake_cache_client.fail = True
return {"count": 7}
result = await resolve_layer_payload(
key="publish-error", policy=EarthLayerCachePolicy(60, 120), builder=builder
)
assert result.state == "bypass"
assert result.payload == {"count": 7}
assert calls == 1
@pytest.mark.asyncio
async def test_local_wait_timeout_leaves_owner_running():
started = asyncio.Event()
finish = asyncio.Event()
policy = EarthLayerCachePolicy(60, 120, build_wait_seconds=0.02)
async def builder():
started.set()
await finish.wait()
return {"count": 1}
owner = asyncio.create_task(
resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
)
await started.wait()
with pytest.raises(HTTPException) as error:
await resolve_layer_payload(key="local-timeout", policy=policy, builder=builder)
assert error.value.status_code == 503
assert not owner.done()
finish.set()
assert (await owner).payload == {"count": 1}
assert not earth_layer_cache._inflight
@pytest.mark.asyncio
async def test_different_keys_can_build_concurrently():
started = set()
both_started = asyncio.Event()
async def build(key):
started.add(key)
if len(started) == 2:
both_started.set()
await asyncio.wait_for(both_started.wait(), 1)
return {"key": key}
results = await asyncio.gather(
*(
resolve_layer_payload(
key=key, policy=EarthLayerCachePolicy(60, 120), builder=lambda key=key: build(key)
)
for key in ("one", "two")
)
)
assert {result.payload["key"] for result in results} == {"one", "two"}

View File

@@ -3,6 +3,7 @@ from types import SimpleNamespace
import pytest
from app.api.v1.news import get_earth_feed
from app.services.earth_news import (
NewsFeedEndpoint,
NewsFeedSource,
@@ -1201,7 +1202,6 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
db=FakeDb(),
)
assert captured["freshness_region"] == "europe"
assert captured["items_region"] == "europe"
assert captured["items_categories"] == {"business", "ecommerce"}
assert captured["items_source_ids"][0] is None
@@ -1237,71 +1237,41 @@ async def test_news_source_test_treats_type_reference_as_non_fetching():
@pytest.mark.asyncio
async def test_earth_news_payload_initializes_empty_database_from_rss(monkeypatch):
async def test_earth_news_feed_get_does_not_fetch_write_or_queue_rss(monkeypatch):
db = object()
source = NewsFeedSource(
id="test-feed",
name="Test Feed",
region="global",
homepage_url="https://example.com",
feed_url="https://example.com/rss.xml",
read_calls = 0
async def fail_side_effect(*_args, **_kwargs):
raise AssertionError("RSS feed reads must not fetch, persist, or enqueue work")
async def empty_news_items(*_args, **_kwargs):
nonlocal read_calls
read_calls += 1
return []
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_side_effect)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fail_side_effect)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fail_side_effect)
monkeypatch.setattr(
"app.services.earth_news_store.list_earth_news_items", empty_news_items
)
item = ParsedNewsItem(
id="test-feed:init",
title="Initial RSS story",
summary="Initial summary",
url="https://example.com/init",
source="Test Feed",
feed_name="Test Feed",
feed_region="global",
homepage_url="https://example.com",
published_at=datetime(2026, 5, 15, 3, 0, tzinfo=UTC),
payload = await get_earth_feed(
lat=None,
lon=None,
region=None,
categories=None,
sources=None,
limit=None,
locale=None,
db=db,
)
item.location_patch = {
"latitude": 20.0,
"longitude": 0.0,
"location_label": "全球",
"location_source": "region_anchor",
"verified": False,
"location_meta": {"target": None, "anchor": {"region": "global"}},
}
upserted = []
enqueued = []
async def fake_get_earth_news_freshness(_db, *, active_region):
return 0, None
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
return [item], [], {"test-feed": {"source_id": "test-feed", "ok": True, "status": "ok", "count": 1}}
async def fake_upsert_earth_news_items(_db, items):
upserted.extend(items)
return len(items)
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
return [item]
async def fake_enqueue_target_location_job(payload, **_kwargs):
enqueued.append(payload)
return True
monkeypatch.setattr("app.services.earth_news.get_sources_for_region", lambda _region: [source])
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
payload = await get_earth_news_payload(db=db)
assert upserted[0].id == "test-feed:init"
assert payload["items"][0]["id"] == "test-feed:init"
assert payload["items"][0]["verified"] is False
assert enqueued[0]["id"] == "test-feed:init"
assert payload["items"] == []
assert read_calls == 1
@pytest.mark.asyncio
async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
async def test_earth_news_payload_does_not_supplement_stale_database_items(monkeypatch):
db = object()
old_item = ParsedNewsItem(
id="db:old",
@@ -1314,41 +1284,17 @@ async def test_earth_news_payload_supplements_stale_database_items(monkeypatch):
homepage_url="https://example.com",
published_at=datetime(2026, 5, 14, 3, 0, tzinfo=UTC),
)
old_item.location_patch = {
"latitude": 20.0,
"longitude": 0.0,
"location_label": "全球",
"location_source": "region_anchor",
"verified": False,
"location_meta": {"target": None, "anchor": {"region": "global"}},
}
fetched = []
async def fail_fetch(*_args, **_kwargs):
raise AssertionError("stale RSS supplementation belongs to the background worker")
async def fake_get_earth_news_freshness(_db, *, active_region):
return 12, datetime(2026, 5, 14, 3, 0, tzinfo=UTC)
async def fake_fetch_rss_items_for_sources(_sources, **_kwargs):
fetched.append(True)
return [old_item], [], {"stored": {"source_id": "stored", "ok": True, "status": "ok", "count": 1}}
async def fake_upsert_earth_news_items(_db, items):
return len(items)
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
return [old_item]
async def fake_enqueue_target_location_job(_payload, **_kwargs):
return True
monkeypatch.setattr("app.services.earth_news_store.get_earth_news_freshness", fake_get_earth_news_freshness)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch_rss_items_for_sources)
monkeypatch.setattr("app.services.earth_news_store.upsert_earth_news_items", fake_upsert_earth_news_items)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fail_fetch)
monkeypatch.setattr("app.services.earth_news_store.list_earth_news_items", fake_list_earth_news_items)
monkeypatch.setattr("app.services.earth_news_queue.enqueue_target_location_job", fake_enqueue_target_location_job)
payload = await get_earth_news_payload(db=db)
payload = await get_earth_news_payload(db=db, source_ids={"stored"})
assert fetched == [True]
assert payload["items"][0]["id"] == "db:old"

View File

@@ -0,0 +1,86 @@
from typing import Any
import httpx
import pytest
from app.services.earth_news import (
NewsFeedEndpoint,
NewsFeedSource,
ParsedNewsItem,
_fetch_single_feed_url,
_serialize_item,
)
async def fetch_feed(
content: bytes, content_type: str = "text/xml"
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
source = NewsFeedSource(
id="ebrun",
name="亿邦动力",
region="asia-pacific",
feed_url="https://example.com/rss.xml",
homepage_url="https://example.com/",
)
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
transport = httpx.MockTransport(
lambda request: httpx.Response(
200, content=content, headers={"content-type": content_type}
)
)
async with httpx.AsyncClient(transport=transport) as client:
return await _fetch_single_feed_url(client, source, feed)
@pytest.mark.asyncio
@pytest.mark.parametrize(
("encoding", "declaration", "content_type"),
[
("gbk", 'encoding="GBK"', "text/xml"),
("gb2312", "encoding='GB2312'", "application/rss+xml"),
("gb18030", 'encoding="GB18030"', "text/xml"),
("gbk", "", "text/xml; charset=gbk"),
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
("utf-8", "", "text/xml"),
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
("utf-16", 'encoding="UTF-16"', "text/xml"),
("utf-32", 'encoding="UTF-32"', "text/xml"),
],
)
async def test_feed_encoding_preserves_chinese_in_display(
encoding: str, declaration: str, content_type: str
):
title = "618电商战报:零售额增长266%"
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
xml = f'''<?xml version="1.0" {declaration}?>
<rss version="2.0"><channel><item>
<title><![CDATA[{title}]]></title>
<description><![CDATA[<p>{summary}</p>]]></description>
<link>https://example.com/story</link>
</item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
assert error is None
assert health["ok"] is True
assert len(items) == 1
assert items[0].title == title
assert items[0].summary == summary
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
assert payload["display_title"] == title
assert payload["display_summary"] == summary
@pytest.mark.asyncio
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
<rss><channel><item><title>电商新闻</title>
<link>https://example.com/story</link></item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode("gbk"))
assert items == []
assert error
assert health["ok"] is False
assert health["status"] == "format_error"

View File

@@ -0,0 +1,525 @@
from contextlib import asynccontextmanager
from datetime import UTC, datetime
import json
import pytest
from app.services.earth_news import NewsFeedEndpoint, NewsFeedSource
from app.services import earth_news_refresh
from app.services.scheduler import (
EARTH_NEWS_REFRESH_JOB_ID,
_earth_news_region_refresh_job_id,
schedule_earth_news_region_refresh,
start_scheduler,
sync_scheduler_with_datasources,
)
class FakeRedis:
def __init__(self) -> None:
self.values: dict[str, str] = {}
def set(self, key: str, value: str, *, nx: bool = False, ex: int | None = None) -> bool:
del ex
if nx and key in self.values:
return False
self.values[key] = value
return True
def exists(self, key: str) -> bool:
return key in self.values
def eval(self, script: str, _key_count: int, key: str, token: str, *_args: object) -> int:
if self.values.get(key) != token:
return 0
if "redis.call('del'" in script:
del self.values[key]
return 1
class FakeStatusRedis:
def __init__(self) -> None:
self.values: dict[str, str] = {}
self.expirations: dict[str, int] = {}
def get(self, key: str):
return self.values.get(key)
def eval(self, script: str, key_count: int, *args):
keys = args[:key_count]
values = args[key_count:]
if "local active_id" in script:
active_key, status_key = keys
refresh_id, encoded, active_ttl, status_ttl, status_prefix = values
active_id = self.values.get(active_key)
if active_id:
active_status = self.values.get(f"{status_prefix}{active_id}")
if active_status:
status = json.loads(active_status)["status"]
if status in {"pending", "running"}:
return [active_id, active_status, 0]
self.values.pop(active_key, None)
self.values[active_key] = refresh_id
self.values[status_key] = encoded
self.expirations[active_key] = int(active_ttl)
self.expirations[status_key] = int(status_ttl)
return [refresh_id, encoded, 1]
active_key, status_key = keys
encoded, refresh_id, status_ttl, status, active_ttl = values
if self.values.get(active_key) != refresh_id:
return 0
self.values[status_key] = encoded
self.expirations[status_key] = int(status_ttl)
if status in {"succeeded", "failed"}:
self.values.pop(active_key, None)
else:
self.expirations[active_key] = int(active_ttl)
return 1
class FakeSession:
def __init__(self) -> None:
self.commits = 0
self.rollbacks = 0
async def commit(self) -> None:
self.commits += 1
async def rollback(self) -> None:
self.rollbacks += 1
@pytest.mark.asyncio
async def test_region_refresh_fetches_only_missing_feed_coverage(monkeypatch):
redis_client = FakeRedis()
db = FakeSession()
feeds = (
NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global"),
NewsFeedEndpoint("feed-b", "Feed B", "https://example.com/b.xml", region="global"),
)
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url=feeds[0].url,
homepage_url="https://example.com",
feeds=feeds,
)
fetched_source_feeds: list[list[str]] = []
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
assert config_payload == {"sources": []}
return [source]
async def fake_freshness(_db, *, active_region):
assert active_region == "global"
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
assert active_region == "global"
assert recent_after is not None
return {("source-a", "feed-a")}
async def fake_fetch(sources, **_kwargs):
fetched_source_feeds.extend([[feed.id for feed in item.feeds] for item in sources])
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
async def fake_record_health(_db, _health):
return None
async def fake_enqueue(_items):
return None
async def fake_upsert(_db, items):
assert items == []
return 0
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr(
"app.services.earth_news.record_earth_news_sources_health", fake_record_health
)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_enqueue)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "completed"
assert result["successful_sources"] == 1
assert fetched_source_feeds == [["feed-b"]]
assert db.commits == 1
assert earth_news_refresh._lock_key("global") not in redis_client.values
@pytest.mark.asyncio
async def test_region_refresh_reports_failure_when_all_sources_time_out(monkeypatch):
redis_client = FakeRedis()
db = FakeSession()
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="global")
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url=feed.url,
homepage_url="https://example.com",
feeds=(feed,),
)
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
return [source]
async def fake_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
return set()
async def fake_fetch(*_args, **_kwargs):
return (
[],
["private upstream details"],
{
"source-a": {
"source_id": "source-a",
"ok": False,
"status": "timeout",
"feed_results": [{"feed_id": "feed-a", "ok": False, "status": "timeout"}],
}
},
)
async def fake_noop(*_args, **_kwargs):
return None
async def fake_upsert(_db, _items):
return 0
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
result = await earth_news_refresh.refresh_earth_news_region("global", force=True)
assert result["state"] == "failed"
assert result["error_code"] == "timeout"
assert db.commits == 1
@pytest.mark.asyncio
async def test_database_commit_failure_sets_refresh_failed_status(monkeypatch):
class CommitFailureSession(FakeSession):
async def commit(self) -> None:
raise RuntimeError("database commit failed with sensitive context")
redis_client = FakeRedis()
db = CommitFailureSession()
feed = NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml", region="europe")
source = NewsFeedSource(
id="source-a",
name="Source A",
region="europe",
feed_url=feed.url,
homepage_url="https://example.com",
feeds=(feed,),
)
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
statuses = []
@asynccontextmanager
async def fake_session_factory():
yield db
async def fake_get_sources_payload(_db):
return {"sources": []}
async def fake_get_sources_for_region(_db, _region, *, config_payload=None):
return [source]
async def fake_freshness(_db, *, active_region):
return 12, datetime.now(UTC)
async def fake_coverage(_db, *, active_region, recent_after):
return set()
async def fake_fetch(*_args, **_kwargs):
return [], [], {"source-a": {"source_id": "source-a", "ok": True, "feed_results": []}}
async def fake_noop(*_args, **_kwargs):
return None
async def fake_upsert(_db, _items):
return 1
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fake_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
monkeypatch.setattr(
"app.services.earth_news.get_earth_news_sources_payload", fake_get_sources_payload
)
monkeypatch.setattr(
"app.services.earth_news.get_configured_sources_for_region", fake_get_sources_for_region
)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_freshness", fake_freshness)
monkeypatch.setattr(earth_news_refresh, "get_earth_news_feed_coverage", fake_coverage)
monkeypatch.setattr("app.services.earth_news._fetch_rss_items_for_sources", fake_fetch)
monkeypatch.setattr("app.services.earth_news.record_earth_news_sources_health", fake_noop)
monkeypatch.setattr("app.services.earth_news._enqueue_unverified_locations", fake_noop)
monkeypatch.setattr(earth_news_refresh, "upsert_earth_news_items", fake_upsert)
monkeypatch.setattr(scheduler_module, "touch_earth_news_refresh_status", lambda *_args: True)
monkeypatch.setattr(
scheduler_module,
"set_earth_news_refresh_status",
lambda refresh_id, region, status, *, error=None: statuses.append(
{"refresh_id": refresh_id, "region": region, "status": status, "error": error}
),
)
scheduler_module._pending_earth_news_refreshes["europe"] = {
"force": True,
"refresh_id": "refresh-commit-failure",
}
await scheduler_module._run_earth_news_region_refresh("europe")
assert [entry["status"] for entry in statuses] == ["running", "failed"]
assert statuses[-1]["error"]["code"] == "refresh_failed"
assert "sensitive context" not in repr(statuses)
assert not redis_client.values
@pytest.mark.asyncio
async def test_region_refresh_skips_when_cross_process_lock_is_held(monkeypatch):
redis_client = FakeRedis()
redis_client.values[earth_news_refresh._lock_key("global")] = "other-owner"
def fail_session_factory():
raise AssertionError("a locked refresh must not open a database session")
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "busy"
def test_failed_feed_enters_cooldown_without_blocking_other_sources():
redis_client = FakeRedis()
source = NewsFeedSource(
id="source-a",
name="Source A",
region="global",
feed_url="https://example.com/a.xml",
homepage_url="https://example.com",
feeds=(NewsFeedEndpoint("feed-a", "Feed A", "https://example.com/a.xml"),),
)
health = {
"source-a": {
"ok": False,
"feed_results": [{"feed_id": "feed-a", "ok": False}],
}
}
earth_news_refresh._record_failed_feed_cooldowns(
redis_client,
active_region="global",
health_by_source=health,
sources=[source],
)
selected, cooling = earth_news_refresh._select_feeds(
[source],
active_region="global",
required_keys={("source-a", "feed-a")},
redis_client=redis_client,
)
assert selected == []
assert cooling == {("source-a", "feed-a")}
@pytest.mark.asyncio
async def test_region_refresh_skips_when_redis_coordination_is_unavailable(monkeypatch):
def fail_redis_client():
raise RuntimeError("redis unavailable")
def fail_session_factory():
raise AssertionError("uncoordinated processes must not start duplicate RSS fetches")
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", fail_redis_client)
monkeypatch.setattr(earth_news_refresh, "async_session_factory", fail_session_factory)
result = await earth_news_refresh.refresh_earth_news_region("global")
assert result["state"] == "coordination_unavailable"
def test_refresh_status_deduplicates_active_region_and_bounds_completed_results(monkeypatch):
redis_client = FakeStatusRedis()
monkeypatch.setattr(earth_news_refresh._RedisClient, "get_client", lambda: redis_client)
first, created = earth_news_refresh.create_earth_news_refresh_status("global")
duplicate, duplicate_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert created is True
assert duplicate_created is False
assert duplicate == first
earth_news_refresh.set_earth_news_refresh_status(first["refresh_id"], "global", "running")
running, running_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert running_created is False
assert running["status"] == "running"
completed = earth_news_refresh.set_earth_news_refresh_status(
first["refresh_id"], "global", "succeeded"
)
assert completed["status"] == "succeeded"
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
second, second_created = earth_news_refresh.create_earth_news_refresh_status("global")
assert second_created is True
assert second["refresh_id"] != first["refresh_id"]
assert earth_news_refresh.get_earth_news_refresh_status(first["refresh_id"]) == completed
assert redis_client.expirations[earth_news_refresh._refresh_active_key("global")] == 120
assert (
redis_client.expirations[earth_news_refresh._refresh_status_key(second["refresh_id"])]
== 600
)
status_keys = [
key
for key in redis_client.values
if key.startswith(earth_news_refresh.REFRESH_STATUS_PREFIX)
]
assert len(status_keys) == 2
pending, pending_created = earth_news_refresh.create_earth_news_refresh_status("europe")
assert pending_created is True
redis_client.values.pop(earth_news_refresh._refresh_active_key("europe"), None)
expired = earth_news_refresh.get_earth_news_refresh_status(pending["refresh_id"])
assert expired["status"] == "failed"
assert expired["error"]["code"] == "expired"
@pytest.mark.asyncio
async def test_earth_news_refresh_scheduler_job_survives_datasource_sync(monkeypatch):
class FakeJob:
def __init__(self, job_id: str) -> None:
self.id = job_id
class FakeScheduler:
def __init__(self) -> None:
self.running = False
self.jobs = {
"obsolete": FakeJob("obsolete"),
_earth_news_region_refresh_job_id("global"): FakeJob(
_earth_news_region_refresh_job_id("global")
),
}
self.added: dict[str, object] = {}
def add_job(self, _func, **kwargs):
self.added.update(kwargs)
self.jobs[kwargs["id"]] = FakeJob(kwargs["id"])
def start(self) -> None:
self.running = True
def get_jobs(self):
return list(self.jobs.values())
def remove_job(self, job_id: str) -> None:
self.jobs.pop(job_id, None)
class FakeResult:
def scalars(self):
return self
def all(self):
return []
class FakeDb:
async def execute(self, _query):
return FakeResult()
fake_scheduler = FakeScheduler()
@asynccontextmanager
async def fake_session_factory():
yield FakeDb()
monkeypatch.setattr("app.services.scheduler.scheduler", fake_scheduler)
monkeypatch.setattr("app.services.scheduler.async_session_factory", fake_session_factory)
start_scheduler()
await sync_scheduler_with_datasources()
assert fake_scheduler.added["id"] == EARTH_NEWS_REFRESH_JOB_ID
assert fake_scheduler.added["next_run_time"] <= datetime.now(UTC)
assert EARTH_NEWS_REFRESH_JOB_ID in fake_scheduler.jobs
assert _earth_news_region_refresh_job_id("global") in fake_scheduler.jobs
assert "obsolete" not in fake_scheduler.jobs
@pytest.mark.asyncio
async def test_region_refresh_requests_are_merged_and_force_is_preserved(monkeypatch):
class FakeScheduler:
running = True
def __init__(self) -> None:
self.jobs: dict[str, tuple[object, dict[str, object]]] = {}
def add_job(self, func, **kwargs):
self.jobs[kwargs["id"]] = (func, kwargs)
scheduler_module = __import__("app.services.scheduler", fromlist=["scheduler"])
fake_scheduler = FakeScheduler()
monkeypatch.setattr(scheduler_module, "scheduler", fake_scheduler)
captured: list[tuple[str, bool]] = []
async def fake_refresh(region: str, *, force: bool = False):
captured.append((region, force))
return {"locked": False, "fetched": 0, "upserted": 0}
monkeypatch.setattr(scheduler_module, "refresh_earth_news_region", fake_refresh)
assert schedule_earth_news_region_refresh("global") is True
assert schedule_earth_news_region_refresh("global", force=True) is False
assert len(fake_scheduler.jobs) == 1
job_id = _earth_news_region_refresh_job_id("global")
func, kwargs = fake_scheduler.jobs.pop(job_id)
await func(**kwargs["kwargs"])
assert captured == [("global", True)]
assert job_id not in fake_scheduler.jobs

View File

@@ -0,0 +1,140 @@
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from app.services import earth_db_change_listener as listener_module
from app.services.earth_db_change_listener import EarthDbChangeDispatcher, EarthDbChangeListener
from app.services.earth_outbox_maintenance import (
DELETE_BATCH_SQL,
compact_consumed_events,
ensure_retention_index,
purge_consumed_events,
)
class FakeConnection:
def __init__(self, *, index_state: str = "ready", batch_counts: list[int] | None = None):
self.index_state = index_state
self.batch_counts = list(batch_counts or [])
self.calls: list[tuple[str, tuple[object, ...]]] = []
self.retained = {"total": 13, "pending": 3}
self.restored = dict(self.retained)
self.rolled_back = False
async def fetchrow(self, query: str, *args: object):
self.calls.append((query, args))
if "pg_index" in query:
if self.index_state == "missing":
return None
return {"indisvalid": self.index_state == "ready", "indisready": True}
if "FROM earth_outbox_retained" in query:
return self.retained
return self.restored
async def fetchval(self, query: str, *args: object):
self.calls.append((query, args))
if query == DELETE_BATCH_SQL:
return self.batch_counts.pop(0)
return True
async def execute(self, query: str, *args: object):
self.calls.append((query, args))
if "CREATE INDEX CONCURRENTLY" in query:
self.index_state = "ready"
return "OK"
def transaction(self):
connection = self
class Transaction:
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, _exc, _traceback):
connection.rolled_back = exc_type is not None
return Transaction()
@pytest.mark.asyncio
async def test_retention_skips_without_index():
conn = FakeConnection(index_state="missing")
assert await purge_consumed_events(conn, retention_days=1) == 0
assert not any(query == DELETE_BATCH_SQL for query, _ in conn.calls)
@pytest.mark.asyncio
async def test_retention_batches_only_consumed_old_rows():
conn = FakeConnection(batch_counts=[1000, 19])
deleted = await purge_consumed_events(
conn, retention_days=1, batch_size=1000, max_batches=10, pause_seconds=0
)
assert deleted == 1019
calls = [args for query, args in conn.calls if query == DELETE_BATCH_SQL]
assert len(calls) == 2
assert all(isinstance(args[0], datetime) and args[0].tzinfo is UTC for args in calls)
assert "consumed_at IS NOT NULL AND consumed_at < $1" in DELETE_BATCH_SQL
assert "FOR UPDATE SKIP LOCKED" in DELETE_BATCH_SQL
@pytest.mark.asyncio
async def test_index_creation_is_concurrent_and_repairs_invalid_index():
conn = FakeConnection(index_state="invalid")
await ensure_retention_index(conn)
statements = [query for query, _ in conn.calls]
assert any("DROP INDEX CONCURRENTLY" in query for query in statements)
assert any("CREATE INDEX CONCURRENTLY" in query for query in statements)
@pytest.mark.asyncio
async def test_compact_retains_pending_and_recent_consumed_in_one_transaction():
conn = FakeConnection()
assert await compact_consumed_events(conn, retention_days=1) == (13, 3)
statements = [query for query, _ in conn.calls]
assert next(i for i, query in enumerate(statements) if query.startswith("LOCK TABLE")) < next(
i for i, query in enumerate(statements) if query.startswith("TRUNCATE TABLE")
)
assert any("consumed_at IS NULL OR consumed_at >= $1" in query for query in statements)
assert "TRUNCATE TABLE earth_data_change_events CONTINUE IDENTITY RESTRICT" in statements
assert not conn.rolled_back
@pytest.mark.asyncio
async def test_compact_verification_failure_rolls_back():
conn = FakeConnection()
conn.restored = {"total": 12, "pending": 2}
with pytest.raises(RuntimeError, match="verification failed"):
await compact_consumed_events(conn, retention_days=1)
assert conn.rolled_back
@pytest.mark.asyncio
async def test_listener_runs_retention_at_most_once_per_interval(monkeypatch):
calls: list[str] = []
async def acquire(_conn):
calls.append("acquire")
return True
async def purge(_conn, *, retention_days):
assert retention_days == 1
calls.append("purge")
async def release(_conn):
calls.append("release")
monkeypatch.setattr(listener_module, "monotonic", lambda: 100.0)
monkeypatch.setattr(listener_module, "try_maintenance_lock", acquire)
monkeypatch.setattr(listener_module, "purge_consumed_events", purge)
monkeypatch.setattr(listener_module, "release_maintenance_lock", release)
monkeypatch.setattr(listener_module, "configured_retention_days", lambda: 1)
dispatcher = EarthDbChangeDispatcher(broadcast_earth_update=lambda _: None)
listener = EarthDbChangeListener(dsn="postgresql://example/db", dispatcher=dispatcher)
listener._connection = FakeConnection()
await listener._maybe_purge_outbox()
await listener._maybe_purge_outbox()
assert calls == ["acquire", "purge", "release"]

View File

@@ -1,64 +1,219 @@
from copy import deepcopy
import httpx
import pytest
from app.services import llm_model_catalog as discovery
from app.services import llm_provider_catalog as catalog
REAL_CLIENT = httpx.AsyncClient
def mock_catalog(monkeypatch, payload):
client_type = httpx.AsyncClient
transport = httpx.MockTransport(lambda request: httpx.Response(200, json=payload))
def mock_http(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
catalog.httpx, "AsyncClient", lambda **kwargs: client_type(transport=transport, **kwargs)
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
async def test_refresh_orders_by_release_date_before_choosing_default(monkeypatch):
mock_catalog(
monkeypatch,
{
"minimax": {
"models": {
"MiniMax-M2": {"release_date": "2025-10-27"},
"MiniMax-M3": {"release_date": "2026-06-01"},
"MiniMax-M2.7": {"release_date": "2026-03-18"},
"undated-model": {},
@pytest.mark.parametrize(
"provider,path,auth",
[
("minimax", "/anthropic/v1/models", "x-api-key"),
("anthropic", "/v1/models", "x-api-key"),
("openai", "/v1/models", "authorization"),
("deepseek", "/v1/models", "authorization"),
("alibaba", "/api/v1/models", "authorization"),
("moonshotai", "/v1/models", "authorization"),
("openrouter", "/api/v1/models", "authorization"),
("opencode-go", "/zen/go/v1/models", "authorization"),
("ollama", "/api/tags", "authorization"),
],
)
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
preset = catalog.get_fallback_llm_provider_preset(provider)
def handle(request):
assert request.url.path == path
assert request.url.host == httpx.URL(preset["base_url"]).host
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
if provider == "alibaba":
assert request.url.params["capabilities"] == "TG"
return httpx.Response(
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
)
if provider == "ollama":
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
return httpx.Response(200, json={"data": [{"id": "latest"}]})
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
assert result["base_url"] == preset["base_url"]
assert "test-key" not in str(result)
@pytest.mark.parametrize(
"provider,base,api,path",
[
(
"minimax",
"https://api.minimax.io/anthropic/v1/",
"anthropic-messages",
"/anthropic/v1/models",
),
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
(
"alibaba",
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
(
"alibaba",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
],
)
def test_urls_preserve_region_and_gateway(provider, base, api, path):
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
assert url.host == httpx.URL(base).host
assert url.path == path
@pytest.mark.asyncio
async def test_anthropic_pagination_and_release_order(monkeypatch):
seen = []
def handle(request):
seen.append(request.url.params.get("after_id"))
if len(seen) == 1:
return httpx.Response(
200,
json={
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
"has_more": True,
"last_id": "old",
},
)
return httpx.Response(
200,
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
assert seen == [None, "old"]
assert result["models"] == ["new", "old"]
@pytest.mark.asyncio
async def test_dashscope_pagination(monkeypatch):
def handle(request):
page = int(request.url.params["page_no"])
return httpx.Response(
200,
json={
"output": {
"total": 2,
"models": [
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
],
}
}
},
},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
assert result["models"] == ["page-2", "page-1"]
@pytest.mark.asyncio
@pytest.mark.parametrize(
"payload",
[
{},
{"data": []},
{"data": [None]},
{"data": [{}]},
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
],
)
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
with pytest.raises(discovery.LLMProviderCatalogError):
await catalog.refresh_llm_provider_preset("minimax", "test-key")
@pytest.mark.asyncio
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
assert len(result["models"]) == 140
assert result["models"][0] == "model-139"
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
@pytest.mark.asyncio
async def test_empty_ollama_catalog_is_valid(monkeypatch):
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
result = await catalog.refresh_llm_provider_preset("ollama")
assert result["models"] == []
assert result["model"] == ""
@pytest.mark.asyncio
async def test_opencode_documented_protocols(monkeypatch):
models = [
"minimax-m3",
"qwen3.8-max",
"gpt-5.6-luna",
"grok-4.6",
"muse-spark-1.3-contributor",
"kimi-k3",
]
mock_http(
monkeypatch,
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
)
refreshed = await catalog.refresh_llm_provider_preset("minimax")
assert refreshed["model"] == "MiniMax-M3"
assert refreshed["models"] == ["MiniMax-M3", "MiniMax-M2.7", "MiniMax-M2", "undated-model"]
assert refreshed["source"] == catalog.MODELS_DEV_URL
result = await catalog.refresh_llm_provider_preset("opencode-go")
assert list(result["model_provider_apis"].values()) == [
"anthropic-messages",
"anthropic-messages",
"openai-responses",
"openai-responses",
"openai-responses",
"openai-completions",
]
@pytest.mark.asyncio
async def test_refresh_does_not_truncate_new_models(monkeypatch):
models = {f"older-{index}": {"release_date": "2025-01-01"} for index in range(85)}
models["latest"] = {"release_date": "2026-06-01"}
mock_catalog(monkeypatch, {"minimax": {"models": models}})
async def test_retry_transient_error_only(monkeypatch):
calls = []
refreshed = await catalog.refresh_llm_provider_preset("minimax")
def handle(request):
calls.append(request)
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
assert refreshed["model"] == "latest"
assert len(refreshed["models"]) == 86
mock_http(monkeypatch, handle)
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 2
calls.clear()
def unauthorized(request):
calls.append(request)
return httpx.Response(401, text="secret-upstream-text")
@pytest.mark.asyncio
@pytest.mark.parametrize("payload", [{}, {"minimax": {"models": {}}}, {"minimax": []}])
async def test_invalid_catalog_fails_instead_of_claiming_fallback_is_fresh(monkeypatch, payload):
mock_catalog(monkeypatch, payload)
with pytest.raises(catalog.LLMProviderCatalogError):
await catalog.refresh_llm_provider_preset("minimax")
@pytest.mark.asyncio
async def test_empty_opencode_catalog_is_a_refresh_failure(monkeypatch):
mock_catalog(monkeypatch, {"data": []})
with pytest.raises(catalog.LLMProviderCatalogError):
await catalog.refresh_llm_provider_preset("opencode-go", api_key="test-key")
mock_http(monkeypatch, unauthorized)
with pytest.raises(httpx.HTTPStatusError) as error:
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 1
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)

View File

@@ -0,0 +1,165 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.api.v1 import settings as api
from app.models.system_setting import SystemSetting
from app.services import llm_model_catalog as discovery
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
REAL_CLIENT = httpx.AsyncClient
@pytest.fixture
def settings_db(monkeypatch, tmp_path):
# Real SQL reads, commits and reloads; do not mock the settings helpers.
engine = create_engine("sqlite://")
SystemSetting.__table__.create(engine)
session = Session(engine)
db = SimpleNamespace(
execute=AsyncMock(side_effect=session.execute),
add=session.add,
commit=AsyncMock(side_effect=session.commit),
refresh=AsyncMock(side_effect=session.refresh),
)
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
yield db
session.close()
engine.dispose()
def use_upstream(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
async def test_fresh_database_lists_all_builtin_presets(settings_db):
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
assert len(result["data"]) == 9
assert all(row["models"] for row in result["data"])
assert "MiniMax-M3" in result["data"][0]["models"]
@pytest.mark.asyncio
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
runtime = {
"ai_provider": {
"default_provider": "minimax",
"service_token": "private-service-token",
"providers": {
"minimax": {
"base_url": "https://saved.test/anthropic",
"model": "old",
"api_key": "saved-secret",
}
},
}
}
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
before = await api.get_setting_payload(settings_db, "external_integrations")
seen = []
def handle(request):
seen.append(request)
assert str(request.url) == "https://draft.test/anthropic/v1/models"
assert request.headers["x-api-key"] == "draft-secret"
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
use_upstream(monkeypatch, handle)
draft = api.AIProviderIntegrationUpdate(
provider="minimax",
base_url="https://draft.test/anthropic",
model="unsaved-model",
api_key="draft-secret",
service_token="unsaved-service-token",
)
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
result = await api.get_ai_provider_presets(None, settings_db)
preset = next(row for row in result["data"] if row["provider"] == "minimax")
assert len(seen) == 1
assert preset["models"] == ["MiniMax-M3", "old"]
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
assert await api.get_setting_payload(settings_db, "external_integrations") == before
assert "secret" not in str(result)
assert "private-service-token" not in str(result)
assert draft.model == "unsaved-model"
stored = await settings_db.execute(
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
)
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
@pytest.mark.asyncio
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
settings_db, monkeypatch
):
await api.save_setting_payload(
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
)
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
commits = settings_db.commit.await_count
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
assert error.value.status_code == 502
assert "private-upstream-secret" not in error.value.detail
assert settings_db.commit.await_count == commits
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
"models"
] == ["saved-model"]
@pytest.mark.asyncio
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
def fail_request(request):
pytest.fail("must not query a private catalog without credentials")
use_upstream(monkeypatch, fail_request)
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("minimax", None, settings_db)
assert error.value.status_code == 400
assert "API Key" in error.value.detail
@pytest.mark.asyncio
@pytest.mark.parametrize(
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
)
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
result = await api._check_ai_provider_lightweight(
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
)
assert result["success"] is False
assert result["connected"] is False
assert "private-upstream-secret" not in result["message"]
@pytest.mark.asyncio
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
assert result["success"] is False
@pytest.mark.asyncio
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
use_upstream(
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
)
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight(
{**preset, "model": "future-model", "api_key": "test-key"}, 5
)
assert result["success"] is True
assert result["url"].endswith("/anthropic/v1/models")

View File

@@ -10,7 +10,7 @@ from app.core.logging import PlanetContextFilter, PlanetFormatter, get_logger
from app.core.request_context import set_request_id
from app.services import business_logs
from app.services import persistent_logs
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup
from app.models.system_log import ObservabilityEvent, ObservabilityEventGroup, SystemLog
def _capture_output(callback):
@@ -100,6 +100,23 @@ def test_business_context_redacts_nested_sensitive_values():
assert context["nested"]["safe"] == "visible"
@pytest.mark.asyncio
async def test_business_log_preserves_events_in_isolated_store(persisted_test_logs):
await business_logs.emit_business_log(
get_logger("tests.persistence"),
event="test.business.completed",
message="Business action completed",
category="test",
context={"api_key": "test-secret", "record_count": 2},
)
logs = [entry for entry in persisted_test_logs if isinstance(entry, SystemLog)]
assert len(logs) == 1
assert logs[0].event == "test.business.completed"
assert logs[0].context == {"api_key": "[REDACTED]", "record_count": 2}
assert any(isinstance(entry, ObservabilityEventGroup) for entry in persisted_test_logs)
def test_observability_fingerprint_normalizes_hls_fragments():
first = persistent_logs.build_observability_fingerprint(
source="earth-client",

View File

@@ -0,0 +1,142 @@
from types import SimpleNamespace
from uuid import UUID
import importlib
import pytest
from fastapi import HTTPException
from app.api.v1 import earth, news
@pytest.mark.asyncio
async def test_post_news_refresh_only_schedules_background_region_check(monkeypatch):
scheduled = []
def request(region: str):
scheduled.append(region)
return {"refresh_id": "refresh-europe", "region": region, "status": "pending"}
async def fail_network_refresh(*_args, **_kwargs):
raise AssertionError("the POST handler must not perform RSS requests")
monkeypatch.setattr("app.services.scheduler.request_earth_news_region_refresh", request)
monkeypatch.setattr("app.services.scheduler.refresh_earth_news_region", fail_network_refresh)
response = await news.request_earth_news_refresh(
news.EarthNewsRefreshRequest(region="Europe", lat=48.8, lon=2.3)
)
assert response == {"refresh_id": "refresh-europe", "region": "europe", "status": "pending"}
assert scheduled == ["europe"]
refresh_route = next(route for route in news.router.routes if route.path == "/refresh")
assert refresh_route.status_code == 202
@pytest.mark.asyncio
async def test_post_news_refresh_infers_region_and_rejects_unknown_region(monkeypatch):
scheduled = []
monkeypatch.setattr(
"app.services.scheduler.request_earth_news_region_refresh",
lambda region: scheduled.append(region)
or {"refresh_id": "refresh-americas", "region": region, "status": "pending"},
)
response = await news.request_earth_news_refresh(
news.EarthNewsRefreshRequest(lat=40.7, lon=-74.0)
)
assert response["region"] == "americas"
assert response["status"] == "pending"
assert scheduled == ["americas"]
with pytest.raises(HTTPException) as error:
await news.request_earth_news_refresh(news.EarthNewsRefreshRequest(region="unknown"))
assert error.value.status_code == 422
@pytest.mark.asyncio
async def test_get_news_refresh_returns_status_and_expired_shape(monkeypatch):
current = {"refresh_id": "abc", "region": "global", "status": "running"}
monkeypatch.setattr(
"app.services.earth_news_refresh.get_earth_news_refresh_status",
lambda _refresh_id: current,
)
response = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
assert response is current
assert response["status"] == "running"
monkeypatch.setattr(
"app.services.earth_news_refresh.get_earth_news_refresh_status", lambda _id: None
)
expired = await news.get_earth_news_refresh(UUID("00000000-0000-0000-0000-000000000abc"))
assert expired == {
"refresh_id": UUID("00000000-0000-0000-0000-000000000abc").hex,
"region": None,
"status": "failed",
"error": {
"code": "expired",
"message": "Refresh status expired. Submit a new refresh.",
},
}
@pytest.mark.asyncio
async def test_database_failure_sets_terminal_failed_status_without_leaking_error(monkeypatch):
scheduler = importlib.import_module("app.services.scheduler")
statuses = []
async def fail_refresh(_region, *, force=False):
raise RuntimeError("postgres://private-token@host/database: commit failed")
monkeypatch.setattr(scheduler, "refresh_earth_news_region", fail_refresh)
monkeypatch.setattr(scheduler, "touch_earth_news_refresh_status", lambda *_args: True)
monkeypatch.setattr(
scheduler,
"set_earth_news_refresh_status",
lambda refresh_id, region, state, *, error=None: statuses.append(
{"refresh_id": refresh_id, "region": region, "status": state, "error": error}
),
)
scheduler._pending_earth_news_refreshes["europe"] = {
"force": True,
"refresh_id": "refresh-db-failure",
}
await scheduler._run_earth_news_region_refresh("europe")
assert [item["status"] for item in statuses] == ["running", "failed"]
assert statuses[-1]["error"] == {
"code": "refresh_failed",
"message": "Could not refresh news. Please try again.",
}
assert "private-token" not in repr(statuses)
@pytest.mark.asyncio
async def test_news_source_save_and_reset_schedule_immediate_refresh(monkeypatch):
scheduled = []
monkeypatch.setattr(
earth,
"save_earth_news_sources_payload",
lambda _db, _payload: _async_result({"status": "updated"}),
)
monkeypatch.setattr(
earth,
"reset_earth_news_sources_payload",
lambda _db: _async_result({"status": "reset"}),
)
monkeypatch.setattr(
"app.services.scheduler.schedule_earth_news_refresh_for_all_regions",
lambda *, force=False: scheduled.append(force),
)
await earth.update_earth_news_sources(
SimpleNamespace(model_dump=lambda: {}), _current_user=None, db=object()
)
await earth.reset_earth_news_sources(_current_user=None, db=object())
assert scheduled == [True, True]
async def _async_result(value):
return value

View File

@@ -0,0 +1,141 @@
from pathlib import Path
import sys
from unittest.mock import patch
import httpx
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
with patch(
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
return_value={},
):
from aiprovider.provider_service import ProviderService
from aiprovider.schemas import SituationalAnalysisRequest
from app.api.v1.settings import _runtime_config_from_ai_payload
def test_custom_openai_gateway_keeps_its_configured_protocol():
config = _runtime_config_from_ai_payload(
{
"default_provider": "openai",
"providers": {
"openai": {
"base_url": "https://gateway.test/v1",
"provider_api": "openai-completions",
"model": "gpt-6-astra",
},
},
}
)["llm_config"]
service = ProviderService({**config, "api_key": "test-key"})
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"model,path",
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
)
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
llm = _runtime_config_from_ai_payload(
{
"default_provider": "opencode-go",
"providers": {
"opencode-go": {
"model": model,
"api_key": "test-key",
"model_provider_apis": {model: "openai-completions"},
}
},
}
)["llm_config"]
service = ProviderService(llm)
seen = []
async def post(**request):
seen.append(request)
assert request["path"] == path
assert request["request_body"]["model"] == model
if path == "/responses":
assert request["request_body"]["store"] is False
assert "max_tokens" not in request["request_body"]
return {
"output": [
{"type": "reasoning", "summary": [{"text": "reason"}]},
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
]
}
return {"content": [{"type": "text", "text": "OK"}]}
monkeypatch.setattr(service, "_post", post)
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
assert len(seen) == 1
assert result.content == "OK"
@pytest.mark.asyncio
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
seen.append(str(request.url))
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
service = ProviderService(
{
"provider": "minimax",
"provider_api": "anthropic-messages",
"base_url": base,
"api_key": "test-key",
"model": "MiniMax-M3",
}
)
result = await service.analyze(
SituationalAnalysisRequest(title="test", objective="reply OK")
)
assert result.content == "OK"
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
@pytest.mark.asyncio
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
import json
seen.append(request)
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
assert request.headers["user-agent"] == "Planet/1.0"
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
service = ProviderService(
{
"provider": "opencode-go",
"provider_api": "anthropic-messages",
"base_url": "https://opencode.ai/zen/go/v1",
"api_key": "test-key",
"model": "minimax-m3",
}
)
await service.analyze(
SituationalAnalysisRequest(
title="test",
objective="reply OK",
thinking={"type": "enabled"},
context={"session_id": session_id},
)
)
session_headers = [request.headers["x-opencode-session"] for request in seen]
assert session_headers[0] == session_headers[1]
assert session_headers[0] != session_headers[2]

View File

@@ -1,7 +1,5 @@
from copy import deepcopy
from types import SimpleNamespace
import httpx
import pytest
from app.api.v1 import settings as settings_api
@@ -26,71 +24,6 @@ from app.api.v1.settings import (
from app.services.llm_provider_catalog import get_fallback_llm_provider_preset
@pytest.mark.asyncio
async def test_refreshed_presets_survive_listing_without_changing_runtime_settings(monkeypatch):
stored = {
"external_integrations": {
"ai_provider": {
"default_provider": "minimax",
"service_token": "internal-token",
"providers": {"minimax": {"model": "custom-model", "api_key": "provider-key"}},
}
}
}
runtime_before = deepcopy(stored["external_integrations"])
async def fake_save(_db, category, payload):
stored[category] = deepcopy(payload)
return stored[category]
async def fake_get_many(_db, categories):
return {category: deepcopy(stored.get(category, {})) for category in categories}
async def fake_refresh(provider, api_key=None):
return {
**get_fallback_llm_provider_preset(provider),
"model": "new-model",
"models": ["new-model", "older-model"],
"source": "https://models.dev/api.json",
}
monkeypatch.setattr(settings_api, "save_setting_payload", fake_save)
monkeypatch.setattr(settings_api, "get_setting_payloads", fake_get_many)
monkeypatch.setattr(settings_api, "refresh_llm_provider_preset", fake_refresh)
user = SimpleNamespace(id=1, role="admin")
await settings_api.refresh_ai_provider_preset("minimax", user, object())
await settings_api.refresh_ai_provider_preset("openai", user, object())
listed = await settings_api.get_ai_provider_presets(user, object())
presets = {preset["provider"]: preset for preset in listed["data"]}
assert presets["minimax"]["models"] == ["new-model", "older-model"]
assert presets["openai"]["model"] == "new-model"
assert presets["anthropic"]["source"] == "fallback"
assert presets["minimax"]["refreshed_at"]
assert stored["external_integrations"] == runtime_before
assert "provider-key" not in str(listed)
assert "internal-token" not in str(listed)
@pytest.mark.asyncio
async def test_failed_preset_refresh_is_an_error_and_does_not_save(monkeypatch):
async def fail_refresh(*args, **kwargs):
raise httpx.ConnectError("upstream error with secret-value")
async def fail_save(*args, **kwargs):
pytest.fail("failed refresh must preserve the last saved preset")
monkeypatch.setattr(settings_api, "refresh_llm_provider_preset", fail_refresh)
monkeypatch.setattr(settings_api, "save_setting_payload", fail_save)
with pytest.raises(settings_api.HTTPException) as error:
await settings_api.refresh_ai_provider_preset("minimax", SimpleNamespace(id=1), object())
assert error.value.status_code == 502
assert "secret-value" not in error.value.detail
@pytest.fixture(autouse=True)
def isolated_ai_provider_env_file(monkeypatch, tmp_path):
env_file = tmp_path / ".env"

View File

@@ -0,0 +1,37 @@
from app.services.tv_streams import (
CCTV4_HOMEPAGE,
CCTV4_STREAM,
UNSUPPORTED_CCTV4_STREAM,
normalize_tv_settings,
normalize_tv_source,
)
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
settings = normalize_tv_settings(None)
assert settings["default_source_id"] == "aljazeera-mubasher"
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert cctv["source_type"] == "hls"
assert cctv["stream_url"] == CCTV4_STREAM
assert "美洲版" in cctv["name"]
assert cctv["use_proxy"] is False
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
def test_saved_legacy_default_is_corrected_without_mutating_input():
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
source = normalize_tv_source(saved)
assert source["source_type"] == "hls"
assert source["stream_url"] == CCTV4_STREAM
assert source["use_proxy"] is False
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert source["stream_url"] == custom["stream_url"]
assert source["source_type"] == "hls"
assert source["use_proxy"] is True
assert settings["auto_fallback"] is False

View File

@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
longitude=-90.05,
metadata={
"organization": "xAI",
"location": "3231 Riverport Rd, Memphis, TN 38109",
"gpu_type": "H100",
"gpu_count": 100000,
"value": "20000",
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
compute_center_locations.set_compute_center_location_cache({})
def test_epoch_country_level_coordinates_are_unresolved():
record = _build_record(
record_id=33,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Meta 100k",
country="United States",
city="",
latitude=39.381266,
longitude=-97.922211,
metadata={
"location": "United States of America",
"raw_country": "United States of America",
},
)
payload = convert_compute_centers_to_geojson([record])
assert payload["features"] == []
assert len(payload["unresolved"]) == 1
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
def test_epoch_city_coordinates_are_not_labeled_precise():
record = _build_record(
record_id=34,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Seoul Cluster",
country="South Korea",
city="",
latitude=37.5665,
longitude=126.978,
metadata={"location": "Seoul", "raw_country": "South Korea"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.is_resolved
assert result.location.location_precision == "city"
assert result.location.needs_confirmation is True
assert result.location.is_estimated is True
assert result.location.city is None
assert result.location.matched_location_name == "Seoul"
assert "Epoch Location: Seoul" in result.location.location_source_note
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
record = _build_record(
record_id=37,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Petrobras Fênix",
country="Brazil",
city="",
latitude=-22.909534,
longitude=-43.209934,
metadata={
"location": "Rio de Janeiro, Brazil",
"raw_country": "Brazil",
},
)
payload = convert_compute_centers_to_geojson([record])
feature = payload["features"][0]
assert feature["properties"]["location_precision"] == "city"
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
"location_source_note"
]
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-35": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-35",
"name": "Confirmed Site",
"latitude": 35.0,
"longitude": 139.0,
"precision": "site",
"confidence": 1.0,
"location_source": "official_facility_registry",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=35,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Japan Cluster",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
"raw_country": "Japan",
},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "stored_compute_center_location"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-36": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-36",
"name": "Legacy Country Point",
"latitude": 36.386493,
"longitude": 138.59223,
"precision": "precise",
"confidence": 1.0,
"location_source": "source_coordinates",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=36,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Legacy Japan Cluster",
country="Japan",
city="",
latitude=36.386493,
longitude=138.59223,
metadata={"location": "Japan", "raw_country": "Japan"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location is None
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_verified_sites_replace_epoch_country_center_coordinates():
specifications = [
(
"epoch_jamstec_zettascaler_2_0_gyoukou",
"Gyoukou 2.0",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_jamstec_zettascaler_2_2_gyoukou",
"Gyoukou 2.2",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_argonne_nl_sophia__formerly_theta_",
"Sophia",
"United States of America",
39.381266,
-97.922211,
[-87.984498648496, 41.718109383287235],
),
]
records = []
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
specifications, start=40
):
record = _build_record(
record_id=index,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name=name,
country=country,
city="",
latitude=latitude,
longitude=longitude,
metadata={"location": country, "raw_country": country},
)
record.source_id = source_id
records.append(record)
payload = convert_compute_centers_to_geojson(records)
assert payload["unresolved"] == []
assert len(payload["features"]) == 3
for feature, specification in zip(payload["features"], specifications):
assert feature["geometry"]["coordinates"] == specification[-1]
assert feature["properties"]["location_precision"] == "site"
assert feature["properties"]["location_source"] == "official_facility_registry"
assert "https://" in feature["properties"]["location_source_note"]
def test_verified_site_beats_epoch_city_level_source_coordinates():
record = _build_record(
record_id=44,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Gyoukou 2.2",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
)
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "official_facility_registry"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (
35.380924,
139.625008,
)
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-32": {
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
@pytest.mark.asyncio
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
@pytest.mark.asyncio
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
llm_candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
@pytest.mark.asyncio
async def test_compute_centers_geojson_endpoint_returns_stats():
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
@pytest.mark.asyncio
async def test_visualization_geo_summary_returns_counts(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
# Exercise the fixture DB, never the running development server's Redis payload.
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
class _FakeSession:
async def execute(self, query):
query_text = str(query).lower()
if "group by collected_data.source" in query_text:
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
if "bgp_incidents" in query_text:
return _ScalarResult(scalar_value=2)
if "bgp_anomalies" in query_text:
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
def _fake_ror(query):
assert query == "Oak Ridge National Laboratory"
return {
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
assert "FORBIDDEN_PRECISIONS" in text, (
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
)
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
from app.core.compute_types import compute_site_type
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
metadata={"location_precision": "site"})
result = convert_compute_centers_to_geojson([record])
assert compute_site_type(record) == kind
assert result["features"][0]["properties"]["site_type"] == kind
record.source = "epoch_ai_gpu"
record.data_type = "gpu_cluster"
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
assert compute_site_type(record) == kind
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind

View File

@@ -0,0 +1,156 @@
from collections.abc import AsyncIterator, Iterator, Sequence
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import Select, create_engine
from sqlalchemy.engine import Result, ScalarResult
from sqlalchemy.orm import Session
from app.api.v1 import visualization
from app.models.collected_data import CollectedData
class SatelliteRows:
def __init__(self, result: ScalarResult[CollectedData]) -> None:
self.result = result
self.closed = False
async def partitions(self, size: int) -> AsyncIterator[Sequence[CollectedData]]:
for batch in self.result.partitions(size):
yield batch
async def close(self) -> None:
self.result.close()
self.closed = True
class SatelliteSession:
def __init__(self, session: Session) -> None:
self.session = session
self.streams: list[SatelliteRows] = []
async def stream_scalars(
self, statement: Select, *, execution_options: dict[str, int]
) -> SatelliteRows:
rows = SatelliteRows(self.session.scalars(statement, execution_options=execution_options))
self.streams.append(rows)
return rows
async def execute(self, statement: Select) -> Result:
return self.session.execute(statement)
@pytest.fixture
def satellite_db() -> Iterator[SatelliteSession]:
engine = create_engine("sqlite://")
CollectedData.__table__.create(engine)
with Session(engine) as session:
yield SatelliteSession(session)
engine.dispose()
def satellite(
record_id: int, *, current: bool = True, task_id: int = 1,
name: str | None = None, source: str = "celestrak_tle",
) -> CollectedData:
return CollectedData(
id=record_id,
source=source,
source_id=str(record_id),
data_type="satellite_tle",
name=name or f"SAT-{record_id}",
is_current=current,
task_id=task_id,
collected_at=datetime(2026, 1, 1, tzinfo=UTC) + timedelta(days=task_id),
extra_data={
"norad_cat_id": record_id + 40000,
"tle_line1": f"line-1-{record_id}",
"tle_line2": f"line-2-{record_id}",
},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("limit", [None, 1, 501])
async def test_streamed_satellites_preserve_all_rows_order_tle_and_limit(
satellite_db: SatelliteSession, limit: int | None,
) -> None:
satellite_db.session.add_all([satellite(record_id) for record_id in range(1, 1004)])
satellite_db.session.add_all([
satellite(2000, name="Unknown"),
satellite(3000, current=False),
satellite(4000, source="other"),
])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=limit, db=satellite_db)
expected_ids = list(range(1003, 0, -1))[:limit]
assert payload["count"] == len(expected_ids)
assert [feature["properties"]["id"] for feature in payload["features"]] == expected_ids
assert [feature["properties"]["tle_line1"] for feature in payload["features"]] == [
f"line-1-{record_id}" for record_id in expected_ids
]
assert len(satellite_db.streams) == 1
assert satellite_db.streams[0].closed
@pytest.mark.asyncio
async def test_streamed_satellites_fall_back_to_latest_non_unknown_task(
satellite_db: SatelliteSession,
) -> None:
satellite_db.session.add_all([
satellite(1, current=False, task_id=1),
satellite(2, current=False, task_id=2),
satellite(3, current=False, task_id=2),
satellite(4, current=False, task_id=3, name="Unknown"),
])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=1, db=satellite_db)
assert payload["count"] == 1
assert payload["features"][0]["properties"]["id"] == 3
assert all(stream.closed for stream in satellite_db.streams)
@pytest.mark.asyncio
async def test_unrenderable_current_rows_do_not_restore_historical_satellites(
satellite_db: SatelliteSession,
) -> None:
current = satellite(2)
current.extra_data = {}
satellite_db.session.add_all([satellite(1, current=False), current])
satellite_db.session.commit()
payload = await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert payload == {"type": "FeatureCollection", "features": [], "count": 0}
assert len(satellite_db.streams) == 1
@pytest.mark.asyncio
async def test_empty_satellite_catalog_returns_empty_collection(
satellite_db: SatelliteSession,
) -> None:
payload = await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert payload == {"type": "FeatureCollection", "features": [], "count": 0}
assert satellite_db.streams[0].closed
@pytest.mark.asyncio
async def test_satellite_cursor_closes_if_conversion_fails(
satellite_db: SatelliteSession, monkeypatch: pytest.MonkeyPatch,
) -> None:
satellite_db.session.add(satellite(1))
satellite_db.session.commit()
def fail_conversion(_records: Sequence[CollectedData]) -> None:
raise ValueError("invalid orbital data")
monkeypatch.setattr(visualization, "convert_satellite_to_geojson", fail_conversion)
with pytest.raises(ValueError, match="invalid orbital data"):
await visualization._build_satellites_geojson(limit=None, db=satellite_db)
assert satellite_db.streams[0].closed

View File

@@ -25,6 +25,25 @@
{
"id": "china-taiwan-penghu",
"name": "Taiwan and Penghu",
"sourceIsoA3": ["TWN"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-hong-kong",
"name": "Hong Kong",
"sourceIsoA3": ["HKG"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-macao",
"name": "Macao",
"sourceIsoA3": ["MAC"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
@@ -57,6 +76,7 @@
{
"id": "kosovo",
"name": "Kosovo",
"sourceIsoA3": ["KOS", "XKS"],
"action": "render_as_disputed_with_parent",
"parentIsoA3": "SRB",
"hoverIsoA3": "SRB",

View File

@@ -1,5 +1,3 @@
version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
@@ -32,7 +30,7 @@ services:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: planet_db
ports:
- "5432:5432"
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
@@ -45,7 +43,7 @@ services:
image: redis:7-alpine
container_name: planet_redis
ports:
- "6379:6379"
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
volumes:
- redis_data:/data
healthcheck:

View File

@@ -8,6 +8,425 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.76.20] — 2026-11-10
Released: 2026-11-10
### Highlights
- 统一 Earth 对象详情、区域新闻与算力卡片的两侧布局,改善连接线、悬停展开和拖放稳定性,并降低卫星数据查询的 ORM 内存开销。
### Added / Fixed / Improved
- 共享布局仲裁器分配左右两侧与纵向走廊,使用换侧、换序滞后减少卡片争位;手动固定优先,空间受限时保留完整卡片。
- 地理原始投影与平滑连线锚点分离,固定状态依据真实背面深度解除;连接线选择合适的入卡边并平衡折线段,减少倒钩与抖动。
- 修复新闻及算力胶囊悬停展开后的尺寸、正文和操作区显示;新闻打字在悬停展开时继续,统一选择框与标签移除样式。
- 连接测试图标补充明暗主题的悬停和按下反馈;卫星查询以 500 条 ORM 批次转换,保留当前/最新任务回退、完整 TLE、顺序与 limit 语义。
- 清理旧布局引用和无效计算,同步中英文布局、渲染及使用文档,增加布局仲裁与卫星分批查询回归。
---
## [0.76.19] — 2026-11-09
Released: 2026-11-09
### Highlights
- 改善 Earth 桌面与移动端卡片、新闻刷新和数据更新链路,降低重复抓取、缓存重建与历史事件维护的开销。
### Added / Fixed / Improved
- 统一区域卡片的展开与胶囊展示、悬停和拖拽跟随,分别应用算力/新闻卡片展开上限;提供细线/粗线引导线及移动端交互改进,修复语言切换后状态提示标题残留。
- 新闻 GET 只读取已保存内容,RSS 补抓改为独立调度;手动刷新提供可追踪状态、超时和重试,完成展示后才提示成功。
- 图层缓存按 key 合并并发构建,续租与发布校验锁持有者;纯新闻变更不再触发 summary 缓存重建。
- BGP 前缀归属使用容错转换和可独立部署的 GiST 索引;Outbox 单次写入事件,跳过纯时间戳变更,并提供有界保留与维护脚本。
- 卫星粒子以球体遮挡判断替代点精灵深度裁切;同步中英文文档,补充回归并隔离前端验证构建产物。
---
## [0.76.18] — 2026-10-02
Released: 2026-10-02
### Highlights
- 收敛 Earth 新闻 breaking 交互、卡片 hover、面板稳定性与 ticker 玻璃扫光,并修复船只地表锚点显隐。
### Added / Fixed / Improved
- breaking 数据、ticker 当前项、面板生命周期和选中态分离;展开面板时冻结卡片 DOM,后台更新在关闭时应用。
- ticker 保留原有容器与滚动,breaking 扫光独立于静态上下边线,连续 breaking 切换不重启;卡片 hover/扫光统一复用等级色相。
- 新闻卡片内容高度、首卡间距和 cruise 选中态统一;船只 marker 复用地表锚点遮挡,保持前后显隐与 BGP/算力对象一致。
- 同步 Earth 前端状态机与渲染层中英文开发文档。
---
## [0.76.17] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 ticker 连续 breaking 新闻切换时扫光动画重启导致的闪烁、回跳和断帧。
### Added / Fixed / Improved
- breaking 等级 class 只负责颜色,稳定的 active 状态负责动画生命周期。
- ticker 与卡片扫光方向统一为左到右,连续 breaking 切换不重置时间轴。
---
## [0.76.16] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 ticker breaking class 每 120ms 重置导致的边框闪烁、扫光停滞和方向反转。
### Added / Fixed / Improved
- watcher 仅在当前 breaking 等级变化时更新 class,动画可以完整播放。
- ticker 与卡片扫光方向统一为左到右,ticker 扫光覆盖完整 ticker 区域。
---
## [0.76.15] — 2026-10-02
Released: 2026-10-02
### Highlights
- 将 ticker 扫光限制在上下边线以内,消除 breaking 边框被扫光伪元素带动闪烁。
### Added / Fixed / Improved
- ticker 扫光与卡片扫光保持独立横向移动,边线保持稳定,只响应 hover/等级颜色。
---
## [0.76.14] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复卡片扫光从中途出现的问题,让反射从卡片左边界完整扫到右边界。
### Added / Fixed / Improved
- 卡片反射改用内部 `background-position` 扫描,不再使用卡片外部 transform 起始。
- ticker 上下边线保持静态,横向扫光独立运行。
---
## [0.76.13] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修正 breaking hover 强度基准,hover 会在自身 breaking 高亮上继续增强。
### Added / Fixed / Improved
- ticker hover 从 breaking 原始色相和强度继续增强,不再回落到普通新闻强度。
- breaking 卡片边框 hover 同样相对自身等级增强,保持红/橙/黄颜色连续。
---
## [0.76.12] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 breaking ticker 和黄卡 hover 被白色混合漂淡的问题。
### Added / Fixed / Improved
- hover 使用共享强度参数与等级色相,玻璃底复用深色 HUD surface。
- 红色 ticker 保持红色边线,黄色卡片保持黄色 tint,不再被白色边框稀释。
---
## [0.76.11] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修正 breaking 卡片整卡扫光层级,并将 ticker hover 亮度对齐共享 HUD 面板。
### Added / Fixed / Improved
- 卡片扫光覆层位于内容之上,整张卡片 hover/focus 时才扫过。
- ticker 上下边线 hover 使用共享 HUD 边框 hover 强度,breaking 只替换色相。
---
## [0.76.10] — 2026-10-02
Released: 2026-10-02
### Highlights
- 将 breaking 卡片扫光限制为当前 hover/focus 卡片,避免所有卡片同时动态闪烁。
### Added / Fixed / Improved
- 卡片反射层默认关闭,只在当前交互卡片上启动对应等级颜色扫光。
- ticker 的全宽 breaking 扫光保持独立,不影响卡片浏览。
---
## [0.76.9] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复 breaking 卡片黄色 hover 变淡,并补齐 ticker 上下边线的 hover 高亮反馈。
### Added / Fixed / Improved
- 卡片 hover 使用不透明等级色相和统一混合强度,红/橙/黄表现一致。
- ticker hover 只强化上下两条边,普通新闻使用蓝色,breaking 使用对应等级颜色。
---
## [0.76.8] — 2026-10-02
Released: 2026-10-02
### Highlights
- 修复展开新闻面板时后台 ticker / realtime 更新导致卡片重建和 hover 抖动的问题。
### Added / Fixed / Improved
- 面板浏览期间冻结卡片 DOM,后台更新只写入最新 payload。
- 关闭面板时一次性应用待处理新闻,保持滚动位置和 hover 稳定。
- 展开面板时暂停隐藏 ticker,手动刷新仍可立即更新面板内容。
---
## [0.76.7] — 2026-10-02
Released: 2026-10-02
### Highlights
- 统一 breaking 新闻卡片的 hover 交互,让通用 hover 行为叠加卡片自身等级颜色。
### Added / Fixed / Improved
- 红、橙、黄 breaking 卡片共用同一套 hover 提亮、边框增强和浮起逻辑。
- 等级颜色通过卡片变量注入,避免为每种颜色复制 hover 样式。
---
## [0.76.6] — 2026-10-02
Released: 2026-10-02
### Highlights
- 扩展 breaking ticker 的全域扫光效果,并拉开新闻卡片与上方状态内容的间距。
### Added / Fixed / Improved
- ticker 反射从局部光斑改为覆盖整条横向区域的扫光/闪烁带。
- 桌面与移动新闻列表首卡预留顶部呼吸空间,保持内容区层次。
---
## [0.76.5] — 2026-10-02
Released: 2026-10-02
### Highlights
- 增加 breaking 新闻的玻璃反射式警灯脉冲,并修复新闻卡片内容被 flex 压缩的问题。
### Added / Fixed / Improved
- ticker、卡片边框和卡片反射统一使用同一 breaking 等级颜色。
- breaking ticker 使用局部红/橙高光脉冲,卡片使用反射带扫过;面板底色保持蓝色。
- 新闻卡片固定非收缩布局并预留最小内容高度,避免只剩媒体名和时间。
- 同步中英文 Earth 前端开发文档,记录视觉状态与 reduced-motion 行为。
---
## [0.76.4] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修正 breaking 新闻状态显示:ticker 仅在当前滚动到 breaking 新闻时变红,展开面板始终保持蓝色。
### Added / Fixed / Improved
- 选中、焦点、关闭动画与 breaking 数据状态彻底分离,关闭新闻栏不再留下橙色边线。
- breaking 新闻按过期时间重新计算,单条新闻卡片继续保留自身 breaking 红色样式。
- 增加 ticker breaking 状态单测和浏览器关闭/重开验证。
---
## [0.76.3] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修复新闻栏关闭后残留橙色边线的问题,明确 breaking、展开面板和新闻选择的状态边界。
### Added / Fixed / Improved
- ticker 保持中性,breaking 等级只作用于展开新闻面板整体背景和阴影。
- 关闭新闻面板时清理选中故事和过期焦点状态,breaking 按新闻过期时间重新计算。
- 增加 breaking 状态单测,保持新闻滚动、选择和面板动画行为不变。
---
## [0.76.2] — 2026-10-01
Released: 2026-10-01
### Highlights
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
### Added / Fixed / Improved
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
---
## [0.76.1] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
### Added / Fixed / Improved
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
---
## [0.76.0] — 2026-09-30
Released: 2026-09-30
### Highlights
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
### Added / Fixed / Improved
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
---
## [0.75.4] — 2026-09-29
Released: 2026-09-29
### Highlights
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
### Improved / Fixed
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
---
## [0.75.3] — 2026-09-27
Released: 2026-09-27
### Highlights
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
### Improved / Fixed
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
---
## [0.75.2] — 2026-09-25
Released: 2026-09-25
### Highlights
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
### Fixed
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
---
## [0.75.1] — 2026-09-25
Released: 2026-09-25
### Highlights
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
### Added / Fixed / Improved
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
---
## [0.75.0] — 2026-09-23
Released: 2026-09-23
### Highlights
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
### Added / Fixed / Improved
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
---
## [0.74.7] — 2026-09-17
Released: 2026-09-17
### Highlights
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
### Added / Fixed / Improved
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
---
## [0.74.6] — 2026-09-16
Released: 2026-09-16
### Highlights
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
### Added / Fixed / Improved
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
---
## [0.74.5] — 2026-09-13
Released: 2026-09-13
### Highlights
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
### Added / Fixed / Improved
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
---
## [0.74.4] — 2026-09-13
Released: 2026-09-13

View File

@@ -79,14 +79,24 @@ git diff --unified=0 HEAD -- <path>
## Harness Commands
Windows desktop launcher tests run separately on Windows with
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
They use temporary desktop folders and mocked restart commands; no UAC prompt
or real service restart is triggered. The WSL bridge regression suite,
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
disposable command substitutes. For a real entry-point check without restarting,
run the repository `planet.cmd --install-only`, then the generated desktop
launcher with the same flag. Installation and mocked checks do not prove a live
elevated restart; report that separately when it has not been exercised.
| Tier | Command | What It Does |
| --- | --- | --- |
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, and documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
Docker image smoke builds are expensive and are off by default:
@@ -151,11 +161,6 @@ PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
controls should reuse the shared component, shared token, or established CSS
owner for that area instead of introducing a page-local lookalike.
- `scripts/harness/frontend-rules-check.sh` warns when semantic
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
approved React and Earth CSS owner files. A warning means the reviewer should
either move the style into the shared owner or document why this is a genuinely
new visual family.
### Earth I18n Harness Rules
@@ -209,6 +214,12 @@ required tool is not visible there, `scripts/harness/lib.sh` asks the user's
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
path. This avoids hardcoding a dotfile while still covering agent environments
that do not inherit the user's normal shell setup.
Validation adds the resolved Bun directory to its process `PATH` so package
scripts can invoke Bun as well. `test_tool_resolution.py` exercises this through
the validation entry point with an isolated login-shell tool path.
The validation build and preview share a temporary output directory that is
removed after the run. This gives the smoke a consistent build while local
development tools may rebuild `frontend/dist`.
Required for full local stack operation:
@@ -257,8 +268,7 @@ modify the host Docker environment.
No automatic hooks are installed in this phase. Manual reminders:
- Run `scripts/harness/quick-check.sh` before handing off small changes.
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
- Choose the verification scope using the [validation guidance in the agent guide](../AGENTS.md#validation).
- Run `scripts/harness/security-check.sh` after touching config, auth,
credentials, docs examples, or generated fixtures.
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
@@ -280,7 +290,7 @@ No automatic hooks are installed in this phase. Manual reminders:
2. Check `CODEMAP.md` for entry points and ownership boundaries.
3. Inspect existing tests and docs before editing.
4. Make the smallest behavior-preserving or feature-scoped change.
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
5. Run the checks for the affected behavior and any gates required for that area.
6. Update relevant docs when behavior, workflow, or operations change.
7. For rendered frontend changes, verify the affected route with Playwright or
the full harness smoke, because `bun run build` alone does not prove page
@@ -291,8 +301,6 @@ No automatic hooks are installed in this phase. Manual reminders:
1. Reproduce with a focused test or command.
2. Patch the owning module, not a caller-side workaround.
3. Run the focused regression test.
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
locally.
### Documentation Change

View File

@@ -14,6 +14,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
## Scope Rules
- Focus documentation changes on the primary document that owns the affected behavior. Do not trigger speculative updates across secondary or architectural docs unless the underlying contract or operator-facing workflow actually changed.
- User-visible workflow changes must update `docs/technical/zh/manual.md` and usually `docs/technical/zh/quickstart.md`.
- If an English counterpart exists for user-facing docs such as `manual.md` or `quickstart.md`, update `docs/technical/en/...` enough that it does not contradict the Chinese source.
- Control console page responsibility changes must update `docs/technical/zh/frontend-admin-frontend-context.md`.
@@ -23,6 +24,7 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
- Collector, datasource, credential, settings, connectivity, scheduler, or API changes must update the relevant backend docs, especially `docs/technical/zh/backend-collectors.md` and any datasource/settings-specific doc.
- When a change turns an old plan assumption into current behavior, update the relevant `docs/plans/*.md` with a status note instead of leaving contradictory instructions.
- Search docs for stale terms introduced by the change, for example old tab names, old route responsibilities, obsolete auth assumptions, or renamed UI labels.
- Internal bug fixes or optimizations that preserve existing APIs and workflows do not require updating user manuals or creating new documentation.
## Public Docs Rules
@@ -42,86 +44,17 @@ If the same action has both a UI and a CLI path (e.g. user creation), describe t
## Recommended Checks
Run the checks that match the affected docs.
### Duplicate Bilingual Docs
Always run the centralized harness documentation gate:
```bash
python - <<'PY'
from pathlib import Path
same = []
for en in sorted(Path("docs/technical/en").glob("*.md")):
zh = Path("docs/technical/zh") / en.name
if zh.exists() and en.read_text() == zh.read_text():
same.append(en.name)
if same:
raise SystemExit("identical en/zh docs: " + ", ".join(same))
print("no identical en/zh docs")
PY
scripts/harness/docs-consistency-check.sh
```
### Language-Less Technical Links
```bash
rg -n "/home/ray/dev/linkong/planet/docs/technical/(?!zh|en)" docs/technical/zh --pcre2
```
This should return no matches.
### Public Docs Registry
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
known = set(re.findall(r"'([^']+\.md)':\s*\{", metadata))
known.add("README.md")
missing = []
for readme in [Path("docs/technical/zh/README.md"), Path("docs/technical/en/README.md")]:
if not readme.exists():
continue
for href in re.findall(r"\]\(([^)]+\.md)\)", readme.read_text()):
path = Path(href)
if "docs/technical/" not in href:
continue
filename = path.name
if filename not in known:
missing.append(f"{readme}: {filename}")
if missing:
raise SystemExit("docs README links missing DOCS_METADATA: " + ", ".join(missing))
print("docs README links are whitelisted")
PY
```
### Public Bilingual Pairs
```bash
python - <<'PY'
import re
from pathlib import Path
metadata = Path("frontend/src/pages/Docs/docs-content.ts").read_text()
filenames = sorted(set(re.findall(r"'([^']+\.md)':\s*\{", metadata)) - {"README.md"})
missing = []
for filename in filenames:
for lang in ("zh", "en"):
path = Path("docs/technical") / lang / filename
if not path.exists():
missing.append(str(path))
if missing:
raise SystemExit("missing bilingual docs: " + ", ".join(missing))
print("public docs have zh/en file pairs")
PY
```
### Raw Filename Link Titles
```bash
rg -n "\[[^]]+\.md\]\(" docs/technical/zh docs/technical/en
```
This should return no matches for polished public docs.
This single command automatically covers:
1. Public docs whitelist and `DOCS_METADATA` synchronization (`frontend/src/pages/Docs/docs-content.ts` vs backend Gatekeeper)
2. Technical doc bilingual file pairs and identical duplicate-content detection
3. Public doc link validity and human-readable link titles (no raw `.md` filenames)
4. Language-scoped internal links (`docs/technical/zh/` links pointing to `docs/technical/zh/`)
5. Supported credential collector contracts and UI guide registration
6. Manual console route coverage against actual admin manifests
7. Documented `?section=` deep-link validity and UI route drift in active docs

View File

@@ -28,8 +28,7 @@ compatibility note, not a replacement for existing rules or architecture docs.
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
| `.codex/skills/*/SKILL.md` | Present | Local workflows for cleanup, docs, goal-driven work, and release. |
## Existing CI Gates
@@ -57,8 +56,7 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
## Release And Deploy Process
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
`.claude/commands/release.md`.
- Release workflow is documented in `.codex/skills/release/SKILL.md`.
- Version-bearing files include `VERSION`, `frontend/package.json`,
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
`docs/version-history.md`.
@@ -102,7 +100,6 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
@@ -137,7 +134,7 @@ index is the routing layer; this table is the coverage/evidence layer.
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
| `uiux` | Related visual surfaces should use one style system per product area. | Manual visual review. | Compare changed controls with the shared component on the rendered page. |
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
@@ -154,8 +151,8 @@ index is the routing layer; this table is the coverage/evidence layer.
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, and admin-config-derived section deep-link consistency. |
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |

View File

@@ -22,12 +22,14 @@
当前重点入口:
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)

View File

@@ -0,0 +1,602 @@
# 算力与资源态势:展示、采集与 AI 迭代实施计划
状态:首版空间卡片原型实现中(2026-09-21)。已接入算力专题入口、45 个参与方目录、年份/口径选择、区域数据卡与来源、成功快照设施查询,以及 Epoch/TOP500 的首批质量修复。旧固定双图布局已移除。历史年度数据使用受控版本导入;各国完整指标覆盖、卡片图表、全榜获取、跨源设施合并、持久化指标管理、变化归因与 A4 AI 解读尚未完成,不视为全部 A1—A3 已验收。后续章节保留完整设计目标,当前实现以技术文档为准。2026-09-14 完成的口径核查仍是内容依据。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
## 0. 当前执行方向与范围
2026-09-21 展示方向修订:UI、地球材质、政治展示规则、地理资源对齐及性能门槛以[Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)为准。下文固定右侧国家面板、底部横贯双图等布局不再作为实施目标。采用从区域锚点引出连接线、卡片围绕地球自适应浮动的方向,先验证空间交互再完善卡片内容。数据来源、口径与证据要求继续有效。
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚:地图显示资源在哪里,图表说明规模和相对位置怎样变化,证据说明数字从哪里来,AI 帮助阅读变化。
采用“专题先交付,公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线,不是首版展示的前置条件;第 6 节总分保持实验性,不能为了首屏有大数字而生成不完整排名。
| 当前要做 | 当前完成形态 | 后续才扩展 |
| --- | --- | --- |
| 算力数据可信 | 修复既有采集,国家/设施/精度/年份不混用 | 大规模自动来源发现、完整证据图谱 |
| 全球分布与国家对比 | 现有算力中心地球 + 可比参与方 + 明确样本覆盖 | 制造、矿产、电力和贸易的跨主题关系 |
| 历史展示 | 两张核对后的图、时间选择、已知事件轨道 | 复杂预测、不确定性传播与可训练参数 |
| 证据与解释 | 来源、参考年、关键字段;基于数据快照的简短 AI 说明 | 自动补证、模型候选、回测、影子运行与提升 |
首版验收路径:打开算力专题 → 看全球公开设施分布 → 选中国和美国 → 同时看算力与创新历史 → 点一个变化或设施 → 查看出处及解释。此路径稳定后,才扩大评价维度。
## 1. 要交付什么
在智能星球现有算力中心图层上增加“算力与资源态势”模式,并在控制台提供对应的数据管理和模型实验工作区。核心问题是:各方拥有什么资源、资源如何形成可用能力、相对位置如何变化、变化有何证据,以及未来什么约束可能改变格局。
首批比较中国和美国;数据结构从第一天支持多方。随后纳入欧洲、日本、韩国、印度及其他数据合格的参与方。地区汇总与成员国不得重复计入全球分母。所有国家和地区字段使用现有 `normalize_country()` 和规范字典,不以网页原文或 AI 判断覆盖项目标签。
交付三个相互连接的产品能力:
| 能力 | 用户获得的结果 | 首版边界 |
| --- | --- | --- |
| 历史观测 | 算力绝对规模、全球份额、创新指数与分差、设施时间线 | 使用可验证观测;缺失年份可见,不插值冒充事实 |
| 变化解释 | 哪些设施、资源、数据修订造成指标变化,每项贡献和证据 | 数学贡献可复算;原因解释区分事实、假设与未解问题 |
| 模型改进 | AI 发现缺口、补证、提出改进,经过验证产生新版本 | 固定工作流先行;不能由大模型直接决定国家分数 |
第一版先交付可信的事实与回放,不以总分完整作为上线前提。完整创新指数分项、有效算力参数和项目交付数据不足时,继续展示已完成的观测能力,综合评分显示“数据不足”,不生成示意国家排名。
不在第一版承诺全球所有设施的完整清单、精确的国家真实算力、每日变化的国家创新能力、可验证的国家综合实力真值,或自动预测某年必然追平。
## 2. 当前项目基础与必须补齐的边界
以下结论来自当前文件检查,未对生产数据库、凭证可用性或线上采集结果做完整审计。
| 已有入口 | 已确认能力 | 本专题需要补齐 |
| --- | --- | --- |
| [采集基类](../../backend/app/services/collectors/base.py)、[数据作业](../../backend/app/services/data_jobs.py) | 采集、转换、任务状态、快照、取消与重试基础 | 文档型数据源、质量门槛、领域规范化、失败保留正式快照 |
| [Epoch 采集器](../../backend/app/services/collectors/epoch_ai.py) | GPU 集群采集入口 | 改读公开 CSV;去掉解析失败返回示例记录;单位、精度、状态和时间字段校正 |
| [TOP500 采集器](../../backend/app/services/collectors/top500.py) | 榜单与详情解析 | 去掉示例回退、固定参考日期、排名作为身份;按当期表头解析单位 |
| [采集记录](../../backend/app/models/collected_data.py)、[快照](../../backend/app/models/data_snapshot.py) | `entity_key`、快照关联、前一记录和变化摘要 | 原始证据版本、发表时间、有效时间、指标注册和可重放的观测集合 |
| [算力位置服务](../../backend/app/services/compute_center_locations.py)、[GeoJSON](../../backend/app/api/v1/visualization.py) | 设施定位、位置置信信息、未定位记录 | 国家统计与地图可定位集合分离;匿名设施不能猜坐标 |
| [算力中心渲染](../../frontend/public/earth/js/compute-centers.js)、[Earth 壳](../../frontend/src/pages/Earth/Earth.tsx) | Three.js 交互层;Earth 由 iframe 加载独立页面 | 历史时点、国家比较、双图、关系层和解释面板 |
| [图层适配器](../../backend/app/services/earth_layer_adapters.py)、[数据库监听](../../backend/app/services/earth_db_change_listener.py) | 通过 PostgreSQL outbox 驱动缓存与 `earth_updates` | 正式指标快照发布事件;草稿与实验结果不改变默认地球 |
| [AI Client](../../backend/app/services/ai_client.py)、[提示词注册](../../backend/app/ai_tasks/prompts.py) | 全局 provider 配置、模型调用、任务提示词及运维覆盖 | 算力专题的固定工作流、结构化输出验证、预算与审计 |
| [搜索工具](../../backend/app/services/ai_tools/web_search.py)、[网页取证](../../backend/app/services/ai_tools/web_fetch.py) | 搜索和基础 HTML 正文提取 | PDF/表格提取、引用位置、下载时大小限制、URL 与重定向的访问限制 |
| [证据辅助函数](../../backend/app/services/ai_tools/evidence_store.py) | 摘要、去重和内容哈希规范化 | 该文件不是持久化证据库,需补建不可变原文存储和证据记录 |
沿用[数据作业与 Outbox 架构](../technical/zh/data-job-earth-sync-architecture.md)及[轻量 Agent 编排计划](agents-light-orchestrator-websearch-plan.md):PostgreSQL 负责可靠账本,Redis 负责缓存;不另起 Kafka、Celery 或另一个独立调度真相源。`aiprovider` 保持协议适配,业务取证、估计、工具权限和模型发布全部由后端负责。
当前 `JobType` 只有采集、清理和 Earth 刷新。新增分析、实验工作流需要显式扩展任务类型、worker 分发和取消策略,不能把现有采集队列描述成已经支持通用 Agent。
## 3. 页面与地球怎样展示
### 3.1 默认工作区
保留 `/earth` 主入口,在现有算力中心工具中增加专题模式,避免增加一个与地球无关联的新大屏。进入后使用以下概念布局;这是规划线框,不是现有页面截图。
```text
┌ 时间/参考期 · 中国/美国/其他参与方 · 指标口径 · 观测/情景 ┐
│ │
│ 3D 地球:设施、资源、已证实关系 国家对比面板 │
│ 绝对量/份额 │
│ 变化贡献 │
│ 证据与缺口 │
├────────────── 历史时间轴 / 事件轨道 ────────────────────┤
│ 算力规模与份额图 综合创新指数与分差图 │
└───────────────────────────────────────────────────────┘
```
桌面默认同时显示两张历史图,用相同国家配色,但各自保留单位和实际观测年份。图表高度优先保证刻度、来源标识与数据点可读,不能通过压缩字体维持布局。较窄屏幕把图表移入可切换的详情区,国家和设施详情使用抽屉;不叠加多个遮挡地图的浮层。
沿用项目一屏高度链和主题。桌面在 1366×768、1920×1080,移动端在约 390×844 验证,另测 125%/150% 浏览器缩放。页面根与中间 flex 容器分别正确设置高度和 `min-height: 0`,仅详情、长表和证据正文内部滚动。
### 3.2 视图职责
| 视图 | 主内容 | 点击或切换后的行为 |
| --- | --- | --- |
| 国家总览 | 可比算力绝对量、份额、子分、实际参考期;总分合格才出现 | 选中参与方,定位地球并联动两图;不重设全局评分尺度 |
| 算力趋势 | 绝对规模、全球份额、双边份额三种独立视图 | 同一来源口径内切换;不把百分比轴和 FLOP/s 轴混合 |
| 创新趋势 | 各方指数;只有分差时只显示中美分差 | 切换年度,查看分项与报告版本,不能反推缺失的单国得分 |
| 变化账单 | 新投产、扩建、退役、效率变化、资料补录、方法修订 | 点选贡献定位资产;逐步展示输入、计算和证据 |
| 设施详情 | 设备、精度、投产状态、功率、运营方、地点精度、历史版本 | 展开相关公告、设备来源及不确定字段 |
| 资源关系 | 已证实的芯片供给、所有权、电力接入或网络连接 | 选一类关系再显示,不能让所有关系默认铺满地球 |
| 情景工作区 | 并网推迟、交付变化、效率假设及结果区间 | 独立 scenario ID;退出恢复正式快照 |
| 控制台研究区 | 数据缺口、待核验证据、采集任务、模型实验和版本 | 使用已有采集与 AI 设置入口;Playground 仍用于调试 |
### 3.3 两张图进入产品的明确规则
总算力图以核对后的报告观测为起点:2020 年美国/中国为 36%/31%,2021 年 34%/33%,2022 年 34%/33%,2023 年 41%/31%。原始来源和报告版本见第 15 节。2024—2025 年暂不填值;2026 年 42%/33% 只保存为待核实主张,不能接到正式历史曲线上。
创新图使用报道明确披露的 2023、2024、2025 年分差 22.02、19.96、17.95 分。报告发表于 2026 年,必须区分报告年份与参考年份。完整分项、年度得分和方法修订仍需采集。图中是综合创新指数,不是纯算力或纯产出。
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读,不进入计算;不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度,不让刷新网页造成“国家实力实时跳动”。
### 3.3a 当前数据合同(2026-09-22)
默认主指标是全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。CAICT 报告核查目前使用官方 PDF 索引文本;本地 PDF 下载未成功,因此不宣称已肉眼核验 PDF 页面。绝对量合同记录 2022 年 302(证据版本 `caict2023`)和 2023 年 435(证据版本 `caict2024`);2024 年报告正式书名为《先进计算暨算力发展指数蓝皮书(2024年)》。官方 URL 已替换为四个镜像来源。
日本全球份额观测为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 `APAC` 与 `OTHER` 不能直接套用报告中的 Japan 或“其他”;`EUROPE` 成员集合尚未与报告口径对齐,仍返回缺失。双边份额分母固定为 US + China,新加入日本不改变分母。主要指标缺失时可显示最近有报告的年份(当前可到 2023),而不是留空为 2026;用户明确选择 2026 时仍显示缺失,不回填最近值。不可比性尚未确认,不展示未经确认的同比变化;覆盖数据保留在辅助区并带 count。
数据 API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现,用户对该问法的选择待后续确认;当前合同不提前新增该模式或声称完成 dataset 计算。
默认“历史回顾”允许查看最新证据重述后的过去;“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时,必须显示“最新观测:2023”,或在严格同年模式显示缺失。
### 3.4 地球编码与交互
- 节点面积按同口径能力编码;跨度过大时可切换对数档位并明示图例。国家颜色只用于身份,不能把颜色本身当作领先或落后。
- 已投产用实心,建设或情景用空心/虚线;置信不足使用独立标记,不能把低置信透明度误读为小算力。
- 来源只到国家或区域时,保留在国家/区域统计和未定位列表;不画在首都,不把行政区域中心当作设施坐标。
- 国家总量与可见设施合计可以不同,分别显示覆盖范围。不得只统计有坐标的节点,也不得把宏观缺口按现有节点比例摊分。
- Epoch 当前公开版本对中国部分集群匿名化并取整;匿名记录只能按允许的粒度使用。新增官方公告可作为独立证据,不能依据匿名数值猜出设施身份或制造地点。
- 供应、所有权和网络关系采用不同图例,显示关系来源与有效期;商业供应关系不等同于实际运输路线。
- 全国可汇总吞吐与最大单集群能力分别展示;地理分散设备不能合并成一个训练集群。
- 时间回放使用稳定实体 ID 更新已有节点。隐藏图层、变更时点或情景时同步清理 hover、locked、tooltip 和 selection;请求过期结果不得覆盖新时点。
- 使用现有 Three.js Interactable、图标和实例化能力。新增壳层必须遵循现有深度顺序,在远距约 50% 缩放检查闪烁;不另建悬浮球面掩盖深度问题。
## 4. 数据需要补充采集什么
### 4.1 P0:决定首版可信度的数据
“已核对入口”只表示资料或文档可查,不表示接口凭证、完整历史和生产连通性已经验证。以下频率是 Planet 建议检查频率,不是来源承诺的更新频率。
| ID / 优先级 | 数据与来源 | 必需字段/粒度 | 采集方式与频率 | 当前缺口与采用条件 |
| --- | --- | --- | --- | --- |
| D01 / P0 | 信通院历年算力报告 | 国家×参考年×口径;绝对规模、全球总量/份额、单位、精度、估算方法、版本 | 已取得报告先受控导入;月度检查新版;PDF 表格提取后核验 | 已核对 2020—2023 份额及部分绝对量;补 2024 以后和其他国家;跨版本方法改变时断开序列 |
| D02 / P0 | Epoch GPU Clusters | 集群/版本;设备数、型号、16/8/32-bit 性能、所在地、状态、投产/退役时间、功率、扩建关系 | 官方公开 CSV,每日条件抓取,按内容哈希去重 | 改造现有 HTML 解析;处理匿名、取整、缺坐标和覆盖偏差;不能声称完整全国清单 |
| D03 / P0 | TOP500 / 可选 Green500 | 稳定系统 ID×榜单期;Rmax/Rpeak、FP64、功率、机构、名次、列表日期 | 每周检查新榜单;按发布期导入历史与详情 | 修复 rank 作为 source ID、固定日期和单位问题;榜单消失不等于退役;不当成 AI 总算力 |
| D04 / P0 | 全球 AI 创新指数原报告及发布材料 | 国家×参考年×指标;单国总分、分项、权重、标准化、成员范围、方法版本 | 文档导入+月度更新检查 | 当前只具备部分新闻转述分差;优先获得分项和年度可比性证据;不能用差值合成各国历史得分 |
| D05 / P0 | 已有位置维表与公开设施公告 | 实体别名、运营方、国家、地点、精度、采用依据、有效期 | 复用位置管线,新增/变化时触发 | 全国汇总独立于定位成败;不把同址不同分期误合并,也不把扩建当第二个全量集群 |
首版最低闭环:D01 和 D04 的已验证历史可先支撑双图,D02/D03/D05 支撑设施地图;两者不互相伪装为同一覆盖范围。若 D04 分项不可取得,创新总分仍作为独立观测,O 分与综合 S 不发布。
首版可以先受控导入已核实的年度数据,不必先完成整套 PDF 自动解析和报告发现。D04 原报告分项继续作为补数任务,不阻塞已经核对的创新分差图;原始摘录、出处、年份和未核实范围必须随数据保存。
### 4.2 P1/P2:有效能力、增长与资源关系
| ID / 优先级 | 补充数据与候选来源 | 粒度与核心字段 | 方式/频率 | 对模型的作用及限制 |
| --- | --- | --- | --- | --- |
| D06 / P1 | Epoch 芯片销售、所有权、硬件数据;厂商规格 | 芯片型号×时间;规格、销售/交付、持有主体、数量、性能换算、估计区间 | 官方数据文件与规格文档;周查/季度快照 | 校验硬件供给;设计者、所有者、实际所在地与使用权分开;不能把出货和已部署重复加总 |
| D07 / P1 | MLPerf Training / Inference、可复现厂商测试 | 配置×任务×软件版本;模型、质量门槛、batch、上下文、时延、吞吐、能耗、可用状态 | 官方结果导入;月查新版本 | 校准特定任务的性能系数;不同 benchmark 版本/质量/负载不直连;实验室成绩不等同生产利用率 |
| D08 / P1 | IEA、电力统计机构、EIA、并网与运营方资料 | 国家/地区/项目×时点;发电、用电、接入 MW、PUE、并网日期、可用负荷 | 国家统计月/季;报告年;项目公告日查 | 电力作为设施约束;国家发电量、用电 TWh 不能直接转换机房可用 MW;中国与美国地方数据可得性不同 |
| D09 / P1 | 项目运营方、许可部门、官方采购和工程进展 | 项目×阶段;公告、开工、设备交付、并网、验收、部分投产、撤销、金额与范围 | 已注册来源每日检查,重要项目每周核验 | 建立投产与延期标签;最终失败项目也保留,避免只收集成功项目;单纯新闻未更新不判定失败 |
| D10 / P1 | 公司投资者关系、SEC EDGAR、境内及其他市场法定披露 | 法人/分部×财季;CapEx、已付款/指引、租赁、合同、地域/业务归属 | 官方 API/报告;日查新披露,按季归档 | 用于资金与项目约束;总 CapEx 不是 AI 投资;合并报表、融资租赁、设备与机房成本须防重复 |
| D11 / P1 | 创新报告分项、公开模型评测、OpenAlex;HF 作为辅助 | 机构/国家×年/月;研发成果、质量归一化产出、可比模型能力、应用指标 | 原报告年;评测周;研究元数据月 | 重建不含重复资源项的 O;合作成果使用明确分摊;不从作者姓名推断国籍;下载量不代表实际使用或产业产出 |
| D12 / P2 | 晶圆制造、先进封装、HBM、关键设备的正式披露 | 工厂/供应方×工艺/产品×季度;已投产能力、良率范围、交付、客户与依赖 | 公司/监管原件;月查、季度快照 | 建立瓶颈和替代关系;晶圆数不能无依据换成 GPU 数;总行业份额不能归给某集群 |
| D13 / P2 | USGS、各国统计、UN Comtrade | 商品×国家×年/月;产量、储量、加工量、贸易流、HS 版本和单位 | 年度报告、月/年贸易数据 | 扩展矿产与资源依赖;储量、开采和加工分开;宽泛服务器税号不能反推高端芯片数量 |
| D14 / P2 | 既有海缆、PeeringDB、BGP 与已证实连接资料 | 设施/网络/关系×有效期;容量声明、连接、依赖、故障 | 复用现有产品,按其真实更新节奏 | 提供连通性和韧性背景;BGP 可见性不是带宽,海缆容量不是集群内部互联性能 |
EIA、SEC、Epoch、MLCommons、IEA、USGS 已核对资料入口;UN Comtrade 门户可见,但本次未验证可调用 API,接入要单独完成认证、额度、字段和历史测试。OpenAlex 需在接入时按最新访问规则配置,不假定无限制匿名访问。所有付费、登录和受限数据只列为可选增强,不作为首版隐藏前提。
### 4.3 首批采购与补数优先顺序
1. 先完成现有 Epoch/TOP500 的真实性与时间口径修复,避免新增来源扩大错误。
2. 导入信通院历史、创新分差和原始出处;追索创新原报告分项及最新版总算力数据。
3. 建立中美重点设施及项目阶段台账,并记录未知量。重点设施用于高影响解释,不冒充全国抽样无偏估计。
4. 用官方规格与 MLPerf 建立少量明确工作负载的换算表;其余硬件维持理论值或范围。
5. 补项目级并网、交付、资金与退役记录,积累可校准的真实标签。
6. 最后扩展制造、HBM、矿产与多方关系;每个新主题必须有对应指标和展示消费者,不只收集而不使用。
## 5. 采集到正式数据的流水线
```mermaid
flowchart LR
A[注册的数据源] --> B[原文与内容哈希]
B --> C[确定性解析或 AI 提取候选]
C --> D[单位/时间/实体/引用校验]
D --> E[版本化有效观测]
D --> F[冲突与待核实队列]
E --> G[确定性指标计算]
G --> H[正式结果快照]
H --> I[Outbox 与缓存更新]
I --> J[地球/双图/变化账单]
```
### 5.1 数据源合同
每个数据源保存来源机构、域名与下载入口、文档类型、凭证配置引用、检查频率、预期粒度、单位与精度、更新时间语义、空结果语义、历史范围、使用许可、维护负责人和解析器版本。授权文件或数据保存在运行数据存储,不提交到仓库;公开 UI 只展示允许公开的摘录或链接。
API/CSV 优先;HTML/PDF 采用固定解析;必须用 AI 时先产生候选,核验后保存确定性映射。扫描件 OCR 必须保留页号、表格区域和原图,数值不能只靠模型复述。下载器在流式读取时限制实际字节量,不能下载完整文件后才截断;按域名限速,遵守来源规则。
### 5.2 事实键与时间
观测键至少包括实体、指标、单位、精度、工作负载、地理范围、归属口径、参考期及来源版本。历史至少区分:
| 时间 | 含义 |
| --- | --- |
| `reference_start/end` 或 `effective_at` | 数据描述的年度、季度或设施事件发生时间 |
| `published_at` | 资料何时首次公开;不知道就保持未知 |
| `retrieved_at` | Planet 何时取得该版本 |
| `recorded_at` / `superseded_at` | 系统何时采用、替换该观测 |
抓取日期不代替投产日期;报告年份不代替参考年份;机构估计、实测、公告、模型估计和人工情景分别标识。只有年份时保留年份精度,不能伪造 1 月 1 日投产;回放按时间范围或“当年”事件显示。
### 5.3 合并与质量门槛
- 稳定身份优先使用官方系统 ID、项目 ID、法人 ID 和可信来源链接。TOP500 排名只作为观测属性。自动实体合并必须可撤销,保存别名和合并理由。
- 同一集群的全量扩建公告、一期/二期、重复转载与另一个数据集的同一设备只计一次。芯片交付和机房投产分别保留,不能合并为两个已投产资产。
- 国家级统计、企业所有权统计、设施地理统计是不同视图。国家公司在海外租用算力不得同时作为两国境内部署能力;全球成员集合固定且版本化。
- 转换单位前先核验浮点精度、稀疏/稠密、峰值/实测和时间量纲。FLOP 与 FLOP/s、TWh 与 MW 均不得互换。
- 解析为空、字段突然缺失、总量异常下降或源结构改变,进入失败/隔离状态,保留上次正式结果。缺榜、缺行或匿名化不直接触发实体退役。
- 多条转载归到同一 `source_family`;媒体数量不构成多份独立证据。冲突先检查口径,再保存多个主张及采用规则,不能简单取平均。
- 缺坐标不影响已知国家统计;未知国家不强行分配。数据覆盖率可描述已登记实体或必填字段的完整率;无法知道总体时不声称“覆盖全球百分之多少”。
- 一位有效数字等取整值保留取整范围;这是观测精度范围,不冒充统计置信区间。
首批历史导入先双人或规则加人工抽查核验高影响记录,保存一套回归样本。后续同版本确定性解析通过固定门槛可自动采用;新来源、新语义或无法解释的大幅变化进入候选队列。
## 6. 数学模型如何工作
### 6.1 分清事实、估计和评价
模型分三层:观测层保存来源说了什么;估计层回答可用能力、交付时间和不确定性;评价层按明确价值取向组合指标。国家综合实力没有一个可直接取得的标签真值,因此不能声称“AI 找到了客观最优的国家评分权重”。
保留 v2 的模型草案:
\[
S_{c,t}=0.40H_{c,t}+0.30O_{c,t}+0.20G_{c,t}+0.10R_{c,t}
\]
H 是资源能力,O 是不重复的创新与应用表现,G 是扩张潜力,R 是韧性;四项均为 0—100 的版本化子分。S 称为“综合态势分”,包含未来潜力,不能称作已投产算力、胜率或国家实力百分比。权重是默认设计参数,需通过敏感性分析和评价目的评审后冻结。
### 6.2 资源能力 H 与两类算力模式
总算力模式使用 D01 的同口径绝对规模;AI 训练模式使用明确工作负载的有效能力;推理模式另选固定模型、质量、上下文与时延要求。不同模式不共用没有物理依据的换算,也不比较彼此总分。
\[
C^{eff}_{c,t,w}=\sum_{j\in deployed(c,t)} C^{peak}_{j,t,p}\,u_{j,t,w}
\]
`p` 是固定精度,`w` 是参考工作负载;`u` 由可比测试、已知运行约束或经校准的估计支持。没有依据时只发布理论能力,或者明示区间;不能统一假设某国芯片只有另一个国家的固定折扣。
内存、互联、软件和电力会影响同一个有效利用系数,不得分别随意乘一串折扣重复惩罚。若已知场地供电上限,可用场地总功率除以 PUE 约束 IT 负荷,再按同配置效率估计可运行容量;已在利用系数中计入的限电不再扣一次。总吞吐与可用于单次大训练的最大连续集群另列。
国家宏观估计和公开集群样本并列校验,不直接把二者相加。若 H 只能覆盖已观测设施,分数标题和参与方排名必须明确限定为这个样本,不外推为全国真实能力。
### 6.3 创新与应用 O
完整创新指数 I 单独显示。取得 D04 分项后,建立指标归属表,逐一标注资源、扩张、韧性或创新;只把不重叠的研发质量、模型/科研成果与应用表现用于 O。不是简单执行“创新总分减去算力分”,因为不同分项的权重和标准化可能不同。
备用指标可来自 D11,但新增指标意味着新定义和新版本,不能偷偷替换原报告。论文数量须去重、处理合作分摊及引用年龄;模型结果要冻结评测版本;应用指标明确平台样本。专利、下载、融资和模型排行榜均不是彼此的等价替代。
\[
D^I_t=I_{US,t}-I_{CN,t},\qquad v^I_t=(D^I_{t-1}-D^I_t)/\Delta years
\]
2023—2025 年披露分差缩小 4.07 分,约为起点分差的 18.5%;不能解释为中国能力提高 18.5%,也不能据此推断算法效率提升。只有差值不能恢复各国绝对得分。跨年方法不一致时分段显示,不能外推追平时间。
### 6.4 扩张潜力 G
\[
G^{raw}_{c,t,h}=\sum_{j\in pipeline(c,t)}P(T_j\leq t+h\mid X_{j,\leq t})\,\Delta C^{eff}_j
\]
默认 `h=12 个月`。X 只含当时可知的项目状态、设备交付、并网、建设和资金证据。分期投产拆分未交付增量;已投产部分进入 H 后从剩余 G 扣除。公告金额不直接转成已投产能力。
初期采用明确的保守/基准/乐观情景,没有历史标签就不提供伪精确概率。标签积累后可比较阶段条件概率、校准的逻辑回归或生存分析;处理尚未到期项目的右删失、取消与部分交付,不能把所有未完成记录标成失败。
### 6.5 韧性 R
\[
R_{c,t}=100\sum_s q_s\,\min(1,C^{eff}_{c,t,s}/C^{eff}_{c,t,base})
\]
s 是对各方一致定义的冲击情景,例如特定供应类别中断或并网延迟;q 为公开、固定的情景权重,除非有概率依据,否则不称为发生概率。基准能力为零或未知时不计算此比值。
关系图用于识别依赖、替代和共同故障源。没有替代来源证据就标为未知;政策公告通常改变未来供应或可用性假设,不自动删除境内现有芯片存量。
### 6.6 标准化与发布条件
数量型正向指标可采用固定基期映射:
\[
N(x;a,b)=100\,clip\left(\frac{\ln(1+x/a)}{\ln(1+b/a)},0,1\right),\quad a,b>0
\]
a、b 与 x 单位一致,来自有覆盖说明的固定参考面板,存入模型版本。成本型、强度型和比例型指标各自定义映射。不得按每日领先国家重定标,也不能因加入一个国家导致全部历史分数被静默重写。
关键输入缺失时 `S=null`,保留可用子分,不填零或自动重分配权重。O、G 或 R 没有可比输入时综合分不上线;原始图表和证据浏览仍可上线。样本少、匿名取整或来源偏差反映在不确定性和适用范围中,不直接扣国家实力分。
不确定性计算分开保存观测精度、参数估计和情景假设。只有有依据的输入分布才运行 Monte Carlo;共享芯片规格、同一公告、同一供应链的误差必须相关采样。无法量化的覆盖偏差直接披露,不能靠窄置信区间掩盖。排名用“无法区分”处理不稳健结果,不展示过多小数制造精确感。
### 6.7 变化与归因
\[
g_{c,t}=C_{c,t}/C_{c,t-1}-1,\quad p_{c,t}=C_{c,t}/\sum_kC_{k,t},\quad b_{CN,t}=C_{CN,t}/(C_{CN,t}+C_{US,t})
\]
全球分母必须完整定义;双边份额不得标为全球份额。按报告披露值,2022—2023 年中国总算力从 302 增至 435 EFlops,约增长 44%,份额从 33% 降至 31%。这是“绝对增长、相对份额下降”,不是存量消失。算力与创新现有历史窗口不同,不能拼成同一时期的因果结论。
同权重、同版本下,分项账单严格满足:
\[
\Delta S=0.40\Delta H+0.30\Delta O+0.20\Delta G+0.10\Delta R
\]
分项内非线性与交互影响可使用固定分组的 Shapley 分解;必须保存分组、基线和近似误差,不能把任意计算顺序产生的贡献当唯一解释。数学归因只解释模型输出,不证明宏观因果。
正式发布中始终分开:现实变化、补录旧事实、来源修订、模型/基期修订。版本升级用“同一输入、不同模型”的桥接结果展示,不能把换模型造成的跳变计入国家增长。
## 7. 项目内 AI 如何参与
### 7.1 运行方式
复用控制台已经配置的模型,通过 `AIProviderClient` 调用;不要求新的模型供应商,也不把供应商名称写死在业务代码中。第一阶段采用后端固定步骤工作流,各角色是任务模板,可以由同一个模型执行,不需要首先建设复杂的多 Agent 协商系统。
在现有提示词注册表中新增以下拟议 task key;提示词版本与运维覆盖的有效内容哈希进入每次运行记录。
| 任务模板 | 输入 | 允许输出 | 关键约束 |
| --- | --- | --- | --- |
| `compute.sources.discover` | 指标缺口、许可范围、已注册来源 | 来源候选、适用指标、采集建议 | 先找原始出处;新域名不直接进入正式定时采集 |
| `compute.evidence.extract` | 原文分块、页码、表格、目标 schema | 带证据定位的字段候选 | 未出现的数值返回缺失;原文中的指令只是待分析内容 |
| `compute.evidence.review` | 候选、对照记录、单位与实体规则 | 冲突类型、支持/反驳证据、待核验项 | 第二个 AI 的同意不是独立事实证明 |
| `compute.gaps.prioritize` | 缺失、来源健康、参数敏感性、采集成本 | 下一步补证清单 | 优先可能影响结论且可查证的缺口,不以国家倾向排序 |
| `compute.change.explain` | 已计算的贡献账单、事实版本 | 有引用的可读解释 | 不能修改计算结果;事实、估计、假设分别表达 |
| `compute.model.propose` | 误差报告、数据质量问题、模型配置 | 有边界的参数/映射/新特征提案 | 必须说明可证伪假设、预期改善、影响范围和回退方案 |
| `compute.model.review` | 候选方案、独立评估结果、影响报告 | 发布建议与限制 | 无权改变测试标签、门槛、保留集或自己批准自己 |
结构化结果由后端 Pydantic schema 校验。当前 AI 返回以文本为主,需要新增结果解析与受限重试;JSON 不合格、引用不匹配、单位冲突时运行失败或保留候选,不能将自然语言当作正式数据。
### 7.2 工具与权限
允许工具包括:参数化只读指标查询、已注册来源搜索/抓取、文档段落读取、规则化单位转换、实体候选查询、固定估计器运行、创建缺口任务、保存提案、提交实验任务。
AI 不直接写正式分数、覆盖原始证据、执行任意 SQL/Python/Shell、修改发布门槛或扩大自己的工具权限。生成的采集映射必须通过回放样本后才保存;新增可执行采集器代码属于普通开发和发布流程,不在生产 Agent 内 `eval`。
网页抓取需校验协议、主机、解析后的目标地址及每次重定向,阻止访问内网、回环和元数据端点;使用专门外部抓取边界,不能把内部业务 API 和外部取证 URL 混用。AI 只得到所需公开证据或经授权的业务摘要,凭证由后端设置解析,不进入提示词、引用或日志。
### 7.3 用户能看见的 AI 结果
每份分析展示三部分:已验证事实、模型解释、仍缺哪些证据。每条数值和关键结论可展开来源。用户可以标记“实体合并错误”“单位/年份错误”“证据不支持”“假设不合理”,形成结构化反馈;收藏、点赞和是否喜欢国家排名不能成为事实标签。
“AI 发现数据不足”可以是成功结果。模型不可用时,确定性采集、计算、双图和地球继续工作,解释面板保留上次结果并标注版本与时间。
## 8. 数学模型如何自我迭代
### 8.1 自动迭代分成三类
| 层次 | 可以怎样改进 | 自动程度 |
| --- | --- | --- |
| 证据和数据 | 补充来源、修正字段、发现重复和过期、改进解析映射 | 已核准来源与已验证规则可自动运行;新语义及高影响冲突先进入候选 |
| 可验证的估计参数 | 项目延期分布、特定硬件/任务效率、记录错误率 | 通过冻结评估、影子运行和预先配置的边界后,可以自动发布小范围参数更新 |
| 评价定义 | H/O/G/R 权重、指标含义、基期、国家范围和价值取向 | 默认生成提案,由模型负责人确认版本;不是机器从不存在的“国家实力真值”中自动学习 |
此处是产品建议的默认发布策略,不是在本次计划工作中申请执行权限。未来可配置自动发布的参数白名单,但不能用“开启自动迭代”笼统授权所有定义变化。
### 8.2 迭代闭环
```mermaid
flowchart TD
A[新事实/人工纠错/成熟预测结果] --> B[确定性误差与质量评估]
B --> C[AI 提出可证伪改进]
C --> D[生成候选配置与锁定实验]
D --> E[按时间回测及国家分组评估]
E --> F{是否通过预设门槛}
F -->|否| G[记录失败原因并保留现行版本]
F -->|是| H[影子运行:结果不改变正式排名]
H --> I[按发布策略评审或受限自动提升]
I --> J[发布不可变版本与变动桥接]
J --> K[监测退化并可原子回滚]
K --> B
```
每次提案只改变一种主要因素,或者明确作为一个联合实验;不能同时换来源、标签、权重和评估方法后宣称单个参数有效。失败提案也保存,避免只展示成功实验。
提案记录至少包括:问题、证据 ID、基线版本、拟议配置差异、适用实体/任务、可检验目标、锁定数据截止时间、主要评估指标、停止条件和回滚版本。可以提出“将并网阶段纳入交付预测”,不能只提出“让某国得分更合理”。
### 8.3 真实反馈标签从哪里来
| 估计对象 | 可观察标签 | 评估方式 |
| --- | --- | --- |
| 文档字段提取 | 人工核对原文的数值、单位、年份、实体和引用位置 | 字段精确率/召回率、关键错误、引用支持率,按语言/文档类型分组 |
| 实体合并 | 官方 ID、项目连续记录或人工确认的同一/不同实体 | 错误合并率和遗漏合并率;高影响错误单列 |
| 项目按期投产 | 后续正式验收/投产/取消记录,保留观测截止 | Brier score、可靠性图;右删失与失访不能计作失败 |
| 新增容量 | 未来实际确认的设备与容量,具有同口径 | 绝对误差、对数误差;同时报告项目等权与容量加权结果 |
| 任务性能系数 | 未参与拟合的可复现实测配置 | 相同任务质量下的预测误差与区间覆盖 |
| 区间预测 | 后续实际值是否落入范围及区间宽度 | 覆盖率与区间评分一起评价,避免靠无限放宽范围“提高准确率” |
不使用 AI 自己生成的摘要作事实标签,不用现行总分拟合下一版总分,不把另一个复合指数直接当国家实力真值。完整创新指数可做外部一致性讨论,但若其包含输入指标或与 O 重叠,就不能作为独立验证集。
### 8.4 回测与数据泄漏控制
预测训练、验证和保留集按真实可获得时间滚动切分,同一项目的后续阶段、同一公告转载和同一扩建链必须归组,避免分散进训练和测试。每个特征都要求 `published_at <= forecast_cutoff`;仅今天抓到而无法证明当时可得的资料,不参加“当时可知”回测。
新模型先与简单基线比较:维持上次状态、沿用公告日期、同阶段历史交付中位数。不得只与一个刻意弱的旧模型比较。参数、标准化锚点和来源质量估计只从训练窗口取得。
大模型的训练记忆可能知道后来结果,所以历史回测中的预测器必须是只读取冻结特征的确定性估计器;不能让 LLM 凭记忆补旧事实。提案生成与真实保留集隔离,实际前瞻影子运行仍是必要证据。
按国家、硬件代际、项目规模、来源语言分别评估,并按项目/运营方/时间块进行重采样,不能把高度相关的重复记录当独立样本。小样本报告区间与不足,不为满足仪表盘而宣称显著改进。
### 8.5 建议发布门槛
以下是实施阶段的初始门槛草案,需在 M0 用试点数据冻结。它们是最低操作条件,不保证统计有效性,AI 无权自行降低。
| 门槛 | 初始建议 |
| --- | --- |
| 硬正确性 | 单位、时间穿越、重复计数、采集失败保留正式值等必测案例全部通过 |
| 提取评估 | 至少 200 条人工核验字段、覆盖中英文及主要文档类型;报告精确率/召回率及区间;关键数值/单位错误不得进入自动采用集 |
| 预测参数自动提升资格 | 至少 50 个有可验证结果的独立项目、覆盖至少 3 个滚动窗口;每个主要参与方至少 10 个,不足则继续影子运行或人工评估 |
| 改善幅度 | 预先指定主要损失相对现行版和简单基线至少改善 5%;按依赖结构重采样的损失差区间须支持改善 |
| 分组保护 | 主要参与方损失退化不超过预先约定范围,初始建议 2%;不能以平均改善掩盖一个国家明显恶化 |
| 区间质量 | 不仅检查覆盖,还检查宽度与评分;对无法量化的来源覆盖偏差单列说明 |
| 影子观察 | 至少连续 4 周稳定运行,并达到所需成熟标签数;12 个月预测不因观察满 4 周就算验证完成 |
| 试验节制 | 初始每月最多 3 个主要候选,锁定真实保留集;多次搜索造成的选择偏差要通过新的前瞻样本再次检查 |
标签暂时不足时,系统可以自动补数、解释、积累预测与结果,但不发布“自我学习后的更准模型”。这不会阻塞第一版事实产品上线。
### 8.6 一个完整迭代例子
以下是流程示例,不是已发生的项目事实或评估成绩。
系统发现一批已订购芯片的项目未如期投产;确定性评估记录预测误差。AI 查询证据后提出:现有交付模型只使用芯片到货时间,遗漏电力接入阶段。它提交一个新增“并网是否完成”特征的候选。
后端按项目原始发表时间构造训练数据,用固定估计器拟合;锁定保留集检验总体和中美分组误差。即便论文或第二个 AI 都认为这个想法合理,只要结果未通过门槛就不发布。通过后先影子运行;正式发布时展示受影响的 G 分、预计容量和模型版本变化,已投产 H 不因推测而下降。
若后续发现并网字段提取错误导致退化,回滚活动模型指针,保存新证据和失败实验。不能删除失败记录或改写过去已发布的预测。
## 9. 后端数据与实现结构
首版采用最小领域实现:沿用 `CollectedData`、`DataSnapshot` 和位置维表,新增经过 schema 校验的国家指标记录与聚合服务。数值口径、参考期、发表时间、来源链接、原文哈希和证据定位可先作为受约束的元数据保存;必要的原件存放在许可允许的运行数据目录。只有现有表确实无法表达查询或历史约束时才增加专用表。
先冻结面向专题的 API 契约,地球与图表只消费该契约。将来拆出指标观测、证据、实验等表时,通过服务层迁移,不让前端依赖临时 JSON 或物理表结构。首版不以第 9.1 节全部表建完作为开工条件。
### 9.1 拟议持久化对象
下表为待新增的逻辑对象,可按最终数据库设计合并有限的小表;不要与现有采集、位置和任务状态形成并行真相源。
| 对象 | 保存内容 | 关联和约束 |
| --- | --- | --- |
| `evidence_documents` | 原文存储引用、原始/提取哈希、来源、发表/抓取时间、许可、提取版本 | 内容不可变;全文不在高频列表接口返回 |
| `metric_definitions` | 粒度、单位、精度、工作负载、归属、方向、适用范围和版本 | 一个稳定 metric ID;模型版本引用具体定义版本 |
| `metric_observations` | 数值/范围、参考期、知识时间、状态、来源版本、取整和缺失原因 | 链接原采集记录;追加式修订,不复制整份原始 JSON |
| `observation_evidence` | 观测与证据的支持/反驳关系、页号、表格/段落、短摘录 | 多对多;来源家族避免转载重复投票 |
| `compute_entities` / `resource_relations` | 稳定实体、别名、分期关系、所有权/供应/连接、有效期 | 复用现有位置维表;不另存一套会漂移的正式坐标 |
| `model_versions` | 指标集合、权重、锚点、参数、代码版本、情景与发布状态 | 不可变版本;活动指针独立且原子切换 |
| `model_runs` / `model_results` | 输入清单哈希、版本、截止时点、随机种子、环境、国家结果、分项和不确定性 | 结果可重放;实验、情景与正式运行分开 |
| `model_proposals` / `model_evaluations` | 配置差异、假设、冻结评估方案、损失、分组结果、决定与回退目标 | 发布必须引用通过的评估,保存失败实验 |
| `agent_runs` | task key、提示词有效哈希、模型、工具调用、引用、预算、结构化输出、终态 | 链接现有任务账本,Agent 自身不拥有另一套调度状态 |
重放保留输入观测 ID/版本清单或不可变清单文件,不能只保存一段 SQL 然后查询已被更改的数据。记录计算实现版本、依赖环境和随机种子;LLM 输出保存为证据,不要求重新调用模型才能复算数值。
历史删除与保留策略需要显式区分:普通缓存可清理;被正式模型版本引用的观测及许可允许保存的证据受保留保护。来源撤回或许可变更时标记可访问状态并保留允许保留的哈希和元数据,不强行公开原文。
### 9.2 待新增模块与现有边界
| 拟议模块 | 职责 |
| --- | --- |
| `backend/app/services/compute_intelligence/observations.py` | 规范化、有效观测选择和数据缺口 |
| `.../metrics.py`、`.../scoring.py` | 纯数值计算、单位/指标注册、标准化和总分 |
| `.../forecasting.py`、`.../evaluation.py` | 固定估计器、时间切分、基线、分组指标与实验 |
| `.../workflows.py`、`.../proposals.py` | 固定 AI 工作流、结构化候选、预算和发布规则 |
| `.../publication.py` | 正式快照、变动桥接、活动版本和 outbox |
| `backend/app/api/v1/compute_intelligence.py` | 参数化只读查询和后台任务入口 |
| `frontend/public/earth/js/compute-intelligence.js` | 专题 UI 状态、图表与现有图层联动,消费后端计算结果 |
| 控制台专题页与现有采集/AI 设置页签 | 证据核验、缺口、模型实验、版本回看 |
具体文件在实现时保持单一职责,优先复用已有服务,不为目录结构创建空文件。新来源继续注册到现有采集器体系及 datasource 配置;新提示词放入现有 `default_prompts.json`,不散落在 renderer 或 `aiprovider`。
### 9.3 拟议 API
下列接口尚不存在,名称在 M0 与当前路由约定核对后冻结。
| 方法与路径(统一前缀 `/api/v1/compute-intelligence`) | 返回内容 |
| --- | --- |
| `GET /overview` | 所选参与方、口径、参考期与知识截止下的结果、适用范围及覆盖说明 |
| `GET /series` | 指定指标的有界历史序列,含缺失、版本、实际观测时间与来源引用 |
| `GET /entities/{id}` | 设施、分期、资源关系和历史观测 |
| `GET /changes` | 数据或模型贡献账单,数据库侧分页 |
| `GET /evidence/{id}` | 根据权限和许可返回证据摘要、定位与允许展示的内容 |
| `GET /models`、`GET /runs/{id}` | 模型版本、输入清单、计算结果与评估摘要 |
| `POST /recompute`、`POST /agents/run` | 建立任务并立即返回任务 ID,允许取消和查看进度 |
| `POST /proposals/{id}/evaluate` | 创建冻结数据与方案的评估任务 |
| `POST /models/{id}/promote`、`POST /models/{id}/rollback` | 按角色及发布策略原子切换活动版本,记录理由和影响 |
查询参数固定包含 `metric_profile`、`entities`、`reference_at`、`knowledge_cutoff`、`model_version` 和可选 `scenario_id`。前端不接收任意表达式或 SQL,不自行补算另一套总分。
### 9.4 发布与性能
后台完整计算并通过质量门槛后,在事务中发布结果快照和活动指针,通过既有 outbox 唤醒刷新。新 adapter 显式声明国家面板/历史图的刷新范围;设施变化才触发相应 `computeCenters` 更新。草稿证据、失败实验和影子运行不改变默认地球。
缓存键包含口径、实体集合、参考期、知识截止、数据清单哈希、模型与情景版本。WebSocket 只通知结果版本与必要增量,客户端拒绝较旧响应;不能每次国家 hover 都重算模型或执行 LLM。
聚合、过滤、排序和分页在数据库完成;预计算国家年度/月度可用序列,地图按分辨率聚合节点。时间滑动去抖并复用已取快照,播放使用相邻状态差量。建议试点验收目标为缓存查询 P95 小于 1 秒、切换已缓存时点小于 500 毫秒,声明基准机器、数据规模和网络条件;未实测前不作为已达到性能。
## 10. 刷新、成本和失败处理
| 工作 | 建议节奏 | 触发条件与成本控制 |
| --- | --- | --- |
| CSV/API 检查 | 日或来源允许的更慢频率 | ETag/Last-Modified/哈希去重,无内容变化不调用 AI |
| 年报与指数发现 | 月度,发布季可提高 | 找到新版本再下载和提取,不把月检写成月度指标 |
| 重点项目证据 | 日查、周复核 | 去重后仅高影响变化进入提取队列 |
| 指标重算 | 有效观测或模型版本改变时 | 只重算受影响实体/时点/指标依赖 |
| AI 解释 | 有意义的结果变化或用户请求 | 相同输入哈希复用,不因页面打开重复生成 |
| 数据缺口排序 | 每周 | 敏感性、覆盖风险、可获得性和预计采集成本共同决定 |
| 参数候选实验 | 月度或成熟标签达到门槛 | 限制候选数量;无足够标签不强制产出升级 |
试点默认每工作流最多 3 次搜索、8 个页面抓取、4 次 LLM 调用、1 次结构化修复重试,最多同时运行 2 个 AI 工作流;均为可配置预算,不是第三方接口能力承诺。按全局模型配置记录 token/费用,设置日预算和单任务截止时间,超额进入延后或停止状态,不无限自递归。
失败处理:采集失败保留上次正式值和新鲜度;AI 不可用仍提供确定性结果;引用失效保留许可允许的原件和哈希;长期资料缺口显示缺失并降低结论覆盖范围;新源冲突进入待核验;模型退化恢复前一活动版本。不能用重试或回滚删除已提交有效证据。
## 11. 分期实施与依赖
工作量是规划估算,按一名熟悉项目的全栈开发者、持续可用的数据核验支持和现有环境可用计算;不是交付承诺。开发时间不包括等待付费授权、原始报告或 12 个月预测结果成熟的时间。
### 11.1 当前优先交付:展示专题
| 顺序 | 修改方向 | 验收内容 | 估计开发工作日 |
| --- | --- | --- | ---: |
| A1 修复数据与最小合同 | Epoch CSV、TOP500 日期/身份/单位、空结果保护;保存历史指标和来源 | 能得到真实当前快照与核验后的年度序列,不要求宏观指标全到最新年 | 2—3 |
| A2 国家对比与历史接口 | 国家汇总、公开样本与宏观统计分离、双图数据、证据摘要 | 绝对量/份额/创新分差有各自定义;缺失与时点可见 | 2—3 |
| A3 地球与图表展示 | 专题模式、国家面板、双图、时间联动、设施/证据详情 | 完成第 0 节完整浏览路径,并通过桌面/移动/缩放验证 | 3—5 |
| A4 可选 AI 阅读辅助 | 复用现有 client 和任务提示词,针对冻结快照生成带引用说明 | 事实与推断分开;可关闭;失败不影响地球与图表 | 1—2 |
A1—A3 是当前必须交付的闭环,约 7—11 个开发工作日;A4 可在闭环稳定后增加。实际数据源不通、现有采集回归问题或主题布局复杂度会改变估算。实现时首先验证 A1,而不是先开发通用 Agent 框架或完整综合评分。
### 11.2 后续平台能力路线
下表是完整能力建设的工作包估算,与 A 路线存在复用和重叠,不应重复相加。A 路线完成后先盘点已具备的 M0—M2 能力,只实施剩余部分。
| 阶段 | 工作包 | 交付物与通过条件 | 估计开发工作日 |
| --- | --- | --- | ---: |
| M0 合同与试点 | 固定口径、成员、来源许可、历史时间语义、角色权限、质量/评估门槛 | 指标注册表、数据源接入清单、黄金核验样本;所有未知明确登记 | 3—5 |
| M1 可信数据 | 修复 Epoch/TOP500;证据与指标观测;导入两组历史;稳定身份与失败保护 | 可查询、可追溯、可重放的历史 API;无示例数据进入正式结果 | 7—10 |
| M2 可见产品 | 地球专题、双图、国家对比、变化账单、证据抽屉和控制台缺口页 | 中美历史回放闭环;移动/缩放和旧图层回归通过;综合分可保持未就绪 | 7—10 |
| M3 资源与潜力 | 芯片、实测、项目、电力/资金台账,场景与分项估计 | H 与可用 G/R 子项具备来源;O 仅在分项齐备后发布;总分门槛满足才开放 | 8—12 |
| M4 AI 工作流 | 固定提取/核验/补证/解释流程、反馈和预算;扩展可靠任务类型 | AI 只能产生有引用候选与解释,失败不会污染正式数据 | 7—10 |
| M5 模型迭代 | 真实标签、时间回测、候选评估、影子运行、发布桥接与回滚 | 能证明某个可观察估计目标改善;不要求每轮必有新模型 | 8—12 |
从零完整建设上述广义平台能力约 40—59 个开发工作日,包含比 A 路线更完整的证据和实验治理;它不是当前展示专题的工期。影子期至少四周,但不成熟的预测标签会延长自动提升资格等待期。
依赖顺序:M0 → M1 → M2;M3 的项目台账应尽早开始积累标签;M4 依赖 M1 的可信证据结构;M5 依赖 M3 的标签和 M4 的审计工作流。多方、矿产和复杂供应链在 M2 之后按数据质量逐个扩展,不等全部主题齐备再上线。
## 12. 验收与测试矩阵
| 范围 | 必测案例 | 通过标准 |
| --- | --- | --- |
| 来源真实性 | HTML 改版、空 CSV、404、部分下载、示例字符串 | 正式数据不被示例或空结果替换;错误原因可见 |
| 单位与精度 | T/P/E 换算,FP64/FP16/FP8,稀疏/稠密,MW/TWh,FLOP/FLOP/s | 不同口径拒绝合并;可比转换有可复算结果 |
| 时间与版本 | 固定抓取日误作投产日、2026 报告回填 2023、未知发布日期 | 严格截止查询无未来信息;历史回顾明确重述 |
| 实体与去重 | 排名换位、扩建、名称变化、匿名化、重复转载、跨源同集群 | 身份稳定;不重复计数;匿名或缺榜不自动退役 |
| 统计与地理 | 缺坐标、未知国家、欧盟与成员国、跨境所有权 | 国家统计不依赖坐标;分母不重叠;归属视图不混用 |
| 两张历史图 | 已核验年份值、缺失年份、2026 待核实主张 | 只展示有证据的观测;不能伪造平直历史或默认预测 |
| 综合评分 | 缺 O/G/R、权重和不为 1、标准化越界、国家集合变化 | 非法配置拒绝;缺失总分为 null;固定版本不静默漂移 |
| 变化账单 | 新投产、补录、源修订、换模型 | 分项贡献之和满足计算容差;类型和版本差异可解释 |
| AI 输出 | 格式错误、虚构引用、原文含指令、模型超时、重复任务 | 输出隔离;工具边界有效;任务幂等并能取消 |
| 模型评估 | 标签穿越、同项目跨集、过拟合小样本、分组退化 | 不通过门槛不发布;保留失败实验 |
| 发布与回滚 | 两个并发发布、计算中断、旧 WS/HTTP 响应 | 原子版本;半成品不可见;客户端不回滚到旧时点 |
| 地球渲染 | 远距/近距、时间播放、层隐藏、节点锁定、场景退出 | 无闪烁和悬空 tooltip;实例复用,不每帧重新建纹理 |
| UI | 桌面、移动、125%/150% 缩放、键盘、无数据/低置信/加载状态 | 一屏工作区无意外全局滚动;关键标签不靠颜色区分 |
实现阶段按变更执行后端精确测试、`scripts/harness/quick-check.sh`、Bun 前端构建和渲染 smoke;地球改变不能只凭构建通过。保持现有公共页面、鉴权、管理路由、安全导航及其他地球图层的回归证据。性能阈值用记录过的基准规模验证,不能把开发空数据测试称为生产验收。
## 13. 文档、发布与完成定义
每阶段 PR 和部署使用已有项目流程,功能旗标分别控制专题 UI、估计分项、AI 工作流与自动参数提升。先开放读取与事实浏览,再启用模型实验;关闭新旗标应回到原有算力图层而不丢历史数据。
发生真实用户工作流变化时更新中英文手册与快速入门;采集、AI、作业与位置变化更新相应后端技术文档;控制台、Earth、图层顺序和样式变化更新对应 context 与规范。新增公共技术文档需要同步中英文和 Docs 注册;本文件是内部中文计划,不注册为已实现的公共手册。
当前展示专题完成定义:用户完成第 0 节浏览路径,可以从国家指标变化回到出处和所用计算版本;重新计算得到同一数值;知道哪些资料尚缺。两张图应和国家、设施、时间、证据联动,不能只是贴在地球旁的静态图片。
后续决策辅助与模型迭代完成定义:情景的假设与结果可复算;AI 的一次成功改进有独立验证目标、评估记录和可回滚版本。只有一段 AI 分析或未经校验的总分,不代表完成决策辅助能力。
## 14. 风险与预先决定的退路
| 风险 | 产品和实施处理 |
| --- | --- |
| 无法取得创新原报告分项 | 保留完整指数独立观测;O 和总分不发布,不伪造拆分 |
| 中国或其他地区公开设施更少 | 显示宏观统计与设施样本两个覆盖视图,扩充本地语言原始来源,不把保密或匿名化当能力下降 |
| 硬件真实利用率不可知 | 理论量先行,明确任务和区间;不把实验室 benchmark 当生产实测 |
| 来源改版、匿名化或许可证变化 | 版本化下载、质量隔离、许可范围内保存证据,必要时停更单一来源 |
| 资金、电力、芯片相互重复计分 | 固定指标归属与约束链;不把投入、存量、产出和事件各加一次 |
| 训练标签不足或结果多年后才成熟 | 先提供情景与证据改进,延后参数自动提升资格 |
| 模型迎合既定国家排名 | 预先定义目标和门槛,评价权重单独评审,不以用户喜好或另一个 AI 打分训练 |
| 已有采集/定位状态被新服务覆盖 | 复用当前真相源,使用明确字段所有者、发布快照和回归测试 |
## 15. 来源与相关文档
以下链接支持数据口径、来源可得性和架构选择;除明确标注的历史值外,本计划的字段、频率、权重、工期和发布门槛均是拟议设计。
- [信通院 2021 年版算力白皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202109271518811781_1.pdf):2020 年总算力份额。
- [信通院 2022 年版白皮书(镜像)](https://13115299.s21i.faiusr.com/61/1/ABUIABA9GAAgvq2fmwYozLrrpwU.pdf):正文第 13—14 页,2021 年份额与口径。
- [信通院 2023 年版白皮书(镜像)](https://www.ahchanye.com/wp-content/uploads/2023/09/2023092815342684.pdf):正文第 13 页的 2022 年份额,以及 302 EFlops 总规模。
- [信通院 2024 年版蓝皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202502051642799257_1.pdf):正文第 10 页的 2023 年份额及后续规模说明。
- [科技日报关于创新指数报告的报道(新浪转载)](https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml):分差与综合指标维度;未替代原报告分项。
- [2026 年 42%/33% 候选出处](https://www.mornai.cn/news/gpu/ai-computing-power-never-sleeps/):待核实商业文章,不作为正式统计。
- [Epoch GPU 集群公开数据](https://epoch.ai/data/gpu-clusters)、[字段与下载说明](https://epoch.ai/data/gpu-clusters-documentation):CSV、地理字段、覆盖与公开版本限制。
- [Epoch 芯片所有权](https://epoch.ai/data/ai-chip-owners):所有权不等于使用权或所在地,交付与投产有时间差。
- [MLPerf Training](https://mlcommons.org/benchmarks/training/)、[MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/):固定任务、质量及测试配置。
- [IEA Energy and AI](https://www.iea.org/reports/energy-and-ai)、[EIA 开放数据](https://www.eia.gov/opendata/):能源资料入口,不能直接替代设施并网证据。
- [SEC EDGAR API 说明](https://www.sec.gov/search-filings/edgar-application-programming-interfaces):公司提交与财务数据访问;需注意财季与标签口径。
- [USGS 矿产概要](https://www.usgs.gov/centers/national-minerals-information-center/mineral-commodity-summaries)、[UN Comtrade 门户](https://comtradeplus.un.org/):矿产与贸易候选数据源。
- [OpenAlex 帮助与数据说明](https://help.openalex.org/):研究元数据候选来源;接入时核验访问与归属规则。
- [业务架构与数据流转](../technical/zh/platform-data-flows.md)、[后端采集器](../technical/zh/backend-collectors.md)、[AI Provider 边界](../technical/zh/agents-aiprovider.md)。
- [智能星球前端上下文](../technical/zh/earth-frontend-context.md)、[渲染图层顺序](../technical/zh/earth-render-layer-order.md)、[图层视觉规范](../technical/zh/earth-layer-style-reference.md)。
- [轻量 Agent 编排](agents-light-orchestrator-websearch-plan.md)、[态势感知基础计划](agents-situational-awareness-foundation-plan.md):复用总体方向,不重复建设 provider 内业务编排。

View File

@@ -0,0 +1,154 @@
# Planet 开发 Agent 模型分配与 Token 控制计划
日期:2026-09-21。状态:已用于本轮地理与空间卡片开发;未改变全局或项目默认模型配置。适用于已授权的开发任务,不代替用户对任务范围的决定。
目标:在固定政治展示规则、现有 UI 逻辑和性能门槛下,降低完成一个合格任务的总成本与返工次数。不是让所有工作都使用低价模型,也不是让最强模型从头包办。
当前重点任务见 [Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)。该计划的功能与验收要求优先;本文件只决定如何组织开发。
## 1. 模型角色
模型定位参考 OpenAI Docs;下表的具体分工是针对 Planet 的工程建议,需要结合实际一次通过率和返工记录调整,不是模型能力保证。
| 模型 | 精确模型 ID | 默认推理强度 | 分配的工作 | 不作为默认用途 |
| --- | --- | --- | --- | --- |
| Astra | `gpt-6-astra` | `high` | 跨系统难题、空间展示方案、复杂坐标/深度/性能诊断、关键设计与独立审查 | 日常文件搜索、机械编辑、反复运行已确定命令 |
| Sol | `gpt-5.6-sol` | `medium`;复杂实现用 `high` | 执行主控、核心代码、共享组件整合、Three.js 状态与生命周期、最终集成 | 逐条整理大批重复记录、重复复述整个项目背景 |
| Terra | `gpt-5.6-terra` | `medium` | 接口已明确的常规实现、局部 UI 修复、数据处理脚本、测试编写、浏览器回归 | 独自决定空间交互方向、坐标模型或跨模块架构 |
| Luna | `gpt-5.6-luna` | `low`;明确的小修改可用 `medium` | 定向检索、清单比对、日志归类、说明文档初稿、固定格式整理 | 核心渲染、数据迁移、几何归属修改、最终审美或性能放行 |
日常推荐组合是 Sol 主控加一个 Terra worker;Luna 和 Astra 按需进入,不要求每个阶段凑齐四种模型。已经完成的资源准备不因采用这份方案重新做一遍。
当前会话的主模型不会因为写入计划自动变成 Sol。真正执行时,主会话模型由实际任务设置决定,子 Agent 在派发时明确指定模型和推理强度,并核对返回的运行配置;不能把角色名称当作已经切换模型的证据。
## 2. 本项目任务分配表
| 工作包 | 主执行 | 独立验证/审查 | 交付和升级条件 |
| --- | --- | --- | --- |
| 现有窗口、连线、巡航、滚动条入口盘点 | Terra | Sol 检查引用与遗漏 | 文件/符号清单、现有行为证据;Luna 可辅助定向搜索,无需读取全仓库 |
| 贴图归档、尺寸、哈希、配准元数据检查 | 直接运行脚本;脚本需新增时 Terra | Sol 抽查实际输出 | S0 已完成,后续只做有变化的核对,不委托模型重复计算哈希 |
| 固定政治规则的技术合同与几何覆盖边界 | Sol | Astra 审查规则是否贯穿各数据路径 | 权属口径已由项目确定,任何模型均不得重新决定;标识不明确先核对,不能猜测几何 |
| 贴图、海岸线、拾取错位定位 | Sol `high` | Astra 在根因不清或涉及多个坐标层时介入 | 用明确经纬点复现,区分错误数据、回退、投影、UV 与半像素偏移 |
| 规则已明确的映射、来源清单、局部转换脚本 | Terra | Sol;关键几何结果由 Astra 抽查 | 只落实批准合同;发现新的几何语义不得自行扩大规则 |
| 多区域空间卡片的第一版运动设计 | Astra `high` | Sol 检查能否复用当前实现 | 明确锚点、投影、避让、换侧、背面及阅读稳定性;产出小合同,不产出整套 UI 大改 |
| 2 张卡片的连接、弹性移动、生命周期原型 | Sol `high` | Terra 实测;Astra 看关键动效证据 | 优先实现最小完整循环;使用参考图和实际运动证据,不用文字自评代替 |
| 扩至 4/6 卡、增删区域和事件整合 | Sol | Terra 跑交互/竞态测试 | 复用原型合同;`main.js` 等共享入口只由指定 owner 写入 |
| 共享窗口行为、标准字体、移动抽屉及深色滚动条 | Terra | Sol 看渲染与旧面板回归 | 修改公共滚动所有者,验证图层等所有相关页;涉及共享状态重构时升级给 Sol |
| 地球柔光、材质层次、节点与背景关系 | Sol | Astra 对照 UE/PPT/海报审查 | 第一轮只改少量确定参数并给实际截图;性能退化交 Sol 诊断,难题交 Astra |
| 卡片内指标、普通图表、加载/缺失/错误状态 | Terra | Sol | 数据合同先冻结;不复活固定全屏大面板,不编造缺失年份 |
| 国家/区域接口和前后端联动 | Terra;跨模块变更由 Sol | Sol | 不硬编码中美为唯一对象;范围筛选、比较对象和组合去重必须一致 |
| 性能基准工具和回归测试 | Terra | Sol 检查方法;Astra 审查疑难退化 | 固定设备、数据与相机操作;日志含帧时间和内存证据,不能只报平均 FPS |
| 单位、日期、缓存、旧请求覆盖等复杂数据故障 | Sol | Astra 仅处理跨系统未解问题 | 先给最小复现和失败测试,避免多个模型同时试不同修复 |
| 中英文手册、资源清单、计划状态同步 | Luna 初稿;技术性较强时 Terra | Sol 对照实际 diff | 已实现与待实施分开;Luna 不能自行宣告功能完成 |
| 最终集成与验收 | Sol | Astra 对关键风险做一次集中审查 | 相关测试、渲染、政治规则及性能证据齐全;普通修订不逐项再次走 Astra |
## 3. 顺序与并行安排
### 第一轮:证据与边界
Sol 读取现有计划、当前工作区差异和必要规则,整理短任务清单。Terra 盘点实际运行的数据提供者、组件和性能基线;独立且批量的文档/引用整理才交 Luna。先读已有检查结果,避免重新下载贴图、重新研究 World Monitor 或重复跑已经通过的检查。
Sol 将确定的组件接口、规则约束、实际截图与未解问题整理成一个短包,Astra 只审查高返工风险的空间方案和坐标问题。Astra 返回决定、理由和验证点,不再从零扫描整个仓库。
### 第二轮:最小空间原型与独立 UI 修复
Sol 实现 2 张卡片的锚点、连线和有阻尼移动;Terra 可以同时处理公共滚动条或写回归测试,前提是文件所有权不冲突。先完成最小可演示运动,再扩展内容和卡片数量。若两者都要改 `controls.js`、`hud.css` 等共享文件,则按顺序交接,不并行写入。
Astra 对原型的实际截图/录屏做一次聚焦审查,检查是否符合“从地球四周延伸”的方向。空间原型尚未确认时,不展开完整指标卡片和全量国家 UI,以免重演方向错误后大规模返工。
### 第三轮:内容、全球扩展与性能
合同稳定后,Terra 接入指标与状态,Sol 负责与渲染/生命周期的整合。性能测试由脚本执行,Terra 整理差异;可复现的退化先交 Sol。只有不能局部解释的 GPU/CPU、坐标、异步问题才交 Astra。
### 第四轮:一次集成验收
Sol 整合并运行所需检查;Luna 根据最终 diff 更新文档初稿。Astra 集中审查关键几何、政治规则完整性、卡片运动与性能结论。已通过范围不反复重测,除非新修改影响该范围。
## 4. 并发上限与文件所有权
当前环境总并发槽位为 4,含主 Agent;执行时仍以实际可用额度为准。默认使用主 Agent 加 1 个 worker。只有存在两项以上真正独立的工作,且主 Agent 同时有有用工作可做时,才扩大到主 Agent 加 2 个 worker;第 4 个槽位只给临时审查或独立批处理。
每项任务指定可写文件范围。`main.js`、`controls.js`、共享 HUD/InfoCard 与跨层状态接口同一时间只有一个写入 owner。其他 Agent 只读检查或提交补丁建议,由 owner 集成。不得并行启动多个浏览器重测相同场景,也不得共享有冲突的登录/本地存储测试状态。
浏览器验证优先使用当前可用的 Browser 控制能力,包括统一 CUA 入口。不能仅因技能目录中缺少单独的 Browser 条目就判定工具不存在;应继续检查已暴露工具并多次重试连接。只有多次确认 Browser 不可用后,才考虑独立 Playwright,并说明具体阻塞。
Agent 完成有边界的任务后停止。只有同一任务需要补充验证时才续用;不能让 worker 在空闲时自动寻找新功能或自行扩大范围。创建用户侧新任务不作为内部拆工方式。
## 5. 节省 Token 的具体做法
1. 派发窄任务,不默认继承整个历史对话。使用简短任务卡加源文件路径;必要历史按最小范围提供。短上下文不能省略政治规则、性能要求和相关视觉证据。
2. 主控保存一份简洁任务账本:当前合同、文件 owner、已完成证据、失败原因、下一步。各 Agent 引用同一事实,不各写一份长计划。
3. 先 `rg` 定位,再读函数和局部 diff;不重复读取巨大 JSON、整仓日志或全部历史文档。大数据留在文件里,由脚本给出数量、哈希、差异和异常样例。
4. 图片、截图和录屏用实际文件查看。不能为了省 token 把视觉任务变成只读文件名或文字概括。
5. 相同来源只研究一次,保存来源、日期和关键结论;只有数据变化、结论不足或任务扩大时重查。
6. 子任务首次任务卡建议控制在约 500–1500 token,结果摘要约 300–800 token;这是上下文组织目标,不是截断必要证据的硬上限。
7. 返回“改了什么、哪些文件、验证证据、剩余阻塞”即可。完整日志和截图放到文件,用路径引用;失败时返回相关片段,不整段倒入主控上下文。
8. 哈希、解压、下载、构建、测试和统计交给确定性工具。不要为了运行一条已知命令另开一个 Agent。
9. 不重复安排多个模型完整实现同一功能。关键风险做一次独立审查,普通修改由测试与 owner 检查。
10. 默认不开 `max` / `ultra` 推理。提高模型或推理强度只针对具体难题,并记录升级理由。
## 6. 升级与停止条件
| 情况 | 处理 |
| --- | --- |
| Luna 遇到不明确需求、需要理解跨文件状态或开始猜测行为 | 交 Terra;若涉及核心状态/几何,直接交 Sol |
| Terra 的任务超出明确合同,或同一实质失败在一次有针对性的修复后仍未解决 | 给 Sol 最小复现、尝试记录和失败证据,不继续盲试 |
| Sol 遇到跨坐标层、性能来源不明或与现有交互语义冲突的问题 | 给 Astra 聚焦诊断任务;可以由 Astra 完成必要关键补丁,不限于口头审查 |
| 已知一开始就是复杂的空间布局或高风险几何设计 | 直接用 Sol/Astra,不为了经过低模型梯级而增加一次无效尝试 |
| 网络、凭证、工具缺失、原始资料不存在 | 不通过升模型解决环境问题;检查环境或记录缺口,继续独立工作 |
| 失败来自缺失用户决定或无法识别的地名/视觉目标 | 提出具体问题,不让多个模型猜不同版本 |
默认允许 worker 一次有证据的修复;是否继续由主控根据新信息决定。超时或 token 消耗增加但没有新增证据时停止扩展,整理已知结果后重新分配,不能无限自递归。
## 7. 审查门槛
Astra 重点进入三个位置:最初高风险空间/几何合同、首次运动原型、最终关键风险验收;中间疑难问题按需增加。它不需要审阅每一处 CSS、每一段翻译或每个测试日志。
Sol 对各 worker 的交付负责。不能以“另一模型说通过”代替运行结果。任何模型都无权改变项目政治口径、降低性能门槛、把未知数据填成确定值,或自行部署和发版。
涉及政治几何和性能结论,使用与实现过程分离的检查证据;普通文档格式或固定映射使用针对性检查即可。视觉方向仍由已确认的用户要求和参考图决定,Astra 的审美判断不替代用户目标。
## 8. 单任务派发模板
```text
任务:一个可独立验收的目标
模型与推理:明确 model ID + reasoning effort
必要约束:相关 rules 模块、政治口径、性能、共享组件、视觉参考
输入:最少文件/符号、当前合同、已有验证结果和图片路径
可写范围:明确文件;共享文件由指定 owner 修改
验收:必须出现的行为、必要测试/渲染/性能证据
停止/升级:哪些条件不能继续猜测或扩展
返回:结果摘要、改动文件、检查结果、证据路径、遗留问题
```
派发有意使用较小上下文时,必须把不可省略的约束写进任务卡。执行任务的 Agent 自行按规则加载相应模块,不因主控已读过就假定子 Agent 也知道。
## 9. 如何验证分配确实更高效
每个工作包记录模型、推理强度、输入/输出 token(环境可提供时)、耗时、工具耗时、修复轮数、一次验收是否通过和最终证据。若读不到准确 token,就标记未知,不用消息长度伪造 token 统计。
比较每个已验收任务的总消耗,包含主控、worker、审查和返工,而不是仅比较某次调用。先完成 3–5 个有代表性的工作包,再调整默认路由;这个数量是试点取样建议,不代表统计显著性或保证节省比例。
API 单价、订阅额度和任务级成本不是同一个指标。本计划不预先承诺固定节省百分比;模型可用性和实际费用以执行时配置为准。低成本模型导致反复修复时应提前升级;高能力模型能明显减少整体返工时可以直接采用。
## 10. 官方依据与执行边界
OpenAI Docs 将 Astra 定位为最具能力、适合困难端到端任务的模型,Sol 面向复杂专业工作,Terra 平衡能力与成本,Luna 面向成本敏感任务;它们并未规定 Planet 各文件应由哪个模型处理。本计划按任务风险给出项目级分工。
- [OpenAI 模型说明](https://developers.openai.com/api/docs/models)
- [Codex 模型选择](https://learn.chatgpt.com/docs/models)
- [子 Agent 配置与模型选择](https://learn.chatgpt.com/docs/agent-configuration/subagents)
只在用户已授权相应开发与委派、且运行环境支持时执行这套拆工。仅制定本计划不会切换当前会话模型、修改默认配置或自动启动 Agent。执行时若某个精确模型不可用,先说明并选择可用替代,不把替代模型冒充原分配。
## 11. 本轮实际执行账本
| 工作包 | 实际分配 | 结果与升级 |
| --- | --- | --- |
| 地理政策、源证据、几何构建与回退 | Sol `high`,`geography_policy` | 版本化规则与校验已实现;如实保留未核验几何,未用模型猜测填补 |
| 空间连接线、卡片生命周期与主入口 | Sol `high`,`spatial_callouts` | 完成原型;六卡避让多次渲染仍有冲突后,交主控接手约束求解,worker 停止同文件写入 |
| 同机性能基线、对照工具、公共滚动条 | Terra `medium`,`performance_baseline` | 保存桌面/模拟手机各 5 次基线;共享滚动容器修复和对照工具已完成 |
| 复杂避让、全球目录/API、视觉参数、集成验证 | 主会话 | 主控整合并审查;性能脚本的场景同步多次失败后也由主控接手修复。独占布局文件,浏览器/构建按顺序执行,避免测试相互干扰 |
| Luna 辅助 | 未启用 | 没有值得额外传递上下文的独立机械任务,不为凑齐模型而派工 |
本轮精确 token 与费用不可得,记为未知;不报告未经测量的节省比例。空间避让存在返工,不能把功能实现或子任务数等同于效率提升。性能问题应在首次稳定复现后采集 CPU/布局证据,再修改热点;本轮最终定位到共享 SVG 长度读取造成的同步布局,单纯调整避让参数不足以解释该成本。性能、运动观感及地理缺项以功能计划的验收状态为准。

View File

@@ -1,715 +0,0 @@
# Earth 天球背景与日月位置实施方案
## 目标
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
1. 用真实天球背景替换当前随机星点
2. 在当前时间下显示太阳与月亮的相对位置
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
本方案优先追求:
- 与当前 Three.js Earth 架构兼容
- 风险可控
- 先落地一版真实感明显提升的 V1
- 为后续更严格的天文参考系升级预留余地
## 当前现状
当前 Earth 的基础条件已经具备:
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
这意味着:
- 不需要重写 Earth
- 可以在现有 scene/world 层新增一个 celestial layer
- 第一阶段不必拆 Earth / satellite / cable 的参考系
## 总体策略
采用“两层现实”设计:
### 1. 世界层(world-space celestial layer)
用于放置:
- 天球背景
- 太阳
- 月亮
- 太阳光方向
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
### 2. 地球层(earth-fixed layer)
继续保持当前结构:
- 海缆
- 登陆点
- 卫星点与轨迹
- BGP 覆盖
- 地球纹理、云层、地形
这些对象继续挂在 `earthObj` 下,不打断现有交互。
## 为什么先这样做
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
如果现在直接做严格惯性参考系改造,会同时影响:
- `earthObj.rotation`
- 卫星轨迹与锁定逻辑
- 海缆与登陆点附着关系
- resetView / autoRotate / hover / click 等交互链路
所以第一阶段只做:
- 真正的天空
- 真正的日月方向
- 不碰现有 Earth 附着对象的语义
## 推荐技术选型
### 天文计算库
推荐:
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
原因:
- 有 JavaScript 版本
- 支持 Sun / Moon 的矢量与坐标变换
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
不作为主选的库:
- [SunCalc](https://github.com/mourner/suncalc)
原因:
- 更偏本地观察者视角的太阳/月亮高度角
- 用于“地面日出日落”很好
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
### Three.js 表现层
推荐组合:
- 天球:内翻球壳 + 星图纹理
- 太阳:`THREE.Sprite`
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
- 太阳光:`THREE.DirectionalLight`
参考:
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
## 天球背景资源与星体数据来源
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
### 1. 背景资源:全天星图贴图
用于 Phase 1 的“真实天空背景”。
推荐优先来源:
- NASA SVS 的 Tycho 全天星图
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
- NASA Deep Star Maps 2020
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议要求:
- 使用官方来源或官方衍生可复用资源
- 等距矩形投影(equirectangular)
- 坐标定义尽量明确为赤道坐标展开
- 分辨率建议至少 `4k`
- 颜色不要过亮,避免压过 Earth HUD 前景
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
建议本地资源目录:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
### 2. 位置数据:星表与天体计算
用于 Phase 2+ 的“位置正确的星体”。
推荐来源分两层:
- 太阳、月亮位置
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
- 恒星位置
- 第一优先:Hipparcos / Tycho
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
- 第二优先:Gaia
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议策略:
- V1:背景球壳只用全天星图,不立即生成全量恒星点
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
这样做的原因:
- 背景球壳负责“天球真实感”
- 亮星点负责“位置正确、可后续标注和高亮”
- 不需要一开始就处理数十万甚至数百万颗星
### 3. 对外部成熟实现的参考结论
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
- 图形渲染使用 `TWGL.js`
- 天文计算使用 `Skyfield` 与 `Astronomia`
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
这给本项目的启发是:
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
- “位置正确的动态天体”则应依赖单独的天文计算链路
- 没有必要在第一版就直接渲染完整星表
因此本项目推荐继续坚持两层拆分:
- 背景层:官方深空图 / 全天星图
- 计算层:太阳、月亮与后续亮星点
## 如何保证星体位置正确
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
### 1. 统一坐标基准
本方案推荐统一使用:
- `J2000` 赤道坐标系作为恒星位置基准
原因:
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
### 2. 背景贴图与点位必须使用同一展开逻辑
如果背景球壳使用赤道坐标全天图,那么:
- 亮星点也必须按赤道坐标贴到同一球面方向
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
否则会出现:
- 背景银河带是对的
- 但太阳/月亮或亮星点飘到不匹配的位置
### 3. RA / Dec 到 Three.js 坐标的落点方式
亮星点和日月方向最终都要转成单位球面向量。
概念步骤:
1. 读取赤经 `RA`
2. 读取赤纬 `Dec`
3. 转成弧度
4. 映射到单位球面向量
5. 再根据 Three.js 当前世界坐标定义做轴向映射
参考公式:
```text
x = cos(dec) * cos(ra)
y = sin(dec)
z = cos(dec) * sin(ra)
```
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
- 验证 `RA = 0h`
- 验证 `RA = 6h`
- 验证北天极
- 验证银河带主方向
然后确定最终的:
- `x/y/z` 对应 Three.js 哪个轴
- 是否需要 `z` 取反
- 是否需要整体再做一个固定 `rotation`
建议把这层显式封装在:
```js
function equatorialToWorldVector(raRad, decRad)
```
不要把轴映射散落在不同模块里。
### 4. 背景球壳与恒星点的关系
推荐最终组合:
- 背景层:全天星图球壳
- 点位层:亮星点
- 动态层:太阳 / 月亮
这样有三个好处:
- 背景层提供密集真实的天空纹理
- 亮星点提供位置正确、可扩展的标注基础
- 太阳/月亮提供与时间相关的真实动态对象
## 数据与资源建议清单
### 推荐首批引入资源
1. 全天星图
- 来源:NASA Tycho all-sky map
- 用途:背景球壳纹理
2. 月亮纹理
- 用途:Phase 4 月相表现
- 路径建议:
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
3. 太阳 glow 贴图
- 用途:太阳 sprite halo
- 路径建议:
- `frontend/public/earth/assets/celestial/sun_glow.png`
### 推荐首批数据文件
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
- `frontend/public/earth/assets/celestial/bright-stars.json`
建议字段:
```json
[
{
"id": 32349,
"name": "Sirius",
"raDeg": 101.2875,
"decDeg": -16.7161,
"mag": -1.46,
"colorIndex": 0.00
}
]
```
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
- 只保留亮星
- 只保留渲染必需字段
- JSON 或二进制轻量格式
## 资源与数据实施路线
### 路线 A:先做可用版本(推荐)
1. 引入 NASA Tycho 全天图
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
2. 实现背景球壳
3. 用 Astronomy Engine 计算太阳/月亮方向
4. 暂不做亮星点
优点:
- 最快见效
- 风险最低
- 就能明显提升天球真实感
### 路线 B:在 A 基础上增强
1. 离线生成 `bright-stars.json`
2. 浏览器端渲染亮星点
3. 后续可加:
- 星座线
- 亮星名称
- 特定星体高亮
优点:
- 背景真实感和“位置正确的可交互星体”同时兼顾
## 代码模块建议细化
### 新增模块
- `frontend/public/earth/js/celestial.js`
- 管理天球背景
- 管理太阳/月亮
- 管理亮星层(后续)
- `frontend/public/earth/js/celestial-data.js`
- 资源路径
- 星图方向配置
- 亮星数据加载(后续)
### 建议函数设计
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date)
export function setCelestialVisibility(visible)
export function disposeCelestialLayer()
function loadStarMapTexture()
function createSkySphere(texture)
function createSunSprite()
function createMoonSprite()
function getSunEquatorialPosition(date)
function getMoonEquatorialPosition(date)
function equatorialToWorldVector(raRad, decRad)
```
### 推荐后续预处理脚本
如要引入亮星层,建议单独做离线脚本:
- `scripts/build_bright_stars.py`
职责:
- 从 Hipparcos / Tycho 源数据读取
- 过滤亮星
- 生成 `bright-stars.json`
这样浏览器端只消费轻量结果,不承担大表解析成本。
## 分阶段实施
## Phase 1:真实天球背景
### 目标
用真实全天星图替换当前随机星点背景。
### 做法
1. 新增一张全天星图纹理
建议路径:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
纹理要求:
- 等距矩形投影
- 赤经/赤纬坐标展开
- 无地平线、无地景遮挡
- 尽量深色、弱干扰,适合大屏 HUD 叠加
2. 新增天球球壳
新增模块:
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
建议接口:
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date, camera, earth)
export function disposeCelestialLayer()
```
3. 实现一个大半径内翻球体
建议参数:
- 半径:`600 ~ 900`
- 材质:`MeshBasicMaterial`
- `side: THREE.BackSide`
- 不受场景光照影响
- 始终围绕场景中心
### 验收标准
- 初始加载后背景不再是随机星点
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
- 不明显干扰海缆/卫星/BGP 的前景识别
## Phase 2:太阳与月亮真实位置
### 目标
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
### 做法
1. 在 `celestial.js` 内封装天体位置计算
建议函数:
```js
function getSunDirection(date)
function getMoonDirection(date)
```
输出统一为 world-space `THREE.Vector3`
2. 太阳显示
- 一个暖色发光 sprite
- 比月亮更大、更亮
- 可选添加柔和 halo
3. 月亮显示
- 一个较小 sprite 或 sphere
- 灰白偏冷色
- 后续 Phase 3 再做月相
4. 更新频率
不要每帧重新做完整天文计算,建议:
- 每 30 秒或 60 秒重算一次真实位置
- 渲染帧内做平滑过渡
### 验收标准
- 页面可见太阳与月亮两个对象
- 时间变化时位置会更新
- 日月不会跟随地球局部旋转而错误附着
## Phase 3:太阳驱动地球受光
### 目标
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
### 做法
1. 替换或接管当前主定向光
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
建议改为:
- 保留环境补光
- 主太阳光方向由 `sunDirection` 决定
2. 太阳光参数建议
- `DirectionalLight` 颜色偏暖白
- 强度略高于当前主光
- 保留一个弱背光作为氛围补偿,避免背面过死黑
3. 先不做物理级大气散射
第一版只要求:
- 亮面与暗面方向真实
- 云层和大气仍保持当前风格
### 验收标准
- 地球明暗面会随太阳方向改变
- 太阳 sprite 和地球亮面方向一致
- 不破坏现有海缆、卫星、BGP 的可见性
## Phase 4:月相与天文细节增强
### 目标
在日月真实位置基础上增加更强的“天文可信度”。
### 可选项
1. 月相
- 根据日月夹角计算 illuminated fraction
- 用月相纹理或 shader 表达盈亏
2. 赤道/黄道辅助线
- 可作为开发调试层,不默认显示
3. 太阳 terminator 增强
- 给地球夜面加入更自然的 night tint
- 未来可叠加城市夜光纹理
4. 天文时间入口
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
### 验收标准
- 月亮不再只是一个静态圆点
- 后续扩展行星或观测模式时无需推倒重来
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
### 目标
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
### 需要处理的问题
- 地球自转角与 UTC 的一致性
- 赤道坐标系、地固坐标系、相机交互层分离
- 卫星轨道显示与 Earth 旋转同步关系
- resetView 和 autoRotate 的语义重定
### 风险
这一步会影响:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
因此不建议与 V1 同时推进。
## 代码改造清单
## 1. 新增文件
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
职责:
- 管理天球背景、太阳、月亮
- 对外暴露 init/update/dispose
## 2. 修改 `constants.js`
文件:
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
新增:
```js
export const CELESTIAL_CONFIG = {
sphereRadius: 800,
updateIntervalMs: 60000,
sunSpriteScale: 28,
moonSpriteScale: 16,
sunLightIntensity: 1.25,
ambientIntensity: 0.28,
backLightIntensity: 0.18,
};
```
## 3. 修改 `main.js`
文件:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
主要改动:
1. `init()` 中:
- 初始化 celestial layer
2. `addLights()` 中:
- 把固定太阳光改成可更新的 celestial sun light
3. `animate()` 中:
- 每帧调 `updateCelestialLayer()`
4. `destroy()` 中:
- 清理 celestial 资源
## 4. 修改 `earth.js`
文件:
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
主要改动:
- `createStars()` 逐步退役
- 第一阶段可先保留作为 fallback
- 当真实星图加载成功后,不再显示随机星点
## 5. 新增资源
目录建议:
- `frontend/public/earth/assets/celestial/`
建议至少包含:
- `starmap_equatorial_4k.jpg`
- `sun_glow.png`
- `moon_albedo_2k.jpg`
## 数据流设计
```mermaid
flowchart TD
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
B --> C["创建天球球壳"]
B --> D["创建太阳 sprite + 主定向光"]
B --> E["创建月亮 sprite"]
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
H --> I["更新 sun sprite / moon sprite 位置"]
H --> J["更新太阳 DirectionalLight 方向"]
J --> K["地球昼夜方向变化"]
```
## 风险与注意事项
### 1. 星图投影方向容易反
这会表现为:
- 星图左右镜像
- 赤经方向颠倒
- 日月位置和背景对不上
建议:
- 先做一个开发调试模式
- 显示赤经/赤纬参考点,快速校正纹理朝向
### 2. 不要让天球跟随 Earth 旋转
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
### 3. 不要每帧做重型天文计算
真实位置更新应节流,否则会浪费 CPU。
### 4. 月亮先求“方向正确”,再求“月相精致”
月相属于第二步优化,不应阻塞 V1 上线。
## 推荐实施顺序
1. 新建 `celestial.js`
2. 用星图球壳替换随机星点
3. 接入 Astronomy Engine
4. 加太阳/月亮 sprite
5. 用太阳方向驱动主光
6. 再决定要不要做月相和更严格参考系
## 最终建议
对于当前 Planet Earth,最稳妥的方案是:
- 先做真实天球背景
- 再做真实太阳/月亮方向
- 再让太阳驱动地球受光
- 暂时不做 Earth 参考系重构
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。

View File

@@ -1,106 +0,0 @@
# Earth Predicted Orbit Plan
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
## Goal
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
- 从当前时刻开始
- 绕地球一圈
- 当前点最亮
- 向后沿轨道逐步衰减
## Current State
当前已经有:
- 卫星历史轨迹
- 锁定卫星
- 轨道高亮与相关联动
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
## Why It Is Valuable
预测轨道可以明显提升:
- 锁定卫星后的空间可读性
- 轨道类型辨识
- 演示解释力
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
## Scope
### Phase 1
- 锁定卫星时显示一整圈预测轨道
- 解锁时隐藏
- 不替代现有普通轨迹系统
### Phase 2
- 根据轨道类型调整采样率
- GEO / MEO / LEO 不同密度
- 进一步减少 fallback 轨迹的比例
## Implementation Direction
### 1. Orbit period
基于 `meanMotion` 估算轨道周期。
### 2. Predicted samples
以固定采样步长从 `now -> now + period` 推算轨迹点。
### 3. Render object lifecycle
预测轨道应是一个独立渲染对象:
- show
- update
- hide
- dispose
### 4. Visual semantics
预测轨道不应与普通尾迹混淆:
- 更稳定
- 更完整
- 透明度沿轨道衰减
- 当前点附近更亮
## Known Risks
### 1. TLE propagation gaps
部分卫星可能出现 SGP4 计算不足,需要 fallback。
### 2. Multiple orbit lines
必须确保:
- 锁定切换前先清旧轨道
- 页面隐藏/销毁时清理
### 3. Performance
GEO 轨道点数高,采样率需要按轨道类型分层。
## Acceptance
1. 锁定单颗卫星时只显示一条预测轨道
2. 解锁后轨道立即清除
3. 不同轨道类型下点数可控
4. 页面切换回来不会闪出旧轨道残留
## Satellite Footprint Follow-Up Items
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog:
1. 为 `iridium-next` 新建独立 footprint adapter。
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint。

Some files were not shown because too many files have changed in this diff Show More