Compare commits

...

14 Commits

Author SHA1 Message Date
rayd1o
a2025dbfe0 release: bump version to 0.75.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-25 02:03:53 +08:00
rayd1o
003b316d42 release: bump version to 0.75.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-25 00:16:30 +08:00
rayd1o
0c40764c58 feat: configure compute comparisons and restore adaptive globe dragging
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-24 22:59:55 +08:00
rayd1o
5bfc790dec release: bump version to 0.75.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-23 21:20:12 +08:00
linkong
60eefb19c9 release: bump version to 0.74.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-17 17:30:54 +08:00
rayd1o
83a10a6c34 release: bump version to 0.74.6
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-16 21:05:40 +08:00
rayd1o
cee1996809 release: bump version to 0.74.5
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-13 14:04:34 +08:00
rayd1o
58671e7bc3 release: bump version to 0.74.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-13 10:27:00 +08:00
rayd1o
a54fcdbeed release: bump version to 0.74.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-09-13 02:17:55 +08:00
rayd1o
1dd2921674 release: bump version to 0.74.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-07-01 23:40:00 +08:00
linkong
d30f7d08c5 release: bump version to 0.74.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-30 18:54:34 +08:00
linkong
5bdb55f3f1 release: bump version to 0.74.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-30 13:52:52 +08:00
linkong
fbecf30513 release: bump version to 0.73.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-29 17:04:05 +08:00
linkong
19d5ac0fee release: bump version to 0.72.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-29 14:05:06 +08:00
365 changed files with 33404 additions and 4519 deletions

View File

@@ -1,139 +0,0 @@
---
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
---
# /cleanup — 垃圾代码审查与清理
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
## 检查范围
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
## 节省上下文规则
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
```bash
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
## 审查清单
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
### 1. 重复逻辑 (Duplicate Logic)
- 完全相同或高度相似的代码块在多处出现
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
### 2. Magic Numbers / Magic Strings
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
### 3. 命名问题
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
- 命名与实际用途不符
- 同一概念在不同地方用不同名字表达
### 4. 死代码 / 无效代码
- 注释掉的旧代码块(3行以上)
- 声明后从未使用的变量/参数/导入
- 永远不会执行的条件分支
### 5. 代码风格问题
- 尾部空白字符(trailing whitespace)
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
- 空行使用不一致(连续多个空行等)
### 6. 其他常见问题
- 私有辅助函数应被 export 但没有,导致调用方重复实现
- 类型/接口重复定义
- 过于冗长的条件表达式可以简化(不改逻辑)
## 执行步骤
### Step 1 — 获取待检查文件列表
```bash
# 无参数时:获取所有未提交修改
git diff HEAD --name-only
# 有参数时:用 $ARGUMENTS 过滤
```
### Step 2 — 逐文件阅读并分析
先从 focused diff 开始:
```bash
git diff --unified=0 HEAD -- <file>
```
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
### Step 3 — 报告问题清单
在修改前,先以列表形式输出所有发现的问题:
```
发现 N 个问题:
[文件] js/foo.js
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
[文件] js/bar.js
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
...
```
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
### Step 4 — 执行修复
对每个问题,使用 Edit 工具进行**最小化修改**:
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
- **命名问题**:重命名变量,更新所有使用处
- **死代码**:直接删除
- **尾部空白/风格**:修正
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
**修复原则:**
- 只改在审查清单中发现的问题,不做额外优化
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
- 改完后用 `grep` 验证旧的坏代码已消失
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
### Step 5 — 输出总结
```
清理完成:
修复了 N 个问题:
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
...
未修改的问题(需人工确认):
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
```
## 约束
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
- **禁止**添加新功能、新抽象、新参数
- **禁止**修改注释内容(只删除注释掉的死代码)
- **禁止**修改测试文件逻辑
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"

View File

@@ -1,104 +0,0 @@
---
description: Create or update repository documentation from current code changes
argument-hint: Optional: topic to document, or leave empty to infer from git diff
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
---
# /docs — Documentation Workflow
## Goal
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
## Repository Rules
Before deciding scope, check whether the repository has a documentation rules file:
```bash
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
### Step 1 — Understand The Change
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
rg --files docs
```
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
### Step 2 — Decide Scope
- Prefer updating an existing relevant document over creating a duplicate.
- Use one document for one coherent topic.
- Split documents only when the change crosses meaningful domains.
- Keep filenames lowercase and hyphenated.
- Apply the repository-specific rules file before writing.
#### Document Audience Routing (Planet)
In this repository, classify the action's performer before picking a target file:
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
### Step 3 — Write
Explain:
- Background/problem: what was wrong or missing before.
- Core design decisions and rationale.
- Operational or user-facing impact.
- Relevant code paths, only when useful for future maintainers.
Style:
- Follow the repository’s existing language and heading conventions.
- Use fenced code blocks with language tags.
- Prefer tables for comparisons or parameter lists.
- Keep snippets concise and relevant.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
### Step 4 — Verify
- Read the completed docs once for clarity and stale statements.
- Verify referenced paths exist with `test -e` or `rg --files`.
- Run applicable checks from `docs/documentation-coverage-rules.md`.
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
### Step 5 — Report
Summarize changed docs and verification:
```md
Updated:
- path/to/doc.md — what changed
Verified:
- checks that passed
- checks that could not be run, if any
```
## Hard Constraints
- Do not leave placeholder docs.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
- Keep docs maintainable and concise.

View File

@@ -1,93 +0,0 @@
---
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
argument-hint: 建议填写任务目标;若同时给出成功标准更好
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
---
# /goal-driven — 目标驱动执行模式
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
适用场景:
- 长周期实现任务
- 高复杂度工程任务
- 可被明确验收的研究、实现、迁移、验证类工作
不适用场景:
- 纯脑暴
- 无法定义成功标准的模糊任务
- 很小的一次性修改
## 输入要求
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
启动时先输出:
```md
Goal
- ...
Criteria for success
- ...
Plan
1. ...
2. ...
3. ...
Verification
- ...
```
## 执行规则
1. 先把任务固化为两个核心块:
- `Goal`
- `Criteria for success`
2. 成功标准必须尽量客观,可验证,可落地。
优先写成:
- 需要交付什么
- 需要通过哪些测试或验证
- 如何判断结果真的完成
3. 进入持续执行循环:
- 完成一个阶段
- 检查当前结果是否满足成功标准
- 若未满足,明确剩余差距并继续推进
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
5. 如果验证失败:
- 明确指出哪条成功标准没满足
- 继续工作,不要把阶段性进展误判为完成
6. 只有在以下情况之一才能停止:
- 成功标准已满足
- 用户明确要求停止
## 执行风格
- 重证据,轻口头判断
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
- 重验收,轻自我感觉
- 优先用测试、日志、产物、对比结果来证明完成
- 对长期任务保持“未达标就继续”的节奏
## 简版模板
```md
Goal: [[[[[在此填写最终目标]]]]]
Criteria for success: [[[[[在此填写成功标准]]]]]
循环执行:
1. 推进任务
2. 检查是否满足成功标准
3. 若未满足,继续工作
4. 直到满足标准或用户明确停止
```

View File

@@ -1,160 +0,0 @@
---
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
---
# /release — Planet 发版工作流
## 版本号规则
| 变更类型 | 版本跳动 | 适用场景 |
|---------|---------|---------|
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
## 必须同步更新的文件
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
- `VERSION`
- `frontend/package.json`(`"version"` 字段)
- `pyproject.toml`(`version =` 字段)
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## 节省上下文规则
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
```bash
git status --short
git diff --stat HEAD
git diff --name-only HEAD
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
```
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
## 执行步骤
### Step 1 — 环境检查
```bash
git branch --show-current # 确认在 dev 分支
git status --short # 检查是否有无关的未暂存修改
cat VERSION # 读取当前版本
```
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
### Step 2 — 确定发版类型与新版本号
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
- **先输出发版计划供用户确认**:
```
发版计划:
类型:bugfix
版本:0.26.2 → 0.26.3
分支:dev
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
### Step 3 — 更新版本号文件
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
1. `VERSION` — 直接替换全部内容为新版本号
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
### Step 4 — 更新 CHANGELOG.md
在文件顶部插入新条目,格式:
```markdown
## [x.x.x] — YYYY-MM-DD
### ✨ Features / 🐛 Fixes / 🔧 Improvements
- ...(只列高信号条目,最多 5 条)
- ...
---
```
日期使用 `date +%Y-%m-%d` 获取今天的日期。
### Step 5 — 更新 docs/version-history.md
- 更新文件头部的"当前开发版本"字段
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
### Step 6 — 验证
针对本次变更范围做最小验证:
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
```bash
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
### Step 7 — 提交前预览
展示将要提交的文件列表:
```bash
git diff --stat HEAD
```
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
### Step 8 — Commit & Push(用户确认后)
```bash
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# 若有代码变更也一并 stage
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
commit message 固定格式:`release: bump version to x.x.x`
### Step 9 — 完成确认
输出摘要:
```
✓ 版本号已更新:0.26.2 → 0.26.3
✓ CHANGELOG 已更新
✓ version-history 已更新
✓ uv.lock 已重新生成
✓ 验证通过
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ 已 push 到 origin/dev
```
## 注意事项
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 772 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

View File

@@ -5,6 +5,15 @@
!VERSION
!backend/
!backend/**
!config/
!config/earth-boundary-pov-policy.china-v1.json
!scripts/
!scripts/build_earth_boundary_china_pov_geojson.py
!scripts/build_earth_boundary_pmtiles.py
!scripts/earth_boundary_policy.py
!resources/
!resources/earth/
!resources/earth/boundary-pov-geometry.china-v1.json
!aiprovider/
!aiprovider/**

5
.gitignore vendored
View File

@@ -160,3 +160,8 @@ data/earth-boundary-sources/
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
# committing thousands of generated tile files.
frontend/public/earth/data/boundaries/
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
resources/earth/candidates/nasa-blue-marble/*.jpg
resources/earth/candidates/natural-earth-3/*.jpg
*:Zone.Identifier

148
AGENTS.md
View File

@@ -1,28 +1,41 @@
# Planet Agent Entry Point
# AGENTS.md
This is the compatibility entry point for coding agents. The older root
`agents.md` file remains authoritative for repository-specific agent behavior;
do not delete or replace it.
**Planet agent harness. Defines behavior for coding agents working in this repository.**
## Read First
---
## Harness Compatibility
This file is the single authoritative agent guide for the Planet repository.
The older lowercase `agents.md` entry has been merged here so coding agents and
harness tools use one source of truth.
### Source Of Truth
- `rules.md` is the mandatory repository rule source. Always load `core`,
`security`, and `workflow`; load only task-relevant modules after that.
- `AGENTS.md` defines the local agent operating mode and evidence gates.
- `project_context.md` is background, not a rule source. Prefer newer
implementation docs when it disagrees with current code.
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
goal-driven work, and release.
- Do not duplicate long workflow text across harness files. Durable constraints
belong in `rules.md`; task procedures belong in skills or scripts.
Read these files before changing code:
1. `rules.md` - mandatory repository rules. Always load `core`, `security`, and
`workflow`; load `docs`, `frontend`, `backend`, `earth`, `ai`, or `release`
when the task touches those areas.
2. `agents.md` - existing agent role, communication, and workflow guidance.
3. `project_context.md` - static project background. Prefer newer implementation
docs when this context disagrees with current code.
4. `README.md` - current architecture, startup, and toolchain summary.
5. `docs/HARNESS.md` - harness workflow, conflict policy, and validation tiers.
6. `CODEMAP.md` - codebase entry points, ownership boundaries, and deeper docs.
1. `rules.md`
2. `AGENTS.md`
3. `project_context.md`
4. `README.md`
5. `docs/HARNESS.md`
6. `CODEMAP.md`
For documentation work, also read `docs/documentation-coverage-rules.md`.
## Start Safely
### Start Safely
Before editing:
Before broad edits:
```bash
git status --short
@@ -40,7 +53,7 @@ git diff --unified=0 HEAD -- <path>
Preserve user changes already present in the worktree.
## Validation
### Validation
Fast local harness validation:
@@ -54,28 +67,117 @@ Full local validation:
scripts/harness/validate.sh
```
`validate.sh` includes the quick check and the frontend Bun build. Docker image
smoke builds are intentionally opt-in:
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
unless disabled by its documented environment flags. Docker image smoke builds
are intentionally opt-in:
```bash
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
## High-Risk Areas
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
current non-interactive environment first. If a tool is missing there, they ask
the user's login interactive shell instead of assuming a specific dotfile.
### High-Risk Areas
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
`destroy` cleanup.
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
rendered smoke evidence for public pages, auth guards, authenticated admin
route/section availability, safe navigation/search/tab interactions, mobile
layout, and 125% / 150% zoom, not only a build.
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
intermediate wrappers keep `min-height: 0`, and only the intended child owns
scrolling.
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
workflows in the backend.
- Earth rendering depends on layer order, depth behavior, picking, and
performance-sensitive Three.js code.
- Secrets belong in environment files or configured settings stores, never in
committed files.
- Backend service code must use structured logging instead of `print()` or
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
## Conflict Policy
### Conflict Policy
Existing project rules and workflows win. If new harness guidance conflicts with
`rules.md`, `agents.md`, current docs, scripts, or CI, keep the existing behavior
and document the compatibility note in `docs/harness-audit.md` or
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
behavior and document the compatibility note in `docs/harness-audit.md` or
`docs/HARNESS.md`.
---
## Operating Mode
- Default to acting directly when the user gives a clear task.
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
and Astra for complex reasoning; the main agent owns final verification.
- Ask before acting only when the missing decision is risky, cannot be
discovered from repository context, and no conservative assumption is safe.
- Read relevant files before editing.
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
focused file reads, tests, builds, linters, and harness scripts.
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
release work, and documentation unless the task requires it.
---
## Evidence Gates
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
mock, browser capture, or visual reference, obtain evidence from the artifact
before interpreting intent or editing code.
- Path resolution is part of the task. If the path cannot be opened, first try
reasonable local equivalents such as WSL/Windows path conversion,
workspace-relative lookup, absolute paths, and attached-file locations.
- Never guess from prompt text, filenames, previous context, logs, OCR, or
memory when a visual artifact was provided but cannot be accessed.
- OCR is acceptable evidence for text-only visual questions or non-multimodal
environments; state that OCR was used as the fallback. Layout, color, spacing,
pixel, and rendering issues need real visual inspection or a clear limitation
note.
- If a visual artifact still cannot be inspected, say so and pause that
visual-dependent part of the work.
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
diff, direct file check, or harness result.
- For UI and rendering changes, verify the rendered result when local tooling
allows it.
---
## Communication
- Match the user's language. Use Chinese for Chinese requests unless the user
asks otherwise.
- Keep updates short and specific: what is being inspected, edited, or verified.
- Final responses should summarize changed files and verification, with blockers
stated plainly.
- Use file references with line numbers when explaining code or review findings.
---
## Quality Bar
- Prefer existing project patterns over new abstractions.
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
a stable path exists.
- Centralize prompts, constants, defaults, and shared request/response handling.
- Do not add secrets, generated runtime output, or local environment files.
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
- Run the smallest relevant verification for the changed scope and report
anything skipped.
---
## Prohibited
- Do not skip visual evidence handling when a visual artifact was provided.
- Do not preserve obsolete harness files just because they already exist.
- Do not invent behavior not present in code, docs, or verified external
sources.
- Do not rewrite unrelated files during cleanup.
- Do not mark a task complete without checking concrete success criteria.

View File

@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
## Runtime Entry Points
@@ -41,11 +42,18 @@ are the source of detail for specific subsystems.
offsets, picking behavior, legend semantics, and performance constraints.
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
wrapping it from harness scripts instead of duplicating its internals.
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
interactive command path without hardcoding `.zshrc`.
## Validation Commands
```bash
scripts/harness/doctor.sh
scripts/harness/security-check.sh
scripts/harness/backend-rules-check.sh
scripts/harness/frontend-rules-check.sh
scripts/harness/docs-consistency-check.sh
scripts/harness/quick-check.sh
scripts/harness/validate.sh
./planet.sh health
@@ -60,8 +68,17 @@ uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py t
cd frontend
bun install --frozen-lockfile
bun run build
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
```
The frontend smoke covers public routes, unauthenticated admin guards,
login-error handling, the Earth iframe entry, and authenticated `super_admin`
admin route/section rendering with mocked API data. Authenticated admin checks
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
check for accidental global horizontal overflow. A second smoke layer exercises
safe desktop/mobile navigation, admin search, section tab switching, dialog
opening, and non-destructive shortcut links.
Optional delivery smoke, when Docker and Helm are available:
```bash
@@ -84,9 +101,9 @@ PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
## Known Sharp Edges
- `project_context.md` includes older roadmap-era assumptions such as Celery,
Kafka, TimescaleDB, MinIO, and UE5 being part of the active local stack. Treat
it as background unless current README/docs/code confirm the same behavior.
- `project_context.md` is static background for agents. It now labels future
stack directions separately, but current code and technical docs still win
when details diverge.
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
active local development shape.
- Local `destroy` is intentionally destructive for Planet-owned Docker and build

View File

@@ -83,7 +83,7 @@
| 组件 | 用途 |
|------|------|
| React 18 | UI 框架 |
| Ant Design Pro | 管理后台组件 |
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
| Axios | HTTP 客户端 |
| Socket.io-client | WebSocket 客户端 |
| ECharts | 统计图表 |
@@ -168,10 +168,12 @@
## 快速启动
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
```bash
# 新机器或空项目首次初始化
./planet.sh init
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
@@ -322,7 +324,7 @@ ipconfig
## 启动容错参数
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
可通过环境变量临时调整:

View File

@@ -1 +1 @@
0.71.1
0.75.2

248
agents.md
View File

@@ -1,248 +0,0 @@
# agents.md
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
---
## Harness Compatibility
Common agent tools should start at `AGENTS.md`. This file remains the existing
behavior guide and must not be replaced by harness docs. For safe repository
orientation, use:
- `rules.md` for mandatory project rules
- `project_context.md` for static background
- `docs/HARNESS.md` for validation tiers and conflict policy
- `CODEMAP.md` for subsystem entry points and ownership boundaries
- `docs/harness-audit.md` for the latest harness compatibility notes
Existing project rules and workflows stay authoritative when they conflict with
new harness guidance.
---
## Identity
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
- Python FastAPI backend
- React Admin dashboard
- Unreal Engine 5 3D visualization
- Multi-source data collection
- Polarized 3D large display (4K, 120Hz)
---
## Communication Style
### Tone
- **Professional but concise**
- Technical accuracy with clarity
- No unnecessary verbosity
- Use code comments sparingly (explain **why**, not **what**)
### When Responding
1. **Answer directly** - 1-3 sentences for simple questions
2. **Use code blocks** for all code snippets
3. **Include file:line_number** references when discussing code
4. **Never** start with "I am an AI assistant" or similar phrases
5. **Never** add unnecessary preambles/postambles
### Examples
**Good:**
```
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
```
**Bad:**
```
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
```
---
## Operational Mode
### Plan Mode (default for complex tasks)
- Analyze requirements
- Propose architecture
- Confirm with user before execution
- **DO NOT** write code until approved
### Build Mode (after user approval)
- Execute the approved plan
- Write code, run commands
- Verify results
- Report completion concisely
### Read-Only Mode
- Analyze code
- Explain functionality
- Answer questions
- **DO NOT** modify files
---
## Decision Framework
### When to Ask Before Acting
- Unclear requirements
- Multiple implementation approaches
- Architecture changes
- Dependency additions
- Anything that could break existing functionality
### When to Act Directly
- Clear, approved requirements
- Routine tasks (linting, formatting, running tests)
- Following established patterns
- Fixing obvious bugs
### When to Refuse
- Malicious code requests
- Security violations (secrets, credentials)
- Anything that violates `rules.md`
---
## Working Principles
### 1. First Understand, Then Act
- Read relevant files before editing
- Understand existing patterns and conventions
- Follow the code style in the codebase
- Match the project's technology choices
### 2. Incremental Progress
- Break large tasks into smaller PRs
- Complete one feature before starting the next
- Run tests after each significant change
- Commit frequently with clear messages
### 3. Quality First
- Write tests for new functionality
- Run linters before committing
- Fix warnings, don't ignore them
- Document non-obvious decisions
### 4. Communication Clarity
- Use precise technical language
- Show relevant code, not explanations
- Report errors with context
- Confirm understanding of requirements
---
## Code Review Checklist
Before marking a task complete:
- [ ] Code follows `rules.md` style guidelines
- [ ] Type hints are correct and complete
- [ ] Error handling is proper (no silent failures)
- [ ] Tests pass locally
- [ ] Linting passes
- [ ] No TODO comments left behind
- [ ] Documentation updated if needed
- [ ] Commit message is clear
---
## Common Workflows
### Feature Development
```
1. Understand requirements
2. Check existing patterns in codebase
3. Design solution (brief mental model)
4. Write code following rules.md
5. Write/run tests
6. Lint and format
7. Commit with clear message
8. Report completion
```
### Bug Fix
```
1. Reproduce the bug (write failing test)
2. Locate the source
3. Fix the issue
4. Verify test passes
5. Check for regressions
6. Commit fix
```
### Refactoring
```
1. Understand current behavior
2. Design target state
3. Make incremental changes
4. Preserve tests
5. Verify functionality
6. Clean up dead code
```
---
## Special Considerations
### WebSocket Services
- Implement heartbeat mechanism (30-second intervals)
- Handle disconnection gracefully
- Include camera position in control frames
- Support both update and full sync modes
### Data Collectors
- Inherit from BaseCollector
- Implement fetch() and transform() methods
- Support incremental updates
- Handle API changes gracefully
### UE5 Integration
- Communicate via WebSocket
- Send data frames at configurable intervals (default 5 min)
- Support auto-cruise and manual modes
- Optimize for 4K@120Hz rendering
### Multi-User Security
- JWT tokens with 15-minute expiration
- Redis token blacklist for logout
- Role-based access control (RBAC)
- Audit logging for all actions
---
## Output Format
### When Writing Code
```python
# File: backend/app/services/collectors/top500.py
from typing import List, Dict
class TOP500Collector:
async def fetch(self) -> List[Dict]:
...
```
### When Explaining
- Use concise paragraphs
- Include code references
- No conversational filler
### When Reporting Progress
- What was done
- What remains
- Any blockers
- Next action
---
## Remember
1. **Rules are hard constraints** - follow `rules.md` absolutely
2. **Context provides understanding** - use `project_context.md` for background
3. **Role defines behavior** - follow `agents.md` for how to work
4. **Quality over speed** - Enterprise systems require precision
5. **Communicate clearly** - Precision in, precision out

View File

@@ -1,4 +1,4 @@
# syntax=docker/dockerfile:1.7
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
FROM ${UV_IMAGE} AS uv
FROM ${PYTHON_IMAGE}
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
COPY --from=uv /uv /uvx /bin/
WORKDIR /app
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
uv sync --frozen --only-group aiprovider
COPY aiprovider /app/aiprovider
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
EXPOSE 8010
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]

View File

@@ -4,7 +4,7 @@
完整使用说明见:
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
当前支持:
@@ -15,6 +15,7 @@
- `AI_PROVIDER=ollama`
- request adapter:
- `AI_PROVIDER_API=openai-completions`
- `AI_PROVIDER_API=openai-responses`
- `AI_PROVIDER_API=anthropic-messages`
- `AI_PROVIDER_API=ollama-generate`

View File

@@ -3,6 +3,7 @@ from __future__ import annotations
import asyncio
import json
from typing import Any
from uuid import NAMESPACE_URL, uuid4, uuid5
import httpx
from fastapi import HTTPException, status
@@ -65,6 +66,7 @@ class ProviderService:
self.model_provider_apis = self._parse_model_provider_apis(
overrides.get("model_provider_apis")
)
self.session_id = str(uuid4())
def get_status(self) -> AIProviderStatusResponse:
enabled = self.provider != "disabled"
@@ -95,6 +97,8 @@ class ProviderService:
)
prompt = self._build_prompt(payload)
if payload.context.get("session_id") is not None:
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
provider_api = self._resolve_model_provider_api(model)
@@ -102,6 +106,10 @@ class ProviderService:
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
content = self._extract_openai_content(data)
content_blocks = self._extract_openai_blocks(data)
elif provider_api == "openai-responses":
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
content_blocks = self._extract_responses_blocks(data)
content = "".join(block.text for block in content_blocks if block.text)
elif provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(
model,
@@ -200,6 +208,39 @@ class ProviderService:
request_body=request_body,
)
async def _request_openai_responses(
self, model: str, prompt: str, system_prompt: str | None = None,
) -> dict[str, Any]:
request_body: dict[str, Any] = {
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
}
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["instructions"] = resolved_system_prompt
return await self._post(
path="/responses",
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
request_body=request_body,
)
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
blocks: list[AIContentBlock] = []
for item in payload.get("output") or []:
if not isinstance(item, dict):
continue
if item.get("type") == "message":
for part in item.get("content") or []:
if not isinstance(part, dict):
continue
text = part.get("text") or part.get("refusal")
if isinstance(text, str) and text:
blocks.append(AIContentBlock(type="text", text=text))
elif item.get("type") == "reasoning":
for part in item.get("summary") or []:
if isinstance(part, dict) and isinstance(part.get("text"), str):
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
return blocks
async def _request_anthropic_messages(
self,
model: str,
@@ -226,7 +267,7 @@ class ProviderService:
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["system"] = resolved_system_prompt
resolved_thinking = self._resolve_anthropic_thinking(thinking)
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
if resolved_thinking:
request_body["thinking"] = resolved_thinking
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
@@ -243,8 +284,12 @@ class ProviderService:
request_body=request_body,
)
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
def _resolve_anthropic_thinking(
self, thinking: dict[str, Any] | None, model: str,
) -> dict[str, Any] | None:
if thinking:
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
return {"type": "adaptive"}
return thinking
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
@@ -295,6 +340,9 @@ class ProviderService:
headers: dict[str, str],
request_body: dict[str, Any],
) -> dict[str, Any]:
headers = {"User-Agent": "Planet/1.0", **headers}
if self.provider == "opencode-go":
headers["x-opencode-session"] = self.session_id
last_error: Exception | None = None
for attempt in range(1, self.http_retry_attempts + 1):
try:

View File

@@ -28,6 +28,11 @@ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
COPY VERSION /app/VERSION
EXPOSE 8000

View File

@@ -12,6 +12,7 @@ from app.api.v1 import (
alerts,
settings,
collected_data,
compute_intelligence,
data_products,
layers,
visualization,
@@ -43,6 +44,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(

View File

@@ -0,0 +1,231 @@
"""Compute observations, administrator configuration and independent facility samples."""
from datetime import UTC, datetime
from typing import Literal
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.core.earth_geography_policy import display_country
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.user import User
from app.schemas.compute_configuration import (
ComputeConfiguration,
ComputeConfigurationReset,
ComputeConfigurationWrite,
)
from app.services.compute_catalog import (
PARTICIPANTS,
participant_members,
source_country_names,
source_country_participants,
)
from app.services.compute_center_locations import refresh_compute_center_location_cache
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
configuration_response,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
router = APIRouter()
SOURCES = ("top500", "epoch_ai_gpu")
def selected_countries(countries: str) -> list[str]:
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
members = participant_members(codes)
flattened = [country for participant in members.values() for country in participant]
if len(flattened) != len(set(flattened)):
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
return codes
@router.get("/series")
async def get_series(
response: Response,
countries: str = "CN,US",
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
envelope = await read_configuration(db)
response.headers["Cache-Control"] = "no-store"
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
result.update(
version=envelope["revision"],
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
)
return result
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response,
_actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
return configuration_response(await read_configuration(db))
@router.put("/configuration")
async def update_configuration(
payload: ComputeConfigurationWrite,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=payload.config, actor=actor,
)
return configuration_response(envelope)
@router.post("/configuration/reset")
async def reset_configuration(
payload: ComputeConfigurationReset,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
)
return configuration_response(envelope)
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
now = datetime.now(UTC)
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
ranked = (
select(
DataSnapshot.id,
func.row_number()
.over(
partition_by=DataSnapshot.source,
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
)
.label("rank"),
)
.where(
DataSnapshot.source.in_(SOURCES),
DataSnapshot.status == "success",
DataSnapshot.completed_at < cutoff,
)
.subquery()
)
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
# Old collector rows have ambiguous units/identity and must not become
# national evidence merely because a new UI can display them.
query = select(CollectedData).where(
CollectedData.snapshot_id.in_(snapshots),
CollectedData.is_valid == 1,
CollectedData.extra_data["compute_contract"].as_string() == "v1",
or_(
CollectedData.source == "top500",
and_(
CollectedData.extra_data["status"].as_string() == "Existing",
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
),
),
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
)
return query
@router.get("/facilities")
async def get_facilities(
countries: str = "CN,US",
year: int | None = Query(None, ge=2000, le=2100),
limit: int = Query(1000, ge=1, le=2000),
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
query = await sample_query(db, codes, year)
country = CollectedData.extra_data["country"].as_string()
source_participants = source_country_participants(codes)
# Canonical source subdivisions retain their raw label on each record but
# share one owner in the comparison, using the same policy as the globe.
owner_cases = [
(country == name, display_country(name))
for name in source_country_names(codes)
if display_country(name) != name
]
display = case(*owner_cases, else_=country) if owner_cases else country
participant_cases = [(country == name, code) for name, code in source_participants.items()]
participant = case(*participant_cases, else_=None)
counts = (
(
await db.execute(
query.with_only_columns(
display.label("country"),
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(display, participant, CollectedData.source)
)
)
.mappings()
.all()
)
participant_counts = (
(
await db.execute(
query.with_only_columns(
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(participant, CollectedData.source)
)
)
.mappings()
.all()
)
records = (
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
.scalars()
.all()
)
if year:
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
else:
await refresh_compute_center_location_cache(db)
payload = convert_compute_centers_to_geojson(records)
for feature in payload["features"]:
feature["properties"]["historical_snapshot"] = year is not None
properties = feature["properties"]
properties["source_country"] = properties.get("country")
properties["country"] = display_country(properties.get("country"))
total = sum(row["count"] for row in counts)
payload.update(
{
"counts": [dict(row) for row in counts],
"participant_counts": [dict(row) for row in participant_counts],
"participants": [PARTICIPANTS[code] for code in codes],
"total_count": total,
"returned_count": len(records),
"truncated": total > limit,
"reference_year": year,
"scope": "public_source_records",
"history_basis": (
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
),
"coverage": "available" if total else "no_verified_snapshot",
"cross_source_deduplicated": False,
}
)
return payload

View File

@@ -3,9 +3,9 @@ from datetime import UTC, datetime
import os
from pathlib import Path
from typing import Optional
from urllib.parse import urlsplit
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
import httpx
from pydantic import BaseModel, EmailStr, Field
from dotenv import dotenv_values
from sqlalchemy import select
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
list_fallback_llm_provider_presets,
refresh_llm_provider_preset,
)
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
DEFAULT_SETTINGS = {
"system": {
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
merged = deepcopy(DEFAULT_SETTINGS[category])
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
defaults = get_fallback_llm_provider_preset(
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
)
else:
defaults = DEFAULT_SETTINGS[category]
merged = deepcopy(defaults)
if payload:
merged.update(payload)
return merged
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
merged = merge_with_defaults(category, payload)
record = await get_setting_record(db, category)
if record is None:
record = SystemSetting(category=category, payload=payload)
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
await db.commit()
await db.refresh(record)
return merge_with_defaults(category, record.payload)
return merged
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
)
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
model_provider_apis = {
**(provider_config.get("model_provider_apis") or {}),
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
}
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
model_provider_apis = {}
return {
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": _resolve_service_token(normalized_ai)[0],
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
"api_key": api_key,
"max_tokens": int(provider_config.get("max_tokens") or 1200),
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
"model_provider_apis": provider_config.get("model_provider_apis") or {},
"preset_models": _get_provider_preset(default_provider).get("models") or [],
"model_provider_apis": model_provider_apis,
},
}
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
}
def _join_provider_url(base_url: str, path: str) -> str:
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
def _extract_model_ids(payload: dict) -> list[str]:
data = payload.get("data") if isinstance(payload, dict) else None
if isinstance(data, list):
return [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
]
models = payload.get("models") if isinstance(payload, dict) else None
if isinstance(models, list):
return [
str(item.get("name") or item.get("model") or item.get("id") or item)
for item in models
if item
]
return []
def _contains_model(model_ids: list[str], model: str) -> bool:
normalized_model = model.strip().lower()
return any(str(item).strip().lower() == normalized_model for item in model_ids)
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
provider = _normalize_provider_id(llm_config.get("provider") or "")
configured_api = (
str(llm_config.get("provider_api") or "").strip()
or ProviderApi.OPENAI_COMPLETIONS.value
)
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
model = str(llm_config.get("model") or "").strip()
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
base_url = str(llm_config.get("base_url") or "").strip()
api_key = str(llm_config.get("api_key") or "").strip()
provider_api = configured_api
model_provider_apis = llm_config.get("model_provider_apis")
if isinstance(model_provider_apis, dict):
provider_api = str(model_provider_apis.get(model) or provider_api)
preset_models = [
str(item)
for item in (llm_config.get("preset_models") or [])
if str(item).strip()
]
if not provider or not base_url or not model:
return {
"success": False,
"connected": False,
"message": "当前 provider/base_url/model 未完整配置。",
"mode": "lightweight_config",
}
result = {
"success": False, "connected": False, "mode": "lightweight_models",
"provider": provider, "model": model,
}
if not base_url or not model:
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
return {
"success": False,
"connected": False,
"message": "当前 provider 未配置 API Key。",
"mode": "lightweight_config",
}
if provider == "opencode-go":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
url = _join_provider_url(base_url, "/api/tags")
headers: dict[str, str] = {}
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
url = _join_provider_url(base_url, "/models")
headers = {
"x-api-key": api_key,
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
}
else:
return {
"success": False,
"connected": False,
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
"mode": "lightweight_unsupported",
}
return {**result, "message": "当前 provider 未配置 API Key。"}
try:
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
response = await client.get(url, headers=headers)
response.raise_for_status()
payload = response.json()
except httpx.HTTPStatusError as exc:
detail = exc.response.text or exc.response.reason_phrase
if exc.response.status_code == 404 and _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
"mode": "lightweight_preset",
"provider": provider,
"provider_api": provider_api,
"model": model,
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
"mode": "lightweight_models",
"url": url,
}
catalog = await fetch_model_catalog(
provider, base_url, provider_api, api_key,
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
)
except Exception as exc:
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: {exc}",
"mode": "lightweight_models",
"url": url,
}
model_ids = _extract_model_ids(payload)
if model_ids and not _contains_model(model_ids, model):
if _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
"mode": "lightweight_models_with_preset_alias",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"连接可用,但模型目录中没有当前模型: {model}",
"mode": "lightweight_models",
"provider": provider,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {**result, "message": catalog_error_message(exc)}
result.update({"url": catalog.url, "models_count": len(catalog.models)})
if model.casefold() not in {item.casefold() for item in catalog.models}:
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
model_apis = llm_config.get("model_provider_apis") or {}
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过",
"mode": "lightweight_models",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
**result, "success": True, "connected": True,
"provider_api": model_apis.get(model) or provider_api,
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
}
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.success",
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
else "settings.ai_provider.connect.failed"),
message="AI provider connection test completed",
category="ai",
service="api",
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
"provider": payload.provider,
"model": payload.model,
"configured": True,
"lightweight_status": lightweight_result.get("status"),
"connected": lightweight_result["connected"],
},
)
return {
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
@router.get("/integrations/ai-provider/presets")
async def get_ai_provider_presets(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return {"data": list_fallback_llm_provider_presets()}
presets = list_fallback_llm_provider_presets()
saved = await get_setting_payloads(
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
)
return {
"data": [
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
for preset in presets
]
}
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
payload: AIProviderIntegrationUpdate | None = None,
):
try:
provider_id = _normalize_provider_id(provider)
api_key = None
if provider_id == "opencode-go":
current_payload = await get_setting_payload(db, "external_integrations")
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
get_fallback_llm_provider_preset(provider_id)
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
current_payload = await get_setting_payload(db, "external_integrations")
if payload is not None:
if _normalize_provider_id(payload.provider) != provider_id:
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
ai_payload = _build_ai_provider_payload(current_payload, payload)
else:
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
raise HTTPException(
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
)
try:
refreshed = await refresh_llm_provider_preset(
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
provider_api=provider_config.get("provider_api"),
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
)
except Exception as exc:
fallback = get_fallback_llm_provider_preset(provider)
fallback["refresh_error"] = str(exc)
return {"data": fallback}
logger.warning(
"LLM provider catalog refresh failed",
extra={
"event": "settings.ai_provider.catalog.failed",
"context": {
"provider": provider_id,
"error_type": type(exc).__name__,
},
},
)
raise HTTPException(
status_code=502,
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
) from exc
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
return {"data": refreshed}
@router.put("/integrations")

View File

@@ -7,6 +7,7 @@ from fastapi.responses import Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_db
from app.services.tv_catalog import get_tv_catalog_page
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
router = APIRouter()
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
@router.get("/streams")
async def list_public_tv_streams(
offset: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
q: str = Query("", max_length=200),
selected_id: str | None = Query(None, max_length=200),
db: AsyncSession = Depends(get_db),
):
return await get_public_tv_payload(db)
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
@router.get("/proxy")

View File

@@ -644,7 +644,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
return "unknown"
unit = str(capacity_unit or "").strip().lower()
if unit in {"pflop/s", "pflops", "pflop"}:
if unit in {"flop/s", "flops"}:
normalized_tflops = capacity_value / 1e12
elif unit in {"pflop/s", "pflops", "pflop"}:
normalized_tflops = capacity_value * 1000
elif unit in {"gflop/s", "gflops", "gflop"}:
normalized_tflops = capacity_value
@@ -672,9 +674,13 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
"""
features: List[Dict[str, Any]] = []
unresolved: List[Dict[str, Any]] = []
withheld_count = 0
for record in records:
metadata = record.extra_data or {}
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
withheld_count += 1
continue
result = resolve_compute_center_location_full(record, metadata)
site_type = (
"supercomputer"
@@ -706,7 +712,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
latitude = location.latitude
longitude = location.longitude
if site_type == "supercomputer":
if metadata.get("compute_contract") == "v1":
capacity_value = _parse_float(get_record_field(record, "value"))
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
elif site_type == "supercomputer":
capacity_value = _parse_float(get_record_field(record, "rmax"))
capacity_unit = "GFlops"
else:
@@ -767,7 +776,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
}
)
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
VESSEL_TYPE_FILTERS = {
@@ -1121,6 +1130,7 @@ async def build_vessel_snapshot_response(
safe_limit = _safe_vessel_limit(limit)
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
snapshot_started_at = datetime.now(UTC)
features, diagnostics = await _load_raw_vessel_snapshot_features(
db,
bbox=bbox,
@@ -1137,6 +1147,7 @@ async def build_vessel_snapshot_response(
"features": features,
"count": len(features),
"stats": _build_vessel_stats(features),
"generated_at": to_iso8601_utc(snapshot_started_at),
"diagnostics": {
**diagnostics,
"filtered_count": len(features),

View File

@@ -161,7 +161,7 @@ async def websocket_endpoint(
if is_anonymous:
channels = [channel for channel in channels if channel in supported_channels]
vessel_subscription = None
if "vessels" in channels and "bbox" in payload_data:
if "vessels" in channels:
try:
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
except ValueError as exc:

View File

@@ -0,0 +1,57 @@
"""Country display ownership for Earth comparisons and cards.
Keep source-country evidence separate; use this projection only for display
grouping so regional source records cannot become competing national keys.
"""
from __future__ import annotations
import json
from functools import lru_cache
from pathlib import Path
from app.core.countries import COUNTRY_ENTRIES, normalize_country
_DISPLAY_ISO_A3 = {
canonical: next(
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
None,
)
for canonical, aliases in COUNTRY_ENTRIES
}
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
@lru_cache(maxsize=1)
def _policy_country_by_code() -> dict[str, str]:
try:
content = _POLICY_PATH.read_text(encoding="utf-8")
except FileNotFoundError as error:
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
policy = json.loads(content)
if policy.get("profile") != "china-pov-v1":
raise ValueError("unsupported Earth display-policy profile")
mappings = {}
for rule in policy["rules"]:
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
if target not in _COUNTRY_BY_ISO_A3:
continue
for code in rule.get("sourceIsoA3", []):
mappings[code] = _COUNTRY_BY_ISO_A3[target]
return mappings
def display_country(value: str | None) -> str | None:
country = normalize_country(value)
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
owner = _policy_country_by_code().get(code)
if owner:
return owner
return country
def display_iso_a3(value: str | None) -> str | None:
country = display_country(value)
return _DISPLAY_ISO_A3.get(country) if country else None

View File

@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
class ProviderApi(StrEnum):
ANTHROPIC_MESSAGES = "anthropic-messages"
OPENAI_COMPLETIONS = "openai-completions"
OPENAI_RESPONSES = "openai-responses"
OLLAMA_GENERATE = "ollama-generate"

View File

@@ -6,7 +6,7 @@ import redis
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jose import JWTError, jwt
from sqlalchemy import text
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
return None
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
try:
parsed_id = int(user_id)
except (TypeError, ValueError):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
result = await db.execute(select(User).where(User.id == parsed_id))
user = result.scalar_one_or_none()
if user is None or not user.is_active:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
return user
async def get_current_user(
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
def blacklist_token(token: str) -> None:

View File

@@ -4,11 +4,14 @@ import asyncio
from datetime import UTC, datetime
from typing import Dict, Any
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
EARTH_UPDATES_CHANNEL = "earth_updates"
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
logger = get_logger(__name__, service="websocket")
class DataBroadcaster:
@@ -114,16 +117,22 @@ class DataBroadcaster:
return
pending = self._pending_vessel_updates
self._pending_vessel_updates = {}
vessels = []
for item in pending.values():
vessel = dict(item)
source = vessel.pop("_source", None)
action = vessel.pop("_action", "upsert")
created = vessel.pop("_created", None)
vessel["source"] = source
vessel["action"] = action
vessel["created"] = created
vessels.append(vessel)
try:
vessels = await self._load_current_vessel_updates(list(pending))
except Exception:
# Preserve newer updates that arrived during the failed database read.
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
raise
if not vessels:
return
vessels = [
{
**vessel,
"action": vessel.get("action", "upsert"),
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
}
for vessel in vessels
]
await manager.broadcast_vessels(
{
"action": "upsert",
@@ -133,12 +142,31 @@ class DataBroadcaster:
}
)
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
from app.db.session import async_session_factory
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
mmsis = [int(key) for key in keys if key.isdigit()]
vessels = []
async with async_session_factory() as db:
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
vessels.extend(await get_current_vessels_by_mmsi(
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
))
present = {int(vessel["mmsi"]) for vessel in vessels}
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
return vessels
async def broadcast_vessels_periodically(self):
while self.running:
try:
await self.flush_vessel_updates()
except Exception:
pass
except Exception as exc:
logger.exception_event(
"Failed to flush vessel updates",
event="vessels.broadcast.failed",
context={"error": str(exc)},
)
await asyncio.sleep(self._vessel_flush_interval)
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):

View File

@@ -63,6 +63,8 @@ class ConnectionManager:
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
if channel == "vessels":
self.vessel_subscriptions.pop(websocket, None)
subscribers = self.channel_subscriptions.get(channel)
if subscribers is not None:
subscribers.discard(websocket)
@@ -88,7 +90,8 @@ class ConnectionManager:
return subscription
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
bbox = config.get("bbox")
global_scope = config.get("scope") == "global"
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
try:
@@ -103,7 +106,7 @@ class ConnectionManager:
raise ValueError("bbox longitude values must be between -180 and 180")
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
raise ValueError("bbox latitude values must be between -90 and 90")
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
zoom = int(config.get("zoom") or 1)
@@ -116,10 +119,11 @@ class ConnectionManager:
if str(item).strip()
}
return {
"scope": "global" if global_scope else "viewport",
"bbox": (lon_min, lat_min, lon_max, lat_max),
"zoom": zoom,
"limit": limit,
"type": vessel_types,
"type": sorted(vessel_types),
"last_sent_at": None,
}
@@ -152,7 +156,9 @@ class ConnectionManager:
vessel
for vessel in vessels
if self._vessel_matches_subscription(vessel, subscription)
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
]
if subscription.get("scope") != "global":
matched = matched[:subscription["limit"]]
if not matched:
continue
subscription["last_sent_at"] = datetime.now(UTC)
@@ -162,16 +168,24 @@ class ConnectionManager:
"timestamp": subscription["last_sent_at"].isoformat(),
"payload": {
**data,
"vessels": matched,
"vessels": [],
"subscription": {
"bbox": list(subscription["bbox"]),
"zoom": subscription["zoom"],
"limit": subscription["limit"],
"scope": subscription.get("scope", "viewport"),
},
},
}
try:
await connection.send_json(message)
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
await connection.send_json({
**message,
"payload": {
**message["payload"],
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
},
})
except Exception:
self.unsubscribe_all(connection)
@@ -180,6 +194,8 @@ class ConnectionManager:
vessel: dict[str, Any],
subscription: dict[str, Any],
) -> bool:
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
return True
try:
lon = float(vessel.get("lon"))
lat = float(vessel.get("lat"))

View File

@@ -0,0 +1,94 @@
"""Validated, source-backed observations for the compute comparison cards."""
from datetime import date
from typing import Annotated, Literal, Self
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
Text = Annotated[str, Field(min_length=1, max_length=500)]
class ConfigurationModel(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
class ComputeSource(ConfigurationModel):
id: Identifier
title: Text
url: HttpUrl
publisher: Text
report_year: Year
published_at: date | None = None
locator: Text
@field_validator("url")
@classmethod
def public_reference(cls, value: HttpUrl) -> HttpUrl:
if value.username or value.password or len(str(value)) > 2000:
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
return value
class ComputeObservation(ConfigurationModel):
year: Year
metric: Literal["global_share", "absolute"]
source_id: Identifier
values: dict[str, Value] = Field(min_length=1, max_length=250)
@model_validator(mode="after")
def comparable_values(self) -> Self:
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
raise ValueError(
"Use countries from the canonical compute catalog, not regional totals"
)
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
raise ValueError("Global shares for one year must total at most 100 percent")
return self
class InnovationObservation(ConfigurationModel):
year: Year
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
source_id: Identifier
class ComputeConfiguration(ConfigurationModel):
reviewed_at: date
sources: list[ComputeSource] = Field(max_length=200)
observations: list[ComputeObservation] = Field(max_length=202)
innovation: list[InnovationObservation] = Field(max_length=101)
@model_validator(mode="after")
def consistent_evidence(self) -> Self:
ids = [source.id for source in self.sources]
if len(ids) != len(set(ids)):
raise ValueError("Report source IDs must be unique")
keys = [(row.metric, row.year) for row in self.observations]
if len(keys) != len(set(keys)):
raise ValueError("Each metric/year may have only one observation")
years = [row.year for row in self.innovation]
if len(years) != len(set(years)):
raise ValueError("Innovation observation years must be unique")
sources = {source.id: source for source in self.sources}
for row in [*self.observations, *self.innovation]:
source = sources.get(row.source_id)
if source is None:
raise ValueError(f"Missing report source: {row.source_id}")
if row.year > source.report_year:
raise ValueError("Observation year must not be after its report year")
if self.reviewed_at > date.today():
raise ValueError("Review date must not be in the future")
return self
class ComputeConfigurationWrite(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)
config: ComputeConfiguration
class ComputeConfigurationReset(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)

View File

@@ -36,6 +36,9 @@ class UserInDB(UserBase):
class UserResponse(UserBase):
# Bootstrap accounts have reserved .local email addresses. Validate email
# on writes, while allowing existing local identities to be serialized.
email: str
id: int
role: str
gatekeeper_groups: list[str] = Field(default_factory=list)

View File

@@ -31,6 +31,7 @@ class BaseCollector(ABC):
frequency_hours: int = 4
data_type: str = "generic"
fail_on_empty: bool = False
preserve_snapshot_on_failure: bool = False
def __init__(self):
self._current_task = None
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
if self.fail_on_empty and not data:
raise RuntimeError(f"Collector {self.name} produced no valid records")
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
raise
except Exception as e:
await db.rollback()
if snapshot_id is not None and self.preserve_snapshot_on_failure:
await self._rollback_incomplete_run(
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
)
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.phase_message = str(e)
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.status = SnapshotStatus.FAILED.value
if self.preserve_snapshot_on_failure:
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {"error": str(e)}
await db.commit()

View File

@@ -0,0 +1,82 @@
"""Quality gates shared by the two compute sources."""
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import normalize_country
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.models.data_snapshot import DataSnapshot
from app.services.collectors.base import BaseCollector
class ComputeCollector(BaseCollector):
fail_on_empty = True
preserve_snapshot_on_failure = True
async def _create_snapshot(
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
) -> int:
previous_count = await db.scalar(
select(DataSnapshot.record_count)
.where(
DataSnapshot.source == self.name,
DataSnapshot.is_current.is_(True),
DataSnapshot.status == "success",
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
if previous_count and len(data) < previous_count * 0.6:
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
return await super()._create_snapshot(db, task_id, data, started_at)
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
if self.name == "top500":
await self._copy_verified_locations(db, data)
return await super()._save_data(db, data, **kwargs)
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
"""Keep historical location keys; copy only unambiguous exact identities.
Never migrate by rank: the machine occupying rank 1 changes over time.
"""
names = [item["name"] for item in data]
locations = (
(
await db.execute(
select(ComputeCenterLocationRecord).where(
ComputeCenterLocationRecord.source == self.name,
ComputeCenterLocationRecord.name.in_(names),
)
)
)
.scalars()
.all()
)
existing = {location.source_id for location in locations}
for item in data:
if item["source_id"] in existing:
continue
country = normalize_country(item["country"])
if not country:
continue
matches = [
location
for location in locations
if location.name == item["name"]
and normalize_country(location.country) == country
and (not location.site or location.site == item.get("metadata", {}).get("site"))
and not location.source_id.startswith("top500_system_")
]
if len(matches) != 1:
continue
old = matches[0]
values = {
column.name: getattr(old, column.name)
for column in old.__table__.columns
if column.name not in {"id", "source_id", "created_at", "updated_at"}
}
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
existing.add(item["source_id"])

View File

@@ -1,119 +1,135 @@
"""Epoch AI GPU Clusters Collector
Collects data from Epoch AI GPU clusters tracking.
https://epoch.ai/data/gpu-clusters
"""
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
import csv
import hashlib
import io
import json
import math
import re
from typing import Dict, Any, List
from datetime import UTC, datetime
from bs4 import BeautifulSoup
from urllib.parse import urlparse
import httpx
from app.services.collectors.base import BaseCollector
from app.core.countries import normalize_country
from app.services.collectors.compute_base import ComputeCollector
from app.services.epoch_location import assess_epoch_location
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
class EpochAIGPUCollector(BaseCollector):
class EpochAIGPUCollector(ComputeCollector):
name = "epoch_ai_gpu"
priority = "P0"
module = "L1"
frequency_hours = 6
data_type = "gpu_cluster"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch Epoch AI GPU clusters data from webpage"""
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
async def fetch(self) -> list[dict]:
configured = self._resolved_url or EPOCH_CSV_URL
parsed = urlparse(configured)
url = (
EPOCH_CSV_URL
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
else configured
)
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.text)
return self.parse_response(response.text)
def parse_response(self, html: str) -> List[Dict[str, Any]]:
"""Parse Epoch AI webpage to extract GPU cluster data"""
data = []
soup = BeautifulSoup(html, "html.parser")
def parse_response(self, text: str) -> list[dict]:
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
if not required.issubset(reader.fieldnames or []):
raise ValueError(
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
)
entries = []
seen = set()
anonymous_counts = {}
for row in reader:
name = (row.get("Name") or "").strip()
country = normalize_country(row.get("Country"))
if not name:
raise ValueError("Epoch row has missing identity")
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
anonymous = "anonym" in name.lower()
if anonymous:
# Public anonymous rows have no facility identity. Preserve
# multiplicity within snapshots, without guessing names.
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
if source_id in seen:
raise ValueError("Epoch identity collision; previous snapshot retained")
seen.add(source_id)
raw_date = row.get("First Operational Date", "")
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
value = None
raw_value = row.get("16-bit OP/s (log)", "")
if raw_value:
exponent = float(raw_value)
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
raise ValueError("Epoch performance exponent out of bounds")
value = 10**exponent
anonymous = anonymous or (country == "中国" and not row.get("Location"))
location = row.get("Location")
location_assessment = assess_epoch_location(
location,
country=country,
raw_country=row.get("Country"),
)
entries.append(
{
"source_id": source_id,
"name": name,
"country": country,
"value": value,
"unit": "FLOP/s",
"reference_date": reference_date,
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
"metadata": {
"compute_contract": "v1",
"precision": "FP16",
"measurement": "theoretical",
"source_url": EPOCH_CSV_URL,
"raw_reference_date": raw_date,
"status": row.get("Status"),
"certainty": row.get("Certainty"),
"operator": row.get("Owner"),
"gpu_type": row.get("Chip type (primary)"),
"gpu_count": row.get("Chip quantity (primary)"),
"location": location,
"location_precision": location_assessment.precision,
"location_precision_reason": location_assessment.reason,
"raw_country": row.get("Country"),
"raw_performance": raw_value,
"performance_encoding": "log10",
"builds_upon": row.get("Builds Upon"),
"superseded_by": row.get("Superseded by"),
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
"include_in_standard_analysis": row.get(
"Include in Standard Analysis", ""
).lower()
== "true",
"rounded": country == "中国",
"anonymous": anonymous,
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
"country_unresolved": country is None,
"sources": [
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
],
},
}
)
return entries
# Try to find data table on the page
tables = soup.find_all("table")
for table in tables:
rows = table.find_all("tr")
for row in rows[1:]: # Skip header
cells = row.find_all(["td", "th"])
if len(cells) >= 5:
try:
cluster_name = cells[0].get_text(strip=True)
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
continue
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
country, city = self._parse_location(location_cell)
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
entry = {
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
"name": cluster_name,
"country": country,
"city": city,
"latitude": "",
"longitude": "",
"value": self._parse_performance(perf_cell),
"unit": "TFlop/s",
"metadata": {
"raw_data": perf_cell,
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}
data.append(entry)
except (ValueError, IndexError, AttributeError):
continue
# If no table found, return sample data
if not data:
data = self._get_sample_data()
return data
def _parse_location(self, location: str) -> tuple:
"""Parse location string into country and city"""
if not location:
return "", ""
if "," in location:
parts = location.rsplit(",", 1)
city = parts[0].strip()
country = parts[1].strip() if len(parts) > 1 else ""
return country, city
return location, ""
def _parse_performance(self, perf: str) -> str:
"""Parse performance string to extract value"""
if not perf:
return "0"
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
if match:
return match.group(1).replace(",", "")
match = re.search(r"([\d,.]+)", perf)
if match:
return match.group(1).replace(",", "")
return "0"
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "epoch_sample_1",
"name": "Sample GPU Cluster",
"country": "United States",
"city": "San Francisco, CA",
"latitude": "",
"longitude": "",
"value": "1000",
"unit": "TFlop/s",
"metadata": {
"note": "Sample data - Epoch AI page structure may vary",
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
},
]
@staticmethod
def coordinate(value: str | None, bound: int) -> float | None:
if not value:
return None
number = float(value)
if not math.isfinite(number) or abs(number) > bound:
raise ValueError("Epoch coordinate out of bounds")
return number

View File

@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
async def fetch(self) -> list[dict[str, Any]]:
request_url = (self._resolved_url or "").strip()
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
"reference_date": datetime.now(UTC).isoformat(),
}
)
if len(normalized) >= max_sources:
if max_sources > 0 and len(normalized) >= max_sources:
break
return normalized

View File

@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
"""
import asyncio
import os
from typing import Dict, Any, List
from datetime import UTC, datetime
import os
from typing import Any, Dict, List
from urllib.parse import urlencode
import httpx
from urllib.parse import urlencode
from app.core.logging import get_logger
from app.services.collectors.base import HTTPCollector
# PeeringDB API key - read from environment variable
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
logger = get_logger(__name__, service="collector")
class PeeringDBIXPCollector(HTTPCollector):
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
if response.status_code == 429:
# Rate limited - wait and retry with exponential backoff
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
if response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
if response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:

View File

@@ -1,17 +1,21 @@
"""Space-Track TLE Collector
"""Space-Track TLE Collector.
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
API documentation: https://www.space-track.org/documentation
"""
import json
from typing import Dict, Any, List
import httpx
from typing import Any, Dict, List
from urllib.parse import urlparse
from app.services.collectors.base import BaseCollector
import httpx
from app.core.data_sources import get_data_sources_config
from app.core.logging import get_logger
from app.core.satellite_tle import build_tle_lines_from_elements
from app.services.collectors.base import BaseCollector
logger = get_logger(__name__, service="collector")
class SpaceTrackTLECollector(BaseCollector):
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
password = settings.SPACETRACK_PASSWORD
if not username or not password:
print("SPACETRACK: No credentials configured, using sample data")
logger.warning_event(
"Space-Track credentials are not configured; using sample data",
event="collector.spacetrack.credentials_missing",
)
return self._get_sample_data()
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
logger.info_event(
"Space-Track TLE fetch started",
event="collector.spacetrack.fetch.start",
)
try:
async with httpx.AsyncClient(
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
"password": password,
},
)
print(f"SPACETRACK: Login response status: {login_response.status_code}")
print(f"SPACETRACK: Login response URL: {login_response.url}")
logger.info_event(
"Space-Track login response received",
event="collector.spacetrack.login.response",
context={"status_code": login_response.status_code},
)
if login_response.status_code == 403:
print("SPACETRACK: Trying alternate login method...")
logger.warning_event(
"Space-Track login returned forbidden; trying alternate method",
event="collector.spacetrack.login.forbidden",
)
async with httpx.AsyncClient(
timeout=120.0,
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
) as alt_client:
await alt_client.get(f"{self.site_root}/")
form_data = {
"username": username,
"password": password,
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
}
alt_login = await alt_client.post(
self.login_url,
data={
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
"password": password,
},
)
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
logger.info_event(
"Space-Track alternate login response received",
event="collector.spacetrack.alt_login.response",
context={"status_code": alt_login.status_code},
)
if alt_login.status_code == 200:
tle_response = await alt_client.get(self.probe_url)
if tle_response.status_code == 200:
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records via alt method")
logger.info_event(
"Space-Track alternate query completed",
event="collector.spacetrack.alt_query.completed",
context={"record_count": len(data)},
)
return data
if login_response.status_code != 200:
print(f"SPACETRACK: Login failed, using sample data")
logger.warning_event(
"Space-Track login failed; using sample data",
event="collector.spacetrack.login.failed",
context={"status_code": login_response.status_code},
)
return self._get_sample_data()
tle_response = await client.get(self.probe_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:
print(f"SPACETRACK: Query failed, using sample data")
return self._get_sample_data()
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records")
return data
except Exception as e:
print(f"SPACETRACK: Error - {e}, using sample data")
return self._get_sample_data()
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
try:
async with httpx.AsyncClient(
timeout=120.0,
follow_redirects=True,
headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "application/json, text/html, */*",
"Accept-Language": "en-US,en;q=0.9",
},
) as client:
# First, visit the main page to get any cookies
await client.get(f"{self.site_root}/")
# Login to get session cookie
login_response = await client.post(
self.login_url,
data={
"identity": username,
"password": password,
},
logger.info_event(
"Space-Track TLE query response received",
event="collector.spacetrack.query.response",
context={"status_code": tle_response.status_code},
)
print(f"SPACETRACK: Login response status: {login_response.status_code}")
print(f"SPACETRACK: Login response URL: {login_response.url}")
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
if login_response.status_code != 200:
print(f"SPACETRACK: Login failed, using sample data")
return self._get_sample_data()
# Query for TLE data (get first 1000 satellites)
tle_response = await client.get(self.query_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:
print(f"SPACETRACK: Query failed, using sample data")
logger.warning_event(
"Space-Track TLE query failed; using sample data",
event="collector.spacetrack.query.failed",
context={"status_code": tle_response.status_code},
)
return self._get_sample_data()
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records")
logger.info_event(
"Space-Track TLE fetch completed",
event="collector.spacetrack.fetch.completed",
context={"record_count": len(data)},
)
return data
except Exception as e:
print(f"SPACETRACK: Error - {e}, using sample data")
logger.warning_event(
"Space-Track TLE fetch failed; using sample data",
event="collector.spacetrack.fetch.failed",
context={"error": str(e)},
)
return self._get_sample_data()
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:

View File

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
"""
import asyncio
import math
import re
from urllib.parse import urljoin
from typing import Dict, Any, List
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
from app.services.collectors.compute_base import ComputeCollector
class TOP500Collector(BaseCollector):
class TOP500Collector(ComputeCollector):
name = "top500"
priority = "P0"
module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
entries = self.parse_response(response.text)
entries = self.parse_response(response.text, str(response.url))
semaphore = asyncio.Semaphore(8)
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
detail_url = ""
if link and link.get("href"):
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
detail_url = urljoin(str(top500_base_url), link.get("href"))
manufacturer = ""
if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
return detail_map
def parse_response(self, html: str) -> List[Dict[str, Any]]:
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
"""Parse TOP500 HTML response"""
data = []
soup = BeautifulSoup(html, "html.parser")
canonical = soup.find("link", rel="canonical")
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
# Find the table with TOP500 data
table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
if table:
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
if not reference_date:
raise ValueError("TOP500 list date missing; previous snapshot retained")
rows = table.select("tr")
for row in rows[1:]: # Skip header row
cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
cores = cells[2].get_text(strip=True).replace(",", "")
rmax_text = cells[3].get_text(strip=True)
rmax = self._parse_performance(rmax_text)
rmax = self._parse_performance(rmax_text, headers[3])
rpeak_text = cells[4].get_text(strip=True)
rpeak = self._parse_performance(rpeak_text)
rpeak = self._parse_performance(rpeak_text, headers[4])
power = ""
if len(cells) >= 6:
power = cells[5].get_text(strip=True).replace(",", "")
identity = re.search(r"/system/(\d+)", detail_url)
if not identity:
raise ValueError("TOP500 stable system ID missing")
entry = {
"source_id": f"top500_{rank}",
"source_id": f"top500_system_{identity[1]}",
"name": system_name,
"country": country,
"city": city,
"latitude": 0.0,
"longitude": 0.0,
"latitude": None,
"longitude": None,
"value": str(rmax),
"unit": "PFlop/s",
"unit": "FLOP/s",
"metadata": {
"compute_contract": "v1",
"precision": "FP64", "measurement": "LINPACK Rmax",
"source_url": detail_url, "list_url": date_url,
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
"rank": rank,
"cores": cores,
"rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
"site": site,
},
"_detail_url": detail_url,
"reference_date": "2025-11-01",
"reference_date": reference_date,
}
data.append(entry)
except (ValueError, IndexError, AttributeError) as e:
continue
# If scraping failed, return sample data for testing
if not data:
data = self._get_sample_data()
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
return data
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
return 0.0
return 0.0
def _parse_performance(self, text: str) -> float:
def _parse_performance(self, text: str, header: str = "") -> float:
"""Parse performance value from text (handles E, P, T suffixes)"""
text = text.strip().upper()
text = text.strip().upper().replace(",", "")
if not re.search(r"[EPTGMK]F", text):
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
if not unit:
raise ValueError("TOP500 performance unit missing")
text = f"{text} {unit[1]}FLOP/S"
multipliers = {
"E": 1e18,
"P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
# Try simple float parsing
try:
return float(text.replace(",", ""))
value = float(text.replace(",", ""))
if not math.isfinite(value) or value < 0:
raise ValueError("Invalid TOP500 performance")
return value
except ValueError:
return 0.0
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "top500_1",
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
"country": "United States",
"city": "Livermore, CA",
"latitude": 37.6819,
"longitude": -121.7681,
"value": "1742.00",
"unit": "PFlop/s",
"metadata": {
"rank": 1,
"cores": "11039616",
"rmax": "1742.00",
"rpeak": "2746.38",
"power": "29581",
"manufacturer": "HPE",
"site": "DOE/NNSA/LLNL",
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
"interconnect": "Slingshot-11",
"installation_year": "2025",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_2",
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
"country": "United States",
"city": "Oak Ridge, TN",
"latitude": 36.0107,
"longitude": -84.2663,
"value": "1353.00",
"unit": "PFlop/s",
"metadata": {
"rank": 2,
"cores": "9066176",
"rmax": "1353.00",
"rpeak": "2055.72",
"power": "24607",
"manufacturer": "HPE",
"site": "DOE/SC/Oak Ridge National Laboratory",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_3",
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
"country": "United States",
"city": "Argonne, IL",
"latitude": 41.3784,
"longitude": -87.8600,
"value": "1012.00",
"unit": "PFlop/s",
"metadata": {
"rank": 3,
"cores": "9264128",
"rmax": "1012.00",
"rpeak": "1980.01",
"power": "38698",
"manufacturer": "Intel",
},
"reference_date": "2025-11-01",
},
]
raise ValueError("Invalid TOP500 performance") from None

View File

@@ -0,0 +1,200 @@
"""Canonical country and mutually exclusive regional compute participants."""
import re
from collections.abc import Iterable
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
from app.core.earth_geography_policy import display_country
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
# Explicit membership prevents a regional total from double-counting a country.
# APAC deliberately excludes China.
COMPARISON_GROUPS = (
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
(
"EUROPE",
"欧洲",
"Europe",
(
"GB",
"DE",
"FR",
"CH",
"ES",
"FI",
"HU",
"IE",
"IT",
"LU",
"MD",
"NL",
"PL",
"RO",
"RU",
"UA",
),
{"latitude": 54.0, "longitude": 20.0},
),
(
"APAC",
"亚太(不含中国)",
"Asia Pacific (excluding China)",
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
{"latitude": 36.0, "longitude": 130.5},
),
(
"OTHER",
"其他地区",
"Other",
(
"CA",
"AO",
"EG",
"KE",
"MZ",
"NG",
"UG",
"ZA",
"BR",
"CO",
"CR",
"DO",
"MX",
"AE",
"SY",
"TR",
),
{"latitude": 12.0, "longitude": 10.0},
),
)
def _country_code(canonical: str, aliases: list[str]) -> str | None:
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
return "GB" if canonical == "英国" else code
def build_country_catalog() -> list[dict]:
"""Build the policy-projected canonical country drilldown catalog."""
countries = []
for canonical, aliases in COUNTRY_ENTRIES:
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
continue
code = _country_code(canonical, aliases)
if code:
countries.append(
{
"code": code,
"name": canonical,
"name_en": aliases[0],
"kind": "country",
"members": [code],
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
"anchor_kind": "regional_label",
}
)
return countries
COUNTRY_CATALOG = build_country_catalog()
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
def _validate_group_members() -> None:
members = [
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
]
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
raise ValueError("compute comparison groups must cover country catalog exactly once")
_validate_group_members()
GROUP_BY_MEMBER = {
member: code
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
}
def build_catalog() -> list[dict]:
"""Return the five default participants before country drilldown entries."""
catalog: list[dict] = []
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
entry = (
dict(COUNTRY_PARTICIPANTS[code])
if code in COUNTRY_PARTICIPANTS
else {
"code": code,
"name": name,
"name_en": name_en,
"kind": "region",
"members": list(members),
"anchor": dict(anchor),
"anchor_kind": "regional_label",
}
)
entry.update(
{
"members": list(members),
"parent_group": name,
"parent_group_code": code,
"order": index,
"default_selected": True,
"color": PALETTE[index % len(PALETTE)],
}
)
catalog.append(entry)
for entry in COUNTRY_CATALOG:
if entry["code"] in DEFAULT_PARTICIPANTS:
continue
country = dict(entry)
group_code = GROUP_BY_MEMBER[country["code"]]
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
country.update(
{
"parent_group": group_name,
"parent_group_code": group_code,
"order": len(catalog),
"default_selected": False,
"color": PALETTE[len(catalog) % len(PALETTE)],
}
)
catalog.append(country)
return catalog
CATALOG = build_catalog()
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
"""Resolve country or region participant codes to canonical country codes."""
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
def source_country_names(codes: Iterable[str]) -> list[str]:
"""Return unique canonical source labels, respecting China and Serbia ownership."""
selected_members = {
member for members in participant_members(codes).values() for member in members
}
selected_names = {COUNTRIES[code] for code in selected_members}
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
"""Map stored source-country labels to their requested participant code."""
owner_by_country = {
member: participant
for participant, members in participant_members(codes).items()
for member in members
}
code_by_name = {name: code for code, name in COUNTRIES.items()}
return {
name: owner_by_country[code_by_name[display_country(name)]]
for name in source_country_names(codes)
}

View File

@@ -3,7 +3,12 @@
This module is a thin domain wrapper that wires up
:mod:`app.services.location` for compute centers:
SourceCoordinates
Confirmed stored location -> trustworthy source coordinates -> exact
official facility registration -> source city/site coordinates.
Epoch AI coordinates are admitted only at the grain asserted by its official
``Location`` column. Country/region/empty locations remain unresolved even
when the CSV happens to contain a latitude/longitude pair.
The online Nominatim step is intentionally reserved for the user-triggered
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -31,6 +36,8 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.core.collected_data_fields import get_record_field
from app.models.collected_data import CollectedData
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.services.compute_center_verified_sites import get_verified_compute_site
from app.services.epoch_location import assess_epoch_location
from app.services.location import (
LocationCandidate,
@@ -166,8 +173,13 @@ def set_compute_center_location_cache(
async def refresh_compute_center_location_cache(
session: AsyncSession,
*,
as_of: datetime | None = None,
) -> dict[str, dict[str, Any]]:
result = await session.execute(select(ComputeCenterLocationRecord))
query = select(ComputeCenterLocationRecord)
if as_of is not None:
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
result = await session.execute(query)
records = result.scalars().all()
cache = {}
for record in records:
@@ -350,6 +362,58 @@ class ROROrganizationResolver:
)
class ComputeCenterSourceCoordinatesResolver:
"""Use source coordinates only at the grain asserted by the source text."""
name = "source_coordinates"
def __init__(self) -> None:
self._default = SourceCoordinatesResolver()
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
if extra.get("source") != "epoch_ai_gpu":
return self._default.resolve(query)
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
return ResolverOutput()
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput(
attempted_queries=(f"source_coordinates:{assessment.precision}",)
)
output = self._default.resolve(query)
if not output.candidates:
return output
candidate = output.candidates[0]
source_location = extra.get("source_location")
source_location_text = (
source_location.strip() if isinstance(source_location, str) else ""
)
source_note = assessment.reason
if source_location_text:
source_note = f"{source_note}; Epoch Location: {source_location_text}"
return ResolverOutput(
candidates=(
LocationCandidate(
**{
**candidate.__dict__,
"precision": assessment.precision,
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
"source_note": source_note,
"needs_confirmation": assessment.precision != "precise",
"matched_location_name": source_location_text
or candidate.matched_location_name,
}
),
),
attempted_queries=output.attempted_queries,
)
class StoredComputeCenterLocationResolver:
"""Resolve a compute center through the DB-backed current-location cache."""
@@ -367,13 +431,26 @@ class StoredComputeCenterLocationResolver:
longitude = parse_float(stored.get("longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
return ResolverOutput()
precision = stored.get("precision") or "city"
if (
extra.get("source") == "epoch_ai_gpu"
and stored.get("location_source") == "source_coordinates"
):
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput()
precision = assessment.precision
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=latitude,
longitude=longitude,
display_name=stored.get("name") or query.name or "Compute center",
precision=stored.get("precision") or "city",
precision=precision,
confidence=float(stored.get("confidence") or 0.85),
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
source=self.name,
@@ -386,6 +463,60 @@ class StoredComputeCenterLocationResolver:
matched_location_name=stored.get("site") or stored.get("name") or query.name,
location_verified_at=stored.get("verified_at"),
suggested_registry_entry=None,
raw_payload={
"stored_location_source": stored.get("location_source"),
"verification_status": stored.get("verification_status"),
},
),
)
)
class VerifiedComputeSiteResolver:
"""Resolve an exact source identity to an evidence-backed facility site."""
name = "official_facility_registry"
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
source = coerce_str(extra.get("source"))
source_id = coerce_str(extra.get("source_id"))
if not source or not source_id:
return ResolverOutput()
site = get_verified_compute_site(source, source_id)
if not site:
return ResolverOutput()
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
source_note = coerce_str(site.get("source_note"))
if source_urls:
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=float(site["latitude"]),
longitude=float(site["longitude"]),
display_name=coerce_str(site.get("matched_location_name"))
or query.name
or "Compute center",
precision=coerce_str(site.get("precision")) or "site",
confidence=1.0,
query=f"official_facility_registry::{source}:{source_id}",
source=self.name,
source_note=source_note or None,
matched_fields=("source", "source_id"),
needs_confirmation=False,
city=coerce_str(site.get("city")) or None,
region=None,
country=coerce_str(site.get("country")) or query.country,
matched_location_name=coerce_str(site.get("matched_location_name"))
or None,
location_verified_at=coerce_str(site.get("verified_at")) or None,
suggested_registry_entry=None,
raw_payload={
"address": site.get("address"),
"coordinate_basis": site.get("coordinate_basis"),
"sources": source_urls,
},
),
)
)
@@ -400,7 +531,7 @@ def _short_system_name(name: Any) -> str:
return head or text
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
name = coerce_str(getattr(record, "name", None))
return {
"source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +548,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
or metadata.get("manufacturer")
),
"organization": coerce_str(metadata.get("organization")),
"source_location": metadata.get("location"),
"source_raw_country": metadata.get("raw_country"),
}
def _context_to_query(
context: dict[str, str],
context: dict[str, Any],
*,
source_lat: float | None = None,
source_lon: float | None = None,
@@ -444,6 +577,8 @@ def _context_to_query(
"site": context.get("site") or "",
"operator": context.get("operator") or "",
"organization": context.get("organization") or "",
"source_location": context.get("source_location"),
"source_raw_country": context.get("source_raw_country"),
},
)
@@ -497,11 +632,36 @@ def _compute_center_query_plan(
return plan
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
stored_confirmed = (
candidate.source == "stored_compute_center_location"
and not candidate.needs_confirmation
and stored_origin != "source_coordinates"
)
if stored_confirmed:
source_rank = 0
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
source_rank = 1
elif candidate.source == "official_facility_registry":
source_rank = 2
elif candidate.source == "source_coordinates":
source_rank = 3
elif candidate.source == "stored_compute_center_location":
source_rank = 4
else:
source_rank = 5
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
return (source_rank, precision_rank, -float(candidate.confidence or 0))
COMPUTE_CENTER_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
StoredComputeCenterLocationResolver(),
VerifiedComputeSiteResolver(),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
" or stored compute-center location."
@@ -510,7 +670,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
VerifiedComputeSiteResolver(),
ROROrganizationResolver(),
NominatimResolver(
query_plan_builder=_compute_center_query_plan,
@@ -518,6 +679,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
geocoder=lambda q: _geocode_online(q),
),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
", ROR organization lookup, or online geocoding."
@@ -531,6 +693,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
_GEOGRAPHY_MODE_BY_SOURCE = {
"source_coordinates": "source_coordinates",
"stored_compute_center_location": "stored_compute_center_location",
"official_facility_registry": "official_facility_registry",
"ror_organization_registry": "ror_organization",
"nominatim_online_geocode": "online_geocode",
}
@@ -539,7 +702,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
def _candidate_to_location(
candidate: LocationCandidate,
*,
context: dict[str, str],
context: dict[str, Any],
) -> ComputeCenterLocation:
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +757,7 @@ def _candidate_to_location(
def _diagnostic_for(
record: Any,
context: dict[str, str],
context: dict[str, Any],
*,
failure_reason: str,
attempted_queries: tuple[str, ...] = (),
@@ -738,7 +901,7 @@ def build_compute_center_location_query(
organization: str | None = None,
) -> LocationQuery:
name_value = coerce_str(name)
context: dict[str, str] = {
context: dict[str, Any] = {
"source": coerce_str(source),
"source_id": coerce_str(source_id),
"name": name_value,
@@ -783,6 +946,23 @@ async def seed_compute_center_locations_from_source_coords(
longitude = parse_float(get_record_field(record, "longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
continue
metadata = record.extra_data or {}
precision = "precise"
confidence = 1.0
needs_confirmation = False
source_note = "Seeded from source-provided compute-center coordinates"
if source_value == "epoch_ai_gpu":
assessment = assess_epoch_location(
metadata.get("location"),
country=get_record_field(record, "country"),
raw_country=metadata.get("raw_country"),
)
if not assessment.is_renderable:
continue
precision = assessment.precision
confidence = 1.0 if precision == "precise" else 0.9
needs_confirmation = precision != "precise"
source_note = assessment.reason
existing = await session.scalar(
select(ComputeCenterLocationRecord)
.where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +970,6 @@ async def seed_compute_center_locations_from_source_coords(
)
if existing:
continue
metadata = record.extra_data or {}
session.add(
ComputeCenterLocationRecord(
source=source_value,
@@ -802,16 +981,16 @@ async def seed_compute_center_locations_from_source_coords(
country=coerce_str(get_record_field(record, "country")) or None,
latitude=latitude,
longitude=longitude,
precision="precise",
confidence=1.0,
precision=precision,
confidence=confidence,
location_source="source_coordinates",
source_note="Seeded from source-provided compute-center coordinates",
source_note=source_note,
raw_payload={
"record_id": getattr(record, "id", None),
"source": source_value,
"source_id": source_id,
},
needs_confirmation=False,
needs_confirmation=needs_confirmation,
verification_status="source_provided",
verified_at=None,
)

View File

@@ -0,0 +1,44 @@
{
"version": 1,
"sites": [
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
"latitude": 35.380924,
"longitude": 139.625008,
"precision": "site",
"location_source": "official_facility_registry",
"country": "日本",
"city": "Yokohama",
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
]
},
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
"latitude": 41.718109383287235,
"longitude": -87.984498648496,
"precision": "site",
"location_source": "official_facility_registry",
"country": "美国",
"city": "Argonne",
"matched_location_name": "Argonne Leadership Computing Facility",
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.alcf.anl.gov/systems/sophia",
"https://docs.alcf.anl.gov/sophia/",
"https://www.alcf.anl.gov/about/visiting-alcf"
]
}
]
}

View File

@@ -0,0 +1,26 @@
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
import json
from copy import deepcopy
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=1)
def _site_index() -> dict[tuple[str, str], dict]:
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
index = {}
for site in payload["sites"]:
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
raise ValueError("Invalid verified compute facility coordinate")
for source_id in site["source_ids"]:
key = (site["source"], source_id)
if key in index:
raise ValueError("Duplicate verified compute facility identity")
index[key] = site
return index
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
site = _site_index().get((source, source_id))
return deepcopy(site) if site else None

View File

@@ -0,0 +1,99 @@
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
import hashlib
from pathlib import Path
from uuid import uuid4
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import COUNTRY_CATALOG
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
CATEGORY = "compute_comparison"
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
def default_envelope() -> dict:
return {
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
"revision": DEFAULT_REVISION,
"is_default": True,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return dict(record.payload) if record else default_envelope()
def configuration_response(envelope: dict) -> dict:
return {
**envelope,
"countries": [
{key: country[key] for key in ("code", "name", "name_en")}
for country in COUNTRY_CATALOG
],
}
async def publish_configuration(
db: AsyncSession,
*,
revision: str,
config: ComputeConfiguration,
actor: User,
reset: bool = False,
) -> dict:
# The unique category plus row lock also serializes concurrent first writes.
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=default_envelope())
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if record.payload["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
envelope = (
default_envelope()
if reset
else {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"is_default": reset,
}
)
before = record.payload
if reset:
await db.delete(record)
else:
record.payload = envelope
# Audit and configuration commit together: a published edit must be attributable.
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="compute_comparison.reset" if reset else "compute_comparison.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": envelope},
)
)
await db.commit()
return envelope

View File

@@ -0,0 +1,122 @@
"""Versioned, source-backed annual observations for compute comparisons."""
from copy import deepcopy
from datetime import date
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import (
CATALOG,
DEFAULT_PARTICIPANTS,
PARTICIPANTS,
participant_members,
)
from app.services.compute_configuration import DEFAULT_CONFIGURATION
def _participant_value(
members: tuple[str, ...], observations: dict[str, float | int], mode: str
) -> tuple[float | int | None, dict]:
covered = [member for member in members if member in observations]
missing = [member for member in members if member not in observations]
coverage = {
"status": "complete" if not missing else "incomplete",
"required_members": list(members),
"covered_members": covered,
"missing_members": missing,
}
if missing:
return None, coverage
if mode == "bilateral_share":
if (
not set(members).issubset({"CN", "US"})
or not {"CN", "US"}.issubset(observations)
or observations["US"] + observations["CN"] <= 0
):
coverage.update(
{
"status": "incompatible",
"reason": "bilateral_share_requires_CN_and_US_observations",
}
)
return None, coverage
return (
round(
sum(observations[member] for member in members)
/ (observations["US"] + observations["CN"])
* 100,
2,
),
coverage,
)
return sum(observations[member] for member in members), coverage
def compute_history(
countries: list[str],
mode: str = "global_share",
config: ComputeConfiguration | None = None,
) -> dict:
"""Aggregate only fully covered, same-year compatible observations."""
config = config if config is not None else DEFAULT_CONFIGURATION
metric = "global_share" if mode == "bilateral_share" else mode
annual = {row.year: row for row in config.observations if row.metric == metric}
gaps = {row.year: row for row in config.innovation}
observed_years = [row.year for row in config.observations] + list(gaps)
years = tuple(
range(
min(observed_years, default=date.today().year),
max([date.today().year, *observed_years]) + 1,
)
)
members_by_participant = participant_members(countries)
rows = []
for year in years:
record = annual.get(year)
observations = record.values if record else {}
values, coverage = {}, {}
for code, members in members_by_participant.items():
values[code], coverage[code] = _participant_value(members, observations, mode)
evidence_id = record.source_id if record else None
rows.append(
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
)
selected_members = {member for members in members_by_participant.values() for member in members}
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
return {
"version": f"compute-history-{config.reviewed_at}",
"provenance": "curated_reports",
"reviewed_at": config.reviewed_at.isoformat(),
"time_basis": "retrospective",
"years": years,
"mode": mode,
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
mode
],
"catalog": deepcopy(CATALOG),
"default_participants": list(DEFAULT_PARTICIPANTS),
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
"compute": rows,
"innovation": [
{
"year": year,
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
"evidence_id": gaps[year].source_id
if year in gaps and has_innovation_scope
else None,
}
for year in years
],
"evidence": {
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
},
"innovation_scope": ["US", "CN"],
"limitations": [
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
"全球份额不从中美或其他局部成员反推剩余地区。",
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
],
}

View File

@@ -0,0 +1,125 @@
{
"reviewed_at": "2026-09-22",
"sources": [
{
"id": "caict-2021",
"title": "中国算力发展指数白皮书(2021)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2021,
"published_at": null,
"locator": "正文 11 页图 4(2020 年全球份额)"
},
{
"id": "caict-2022",
"title": "中国算力发展指数白皮书(2022)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2022,
"published_at": null,
"locator": "正文 13 页图 5(2021 年全球份额)"
},
{
"id": "caict-2023",
"title": "中国算力发展指数白皮书(2023)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2023,
"published_at": null,
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
},
{
"id": "caict-2024",
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2024,
"published_at": null,
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
},
{
"id": "innovation-2026",
"title": "全球人工智能创新指数报告 2026:科技日报报道",
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
"publisher": "科技日报(新浪转载)",
"report_year": 2026,
"published_at": "2026-07-17",
"locator": "2023–2025 年美国减中国的指数分差"
}
],
"observations": [
{
"year": 2020,
"metric": "global_share",
"source_id": "caict-2021",
"values": {
"US": 36,
"CN": 31,
"JP": 6
}
},
{
"year": 2021,
"metric": "global_share",
"source_id": "caict-2022",
"values": {
"US": 34,
"CN": 33,
"JP": 5
}
},
{
"year": 2022,
"metric": "global_share",
"source_id": "caict-2023",
"values": {
"US": 34,
"CN": 33,
"JP": 4
}
},
{
"year": 2023,
"metric": "global_share",
"source_id": "caict-2024",
"values": {
"US": 41,
"CN": 31,
"JP": 4
}
},
{
"year": 2022,
"metric": "absolute",
"source_id": "caict-2023",
"values": {
"CN": 302
}
},
{
"year": 2023,
"metric": "absolute",
"source_id": "caict-2024",
"values": {
"CN": 435
}
}
],
"innovation": [
{
"year": 2023,
"value": 22.02,
"source_id": "innovation-2026"
},
{
"year": 2024,
"value": 19.96,
"source_id": "innovation-2026"
},
{
"year": 2025,
"value": 17.95,
"source_id": "innovation-2026"
}
]
}

View File

@@ -45,7 +45,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
@@ -56,7 +56,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),

View File

@@ -6,8 +6,10 @@ import asyncio
import hashlib
import json
import shutil
import subprocess
import sys
from datetime import UTC, datetime
from functools import lru_cache
from pathlib import Path
from typing import Any
from uuid import uuid4
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = (
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
)
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
POLICY_FALLBACK_PATH = (
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
)
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
@lru_cache(maxsize=16)
def _read_versioned_asset_cached(
path: Path, mtime_ns: int, size_bytes: int
) -> tuple[dict[str, Any], str]:
payload = _read_json(path)
return payload, _stable_json_hash(payload)
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
if not path.exists():
return {}, ""
stat = path.stat()
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
path = REPO_ROOT / source["path"]
payload = _read_json(path)
features = payload.get("features") if isinstance(payload, dict) else None
return {
"type": "FeatureCollection",
"features": features if isinstance(features, list) else [],
}
def _write_high_precision_geojson_manifest(
sources: list[dict[str, Any]],
build_input_hash: str,
missing_tools: list[str],
) -> dict[str, Any]:
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
admin0_payload = _load_source_feature_collection(admin0)
coastline_payload = _load_source_feature_collection(coastline)
claim_payload = _load_source_feature_collection(claim_lines)
for feature in coastline_payload["features"]:
props = feature.setdefault("properties", {})
if isinstance(props, dict):
props["PLANET_LAYER"] = "coastline"
base_payload = {
"type": "FeatureCollection",
"features": [*admin0_payload["features"], *coastline_payload["features"]],
}
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
_write_json(base_path, base_payload)
_write_json(hover_path, admin0_payload)
_write_json(claim_path, claim_payload)
manifest = {
"version": "natural-earth-v1",
"builtAt": _utc_now_iso(),
"tileProvider": "geojson-high-precision",
"format": "geojson-directory",
"buildInputHash": build_input_hash,
"base": base_path.name,
"hoverIndex": hover_path.name,
"claimLine": claim_path.name,
"sourceFeatureCount": {
"admin0": len(admin0_payload["features"]),
"coastline": len(coastline_payload["features"]),
"claimLines": len(claim_payload["features"]),
},
"pmtiles": None,
"missingTools": missing_tools,
}
result = subprocess.run(
[
sys.executable,
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
"--admin0-source", admin0["path"],
"--coastline-source", coastline["path"],
"--claims-source", claim_lines["path"],
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
"--build-input-hash", build_input_hash,
"--pov-policy", _relative(POV_POLICY_PATH),
"--pov-geometry", str(POV_GEOMETRY_PATH),
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
)
if result.returncode:
raise EarthBoundaryBuildError(
result.stderr.strip() or "Earth boundary policy build failed",
code="policy_build_failed",
)
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
manifest["missingTools"] = missing_tools
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
return manifest
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
"build_config": BUILD_CONFIG,
}
)
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
bool(boundary_manifest)
and boundary_manifest.get("buildInputHash") == build_input_hash
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and _policy_verified(boundary_manifest)
and PMTILES_ARTIFACT_PATH.exists()
and _pmtiles_hover_exists(boundary_manifest)
)
def _policy_verified(manifest: dict[str, Any]) -> bool:
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
coverage = manifest.get("povPolicy") or {}
return (
policy.get("profile") == "china-pov-v1"
and coverage.get("profile") == policy["profile"]
and coverage.get("sha256") == policy_hash
and coverage.get("reviewedGeometrySourceCommit")
== geometry.get("sourceCommit")
and coverage.get("geometrySha256") == geometry_hash
and coverage.get("applied") is True
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
<= set(coverage.get("appliedRuleIds") or [])
)
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
paths = [manifest.get("base"), manifest.get("hoverIndex")]
if manifest.get("chinaClaims", {}).get("available"):
paths.append(manifest.get("claimLine"))
for asset in paths:
if not isinstance(asset, str) or Path(asset).name != asset:
return False
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
return False
return True
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
hover = manifest.get("hoverIndex")
return (
isinstance(hover, str)
and Path(hover).name == hover
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
)
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
high_precision_ready = (
manifest_exists
and _policy_verified(boundary_manifest)
and (
(
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and pmtiles_exists
and _pmtiles_hover_exists(boundary_manifest)
)
or (
boundary_manifest.get("tileProvider") == "geojson-high-precision"
and _geojson_artifacts_exist(boundary_manifest)
)
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
)
)
legacy_exists = LEGACY_GEOJSON_PATH.exists()
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
_versioned_asset(POLICY_FALLBACK_PATH)[0]
)
provider = (
boundary_manifest.get("tileProvider")
if high_precision_ready
else "legacy-geojson" if legacy_exists else "missing"
else "policy-geojson-fallback" if legacy_exists else "missing"
)
return {
"provider": provider,
"high_precision_ready": high_precision_ready,
"fallback_available": legacy_exists,
"policy_publishable": bool(
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
),
"config_source": config_source,
"config_path": _relative(LOCAL_CONFIG_PATH),
"config_exists": LOCAL_CONFIG_PATH.exists(),
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
"tileProvider": boundary_manifest.get("tileProvider"),
"buildInputHash": boundary_manifest.get("buildInputHash"),
"builtAt": boundary_manifest.get("builtAt"),
"povPolicy": boundary_manifest.get("povPolicy"),
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
},
"pmtiles": {
"path": _relative(PMTILES_ARTIFACT_PATH),
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
},
"legacy": {
"path": _relative(LEGACY_GEOJSON_PATH),
"path": _relative(POLICY_FALLBACK_PATH),
"exists": legacy_exists,
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
},
"source_manifest": {
"path": _relative(SOURCE_MANIFEST_PATH),
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
build_input_hash,
"--pov-policy",
_relative(POV_POLICY_PATH),
"--pov-geometry",
str(POV_GEOMETRY_PATH),
]
)
if progress_callback:

View File

@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
source_has_adapter = bool(get_earth_update_layers_for_source(source))
effective_source = source if source_has_adapter else (table_name if table_name else source)
operation = payload.get("operation")
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
refresh_strategy = "reload"
update: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
operation = payload.get("operation")
if operation:
self.operations.add(str(operation))
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
self.refresh_strategy = "reload"
entity_keys = payload.get("entity_keys")
if not isinstance(entity_keys, list):
entity_key = payload.get("entity_key")
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
if not update:
return False
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
from app.core.websocket.broadcaster import broadcaster
keys = payload.get("entity_keys")
if not isinstance(keys, list):
keys = [payload.get("entity_key")]
broadcaster.enqueue_vessel_update({
"action": "remove",
"vessels": [{"mmsi": key} for key in keys if key is not None],
})
source = update["source"]
pending = self._pending.get(source)
if pending is None:

View File

@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
layers=("vessels",),
cache_patterns=("vessels*", "summary*"),
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
refresh_strategy="delta",
),
EarthLayerAdapter(
sources=frozenset(

View File

@@ -865,9 +865,9 @@ def _get_locale_text(
if isinstance(fallback, dict):
value = _coerce_str(fallback.get(key))
if value:
if locale == "en-US" and _contains_cjk_text(value):
return ""
return value
if locale == "en-US" and _is_chinese_language(item.content_language):
return item.title if key == "title" else item.summary
return ""

View File

@@ -0,0 +1,198 @@
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
from __future__ import annotations
from dataclasses import dataclass
import math
import re
from typing import Any
from app.core.countries import normalize_country
_COUNTRY_LABEL_OVERRIDES = {
"korea republic of": "韩国",
}
_REGION_ONLY_LABELS = {
"europe",
"iowa",
"maryland",
"new mexico",
"texas",
"the nordics",
"virginia",
"washington",
}
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
re.IGNORECASE,
)
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
re.IGNORECASE | re.DOTALL,
)
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
)
_NUMBERED_STREET_PATTERN = re.compile(
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
r"highway|hwy|way|parkway|pkwy|chome)\b|"
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
re.IGNORECASE,
)
_POSTAL_CODE_PATTERN = re.compile(
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
re.IGNORECASE,
)
_SITE_CONTEXT_PATTERN = re.compile(
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
re.IGNORECASE,
)
_COORDINATE_CUE_PATTERN = re.compile(
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
re.IGNORECASE | re.DOTALL,
)
@dataclass(frozen=True)
class EpochLocationAssessment:
precision: str
reason: str
@property
def is_renderable(self) -> bool:
return self.precision in {"precise", "site", "city"}
def _normalized_label(value: Any) -> str:
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
def _country_from_label(value: Any) -> str | None:
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
_normalized_label(value)
)
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
if isinstance(raw, bool):
return None
try:
number = float(raw)
except (TypeError, ValueError):
return None
return number if math.isfinite(number) and abs(number) <= bound else None
def _has_explicit_coordinate_pair(text: str) -> bool:
hemispheres = {}
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
hemisphere = match.group("hemisphere").upper()
degrees = int(match.group("degrees"))
minutes = int(match.group("minutes"))
seconds = float(match.group("seconds") or 0)
bound = 90 if hemisphere in {"N", "S"} else 180
if degrees <= bound and minutes < 60 and seconds < 60:
hemispheres[hemisphere] = degrees
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
return True
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
if labeled_pair:
latitude = float(labeled_pair.group("latitude"))
longitude = float(labeled_pair.group("longitude"))
return abs(latitude) <= 90 and abs(longitude) <= 180
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
if not decimal_pair:
return False
latitude, longitude = map(float, decimal_pair.groups())
return abs(latitude) <= 90 and abs(longitude) <= 180
def _has_address_evidence(text: str) -> bool:
if _NUMBERED_STREET_PATTERN.search(text):
return True
return bool(
_POSTAL_CODE_PATTERN.search(text)
and ("\n" in text or text.count(",") >= 2)
and _SITE_CONTEXT_PATTERN.search(text)
)
def assess_epoch_location(
location: Any,
*,
country: Any = None,
raw_country: Any = None,
) -> EpochLocationAssessment:
"""Classify source coordinates without upgrading their source text.
Epoch's CSV supplies coordinates even when ``Location`` contains only a
country. Those coordinates are useful source evidence, but are not a
facility point and must not enter the renderable location pipeline.
"""
if isinstance(location, dict):
latitude = _coordinate(location, ("lat", "latitude"), 90)
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
has_coordinates = latitude is not None and longitude is not None
return EpochLocationAssessment(
"precise" if has_coordinates else "unknown",
"complete explicit coordinate object"
if has_coordinates
else "coordinate object lacks a complete valid latitude/longitude pair",
)
text = str(location or "").strip()
if not text:
return EpochLocationAssessment("unknown", "Epoch Location is empty")
normalized = _normalized_label(text)
normalized_country = normalize_country(country)
source_country = normalize_country(raw_country)
location_country = _country_from_label(text)
if location_country or (
raw_country and normalized == _normalized_label(raw_country)
):
matched = location_country or source_country or normalized_country
return EpochLocationAssessment(
"country",
f"Epoch Location identifies country only: {matched or text}",
)
if normalized in _REGION_ONLY_LABELS:
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
if (
len(comma_parts) == 2
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
and _country_from_label(comma_parts[1])
):
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
if normalized in _UNLOCATABLE_LABELS:
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
if normalized.isdigit():
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
if _has_explicit_coordinate_pair(text):
return EpochLocationAssessment(
"precise", "Epoch Location contains a complete valid coordinate pair"
)
if _COORDINATE_CUE_PATTERN.search(text):
return EpochLocationAssessment(
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
)
if _has_address_evidence(text):
return EpochLocationAssessment("precise", "Epoch Location contains an address")
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
return EpochLocationAssessment("city", "Epoch Location identifies a locality")

View File

@@ -0,0 +1,186 @@
"""Authenticated model discovery shared by refresh and connection checks."""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime
from urllib.parse import urlsplit, urlunsplit
import httpx
CATALOG_TIMEOUT_SECONDS = 30
CATALOG_MAX_PAGES = 100
CATALOG_PAGE_SIZE = 100
CATALOG_REQUEST_ATTEMPTS = 2
CATALOG_RETRY_DELAY_SECONDS = 0.2
SUPPORTED_PROVIDER_APIS = {
"openai-completions",
"openai-responses",
"anthropic-messages",
"ollama-generate",
}
class LLMProviderCatalogError(RuntimeError):
"""A safe, user-facing catalog failure with no upstream response or credentials."""
@dataclass(frozen=True)
class ModelCatalog:
url: str
models: list[str]
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
parts = urlsplit(base_url.strip())
if parts.scheme not in {"http", "https"} or not parts.hostname:
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
if parts.username or parts.password or parts.query or parts.fragment:
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
if provider_api not in SUPPORTED_PROVIDER_APIS:
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
path = parts.path.rstrip("/")
if provider_api == "ollama-generate":
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
path = "/api/v1/models"
else:
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
path += "/v1"
path += "/models"
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
def catalog_error_message(exc: Exception) -> str:
if isinstance(exc, LLMProviderCatalogError):
return str(exc)
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
return "模型目录请求超时,请检查网络后重试。"
if isinstance(exc, httpx.HTTPStatusError):
code = exc.response.status_code
messages = {
401: "API Key 验证失败,请检查当前供应商的凭证。",
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
429: "供应商请求限流,请稍后重试。",
}
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
if isinstance(exc, httpx.RequestError):
return "无法连接模型目录,请检查基础地址和网络。"
return "模型目录响应无效,请稍后重试。"
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
if not isinstance(payload, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
envelope = payload.get("output", payload)
if not isinstance(envelope, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
rows = envelope.get("data", envelope.get("models"))
if not isinstance(rows, list):
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
if any(not isinstance(row, dict) for row in rows):
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
return rows, envelope
def _model_id(row: dict[str, object]) -> str:
value = row.get("id") or row.get("model") or row.get("name")
if not isinstance(value, str) or not value.strip():
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
return value.strip()
def _model_date(row: dict[str, object]) -> float:
value = row.get("created_at") or row.get("published_time") or row.get("created")
if isinstance(value, (int, float)):
return float(value)
if isinstance(value, str):
try:
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
except ValueError:
pass
return 0
async def _get_catalog_page(
client: httpx.AsyncClient,
url: str,
headers: dict[str, str],
params: dict[str, str | int],
) -> object:
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
try:
response = await client.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
except httpx.HTTPStatusError as exc:
if attempt or exc.response.status_code not in {502, 503, 504}:
raise
except httpx.TransportError:
if attempt:
raise
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
raise LLMProviderCatalogError("模型目录请求失败。")
async def fetch_model_catalog(
provider: str,
base_url: str,
provider_api: str,
api_key: str = "",
anthropic_version: str = "2023-06-01",
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
) -> ModelCatalog:
url = model_catalog_url(provider, base_url, provider_api)
public_catalog = provider in {"opencode-go", "openrouter"}
if not api_key and provider_api != "ollama-generate" and not public_catalog:
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
if provider_api == "anthropic-messages" and provider not in {
"opencode-go",
"openrouter",
"alibaba",
"moonshotai",
}:
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
elif api_key:
headers["Authorization"] = f"Bearer {api_key}"
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
params: dict[str, str | int] = {}
if native_dashscope:
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
rows_by_id: dict[str, dict[str, object]] = {}
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
# Bound the complete pagination/retry cycle, not just each individual request.
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
for page in range(CATALOG_MAX_PAGES):
payload = await _get_catalog_page(client, url, headers, params)
rows, envelope = _model_rows(payload)
previous_count = len(rows_by_id)
for row in rows:
rows_by_id[_model_id(row)] = row
has_more = envelope.get("has_more") is True
if native_dashscope:
total = envelope.get("total")
if not isinstance(total, int) or total < 0:
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
has_more = len(rows_by_id) < total
params["page_no"] = page + 2
elif has_more:
cursor = envelope.get("last_id")
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
params["after_id"] = cursor
if not has_more:
models = sorted(
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
)
# An empty Ollama catalog is valid: no models have been installed yet.
if not models and provider_api != "ollama-generate":
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
return ModelCatalog(url=url, models=models)
if len(rows_by_id) == previous_count:
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")

View File

@@ -3,17 +3,30 @@
from __future__ import annotations
from typing import Any
from urllib.parse import urlsplit
import httpx
MODELS_DEV_URL = "https://models.dev/api.json"
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
from app.services.llm_model_catalog import fetch_model_catalog
OPENCODE_GO_MODEL_PROVIDER_APIS = {
"minimax-m3": "anthropic-messages",
"qwen3.8-max": "anthropic-messages",
"qwen3.8-flash": "anthropic-messages",
"qwen3.7-max": "anthropic-messages",
"qwen3.7-plus": "anthropic-messages",
"qwen3.6-plus": "anthropic-messages",
"grok-4.6": "openai-responses",
"gpt-5.6-luna": "openai-responses",
"muse-spark-1.3-contributor": "openai-responses",
"muse-spark-1.2-contributor": "openai-responses",
"minimax-m2.7": "anthropic-messages",
"minimax-m2.5": "anthropic-messages",
}
OPENCODE_GO_FALLBACK_MODELS = [
"minimax-m3",
"kimi-k3",
"glm-5.3",
"qwen3.8-max",
"gpt-5.6-luna",
"minimax-m2.7",
"minimax-m2.5",
"kimi-k2.6",
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
]
OPENAI_MODEL_PROVIDER_APIS = {
model: "openai-responses"
for model in [
"gpt-6-astra",
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
"gpt-5.1",
"gpt-5.1-codex",
]
}
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"minimax": {
"provider": "minimax",
"label": "MiniMax",
"provider_api": "anthropic-messages",
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-M2.7",
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
"model": "MiniMax-M3",
"models": [
"MiniMax-M3",
"MiniMax-M2.7",
"MiniMax-M2.7-highspeed",
"MiniMax-M2.5",
"MiniMax-M2.5-highspeed",
"MiniMax-M2.1",
"MiniMax-M2.1-highspeed",
"MiniMax-M2",
],
"api_key_env": "MINIMAX_API_KEY",
"source": "fallback",
},
"openai": {
"provider": "openai",
"label": "OpenAI",
"provider_api": "openai-completions",
"provider_api": "openai-responses",
"base_url": "https://api.openai.com/v1",
"model": "gpt-5.1",
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
"model": "gpt-6-astra",
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
"api_key_env": "OPENAI_API_KEY",
"source": "fallback",
},
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Anthropic",
"provider_api": "anthropic-messages",
"base_url": "https://api.anthropic.com/v1",
"model": "claude-sonnet-4-6",
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
"model": "claude-opus-5",
"models": ["claude-opus-5", "claude-sonnet-4-6"],
"api_key_env": "ANTHROPIC_API_KEY",
"source": "fallback",
},
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "DeepSeek",
"provider_api": "openai-completions",
"base_url": "https://api.deepseek.com/v1",
"model": "deepseek-chat",
"models": ["deepseek-chat", "deepseek-reasoner"],
"model": "deepseek-flash",
"models": ["deepseek-flash", "deepseek-v4-pro"],
"api_key_env": "DEEPSEEK_API_KEY",
"source": "fallback",
},
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Alibaba Qwen / DashScope",
"provider_api": "openai-completions",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"model": "qwen3-max",
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
"model": "qwen3.8-max",
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
"api_key_env": "DASHSCOPE_API_KEY",
"source": "fallback",
},
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Moonshot AI / Kimi",
"provider_api": "openai-completions",
"base_url": "https://api.moonshot.ai/v1",
"model": "kimi-k2.5",
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
"model": "kimi-k3",
"models": ["kimi-k3", "kimi-k2.5"],
"api_key_env": "MOONSHOT_API_KEY",
"source": "fallback",
},
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
},
}
MODELS_DEV_PROVIDER_KEYS = {
"minimax": "minimax",
"openai": "openai",
"anthropic": "anthropic",
"deepseek": "deepseek",
"alibaba": "alibaba",
"moonshotai": "moonshotai",
"openrouter": "openrouter",
}
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
}
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
async def refresh_llm_provider_preset(
provider: str,
api_key: str | None = None,
*,
base_url: str | None = None,
provider_api: str | None = None,
anthropic_version: str = "2023-06-01",
) -> dict[str, Any]:
fallback = get_fallback_llm_provider_preset(provider)
if fallback["provider"] == "opencode-go":
headers = {"User-Agent": "Planet/1.0"}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
OPENCODE_GO_MODELS_URL,
headers=headers,
)
response.raise_for_status()
payload = response.json()
data = payload.get("data") if isinstance(payload, dict) else []
model_ids = [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
][:120]
if not model_ids:
model_ids = fallback["models"]
return {
**fallback,
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
"models": model_ids,
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
"source": OPENCODE_GO_MODELS_URL,
}
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
if not models_dev_key:
return fallback
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
MODELS_DEV_URL,
headers={"User-Agent": "Planet/1.0"},
)
response.raise_for_status()
catalog = response.json()
upstream = catalog.get(models_dev_key)
if not isinstance(upstream, dict):
return fallback
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
model_ids = list(upstream_models.keys())[:80]
base_url = upstream.get("api") or fallback["base_url"]
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
base_url = "https://api.deepseek.com/v1"
refreshed = {
resolved_base_url = base_url or fallback["base_url"]
resolved_api = provider_api or fallback["provider_api"]
catalog = await fetch_model_catalog(
fallback["provider"],
resolved_base_url,
resolved_api,
api_key or "",
anthropic_version,
)
model_provider_apis = (
_opencode_go_model_provider_apis(catalog.models)
if fallback["provider"] == "opencode-go"
else fallback.get("model_provider_apis", {})
)
if (
fallback["provider"] == "openai"
and urlsplit(resolved_base_url).hostname != "api.openai.com"
):
model_provider_apis = {}
return {
**fallback,
"label": upstream.get("name") or fallback["label"],
"base_url": base_url,
"model": model_ids[0] if model_ids else fallback["model"],
"models": model_ids or fallback["models"],
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
"source": MODELS_DEV_URL,
"base_url": resolved_base_url,
"provider_api": resolved_api,
"model": catalog.models[0] if catalog.models else "",
"models": catalog.models,
"model_provider_apis": model_provider_apis,
"source": catalog.url,
}
return refreshed

View File

@@ -635,6 +635,7 @@ async def _run_assistant_message(
constraints=_collect_constraints(payload.constraints),
context={
"source": "playground",
"session_id": session_id,
"preset": payload.selected_preset_key,
"conversation_history": conversation_history,
"history_size": len(conversation_history),

View File

@@ -0,0 +1,148 @@
"""Search and paginate the public live TV catalog at the database boundary."""
from typing import Any
from sqlalchemy import Select, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.models.collected_data import CollectedData
from app.services.tv_streams import (
TV_LIVE_SOURCE_COLLECTOR,
TV_LIVE_SOURCE_DATA_TYPE,
_build_collected_tv_source,
build_public_tv_payload,
get_tv_settings_payload,
)
def _source_key():
return func.coalesce(
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
func.nullif(CollectedData.source_id, ""),
CollectedData.entity_key,
)
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
metadata = CollectedData.extra_data
source_id = _source_key()
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
ranked = (
select(
CollectedData.id,
func.row_number()
.over(
partition_by=source_id,
order_by=CollectedData.id.desc(),
)
.label("source_rank"),
)
.where(
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
CollectedData.is_current.is_(True),
CollectedData.is_valid == 1,
CollectedData.deleted_at.is_(None),
enabled.notin_(("false", "0", "no", "off")),
source_id.notin_(configured_ids),
)
.subquery()
)
return (
select(CollectedData)
.join(ranked, ranked.c.id == CollectedData.id)
.where(ranked.c.source_rank == 1)
)
def _filter_catalog_query(
query: Select[tuple[CollectedData]], terms: list[str]
) -> Select[tuple[CollectedData]]:
metadata = CollectedData.extra_data
searchable = func.lower(
func.concat_ws(
" ",
CollectedData.name,
CollectedData.title,
CollectedData.source_id,
metadata["name"].as_string(),
metadata["provider"].as_string(),
metadata["region"].as_string(),
metadata["country"].as_string(),
metadata["language"].as_string(),
)
)
for term in terms:
query = query.where(searchable.contains(term, autoescape=True))
return query
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
searchable = " ".join(
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
).lower()
return all(term in searchable for term in terms)
async def get_tv_catalog_page(
db: AsyncSession,
*,
offset: int = 0,
limit: int = 50,
q: str = "",
selected_id: str | None = None,
) -> dict[str, Any]:
settings = await get_tv_settings_payload(db)
payload = build_public_tv_payload(settings, [])
configured = payload["sources"]
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
if selected_id:
selected = next((source for source in configured if source["id"] == selected_id), None)
if selected is None:
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
selected = _build_collected_tv_source(record, 0) if record else None
if selected:
payload["selected_source"] = selected
summary = (
await db.execute(
select(func.count(), func.max(CollectedData.collected_at))
.select_from(CollectedData)
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
)
).one()
total_collected, latest_update = summary
terms = q.lower().split()
matched_configured = [source for source in configured if _matches_source(source, terms)]
filtered_query = _filter_catalog_query(query, terms)
matched_collected = (
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
if terms
else total_collected
)
sources = matched_configured[offset : offset + limit]
remaining = limit - len(sources)
if remaining:
rows = await db.scalars(
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
.offset(max(0, offset - len(matched_configured)))
.limit(remaining)
)
sources.extend(
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
)
total = len(matched_configured) + matched_collected
next_offset = offset + len(sources)
return {
**payload,
"sources": sources,
"source_count": len(configured) + total_collected,
"latest_updated_at": (
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
),
"total": total,
"offset": offset,
"limit": limit,
"has_more": next_offset < total,
"next_offset": next_offset if next_offset < total else None,
}

View File

@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
from app.models.collected_data import CollectedData
from app.models.system_setting import SystemSetting
DEFAULT_TV_SOURCE_ID = "cgtn-en"
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
TV_SETTINGS_CATEGORY = "tv"
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
UNSUPPORTED_CCTV4_STREAM = (
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
)
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
DEFAULT_TV_SETTINGS = {
"default_source_id": DEFAULT_TV_SOURCE_ID,
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
"sources": [
{
"id": "cctv4",
"name": "CCTV-4 中文国际",
"name": "CCTV-4 中文国际(美洲版)",
"provider": "CCTV",
"region": "China",
"language": "zh-CN",
"source_type": "hls",
"embed_url": "https://tv.cctv.com/live/cctv4/",
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
"homepage_url": "https://tv.cctv.com/live/cctv4/",
"embed_url": "",
"stream_url": CCTV4_STREAM,
"use_proxy": False,
"homepage_url": CCTV4_HOMEPAGE,
"poster_url": "",
"is_enabled": True,
"is_fallback": True,
"sort_order": 10,
"collector_source": None,
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
"notes": CCTV4_PLAYBACK_NOTE,
},
{
"id": "reuters-tv",
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
payload = dict(source or {})
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
payload.get("stream_url")
) == UNSUPPORTED_CCTV4_STREAM:
payload.update(
name="CCTV-4 中文国际(美洲版)",
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
use_proxy=False,
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
)
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
source_type = _clean_text(payload.get("source_type")).lower()
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
"source_type": source_type,
"embed_url": _clean_url(payload.get("embed_url")),
"stream_url": _clean_url(payload.get("stream_url")),
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
"homepage_url": _clean_url(payload.get("homepage_url")),
"poster_url": _clean_url(payload.get("poster_url")),
"youtube_video_id": youtube_video_id,
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
]
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
default_source = next(
source for source in DEFAULT_TV_SETTINGS["sources"]
if source["id"] == DEFAULT_TV_SOURCE_ID
)
normalized_sources.append(
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
normalize_tv_source(default_source, index=len(normalized_sources))
)
default_source_exists = any(

View File

@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
return [item.to_dict() for item in result.scalars().all()]
async def get_current_vessels_by_mmsi(
db: AsyncSession, mmsis: list[int]
) -> list[dict[str, Any]]:
"""Read canonical render state for the vessels changed by a stream flush."""
if not mmsis:
return []
result = await db.execute(
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
)
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
async def aggregate_vessel_observations(
db: AsyncSession,
observations: Iterable[AISRawObservation],

View File

@@ -0,0 +1,72 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from fastapi import FastAPI
from pydantic import ValidationError
from app.api.v1.auth import router
from app.core import security
from app.db.session import DEFAULT_LOGIN_USERS, get_db
from app.models.user import User
from app.schemas.user import UserRegister
async def identity_request(monkeypatch, user, token=None):
result = MagicMock()
result.scalar_one_or_none.return_value = user
db = AsyncMock()
db.execute.return_value = result
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
app = FastAPI()
app.include_router(router, prefix="/auth")
async def override_db():
yield db
app.dependency_overrides[get_db] = override_db
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
return await client.get(
"/auth/me",
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
created = datetime(2026, 9, 21, tzinfo=UTC)
user = User(
id=1, username=default["username"], email=default["email"],
role=default["role"], password_hash="not-exposed", is_active=True,
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
)
response = await identity_request(monkeypatch, user)
assert response.status_code == 200
payload = response.json()
assert payload["email"] == default["email"]
assert payload["email_verified"] is True
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
assert payload["gatekeeper_groups"] == ["ops"]
assert "password_hash" not in payload
@pytest.mark.asyncio
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
response = await identity_request(monkeypatch, user)
assert response.status_code == 401
@pytest.mark.asyncio
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
assert response.status_code == 401
def test_registration_still_validates_email():
with pytest.raises(ValidationError):
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")

View File

@@ -79,7 +79,8 @@ class TestTOP500Collector:
"""Test parsing empty response"""
collector = TOP500Collector()
result = collector.parse_response("<html><body><table></table></body></html>")
assert len(result) > 0
assert result == []
assert collector.fail_on_empty
def test_parse_response_single_item(self):
"""Test parsing single item response"""
@@ -97,9 +98,9 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["source_id"] == "top500_1"
assert result[0]["source_id"] == "top500_system_1"
assert result[0]["name"] == "Test Supercomputer"
assert result[0]["country"] == "USA"
assert result[0]["metadata"]["rank"] == 1
@@ -113,7 +114,7 @@ class TestTOP500Collector:
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
<tr>
<td>1</td>
<td>Valid\nVendor\nSite\nUSA</td>
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
<td>1000</td>
<td>10 PFLOP/s</td>
<td>12 PFLOP/s</td>
@@ -129,7 +130,7 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["name"] == "Valid"

View File

@@ -0,0 +1,222 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.system_log import AuditLog
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
default_envelope,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import FastAPI, HTTPException, Response
from httpx import ASGITransport, AsyncClient
from pydantic import ValidationError
def default_config():
return DEFAULT_CONFIGURATION.model_dump(mode="json")
@pytest.mark.parametrize(
"mutation",
[
lambda value: value["observations"][0]["values"].update(CN=101),
lambda value: value["observations"][0]["values"].update(CN=-1),
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
lambda value: value["observations"][0]["values"].update(CN=True),
lambda value: value["observations"][0]["values"].update(EUROPE=10),
lambda value: value["observations"][0].update(source_id="unknown"),
lambda value: value["observations"][0].update(year=2026),
lambda value: value["observations"].append(value["observations"][0]),
lambda value: value["sources"].append(value["sources"][0]),
lambda value: value["innovation"].append(value["innovation"][0]),
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
],
)
def test_rejects_invalid_or_ambiguous_evidence(mutation):
payload = default_config()
mutation(payload)
with pytest.raises(ValidationError):
ComputeConfiguration.model_validate(payload)
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
payload = default_config()
payload["sources"].append(
{
"id": "new-report",
"title": "New report",
"url": "https://example.com/report",
"publisher": "Research group",
"report_year": 2026,
"locator": "Figure 2",
}
)
payload["observations"].append(
{
"year": 2025,
"metric": "global_share",
"source_id": "new-report",
"values": {"CN": 25, "US": 50, "JP": 0},
}
)
config = ComputeConfiguration.model_validate(payload)
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
row = next(row for row in result["compute"] if row["year"] == 2025)
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
payload = default_config()
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
result = compute_history(
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
)
assert result["compute"][0]["values"] == {"CN": None, "US": None}
payload.update(sources=[], observations=[], innovation=[])
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
assert all(row["values"]["CN"] is None for row in result["compute"])
assert result["evidence"] == {}
def fake_db(record=None):
db = AsyncMock()
db.add = MagicMock()
result = MagicMock()
result.scalar_one_or_none.return_value = record
result.scalar_one.return_value = record
db.execute.return_value = result
return db
@pytest.mark.asyncio
async def test_fresh_install_uses_defaults_without_writing_database():
db = fake_db()
assert await read_configuration(db) == default_envelope()
db.add.assert_not_called()
db.commit.assert_not_awaited()
@pytest.mark.asyncio
async def test_series_reads_persisted_override_and_disables_http_cache():
envelope = default_envelope()
envelope["is_default"] = False
envelope["revision"] = "custom"
envelope["config"]["observations"][3]["values"]["CN"] = 30
response = Response()
data = await get_series(
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
)
assert data["compute"][3]["values"]["CN"] == 30
assert data["version"] == "custom"
assert data["provenance"] == "admin_curated_reports"
assert response.headers["Cache-Control"] == "no-store"
@pytest.mark.asyncio
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
actor = SimpleNamespace(id=1, username="admin")
old = deepcopy(record.payload)
result = await publish_configuration(
db,
revision=old["revision"],
config=DEFAULT_CONFIGURATION,
actor=actor,
)
assert result["revision"] != old["revision"]
audit = db.add.call_args.args[0]
assert isinstance(audit, AuditLog)
assert audit.details == {"before": old, "after": result}
assert audit.actor_id == actor.id
db.commit.assert_awaited_once()
with pytest.raises(HTTPException) as failure:
await publish_configuration(
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
)
assert failure.value.status_code == 409
assert record.payload == result
db.rollback.assert_awaited_once()
db.commit.assert_awaited_once()
@pytest.mark.asyncio
async def test_reset_changes_revision_and_restores_bundled_data():
record = SimpleNamespace(
payload={**default_envelope(), "revision": "custom", "is_default": False}
)
result = await publish_configuration(
fake_db(record),
revision="custom",
config=DEFAULT_CONFIGURATION,
actor=SimpleNamespace(id=1, username="admin"),
reset=True,
)
assert result["is_default"] is True
assert result["config"] == default_config()
assert result["revision"] != "custom"
@pytest.mark.asyncio
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
async def test_non_admin_cannot_configure(role):
with pytest.raises(HTTPException) as failure:
await configuration_admin(SimpleNamespace(role=role))
assert failure.value.status_code == 403
@pytest.mark.asyncio
async def test_http_configuration_auth_validation_and_public_series():
app = FastAPI()
app.include_router(router)
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
assert (await client.get("/series")).status_code == 200
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=1,
username="admin",
role="admin",
)
loaded = (await client.get("/configuration")).json()
invalid = deepcopy(loaded["config"])
invalid["observations"][0]["values"]["CN"] = 200
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": invalid,
},
)
assert response.status_code == 422
db.commit.assert_not_awaited()
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": loaded["config"],
},
)
assert response.status_code == 200
assert response.json()["revision"] != loaded["revision"]
assert (
await client.post(
"/configuration/reset",
json={
"revision": loaded["revision"],
},
)
).status_code == 409

View File

@@ -0,0 +1,268 @@
import csv
import io
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import HTTPException
from sqlalchemy.dialects import postgresql
from app.api.v1.compute_intelligence import sample_query, selected_countries
from app.services.collectors.epoch_ai import EpochAIGPUCollector
from app.services.collectors.top500 import TOP500Collector
from app.services.compute_history import compute_history
def epoch_csv(**overrides):
row = {
"Name": "Cluster Alpha",
"Country": "United States of America",
"Status": "Existing",
"First Operational Date": "2023-05-02",
"16-bit OP/s (log)": "18",
"Location": "3231 Riverport Rd, Memphis, TN 38109",
"latitude": "35",
"longitude": "-90",
**overrides,
}
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
return output.getvalue()
def test_epoch_csv_preserves_precision_date_and_missing_values():
collector = EpochAIGPUCollector()
row = collector.parse_response(epoch_csv())[0]
assert row["value"] == 1e18
assert row["unit"] == "FLOP/s"
assert row["metadata"]["precision"] == "FP16"
assert row["metadata"]["location_precision"] == "precise"
assert row["metadata"]["raw_country"] == "United States of America"
assert row["reference_date"] == "2023-05-02"
missing = collector.parse_response(
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
)[0]
assert missing["value"] is None
assert missing["reference_date"] is None
@pytest.mark.parametrize(
("location", "country", "expected"),
[
("United States of America", "United States of America", "country"),
("Japan", "Japan", "country"),
("Korea (Republic of)", "South Korea", "country"),
("Texas", "United States of America", "region"),
("Iowa, USA", "United States of America", "region"),
("Seoul", "South Korea", "city"),
("Paris, France", "France", "city"),
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
("Cluster 2, Texas", "United States of America", "site"),
("Campus 7, Tokyo", "Japan", "site"),
("Building 240, Argonne", "United States of America", "site"),
("7", "United States of America", "unknown"),
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
("35.380924,139.625008", "Japan", "precise"),
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
],
)
def test_epoch_location_precision_follows_location_field(location, country, expected):
row = EpochAIGPUCollector().parse_response(
epoch_csv(Location=location, Country=country)
)[0]
assert row["metadata"]["location_precision"] == expected
@pytest.mark.parametrize(
("location", "expected"),
[
({"lat": 35.0, "lng": 139.0}, "precise"),
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
({"lat": 35.0}, "unknown"),
({"lng": 139.0}, "unknown"),
({"lat": "invalid", "lng": 139.0}, "unknown"),
],
)
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
from app.services.epoch_location import assess_epoch_location
assert assess_epoch_location(location).precision == expected
def test_epoch_anonymous_rows_are_not_geocoded():
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
assert row["latitude"] is None and row["longitude"] is None
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
@pytest.mark.parametrize(
"payload",
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
)
def test_epoch_invalid_payload_fails_closed(payload):
with pytest.raises(ValueError):
EpochAIGPUCollector().parse_response(payload)
def test_epoch_duplicate_identity_rejected():
payload = epoch_csv()
with pytest.raises(ValueError, match="identity collision"):
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
assert len(rows) == 2
assert rows[0]["source_id"] != rows[1]["source_id"]
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
def test_top500_units_and_stable_identity():
collector = TOP500Collector()
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
first = collector.parse_response(
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
)[0]
second = collector.parse_response(
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
)[0]
assert first["source_id"] == second["source_id"] == "top500_system_123"
assert float(first["value"]) == 1.5e15
assert first["metadata"]["rpeak_flops"] == 2e15
assert first["reference_date"] == "2025-11-01"
assert second["reference_date"] == "2026-06-01"
with pytest.raises(ValueError, match="date missing"):
collector.parse_response(html.format(rank=1))
@pytest.mark.asyncio
async def test_large_source_loss_does_not_create_snapshot():
db = AsyncMock()
db.scalar.return_value = 500
db.add = MagicMock()
with pytest.raises(ValueError, match="40%"):
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
db.add.assert_not_called()
db.commit.assert_not_awaited()
def test_history_missing_years_and_denominators():
global_data = compute_history(["CN", "US"])
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
bilateral = compute_history(["CN", "US"], "bilateral_share")
assert sum(bilateral["compute"][3]["values"].values()) == 100
assert bilateral["denominator"] == "US+CN"
absolute = compute_history(["US", "CN"], "absolute")
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
assert global_data["innovation"][-2]["value"] == 17.95
assert global_data["innovation"][-1]["value"] is None
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
@pytest.mark.asyncio
async def test_snapshot_query_uses_observation_cutoff_not_location():
query = await sample_query(AsyncMock(), ["CN"], 2023)
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "2024-01-01" in sql
assert "row_number() OVER (PARTITION BY" in sql
assert "compute_contract" in sql and "中国" in sql
assert "latitude" not in sql and "longitude" not in sql
def test_country_input_is_bounded_and_deduplicated():
assert selected_countries("CN,CN,US") == ["CN", "US"]
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
with pytest.raises(HTTPException):
selected_countries("invalid")
with pytest.raises(HTTPException, match="overlapping"):
selected_countries("EUROPE,DE")
def test_catalog_supports_global_participants_without_inventing_history():
from app.services.compute_catalog import (
CATALOG,
COMPARISON_GROUPS,
COUNTRIES,
source_country_names,
source_country_participants,
)
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
{entry["code"] for entry in CATALOG}
)
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
owners = source_country_names(["CN"])
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
assert [entry["kind"] for entry in CATALOG[:5]] == [
"country",
"country",
"region",
"region",
"region",
]
group_members = [
member
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
]
assert len(group_members) == len(set(group_members))
assert set(group_members) == set(COUNTRIES)
data = compute_history(["JP", "KR"])
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
assert all(row["values"]["KR"] is None for row in data["compute"])
assert all(row["value"] is None for row in data["innovation"])
assert len(data["catalog"]) > 30
def test_region_history_requires_complete_same_year_member_coverage():
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
assert row_2023["values"]["CN"] == 31
assert row_2023["values"]["US"] == 41
assert row_2023["values"]["EUROPE"] is None
assert row_2023["values"]["APAC"] is None
assert row_2023["values"]["OTHER"] is None
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
def test_anonymous_records_never_enter_the_location_repair_queue():
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.models.collected_data import CollectedData
record = CollectedData(
source="epoch_ai_gpu",
source_id="anonymous",
name="Anonymous",
extra_data={"compute_contract": "v1", "anonymous": True},
)
result = convert_compute_centers_to_geojson([record])
assert result["features"] == []
assert result["unresolved"] == []
assert result["withheld_count"] == 1
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
absolute = compute_history(["CN"], "absolute")
observed = {row["year"]: row for row in absolute["compute"]}
assert observed[2022]["values"]["CN"] == 302
assert observed[2022]["evidence_id"] == "caict-2023"
assert observed[2023]["evidence_id"] == "caict-2024"
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
for row in absolute["compute"]:
if row["evidence_id"]:
report = absolute["evidence"][row["evidence_id"]]
assert report["url"].startswith("https://www.caict.ac.cn/")

View File

@@ -0,0 +1,18 @@
from app.services.compute_center_verified_sites import get_verified_compute_site
def test_registry_requires_exact_facility_identity_not_operator_name():
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
assert site["city"] == "Yokohama"
assert site["precision"] == "site"
assert site["coordinate_basis"] == "official_access_map_center"
assert site["sources"]
site["latitude"] = 0
assert get_verified_compute_site(
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
)["latitude"] != 0

View File

@@ -1,4 +1,5 @@
import json
from pathlib import Path
import pytest
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
def write_geojson(path, name="Test"):
names = [name]
if name == "earth_admin0_boundaries":
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
path.write_text(
json.dumps(
{
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
"features": [
{
"type": "Feature",
"properties": {"name": name},
"properties": {
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
},
}
for item in names
],
}
),
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
config = repo / "config/earth-boundary-sources.local.json"
example = repo / "config/earth-boundary-sources.example.json"
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
path.mkdir(parents=True, exist_ok=True)
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
geometry.parent.mkdir(parents=True, exist_ok=True)
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
geometry.write_bytes(source_geometry.read_bytes())
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
return {
"repo": repo,
"config": config,
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
}
def write_fallback(path, policy_path, geometry_path):
policy = earth_boundaries._read_json(policy_path)
geometry = earth_boundaries._read_json(geometry_path)
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(policy),
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
}
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "legacy-geojson"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
assert status["provider"] == "missing"
assert status["fallback_available"] is False
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
}
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "pmtiles-mvt"
assert status["high_precision_ready"] is True
assert status["policy_publishable"] is False
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["high_precision_ready"] is False
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
assert status["high_precision_ready"] is True
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
assert paths["manifest"].exists()
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
assert manifest["povPolicy"]["applied"] is True
assert manifest["povPolicy"]["coverage"] == "partial"
assert manifest["povPolicy"]["publicationReady"] is False
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
by_rule = {
feature["properties"].get("PLANET_POLICY_RULE"): feature
for feature in hover["features"]
}
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
assert all(
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
for feature in hover["features"]
)
assert any(phase == "download" for _progress, phase, _message in progress_events)
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
assert removed.isdisjoint(DEFAULT_DATASOURCES)
for source in removed:
assert collector_registry.get(source) is None
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
repo = Path(__file__).resolve().parents[2]
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
payload = json.loads(path.read_text(encoding="utf-8"))
matches = [
feature for feature in payload["features"]
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
]
assert len(matches) == 1
jeju = matches[0]
assert jeju["properties"]["ISO_A3"] == "KOR"
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
ring = jeju["geometry"]["coordinates"][0]
assert len(ring) == 61
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)

View File

@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
assert vessel_update is not None
assert vessel_update["source"] == "vessel_position"
assert vessel_update["layers"] == ["vessels"]
assert vessel_update["refresh_strategy"] == "reload"
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
from app.services.earth_db_change_listener import PendingEarthDbChange
for source in ("aisstream_vessels", "barentswatch_vessels"):
update = build_earth_update_from_db_payload({
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
})
assert update["refresh_strategy"] == "delta"
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
pending.add({"operation": "UPDATE"})
pending.add({"operation": "DELETE"})
assert pending.refresh_strategy == "reload"
def test_build_earth_update_maps_interactable_delete_to_delta():

View File

@@ -0,0 +1,41 @@
import json
from pathlib import Path
import pytest
from app.core.earth_geography_policy import display_country, display_iso_a3
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
from app.core import earth_geography_policy as policy
policy._policy_country_by_code.cache_clear()
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
try:
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
policy.display_country("China")
finally:
policy._policy_country_by_code.cache_clear()
def test_display_country_groups_china_regions_and_kosovo() -> None:
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
assert display_country(source) == "中国"
assert display_iso_a3(source) == "CHN"
for source in ("Kosovo", "XK", "SRB"):
assert display_country(source) == "塞尔维亚"
assert display_iso_a3(source) == "SRB"
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
assert display_country("Japan") == "日本"
assert display_iso_a3("Japan") == "JPN"
assert display_country("unexpected non-country") is None
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy = json.loads(path.read_text(encoding="utf-8"))
for rule in policy["rules"]:
for code in rule.get("sourceIsoA3", []):
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))

View File

@@ -384,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
assert items[0].content_language == "zh-CN"
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
assert payload_en["display_title"] == ""
items[0].localizations["en-US"] = {
"title": "Chinese e-commerce platform reports quarterly growth",
"summary": "The platform said cross-border orders rose year over year.",
}
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
def test_default_news_sources_include_business_and_ecommerce_sources():

View File

@@ -0,0 +1,219 @@
from copy import deepcopy
import httpx
import pytest
from app.services import llm_model_catalog as discovery
from app.services import llm_provider_catalog as catalog
REAL_CLIENT = httpx.AsyncClient
def mock_http(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"provider,path,auth",
[
("minimax", "/anthropic/v1/models", "x-api-key"),
("anthropic", "/v1/models", "x-api-key"),
("openai", "/v1/models", "authorization"),
("deepseek", "/v1/models", "authorization"),
("alibaba", "/api/v1/models", "authorization"),
("moonshotai", "/v1/models", "authorization"),
("openrouter", "/api/v1/models", "authorization"),
("opencode-go", "/zen/go/v1/models", "authorization"),
("ollama", "/api/tags", "authorization"),
],
)
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
preset = catalog.get_fallback_llm_provider_preset(provider)
def handle(request):
assert request.url.path == path
assert request.url.host == httpx.URL(preset["base_url"]).host
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
if provider == "alibaba":
assert request.url.params["capabilities"] == "TG"
return httpx.Response(
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
)
if provider == "ollama":
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
return httpx.Response(200, json={"data": [{"id": "latest"}]})
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
assert result["base_url"] == preset["base_url"]
assert "test-key" not in str(result)
@pytest.mark.parametrize(
"provider,base,api,path",
[
(
"minimax",
"https://api.minimax.io/anthropic/v1/",
"anthropic-messages",
"/anthropic/v1/models",
),
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
(
"alibaba",
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
(
"alibaba",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
],
)
def test_urls_preserve_region_and_gateway(provider, base, api, path):
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
assert url.host == httpx.URL(base).host
assert url.path == path
@pytest.mark.asyncio
async def test_anthropic_pagination_and_release_order(monkeypatch):
seen = []
def handle(request):
seen.append(request.url.params.get("after_id"))
if len(seen) == 1:
return httpx.Response(
200,
json={
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
"has_more": True,
"last_id": "old",
},
)
return httpx.Response(
200,
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
assert seen == [None, "old"]
assert result["models"] == ["new", "old"]
@pytest.mark.asyncio
async def test_dashscope_pagination(monkeypatch):
def handle(request):
page = int(request.url.params["page_no"])
return httpx.Response(
200,
json={
"output": {
"total": 2,
"models": [
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
],
}
},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
assert result["models"] == ["page-2", "page-1"]
@pytest.mark.asyncio
@pytest.mark.parametrize(
"payload",
[
{},
{"data": []},
{"data": [None]},
{"data": [{}]},
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
],
)
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
with pytest.raises(discovery.LLMProviderCatalogError):
await catalog.refresh_llm_provider_preset("minimax", "test-key")
@pytest.mark.asyncio
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
assert len(result["models"]) == 140
assert result["models"][0] == "model-139"
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
@pytest.mark.asyncio
async def test_empty_ollama_catalog_is_valid(monkeypatch):
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
result = await catalog.refresh_llm_provider_preset("ollama")
assert result["models"] == []
assert result["model"] == ""
@pytest.mark.asyncio
async def test_opencode_documented_protocols(monkeypatch):
models = [
"minimax-m3",
"qwen3.8-max",
"gpt-5.6-luna",
"grok-4.6",
"muse-spark-1.3-contributor",
"kimi-k3",
]
mock_http(
monkeypatch,
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
)
result = await catalog.refresh_llm_provider_preset("opencode-go")
assert list(result["model_provider_apis"].values()) == [
"anthropic-messages",
"anthropic-messages",
"openai-responses",
"openai-responses",
"openai-responses",
"openai-completions",
]
@pytest.mark.asyncio
async def test_retry_transient_error_only(monkeypatch):
calls = []
def handle(request):
calls.append(request)
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
mock_http(monkeypatch, handle)
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 2
calls.clear()
def unauthorized(request):
calls.append(request)
return httpx.Response(401, text="secret-upstream-text")
mock_http(monkeypatch, unauthorized)
with pytest.raises(httpx.HTTPStatusError) as error:
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 1
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)

View File

@@ -0,0 +1,165 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.api.v1 import settings as api
from app.models.system_setting import SystemSetting
from app.services import llm_model_catalog as discovery
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
REAL_CLIENT = httpx.AsyncClient
@pytest.fixture
def settings_db(monkeypatch, tmp_path):
# Real SQL reads, commits and reloads; do not mock the settings helpers.
engine = create_engine("sqlite://")
SystemSetting.__table__.create(engine)
session = Session(engine)
db = SimpleNamespace(
execute=AsyncMock(side_effect=session.execute),
add=session.add,
commit=AsyncMock(side_effect=session.commit),
refresh=AsyncMock(side_effect=session.refresh),
)
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
yield db
session.close()
engine.dispose()
def use_upstream(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
async def test_fresh_database_lists_all_builtin_presets(settings_db):
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
assert len(result["data"]) == 9
assert all(row["models"] for row in result["data"])
assert "MiniMax-M3" in result["data"][0]["models"]
@pytest.mark.asyncio
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
runtime = {
"ai_provider": {
"default_provider": "minimax",
"service_token": "private-service-token",
"providers": {
"minimax": {
"base_url": "https://saved.test/anthropic",
"model": "old",
"api_key": "saved-secret",
}
},
}
}
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
before = await api.get_setting_payload(settings_db, "external_integrations")
seen = []
def handle(request):
seen.append(request)
assert str(request.url) == "https://draft.test/anthropic/v1/models"
assert request.headers["x-api-key"] == "draft-secret"
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
use_upstream(monkeypatch, handle)
draft = api.AIProviderIntegrationUpdate(
provider="minimax",
base_url="https://draft.test/anthropic",
model="unsaved-model",
api_key="draft-secret",
service_token="unsaved-service-token",
)
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
result = await api.get_ai_provider_presets(None, settings_db)
preset = next(row for row in result["data"] if row["provider"] == "minimax")
assert len(seen) == 1
assert preset["models"] == ["MiniMax-M3", "old"]
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
assert await api.get_setting_payload(settings_db, "external_integrations") == before
assert "secret" not in str(result)
assert "private-service-token" not in str(result)
assert draft.model == "unsaved-model"
stored = await settings_db.execute(
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
)
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
@pytest.mark.asyncio
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
settings_db, monkeypatch
):
await api.save_setting_payload(
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
)
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
commits = settings_db.commit.await_count
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
assert error.value.status_code == 502
assert "private-upstream-secret" not in error.value.detail
assert settings_db.commit.await_count == commits
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
"models"
] == ["saved-model"]
@pytest.mark.asyncio
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
def fail_request(request):
pytest.fail("must not query a private catalog without credentials")
use_upstream(monkeypatch, fail_request)
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("minimax", None, settings_db)
assert error.value.status_code == 400
assert "API Key" in error.value.detail
@pytest.mark.asyncio
@pytest.mark.parametrize(
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
)
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
result = await api._check_ai_provider_lightweight(
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
)
assert result["success"] is False
assert result["connected"] is False
assert "private-upstream-secret" not in result["message"]
@pytest.mark.asyncio
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
assert result["success"] is False
@pytest.mark.asyncio
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
use_upstream(
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
)
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight(
{**preset, "model": "future-model", "api_key": "test-key"}, 5
)
assert result["success"] is True
assert result["url"].endswith("/anthropic/v1/models")

View File

@@ -0,0 +1,141 @@
from pathlib import Path
import sys
from unittest.mock import patch
import httpx
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
with patch(
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
return_value={},
):
from aiprovider.provider_service import ProviderService
from aiprovider.schemas import SituationalAnalysisRequest
from app.api.v1.settings import _runtime_config_from_ai_payload
def test_custom_openai_gateway_keeps_its_configured_protocol():
config = _runtime_config_from_ai_payload(
{
"default_provider": "openai",
"providers": {
"openai": {
"base_url": "https://gateway.test/v1",
"provider_api": "openai-completions",
"model": "gpt-6-astra",
},
},
}
)["llm_config"]
service = ProviderService({**config, "api_key": "test-key"})
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"model,path",
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
)
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
llm = _runtime_config_from_ai_payload(
{
"default_provider": "opencode-go",
"providers": {
"opencode-go": {
"model": model,
"api_key": "test-key",
"model_provider_apis": {model: "openai-completions"},
}
},
}
)["llm_config"]
service = ProviderService(llm)
seen = []
async def post(**request):
seen.append(request)
assert request["path"] == path
assert request["request_body"]["model"] == model
if path == "/responses":
assert request["request_body"]["store"] is False
assert "max_tokens" not in request["request_body"]
return {
"output": [
{"type": "reasoning", "summary": [{"text": "reason"}]},
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
]
}
return {"content": [{"type": "text", "text": "OK"}]}
monkeypatch.setattr(service, "_post", post)
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
assert len(seen) == 1
assert result.content == "OK"
@pytest.mark.asyncio
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
seen.append(str(request.url))
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
service = ProviderService(
{
"provider": "minimax",
"provider_api": "anthropic-messages",
"base_url": base,
"api_key": "test-key",
"model": "MiniMax-M3",
}
)
result = await service.analyze(
SituationalAnalysisRequest(title="test", objective="reply OK")
)
assert result.content == "OK"
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
@pytest.mark.asyncio
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
import json
seen.append(request)
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
assert request.headers["user-agent"] == "Planet/1.0"
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
service = ProviderService(
{
"provider": "opencode-go",
"provider_api": "anthropic-messages",
"base_url": "https://opencode.ai/zen/go/v1",
"api_key": "test-key",
"model": "minimax-m3",
}
)
await service.analyze(
SituationalAnalysisRequest(
title="test",
objective="reply OK",
thinking={"type": "enabled"},
context={"session_id": session_id},
)
)
session_headers = [request.headers["x-opencode-session"] for request in seen]
assert session_headers[0] == session_headers[1]
assert session_headers[0] != session_headers[2]

View File

@@ -0,0 +1,160 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock
import pytest
from sqlalchemy import create_engine, event
from sqlalchemy.orm import Session
from app.db.session import Base
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.system_setting import SystemSetting
from app.models.task import CollectionTask
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.tv_catalog import get_tv_catalog_page
from app.services.tv_streams import normalize_tv_settings
class CatalogSession:
"""Execute the real catalog queries against an isolated SQLite database."""
def __init__(self, session):
self.session = session
async def execute(self, query):
return self.session.execute(query)
async def scalar(self, query):
return self.session.scalar(query)
async def scalars(self, query):
return self.session.scalars(query)
@pytest.fixture
def catalog_db():
engine = create_engine("sqlite:///:memory:")
@event.listens_for(engine, "connect")
def register_functions(connection, _record):
connection.create_function(
"concat_ws",
-1,
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
)
Base.metadata.create_all(
engine,
tables=[
CollectionTask.__table__,
DataSnapshot.__table__,
CollectedData.__table__,
SystemSetting.__table__,
],
)
with Session(engine) as session:
for index in range(135):
session.add(
CollectedData(
source="news_live_streams",
source_id=f"channel-{index:03}",
data_type="news_live_stream",
name=f"Channel {index:03}",
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
is_current=True,
is_valid=1,
extra_data={
"stream_url": "https://example.invalid/live.m3u8",
"region": "Canada",
},
)
)
session.flush()
yield CatalogSession(session)
engine.dispose()
@pytest.mark.asyncio
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
first = await get_tv_catalog_page(catalog_db, limit=50)
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
assert len(set(ids)) == first["source_count"] == 145
assert ids[-1] == "channel-134"
assert third["next_offset"] is None and not third["has_more"]
beyond = await get_tv_catalog_page(catalog_db, offset=200)
assert beyond["sources"] == [] and not beyond["has_more"]
@pytest.mark.asyncio
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
assert payload["total"] == 1 and payload["source_count"] == 145
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
@pytest.mark.asyncio
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
assert payload["selected_source"]["id"] == "channel-134"
assert "channel-134" not in [source["id"] for source in payload["sources"]]
assert payload["default_source_id"] == "aljazeera-mubasher"
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
@pytest.mark.asyncio
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
for name, values in [
("Disabled", {"extra_data": {"is_enabled": False}}),
("Historical", {"is_current": False}),
("Invalid", {"is_valid": 0}),
("Deleted", {"deleted_at": datetime.now(UTC)}),
("Replacement", {"source_id": "channel-134"}),
]:
record = dict(
source="news_live_streams",
source_id=name,
name=name,
data_type="news_live_stream",
is_current=True,
is_valid=1,
)
catalog_db.session.add(CollectedData(**{**record, **values}))
catalog_db.session.flush()
payload = await get_tv_catalog_page(catalog_db, q="replacement")
assert payload["source_count"] == 145
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
assert settings["default_source_id"] == "aljazeera-mubasher"
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
@pytest.mark.asyncio
@pytest.mark.parametrize(
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
)
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
monkeypatch, config, expected
):
collector = NewsLiveStreamsCollector()
channels = [
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
]
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
streams = [
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
for channel in channels
]
monkeypatch.setattr(
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
)
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
assert len(records) == expected
assert all(record["source_id"] != "sport" for record in records)

View File

@@ -0,0 +1,37 @@
from app.services.tv_streams import (
CCTV4_HOMEPAGE,
CCTV4_STREAM,
UNSUPPORTED_CCTV4_STREAM,
normalize_tv_settings,
normalize_tv_source,
)
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
settings = normalize_tv_settings(None)
assert settings["default_source_id"] == "aljazeera-mubasher"
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert cctv["source_type"] == "hls"
assert cctv["stream_url"] == CCTV4_STREAM
assert "美洲版" in cctv["name"]
assert cctv["use_proxy"] is False
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
def test_saved_legacy_default_is_corrected_without_mutating_input():
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
source = normalize_tv_source(saved)
assert source["source_type"] == "hls"
assert source["stream_url"] == CCTV4_STREAM
assert source["use_proxy"] is False
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert source["stream_url"] == custom["stream_url"]
assert source["source_type"] == "hls"
assert source["use_proxy"] is True
assert settings["auto_fallback"] is False

View File

@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
longitude=-90.05,
metadata={
"organization": "xAI",
"location": "3231 Riverport Rd, Memphis, TN 38109",
"gpu_type": "H100",
"gpu_count": 100000,
"value": "20000",
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
compute_center_locations.set_compute_center_location_cache({})
def test_epoch_country_level_coordinates_are_unresolved():
record = _build_record(
record_id=33,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Meta 100k",
country="United States",
city="",
latitude=39.381266,
longitude=-97.922211,
metadata={
"location": "United States of America",
"raw_country": "United States of America",
},
)
payload = convert_compute_centers_to_geojson([record])
assert payload["features"] == []
assert len(payload["unresolved"]) == 1
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
def test_epoch_city_coordinates_are_not_labeled_precise():
record = _build_record(
record_id=34,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Seoul Cluster",
country="South Korea",
city="",
latitude=37.5665,
longitude=126.978,
metadata={"location": "Seoul", "raw_country": "South Korea"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.is_resolved
assert result.location.location_precision == "city"
assert result.location.needs_confirmation is True
assert result.location.is_estimated is True
assert result.location.city is None
assert result.location.matched_location_name == "Seoul"
assert "Epoch Location: Seoul" in result.location.location_source_note
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
record = _build_record(
record_id=37,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Petrobras Fênix",
country="Brazil",
city="",
latitude=-22.909534,
longitude=-43.209934,
metadata={
"location": "Rio de Janeiro, Brazil",
"raw_country": "Brazil",
},
)
payload = convert_compute_centers_to_geojson([record])
feature = payload["features"][0]
assert feature["properties"]["location_precision"] == "city"
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
"location_source_note"
]
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-35": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-35",
"name": "Confirmed Site",
"latitude": 35.0,
"longitude": 139.0,
"precision": "site",
"confidence": 1.0,
"location_source": "official_facility_registry",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=35,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Japan Cluster",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
"raw_country": "Japan",
},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "stored_compute_center_location"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-36": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-36",
"name": "Legacy Country Point",
"latitude": 36.386493,
"longitude": 138.59223,
"precision": "precise",
"confidence": 1.0,
"location_source": "source_coordinates",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=36,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Legacy Japan Cluster",
country="Japan",
city="",
latitude=36.386493,
longitude=138.59223,
metadata={"location": "Japan", "raw_country": "Japan"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location is None
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_verified_sites_replace_epoch_country_center_coordinates():
specifications = [
(
"epoch_jamstec_zettascaler_2_0_gyoukou",
"Gyoukou 2.0",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_jamstec_zettascaler_2_2_gyoukou",
"Gyoukou 2.2",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_argonne_nl_sophia__formerly_theta_",
"Sophia",
"United States of America",
39.381266,
-97.922211,
[-87.984498648496, 41.718109383287235],
),
]
records = []
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
specifications, start=40
):
record = _build_record(
record_id=index,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name=name,
country=country,
city="",
latitude=latitude,
longitude=longitude,
metadata={"location": country, "raw_country": country},
)
record.source_id = source_id
records.append(record)
payload = convert_compute_centers_to_geojson(records)
assert payload["unresolved"] == []
assert len(payload["features"]) == 3
for feature, specification in zip(payload["features"], specifications):
assert feature["geometry"]["coordinates"] == specification[-1]
assert feature["properties"]["location_precision"] == "site"
assert feature["properties"]["location_source"] == "official_facility_registry"
assert "https://" in feature["properties"]["location_source_note"]
def test_verified_site_beats_epoch_city_level_source_coordinates():
record = _build_record(
record_id=44,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Gyoukou 2.2",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
)
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "official_facility_registry"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (
35.380924,
139.625008,
)
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-32": {
@@ -637,7 +873,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -741,7 +981,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]

View File

@@ -1,5 +1,9 @@
import pytest
import importlib
import json
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app.core.websocket.manager import ConnectionManager
from app.core.websocket.broadcaster import DataBroadcaster
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
broadcaster = DataBroadcaster()
async def load_current(keys):
assert keys == ["1"]
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
broadcaster.enqueue_vessel_update(
{
"source": "aisstream_vessels",
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
assert sent[0]["vessels"] == [
{
"mmsi": 1,
"lat": 59.1,
"lon": 10.1,
"source": "aisstream_vessels",
"lat": 60.0,
"lon": 11.0,
"source": "barentswatch_vessels",
"action": "upsert",
"created": None,
}
]
@pytest.mark.asyncio
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
manager = ConnectionManager()
socket = FakeWebSocket()
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
json.dumps(config)
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
await manager.broadcast_vessels({"vessels": vessels})
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
manager.unsubscribe(socket, ["vessels"])
await manager.broadcast_vessels({"vessels": vessels})
assert len(socket.sent) == 3
@pytest.mark.asyncio
async def test_global_vessel_removal_does_not_require_coordinates():
manager = ConnectionManager()
socket = FakeWebSocket()
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
websocket_module = importlib.import_module("app.api.v1.websocket")
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
app = FastAPI()
app.include_router(websocket_module.router)
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
assert socket.receive_json()["type"] == "connection_established"
socket.send_json({
"type": "subscribe",
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
})
response = socket.receive_json()
assert response["type"] == "subscription_confirmed"
assert response["data"]["vessels"]["scope"] == "global"
assert response["data"]["vessels"]["type"] == []
@pytest.mark.asyncio
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
broadcaster = DataBroadcaster()
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
async def fail_read(_keys):
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
raise RuntimeError("database unavailable")
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
with pytest.raises(RuntimeError, match="database unavailable"):
await broadcaster.flush_vessel_updates()
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61

View File

@@ -25,6 +25,25 @@
{
"id": "china-taiwan-penghu",
"name": "Taiwan and Penghu",
"sourceIsoA3": ["TWN"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-hong-kong",
"name": "Hong Kong",
"sourceIsoA3": ["HKG"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-macao",
"name": "Macao",
"sourceIsoA3": ["MAC"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
@@ -57,6 +76,7 @@
{
"id": "kosovo",
"name": "Kosovo",
"sourceIsoA3": ["KOS", "XKS"],
"action": "render_as_disputed_with_parent",
"parentIsoA3": "SRB",
"hoverIsoA3": "SRB",

View File

@@ -1,5 +1,3 @@
version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
@@ -32,7 +30,7 @@ services:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: planet_db
ports:
- "5432:5432"
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
@@ -45,7 +43,7 @@ services:
image: redis:7-alpine
container_name: planet_redis
ports:
- "6379:6379"
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
volumes:
- redis_data:/data
healthcheck:

View File

@@ -8,6 +8,225 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.75.2] — 2026-09-25
Released: 2026-09-25
### Highlights
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
### Fixed
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
---
## [0.75.1] — 2026-09-25
Released: 2026-09-25
### Highlights
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
### Added / Fixed / Improved
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
---
## [0.75.0] — 2026-09-23
Released: 2026-09-23
### Highlights
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
### Added / Fixed / Improved
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
---
## [0.74.7] — 2026-09-17
Released: 2026-09-17
### Highlights
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
### Added / Fixed / Improved
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
---
## [0.74.6] — 2026-09-16
Released: 2026-09-16
### Highlights
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
### Added / Fixed / Improved
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
---
## [0.74.5] — 2026-09-13
Released: 2026-09-13
### Highlights
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
### Added / Fixed / Improved
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
---
## [0.74.4] — 2026-09-13
Released: 2026-09-13
### Highlights
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
### Added / Fixed / Improved
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
---
## [0.74.3] — 2026-09-13
Released: 2026-09-13
### Highlights
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
### Added / Fixed / Improved
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
---
## [0.74.2] — 2026-07-01
Released: 2026-07-01
### Highlights
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`,删除重复维护的旧 Claude command 入口。
- 强化视觉证据规则:截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置,而不是跳过后猜测。
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback,同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
### Added / Fixed / Improved
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容,保留最新单一入口和 harness 验证说明。
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate,补齐路径解析、访问失败报告和 OCR fallback 边界。
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口,并更新文档受众计划中的旧路径引用。
---
## [0.74.1] — 2026-06-30
Released: 2026-06-30
### Highlights
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内,移除旧的全局“选择资产/上传”工具栏。
- 新增字段级图片拖拽反馈,拖到对应字段时直接提示将图片复制为 Logo 或标题图。
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式,并同步中英文使用手册、快速开始和控制台上下文文档。
### Added / Fixed / Improved
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
- `FieldGrid` 支持按字段注入自定义输入控件,同时复用统一草稿提交路径。
- 品牌上传拖拽态改为低饱和 tactile 配色,上传按钮保持蓝色轻立体样式,容器内上下/右侧留白对齐为 3px。
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
---
## [0.74.0] — 2026-06-30
Released: 2026-06-30
### Highlights
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口,减少英文界面中文残留。
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示,避免英文态裁切或错位。
- 将容易遗漏的 i18n 入口和视觉回归加入 harness,让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
### Added / Fixed / Improved
- 新增 Earth runtime i18n 入口,统一高清材质、启动状态、错误提示和图层状态文案来源。
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
- 更新控制台与 Earth 布局规则,保留 brand 尺寸语义,同时让标题、副标题和通知胶囊按内容完整显示。
- 扩展 frontend smoke 与 harness 文档,固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
- 更新 Earth 新闻本地化服务与测试,确保英文界面新闻内容不再回退中文 UI 文案。
---
## [0.73.0] — 2026-06-29
Released: 2026-06-29
### Highlights
- 新增前端统一 i18n 基础设施,让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
- 控制台侧边栏偏好面板接入语言与主题切换,并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
- 扩展 harness 与 smoke 覆盖,确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
### Added / Fixed / Improved
- 新增 `frontend/src/i18n/`,用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译,并覆盖动态计数字符串。
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership,避免首屏溢出和状态词裁切。
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则,记录语言迁移边界、状态指示器布局约束和一屏验证要求。
---
## [0.72.0] — 2026-06-29
Released: 2026-06-29
### Highlights
- 将 agent 入口收敛到单一 `AGENTS.md`,并让 harness 明确阻止小写入口再次分叉。
- 新增完整本地 harness 验证层,覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
- 扩展 Earth News 与控制台 smoke,确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
### Added / Fixed / Improved
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本,并将未跟踪 harness 设施纳入发布。
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出,改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档,明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
---
## [0.71.1] — 2026-06-26
Released: 2026-06-26

View File

@@ -9,7 +9,7 @@ or release workflows.
Existing project rules are authoritative:
1. `rules.md`
2. `agents.md`
2. `AGENTS.md`
3. Current implementation docs under `docs/technical/`
4. Existing scripts, especially `planet.sh`
5. Existing Gitea workflow files under `.gitea/workflows/`
@@ -18,6 +18,17 @@ When harness guidance conflicts with any of the above, keep the existing rule,
do not overwrite the existing workflow, and add a compatibility note here or in
`docs/harness-audit.md`.
For frontend or documentation audits, also read the Rules Coverage Evidence
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
static checks, Playwright smoke coverage, and remaining manual review areas, so
an agent can distinguish a proved harness pass from a rule that still needs
human-quality inspection.
When the user describes work with product words rather than module names, use
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
`地球`, `模型供应商`, and `发版` to the required rule modules.
## Starting Work
Recommended startup flow:
@@ -68,11 +79,25 @@ git diff --unified=0 HEAD -- <path>
## Harness Commands
Windows desktop launcher tests run separately on Windows with
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
They use temporary desktop folders and mocked restart commands; no UAC prompt
or real service restart is triggered. The WSL bridge regression suite,
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
disposable command substitutes. For a real entry-point check without restarting,
run the repository `planet.cmd --install-only`, then the generated desktop
launcher with the same flag. Installation and mocked checks do not prove a live
elevated restart; report that separately when it has not been exercised.
| Tier | Command | What It Does |
| --- | --- | --- |
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, optional Helm checks, and opt-in Docker image smoke builds. |
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
Docker image smoke builds are expensive and are off by default:
@@ -80,6 +105,106 @@ Docker image smoke builds are expensive and are off by default:
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
Frontend Playwright smoke runs by default in full validation after the frontend
build. It starts a local Vite preview and checks the `/` to Earth redirect,
public pages, unknown-route login fallback, protected admin route login
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
mocked API content, Docs detail page
language/theme/search interactions, every Docs catalog slug exposed by the
frontend/backend metadata, the Earth iframe entry point, login error handling,
register + email verification, password reset, standalone email verification,
and authenticated `super_admin` rendering for every admin route plus core
`section` deep links derived from the actual admin route and section config.
Authenticated admin
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
mobile passes also fail on global horizontal overflow so table/detail panels
must keep overflow ownership inside their own scroll regions. To enforce the
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
hard rendered check: the shell must resolve to the viewport height through the
root 100% height chain, `#root`/document/body must not gain vertical overflow,
and the desktop sidebar account/preferences area must remain inside the first
viewport while the nav owns any excess scrolling. The smoke also
derives the sidebar menu from the actual admin route manifest and clicks every
visible `super_admin` menu entry on both desktop and mobile viewports, then
exercises safe interaction paths for admin search, section tabs, the AI settings
shortcut, logs view switching, user dialog opening, and data distribution toggles.
It also exercises Earth News source testing, add/cancel source draft behavior,
and manual news group creation against mocked `/earth/news-*` APIs.
Documented AI and collector
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
`/collection-management?section=collector_credentials` are part of the rendered
smoke surface:
```bash
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
```
### Visual Evidence And Style Consistency
- Treat user-provided screenshots and images as primary visual evidence. If a
screenshot contradicts written text, inspect the image first and explicitly
call out the mismatch before deciding what to change.
- Path resolution is part of visual evidence handling. If a referenced
screenshot path cannot be opened, try reasonable local equivalents first:
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
thread attachments, repository files, and obvious local attachment/download
locations.
- If the image still cannot be found or opened, report the exact path/access
blocker instead of guessing. Do not infer image content from the filename, alt
text, surrounding prose, logs, or memory.
- OCR is acceptable evidence for text-only questions or non-multimodal
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
and rendering issues still require a real visual inspection or an explicit
"could not verify visually" note.
- Same-category UI surfaces must use one visual system per product area. Badges,
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
controls should reuse the shared component, shared token, or established CSS
owner for that area instead of introducing a page-local lookalike.
- `scripts/harness/frontend-rules-check.sh` warns when semantic
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
approved React and Earth CSS owner files. A warning means the reviewer should
either move the style into the shared owner or document why this is a genuinely
new visual family.
### Earth I18n Harness Rules
Earth i18n work must validate rendered behavior, not only static text lookup.
Agents often miss dynamic strings that are created after initial page load, so
the smoke treats these as first-class i18n surfaces:
- **Visible text and attributes**: translated checks must include `innerText`
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
buttons are user-facing copy, not implementation details.
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
BGP markers, compute centers, vessels, and news must render field labels,
status values, source tags, action buttons, and disabled/tooltips in the active
language.
- **English content safety**: English mode must not fall back to Chinese news
titles, summaries, feed names, measure words, or generic status labels. If no
English localization exists, hide the item or use a neutral English fallback.
- **Brand assets**: locale switching must update both text and image assets.
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
for English while keeping the same top-left layout and logo position.
- **Controls and state**: switch/segmented-control visuals must follow the real
checked/pressed state after both direct clicks and programmatic panel changes.
A control is not valid if the state changes but the thumb, active pill, or
`aria-*` state stays stale.
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
must enter through `earthMessage(...)` plus the centralized
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
or `earth:status` events.
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
overflow their panel. Prefer a slightly wider owning panel for important
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
translated tooltip.
Current frontend smoke explicitly covers the Earth English locale flow: brand
image swap, settings language controls, panel switch visual sync, English news
filtering, English detail-card text and tooltips, and TV default/source labels.
## Environment Requirements
Required for normal development:
@@ -89,6 +214,12 @@ Required for normal development:
- `bun` for frontend dependency and build execution
- Python resolved by `uv` from the root `pyproject.toml`
Harness command lookup first checks the current non-interactive `PATH`. If a
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
path. This avoids hardcoding a dotfile while still covering agent environments
that do not inherit the user's normal shell setup.
Required for full local stack operation:
- Docker and Docker Compose
@@ -99,9 +230,26 @@ Optional for delivery smoke:
- Docker daemon for image builds
- Helm for chart lint/template checks
If a required local tool is missing, do not install system software
automatically. Report the gap and point to `./planet.sh init` or
`scripts/bootstrap-dev.sh` as the existing bootstrap path.
For routine harness validation, do not install missing system software
automatically. Report the gap and point to the explicit bootstrap entry points.
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
This bootstrap behavior is intentional; do not invoke it merely to make harness
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
Docker bootstrap regression checks use isolated command stubs and never install
packages or modify the host daemon:
```bash
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
uv run --frozen --project . python scripts/harness/test_database_startup.py
```
Database startup regressions also run in quick-check. They cover Compose
reconciliation of existing containers, visible startup errors, published-port
checks, bounded recreation that preserves volumes, and the backend connection
gate before schema initialization. Their command stubs and driver mocks do not
modify the host Docker environment.
## What Agents Must Not Change Automatically
@@ -121,6 +269,14 @@ No automatic hooks are installed in this phase. Manual reminders:
- Run `scripts/harness/quick-check.sh` before handing off small changes.
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
- Run `scripts/harness/security-check.sh` after touching config, auth,
credentials, docs examples, or generated fixtures.
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
catch direct stdout/debugger calls before they reach runtime logs.
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
route, package-manager, debug-output, and UI rule warnings.
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
route changes.
- Add focused tests before modifying backend service behavior or frontend
workflows.
- For docs changes, run the checks listed in
@@ -136,6 +292,9 @@ No automatic hooks are installed in this phase. Manual reminders:
4. Make the smallest behavior-preserving or feature-scoped change.
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
6. Update relevant docs when behavior, workflow, or operations change.
7. For rendered frontend changes, verify the affected route with Playwright or
the full harness smoke, because `bun run build` alone does not prove page
usability.
### Bug Fix
@@ -149,7 +308,8 @@ No automatic hooks are installed in this phase. Manual reminders:
1. Read `docs/documentation-coverage-rules.md`.
2. Route docs by audience: UI users, operations, or second-party developers.
3. Keep Chinese and English public docs consistent when a public doc pair exists.
3. Keep Chinese and English technical docs paired by filename; public Docs also
need matching frontend/backend metadata when exposed in the product Docs UI.
4. Run the repository-specific docs checks that match the changed files.
### Release Or Delivery Change
@@ -161,7 +321,15 @@ release process.
## Implementation Notes
- `docs/harness-audit.md` records the discovery pass that led to this harness.
- `AGENTS.md` is a compatibility entry point for tools that expect the uppercase
filename. The existing `agents.md` file remains in place.
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
`agents.md` entry has been merged into it and should remain absent.
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
`docs/technical/{zh,en}/`.
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
with mocked API data. It proves route shells, auth guards, and primary admin
sections render, but it is not a replacement for feature-specific browser QA
against a real backend.
- Frontend smoke prints phase-level progress by default. Use
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
progress lines.

View File

@@ -25,12 +25,11 @@ compatibility note, not a replacement for existing rules or architecture docs.
| File | Status | Notes |
| --- | --- | --- |
| `agents.md` | Present | Existing root agent behavior guide. It references `rules.md` and `project_context.md`. |
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
| `AGENTS.md` | Added by harness | Compatibility entry point that points to existing rules and harness docs. |
## Existing CI Gates
@@ -69,13 +68,12 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
## Missing Or Unclear Areas
- README previously listed `AGENTS.md` in the project tree while only lowercase
`agents.md` existed. The harness adds uppercase `AGENTS.md` as a compatibility
wrapper and preserves `agents.md`.
- `project_context.md` includes older roadmap assumptions such as Celery, Kafka,
TimescaleDB, MinIO, and UE5 as active stack elements. The current README and
technical docs describe Web Earth, React admin, FastAPI, PostgreSQL/Redis, and
`aiprovider` as the active local development shape.
- The older lowercase `agents.md` entry has been merged into uppercase
`AGENTS.md` so coding agents and harness tools use one source of truth.
- `project_context.md` originally included older roadmap assumptions such as
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
harness pass updated it to separate active stack facts from future directions;
current code and technical docs still remain authoritative when details drift.
- No safe automatic hook system was already configured. This phase documents
manual reminders instead of adding hooks.
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
@@ -84,19 +82,80 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
| Conflict Or Tension | Resolution |
| --- | --- |
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Added a minimal uppercase compatibility entry and preserved the existing lowercase guide. |
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
## Rules Coverage Evidence
This matrix records how the current harness checks the `rules.md` modules that
matter for this frontend and documentation pass. "Automated" means the listed
command fails when the rule regresses. "Smoke" means the rendered product route
or interaction is opened with Playwright. "Manual" means the rule is still a
judgment call and must be inspected during review.
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
the user describes work with Chinese/product terms instead of module names. The
index is the routing layer; this table is the coverage/evidence layer.
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
| --- | --- | --- | --- |
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
## Harness Files Added
| File | Purpose |
| --- | --- |
| `AGENTS.md` | Compatibility agent entry point. |
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
| `CODEMAP.md` | High-level codebase map and validation references. |
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |

View File

@@ -22,6 +22,10 @@
当前重点入口:
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)

View File

@@ -0,0 +1,62 @@
# 控制台 i18n 接入计划
**状态**:基础设施已落地,大型业务页迁移继续进行
**创建日期**:2026-06-29
**核心目标**:把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系,让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
## 背景
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口,但语言状态只保存在 `docs-lang`,不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主,导致用户切到英文文档后,控制台仍是中文。
本计划把前端语言偏好收敛到 `planet-locale`,默认 `zh-CN`,支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口,通过前端映射与全局 locale 对齐。
## 设计决策
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层,避免长期维护自研插值、hook 和资源加载逻辑。
- 前端统一语言枚举为 `zh-CN` / `en-US`;Docs 请求继续转换为 `zh` / `en`,新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
- 语言偏好首版只保存在浏览器 `localStorage`,不新增后端用户设置字段。
- `docs-lang` 保留为兼容读取和写入项,让已访问过 Docs 的浏览器能平滑迁移。
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key;大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
## 分期
### P1:统一语言基础设施
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
- 在 `frontend/src/main.tsx` 里初始化 i18n,并同步 `document.documentElement.lang`。
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
### P2:高复用界面迁移
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
- 搜索索引按当前语言展示,同时保留中英文关键词以免降低可发现性。
- 用户管理页作为独立业务页示范,迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化,以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
### P3:大型业务页收敛
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
- 过渡期保留 legacy UI 翻译桥,只处理 admin/auth 容器里的精确静态文本和属性。
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
### P4:移除过渡桥
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时,删除 legacy UI 翻译桥。
- 增加 key 完整性检查,确保 `zh-CN` 和 `en-US` 资源结构一致。
## 验证
- `cd frontend && bun run build`
- `scripts/harness/frontend-rules-check.sh`
- `scripts/harness/docs-consistency-check.sh`
- `scripts/harness/quick-check.sh`
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
## 相关文件
- `frontend/src/i18n/`:统一 locale、资源和过渡桥。
- `frontend/src/pages/Docs/Docs.tsx`:Docs 语言状态改为读取全局 locale。
- `frontend/src/admin/components/layout/AdminLayout.tsx`:控制台侧边栏语言切换、导航和搜索文案。
- `frontend/src/admin/search/indexers.ts`:Admin 搜索目标本地化。
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`:当前实现上下文。

View File

@@ -0,0 +1,602 @@
# 算力与资源态势:展示、采集与 AI 迭代实施计划
状态:首版空间卡片原型实现中(2026-09-21)。已接入算力专题入口、45 个参与方目录、年份/口径选择、区域数据卡与来源、成功快照设施查询,以及 Epoch/TOP500 的首批质量修复。旧固定双图布局已移除。历史年度数据使用受控版本导入;各国完整指标覆盖、卡片图表、全榜获取、跨源设施合并、持久化指标管理、变化归因与 A4 AI 解读尚未完成,不视为全部 A1—A3 已验收。后续章节保留完整设计目标,当前实现以技术文档为准。2026-09-14 完成的口径核查仍是内容依据。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
## 0. 当前执行方向与范围
2026-09-21 展示方向修订:UI、地球材质、政治展示规则、地理资源对齐及性能门槛以[Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)为准。下文固定右侧国家面板、底部横贯双图等布局不再作为实施目标。采用从区域锚点引出连接线、卡片围绕地球自适应浮动的方向,先验证空间交互再完善卡片内容。数据来源、口径与证据要求继续有效。
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚:地图显示资源在哪里,图表说明规模和相对位置怎样变化,证据说明数字从哪里来,AI 帮助阅读变化。
采用“专题先交付,公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线,不是首版展示的前置条件;第 6 节总分保持实验性,不能为了首屏有大数字而生成不完整排名。
| 当前要做 | 当前完成形态 | 后续才扩展 |
| --- | --- | --- |
| 算力数据可信 | 修复既有采集,国家/设施/精度/年份不混用 | 大规模自动来源发现、完整证据图谱 |
| 全球分布与国家对比 | 现有算力中心地球 + 可比参与方 + 明确样本覆盖 | 制造、矿产、电力和贸易的跨主题关系 |
| 历史展示 | 两张核对后的图、时间选择、已知事件轨道 | 复杂预测、不确定性传播与可训练参数 |
| 证据与解释 | 来源、参考年、关键字段;基于数据快照的简短 AI 说明 | 自动补证、模型候选、回测、影子运行与提升 |
首版验收路径:打开算力专题 → 看全球公开设施分布 → 选中国和美国 → 同时看算力与创新历史 → 点一个变化或设施 → 查看出处及解释。此路径稳定后,才扩大评价维度。
## 1. 要交付什么
在智能星球现有算力中心图层上增加“算力与资源态势”模式,并在控制台提供对应的数据管理和模型实验工作区。核心问题是:各方拥有什么资源、资源如何形成可用能力、相对位置如何变化、变化有何证据,以及未来什么约束可能改变格局。
首批比较中国和美国;数据结构从第一天支持多方。随后纳入欧洲、日本、韩国、印度及其他数据合格的参与方。地区汇总与成员国不得重复计入全球分母。所有国家和地区字段使用现有 `normalize_country()` 和规范字典,不以网页原文或 AI 判断覆盖项目标签。
交付三个相互连接的产品能力:
| 能力 | 用户获得的结果 | 首版边界 |
| --- | --- | --- |
| 历史观测 | 算力绝对规模、全球份额、创新指数与分差、设施时间线 | 使用可验证观测;缺失年份可见,不插值冒充事实 |
| 变化解释 | 哪些设施、资源、数据修订造成指标变化,每项贡献和证据 | 数学贡献可复算;原因解释区分事实、假设与未解问题 |
| 模型改进 | AI 发现缺口、补证、提出改进,经过验证产生新版本 | 固定工作流先行;不能由大模型直接决定国家分数 |
第一版先交付可信的事实与回放,不以总分完整作为上线前提。完整创新指数分项、有效算力参数和项目交付数据不足时,继续展示已完成的观测能力,综合评分显示“数据不足”,不生成示意国家排名。
不在第一版承诺全球所有设施的完整清单、精确的国家真实算力、每日变化的国家创新能力、可验证的国家综合实力真值,或自动预测某年必然追平。
## 2. 当前项目基础与必须补齐的边界
以下结论来自当前文件检查,未对生产数据库、凭证可用性或线上采集结果做完整审计。
| 已有入口 | 已确认能力 | 本专题需要补齐 |
| --- | --- | --- |
| [采集基类](../../backend/app/services/collectors/base.py)、[数据作业](../../backend/app/services/data_jobs.py) | 采集、转换、任务状态、快照、取消与重试基础 | 文档型数据源、质量门槛、领域规范化、失败保留正式快照 |
| [Epoch 采集器](../../backend/app/services/collectors/epoch_ai.py) | GPU 集群采集入口 | 改读公开 CSV;去掉解析失败返回示例记录;单位、精度、状态和时间字段校正 |
| [TOP500 采集器](../../backend/app/services/collectors/top500.py) | 榜单与详情解析 | 去掉示例回退、固定参考日期、排名作为身份;按当期表头解析单位 |
| [采集记录](../../backend/app/models/collected_data.py)、[快照](../../backend/app/models/data_snapshot.py) | `entity_key`、快照关联、前一记录和变化摘要 | 原始证据版本、发表时间、有效时间、指标注册和可重放的观测集合 |
| [算力位置服务](../../backend/app/services/compute_center_locations.py)、[GeoJSON](../../backend/app/api/v1/visualization.py) | 设施定位、位置置信信息、未定位记录 | 国家统计与地图可定位集合分离;匿名设施不能猜坐标 |
| [算力中心渲染](../../frontend/public/earth/js/compute-centers.js)、[Earth 壳](../../frontend/src/pages/Earth/Earth.tsx) | Three.js 交互层;Earth 由 iframe 加载独立页面 | 历史时点、国家比较、双图、关系层和解释面板 |
| [图层适配器](../../backend/app/services/earth_layer_adapters.py)、[数据库监听](../../backend/app/services/earth_db_change_listener.py) | 通过 PostgreSQL outbox 驱动缓存与 `earth_updates` | 正式指标快照发布事件;草稿与实验结果不改变默认地球 |
| [AI Client](../../backend/app/services/ai_client.py)、[提示词注册](../../backend/app/ai_tasks/prompts.py) | 全局 provider 配置、模型调用、任务提示词及运维覆盖 | 算力专题的固定工作流、结构化输出验证、预算与审计 |
| [搜索工具](../../backend/app/services/ai_tools/web_search.py)、[网页取证](../../backend/app/services/ai_tools/web_fetch.py) | 搜索和基础 HTML 正文提取 | PDF/表格提取、引用位置、下载时大小限制、URL 与重定向的访问限制 |
| [证据辅助函数](../../backend/app/services/ai_tools/evidence_store.py) | 摘要、去重和内容哈希规范化 | 该文件不是持久化证据库,需补建不可变原文存储和证据记录 |
沿用[数据作业与 Outbox 架构](../technical/zh/data-job-earth-sync-architecture.md)及[轻量 Agent 编排计划](agents-light-orchestrator-websearch-plan.md):PostgreSQL 负责可靠账本,Redis 负责缓存;不另起 Kafka、Celery 或另一个独立调度真相源。`aiprovider` 保持协议适配,业务取证、估计、工具权限和模型发布全部由后端负责。
当前 `JobType` 只有采集、清理和 Earth 刷新。新增分析、实验工作流需要显式扩展任务类型、worker 分发和取消策略,不能把现有采集队列描述成已经支持通用 Agent。
## 3. 页面与地球怎样展示
### 3.1 默认工作区
保留 `/earth` 主入口,在现有算力中心工具中增加专题模式,避免增加一个与地球无关联的新大屏。进入后使用以下概念布局;这是规划线框,不是现有页面截图。
```text
┌ 时间/参考期 · 中国/美国/其他参与方 · 指标口径 · 观测/情景 ┐
│ │
│ 3D 地球:设施、资源、已证实关系 国家对比面板 │
│ 绝对量/份额 │
│ 变化贡献 │
│ 证据与缺口 │
├────────────── 历史时间轴 / 事件轨道 ────────────────────┤
│ 算力规模与份额图 综合创新指数与分差图 │
└───────────────────────────────────────────────────────┘
```
桌面默认同时显示两张历史图,用相同国家配色,但各自保留单位和实际观测年份。图表高度优先保证刻度、来源标识与数据点可读,不能通过压缩字体维持布局。较窄屏幕把图表移入可切换的详情区,国家和设施详情使用抽屉;不叠加多个遮挡地图的浮层。
沿用项目一屏高度链和主题。桌面在 1366×768、1920×1080,移动端在约 390×844 验证,另测 125%/150% 浏览器缩放。页面根与中间 flex 容器分别正确设置高度和 `min-height: 0`,仅详情、长表和证据正文内部滚动。
### 3.2 视图职责
| 视图 | 主内容 | 点击或切换后的行为 |
| --- | --- | --- |
| 国家总览 | 可比算力绝对量、份额、子分、实际参考期;总分合格才出现 | 选中参与方,定位地球并联动两图;不重设全局评分尺度 |
| 算力趋势 | 绝对规模、全球份额、双边份额三种独立视图 | 同一来源口径内切换;不把百分比轴和 FLOP/s 轴混合 |
| 创新趋势 | 各方指数;只有分差时只显示中美分差 | 切换年度,查看分项与报告版本,不能反推缺失的单国得分 |
| 变化账单 | 新投产、扩建、退役、效率变化、资料补录、方法修订 | 点选贡献定位资产;逐步展示输入、计算和证据 |
| 设施详情 | 设备、精度、投产状态、功率、运营方、地点精度、历史版本 | 展开相关公告、设备来源及不确定字段 |
| 资源关系 | 已证实的芯片供给、所有权、电力接入或网络连接 | 选一类关系再显示,不能让所有关系默认铺满地球 |
| 情景工作区 | 并网推迟、交付变化、效率假设及结果区间 | 独立 scenario ID;退出恢复正式快照 |
| 控制台研究区 | 数据缺口、待核验证据、采集任务、模型实验和版本 | 使用已有采集与 AI 设置入口;Playground 仍用于调试 |
### 3.3 两张图进入产品的明确规则
总算力图以核对后的报告观测为起点:2020 年美国/中国为 36%/31%,2021 年 34%/33%,2022 年 34%/33%,2023 年 41%/31%。原始来源和报告版本见第 15 节。2024—2025 年暂不填值;2026 年 42%/33% 只保存为待核实主张,不能接到正式历史曲线上。
创新图使用报道明确披露的 2023、2024、2025 年分差 22.02、19.96、17.95 分。报告发表于 2026 年,必须区分报告年份与参考年份。完整分项、年度得分和方法修订仍需采集。图中是综合创新指数,不是纯算力或纯产出。
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读,不进入计算;不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度,不让刷新网页造成“国家实力实时跳动”。
### 3.3a 当前数据合同(2026-09-22)
默认主指标是全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。CAICT 报告核查目前使用官方 PDF 索引文本;本地 PDF 下载未成功,因此不宣称已肉眼核验 PDF 页面。绝对量合同记录 2022 年 302(证据版本 `caict2023`)和 2023 年 435(证据版本 `caict2024`);2024 年报告正式书名为《先进计算暨算力发展指数蓝皮书(2024年)》。官方 URL 已替换为四个镜像来源。
日本全球份额观测为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 `APAC` 与 `OTHER` 不能直接套用报告中的 Japan 或“其他”;`EUROPE` 成员集合尚未与报告口径对齐,仍返回缺失。双边份额分母固定为 US + China,新加入日本不改变分母。主要指标缺失时可显示最近有报告的年份(当前可到 2023),而不是留空为 2026;用户明确选择 2026 时仍显示缺失,不回填最近值。不可比性尚未确认,不展示未经确认的同比变化;覆盖数据保留在辅助区并带 count。
数据 API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现,用户对该问法的选择待后续确认;当前合同不提前新增该模式或声称完成 dataset 计算。
默认“历史回顾”允许查看最新证据重述后的过去;“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时,必须显示“最新观测:2023”,或在严格同年模式显示缺失。
### 3.4 地球编码与交互
- 节点面积按同口径能力编码;跨度过大时可切换对数档位并明示图例。国家颜色只用于身份,不能把颜色本身当作领先或落后。
- 已投产用实心,建设或情景用空心/虚线;置信不足使用独立标记,不能把低置信透明度误读为小算力。
- 来源只到国家或区域时,保留在国家/区域统计和未定位列表;不画在首都,不把行政区域中心当作设施坐标。
- 国家总量与可见设施合计可以不同,分别显示覆盖范围。不得只统计有坐标的节点,也不得把宏观缺口按现有节点比例摊分。
- Epoch 当前公开版本对中国部分集群匿名化并取整;匿名记录只能按允许的粒度使用。新增官方公告可作为独立证据,不能依据匿名数值猜出设施身份或制造地点。
- 供应、所有权和网络关系采用不同图例,显示关系来源与有效期;商业供应关系不等同于实际运输路线。
- 全国可汇总吞吐与最大单集群能力分别展示;地理分散设备不能合并成一个训练集群。
- 时间回放使用稳定实体 ID 更新已有节点。隐藏图层、变更时点或情景时同步清理 hover、locked、tooltip 和 selection;请求过期结果不得覆盖新时点。
- 使用现有 Three.js Interactable、图标和实例化能力。新增壳层必须遵循现有深度顺序,在远距约 50% 缩放检查闪烁;不另建悬浮球面掩盖深度问题。
## 4. 数据需要补充采集什么
### 4.1 P0:决定首版可信度的数据
“已核对入口”只表示资料或文档可查,不表示接口凭证、完整历史和生产连通性已经验证。以下频率是 Planet 建议检查频率,不是来源承诺的更新频率。
| ID / 优先级 | 数据与来源 | 必需字段/粒度 | 采集方式与频率 | 当前缺口与采用条件 |
| --- | --- | --- | --- | --- |
| D01 / P0 | 信通院历年算力报告 | 国家×参考年×口径;绝对规模、全球总量/份额、单位、精度、估算方法、版本 | 已取得报告先受控导入;月度检查新版;PDF 表格提取后核验 | 已核对 2020—2023 份额及部分绝对量;补 2024 以后和其他国家;跨版本方法改变时断开序列 |
| D02 / P0 | Epoch GPU Clusters | 集群/版本;设备数、型号、16/8/32-bit 性能、所在地、状态、投产/退役时间、功率、扩建关系 | 官方公开 CSV,每日条件抓取,按内容哈希去重 | 改造现有 HTML 解析;处理匿名、取整、缺坐标和覆盖偏差;不能声称完整全国清单 |
| D03 / P0 | TOP500 / 可选 Green500 | 稳定系统 ID×榜单期;Rmax/Rpeak、FP64、功率、机构、名次、列表日期 | 每周检查新榜单;按发布期导入历史与详情 | 修复 rank 作为 source ID、固定日期和单位问题;榜单消失不等于退役;不当成 AI 总算力 |
| D04 / P0 | 全球 AI 创新指数原报告及发布材料 | 国家×参考年×指标;单国总分、分项、权重、标准化、成员范围、方法版本 | 文档导入+月度更新检查 | 当前只具备部分新闻转述分差;优先获得分项和年度可比性证据;不能用差值合成各国历史得分 |
| D05 / P0 | 已有位置维表与公开设施公告 | 实体别名、运营方、国家、地点、精度、采用依据、有效期 | 复用位置管线,新增/变化时触发 | 全国汇总独立于定位成败;不把同址不同分期误合并,也不把扩建当第二个全量集群 |
首版最低闭环:D01 和 D04 的已验证历史可先支撑双图,D02/D03/D05 支撑设施地图;两者不互相伪装为同一覆盖范围。若 D04 分项不可取得,创新总分仍作为独立观测,O 分与综合 S 不发布。
首版可以先受控导入已核实的年度数据,不必先完成整套 PDF 自动解析和报告发现。D04 原报告分项继续作为补数任务,不阻塞已经核对的创新分差图;原始摘录、出处、年份和未核实范围必须随数据保存。
### 4.2 P1/P2:有效能力、增长与资源关系
| ID / 优先级 | 补充数据与候选来源 | 粒度与核心字段 | 方式/频率 | 对模型的作用及限制 |
| --- | --- | --- | --- | --- |
| D06 / P1 | Epoch 芯片销售、所有权、硬件数据;厂商规格 | 芯片型号×时间;规格、销售/交付、持有主体、数量、性能换算、估计区间 | 官方数据文件与规格文档;周查/季度快照 | 校验硬件供给;设计者、所有者、实际所在地与使用权分开;不能把出货和已部署重复加总 |
| D07 / P1 | MLPerf Training / Inference、可复现厂商测试 | 配置×任务×软件版本;模型、质量门槛、batch、上下文、时延、吞吐、能耗、可用状态 | 官方结果导入;月查新版本 | 校准特定任务的性能系数;不同 benchmark 版本/质量/负载不直连;实验室成绩不等同生产利用率 |
| D08 / P1 | IEA、电力统计机构、EIA、并网与运营方资料 | 国家/地区/项目×时点;发电、用电、接入 MW、PUE、并网日期、可用负荷 | 国家统计月/季;报告年;项目公告日查 | 电力作为设施约束;国家发电量、用电 TWh 不能直接转换机房可用 MW;中国与美国地方数据可得性不同 |
| D09 / P1 | 项目运营方、许可部门、官方采购和工程进展 | 项目×阶段;公告、开工、设备交付、并网、验收、部分投产、撤销、金额与范围 | 已注册来源每日检查,重要项目每周核验 | 建立投产与延期标签;最终失败项目也保留,避免只收集成功项目;单纯新闻未更新不判定失败 |
| D10 / P1 | 公司投资者关系、SEC EDGAR、境内及其他市场法定披露 | 法人/分部×财季;CapEx、已付款/指引、租赁、合同、地域/业务归属 | 官方 API/报告;日查新披露,按季归档 | 用于资金与项目约束;总 CapEx 不是 AI 投资;合并报表、融资租赁、设备与机房成本须防重复 |
| D11 / P1 | 创新报告分项、公开模型评测、OpenAlex;HF 作为辅助 | 机构/国家×年/月;研发成果、质量归一化产出、可比模型能力、应用指标 | 原报告年;评测周;研究元数据月 | 重建不含重复资源项的 O;合作成果使用明确分摊;不从作者姓名推断国籍;下载量不代表实际使用或产业产出 |
| D12 / P2 | 晶圆制造、先进封装、HBM、关键设备的正式披露 | 工厂/供应方×工艺/产品×季度;已投产能力、良率范围、交付、客户与依赖 | 公司/监管原件;月查、季度快照 | 建立瓶颈和替代关系;晶圆数不能无依据换成 GPU 数;总行业份额不能归给某集群 |
| D13 / P2 | USGS、各国统计、UN Comtrade | 商品×国家×年/月;产量、储量、加工量、贸易流、HS 版本和单位 | 年度报告、月/年贸易数据 | 扩展矿产与资源依赖;储量、开采和加工分开;宽泛服务器税号不能反推高端芯片数量 |
| D14 / P2 | 既有海缆、PeeringDB、BGP 与已证实连接资料 | 设施/网络/关系×有效期;容量声明、连接、依赖、故障 | 复用现有产品,按其真实更新节奏 | 提供连通性和韧性背景;BGP 可见性不是带宽,海缆容量不是集群内部互联性能 |
EIA、SEC、Epoch、MLCommons、IEA、USGS 已核对资料入口;UN Comtrade 门户可见,但本次未验证可调用 API,接入要单独完成认证、额度、字段和历史测试。OpenAlex 需在接入时按最新访问规则配置,不假定无限制匿名访问。所有付费、登录和受限数据只列为可选增强,不作为首版隐藏前提。
### 4.3 首批采购与补数优先顺序
1. 先完成现有 Epoch/TOP500 的真实性与时间口径修复,避免新增来源扩大错误。
2. 导入信通院历史、创新分差和原始出处;追索创新原报告分项及最新版总算力数据。
3. 建立中美重点设施及项目阶段台账,并记录未知量。重点设施用于高影响解释,不冒充全国抽样无偏估计。
4. 用官方规格与 MLPerf 建立少量明确工作负载的换算表;其余硬件维持理论值或范围。
5. 补项目级并网、交付、资金与退役记录,积累可校准的真实标签。
6. 最后扩展制造、HBM、矿产与多方关系;每个新主题必须有对应指标和展示消费者,不只收集而不使用。
## 5. 采集到正式数据的流水线
```mermaid
flowchart LR
A[注册的数据源] --> B[原文与内容哈希]
B --> C[确定性解析或 AI 提取候选]
C --> D[单位/时间/实体/引用校验]
D --> E[版本化有效观测]
D --> F[冲突与待核实队列]
E --> G[确定性指标计算]
G --> H[正式结果快照]
H --> I[Outbox 与缓存更新]
I --> J[地球/双图/变化账单]
```
### 5.1 数据源合同
每个数据源保存来源机构、域名与下载入口、文档类型、凭证配置引用、检查频率、预期粒度、单位与精度、更新时间语义、空结果语义、历史范围、使用许可、维护负责人和解析器版本。授权文件或数据保存在运行数据存储,不提交到仓库;公开 UI 只展示允许公开的摘录或链接。
API/CSV 优先;HTML/PDF 采用固定解析;必须用 AI 时先产生候选,核验后保存确定性映射。扫描件 OCR 必须保留页号、表格区域和原图,数值不能只靠模型复述。下载器在流式读取时限制实际字节量,不能下载完整文件后才截断;按域名限速,遵守来源规则。
### 5.2 事实键与时间
观测键至少包括实体、指标、单位、精度、工作负载、地理范围、归属口径、参考期及来源版本。历史至少区分:
| 时间 | 含义 |
| --- | --- |
| `reference_start/end` 或 `effective_at` | 数据描述的年度、季度或设施事件发生时间 |
| `published_at` | 资料何时首次公开;不知道就保持未知 |
| `retrieved_at` | Planet 何时取得该版本 |
| `recorded_at` / `superseded_at` | 系统何时采用、替换该观测 |
抓取日期不代替投产日期;报告年份不代替参考年份;机构估计、实测、公告、模型估计和人工情景分别标识。只有年份时保留年份精度,不能伪造 1 月 1 日投产;回放按时间范围或“当年”事件显示。
### 5.3 合并与质量门槛
- 稳定身份优先使用官方系统 ID、项目 ID、法人 ID 和可信来源链接。TOP500 排名只作为观测属性。自动实体合并必须可撤销,保存别名和合并理由。
- 同一集群的全量扩建公告、一期/二期、重复转载与另一个数据集的同一设备只计一次。芯片交付和机房投产分别保留,不能合并为两个已投产资产。
- 国家级统计、企业所有权统计、设施地理统计是不同视图。国家公司在海外租用算力不得同时作为两国境内部署能力;全球成员集合固定且版本化。
- 转换单位前先核验浮点精度、稀疏/稠密、峰值/实测和时间量纲。FLOP 与 FLOP/s、TWh 与 MW 均不得互换。
- 解析为空、字段突然缺失、总量异常下降或源结构改变,进入失败/隔离状态,保留上次正式结果。缺榜、缺行或匿名化不直接触发实体退役。
- 多条转载归到同一 `source_family`;媒体数量不构成多份独立证据。冲突先检查口径,再保存多个主张及采用规则,不能简单取平均。
- 缺坐标不影响已知国家统计;未知国家不强行分配。数据覆盖率可描述已登记实体或必填字段的完整率;无法知道总体时不声称“覆盖全球百分之多少”。
- 一位有效数字等取整值保留取整范围;这是观测精度范围,不冒充统计置信区间。
首批历史导入先双人或规则加人工抽查核验高影响记录,保存一套回归样本。后续同版本确定性解析通过固定门槛可自动采用;新来源、新语义或无法解释的大幅变化进入候选队列。
## 6. 数学模型如何工作
### 6.1 分清事实、估计和评价
模型分三层:观测层保存来源说了什么;估计层回答可用能力、交付时间和不确定性;评价层按明确价值取向组合指标。国家综合实力没有一个可直接取得的标签真值,因此不能声称“AI 找到了客观最优的国家评分权重”。
保留 v2 的模型草案:
\[
S_{c,t}=0.40H_{c,t}+0.30O_{c,t}+0.20G_{c,t}+0.10R_{c,t}
\]
H 是资源能力,O 是不重复的创新与应用表现,G 是扩张潜力,R 是韧性;四项均为 0—100 的版本化子分。S 称为“综合态势分”,包含未来潜力,不能称作已投产算力、胜率或国家实力百分比。权重是默认设计参数,需通过敏感性分析和评价目的评审后冻结。
### 6.2 资源能力 H 与两类算力模式
总算力模式使用 D01 的同口径绝对规模;AI 训练模式使用明确工作负载的有效能力;推理模式另选固定模型、质量、上下文与时延要求。不同模式不共用没有物理依据的换算,也不比较彼此总分。
\[
C^{eff}_{c,t,w}=\sum_{j\in deployed(c,t)} C^{peak}_{j,t,p}\,u_{j,t,w}
\]
`p` 是固定精度,`w` 是参考工作负载;`u` 由可比测试、已知运行约束或经校准的估计支持。没有依据时只发布理论能力,或者明示区间;不能统一假设某国芯片只有另一个国家的固定折扣。
内存、互联、软件和电力会影响同一个有效利用系数,不得分别随意乘一串折扣重复惩罚。若已知场地供电上限,可用场地总功率除以 PUE 约束 IT 负荷,再按同配置效率估计可运行容量;已在利用系数中计入的限电不再扣一次。总吞吐与可用于单次大训练的最大连续集群另列。
国家宏观估计和公开集群样本并列校验,不直接把二者相加。若 H 只能覆盖已观测设施,分数标题和参与方排名必须明确限定为这个样本,不外推为全国真实能力。
### 6.3 创新与应用 O
完整创新指数 I 单独显示。取得 D04 分项后,建立指标归属表,逐一标注资源、扩张、韧性或创新;只把不重叠的研发质量、模型/科研成果与应用表现用于 O。不是简单执行“创新总分减去算力分”,因为不同分项的权重和标准化可能不同。
备用指标可来自 D11,但新增指标意味着新定义和新版本,不能偷偷替换原报告。论文数量须去重、处理合作分摊及引用年龄;模型结果要冻结评测版本;应用指标明确平台样本。专利、下载、融资和模型排行榜均不是彼此的等价替代。
\[
D^I_t=I_{US,t}-I_{CN,t},\qquad v^I_t=(D^I_{t-1}-D^I_t)/\Delta years
\]
2023—2025 年披露分差缩小 4.07 分,约为起点分差的 18.5%;不能解释为中国能力提高 18.5%,也不能据此推断算法效率提升。只有差值不能恢复各国绝对得分。跨年方法不一致时分段显示,不能外推追平时间。
### 6.4 扩张潜力 G
\[
G^{raw}_{c,t,h}=\sum_{j\in pipeline(c,t)}P(T_j\leq t+h\mid X_{j,\leq t})\,\Delta C^{eff}_j
\]
默认 `h=12 个月`。X 只含当时可知的项目状态、设备交付、并网、建设和资金证据。分期投产拆分未交付增量;已投产部分进入 H 后从剩余 G 扣除。公告金额不直接转成已投产能力。
初期采用明确的保守/基准/乐观情景,没有历史标签就不提供伪精确概率。标签积累后可比较阶段条件概率、校准的逻辑回归或生存分析;处理尚未到期项目的右删失、取消与部分交付,不能把所有未完成记录标成失败。
### 6.5 韧性 R
\[
R_{c,t}=100\sum_s q_s\,\min(1,C^{eff}_{c,t,s}/C^{eff}_{c,t,base})
\]
s 是对各方一致定义的冲击情景,例如特定供应类别中断或并网延迟;q 为公开、固定的情景权重,除非有概率依据,否则不称为发生概率。基准能力为零或未知时不计算此比值。
关系图用于识别依赖、替代和共同故障源。没有替代来源证据就标为未知;政策公告通常改变未来供应或可用性假设,不自动删除境内现有芯片存量。
### 6.6 标准化与发布条件
数量型正向指标可采用固定基期映射:
\[
N(x;a,b)=100\,clip\left(\frac{\ln(1+x/a)}{\ln(1+b/a)},0,1\right),\quad a,b>0
\]
a、b 与 x 单位一致,来自有覆盖说明的固定参考面板,存入模型版本。成本型、强度型和比例型指标各自定义映射。不得按每日领先国家重定标,也不能因加入一个国家导致全部历史分数被静默重写。
关键输入缺失时 `S=null`,保留可用子分,不填零或自动重分配权重。O、G 或 R 没有可比输入时综合分不上线;原始图表和证据浏览仍可上线。样本少、匿名取整或来源偏差反映在不确定性和适用范围中,不直接扣国家实力分。
不确定性计算分开保存观测精度、参数估计和情景假设。只有有依据的输入分布才运行 Monte Carlo;共享芯片规格、同一公告、同一供应链的误差必须相关采样。无法量化的覆盖偏差直接披露,不能靠窄置信区间掩盖。排名用“无法区分”处理不稳健结果,不展示过多小数制造精确感。
### 6.7 变化与归因
\[
g_{c,t}=C_{c,t}/C_{c,t-1}-1,\quad p_{c,t}=C_{c,t}/\sum_kC_{k,t},\quad b_{CN,t}=C_{CN,t}/(C_{CN,t}+C_{US,t})
\]
全球分母必须完整定义;双边份额不得标为全球份额。按报告披露值,2022—2023 年中国总算力从 302 增至 435 EFlops,约增长 44%,份额从 33% 降至 31%。这是“绝对增长、相对份额下降”,不是存量消失。算力与创新现有历史窗口不同,不能拼成同一时期的因果结论。
同权重、同版本下,分项账单严格满足:
\[
\Delta S=0.40\Delta H+0.30\Delta O+0.20\Delta G+0.10\Delta R
\]
分项内非线性与交互影响可使用固定分组的 Shapley 分解;必须保存分组、基线和近似误差,不能把任意计算顺序产生的贡献当唯一解释。数学归因只解释模型输出,不证明宏观因果。
正式发布中始终分开:现实变化、补录旧事实、来源修订、模型/基期修订。版本升级用“同一输入、不同模型”的桥接结果展示,不能把换模型造成的跳变计入国家增长。
## 7. 项目内 AI 如何参与
### 7.1 运行方式
复用控制台已经配置的模型,通过 `AIProviderClient` 调用;不要求新的模型供应商,也不把供应商名称写死在业务代码中。第一阶段采用后端固定步骤工作流,各角色是任务模板,可以由同一个模型执行,不需要首先建设复杂的多 Agent 协商系统。
在现有提示词注册表中新增以下拟议 task key;提示词版本与运维覆盖的有效内容哈希进入每次运行记录。
| 任务模板 | 输入 | 允许输出 | 关键约束 |
| --- | --- | --- | --- |
| `compute.sources.discover` | 指标缺口、许可范围、已注册来源 | 来源候选、适用指标、采集建议 | 先找原始出处;新域名不直接进入正式定时采集 |
| `compute.evidence.extract` | 原文分块、页码、表格、目标 schema | 带证据定位的字段候选 | 未出现的数值返回缺失;原文中的指令只是待分析内容 |
| `compute.evidence.review` | 候选、对照记录、单位与实体规则 | 冲突类型、支持/反驳证据、待核验项 | 第二个 AI 的同意不是独立事实证明 |
| `compute.gaps.prioritize` | 缺失、来源健康、参数敏感性、采集成本 | 下一步补证清单 | 优先可能影响结论且可查证的缺口,不以国家倾向排序 |
| `compute.change.explain` | 已计算的贡献账单、事实版本 | 有引用的可读解释 | 不能修改计算结果;事实、估计、假设分别表达 |
| `compute.model.propose` | 误差报告、数据质量问题、模型配置 | 有边界的参数/映射/新特征提案 | 必须说明可证伪假设、预期改善、影响范围和回退方案 |
| `compute.model.review` | 候选方案、独立评估结果、影响报告 | 发布建议与限制 | 无权改变测试标签、门槛、保留集或自己批准自己 |
结构化结果由后端 Pydantic schema 校验。当前 AI 返回以文本为主,需要新增结果解析与受限重试;JSON 不合格、引用不匹配、单位冲突时运行失败或保留候选,不能将自然语言当作正式数据。
### 7.2 工具与权限
允许工具包括:参数化只读指标查询、已注册来源搜索/抓取、文档段落读取、规则化单位转换、实体候选查询、固定估计器运行、创建缺口任务、保存提案、提交实验任务。
AI 不直接写正式分数、覆盖原始证据、执行任意 SQL/Python/Shell、修改发布门槛或扩大自己的工具权限。生成的采集映射必须通过回放样本后才保存;新增可执行采集器代码属于普通开发和发布流程,不在生产 Agent 内 `eval`。
网页抓取需校验协议、主机、解析后的目标地址及每次重定向,阻止访问内网、回环和元数据端点;使用专门外部抓取边界,不能把内部业务 API 和外部取证 URL 混用。AI 只得到所需公开证据或经授权的业务摘要,凭证由后端设置解析,不进入提示词、引用或日志。
### 7.3 用户能看见的 AI 结果
每份分析展示三部分:已验证事实、模型解释、仍缺哪些证据。每条数值和关键结论可展开来源。用户可以标记“实体合并错误”“单位/年份错误”“证据不支持”“假设不合理”,形成结构化反馈;收藏、点赞和是否喜欢国家排名不能成为事实标签。
“AI 发现数据不足”可以是成功结果。模型不可用时,确定性采集、计算、双图和地球继续工作,解释面板保留上次结果并标注版本与时间。
## 8. 数学模型如何自我迭代
### 8.1 自动迭代分成三类
| 层次 | 可以怎样改进 | 自动程度 |
| --- | --- | --- |
| 证据和数据 | 补充来源、修正字段、发现重复和过期、改进解析映射 | 已核准来源与已验证规则可自动运行;新语义及高影响冲突先进入候选 |
| 可验证的估计参数 | 项目延期分布、特定硬件/任务效率、记录错误率 | 通过冻结评估、影子运行和预先配置的边界后,可以自动发布小范围参数更新 |
| 评价定义 | H/O/G/R 权重、指标含义、基期、国家范围和价值取向 | 默认生成提案,由模型负责人确认版本;不是机器从不存在的“国家实力真值”中自动学习 |
此处是产品建议的默认发布策略,不是在本次计划工作中申请执行权限。未来可配置自动发布的参数白名单,但不能用“开启自动迭代”笼统授权所有定义变化。
### 8.2 迭代闭环
```mermaid
flowchart TD
A[新事实/人工纠错/成熟预测结果] --> B[确定性误差与质量评估]
B --> C[AI 提出可证伪改进]
C --> D[生成候选配置与锁定实验]
D --> E[按时间回测及国家分组评估]
E --> F{是否通过预设门槛}
F -->|否| G[记录失败原因并保留现行版本]
F -->|是| H[影子运行:结果不改变正式排名]
H --> I[按发布策略评审或受限自动提升]
I --> J[发布不可变版本与变动桥接]
J --> K[监测退化并可原子回滚]
K --> B
```
每次提案只改变一种主要因素,或者明确作为一个联合实验;不能同时换来源、标签、权重和评估方法后宣称单个参数有效。失败提案也保存,避免只展示成功实验。
提案记录至少包括:问题、证据 ID、基线版本、拟议配置差异、适用实体/任务、可检验目标、锁定数据截止时间、主要评估指标、停止条件和回滚版本。可以提出“将并网阶段纳入交付预测”,不能只提出“让某国得分更合理”。
### 8.3 真实反馈标签从哪里来
| 估计对象 | 可观察标签 | 评估方式 |
| --- | --- | --- |
| 文档字段提取 | 人工核对原文的数值、单位、年份、实体和引用位置 | 字段精确率/召回率、关键错误、引用支持率,按语言/文档类型分组 |
| 实体合并 | 官方 ID、项目连续记录或人工确认的同一/不同实体 | 错误合并率和遗漏合并率;高影响错误单列 |
| 项目按期投产 | 后续正式验收/投产/取消记录,保留观测截止 | Brier score、可靠性图;右删失与失访不能计作失败 |
| 新增容量 | 未来实际确认的设备与容量,具有同口径 | 绝对误差、对数误差;同时报告项目等权与容量加权结果 |
| 任务性能系数 | 未参与拟合的可复现实测配置 | 相同任务质量下的预测误差与区间覆盖 |
| 区间预测 | 后续实际值是否落入范围及区间宽度 | 覆盖率与区间评分一起评价,避免靠无限放宽范围“提高准确率” |
不使用 AI 自己生成的摘要作事实标签,不用现行总分拟合下一版总分,不把另一个复合指数直接当国家实力真值。完整创新指数可做外部一致性讨论,但若其包含输入指标或与 O 重叠,就不能作为独立验证集。
### 8.4 回测与数据泄漏控制
预测训练、验证和保留集按真实可获得时间滚动切分,同一项目的后续阶段、同一公告转载和同一扩建链必须归组,避免分散进训练和测试。每个特征都要求 `published_at <= forecast_cutoff`;仅今天抓到而无法证明当时可得的资料,不参加“当时可知”回测。
新模型先与简单基线比较:维持上次状态、沿用公告日期、同阶段历史交付中位数。不得只与一个刻意弱的旧模型比较。参数、标准化锚点和来源质量估计只从训练窗口取得。
大模型的训练记忆可能知道后来结果,所以历史回测中的预测器必须是只读取冻结特征的确定性估计器;不能让 LLM 凭记忆补旧事实。提案生成与真实保留集隔离,实际前瞻影子运行仍是必要证据。
按国家、硬件代际、项目规模、来源语言分别评估,并按项目/运营方/时间块进行重采样,不能把高度相关的重复记录当独立样本。小样本报告区间与不足,不为满足仪表盘而宣称显著改进。
### 8.5 建议发布门槛
以下是实施阶段的初始门槛草案,需在 M0 用试点数据冻结。它们是最低操作条件,不保证统计有效性,AI 无权自行降低。
| 门槛 | 初始建议 |
| --- | --- |
| 硬正确性 | 单位、时间穿越、重复计数、采集失败保留正式值等必测案例全部通过 |
| 提取评估 | 至少 200 条人工核验字段、覆盖中英文及主要文档类型;报告精确率/召回率及区间;关键数值/单位错误不得进入自动采用集 |
| 预测参数自动提升资格 | 至少 50 个有可验证结果的独立项目、覆盖至少 3 个滚动窗口;每个主要参与方至少 10 个,不足则继续影子运行或人工评估 |
| 改善幅度 | 预先指定主要损失相对现行版和简单基线至少改善 5%;按依赖结构重采样的损失差区间须支持改善 |
| 分组保护 | 主要参与方损失退化不超过预先约定范围,初始建议 2%;不能以平均改善掩盖一个国家明显恶化 |
| 区间质量 | 不仅检查覆盖,还检查宽度与评分;对无法量化的来源覆盖偏差单列说明 |
| 影子观察 | 至少连续 4 周稳定运行,并达到所需成熟标签数;12 个月预测不因观察满 4 周就算验证完成 |
| 试验节制 | 初始每月最多 3 个主要候选,锁定真实保留集;多次搜索造成的选择偏差要通过新的前瞻样本再次检查 |
标签暂时不足时,系统可以自动补数、解释、积累预测与结果,但不发布“自我学习后的更准模型”。这不会阻塞第一版事实产品上线。
### 8.6 一个完整迭代例子
以下是流程示例,不是已发生的项目事实或评估成绩。
系统发现一批已订购芯片的项目未如期投产;确定性评估记录预测误差。AI 查询证据后提出:现有交付模型只使用芯片到货时间,遗漏电力接入阶段。它提交一个新增“并网是否完成”特征的候选。
后端按项目原始发表时间构造训练数据,用固定估计器拟合;锁定保留集检验总体和中美分组误差。即便论文或第二个 AI 都认为这个想法合理,只要结果未通过门槛就不发布。通过后先影子运行;正式发布时展示受影响的 G 分、预计容量和模型版本变化,已投产 H 不因推测而下降。
若后续发现并网字段提取错误导致退化,回滚活动模型指针,保存新证据和失败实验。不能删除失败记录或改写过去已发布的预测。
## 9. 后端数据与实现结构
首版采用最小领域实现:沿用 `CollectedData`、`DataSnapshot` 和位置维表,新增经过 schema 校验的国家指标记录与聚合服务。数值口径、参考期、发表时间、来源链接、原文哈希和证据定位可先作为受约束的元数据保存;必要的原件存放在许可允许的运行数据目录。只有现有表确实无法表达查询或历史约束时才增加专用表。
先冻结面向专题的 API 契约,地球与图表只消费该契约。将来拆出指标观测、证据、实验等表时,通过服务层迁移,不让前端依赖临时 JSON 或物理表结构。首版不以第 9.1 节全部表建完作为开工条件。
### 9.1 拟议持久化对象
下表为待新增的逻辑对象,可按最终数据库设计合并有限的小表;不要与现有采集、位置和任务状态形成并行真相源。
| 对象 | 保存内容 | 关联和约束 |
| --- | --- | --- |
| `evidence_documents` | 原文存储引用、原始/提取哈希、来源、发表/抓取时间、许可、提取版本 | 内容不可变;全文不在高频列表接口返回 |
| `metric_definitions` | 粒度、单位、精度、工作负载、归属、方向、适用范围和版本 | 一个稳定 metric ID;模型版本引用具体定义版本 |
| `metric_observations` | 数值/范围、参考期、知识时间、状态、来源版本、取整和缺失原因 | 链接原采集记录;追加式修订,不复制整份原始 JSON |
| `observation_evidence` | 观测与证据的支持/反驳关系、页号、表格/段落、短摘录 | 多对多;来源家族避免转载重复投票 |
| `compute_entities` / `resource_relations` | 稳定实体、别名、分期关系、所有权/供应/连接、有效期 | 复用现有位置维表;不另存一套会漂移的正式坐标 |
| `model_versions` | 指标集合、权重、锚点、参数、代码版本、情景与发布状态 | 不可变版本;活动指针独立且原子切换 |
| `model_runs` / `model_results` | 输入清单哈希、版本、截止时点、随机种子、环境、国家结果、分项和不确定性 | 结果可重放;实验、情景与正式运行分开 |
| `model_proposals` / `model_evaluations` | 配置差异、假设、冻结评估方案、损失、分组结果、决定与回退目标 | 发布必须引用通过的评估,保存失败实验 |
| `agent_runs` | task key、提示词有效哈希、模型、工具调用、引用、预算、结构化输出、终态 | 链接现有任务账本,Agent 自身不拥有另一套调度状态 |
重放保留输入观测 ID/版本清单或不可变清单文件,不能只保存一段 SQL 然后查询已被更改的数据。记录计算实现版本、依赖环境和随机种子;LLM 输出保存为证据,不要求重新调用模型才能复算数值。
历史删除与保留策略需要显式区分:普通缓存可清理;被正式模型版本引用的观测及许可允许保存的证据受保留保护。来源撤回或许可变更时标记可访问状态并保留允许保留的哈希和元数据,不强行公开原文。
### 9.2 待新增模块与现有边界
| 拟议模块 | 职责 |
| --- | --- |
| `backend/app/services/compute_intelligence/observations.py` | 规范化、有效观测选择和数据缺口 |
| `.../metrics.py`、`.../scoring.py` | 纯数值计算、单位/指标注册、标准化和总分 |
| `.../forecasting.py`、`.../evaluation.py` | 固定估计器、时间切分、基线、分组指标与实验 |
| `.../workflows.py`、`.../proposals.py` | 固定 AI 工作流、结构化候选、预算和发布规则 |
| `.../publication.py` | 正式快照、变动桥接、活动版本和 outbox |
| `backend/app/api/v1/compute_intelligence.py` | 参数化只读查询和后台任务入口 |
| `frontend/public/earth/js/compute-intelligence.js` | 专题 UI 状态、图表与现有图层联动,消费后端计算结果 |
| 控制台专题页与现有采集/AI 设置页签 | 证据核验、缺口、模型实验、版本回看 |
具体文件在实现时保持单一职责,优先复用已有服务,不为目录结构创建空文件。新来源继续注册到现有采集器体系及 datasource 配置;新提示词放入现有 `default_prompts.json`,不散落在 renderer 或 `aiprovider`。
### 9.3 拟议 API
下列接口尚不存在,名称在 M0 与当前路由约定核对后冻结。
| 方法与路径(统一前缀 `/api/v1/compute-intelligence`) | 返回内容 |
| --- | --- |
| `GET /overview` | 所选参与方、口径、参考期与知识截止下的结果、适用范围及覆盖说明 |
| `GET /series` | 指定指标的有界历史序列,含缺失、版本、实际观测时间与来源引用 |
| `GET /entities/{id}` | 设施、分期、资源关系和历史观测 |
| `GET /changes` | 数据或模型贡献账单,数据库侧分页 |
| `GET /evidence/{id}` | 根据权限和许可返回证据摘要、定位与允许展示的内容 |
| `GET /models`、`GET /runs/{id}` | 模型版本、输入清单、计算结果与评估摘要 |
| `POST /recompute`、`POST /agents/run` | 建立任务并立即返回任务 ID,允许取消和查看进度 |
| `POST /proposals/{id}/evaluate` | 创建冻结数据与方案的评估任务 |
| `POST /models/{id}/promote`、`POST /models/{id}/rollback` | 按角色及发布策略原子切换活动版本,记录理由和影响 |
查询参数固定包含 `metric_profile`、`entities`、`reference_at`、`knowledge_cutoff`、`model_version` 和可选 `scenario_id`。前端不接收任意表达式或 SQL,不自行补算另一套总分。
### 9.4 发布与性能
后台完整计算并通过质量门槛后,在事务中发布结果快照和活动指针,通过既有 outbox 唤醒刷新。新 adapter 显式声明国家面板/历史图的刷新范围;设施变化才触发相应 `computeCenters` 更新。草稿证据、失败实验和影子运行不改变默认地球。
缓存键包含口径、实体集合、参考期、知识截止、数据清单哈希、模型与情景版本。WebSocket 只通知结果版本与必要增量,客户端拒绝较旧响应;不能每次国家 hover 都重算模型或执行 LLM。
聚合、过滤、排序和分页在数据库完成;预计算国家年度/月度可用序列,地图按分辨率聚合节点。时间滑动去抖并复用已取快照,播放使用相邻状态差量。建议试点验收目标为缓存查询 P95 小于 1 秒、切换已缓存时点小于 500 毫秒,声明基准机器、数据规模和网络条件;未实测前不作为已达到性能。
## 10. 刷新、成本和失败处理
| 工作 | 建议节奏 | 触发条件与成本控制 |
| --- | --- | --- |
| CSV/API 检查 | 日或来源允许的更慢频率 | ETag/Last-Modified/哈希去重,无内容变化不调用 AI |
| 年报与指数发现 | 月度,发布季可提高 | 找到新版本再下载和提取,不把月检写成月度指标 |
| 重点项目证据 | 日查、周复核 | 去重后仅高影响变化进入提取队列 |
| 指标重算 | 有效观测或模型版本改变时 | 只重算受影响实体/时点/指标依赖 |
| AI 解释 | 有意义的结果变化或用户请求 | 相同输入哈希复用,不因页面打开重复生成 |
| 数据缺口排序 | 每周 | 敏感性、覆盖风险、可获得性和预计采集成本共同决定 |
| 参数候选实验 | 月度或成熟标签达到门槛 | 限制候选数量;无足够标签不强制产出升级 |
试点默认每工作流最多 3 次搜索、8 个页面抓取、4 次 LLM 调用、1 次结构化修复重试,最多同时运行 2 个 AI 工作流;均为可配置预算,不是第三方接口能力承诺。按全局模型配置记录 token/费用,设置日预算和单任务截止时间,超额进入延后或停止状态,不无限自递归。
失败处理:采集失败保留上次正式值和新鲜度;AI 不可用仍提供确定性结果;引用失效保留许可允许的原件和哈希;长期资料缺口显示缺失并降低结论覆盖范围;新源冲突进入待核验;模型退化恢复前一活动版本。不能用重试或回滚删除已提交有效证据。
## 11. 分期实施与依赖
工作量是规划估算,按一名熟悉项目的全栈开发者、持续可用的数据核验支持和现有环境可用计算;不是交付承诺。开发时间不包括等待付费授权、原始报告或 12 个月预测结果成熟的时间。
### 11.1 当前优先交付:展示专题
| 顺序 | 修改方向 | 验收内容 | 估计开发工作日 |
| --- | --- | --- | ---: |
| A1 修复数据与最小合同 | Epoch CSV、TOP500 日期/身份/单位、空结果保护;保存历史指标和来源 | 能得到真实当前快照与核验后的年度序列,不要求宏观指标全到最新年 | 2—3 |
| A2 国家对比与历史接口 | 国家汇总、公开样本与宏观统计分离、双图数据、证据摘要 | 绝对量/份额/创新分差有各自定义;缺失与时点可见 | 2—3 |
| A3 地球与图表展示 | 专题模式、国家面板、双图、时间联动、设施/证据详情 | 完成第 0 节完整浏览路径,并通过桌面/移动/缩放验证 | 3—5 |
| A4 可选 AI 阅读辅助 | 复用现有 client 和任务提示词,针对冻结快照生成带引用说明 | 事实与推断分开;可关闭;失败不影响地球与图表 | 1—2 |
A1—A3 是当前必须交付的闭环,约 7—11 个开发工作日;A4 可在闭环稳定后增加。实际数据源不通、现有采集回归问题或主题布局复杂度会改变估算。实现时首先验证 A1,而不是先开发通用 Agent 框架或完整综合评分。
### 11.2 后续平台能力路线
下表是完整能力建设的工作包估算,与 A 路线存在复用和重叠,不应重复相加。A 路线完成后先盘点已具备的 M0—M2 能力,只实施剩余部分。
| 阶段 | 工作包 | 交付物与通过条件 | 估计开发工作日 |
| --- | --- | --- | ---: |
| M0 合同与试点 | 固定口径、成员、来源许可、历史时间语义、角色权限、质量/评估门槛 | 指标注册表、数据源接入清单、黄金核验样本;所有未知明确登记 | 3—5 |
| M1 可信数据 | 修复 Epoch/TOP500;证据与指标观测;导入两组历史;稳定身份与失败保护 | 可查询、可追溯、可重放的历史 API;无示例数据进入正式结果 | 7—10 |
| M2 可见产品 | 地球专题、双图、国家对比、变化账单、证据抽屉和控制台缺口页 | 中美历史回放闭环;移动/缩放和旧图层回归通过;综合分可保持未就绪 | 7—10 |
| M3 资源与潜力 | 芯片、实测、项目、电力/资金台账,场景与分项估计 | H 与可用 G/R 子项具备来源;O 仅在分项齐备后发布;总分门槛满足才开放 | 8—12 |
| M4 AI 工作流 | 固定提取/核验/补证/解释流程、反馈和预算;扩展可靠任务类型 | AI 只能产生有引用候选与解释,失败不会污染正式数据 | 7—10 |
| M5 模型迭代 | 真实标签、时间回测、候选评估、影子运行、发布桥接与回滚 | 能证明某个可观察估计目标改善;不要求每轮必有新模型 | 8—12 |
从零完整建设上述广义平台能力约 40—59 个开发工作日,包含比 A 路线更完整的证据和实验治理;它不是当前展示专题的工期。影子期至少四周,但不成熟的预测标签会延长自动提升资格等待期。
依赖顺序:M0 → M1 → M2;M3 的项目台账应尽早开始积累标签;M4 依赖 M1 的可信证据结构;M5 依赖 M3 的标签和 M4 的审计工作流。多方、矿产和复杂供应链在 M2 之后按数据质量逐个扩展,不等全部主题齐备再上线。
## 12. 验收与测试矩阵
| 范围 | 必测案例 | 通过标准 |
| --- | --- | --- |
| 来源真实性 | HTML 改版、空 CSV、404、部分下载、示例字符串 | 正式数据不被示例或空结果替换;错误原因可见 |
| 单位与精度 | T/P/E 换算,FP64/FP16/FP8,稀疏/稠密,MW/TWh,FLOP/FLOP/s | 不同口径拒绝合并;可比转换有可复算结果 |
| 时间与版本 | 固定抓取日误作投产日、2026 报告回填 2023、未知发布日期 | 严格截止查询无未来信息;历史回顾明确重述 |
| 实体与去重 | 排名换位、扩建、名称变化、匿名化、重复转载、跨源同集群 | 身份稳定;不重复计数;匿名或缺榜不自动退役 |
| 统计与地理 | 缺坐标、未知国家、欧盟与成员国、跨境所有权 | 国家统计不依赖坐标;分母不重叠;归属视图不混用 |
| 两张历史图 | 已核验年份值、缺失年份、2026 待核实主张 | 只展示有证据的观测;不能伪造平直历史或默认预测 |
| 综合评分 | 缺 O/G/R、权重和不为 1、标准化越界、国家集合变化 | 非法配置拒绝;缺失总分为 null;固定版本不静默漂移 |
| 变化账单 | 新投产、补录、源修订、换模型 | 分项贡献之和满足计算容差;类型和版本差异可解释 |
| AI 输出 | 格式错误、虚构引用、原文含指令、模型超时、重复任务 | 输出隔离;工具边界有效;任务幂等并能取消 |
| 模型评估 | 标签穿越、同项目跨集、过拟合小样本、分组退化 | 不通过门槛不发布;保留失败实验 |
| 发布与回滚 | 两个并发发布、计算中断、旧 WS/HTTP 响应 | 原子版本;半成品不可见;客户端不回滚到旧时点 |
| 地球渲染 | 远距/近距、时间播放、层隐藏、节点锁定、场景退出 | 无闪烁和悬空 tooltip;实例复用,不每帧重新建纹理 |
| UI | 桌面、移动、125%/150% 缩放、键盘、无数据/低置信/加载状态 | 一屏工作区无意外全局滚动;关键标签不靠颜色区分 |
实现阶段按变更执行后端精确测试、`scripts/harness/quick-check.sh`、Bun 前端构建和渲染 smoke;地球改变不能只凭构建通过。保持现有公共页面、鉴权、管理路由、安全导航及其他地球图层的回归证据。性能阈值用记录过的基准规模验证,不能把开发空数据测试称为生产验收。
## 13. 文档、发布与完成定义
每阶段 PR 和部署使用已有项目流程,功能旗标分别控制专题 UI、估计分项、AI 工作流与自动参数提升。先开放读取与事实浏览,再启用模型实验;关闭新旗标应回到原有算力图层而不丢历史数据。
发生真实用户工作流变化时更新中英文手册与快速入门;采集、AI、作业与位置变化更新相应后端技术文档;控制台、Earth、图层顺序和样式变化更新对应 context 与规范。新增公共技术文档需要同步中英文和 Docs 注册;本文件是内部中文计划,不注册为已实现的公共手册。
当前展示专题完成定义:用户完成第 0 节浏览路径,可以从国家指标变化回到出处和所用计算版本;重新计算得到同一数值;知道哪些资料尚缺。两张图应和国家、设施、时间、证据联动,不能只是贴在地球旁的静态图片。
后续决策辅助与模型迭代完成定义:情景的假设与结果可复算;AI 的一次成功改进有独立验证目标、评估记录和可回滚版本。只有一段 AI 分析或未经校验的总分,不代表完成决策辅助能力。
## 14. 风险与预先决定的退路
| 风险 | 产品和实施处理 |
| --- | --- |
| 无法取得创新原报告分项 | 保留完整指数独立观测;O 和总分不发布,不伪造拆分 |
| 中国或其他地区公开设施更少 | 显示宏观统计与设施样本两个覆盖视图,扩充本地语言原始来源,不把保密或匿名化当能力下降 |
| 硬件真实利用率不可知 | 理论量先行,明确任务和区间;不把实验室 benchmark 当生产实测 |
| 来源改版、匿名化或许可证变化 | 版本化下载、质量隔离、许可范围内保存证据,必要时停更单一来源 |
| 资金、电力、芯片相互重复计分 | 固定指标归属与约束链;不把投入、存量、产出和事件各加一次 |
| 训练标签不足或结果多年后才成熟 | 先提供情景与证据改进,延后参数自动提升资格 |
| 模型迎合既定国家排名 | 预先定义目标和门槛,评价权重单独评审,不以用户喜好或另一个 AI 打分训练 |
| 已有采集/定位状态被新服务覆盖 | 复用当前真相源,使用明确字段所有者、发布快照和回归测试 |
## 15. 来源与相关文档
以下链接支持数据口径、来源可得性和架构选择;除明确标注的历史值外,本计划的字段、频率、权重、工期和发布门槛均是拟议设计。
- [信通院 2021 年版算力白皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202109271518811781_1.pdf):2020 年总算力份额。
- [信通院 2022 年版白皮书(镜像)](https://13115299.s21i.faiusr.com/61/1/ABUIABA9GAAgvq2fmwYozLrrpwU.pdf):正文第 13—14 页,2021 年份额与口径。
- [信通院 2023 年版白皮书(镜像)](https://www.ahchanye.com/wp-content/uploads/2023/09/2023092815342684.pdf):正文第 13 页的 2022 年份额,以及 302 EFlops 总规模。
- [信通院 2024 年版蓝皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202502051642799257_1.pdf):正文第 10 页的 2023 年份额及后续规模说明。
- [科技日报关于创新指数报告的报道(新浪转载)](https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml):分差与综合指标维度;未替代原报告分项。
- [2026 年 42%/33% 候选出处](https://www.mornai.cn/news/gpu/ai-computing-power-never-sleeps/):待核实商业文章,不作为正式统计。
- [Epoch GPU 集群公开数据](https://epoch.ai/data/gpu-clusters)、[字段与下载说明](https://epoch.ai/data/gpu-clusters-documentation):CSV、地理字段、覆盖与公开版本限制。
- [Epoch 芯片所有权](https://epoch.ai/data/ai-chip-owners):所有权不等于使用权或所在地,交付与投产有时间差。
- [MLPerf Training](https://mlcommons.org/benchmarks/training/)、[MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/):固定任务、质量及测试配置。
- [IEA Energy and AI](https://www.iea.org/reports/energy-and-ai)、[EIA 开放数据](https://www.eia.gov/opendata/):能源资料入口,不能直接替代设施并网证据。
- [SEC EDGAR API 说明](https://www.sec.gov/search-filings/edgar-application-programming-interfaces):公司提交与财务数据访问;需注意财季与标签口径。
- [USGS 矿产概要](https://www.usgs.gov/centers/national-minerals-information-center/mineral-commodity-summaries)、[UN Comtrade 门户](https://comtradeplus.un.org/):矿产与贸易候选数据源。
- [OpenAlex 帮助与数据说明](https://help.openalex.org/):研究元数据候选来源;接入时核验访问与归属规则。
- [业务架构与数据流转](../technical/zh/platform-data-flows.md)、[后端采集器](../technical/zh/backend-collectors.md)、[AI Provider 边界](../technical/zh/agents-aiprovider.md)。
- [智能星球前端上下文](../technical/zh/earth-frontend-context.md)、[渲染图层顺序](../technical/zh/earth-render-layer-order.md)、[图层视觉规范](../technical/zh/earth-layer-style-reference.md)。
- [轻量 Agent 编排](agents-light-orchestrator-websearch-plan.md)、[态势感知基础计划](agents-situational-awareness-foundation-plan.md):复用总体方向,不重复建设 provider 内业务编排。

View File

@@ -0,0 +1,154 @@
# Planet 开发 Agent 模型分配与 Token 控制计划
日期:2026-09-21。状态:已用于本轮地理与空间卡片开发;未改变全局或项目默认模型配置。适用于已授权的开发任务,不代替用户对任务范围的决定。
目标:在固定政治展示规则、现有 UI 逻辑和性能门槛下,降低完成一个合格任务的总成本与返工次数。不是让所有工作都使用低价模型,也不是让最强模型从头包办。
当前重点任务见 [Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)。该计划的功能与验收要求优先;本文件只决定如何组织开发。
## 1. 模型角色
模型定位参考 OpenAI Docs;下表的具体分工是针对 Planet 的工程建议,需要结合实际一次通过率和返工记录调整,不是模型能力保证。
| 模型 | 精确模型 ID | 默认推理强度 | 分配的工作 | 不作为默认用途 |
| --- | --- | --- | --- | --- |
| Astra | `gpt-6-astra` | `high` | 跨系统难题、空间展示方案、复杂坐标/深度/性能诊断、关键设计与独立审查 | 日常文件搜索、机械编辑、反复运行已确定命令 |
| Sol | `gpt-5.6-sol` | `medium`;复杂实现用 `high` | 执行主控、核心代码、共享组件整合、Three.js 状态与生命周期、最终集成 | 逐条整理大批重复记录、重复复述整个项目背景 |
| Terra | `gpt-5.6-terra` | `medium` | 接口已明确的常规实现、局部 UI 修复、数据处理脚本、测试编写、浏览器回归 | 独自决定空间交互方向、坐标模型或跨模块架构 |
| Luna | `gpt-5.6-luna` | `low`;明确的小修改可用 `medium` | 定向检索、清单比对、日志归类、说明文档初稿、固定格式整理 | 核心渲染、数据迁移、几何归属修改、最终审美或性能放行 |
日常推荐组合是 Sol 主控加一个 Terra worker;Luna 和 Astra 按需进入,不要求每个阶段凑齐四种模型。已经完成的资源准备不因采用这份方案重新做一遍。
当前会话的主模型不会因为写入计划自动变成 Sol。真正执行时,主会话模型由实际任务设置决定,子 Agent 在派发时明确指定模型和推理强度,并核对返回的运行配置;不能把角色名称当作已经切换模型的证据。
## 2. 本项目任务分配表
| 工作包 | 主执行 | 独立验证/审查 | 交付和升级条件 |
| --- | --- | --- | --- |
| 现有窗口、连线、巡航、滚动条入口盘点 | Terra | Sol 检查引用与遗漏 | 文件/符号清单、现有行为证据;Luna 可辅助定向搜索,无需读取全仓库 |
| 贴图归档、尺寸、哈希、配准元数据检查 | 直接运行脚本;脚本需新增时 Terra | Sol 抽查实际输出 | S0 已完成,后续只做有变化的核对,不委托模型重复计算哈希 |
| 固定政治规则的技术合同与几何覆盖边界 | Sol | Astra 审查规则是否贯穿各数据路径 | 权属口径已由项目确定,任何模型均不得重新决定;标识不明确先核对,不能猜测几何 |
| 贴图、海岸线、拾取错位定位 | Sol `high` | Astra 在根因不清或涉及多个坐标层时介入 | 用明确经纬点复现,区分错误数据、回退、投影、UV 与半像素偏移 |
| 规则已明确的映射、来源清单、局部转换脚本 | Terra | Sol;关键几何结果由 Astra 抽查 | 只落实批准合同;发现新的几何语义不得自行扩大规则 |
| 多区域空间卡片的第一版运动设计 | Astra `high` | Sol 检查能否复用当前实现 | 明确锚点、投影、避让、换侧、背面及阅读稳定性;产出小合同,不产出整套 UI 大改 |
| 2 张卡片的连接、弹性移动、生命周期原型 | Sol `high` | Terra 实测;Astra 看关键动效证据 | 优先实现最小完整循环;使用参考图和实际运动证据,不用文字自评代替 |
| 扩至 4/6 卡、增删区域和事件整合 | Sol | Terra 跑交互/竞态测试 | 复用原型合同;`main.js` 等共享入口只由指定 owner 写入 |
| 共享窗口行为、标准字体、移动抽屉及深色滚动条 | Terra | Sol 看渲染与旧面板回归 | 修改公共滚动所有者,验证图层等所有相关页;涉及共享状态重构时升级给 Sol |
| 地球柔光、材质层次、节点与背景关系 | Sol | Astra 对照 UE/PPT/海报审查 | 第一轮只改少量确定参数并给实际截图;性能退化交 Sol 诊断,难题交 Astra |
| 卡片内指标、普通图表、加载/缺失/错误状态 | Terra | Sol | 数据合同先冻结;不复活固定全屏大面板,不编造缺失年份 |
| 国家/区域接口和前后端联动 | Terra;跨模块变更由 Sol | Sol | 不硬编码中美为唯一对象;范围筛选、比较对象和组合去重必须一致 |
| 性能基准工具和回归测试 | Terra | Sol 检查方法;Astra 审查疑难退化 | 固定设备、数据与相机操作;日志含帧时间和内存证据,不能只报平均 FPS |
| 单位、日期、缓存、旧请求覆盖等复杂数据故障 | Sol | Astra 仅处理跨系统未解问题 | 先给最小复现和失败测试,避免多个模型同时试不同修复 |
| 中英文手册、资源清单、计划状态同步 | Luna 初稿;技术性较强时 Terra | Sol 对照实际 diff | 已实现与待实施分开;Luna 不能自行宣告功能完成 |
| 最终集成与验收 | Sol | Astra 对关键风险做一次集中审查 | 相关测试、渲染、政治规则及性能证据齐全;普通修订不逐项再次走 Astra |
## 3. 顺序与并行安排
### 第一轮:证据与边界
Sol 读取现有计划、当前工作区差异和必要规则,整理短任务清单。Terra 盘点实际运行的数据提供者、组件和性能基线;独立且批量的文档/引用整理才交 Luna。先读已有检查结果,避免重新下载贴图、重新研究 World Monitor 或重复跑已经通过的检查。
Sol 将确定的组件接口、规则约束、实际截图与未解问题整理成一个短包,Astra 只审查高返工风险的空间方案和坐标问题。Astra 返回决定、理由和验证点,不再从零扫描整个仓库。
### 第二轮:最小空间原型与独立 UI 修复
Sol 实现 2 张卡片的锚点、连线和有阻尼移动;Terra 可以同时处理公共滚动条或写回归测试,前提是文件所有权不冲突。先完成最小可演示运动,再扩展内容和卡片数量。若两者都要改 `controls.js`、`hud.css` 等共享文件,则按顺序交接,不并行写入。
Astra 对原型的实际截图/录屏做一次聚焦审查,检查是否符合“从地球四周延伸”的方向。空间原型尚未确认时,不展开完整指标卡片和全量国家 UI,以免重演方向错误后大规模返工。
### 第三轮:内容、全球扩展与性能
合同稳定后,Terra 接入指标与状态,Sol 负责与渲染/生命周期的整合。性能测试由脚本执行,Terra 整理差异;可复现的退化先交 Sol。只有不能局部解释的 GPU/CPU、坐标、异步问题才交 Astra。
### 第四轮:一次集成验收
Sol 整合并运行所需检查;Luna 根据最终 diff 更新文档初稿。Astra 集中审查关键几何、政治规则完整性、卡片运动与性能结论。已通过范围不反复重测,除非新修改影响该范围。
## 4. 并发上限与文件所有权
当前环境总并发槽位为 4,含主 Agent;执行时仍以实际可用额度为准。默认使用主 Agent 加 1 个 worker。只有存在两项以上真正独立的工作,且主 Agent 同时有有用工作可做时,才扩大到主 Agent 加 2 个 worker;第 4 个槽位只给临时审查或独立批处理。
每项任务指定可写文件范围。`main.js`、`controls.js`、共享 HUD/InfoCard 与跨层状态接口同一时间只有一个写入 owner。其他 Agent 只读检查或提交补丁建议,由 owner 集成。不得并行启动多个浏览器重测相同场景,也不得共享有冲突的登录/本地存储测试状态。
浏览器验证优先使用当前可用的 Browser 控制能力,包括统一 CUA 入口。不能仅因技能目录中缺少单独的 Browser 条目就判定工具不存在;应继续检查已暴露工具并多次重试连接。只有多次确认 Browser 不可用后,才考虑独立 Playwright,并说明具体阻塞。
Agent 完成有边界的任务后停止。只有同一任务需要补充验证时才续用;不能让 worker 在空闲时自动寻找新功能或自行扩大范围。创建用户侧新任务不作为内部拆工方式。
## 5. 节省 Token 的具体做法
1. 派发窄任务,不默认继承整个历史对话。使用简短任务卡加源文件路径;必要历史按最小范围提供。短上下文不能省略政治规则、性能要求和相关视觉证据。
2. 主控保存一份简洁任务账本:当前合同、文件 owner、已完成证据、失败原因、下一步。各 Agent 引用同一事实,不各写一份长计划。
3. 先 `rg` 定位,再读函数和局部 diff;不重复读取巨大 JSON、整仓日志或全部历史文档。大数据留在文件里,由脚本给出数量、哈希、差异和异常样例。
4. 图片、截图和录屏用实际文件查看。不能为了省 token 把视觉任务变成只读文件名或文字概括。
5. 相同来源只研究一次,保存来源、日期和关键结论;只有数据变化、结论不足或任务扩大时重查。
6. 子任务首次任务卡建议控制在约 500–1500 token,结果摘要约 300–800 token;这是上下文组织目标,不是截断必要证据的硬上限。
7. 返回“改了什么、哪些文件、验证证据、剩余阻塞”即可。完整日志和截图放到文件,用路径引用;失败时返回相关片段,不整段倒入主控上下文。
8. 哈希、解压、下载、构建、测试和统计交给确定性工具。不要为了运行一条已知命令另开一个 Agent。
9. 不重复安排多个模型完整实现同一功能。关键风险做一次独立审查,普通修改由测试与 owner 检查。
10. 默认不开 `max` / `ultra` 推理。提高模型或推理强度只针对具体难题,并记录升级理由。
## 6. 升级与停止条件
| 情况 | 处理 |
| --- | --- |
| Luna 遇到不明确需求、需要理解跨文件状态或开始猜测行为 | 交 Terra;若涉及核心状态/几何,直接交 Sol |
| Terra 的任务超出明确合同,或同一实质失败在一次有针对性的修复后仍未解决 | 给 Sol 最小复现、尝试记录和失败证据,不继续盲试 |
| Sol 遇到跨坐标层、性能来源不明或与现有交互语义冲突的问题 | 给 Astra 聚焦诊断任务;可以由 Astra 完成必要关键补丁,不限于口头审查 |
| 已知一开始就是复杂的空间布局或高风险几何设计 | 直接用 Sol/Astra,不为了经过低模型梯级而增加一次无效尝试 |
| 网络、凭证、工具缺失、原始资料不存在 | 不通过升模型解决环境问题;检查环境或记录缺口,继续独立工作 |
| 失败来自缺失用户决定或无法识别的地名/视觉目标 | 提出具体问题,不让多个模型猜不同版本 |
默认允许 worker 一次有证据的修复;是否继续由主控根据新信息决定。超时或 token 消耗增加但没有新增证据时停止扩展,整理已知结果后重新分配,不能无限自递归。
## 7. 审查门槛
Astra 重点进入三个位置:最初高风险空间/几何合同、首次运动原型、最终关键风险验收;中间疑难问题按需增加。它不需要审阅每一处 CSS、每一段翻译或每个测试日志。
Sol 对各 worker 的交付负责。不能以“另一模型说通过”代替运行结果。任何模型都无权改变项目政治口径、降低性能门槛、把未知数据填成确定值,或自行部署和发版。
涉及政治几何和性能结论,使用与实现过程分离的检查证据;普通文档格式或固定映射使用针对性检查即可。视觉方向仍由已确认的用户要求和参考图决定,Astra 的审美判断不替代用户目标。
## 8. 单任务派发模板
```text
任务:一个可独立验收的目标
模型与推理:明确 model ID + reasoning effort
必要约束:相关 rules 模块、政治口径、性能、共享组件、视觉参考
输入:最少文件/符号、当前合同、已有验证结果和图片路径
可写范围:明确文件;共享文件由指定 owner 修改
验收:必须出现的行为、必要测试/渲染/性能证据
停止/升级:哪些条件不能继续猜测或扩展
返回:结果摘要、改动文件、检查结果、证据路径、遗留问题
```
派发有意使用较小上下文时,必须把不可省略的约束写进任务卡。执行任务的 Agent 自行按规则加载相应模块,不因主控已读过就假定子 Agent 也知道。
## 9. 如何验证分配确实更高效
每个工作包记录模型、推理强度、输入/输出 token(环境可提供时)、耗时、工具耗时、修复轮数、一次验收是否通过和最终证据。若读不到准确 token,就标记未知,不用消息长度伪造 token 统计。
比较每个已验收任务的总消耗,包含主控、worker、审查和返工,而不是仅比较某次调用。先完成 3–5 个有代表性的工作包,再调整默认路由;这个数量是试点取样建议,不代表统计显著性或保证节省比例。
API 单价、订阅额度和任务级成本不是同一个指标。本计划不预先承诺固定节省百分比;模型可用性和实际费用以执行时配置为准。低成本模型导致反复修复时应提前升级;高能力模型能明显减少整体返工时可以直接采用。
## 10. 官方依据与执行边界
OpenAI Docs 将 Astra 定位为最具能力、适合困难端到端任务的模型,Sol 面向复杂专业工作,Terra 平衡能力与成本,Luna 面向成本敏感任务;它们并未规定 Planet 各文件应由哪个模型处理。本计划按任务风险给出项目级分工。
- [OpenAI 模型说明](https://developers.openai.com/api/docs/models)
- [Codex 模型选择](https://learn.chatgpt.com/docs/models)
- [子 Agent 配置与模型选择](https://learn.chatgpt.com/docs/agent-configuration/subagents)
只在用户已授权相应开发与委派、且运行环境支持时执行这套拆工。仅制定本计划不会切换当前会话模型、修改默认配置或自动启动 Agent。执行时若某个精确模型不可用,先说明并选择可用替代,不把替代模型冒充原分配。
## 11. 本轮实际执行账本
| 工作包 | 实际分配 | 结果与升级 |
| --- | --- | --- |
| 地理政策、源证据、几何构建与回退 | Sol `high`,`geography_policy` | 版本化规则与校验已实现;如实保留未核验几何,未用模型猜测填补 |
| 空间连接线、卡片生命周期与主入口 | Sol `high`,`spatial_callouts` | 完成原型;六卡避让多次渲染仍有冲突后,交主控接手约束求解,worker 停止同文件写入 |
| 同机性能基线、对照工具、公共滚动条 | Terra `medium`,`performance_baseline` | 保存桌面/模拟手机各 5 次基线;共享滚动容器修复和对照工具已完成 |
| 复杂避让、全球目录/API、视觉参数、集成验证 | 主会话 | 主控整合并审查;性能脚本的场景同步多次失败后也由主控接手修复。独占布局文件,浏览器/构建按顺序执行,避免测试相互干扰 |
| Luna 辅助 | 未启用 | 没有值得额外传递上下文的独立机械任务,不为凑齐模型而派工 |
本轮精确 token 与费用不可得,记为未知;不报告未经测量的节省比例。空间避让存在返工,不能把功能实现或子任务数等同于效率提升。性能问题应在首次稳定复现后采集 CPU/布局证据,再修改热点;本轮最终定位到共享 SVG 长度读取造成的同步布局,单纯调整避让参数不足以解释该成本。性能、运动观感及地理缺项以功能计划的验收状态为准。

View File

@@ -2,7 +2,8 @@
**状态**:待实施
**创建日期**:2026-05-12
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.claude/commands/docs.md`,让以后写文档时自动按受众归档。
**校正日期**:2026-06-26,控制台深链已从旧 tab 查询口径更新为当前 `?section=` 口径。
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.codex/skills/docs/SKILL.md`,让以后写文档时自动按受众归档。
## 背景
@@ -31,12 +32,12 @@
3. **登录与找回密码** — 登录页、忘记密码流程
4. **账户设置** — 修改密码、修改邮箱(需重新验证)、查看权限组、登出
5. **Console 总览** — 左侧菜单结构、各路由用途
6. **配置数据采集器** — `/collection-management?tab=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
7. **配置 AI 凭证** — `/ai?tab=providers`:默认 provider、模型、Base URL、API Key、本地代理;工具 tab(WebSearch、OCR)
6. **配置数据采集器** — `/collection-management?section=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
7. **配置 AI 凭证** — `/ai?section=integrations`:默认 provider、模型、Base URL、API Key、本地代理;工具 section(WebSearch、OCR)位于 `/ai?section=tools`
8. **系统设置** — `/settings` 其他子 tab(系统设置、电视直播源、SMTP 邮件)
9. **用户管理(管理员)** — `/users`:创建、删除、改角色、Gatekeeper 权限组
10. **数据探索** — `/datasources`、`/data`、`/bgp`、`/alerts/*`
11. **AI 测试台** — `/ai?tab=playground`
11. **AI 测试台** — `/ai?section=playground`
12. **Earth 公开页面** — 现 manual.md 的 Earth 章节原样保留(图层、图例、搜索、位置候选、设置、视角、动捕、巡航、移动端)
13. **Docs 文档站** — 当前 Docs 章节保留(权限组说明)
@@ -48,7 +49,7 @@
- 打开管理员给你的 URL
- 注册账号 + 邮箱验证
- 登录后第一次做什么(建议先到 `/collection-management?tab=collector_credentials` 配一个 collector,再到 `/ai` 配模型)
- 登录后第一次做什么(建议先到 `/collection-management?section=collector_credentials` 配一个 collector,再到 `/ai?section=integrations` 配模型)
- 看 Earth
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节,**不**再单独出 `ops-quickstart.md`,避免新增维护点。
@@ -79,7 +80,7 @@
> - 新增客户可见 UI 流 → 同时更新 `manual.md` zh+en 与 `docs-content.ts`
> - 新增 ops 命令或脚本 → 只更新 `ops-runbook.md` zh+en
## .claude/commands/docs.md 增量
## `.codex/skills/docs/SKILL.md` 增量
在 "Step 2 — Decide Scope" 后插一段:
@@ -99,7 +100,7 @@
- `docs/technical/zh/quickstart.md` & `en/quickstart.md` — 重写
- `docs/technical/zh/ops-runbook.md` & `en/ops-runbook.md` *(新)*
- `docs/documentation-coverage-rules.md` — 加受众分层段
- `.claude/commands/docs.md` — 加 Document Audience Routing 段
- `.codex/skills/docs/SKILL.md` — 加 Document Audience Routing 段
- `frontend/src/pages/Docs/docs-content.ts` — 注册 `ops-runbook` 到 `DOCS_METADATA`(`docs_admin` 组)
## 依赖

View File

@@ -0,0 +1,237 @@
# Earth 地理资源与空间卡片展示实施计划
日期:2026-09-21。状态:本地可运行原型。贴图已替换并归档旧图;固定规则、部分核验几何、全球参与方目录、空间卡片、共享移动滚动条与轻量材质调整已实现。完整政治几何、逐段海岸对齐、视觉认可及性能门槛尚未全部验收,不代表可发布。
本计划以已确认的展示方向为准,替代算力专题旧方案中的固定右侧栏、横贯底部双图和隐藏其他面板的布局。空间卡片与算力目录已有本地可运行原型,但仍需完成最终 QA;算力数据采集、指标证据和长期模型设计继续参见[算力与资源态势计划](compute-resource-intelligence-plan.md)。
## 1. 目标与工作边界
地球是空间信息的主体。打开区域对比后,多个区域分别引出连接线,牵引周围的浮动信息卡。地球转动时,地理锚点跟随球体,卡片保持正向可读,通过平滑、有阻尼的运动调整位置并互相避让,形成信息从地球四周延伸出来的感觉。
地球质感参考 UE 3D 截图在易拉宝与 PPT 中的表现:球面有层次、边缘与明暗交界柔和,背景退后,数据与球体构成同一空间。保持当前操作流畅度是硬验收条件。
已授权实施范围包括资源、空间卡片、共享 UI 与性能验证。继续使用现有 Three.js 引擎和资源加载机制。资源版本与实际转换参数以[地球资源说明](../../resources/earth/README.md)和清单为准。
### 当前实现与未完成项
本轮已实现卡片运动、配置控制、内容承载及中性地球材质调整。配置窗口与卡片显示开关已分离,关闭配置不再停用卡片。卡片默认常显基础信息:36px 主数值、年份/指标/口径说明、设施样本与 Epoch 来源、趋势首末年份及单位、证据入口和区域成员数;长列表仅在需要时展开完整图表。桌面卡片约宽 268px,高度随内容自适应,国家约 214px、区域约 223px;展开卡片支持用户调整至最高 480px,位置/尺寸可保存并恢复自动布局。手机保留同等摘要信息。高清 overlay 使用白色 MeshBasicMaterial、opacity 1、toneMapped:false;r128/r160 按 active API 使用对应 sRGB 读写路径。没有新增 GPU draw pass 或高分辨率纹理,但新增并预加载 45 个本地 SVG 旗帜资源。
旧版底边换侧的连续跳动已定位并由路径规划与稳定换侧策略处理。新实现由 `spatial-callout-controller.js` 管理,布局采用事件驱动并保留有效旧矩形;投影连线持续跟随地球,另一侧候选需持续 1800ms 才换侧,12 秒阅读间隔用于抑制不必要的替换。悬停、焦点和详情阅读会锁定卡片,用户拖动复用 `hud-drag.js`,现有 controls 也消费该共享模块。多卡布局通过实际空闲矩形安排最终位置,优先避开已有功能窗;局部 UI QA 已完成,全球性能与最终发布回归仍按后续门槛验证。下方性能数字和旧尺寸的回归记录属于此前原型,不能直接作为当前大卡片的性能验收结论。
- 底图默认“原始底图”(菜单首项),显示菜单可切换并保存;Natural Earth I 保留,NASA 21K 仅作源归档。运行时统一 8K,按需只保留一张 GPU 纹理;成功后释放旧图,失败保留旧图,过期响应不能覆盖当前图源,不新增渲染层。星空逐字节恢复 commit `83a10a6c`,不再校准密度和颜色。高清 overlay 使用白色 MeshBasicMaterial;r128/r160 按 active API 选择对应 sRGB 读写。没有新增长驻高分辨率纹理、后处理或 GPU draw pass;算力卡另新增并预加载 45 个本地 SVG 旗帜资源。Natural Earth 原图偏浅,不把不存在的细节写成恢复结果。
- 地理政策由版本化清单贯穿规范编码、后台聚合、高精几何及低精回退。中国、台湾、港澳、科索沃等已有回归;已核验几何及济州岛补全见资源说明。高精技术就绪与政策可发布分开:`high_precision_ready=true`,`policy_publishable=false`。东沙、中沙、曾母暗沙点注记及经核验海上主张线仍有缺项。
- 规范目录覆盖 45 个国家编码,界面分组为中国、美国、欧洲、不含中国的亚太和其他;分组不当作统计合计。当前空间原型最多同时显示 6 张,后端最多接受 12 个参与方。没有历史观测的地区返回 `null` 历史与覆盖信息;设施样本数永远不作为计算出的国家总量。
- 桌面卡片复用共享 HUD 和连接线,保留稳定 DOM 身份;选择器和来源窗使用共享拖动。背面卡片保留内容并隐去连接线。旋转通过原动画循环更新,投影连线持续更新;布局按事件分帧处理并保留有效旧矩形。`main.js` 的 `isDragging`、`activeTouchPoints`、`hasActiveGlobeInertia` 接入运动模式:手动/惯性快速模式以 72ms/12px 为响应预算,在当前帧完成受限布局,停止后一次 settle;slow auto 保留 1.8 秒换侧和 12 秒阅读保持,用户拖动放置优先。
- 手机复用公共抽屉与深色滚动所有者,使用适合手机阅读的字号;目前是抽屉内紧凑卡片列表,尚未实现手机环绕空间卡片。桌面原型没有强制隐藏新闻、媒体或统计窗口。
- 2/4/6 卡片、背面投影、共享窗口避让、饱和纵列与共享连线动画已有 11 项单元测试。浏览器已检查 1366×768、1920×1080、390×844 及 125%/150% 对应的有效 CSS 视口;单元测试不替代运动观感验收。
### 本轮空间卡片 UI 定向验收(2026-09-21)
本计划保留此前的历史验证记录;它们不代表当前地图源切换或全球 Earth 验收已经完成。当前实现说明只记录所需资源生命周期与 UI 约束,Browser/全球性能结论仍按后续门槛单独复核。
以上是本轮卡片 UI 与局部资产的定向证据。全球地理政策、逐段海岸对齐、完整 Earth 渲染性能、真实设备性能及发布级回归仍按本计划后续门槛单独验收。
## 2. 优先级与不可退让的要求
| 优先级 | 要求 | 执行原则 |
| --- | --- | --- |
| P0 | 固定政治展示规则 | 外部数据、更新、回退及 AI 均不能覆盖项目规则 |
| P0 | 性能不退化 | 在同设备、同数据、同场景下验证;可复现的退化不能用视觉提升抵消 |
| P1 | 地理对应一致 | 贴图定位、海岸线、区域归属、拾取和卡片锚点一致 |
| P1 | 地球关联的空间卡片 | 使用现有连接线与展示机制扩展,不恢复固定大屏布局 |
| P1 | 共享 UI 与移动端一致 | 共用字体、标题栏、按钮、滚动容器及窗口行为 |
| P2 | 卡片内图表和内容完善 | 先验收空间关系与运动,再充实图表和解释 |
## 3. 固定政治展示规则
项目展示采用固定中国视角:中国相关领土及争议区域按项目既定中国视角处理,台湾按中国处理;科索沃按塞尔维亚处理。该口径先于数据源选择,不由数据源默认分类、模型输出或换源结果决定。
实施时建立单一、版本化的规则清单与几何修正包。名称映射只是其中一部分,必须覆盖国家编码、Polygon/MultiPolygon 归属、边界绘制、区域填色、鼠标拾取、高亮、统计及卡片归属。处理后的结果才可以成为运行资源;未经规则校验的新来源不得替换正式资源,旧资源回退也必须通过同一规则。
规则清单应包含:稳定规则 ID、目标区域的明确标识、目标归属、显示名称、几何来源、合并/裁切操作、版本及回归样例。不能只依赖模糊地名字符串替换。语音中未能确认的具体岛礁名称必须在修正清单中明确定位,不能猜测名字后直接移动多边形。
海岸线、陆地行政区域和海上主张线是不同几何层。项目口径在各层一致执行;海上主张线不能作为陆地面直接填充,也不能代替具体岛礁的几何和归属规则。
必须具备的失败测试:外部源恢复原始分类、别名变化、同区域不同编码、几何修改后拾取仍用旧数据、低精度回退覆盖规则。任何一项失败,都不发布该资源版本。
## 4. 已查明的现状
### 4.1 贴图与国界
- 原运行贴图为 `frontend/public/earth/assets/8k_earth_daymap.jpg`,8192×4096;代码直接加载 JPG,未读取该文件配套的地理定位元数据。
- `country-boundaries.js` 使用独立 GeoJSON/PMTiles 几何绘线与拾取;高精度是一个选择状态,存在退回低精度的路径,按钮名称不等于实际正在使用的文件。
- 本机此前检查到 `natural-earth-v1`、`geojson-high-precision` 构建清单,以及独立 base、hover、claim 文件;该构建不能直接等同于完整应用了项目政治几何规则。实施前需要重新读取实际运行版本。
- 已提供的两张问题截图中,一张出现海岸折线与影像陆地错位;另一张有可见岛屿但没有对应线框。截图只能证明现象,尚不能断言根因一定是 UV、投影、低精度回退或缺岛。
- 精度检查必须同时检查“看得到轮廓”和“能选中对应区域”,二者不能各用一套不一致的几何。
### 4.2 World Monitor 对照
本机 World Monitor `463aec0f` 使用独立地表图片与 `countries.geojson`;经纬度归属由点落入多边形的计算决定,二维底图和自带的交互区域也分开加载。
本次文件统计:World Monitor 的中国、日本、韩国分别为 200、60、14 个坐标点;Planet 的低精度分别为 240、65、19;本机高精度文件分别为 14,118、6,952、3,060。点数不等于测绘准确度,但表明不能靠直接换用 World Monitor 的简化国家文件解决近景海岸问题。它的原始归属处理也不能代替本项目规则。
### 4.3 已有共享机制
| 入口 | 当前作用 | 计划复用方式 |
| --- | --- | --- |
| `frontend/public/earth/js/callout-connector.js` | 连接线、元素锚点与路径 | 扩展为多个稳定区域锚点与卡片配对 |
| `frontend/public/earth/js/presentation-controller.js` | 展示生命周期与连接器更新 | 在单项展示的基础上设计受控的多项展示,不破坏现有模式 |
| `frontend/public/earth/js/cruise-sequencer.js` | 巡航时序 | 保留当前巡航行为,区域对比明确自己的生命周期 |
| `frontend/public/earth/js/info-card.js` | 公用详情与移动详情 | 抽出/复用内容渲染能力,不复制成另一套卡片组件 |
| `frontend/public/earth/js/hud-panels.js`、`controls.js` | 面板收起、显示、拖动与约束 | 工具窗继续遵循这些规则;明确 createHUDPanel 本身不包办所有拖动/缩放 |
| `frontend/public/earth/css/hud.css`、`info-panel.css` | 公用外观、字号、滚动与移动抽屉 | 去掉专题私设的大字号、独立控件和白色滚动条 |
## 5. 地理资源:先采用配套贴图的简单方案
### 5.1 选择与准备
默认运行底图为“原始底图”(菜单首项),显示菜单可切换并保存底图源;Natural Earth I 保留可用。NASA 21K 仅保留为源归档,运行包统一使用 8192×4096 的单张 GPU 贴图。官方源版本与许可信息仍以资源清单为准,不能把归档源误写成运行资源。
采用不带政治标注的物理地表图。按需只保留一张运行 GPU 贴图;新源成功加载后释放旧图,失败保留旧图,过期响应不得覆盖当前图源。源包保留 TIFF、世界文件和说明,运行资源统一为 8192×4096 的 Web 图。转换只做尺寸与格式适配,不用生成式图像重绘海岸、岛屿或国界,不随意裁切、旋转或拉伸经纬范围;底图切换不新增渲染层。
下载的原始大文件保存在本地 `downloads/earth/`,不纳入前端发布包。可追溯清单、配准信息与旧贴图归档放在 `resources/earth/`;运行图放在现有 `frontend/public/earth/assets/`。旧图必须先按字节完整归档并核对 SHA256,之后才能替换运行路径。
### 5.2 对齐验证
1. 读取 TIFF/TFW 的像元大小、原点、行列方向、地理范围和 CRS;同时记录 JPEG 派生后的像元中心定位,避免半像素偏移。
2. 明确运行中加载的贴图、边界文件、资源版本与是否发生回退。
3. 使用同一投影和球面经纬转换叠加成套矢量,检查赤道、零经线、日期变更线和高纬区域。
4. 在截图对应海岸、近海岛屿、台湾周边和其他不同经度的典型海岸逐点验证,区分整体偏移、局部概化和缺失几何。
5. 政治修正后的同一份区域几何用于标注、高亮和拾取。所有需要烘焙的线也从这份数据导出,不能在图片和矢量上分别手工描线。
6. 小岛、岛礁或海岸不满足要求时,先补充对应区域数据并重新配准;仅当这条简化路线确实不够用时,再考虑局部细节或分级资源。
成套资源声明配准不等于本项目已经逐像素验收。不能承诺卫星影像每一处潮线都与概化海岸线完全一致;验收需保存具体偏差和缺失清单,不能靠强行拉伸边界迎合图片。
### 5.3 常态渲染与交互
优先预合成常态显示所需的边界,以减少额外线段绘制。需要独立国界开关时,评估复用现有覆盖层或同定位的透明边界层;不默认再常驻一张 8K 纹理。矢量数据继续承担区域判定和选中高亮,不在每帧重新解释整份 GeoJSON。
本阶段不以替换 Three.js、引入新 GIS 平台或建设全球在线瓦片服务作为前置条件。
## 6. 整体视觉方向
参照资产:`Planet-Handoff-20260916` 中的完整易拉宝效果稿和卡片展示 PPT,尤其 PPT 内来自 UE 3D 的原始星球画面。海报中的线点含概念叠加,不能拿来充当精确地理数据。
- 地球保持正圆与正确地理位置,视觉上成为主体;不把纹理精细度等同于画面品质。
- 球面采用柔和、连续的明暗层次,边缘有薄而自然的大气过渡;地理细节仍然可读。
- 降低背景星点对比与密度造成的干扰,让背景退后;变化以参考图与实际渲染对照决定。
- 数据节点与连接线有轻重层级,避免强霓虹、油亮塑料感、浓重太空雾和粗硬轮廓。
- 优先在现有材质、光照与透明度范围内调整;新增后处理、辉光或高分辨率透明层须先证明成本。
- 用同一位置、距离、光照条件对比前后效果,近景与远景都检查边缘、深度冲突和细节。
## 7. 区域卡片:从地球四周延伸出来
### 7.1 行为模型
一个区域对应稳定的地理锚点、连接线和卡片 ID。锚点随球体旋转;卡片处在屏幕空间,保持正向、稳定大小和内容身份。转动时变化的是屏幕位置与线的路径,不重建卡片、不交换身份、不清空正在阅读的内容。
卡片根据区域投影分布到球体周围,保留疏密和空间关系,不固定平均分成八个方向,不横排占满屏幕。使用有限的避让和有阻尼的缓动,最终能够安定;不持续抖动或为了追踪锚点逐帧跳位。
### 7.2 布局与生命周期
- 沿用现有连线的起点/终点接口和动画语言,先验证 2 张、4 张、6 张卡片;6 张是原型测试档位,不是已确定的产品上限。
- 保持地域锚点与区域绑定;算法可以移动卡片,但不能移动真实地理坐标来换取好看的布局。
- 优先避让其他卡片和已有工具窗,不自动隐藏新闻、直播、统计、图层面板。
- 对卡片换侧设置稳定区间,避免锚点越过某条中心线就立即左右互换。
- 背面区域不画成穿过地球的前景锚点。建议保留卡片身份,用边缘提示表达背面方位;完整隐藏、收束或边缘提示的最终表现用短动画原型比较后决定。
- 用户正在操作的卡片短暂锁定其屏幕位置;是否允许长期固定/拖动偏移,先以现有交互能力做原型验证,不增加未经确认的复杂模式。
- 增删区域、切换年份、退出对比、隐藏相关图层和请求失败都要有明确的清理与恢复逻辑;旧响应不能复活已关闭卡片。
- 区域名称锚点用于对比展示;不能把区域中心点画成设施位置。
### 7.3 卡片内容与参与方
先验证空间展示,再完善图表。首轮卡片可以只包含名称、一个有来源的关键指标、参考期及来源入口;固定侧栏和横贯底部双图的旧设计不再继续扩展,普通浮窗加分页的方案也不是最终展示方向。
参与方必须支持全球扩展,不能在接口和 UI 中永久硬编码成中美两项。日本、韩国等可以作为独立对象;亚太、欧洲可用于选择菜单的分组或区域筛选。具体默认组合与区域汇总口径还需在完整 UI 原型中确定,不把早期讨论的菜单建议当成已确认设计。组合区域与成员国不能重复计入同一分母。
缺少同口径历史数据的参与方仍可展示公开设施覆盖及明确的缺失状态,不伪造总量、得分或排名。全国总算力、公开 AI 集群样本、TOP500 FP64 和创新指数继续分开。2026 年未核验值不得加入正式历史。A4 AI 解读不阻塞空间展示验收。
## 8. 共享窗口与移动端
设置、来源、图层等操作窗口继续复用已有标题栏、按钮、拖动和按能力启用的缩放逻辑。不得用单独字号和固定网格重新构建一套视觉体系。数据更新和语言切换应保留窗口位置、尺寸、选中对象和滚动进度。
移动端复用原底部抽屉与详情展示。区域卡片数量受可用空间控制,保证地球与主要交互可见;可收束为可切换的关联卡片,但具体排列需要实机原型确认。不能把桌面全部卡片原尺寸堆到手机上。
白色滚动条问题从 `.earth-mobile-drawer-content` 及实际滚动所有者核对,统一复用 Earth 的深色滚动条样式与滚动容器。覆盖图层、搜索、态势、算力、新闻和设置页;底部导航保持固定,避免嵌套重复滚动,不仅对算力页补一条局部 CSS。
## 9. 性能基线与硬验收
### 9.1 基线
资源替换前的画面可通过归档旧贴图复现。保存设备、浏览器、GPU、视口、DPR、图层开关、数据量、相机路径与资源版本。在相同条件下重复测试,至少记录 5 次稳定运行,并区分冷启动、热启动和稳定旋转。
观察:首个可操作画面时间、首个有效地球画面时间、帧时间 p50/p95/p99、长帧/长任务、交互延迟、JS 堆与纹理内存估算、draw calls、纹理/对象数、网络字节及请求次数。只比较平均 FPS 不足以证明没有卡顿。
### 9.2 实现预算
- 运行贴图先维持 8192×4096。相同格式下纹理像素量不增加;RGBA8 粗估基础约 128 MiB,完整 mipmap 约 171 MiB,不能把 JPEG 文件变小当成显存同比下降。
- 保持已有 mesh、图标实例与纹理缓存;旋转不重新解码图片或构造国家几何。
- 地理投影和轻量位置更新与渲染帧同步;碰撞布局只在锚点明显变化、卡片变化或窗口尺寸变化时重算,不让每张卡片各起一套动画循环。
- 卡片 DOM 保持稳定,图表仅在数据/尺寸变化时更新;避免每帧测量布局后又写布局造成同步回流。
- 网络、缓存、监听器、纹理及卡片数量均有边界,关闭/重开多轮不得持续增长。
- 默认不增加新的高分辨率常驻覆盖层、昂贵后处理或全部区域的最高精度常驻数据。
### 9.3 判定
同条件下不能出现可复现的流畅度、加载、交互或资源占用退化。测量噪声必须记录并重复验证,不能事先允许固定比例退化来降低要求。超出预算先调整效果、卡片密度或实现,再复测。桌面与移动分别验收;没有真实手机数据时只能说明模拟检查通过,不能宣布手机性能达标。
### 9.4 内容卡扩容前的原型验证记录(2026-09-21)
同机 Headless Edge、桌面 1440×900 / 手机尺寸模拟 390×844、DPR 1、自动旋转,每种尺寸 5 次;预热 1 秒、帧时间采样 2 秒。该短采样用于定位,不能代替真实设备长时交互验收。原始基线标识 `2026-09-20T18-30-56-184Z`,最新对照标识 `2026-09-20T21-07-40-495Z`(UTC);完整 JSON 留在本机临时验证目录 `planet-earth-performance-baseline`。
| 项目 | 原基线 | 本轮结果 | 判断 |
| --- | --- | --- | --- |
| 桌面无卡 p95 帧时间,5 次中位数 | 7.09 ms | 7.10 ms | 接近;不能据此宣称完整性能通过 |
| 手机尺寸模拟无卡 p95 | 7.00 ms | 7.20 ms | 未做真实手机测量 |
| 桌面 2 / 4 / 6 卡 p95,5 次中位数 | 无同场景旧实现对照 | 7.10 / 7.10 / 14.00 ms | 六卡比本轮无卡场景有明确成本,仍需优化 |
| 六卡采样窗口长任务 | — | 5 次均为 0 | 没有 50 ms 长任务不等于没有掉帧 |
| 桌面 DOMContentLoaded 中位数 | 1.262 s | 1.627 s | 已移除 Google Fonts 的约 10 秒阻塞;仍不宣称加载性能与基线完全持平 |
| 桌面基础 JS 堆中位数 | 约 50.2 MiB | 约 53.2 MiB | 不是显存;未替代纹理/绘制统计 |
每轮另做 5 次专题开关并强制 GC 检查,关闭后的保留堆约 25 MB,首末差约 0.28–0.33 MB;这不构成长期无泄漏证明。浏览器单独核对关闭后的卡片悬停/焦点监听器为 0,来源窗与拖出容器的选择器均能关闭。
最终 Bun 构建、前端规则检查、文档一致性检查与 `git diff --check` 通过。既有前端 smoke(使用其 API mocks)通过 321 项路由/交互检查,包括公共页面、鉴权、管理导航、移动布局和管理页缩放;Earth 空间卡片另用真实本地接口检查,不能混同两种证据。
布局求解已分帧执行、减少重复候选评分、优先前景区域,在合格布局找到后停止重复试排;工具栏按可见按钮而非透明容器避让。CPU 采样发现共享 SVG 的 `getTotalLength()` 同步布局热点,已改为按已知折线点计算长度,保留原绘线动画。11 项布局/投影/连线测试通过。1366×768 浏览器中 4/6 卡、旋转后、稳定后及工具栏展开时未发现与卡片、选择器或现有 HUD 的矩形交叠;1920×1080、1093×614、911×512 同样完成六卡边界检查,后两项是浏览器缩放的等效 CSS 视口,不冒充真实设备缩放测试。46 项相关后端测试通过,高精与回退的台湾、藏南、阿克赛钦、科索沃、济州岛浏览器拾取均符合规则。官方几何缺项、完整海岸对齐、六卡性能与真实手机仍未通过最终验收。
## 10. 分阶段实施与交付门槛
开发模型分工、文件所有权、并行限制和上下文控制按[开发 Agent 模型分配计划](development-agent-model-allocation-plan.md)执行:Sol 主控与核心实现,Terra 承担明确子任务,Luna 处理有边界的辅助工作,Astra 负责高风险设计及疑难审查。模型分工不改变本计划的政治规则、视觉方向和性能验收门槛。
| 阶段 | 工作 | 必须交付的证据 |
| --- | --- | --- |
| S0 资源准备 | 新贴图、配准信息、旧图归档、来源清单 | 新旧尺寸/哈希、解码与球面渲染检查;本轮范围 |
| S1 规则与配准 | 固定政治清单、实际提供者诊断、配套边界对齐 | 规则测试、截图问题点对照、回退测试;不能只报高精度开关已打开 |
| S2 空间运动原型 | 复用连接器,多区域锚点、卡片稳定身份与避让 | 转动/缩放/换侧/背面/退出短录屏,2/4/6 卡档位及性能差值 |
| S3 视觉与组件 | 柔光质感、背景层次、共享字体与移动滚动 | UE/PPT/海报参照对照、多个视角、桌面/手机及缩放截图 |
| S4 内容联动 | 多参与方、真实指标与来源、时间/设施联动 | 可操作路径、缺失/错误状态、来源与口径验证 |
| S5 回归与发布准备 | 性能复测、旧图层/巡航/工具窗回归、同步文档 | 完整验收记录;需要发布时另走仓库发布流程 |
S1 未通过,不能把新图片称为边界问题已解决。S2 的空间交互未确认,不能直接投入完整图表和全部国家卡片。性能或政治规则任何一项未通过,后续阶段不能据此宣布完成。
## 11. 验收清单
- 政治:标签、几何、填色、拾取、统计、卡片与回退源口径一致;科索沃等明确规则有专门用例。
- 地理:截图所指海岸和小岛、台湾周边、日期变更线、高纬度;记录偏差而非只肉眼说“差不多”。
- 视觉:球体不变形,边缘柔和但地理清楚,卡片从区域延伸且不压满地球;远距无闪烁和黑块。
- 交互:转动、缩放、选中、背面、卡片增删、触摸、键盘、语言切换、退出、窗口大小变化及旧响应竞态。
- 共存:新闻、媒体、统计、图层及原详情卡保持可用;不通过强制隐藏制造布局空间。
- 移动:公共抽屉无白色原生滚动条,底部导航固定,只有预期内容区滚动,文字和操作不被截断。
- 响应式:1366×768、1920×1080、约 390×844,以及 125%/150% 缩放;模拟与实机结果分别记录。
- 回归:公共页面、鉴权、管理路由、导航/搜索/标签页、既有 BGP/新闻/动捕巡航与旧图层。
- 性能:与归档基线同条件比较,保存帧时间、内存、网络和交互证据;循环打开关闭验证资源释放。
- 文档:更新实际变更的中英文手册、Earth context、图层规则和资源清单;计划与已实现行为不能混写。
## 12. 来源、恢复与完成定义
- [Natural Earth 栅格与矢量配准说明](https://www.naturalearthdata.com/downloads/10m-raster-data/)
- [Natural Earth I 资源页面](https://www.naturalearthdata.com/downloads/10m-raster-data/10m-natural-earth-1/)
- [Natural Earth 使用条款](https://www.naturalearthdata.com/about/terms-of-use/)
- [现有 Presentation 解耦计划](earth-presentation-decoupled-architecture-plan.md)
- [原高精度边界计划及历史说明](earth-high-precision-boundary-tiles-plan.md)
- [地球前端上下文](../technical/zh/earth-frontend-context.md)
- [渲染层次规范](../technical/zh/earth-render-layer-order.md)
旧贴图恢复路径和 SHA256 见资源清单。先核对归档哈希,再将归档副本复制回原运行资源路径并重建预览;不以回退图片为由撤销其他用户改动。新资源和旧归档均不得无记录覆盖。
最终完成意味着:展示规则固定且经验证,地图与区域选择对应,多个卡片确实由地球锚点牵引并稳定运动,操作组件和手机滚动保持一致,实际性能不低于基线。单张效果图、构建通过或“高精度”名称都不能单独证明完成。

View File

@@ -2,6 +2,8 @@
## 当前状态
最新实现继续保留分桶 `Points` 与全局数据范围,并已接通 `vessels` 全局实时订阅:后端读取确认状态后按 MMSI 拆包推送,前端原位修改缓冲;快照用于首次加载、重连和删除校准。旧快照不能覆盖较新的更新或删除,旋转与缩放仍不触发视口请求。下方 Sprite 问题分析和阶段方案保留为历史背景,当前约束以[地球前端上下文](../technical/zh/earth-frontend-context.md)为准。
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad,而是更稳的分桶 `THREE.Points` 方案:
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。

View File

@@ -51,6 +51,7 @@ That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-com
Supported request adapters:
- `openai-completions`
- `openai-responses`
- `anthropic-messages`
- `ollama-generate`
@@ -95,12 +96,37 @@ The AI settings page uses:
- `POST /api/v1/settings/integrations/ai-provider/connect`
- `GET /api/v1/settings/integrations/ai-provider/secrets`
- `GET /api/v1/settings/integrations/ai-provider/presets`
- `POST /api/v1/settings/integrations/ai-provider/presets/{provider}/refresh`
- `GET /api/v1/settings/ai-prompts`
- `PUT /api/v1/settings/ai-prompts/{task_key}`
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
Model refresh and lightweight checks share `backend/app/services/llm_model_catalog.py` and query provider endpoints directly, without models.dev. Refresh uses the draft base URL, protocol and credentials, or saved configuration when no draft is supplied. Regional hosts and custom gateway paths are preserved. Missing required credentials return 400; upstream failures return safe errors and retain the previous catalog.
Successful catalogs and `refreshed_at` are stored under `llm_provider_preset:<provider>` in `system_settings`. Dynamic categories use the provider preset as their defaults and are validated before committing. Listing prefers saved catalogs and otherwise shows bundled suggestions. Refresh never modifies the active model, protocol, URL or credentials in `external_integrations`. The frontend retains the draft and previously loaded models when listing fails, with an explicit error message.
### Model Catalog Endpoints
These endpoints were checked against official documentation on 2026-09-13. The table lists discovery URLs; the form still takes the generation base URL. Credentials must belong to the configured service region.
| Provider | Model list GET endpoint | Authentication and parsing |
| --- | --- | --- |
| [MiniMax](https://platform.minimax.io/docs/api-reference/models/anthropic/list-models) | `https://api.minimaxi.com/anthropic/v1/models`; international: `api.minimax.io` | `x-api-key`; `data[].id`; `has_more/last_id` pagination |
| [OpenAI](https://developers.openai.com/api/reference/resources/models/methods/list) | `https://api.openai.com/v1/models` | Bearer; `data[].id` |
| [Anthropic](https://platform.claude.com/docs/en/api/models/list) | `https://api.anthropic.com/v1/models` | `x-api-key`, `anthropic-version`; paginate with `after_id` |
| [DeepSeek](https://api-docs.deepseek.com/api/list-models) | `https://api.deepseek.com/v1/models` | Bearer; `data[].id` |
| [Alibaba Model Studio](https://help.aliyun.com/zh/model-studio/list-models) | `/api/v1/models` on the configured regional host | Bearer; `output.models[].model`; `page_no/page_size/output.total`; filter `capabilities=TG` |
| [Moonshot / Kimi](https://platform.kimi.ai/docs/api/list-models) | `https://api.moonshot.ai/v1/models`; China: `api.moonshot.cn` | Bearer; `data[].id` |
| [OpenRouter](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties) | `https://openrouter.ai/api/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
| [OpenCode Go](https://opencode.ai/docs/go/#models) | `https://opencode.ai/zen/go/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
| [Ollama](https://docs.ollama.com/api/tags) | `/api/tags` on the configured server | Local servers need no key; `models[].model/name`; empty means no installed models |
New Model Studio endpoints for Beijing, Tokyo, Frankfurt and Virginia require the actual workspace host, such as `<WorkspaceId>.cn-beijing.maas.aliyuncs.com`. Singapore uses `dashscope-intl.aliyuncs.com`; Hong Kong uses `cn-hongkong.dashscope.aliyuncs.com`. Discovery changes only the path, never guesses a workspace or switches credential regions. If a legacy Beijing host no longer accepts the account, update the base URL from the provider console.
Pagination and retries have a 30-second overall deadline. Network failures and 502/503/504 allow up to two attempts; authentication failures are not retried. Models are sorted by supplied creation/release dates, preserving upstream order on ties, without arbitrary list truncation. Public catalog membership does not establish account-specific generation permission.
Admin keeps the AI page aligned with the legacy information architecture:
- `Model Providers`
@@ -283,22 +309,11 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
### Lightweight Connectivity Testing
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
- Send full model requests only when the user explicitly runs Playground or a business task.
Connectivity results should be explicit:
- `ok`: authentication, route, and model catalog are usable.
- `warning`: service is reachable, but the current model is missing from the catalog or capability metadata is incomplete.
- `error`: authentication, network, protocol, or model lookup failed.
Toast titles must match the result; failures must not be titled as a successful connection.
The Admin plug button checks proxy configuration and queries the current provider catalog. It returns `success=true` only when discovery succeeds and contains the selected model. A 404, authentication error, invalid response or missing model never becomes success based on bundled suggestions. The message distinguishes discovery from generation; Playground or a business task verifies actual generation. Testing never saves the draft or switches the default provider.
### OpenCode Go Routing Model
Subscription channels such as OpenCode Go should not be handled by hard-coded frontend model sets. Prefer provider catalog or backend capability discovery that records per-model capabilities such as `chat_completions`, `anthropic_messages`, `models_endpoint`, and whether a subscription key is required. The frontend should display capabilities; the backend should map provider, base URL, model, and adapter into the real request.
OpenCode Go protocol mappings are owned by the backend catalog: MiniMax M3/M2.7/M2.5 and the verified Qwen3.6/3.7/3.8 models use Anthropic Messages; GPT-5.6 Luna, Grok 4.6 and Muse Spark Contributor use Responses; other supported models use Chat Completions. The official `/models` response currently supplies IDs only, so new models still require checking the documented endpoint table. Selecting a model updates its protocol in the draft, and backend mappings supersede known stale mappings. Responses uses `input`, `max_output_tokens` and `store=false`, and parses text and reasoning summaries. OpenCode calls carry an application User-Agent and a stable Playground conversation identifier. Generic `thinking=enabled` is translated to MiniMax M3’s `adaptive` format.
#### Settings Page Behavior

View File

@@ -99,7 +99,7 @@ AIS vessel collectors use a different persistence path from regular `CollectedDa
Earth boundaries are no longer data collectors. They are Earth static rendering assets: the Earth Assets settings panel owns source configuration, and `/api/v1/earth/boundaries/*` builds `frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles`. When no high-precision PMTiles artifact is available locally, the frontend uses the bundled low-precision GeoJSON fallback and does not write boundary records to `CollectedData`.
TOP500 and Epoch AI compute sources do not always provide usable coordinates. The unified Earth compute-center endpoint uses only valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without coordinates are returned as `unresolved` instead of being rendered from a local registry, country centroid, or guessed city. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
TOP500 and Epoch AI compute sources do not always provide usable coordinates. Epoch's official CSV `Location` may contain only a country or be empty even when a row has latitude/longitude; it is never treated as a precise site without the required source semantics. The unified Earth compute-center endpoint uses valid source-provided coordinates or `compute_center_locations` dimension-table coordinates during the main map startup path; records without renderable coordinates remain in the complete `unresolved` list instead of being rendered from a local registry, country centroid, or guessed headquarters. `precise`/`site` keep source coordinates; city-level and confirmation-required points may receive only a deterministic display offset keyed by `source` + `source_id`, with the original coordinate untouched. When users manually collect candidates, the backend queries ROR and Nominatim/OpenStreetMap from source fields; accepted candidates are saved into `compute_center_locations` and rendered from that table on the next layer refresh.
Admin collection management follows the business hierarchy instead of flattening every endpoint into one table:
@@ -111,6 +111,12 @@ Snapshot lists should not show every snapshot of the same collector as separate
Credential guides are maintained by `backend/app/services/credential_guides.py`. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.
### News Live Stream Catalog
The IPTV-org adapter in `news_live_streams` retains its news-category filters. Its default `max_sources` is `0`, meaning all matching channels; an explicit positive value still limits collection. Existing configurations retaining the old `120` limit must change it to `0` and collect again to populate the complete catalog.
`GET /api/v1/tv/streams` uses database pagination in `tv_catalog.py`: `offset` defaults to `0`, `limit` defaults to `50` with a maximum of `100`, and whitespace-separated `q` terms match channel names, providers, regions, and languages. Built-in and configured sources come first. Collected sources are deduplicated by channel ID, exclude configured overrides, and are filtered, counted, sorted, and paginated in SQL. The response's `total` counts matches, while `source_count` counts the complete available catalog; use `has_more` and `next_offset` for subsequent pages. `selected_id` can also return the selected channel outside the current page without consuming its quota. `tv_streams.py` remains the owner of default and fallback sources.
## IV. Data Format (stored in CollectedData table)
```python
@@ -352,7 +358,11 @@ GET /api/v1/visualization/vessels/{mmsi}/conflicts
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, and caps `limit` at `5000`. The Earth frontend uses a global bbox for current state and does not refetch on camera viewport changes. The endpoint reads `vessel_current_state` and reports `diagnostics.source = "vessel_current_state"`. The old `/api/v1/visualization/geo/vessels` route has been removed.
High-frequency AIS updates must not become per-delta full-layer rebuilds. If Earth uses the `/ws` `vessels` channel, it should send low-frequency reload/dirty hints and let the frontend merge snapshot refreshes. Tracks and conflicts still read historical facts through the single-vessel APIs.
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
### Layer APIs And Global Stats
@@ -450,3 +460,15 @@ POST /api/v1/datasources/trigger-batch
The request body may pass `source_ids` for selected rows. Without `source_ids`, the backend filters by `product`, `module`, `is_active`, `run_status`, `collected`, `credential_status`, and `q`. The endpoint skips disabled sources, sources already running without `force`, and sources still inside their frequency window, then returns `triggered`, `skipped`, and `failed` groups.
**Core file**: `backend/app/api/v1/datasources.py`
## Compute source contracts and topic APIs
Epoch uses its official public CSV, retaining theoretical FP16 performance, original dates, status, anonymization/rounding flags, and upgrade relationships. Anonymous rows use content fingerprints plus equal-row ordinals to preserve snapshot multiplicity, without claiming persistent facility identity or assigning coordinates. Unknown countries remain flagged and are excluded from selected-country counts. TOP500 uses system-detail IDs, reads the edition from the list URL, and converts header/cell units into FLOP/s with an explicit FP64 LINPACK basis. Public pages may expose only part of an edition; this does not establish full TOP500 coverage.
Neither collector returns sample data on failure. Empty fetches or transformed results, schema errors, and losses exceeding 40% of the previous snapshot fail while retaining prior data. Incomplete writes are rolled back. TOP500 location migration copies only a unique exact name and canonical-country match to the new system ID, retaining the historical key; rank is never a migration key.
`compute_history_defaults.json` ships the original report data. `compute_configuration.py` reads a complete override from the `compute_comparison` category in `system_settings`; absent records use bundled defaults without migrations or manual initialization. `compute_history.py` performs same-year aggregation. Reference periods and publication dates remain distinct, and unknown publication days remain unspecified. `/api/v1/data-products/compute/series` reads configuration on each request with `Cache-Control: no-store`, offering global share, fixed US + China bilateral share and FP32 absolute observations. Innovation gaps remain independent. An empty override stays empty instead of silently restoring observations.
Under the same prefix, `GET/PUT /configuration` and `POST /configuration/reset` require admin/super_admin. Writes accept `{revision, config}` and reset accepts `revision`. Pydantic validates canonical countries, finite values, share totals, duplicate metric/year pairs and report references; each metric/year has one report basis. PostgreSQL category uniqueness and row locking serialize first writes and updates; stale revisions return 409. Configuration and before/after audit records commit in one transaction. Reset removes the override so subsequent reads use the current bundled defaults. `version` is a configuration revision, while `reviewed_at` is the supplied review date. Admin-curated data is not automatically labeled official. Source URLs are references, not scrape instructions.
`/api/v1/data-products/compute/facilities` reads successful snapshots with `compute_contract=v1`, applying country filters and per-source counts in SQL. Epoch excludes planned, decommissioned, superseded, and suspected duplicate records; TOP500 means list observations. Cross-source deduplication remains pending. Historical cutoffs use collection time, never pretending a recently fetched old source was available earlier. Geographic results are bounded and report truncation; total record counts are independent of this limit. Official site helpers are keyed by exact `source` + `source_id`; their access-map centers are site-level references only and retain their official URL in `source_note`.

View File

@@ -70,7 +70,10 @@ The unified event model is `earth.layer.changed`:
| --- | --- |
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
| `delta` | `earth_interactables` upserts/removes by id; ordinary vessel writes update confirmed state by MMSI through the dedicated `vessels` channel without clearing the layer. |
Vessel deletion still emits a `reload` reconciliation hint; individual `vessel_current_state` deletions also enter the vessel remove channel. Source notifications identify changed MMSIs, while transmitted values come from current state. Global subscriptions use `scope: "global"`; message-size limits split frames instead of discarding remaining vessels.
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.

View File

@@ -8,7 +8,7 @@ The console now separates the "data source catalog" from "collector configuratio
- Lists all data sources, including built-in and custom sources.
- Clicking a name only opens an information drawer.
- Focuses on status, manual collection, and running collection tasks.
- `/collection-management?tab=collector_credentials`
- `/collection-management?section=collector_credentials`
- Displays as "Collectors".
- Owns endpoint, headers, base parameters, and credentials.
- Every collector exposes a connection button for health checks.
@@ -321,7 +321,7 @@ The new vessel list entry point is no longer the legacy `/api/v1/visualization/g
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
```
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. If the `/ws` `vessels` channel is connected, it should act as a reload/dirty hint for merged refreshes, not as a per-AIS-delta full-layer rebuild path.
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](backend-collectors.md).
## Custom REST / WebSocket Mapping Runtime

View File

@@ -1,5 +1,19 @@
# Earth Frontend Context
## Configurable compute observations
Compute cards read annual observations from `/api/v1/data-products/compute/series`; the admin Compute comparison section is the maintenance entry point. `compute-topic.js` fetches with no-store, listens for same-origin `planet-compute-configuration` notifications, and checks every 60 seconds while visible and enabled, plus when the page becomes visible. Observation refresh does not reload facility layers. `destroy()` releases the timer, channel and visibility listener. Country membership, full regional coverage and missing-value semantics remain backend-owned.
## Surface texture provenance
On 2026-09-21, `assets/8k_earth_daymap.jpg` was replaced with an 8192×4096 Web derivative of the paired Natural Earth I raster. Its loading path and rendering code remain unchanged. The source is a 16200×8100 EPSG:4326 image; the previous JPEG is preserved byte-for-byte under `resources/earth/archive/2026-09-21/`. Source/license records, the catalogue versus embedded-version discrepancy, SHA256 hashes, JPEG parameters, and recalculated world-file metadata live in `resources/earth/natural-earth-1/`. Large originals remain in Git-ignored `downloads/earth/`, outside the frontend release bundle.
The base-map selector defaults to Original basemap, listed first and saves the source chosen in the display menu. Natural Earth I remains available alongside “Original basemap”, a byte-identical copy of the September 21, 2026 archive; NASA 21K is source archive material only, while runtime delivery stays at one 8K texture. A new source replaces the old GPU texture only after successful loading and releases the old one then; failures retain the old texture and stale responses cannot overwrite the current source. This is a source switch, not an additional rendering layer. The physical texture is JPEG sRGB, does not bake political borders, and detailed boundary alignment is not yet validated. The starfield is restored byte-for-byte to commit `83a10a6c`: `celestial.js` matches that commit, `skyTint` and `skyBrightness` are removed, and `brightStarOpacity` is back to `0.88`; star density and color are no longer calibrated. The HD overlay uses a white `MeshBasicMaterial` with opacity `1` and `toneMapped:false`, with `texture-colors.js` centralizing sRGB read/write. The 0.128 path uses `encoding` plus `LinearTosRGB`; the 0.160.1 path uses `colorSpace` plus the normal colorspace fragment. There is no Phong white-light or colored twilight overlay, blue night tint, `cameraBoost`, or shoulder curve; daylight neutral scaling keeps brightness at or below the source maximum, and dark mode only applies neutral scaling. The original texture asset is unchanged. Natural Earth is intrinsically pale; this does not claim missing detail was restored. Earth still supports the Three 0.128 import-map runtime and frontend package Three 0.160.1 runtime, and WebGL1 derivatives remain enabled for `fwidth`.
Day/night shading uses a narrow smooth transition centered on the solar horizon (normal dot product ±0.12). The day side is capped at the source brightness; night-side linear illumination is 0.24, rising smoothly to at most 0.60 in source shadows to retain dark ocean and forest detail. Shading uses a neutral scalar without added exposure, colored veils, rendering passes, or textures.
Basemaps and TV channels share `hud-select-menu.js` for the HUD popover, list, selection mark, keyboard navigation, and positioning; `tv-source-menu.js` only adapts channel copy. Basemaps disable search, pagination, and counts. The trigger owner identifies popover clicks as internal to the mobile drawer.
This document describes the current real structure of the Earth display frontend. The focus is on helping future changes to the HUD, layers, media panel, real terrain, and BGP visualization avoid repeating past structural and state-sync pitfalls.
Related references:
@@ -75,7 +89,11 @@ This is currently the most critical UI control entry point for the Earth fronten
Earth settings are now grouped by `data-settings-tab` and `data-settings-tab-panel`. Desktop and mobile share the same category semantics: Runtime, Display, Panels, Motion, Shortcuts, and System. When adding a setting, first choose its category, then add the DOM, persistence field, and restore logic; do not keep growing one long undifferentiated panel.
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories to `/api/v1/news/earth-feed?categories=...&locale=zh-CN`, so Web and UE clients share the same backend category filtering path.
Earth runs inside an independent iframe / static application, so it cannot directly reuse React Admin's `react-i18next` context. `public/earth` uses its own [i18n.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/i18n.js) runtime to read and write the global `planet-locale`, while keeping the legacy `docs-lang` in sync. This keeps Docs, the console, and Earth on the same language preference. The language switch belongs in the Settings `System` tab, and desktop/mobile both use the same `data-earth-locale` buttons; do not put language selection into layer, display, or runtime-mode settings.
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories and current locale to `/api/v1/news/earth-feed?categories=...&locale=...`, so Web and UE clients share the same backend category filtering path.
In English mode, Earth news must render only English title/summary text. Chinese source items without `en-US` localization are filtered from the visible cards/ticker/cruise until the backend enrichment finishes, and source/feed labels fall back to English-safe names instead of rendering Chinese labels.
The news panel, ticker, and news cruise must consume `items` / `cruise_items` from the same `/api/v1/news/earth-feed` response instead of keeping separate regional caches. `news.js` builds a refresh request key from region, category, source, and limit; only concurrent requests with the same key reuse the promise, and stale responses from an older region are dropped by token. Source filtering is also region-scoped: when the user moves from Asia Pacific to Europe or another region, source IDs saved for the old region must not be appended to the next fetch. After the new payload arrives, the saved source list is intersected with the available `sources`; if the intersection is empty, the current region falls back to all available sources. This keeps the ticker, panel, and cruise cards aligned after region switches.
@@ -155,6 +173,14 @@ Each module is responsible for its own:
`tv.js` owns the live / aggregation-news tabs inside `media-panel`. Toolbar open and tab-switch actions write back through `earth:tv-visibility-change` and `earth:tv-tab-change`: panel visibility remains viewport-scoped at `views.<scope>.panelVisibility.media-panel`, while the active tab is stored at `shared.mediaPanelActiveTab`. Refreshing the page therefore restores the user's last live/news state. Temporary hides from `closeTransientMobileOverlays()` carry `persist:false` and do not overwrite the preference.
`tv-source-menu.js` reuses HUD and legend-list styling in a popover with search, a scrolling list, and a fixed count footer. `tv.js` requests 50 entries from `/api/v1/tv/streams` and caches channel details; search runs against the complete backend catalog, and pagination uses `next_offset`. Cancellation and a request generation counter prevent stale results from replacing a newer search. Catalog refresh uses `selected_id` to restore a channel outside the first page.
`tv-playback-policy.js` owns live recovery decisions: all non-fatal HLS events remain with the player, while fatal network/media failures receive at most three recovery attempts. Switching requires automatic fallback to be enabled, no explicit user channel selection, and an embeddable backup. Async callbacks in `tv.js` verify player identity and channel; native video errors no longer bypass HLS recovery. `tv_streams.py` replaces the old bundled CCTV-4 CDRM URL, including saved copies, with WCETV's America-edition HLS stream, names the edition explicitly, and preserves custom streams. An HTTP success does not establish video decodability.
Sources with verified CORS support may explicitly set `use_proxy:false` for direct browser playback without segment forwarding. Other sources continue to use the TV proxy by default. The bundled WCETV source uses direct playback.
Source information reuses `createHUDPanel` and its existing 0.22-second CSS transition. The shared controller measures content into `--hud-body-content-height`, avoiding the non-interpolable change from `max-height: none` to zero. ResizeObserver updates content measurements and compensates the top position of upward-expanding dragged panels during animation, keeping their bottom edge fixed without disabling the body transition. The shell's top transition is paused during compensation to prevent drift from double interpolation.
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
@@ -175,6 +201,8 @@ The compute-center layer row has a notification badge for GeoJSON `unresolved` r
Location candidate state in the details card is cached in [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) by `entityType:entityId`. If the user closes the details card or unresolved queue and reopens the same compute center / BGP collector, previously collected candidates and status text are restored. Header-level `一键采用` prefers cached candidates, avoiding repeated online geocoding or LLM factcheck calls. After a location is saved, that entity's candidate list is cleared to a "refreshing layer" status so stale candidates do not keep misleading the user.
`runUnresolvedComputeCenterBatch()` owns a queue of entity contexts rather than panel DOM. `locationCollectStateCache` retains candidates, progress, and save results so rebuilt cards can hydrate the current state. `earth:compute-center-location-batch-change` updates the badge, `savedCandidate` excludes completed entries, and a ✓ entry remains after completion. Queue state lives only in the current Earth page: refresh, close, or route navigation interrupts unfinished work, while saved coordinates remain authoritative in the backend.
The `预览 / 保存` buttons on each candidate row use a single delegated `click` handler per candidate root (the `[data-collect-cache-key]` block in the details card, or `[data-unresolved-item]` in the unresolved queue), guarded by a `data-candidate-actions-bound` flag so it cannot be double-bound. Direct `pointerup` / `click` listeners on individual buttons and overlapping delegated handlers were removed. Candidate objects are no longer JSON-stringified into an HTML attribute and parsed back; buttons only carry `data-candidate-index`, and the handler resolves the candidate object from a module-level `Map` keyed by cache-key. This removes the entire class of failures caused by HTML entity escaping of `&` / `<` / `"` in candidate fields. Clicking `预览` dispatches `earth:preview-location-candidate`; `main.js`'s `previewLocationCandidate()` calls `showComputeCenterLocationPreview()`, which attaches a hollow breathing-ring sprite pair at the candidate coordinates (visually mirroring the BGP event ring) and focuses the camera on the candidate. Previewing another candidate replaces the ring; saving clears it and `spawnSavedComputeCenterLocation()` immediately spawns the formal compute-center interactable. Note that `main.js` has no module-level `earth` variable — every location-save / preview handler must call `const earth = getEarth();` first, otherwise the event handler throws a `ReferenceError` that the surrounding `.catch` swallows, producing the failure mode where the button "does nothing".
The `earth:compute-center-location-saved` reconciliation pipeline is deliberately silent on background-refresh failures. `spawnComputeCenterAfterLocationSave()` already presents the success toast and locked state; `refreshComputeCentersAfterLocationSave()` only reloads backend data when the scene is ready and no longer emits its own `已保存` toast. `handleComputeCenterLocationSaved()` runs refresh in the background after a successful spawn; only when spawn returns `null` (scene not ready) or throws does refresh take over the success toast. A refresh error is only `console.warn`'d — it must never surface as a `保存失败` message, because the save itself succeeded and the refresh is a follow-up sync.
@@ -187,7 +215,11 @@ The backend snapshot endpoint still requires `bbox`, but the Earth runtime treat
Vessel markers are rendered through `createInteractableLayer()` as batched `THREE.Points`, with `cluster` and `avoidance` explicitly disabled. Dense waterways may overlap. Dragging or inertial rotation skips hover picking; normal hover uses screen-space nearest-point picking. Do not reconnect vessels to dynamic screen clustering or per-frame Points rebuilds, because those interaction costs are what make a 3000-marker layer feel heavy.
If the `/ws` `vessels` channel is used by Earth, it should be a low-frequency reload/dirty hint only. Do not create multiple viewport subscriptions, and do not turn every AIS delta into a full layer rebuild.
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
The legacy `/api/v1/visualization/geo/vessels` route has been removed. Frontend code should keep using `PATHS.vesselsApi` and can verify the current-state path through `diagnostics.source == "vessel_current_state"`.
@@ -217,6 +249,14 @@ The cruise sequencer handles generic logic: current target, queue order, camera
All material, layer, satellite, BGP, cable, terrain, celestial, and other style parameters are maintained here. Do not scatter magic numbers in module files.
## Render hot paths
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object.
Satellite breathing runs in vertex shaders. `satellite-position-worker.js` moves full SGP4 position and initial trail calculations off the main thread; the main thread applies snapshots to interaction coordinates and render buffers. `satellite-propagation.js` supplies the same mathematics to the Worker, predicted orbits, and synchronous fallback. Reload, clear, and altitude changes must terminate prior work so stale snapshots cannot resurrect cleared layers.
The `i18n.js` MutationObserver translates only added subtrees, changed text, or changed attributes. Local clocks, status labels, and download progress must not synchronize all locale controls. Global locale synchronization belongs to initialization, locale changes, or subtrees containing new locale controls.
## Current Style Layers
CSS files in `frontend/public/earth/css/` each correspond to a specific component scope. Do not write global Earth styles into `base.css` unless they genuinely apply to everything.
@@ -242,6 +282,8 @@ Earth settings are stored in `localStorage`. The key is typically a namespaced s
Settings that affect visual layers and surface interaction (terrain opacity, day/night mode, satellite display style, satellite idle breathing, real satellite altitude, track display, hover tooltip mode, etc.) are read during initialization and applied immediately.
Language preference is the exception: Earth language is not a private `planet.earth.settings.v2` field. It shares `planet-locale` with Docs/Admin. When language changes, `i18n.js` updates `document.documentElement.lang`, translates static and dynamically inserted DOM, synchronizes language button state, and passes the current locale to news requests. The news module refreshes after a language change so an English UI does not reuse a Chinese news payload.
The surface hover tooltip preference is persisted by `controls.js` as `shared.surfaceHoverInfoMode`, while `main.js` composes the actual tooltip in the globe-surface hover branch. `Country` shows country details only when a country polygon is hit and stays silent over ocean; `Position` shows latitude, longitude, and sampled terrain elevation and clears country-boundary hover; `Full` shows country + position on land and position over ocean.
The real satellite altitude preference is persisted by `controls.js`, while the rendering state lives in `satellites.js`. When enabled, the real radius from SGP4 is compressed logarithmically into the current Earth visual radius range. When disabled, satellite dots, trails, and predicted orbits all return to the legacy same-sphere display. Toggling this setting must refresh satellite positions and clear trail buffers so a trail never mixes both height models. `maxRealAltitudeOffset = 25` is a visual cap tuned for the current camera and `earthRadius = 100`: GEO / MEO remain clearly higher than LEO, but the highest orbits stay within about 25% beyond the globe radius so selection targets, red trails, and the globe do not feel disconnected.
@@ -327,7 +369,10 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps the projection-based behavior for high-frequency realtime layers such as vessels, and `none` disables clustering. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
`globe-navigation.js` maps zoom to `sqrt(R² + (D₀² - R²) / zoom²)` with minimum surface clearance and the matching inverse. The 100% camera pose remains unchanged; 500% stays outside the globe. Mouse and single-touch drags choose a mode on pointer-down from the actual camera frustum. A fully visible globe retains legacy screen-delta sensitivity, 0.18 smoothing and release inertia, including movement outside its silhouette. A viewport-clipped globe uses pixel-aligned ray/sphere grabbing only when the initial pointer hits its surface. Subsequent missed intersections pause surface updates and clear fling speed instead of using a closest-point substitute that can reverse rotation. Grabs starting in space use screen deltas. The mode stays fixed throughout the drag; auto-rotation is suspended during dragging and pinching. Returning from a pinch to one finger reselects the mode and rebases the anchor. Click suppression uses displacement from pointer-down, and cancellation clears inertia.
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps projection-based clustering, and `none` disables it. Vessels explicitly disable clustering and avoidance so incremental updates do not rebuild cluster topology. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
@@ -352,7 +397,7 @@ For future Earth changes:
The Earth frontend and the console frontend are not the same UI system:
- Console frontend: React + Ant Design workbench
- Console frontend: React + Tactile UI / Radix primitives / lucide workbench
- Earth frontend: native HUD + Three.js display under `public/earth`
Therefore:
@@ -363,3 +408,46 @@ Therefore:
For console structure, see:
- [Admin Frontend Context](/home/ray/dev/linkong/planet/docs/technical/en/frontend-admin-frontend-context.md)
## Spatial compute cards
`compute-topic.js` owns periods, measures, participants, and evidence. `compute_catalog.py` supplies canonical names, picker groups, and regional label anchors for 45 currently registered participants. Unknown historical values remain missing. APIs are no longer limited to CN/US. Group labels never become overlapping statistical denominators, and regional coordinates must not become facility locations.
Earth's existing Roboto Condensed face is now served locally from `assets/fonts/`, with its source and license in that directory and the shared declaration in `css/fonts.css`. The remote Google Fonts stylesheet no longer blocks the page; font family and weights are preserved.
`spatial-callouts.js` delegates layout to `spatial-callout-controller.js` and reuses `CalloutConnector` with stable region IDs. Layout is event-driven and retains valid old rectangles while work is pending; the projected source line follows the globe continuously. The layout reference is the direction of the actual source exit point, so a card stays near the object side instead of being pulled from a top-left geographic point to an opposite corner. Neighboring same-sector placements, shorter connectors, and fewer crossings are preferred; the solver may exchange free positions for at most three rounds rather than relayout everything each frame. In fullscreen Earth, nearby foreground positions are allowed so cards of normal size do not cross the whole globe merely to avoid it. In slow mode, sustained direction drift or connector crossings must persist for 1.8 seconds before proposing a joint reflow. Automatic proposals are at least 8 seconds apart and need an improvement of at least 18% plus a minimum absolute gain. Manual/inertial input retains the 72ms/12px response budget. Expansion and manual placement grant 12 seconds of reading time; hover, keyboard focus and dragging continue to lock a card, while residual pointer focus does not lock it forever. Manual placement is a preference: after its hold, automatic repair requires at least 45 degrees of source direction change. Restored positions are not permanent pins. Rear connectors retain the globe-outline cue without changing card identity. Local position/size persistence and reset-to-auto are part of the controller. A `ResizeObserver` handles resizing and collapse; configuration and source windows retain shared drag handles.
Object presentation is shared through `PresentationController` and the new object-callout anchor/layout path. A click on a BGP, compute, vessel, satellite, or cable object draws the connector before placing the adjacent card. The source anchor begins at the actual click or original region label, then follows a visible highlight geometry sample as rotation approaches occlusion. The stable `layoutReferencePoint` is derived from the actual connector exit direction, so `packDirectionalEdgeCallouts` keeps cards on the corresponding screen edge and leaves the globe's center open; a card may move inward only when that edge lane is unavailable, and it may never cover its own connector exit point. The bounded joint assignment considers up to 24 beam states and 32 candidates, avoids existing HUD rectangles, then allows at most three safe exchanges to reduce directional crossings; the previous solver remains a fallback and the work is not a full relayout every frame. A slow source-direction gate also checks whether the actual card target reverses more than 90 degrees from the source before committing a change. Manual drag and hover locks take priority. The source point itself remains a per-frame visual connection target; fixed objects track live positions while cards remain draggable. Rear-side objects retain a globe-outline direction cue, a subdued connector, and a hollow source point; these are orientation guides rather than geographic points. Country candidates are capped at 96 with a hard limit of 128 and cached, so GeoJSON is not scanned every frame. Compute cards keep their original point independent from dynamic source-point movement, and the shared card endpoint slides along the obstacle perimeter. Cable presentation prefers the true ray-hit local point, then samples highlighted geometry by `cableId`. Mobile keeps the existing mini popup and opens drawer details. A stale asynchronous presentation must never dismiss a newer selection.
`configVisible` and `cardsEnabled` are separate. X/Escape closes only configuration; disabling the switch releases connectors and restores the previous map query/layer state. Configuration and source windows bind HUD typography directly so reparenting cannot enlarge their text. The main card presents the selected compute metric as a large share percentage or an absolute value with unit, observation year, historical context, and source entry. Missing comparable data is explicit; no empty trend chart is rendered without observations. Facility records and coverage remain complete in the expanded auxiliary area, where sample counts are evidence context rather than a compute value or global coverage rate. Long lists open only when needed. Mobile preserves the same single metric meaning in its accordion. Local position/size persistence and reset-to-auto remain available. Stable DOM, dimensions, and reading state survive updates, and stale responses cannot re-enable cards.
Motion presentation uses the shared `main.js` input state. Sustained drift or crossings trigger a joint assignment, rather than each card competing independently for leftover slots. `callout-region-path.js` provides the short connection and terminal stub used both for rendering and intersection scoring. Candidate paths are precomputed and layout states are cloned only after pruning. Existing cards remain visible during planning; queued reflows survive frame throttling.
The shared `CalloutConnector` calculates polyline length from its points instead of reading `getTotalLength()` after each SVG write, avoiding repeated synchronous layout during motion. Initial draw animation is retained, so existing cruise presentations receive the same optimization.
On mobile, compute uses a compact regional overview in the existing drawer. Rows are an in-place accordion: tapping the same header collapses it, tapping another collapses the previous row, and at most one expanded card is shown. The expanded card keeps trends, region members, facilities, media, and sources; there is no region subpage, while Settings remains a separate on-demand page. The shared HUD collapsible panel animation is reused and respects reduced motion. Summary loading remains active when map cards are off; enabling cards shares the map loading event and cancels stale requests. Drawer handlers use `composedPath()` so inner HTML replacement cannot collapse the active row. Facility names remain in a scroll region capped at 240px, with left-aligned wrapping rows and at least 42px touch height.
`/series` retains source-backed annual observations; the default primary metric is global compute-device scale split into basic, intelligent, and supercomputer capacity for 2020–2023. Reviewed absolute observations are 302 for 2022 (`caict2023`) and 435 for 2023 (`caict2024`); Japan's global share is 6%/5%/4%/4% for 2020–2023. Product groups APAC excluding China and Other do not map directly to report Japan or “other”; Europe remains `null` until membership aligns with the report. Bilateral share keeps a fixed US + China denominator. Missing primary values may show the latest reported year (currently 2023), while an explicit 2026 selection stays missing. Unconfirmed comparability does not produce year-over-year change. `/facilities` filters and counts unique canonical country codes in successful collection snapshots; coverage counts remain auxiliary, and are not capacity or a global coverage rate. `compute_catalog.py` groups China, US, Europe, APAC excluding China, and Other, covering 45 canonical country codes; these display groups never become statistical denominators. `earth_geography_policy.py` shares display ownership with map policy, grouping Taiwan/Hong Kong/Macao source subdivisions under China while retaining `source_country`. API review metadata is `reviewed_at=2026-09-22` with the matching version date. An independent Epoch share mode is not implemented. Innovation differences are supplied only when both CN and US are selected.
### Boundary surface attachment
Normal borders, coastlines, claim lines, highlights and landing points use `earth-surface.js` to share the HD sphere UV-triangle surface. Borders draw after the texture but before landing points. Highlight lines are always transparent-queue objects with depth testing, preventing through-globe display or overpainting markers. Geometry is computed and cached at construction, not reprojected per frame. See the [render-layer specification](earth-render-layer-order.md).
### News ticker edge fading
A separate pseudo-element paints the ticker background and horizontal rules, fading to full transparency outside the text bounds. The shell has no rectangular backdrop filter, box shadow, or text mask. The region label sizes to its content and the scrolling text follows directly without fading. Alert states change only the gradient rule color so they cannot reintroduce rectangular shadow edges.
Compute cards directly reuse the shared `.hud-panel` background, border, shadow, and glass treatment. Summary, essentials, and detail areas add no separate black surface. Their headers share inset spacing with the layer panel through `.hud-panel__header--inset` / `.layer-panel-header`; primary numbers remain prominent and essential information stays visible.
Saved placement restoration migrates legacy central positions: only old central coordinate locks without the current position marker return to automatic layout, retaining dimensions. Existing edge positions are preserved and versioned. New drags save `positionVersion=2` and continue to honor deliberate placement. The existing storage key is retained instead of clearing every personalized position.
### First frame and spatial card entrance
`earth-runtime-plugin.mjs` bundles local Earth modules into content-hashed entries and shared chunks for both development and production. The existing import map still pins third-party versions; relative assets and workers retain the `/earth/js/` base. `earth-preload.js` requests the selected full-size map early. `earth-texture-image.js` decodes an ImageBitmap asynchronously and transfers that same image to Three.js without resampling or changing texture color/sampling settings. Replaced and destroyed textures release their bitmaps. The full-resolution surface precedes independent concurrent data layers; data refresh retains it. Versioned boundary geometry, picking indexes and claim lines use HTTP caching and parallel requests. The live player engine is imported on demand; media and realtime connections start after the surface frame.
The root attributes `data-earth-first-frame-ms` and `data-earth-surface-ready-ms` record navigation-to-render timings. Measure cold and warm caches separately; local warm results are not a guarantee for all first-visit networks.
Positioned cards stay hidden while the shared connector draws for 420 ms, then fade in using the HUD 180 ms opacity timing. Targets remain fixed during drawing; subsequent motion does not replay entrance. Reduced-motion preferences skip this animation.
Object clicks, search focus and gesture confirmation share `PresentationController`. Cable selection updates selection/legend state without independently opening the card. InfoCard reveal callbacks are versioned so staging or hiding invalidates older callbacks. Connector completion gates card reveal; cancelled cruises no longer hide a newer selection. Object details reuse the regional-card edge candidates and collision scoring, with dimensions measured while staged. Selecting a mobile search result collapses the search drawer so the focused object, connector and compact card remain visible.

View File

@@ -143,6 +143,8 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
## Submarine Cables and Landing Points
`cable-batches.js` batches cable lines and landing points separately. The Sprite material and size parameters below remain owned by picking and selection proxies in `cables.js`; their color, opacity, scale, and visibility update the style texture or instance attributes. Drawing uses `LineSegments` and instanced billboards with the existing textures, colors, render order, and globe occlusion rules.
| Name | Variable | Current Value | Location / Notes |
| --- | --- | --- | --- |
| Default cable color | `CABLE_COLORS.default` | `0xffff44` | Used when no data color available |

View File

@@ -144,6 +144,8 @@ The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. T
- `categories`: comma-separated news category keys, for example `business,ecommerce`. Omit it when all categories are selected.
- `locale`: display locale, currently `zh-CN` or `en-US`, defaulting to `zh-CN`. Chinese RSS items are stored as Chinese source content and enriched with `en-US`; English RSS items are enriched with `zh-CN`.
For `locale=en-US`, the API must not fall back to Chinese source title/summary. If a Chinese source item has not yet received an `en-US` localization, `display_title` and `display_summary` stay empty so the Earth client can show an English pending/empty state instead of mixing languages.
Examples:
```http

View File

@@ -1,5 +1,7 @@
# Earth Render Layer Order
The compute topic reuses the compute-center Interactable icons, depth behavior, and layer order. Country/year changes replace that layer's data and clear prior selection. Its HUD is screen UI excluded from Earth picking in `main.js`. Regional cards are tethered in screen space; rear anchors do not draw connectors through the globe. Facility details reuse the existing card and mobile details drawer. The existing outer atmosphere shell remains, while the starfield is restored to commit `83a10a6c` (`brightStarOpacity:0.88`; no tint/brightness calibration). The HD overlay is a white `MeshBasicMaterial` with opacity 1 and `toneMapped:false`; `texture-colors.js` selects the r128 `encoding`/`LinearTosRGB` path or r160 `colorSpace`/colorspace fragment. No Phong white light, colored twilight, blue night tint, camera boost, or shoulder curve is added. No new post-processing, GPU draw pass, or high-resolution texture was added; the compute topic adds 45 local SVG flag assets.
This document records the current Earth renderer's layer order and the intent of each layer. When adjusting `renderOrder`, radius offsets, depth strategy, or pointer interaction, update this document accordingly.
Note: the layer control panel order and the registration / startup load order are two separate semantics.
@@ -20,12 +22,12 @@ Note: the layer control panel order and the registration / startup load order ar
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
| 1 | Submarine cables | `cables.js` | `CABLE_CONFIG.line.renderOrder` | Cable picking path | Preserves existing cable layer level. |
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order remains `1`; cable altitude offset stays `0.2`, while landing-point anchors use the HD surface offset `0.48` | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
| 2.29 | Country border hover glow | `country-boundaries.js` | Hover radius + glow offset | `depthTest: false`, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 2.3 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: false`, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
| 0.97–0.99 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
| 0.98 | Country border hover glow | `country-boundaries.js` | Same surface mapping, no extra lift | `depthTest: true`, transparent render queue, raycast disabled | Additive glow to reinforce border edge and terrain hover visibility. |
| 0.99 | Country border hover line | `country-boundaries.js` | `hoverAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | `depthTest: true`, transparent render queue, raycast disabled | Neon red-orange hover line; aligned with the normal borders and HD texture shell to avoid ghosting or floating; China and Taiwan share the same highlight group. |
| 3 | Satellite footprint fill / Iridium coverage ring | `satellites.js`, `iridium-footprint-adapter.js` | `GROUND_FOOTPRINT_RENDER_ORDER` | depth-tested; Iridium adapter fill / ring use the same renderOrder | Footprint above land / texture / terrain and country borders, below compute centers and satellites. |
| 3-4.5 | BGP collectors, event rings, and event markers | `bgp.js`, `interactable.js` | Collector markers use `BGP_COLLECTOR_RENDER_ORDER = 4.4`; event markers use `BGP_EVENT_RENDER_ORDER = 4.5`; overlays remain in BGP-owned groups | Screen-space Interactable picking; BGP collectors/events may use collision avoidance where appropriate | BGP collectors share the surface-facility band with vessels; BGP events share the compute-center band. |
| 4.3 | AIS vessel track lines | `vessels.js` | `VESSEL_RENDER_ORDER - 0.1`; `CONFIG.earthRadius + VESSEL_CONFIG.track.altitudeOffset` | Follows vessel selection, not independently picked | Recent track for the selected vessel, below vessel markers. |
@@ -36,6 +38,14 @@ Note: the layer control panel order and the registration / startup load order ar
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
| 98-100 | Sun / moon halo and sprite | `celestial.js` | Fixed renderOrder | Celestial picking disabled | Foreground celestial sprites. |
## Full-set rendering and updates
- Batching changes GPU submission, not the number of satellites, cable segments, or landing points. It does not filter data by the visible hemisphere. Every adjacent cable vertex pair remains present, and picking still returns the original business object.
- Satellite foreground and backdrop remain two `Points` draws. Vertex shaders compute breathing from static per-point parameters and a shared frame-time uniform; existing hover/locked state still controls point masking.
- `satellite-position-worker.js` uses the same Three.js / SGP4 versions as the main thread. `satellite-propagation.js` owns shared orbit, display-altitude, and fallback calculations. Full positions and initial trail samples use transferable arrays; only one calculation is in flight. Reload, clear, and altitude-mode changes terminate the previous Worker.
- Worker startup failure or `SATELLITE_CONFIG.workerStartupTimeoutMs` expiry falls back to the shared synchronous calculation. Counts, APIs, the existing update interval, and trail length are unchanged.
- Verify full draw ranges, per-item picking, locked overlays, trails, rear-side occlusion, visibility toggles, and resource release after clearing. Frame-time comparisons require identical data, view, and resolution.
## Toggle Behavior
| Toggle | Behavior |
@@ -65,7 +75,16 @@ Maintenance rules:
| --- | --- |
| Earth coordinate hover | When HD texture is visible, uses the HD texture overlay as the surface picking target; otherwise uses the Earth base sphere. |
| Country border hover | Converts surface pick coordinates to lat/lon, then uses GeoJSON point-in-polygon; the border hover line itself does not receive raycasts. |
| Country border hover visual | On hover, dims normal border lines and draws no-depth-test glow and solid lines. |
| Country border hover visual | On hover, dims normal border lines and draws depth-tested glow and solid lines below landing points. |
| China / Taiwan hover | `CHN` and `TWN` are grouped in the same hover highlight group; the tooltip still shows the actually-hit feature. |
| Terrain | Acts as a visual layer only; `terrain.raycast` is disabled. |
| Satellites | Uses screen-space satellite picking to prevent footprints or surface layers from blocking satellite clicks. |
### Surface attachment and landing point order
`earth-surface.js` projects geographic vertices onto the high-resolution SphereGeometry UV triangles at geometry construction time. Boundaries, coastlines, claim lines, hover outlines and landing points share this surface mapping. Long edges are subdivided using the surface grid spacing. Landing-point anchors now use the texture shell offset of 0.48; cable line offsets remain 0.2. The draw order is texture 0.96, boundaries/highlights 0.97–0.99, then landing points 1. Hover outlines always use the transparent queue with depth testing, so they do not draw through the globe or over landing markers. Geography is unchanged, and cached geometry is not rebuilt during rotation.
### Startup loading versus drawing order
The selected full-resolution surface loads before independent data requests; boundary construction no longer blocks its first frame. This does not change renderOrder, depth behavior, geometry precision or image resolution. Data refresh retains the surface, with texture disposal limited to successful replacement or scene teardown.

View File

@@ -1,5 +1,9 @@
# Admin Frontend Context
## Compute comparison configuration
The Earth content `compute_comparison` section mounts `ComputeComparisonSettings.tsx`. Structured forms maintain report sources, annual country observations and innovation gaps using the shared Panel, Input, Select, Button and confirmation dialog; operators do not edit JSON. Drafts survive section switches, full-page unload warns about unsaved work, and discarding a draft or restoring defaults requires confirmation. Management data loads on activation. Publishing sends the backend revision to prevent overwriting concurrent edits, and failures preserve the draft. The content area scrolls internally while the heading and publish controls stay in the first viewport. `compute-comparison-model.ts` owns types and same-origin update notifications; `compute-comparison.css` only owns this section's layout.
This document describes the current real structure of the console frontend. The goal is to help future page development, table refactoring, layout governance, and state consolidation quickly find the right entry points.
Related references:
@@ -98,6 +102,8 @@ Admin status labels should use [StatusText](/home/ray/dev/linkong/planet/fronten
`StatusText` is an indicator-light pill: the pill background and border stay on the component base color, while only the dot and text use the status color. `Badge` does not carry the indicator-light meaning, so it may use a light same-tone background and border for stronger hierarchy.
Status indicators must show the full state word. In lists, hierarchy groups, and detail headers, the title/description area should shrink or wrap while the status pill keeps content-sized width and does not get compressed by flex/grid layout; do not truncate state words such as `Configured` or `Available` just to save horizontal space.
| Tone | Color variable | Meaning | Examples |
| --- | --- | --- | --- |
| `success` | `--an-success` | available, successful, connected, enabled | log source `Available`, collection `Success` |
@@ -155,7 +161,6 @@ Purpose:
Current usage:
- Admin data sources, collected data, collection management, logs, alerts, and BGP pages
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
### 3. `TableScrollRegion`
@@ -217,7 +222,31 @@ Current constraints:
- Prefer CSS variable overrides for colors instead of hard-coding theme colors in feature components
- Best for a small set of mutually exclusive choices; do not use it as a long list, navigation menu, or select replacement
### 6. `MarkdownRenderer`
### 6. Console i18n
Files:
- [i18n/index.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/index.ts)
- [i18n/locale.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/locale.ts)
- [i18n/resources.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/resources.ts)
- [LegacyI18nBridge.tsx](/home/ray/dev/linkong/planet/frontend/src/i18n/LegacyI18nBridge.tsx)
Purpose:
- Share one `zh-CN` / `en-US` language state across the console, auth pages, and Docs UI
- Store the language preference in `planet-locale` while keeping compatibility with the old `docs-lang`
- Keep Docs API requests mapped to the backend's existing `zh` / `en` document interface
- Provide language switchers in the console sidebar preferences panel and auth panel
Current constraints:
- New console copy should be added to `resources.ts`, then consumed with `useTranslation()` or `useLocale()`
- Routes, menus, search indexes, and shared components must use explicit translation keys
- `LegacyI18nBridge` is transitional and only handles exact static text and attributes inside admin/auth containers
- Business data, raw logs, API field names, provider ids, commands, and Markdown body content are not translated by the legacy bridge
- Future large-page migrations should shrink the legacy dictionary rather than grow it
### 7. `MarkdownRenderer`
File:
@@ -275,17 +304,16 @@ File:
Responsibilities:
- `/ai` now owns LLM Provider, AI Tool configuration, and the testbench instead of nesting them under `/settings`
- The `模型供应商` tab manages default provider, model, base URL, provider key, local `aiprovider` proxy, and connection test; provider and model fields use editable comboboxes so users can manually enter new providers/models if the models.dev catalog stops updating
- The `工具` tab first selects a tool from a dropdown menu, then renders that tool's configuration; it currently includes WebSearch and OCR
- `/ai` now owns LLM Provider, AI Tool configuration, and Playground instead of nesting them under `/settings`
- The `模型供应商` section manages default provider, model, base URL, provider key, local `aiprovider` proxy, and connection test; provider and model fields use editable comboboxes so users can manually enter new providers/models if the models.dev catalog stops updating
- The `工具调用` section first selects a tool from a dropdown menu, then renders that tool's configuration; it currently includes WebSearch and OCR
- WebSearch configuration includes provider, search key, base URL, timeout, result count, and advanced provider options
- OCR configuration includes provider, Base URL, API key, model/engine, languages, timeout, file-size limit, and output format
- The `测试台` tab embeds the former Playground real session, preset prompts, and AI Provider status debugging
- The page reuses the Settings single-screen tabs, panel card, and internal scrolling style
- The `Playground` section embeds the former Playground real session, preset prompts, and AI Provider status debugging
- The page reuses the Settings single-screen section, panel card, and internal scrolling style
- AI Provider and WebSearch connection tests use `ConnectionTestInput`, with the connector icon fixed at the end of the Base URL input; when WebSearch is disabled, every configuration field and the test entry point are greyed out except the switch
Legacy `/settings?tab=ai` should redirect to `/ai?tab=providers`.
Legacy `/playground` should redirect to `/ai?tab=playground`.
AI configuration no longer lives under `/settings`; `/playground` should redirect to `/ai?section=playground`.
### 3. Business Data Gateway
@@ -342,11 +370,11 @@ Current page boundary:
- Endpoint, headers, and config are displayed here, not edited.
- Credential-bearing collectors point users to `Collection Management -> Collectors`.
Keep this boundary: do not put custom datasource editing, built-in endpoint overrides, or credential forms back into `/datasources`. Those configuration entry points live at `/collection-management?tab=collector_credentials`.
Keep this boundary: do not put custom datasource editing, built-in endpoint overrides, or credential forms back into `/datasources`. Those configuration entry points live at `/collection-management?section=collector_credentials`.
### Collectors Page
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` section is shown as `Collectors` under `/collection-management`.
The `System Display` section under `/settings` includes the `Demo Mode` switch. When enabled, Earth OOBE ignores existing current collected data and the local `browse first` temporary skip state, then opens the initialization guide directly. This switch is only for demos and acceptance checks; it does not change datasources, collection queues, or Earth content resources.
@@ -363,6 +391,7 @@ Current boundary:
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), but its ownership is separate from System Settings:
- `TV Livestream` owns the Earth media-panel source configuration.
- `Branding` owns Earth HUD brand assets. `Logo URL` and `Title Image URL` use inline upload controls inside the fields; the upload buttons keep the primary `TactileButton` style, and dropping an image onto the matching field shows a low-saturation drag reaction instead of the old global asset-picker toolbar.
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
@@ -391,6 +420,7 @@ These principles have been repeatedly validated in the project:
4. Do not use `overflow: hidden` to mask structural issues
5. Do not compress the main work area to make summary cards show completely
6. Custom scrollbars must be floating overlays; they must not squeeze content width
7. The Admin shell relies on the root `height: 100%` chain and should not use exact `100vh` sizing at the workspace root
For detailed experience, see:
@@ -411,7 +441,7 @@ Do not write local CSS patches first, then retrofit the structure.
The console frontend and the Earth frontend are not the same system:
- Console frontend: React + Ant Design workbench
- Console frontend: React + Tactile UI / Radix primitives / lucide workbench
- Earth frontend: independent native HUD system under `public/earth`
Therefore:

Some files were not shown because too many files have changed in this diff Show More