Compare commits
17 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
67548d8278 | ||
|
|
4b8feeb87e | ||
|
|
a2025dbfe0 | ||
|
|
003b316d42 | ||
|
|
0c40764c58 | ||
|
|
5bfc790dec | ||
|
|
60eefb19c9 | ||
|
|
83a10a6c34 | ||
|
|
cee1996809 | ||
|
|
58671e7bc3 | ||
|
|
a54fcdbeed | ||
|
|
1dd2921674 | ||
|
|
d30f7d08c5 | ||
|
|
5bdb55f3f1 | ||
|
|
fbecf30513 | ||
|
|
19d5ac0fee | ||
|
|
3265d22af5 |
@@ -1,139 +0,0 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
description: Create or update repository documentation from current code changes
|
||||
argument-hint: Optional: topic to document, or leave empty to infer from git diff
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — Documentation Workflow
|
||||
|
||||
## Goal
|
||||
|
||||
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
|
||||
|
||||
## Repository Rules
|
||||
|
||||
Before deciding scope, check whether the repository has a documentation rules file:
|
||||
|
||||
```bash
|
||||
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
|
||||
```
|
||||
|
||||
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Understand The Change
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
rg --files docs
|
||||
```
|
||||
|
||||
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — Decide Scope
|
||||
|
||||
- Prefer updating an existing relevant document over creating a duplicate.
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents only when the change crosses meaningful domains.
|
||||
- Keep filenames lowercase and hyphenated.
|
||||
- Apply the repository-specific rules file before writing.
|
||||
|
||||
#### Document Audience Routing (Planet)
|
||||
|
||||
In this repository, classify the action's performer before picking a target file:
|
||||
|
||||
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
|
||||
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
|
||||
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
|
||||
|
||||
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
|
||||
|
||||
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
|
||||
|
||||
### Step 3 — Write
|
||||
|
||||
Explain:
|
||||
|
||||
- Background/problem: what was wrong or missing before.
|
||||
- Core design decisions and rationale.
|
||||
- Operational or user-facing impact.
|
||||
- Relevant code paths, only when useful for future maintainers.
|
||||
|
||||
Style:
|
||||
|
||||
- Follow the repository’s existing language and heading conventions.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Prefer tables for comparisons or parameter lists.
|
||||
- Keep snippets concise and relevant.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
### Step 4 — Verify
|
||||
|
||||
- Read the completed docs once for clarity and stale statements.
|
||||
- Verify referenced paths exist with `test -e` or `rg --files`.
|
||||
- Run applicable checks from `docs/documentation-coverage-rules.md`.
|
||||
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
|
||||
|
||||
### Step 5 — Report
|
||||
|
||||
Summarize changed docs and verification:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- path/to/doc.md — what changed
|
||||
|
||||
Verified:
|
||||
- checks that passed
|
||||
- checks that could not be run, if any
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- Do not leave placeholder docs.
|
||||
- Do not duplicate bilingual files byte-for-byte.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
|
||||
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
|
||||
- Keep docs maintainable and concise.
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -1,160 +0,0 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
BIN
.codex/screenshots/earth-i18n-current-page.png
Normal file
|
After Width: | Height: | Size: 1.2 MiB |
BIN
.codex/screenshots/earth-i18n-hd-texture-status.png
Normal file
|
After Width: | Height: | Size: 4.4 KiB |
BIN
.codex/screenshots/i18n-admin-data-sidebar.png
Normal file
|
After Width: | Height: | Size: 109 KiB |
BIN
.codex/screenshots/i18n-ai-tool-calls.png
Normal file
|
After Width: | Height: | Size: 124 KiB |
BIN
.codex/screenshots/i18n-earth-brand-config.png
Normal file
|
After Width: | Height: | Size: 170 KiB |
BIN
.codex/screenshots/i18n-earth-hud-brand.png
Normal file
|
After Width: | Height: | Size: 772 KiB |
BIN
.codex/screenshots/i18n-settings-notifications.png
Normal file
|
After Width: | Height: | Size: 95 KiB |
BIN
.codex/screenshots/i18n-settings-system.png
Normal file
|
After Width: | Height: | Size: 96 KiB |
BIN
.codex/screenshots/sidebar-left-align.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
@@ -5,6 +5,15 @@
|
||||
!VERSION
|
||||
!backend/
|
||||
!backend/**
|
||||
!config/
|
||||
!config/earth-boundary-pov-policy.china-v1.json
|
||||
!scripts/
|
||||
!scripts/build_earth_boundary_china_pov_geojson.py
|
||||
!scripts/build_earth_boundary_pmtiles.py
|
||||
!scripts/earth_boundary_policy.py
|
||||
!resources/
|
||||
!resources/earth/
|
||||
!resources/earth/boundary-pov-geometry.china-v1.json
|
||||
!aiprovider/
|
||||
!aiprovider/**
|
||||
|
||||
|
||||
5
.gitignore
vendored
@@ -160,3 +160,8 @@ data/earth-boundary-sources/
|
||||
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
|
||||
# committing thousands of generated tile files.
|
||||
frontend/public/earth/data/boundaries/
|
||||
|
||||
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
|
||||
resources/earth/candidates/nasa-blue-marble/*.jpg
|
||||
resources/earth/candidates/natural-earth-3/*.jpg
|
||||
*:Zone.Identifier
|
||||
|
||||
183
AGENTS.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# AGENTS.md
|
||||
|
||||
**Planet agent harness. Defines behavior for coding agents working in this repository.**
|
||||
|
||||
---
|
||||
|
||||
## Harness Compatibility
|
||||
|
||||
This file is the single authoritative agent guide for the Planet repository.
|
||||
The older lowercase `agents.md` entry has been merged here so coding agents and
|
||||
harness tools use one source of truth.
|
||||
|
||||
### Source Of Truth
|
||||
|
||||
- `rules.md` is the mandatory repository rule source. Always load `core`,
|
||||
`security`, and `workflow`; load only task-relevant modules after that.
|
||||
- `AGENTS.md` defines the local agent operating mode and evidence gates.
|
||||
- `project_context.md` is background, not a rule source. Prefer newer
|
||||
implementation docs when it disagrees with current code.
|
||||
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
|
||||
goal-driven work, and release.
|
||||
- Do not duplicate long workflow text across harness files. Durable constraints
|
||||
belong in `rules.md`; task procedures belong in skills or scripts.
|
||||
|
||||
Read these files before changing code:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. `project_context.md`
|
||||
4. `README.md`
|
||||
5. `docs/HARNESS.md`
|
||||
6. `CODEMAP.md`
|
||||
|
||||
For documentation work, also read `docs/documentation-coverage-rules.md`.
|
||||
|
||||
### Start Safely
|
||||
|
||||
Before broad edits:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Use focused context commands before reading large files:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
Preserve user changes already present in the worktree.
|
||||
|
||||
### Validation
|
||||
|
||||
Fast local harness validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/quick-check.sh
|
||||
```
|
||||
|
||||
Full local validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
|
||||
unless disabled by its documented environment flags. Docker image smoke builds
|
||||
are intentionally opt-in:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
|
||||
current non-interactive environment first. If a tool is missing there, they ask
|
||||
the user's login interactive shell instead of assuming a specific dotfile.
|
||||
|
||||
### High-Risk Areas
|
||||
|
||||
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
|
||||
`destroy` cleanup.
|
||||
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
|
||||
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
|
||||
rendered smoke evidence for public pages, auth guards, authenticated admin
|
||||
route/section availability, safe navigation/search/tab interactions, mobile
|
||||
layout, and 125% / 150% zoom, not only a build.
|
||||
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
|
||||
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
|
||||
intermediate wrappers keep `min-height: 0`, and only the intended child owns
|
||||
scrolling.
|
||||
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
|
||||
workflows in the backend.
|
||||
- Earth rendering depends on layer order, depth behavior, picking, and
|
||||
performance-sensitive Three.js code.
|
||||
- Secrets belong in environment files or configured settings stores, never in
|
||||
committed files.
|
||||
- Backend service code must use structured logging instead of `print()` or
|
||||
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
|
||||
|
||||
### Conflict Policy
|
||||
|
||||
Existing project rules and workflows win. If new harness guidance conflicts with
|
||||
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
|
||||
behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
`docs/HARNESS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
|
||||
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
|
||||
and Astra for complex reasoning; the main agent owns final verification.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
|
||||
focused file reads, tests, builds, linters, and harness scripts.
|
||||
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
|
||||
release work, and documentation unless the task requires it.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gates
|
||||
|
||||
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
|
||||
mock, browser capture, or visual reference, obtain evidence from the artifact
|
||||
before interpreting intent or editing code.
|
||||
- Path resolution is part of the task. If the path cannot be opened, first try
|
||||
reasonable local equivalents such as WSL/Windows path conversion,
|
||||
workspace-relative lookup, absolute paths, and attached-file locations.
|
||||
- Never guess from prompt text, filenames, previous context, logs, OCR, or
|
||||
memory when a visual artifact was provided but cannot be accessed.
|
||||
- OCR is acceptable evidence for text-only visual questions or non-multimodal
|
||||
environments; state that OCR was used as the fallback. Layout, color, spacing,
|
||||
pixel, and rendering issues need real visual inspection or a clear limitation
|
||||
note.
|
||||
- If a visual artifact still cannot be inspected, say so and pause that
|
||||
visual-dependent part of the work.
|
||||
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
|
||||
diff, direct file check, or harness result.
|
||||
- For UI and rendering changes, verify the rendered result when local tooling
|
||||
allows it.
|
||||
|
||||
---
|
||||
|
||||
## Communication
|
||||
|
||||
- Match the user's language. Use Chinese for Chinese requests unless the user
|
||||
asks otherwise.
|
||||
- Keep updates short and specific: what is being inspected, edited, or verified.
|
||||
- Final responses should summarize changed files and verification, with blockers
|
||||
stated plainly.
|
||||
- Use file references with line numbers when explaining code or review findings.
|
||||
|
||||
---
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Prefer existing project patterns over new abstractions.
|
||||
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
|
||||
a stable path exists.
|
||||
- Centralize prompts, constants, defaults, and shared request/response handling.
|
||||
- Do not add secrets, generated runtime output, or local environment files.
|
||||
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
|
||||
- Run the smallest relevant verification for the changed scope and report
|
||||
anything skipped.
|
||||
|
||||
---
|
||||
|
||||
## Prohibited
|
||||
|
||||
- Do not skip visual evidence handling when a visual artifact was provided.
|
||||
- Do not preserve obsolete harness files just because they already exist.
|
||||
- Do not invent behavior not present in code, docs, or verified external
|
||||
sources.
|
||||
- Do not rewrite unrelated files during cleanup.
|
||||
- Do not mark a task complete without checking concrete success criteria.
|
||||
110
CODEMAP.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Code Map
|
||||
|
||||
This map gives agents and maintainers a quick orientation without replacing the
|
||||
deeper architecture docs. Current implementation docs under `docs/technical/`
|
||||
are the source of detail for specific subsystems.
|
||||
|
||||
## Top-Level Areas
|
||||
|
||||
| Path | Role | Notes |
|
||||
| --- | --- | --- |
|
||||
| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
|
||||
| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
|
||||
| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
|
||||
| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
|
||||
| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
|
||||
| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
| Runtime | Entry Point | Validation |
|
||||
| --- | --- | --- |
|
||||
| Local full stack | `./planet.sh start` | `./planet.sh health` |
|
||||
| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
|
||||
| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
|
||||
| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
|
||||
| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
|
||||
| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
|
||||
|
||||
## Ownership Boundaries
|
||||
|
||||
- Backend owns business state, auth, evidence collection, prompt selection, AI
|
||||
task orchestration, and database persistence.
|
||||
- `aiprovider` owns provider identity, request adapter style, model gateway
|
||||
retries, and health/status endpoints only.
|
||||
- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
|
||||
and client-side state that mirrors backend truth.
|
||||
- Web Earth rendering changes must preserve documented layer order, altitude
|
||||
offsets, picking behavior, legend semantics, and performance constraints.
|
||||
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
|
||||
wrapping it from harness scripts instead of duplicating its internals.
|
||||
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
|
||||
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
|
||||
interactive command path without hardcoding `.zshrc`.
|
||||
|
||||
## Validation Commands
|
||||
|
||||
```bash
|
||||
scripts/harness/doctor.sh
|
||||
scripts/harness/security-check.sh
|
||||
scripts/harness/backend-rules-check.sh
|
||||
scripts/harness/frontend-rules-check.sh
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
scripts/harness/quick-check.sh
|
||||
scripts/harness/validate.sh
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
CI-equivalent local checks:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
|
||||
|
||||
cd frontend
|
||||
bun install --frozen-lockfile
|
||||
bun run build
|
||||
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
|
||||
```
|
||||
|
||||
The frontend smoke covers public routes, unauthenticated admin guards,
|
||||
login-error handling, the Earth iframe entry, and authenticated `super_admin`
|
||||
admin route/section rendering with mocked API data. Authenticated admin checks
|
||||
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
|
||||
check for accidental global horizontal overflow. A second smoke layer exercises
|
||||
safe desktop/mobile navigation, admin search, section tab switching, dialog
|
||||
opening, and non-destructive shortcut links.
|
||||
|
||||
Optional delivery smoke, when Docker and Helm are available:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
## Deeper Docs
|
||||
|
||||
| Topic | Start Here |
|
||||
| --- | --- |
|
||||
| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
|
||||
| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
|
||||
| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
|
||||
| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
|
||||
| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Harness workflow | `docs/HARNESS.md` |
|
||||
|
||||
## Known Sharp Edges
|
||||
|
||||
- `project_context.md` is static background for agents. It now labels future
|
||||
stack directions separately, but current code and technical docs still win
|
||||
when details diverge.
|
||||
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
|
||||
active local development shape.
|
||||
- Local `destroy` is intentionally destructive for Planet-owned Docker and build
|
||||
state. Never run it as a validation shortcut.
|
||||
@@ -83,7 +83,7 @@
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| React 18 | UI 框架 |
|
||||
| Ant Design Pro | 管理后台组件 |
|
||||
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
@@ -168,10 +168,12 @@
|
||||
|
||||
## 快速启动
|
||||
|
||||
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
|
||||
|
||||
```bash
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
|
||||
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
|
||||
|
||||
@@ -322,7 +324,7 @@ ipconfig
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
|
||||
231
agents.md
@@ -1,231 +0,0 @@
|
||||
# agents.md
|
||||
|
||||
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
|
||||
|
||||
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
|
||||
- Python FastAPI backend
|
||||
- React Admin dashboard
|
||||
- Unreal Engine 5 3D visualization
|
||||
- Multi-source data collection
|
||||
- Polarized 3D large display (4K, 120Hz)
|
||||
|
||||
---
|
||||
|
||||
## Communication Style
|
||||
|
||||
### Tone
|
||||
- **Professional but concise**
|
||||
- Technical accuracy with clarity
|
||||
- No unnecessary verbosity
|
||||
- Use code comments sparingly (explain **why**, not **what**)
|
||||
|
||||
### When Responding
|
||||
1. **Answer directly** - 1-3 sentences for simple questions
|
||||
2. **Use code blocks** for all code snippets
|
||||
3. **Include file:line_number** references when discussing code
|
||||
4. **Never** start with "I am an AI assistant" or similar phrases
|
||||
5. **Never** add unnecessary preambles/postambles
|
||||
|
||||
### Examples
|
||||
|
||||
**Good:**
|
||||
```
|
||||
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
|
||||
```
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operational Mode
|
||||
|
||||
### Plan Mode (default for complex tasks)
|
||||
- Analyze requirements
|
||||
- Propose architecture
|
||||
- Confirm with user before execution
|
||||
- **DO NOT** write code until approved
|
||||
|
||||
### Build Mode (after user approval)
|
||||
- Execute the approved plan
|
||||
- Write code, run commands
|
||||
- Verify results
|
||||
- Report completion concisely
|
||||
|
||||
### Read-Only Mode
|
||||
- Analyze code
|
||||
- Explain functionality
|
||||
- Answer questions
|
||||
- **DO NOT** modify files
|
||||
|
||||
---
|
||||
|
||||
## Decision Framework
|
||||
|
||||
### When to Ask Before Acting
|
||||
- Unclear requirements
|
||||
- Multiple implementation approaches
|
||||
- Architecture changes
|
||||
- Dependency additions
|
||||
- Anything that could break existing functionality
|
||||
|
||||
### When to Act Directly
|
||||
- Clear, approved requirements
|
||||
- Routine tasks (linting, formatting, running tests)
|
||||
- Following established patterns
|
||||
- Fixing obvious bugs
|
||||
|
||||
### When to Refuse
|
||||
- Malicious code requests
|
||||
- Security violations (secrets, credentials)
|
||||
- Anything that violates `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## Working Principles
|
||||
|
||||
### 1. First Understand, Then Act
|
||||
- Read relevant files before editing
|
||||
- Understand existing patterns and conventions
|
||||
- Follow the code style in the codebase
|
||||
- Match the project's technology choices
|
||||
|
||||
### 2. Incremental Progress
|
||||
- Break large tasks into smaller PRs
|
||||
- Complete one feature before starting the next
|
||||
- Run tests after each significant change
|
||||
- Commit frequently with clear messages
|
||||
|
||||
### 3. Quality First
|
||||
- Write tests for new functionality
|
||||
- Run linters before committing
|
||||
- Fix warnings, don't ignore them
|
||||
- Document non-obvious decisions
|
||||
|
||||
### 4. Communication Clarity
|
||||
- Use precise technical language
|
||||
- Show relevant code, not explanations
|
||||
- Report errors with context
|
||||
- Confirm understanding of requirements
|
||||
|
||||
---
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
Before marking a task complete:
|
||||
|
||||
- [ ] Code follows `rules.md` style guidelines
|
||||
- [ ] Type hints are correct and complete
|
||||
- [ ] Error handling is proper (no silent failures)
|
||||
- [ ] Tests pass locally
|
||||
- [ ] Linting passes
|
||||
- [ ] No TODO comments left behind
|
||||
- [ ] Documentation updated if needed
|
||||
- [ ] Commit message is clear
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Feature Development
|
||||
```
|
||||
1. Understand requirements
|
||||
2. Check existing patterns in codebase
|
||||
3. Design solution (brief mental model)
|
||||
4. Write code following rules.md
|
||||
5. Write/run tests
|
||||
6. Lint and format
|
||||
7. Commit with clear message
|
||||
8. Report completion
|
||||
```
|
||||
|
||||
### Bug Fix
|
||||
```
|
||||
1. Reproduce the bug (write failing test)
|
||||
2. Locate the source
|
||||
3. Fix the issue
|
||||
4. Verify test passes
|
||||
5. Check for regressions
|
||||
6. Commit fix
|
||||
```
|
||||
|
||||
### Refactoring
|
||||
```
|
||||
1. Understand current behavior
|
||||
2. Design target state
|
||||
3. Make incremental changes
|
||||
4. Preserve tests
|
||||
5. Verify functionality
|
||||
6. Clean up dead code
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Special Considerations
|
||||
|
||||
### WebSocket Services
|
||||
- Implement heartbeat mechanism (30-second intervals)
|
||||
- Handle disconnection gracefully
|
||||
- Include camera position in control frames
|
||||
- Support both update and full sync modes
|
||||
|
||||
### Data Collectors
|
||||
- Inherit from BaseCollector
|
||||
- Implement fetch() and transform() methods
|
||||
- Support incremental updates
|
||||
- Handle API changes gracefully
|
||||
|
||||
### UE5 Integration
|
||||
- Communicate via WebSocket
|
||||
- Send data frames at configurable intervals (default 5 min)
|
||||
- Support auto-cruise and manual modes
|
||||
- Optimize for 4K@120Hz rendering
|
||||
|
||||
### Multi-User Security
|
||||
- JWT tokens with 15-minute expiration
|
||||
- Redis token blacklist for logout
|
||||
- Role-based access control (RBAC)
|
||||
- Audit logging for all actions
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
### When Writing Code
|
||||
```python
|
||||
# File: backend/app/services/collectors/top500.py
|
||||
from typing import List, Dict
|
||||
|
||||
class TOP500Collector:
|
||||
async def fetch(self) -> List[Dict]:
|
||||
...
|
||||
```
|
||||
|
||||
### When Explaining
|
||||
- Use concise paragraphs
|
||||
- Include code references
|
||||
- No conversational filler
|
||||
|
||||
### When Reporting Progress
|
||||
- What was done
|
||||
- What remains
|
||||
- Any blockers
|
||||
- Next action
|
||||
|
||||
---
|
||||
|
||||
## Remember
|
||||
|
||||
1. **Rules are hard constraints** - follow `rules.md` absolutely
|
||||
2. **Context provides understanding** - use `project_context.md` for background
|
||||
3. **Role defines behavior** - follow `agents.md` for how to work
|
||||
4. **Quality over speed** - Enterprise systems require precision
|
||||
5. **Communicate clearly** - Precision in, precision out
|
||||
@@ -1,4 +1,4 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=openai-responses`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import NAMESPACE_URL, uuid4, uuid5
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
@@ -65,6 +66,7 @@ class ProviderService:
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
self.session_id = str(uuid4())
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -95,6 +97,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
if payload.context.get("session_id") is not None:
|
||||
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
|
||||
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
@@ -102,6 +106,10 @@ class ProviderService:
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif provider_api == "openai-responses":
|
||||
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
|
||||
content_blocks = self._extract_responses_blocks(data)
|
||||
content = "".join(block.text for block in content_blocks if block.text)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
@@ -200,6 +208,39 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_openai_responses(
|
||||
self, model: str, prompt: str, system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body: dict[str, Any] = {
|
||||
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["instructions"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/responses",
|
||||
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in payload.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "message":
|
||||
for part in item.get("content") or []:
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
text = part.get("text") or part.get("refusal")
|
||||
if isinstance(text, str) and text:
|
||||
blocks.append(AIContentBlock(type="text", text=text))
|
||||
elif item.get("type") == "reasoning":
|
||||
for part in item.get("summary") or []:
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
|
||||
return blocks
|
||||
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
@@ -226,7 +267,7 @@ class ProviderService:
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
@@ -243,8 +284,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
def _resolve_anthropic_thinking(
|
||||
self, thinking: dict[str, Any] | None, model: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
|
||||
return {"type": "adaptive"}
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
@@ -295,6 +340,9 @@ class ProviderService:
|
||||
headers: dict[str, str],
|
||||
request_body: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
headers = {"User-Agent": "Planet/1.0", **headers}
|
||||
if self.provider == "opencode-go":
|
||||
headers["x-opencode-session"] = self.session_id
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.http_retry_attempts + 1):
|
||||
try:
|
||||
|
||||
@@ -28,6 +28,11 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
|
||||
COPY backend /app/backend
|
||||
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
|
||||
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
|
||||
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
|
||||
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
|
||||
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
|
||||
COPY VERSION /app/VERSION
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
@@ -12,6 +12,7 @@ from app.api.v1 import (
|
||||
alerts,
|
||||
settings,
|
||||
collected_data,
|
||||
compute_intelligence,
|
||||
data_products,
|
||||
layers,
|
||||
visualization,
|
||||
@@ -43,6 +44,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
|
||||
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
|
||||
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
|
||||
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(
|
||||
|
||||
231
backend/app/api/v1/compute_intelligence.py
Normal file
@@ -0,0 +1,231 @@
|
||||
"""Compute observations, administrator configuration and independent facility samples."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.core.earth_geography_policy import display_country
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import (
|
||||
ComputeConfiguration,
|
||||
ComputeConfigurationReset,
|
||||
ComputeConfigurationWrite,
|
||||
)
|
||||
from app.services.compute_catalog import (
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
from app.services.compute_center_locations import refresh_compute_center_location_cache
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
configuration_response,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
router = APIRouter()
|
||||
SOURCES = ("top500", "epoch_ai_gpu")
|
||||
|
||||
|
||||
def selected_countries(countries: str) -> list[str]:
|
||||
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
|
||||
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
|
||||
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
|
||||
members = participant_members(codes)
|
||||
flattened = [country for participant in members.values() for country in participant]
|
||||
if len(flattened) != len(set(flattened)):
|
||||
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
|
||||
return codes
|
||||
|
||||
|
||||
@router.get("/series")
|
||||
async def get_series(
|
||||
response: Response,
|
||||
countries: str = "CN,US",
|
||||
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
envelope = await read_configuration(db)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
|
||||
result.update(
|
||||
version=envelope["revision"],
|
||||
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response,
|
||||
_actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return configuration_response(await read_configuration(db))
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def update_configuration(
|
||||
payload: ComputeConfigurationWrite,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=payload.config, actor=actor,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
@router.post("/configuration/reset")
|
||||
async def reset_configuration(
|
||||
payload: ComputeConfigurationReset,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
|
||||
now = datetime.now(UTC)
|
||||
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
|
||||
ranked = (
|
||||
select(
|
||||
DataSnapshot.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=DataSnapshot.source,
|
||||
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
|
||||
)
|
||||
.label("rank"),
|
||||
)
|
||||
.where(
|
||||
DataSnapshot.source.in_(SOURCES),
|
||||
DataSnapshot.status == "success",
|
||||
DataSnapshot.completed_at < cutoff,
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
|
||||
# Old collector rows have ambiguous units/identity and must not become
|
||||
# national evidence merely because a new UI can display them.
|
||||
query = select(CollectedData).where(
|
||||
CollectedData.snapshot_id.in_(snapshots),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.extra_data["compute_contract"].as_string() == "v1",
|
||||
or_(
|
||||
CollectedData.source == "top500",
|
||||
and_(
|
||||
CollectedData.extra_data["status"].as_string() == "Existing",
|
||||
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
|
||||
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
|
||||
),
|
||||
),
|
||||
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
|
||||
)
|
||||
return query
|
||||
|
||||
|
||||
@router.get("/facilities")
|
||||
async def get_facilities(
|
||||
countries: str = "CN,US",
|
||||
year: int | None = Query(None, ge=2000, le=2100),
|
||||
limit: int = Query(1000, ge=1, le=2000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
query = await sample_query(db, codes, year)
|
||||
country = CollectedData.extra_data["country"].as_string()
|
||||
source_participants = source_country_participants(codes)
|
||||
# Canonical source subdivisions retain their raw label on each record but
|
||||
# share one owner in the comparison, using the same policy as the globe.
|
||||
owner_cases = [
|
||||
(country == name, display_country(name))
|
||||
for name in source_country_names(codes)
|
||||
if display_country(name) != name
|
||||
]
|
||||
display = case(*owner_cases, else_=country) if owner_cases else country
|
||||
participant_cases = [(country == name, code) for name, code in source_participants.items()]
|
||||
participant = case(*participant_cases, else_=None)
|
||||
counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
display.label("country"),
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(display, participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
participant_counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
records = (
|
||||
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if year:
|
||||
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
|
||||
else:
|
||||
await refresh_compute_center_location_cache(db)
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
for feature in payload["features"]:
|
||||
feature["properties"]["historical_snapshot"] = year is not None
|
||||
properties = feature["properties"]
|
||||
properties["source_country"] = properties.get("country")
|
||||
properties["country"] = display_country(properties.get("country"))
|
||||
total = sum(row["count"] for row in counts)
|
||||
payload.update(
|
||||
{
|
||||
"counts": [dict(row) for row in counts],
|
||||
"participant_counts": [dict(row) for row in participant_counts],
|
||||
"participants": [PARTICIPANTS[code] for code in codes],
|
||||
"total_count": total,
|
||||
"returned_count": len(records),
|
||||
"truncated": total > limit,
|
||||
"reference_year": year,
|
||||
"scope": "public_source_records",
|
||||
"history_basis": (
|
||||
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
|
||||
),
|
||||
"coverage": "available" if total else "no_verified_snapshot",
|
||||
"cross_source_deduplicated": False,
|
||||
}
|
||||
)
|
||||
return payload
|
||||
@@ -3,9 +3,9 @@ from datetime import UTC, datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
|
||||
defaults = get_fallback_llm_provider_preset(
|
||||
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
|
||||
)
|
||||
else:
|
||||
defaults = DEFAULT_SETTINGS[category]
|
||||
merged = deepcopy(defaults)
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
|
||||
|
||||
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
|
||||
merged = merge_with_defaults(category, payload)
|
||||
record = await get_setting_record(db, category)
|
||||
if record is None:
|
||||
record = SystemSetting(category=category, payload=payload)
|
||||
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return merge_with_defaults(category, record.payload)
|
||||
return merged
|
||||
|
||||
|
||||
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
|
||||
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
|
||||
)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
|
||||
model_provider_apis = {
|
||||
**(provider_config.get("model_provider_apis") or {}),
|
||||
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
|
||||
}
|
||||
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": _resolve_service_token(normalized_ai)[0],
|
||||
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
"model_provider_apis": model_provider_apis,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = (
|
||||
str(llm_config.get("provider_api") or "").strip()
|
||||
or ProviderApi.OPENAI_COMPLETIONS.value
|
||||
)
|
||||
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
base_url = str(llm_config.get("base_url") or "").strip()
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
result = {
|
||||
"success": False, "connected": False, "mode": "lightweight_models",
|
||||
"provider": provider, "model": model,
|
||||
}
|
||||
if not base_url or not model:
|
||||
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
|
||||
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
return {**result, "message": "当前 provider 未配置 API Key。"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
catalog = await fetch_model_catalog(
|
||||
provider, base_url, provider_api, api_key,
|
||||
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用,但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {**result, "message": catalog_error_message(exc)}
|
||||
result.update({"url": catalog.url, "models_count": len(catalog.models)})
|
||||
if model.casefold() not in {item.casefold() for item in catalog.models}:
|
||||
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
|
||||
model_apis = llm_config.get("model_provider_apis") or {}
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
**result, "success": True, "connected": True,
|
||||
"provider_api": model_apis.get(model) or provider_api,
|
||||
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
|
||||
}
|
||||
|
||||
|
||||
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
|
||||
else "settings.ai_provider.connect.failed"),
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
"connected": lightweight_result["connected"],
|
||||
},
|
||||
)
|
||||
return {
|
||||
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
presets = list_fallback_llm_provider_presets()
|
||||
saved = await get_setting_payloads(
|
||||
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
|
||||
)
|
||||
return {
|
||||
"data": [
|
||||
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
|
||||
for preset in presets
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
payload: AIProviderIntegrationUpdate | None = None,
|
||||
):
|
||||
try:
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
get_fallback_llm_provider_preset(provider_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
if payload is not None:
|
||||
if _normalize_provider_id(payload.provider) != provider_id:
|
||||
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
|
||||
ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
else:
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
|
||||
)
|
||||
try:
|
||||
refreshed = await refresh_llm_provider_preset(
|
||||
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
|
||||
provider_api=provider_config.get("provider_api"),
|
||||
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
|
||||
)
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
logger.warning(
|
||||
"LLM provider catalog refresh failed",
|
||||
extra={
|
||||
"event": "settings.ai_provider.catalog.failed",
|
||||
"context": {
|
||||
"provider": provider_id,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
|
||||
) from exc
|
||||
|
||||
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
|
||||
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
|
||||
return {"data": refreshed}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
|
||||
@@ -7,6 +7,7 @@ from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=100),
|
||||
q: str = Query("", max_length=200),
|
||||
selected_id: str | None = Query(None, max_length=200),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
|
||||
@@ -644,7 +644,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
if unit in {"flop/s", "flops"}:
|
||||
normalized_tflops = capacity_value / 1e12
|
||||
elif unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value
|
||||
@@ -672,9 +674,13 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
"""
|
||||
features: List[Dict[str, Any]] = []
|
||||
unresolved: List[Dict[str, Any]] = []
|
||||
withheld_count = 0
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
|
||||
withheld_count += 1
|
||||
continue
|
||||
result = resolve_compute_center_location_full(record, metadata)
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
@@ -706,7 +712,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
latitude = location.latitude
|
||||
longitude = location.longitude
|
||||
|
||||
if site_type == "supercomputer":
|
||||
if metadata.get("compute_contract") == "v1":
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
|
||||
elif site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
@@ -767,7 +776,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
|
||||
|
||||
|
||||
VESSEL_TYPE_FILTERS = {
|
||||
@@ -1121,6 +1130,7 @@ async def build_vessel_snapshot_response(
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
|
||||
snapshot_started_at = datetime.now(UTC)
|
||||
features, diagnostics = await _load_raw_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
@@ -1137,6 +1147,7 @@ async def build_vessel_snapshot_response(
|
||||
"features": features,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
"generated_at": to_iso8601_utc(snapshot_started_at),
|
||||
"diagnostics": {
|
||||
**diagnostics,
|
||||
"filtered_count": len(features),
|
||||
|
||||
@@ -161,7 +161,7 @@ async def websocket_endpoint(
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
if "vessels" in channels and "bbox" in payload_data:
|
||||
if "vessels" in channels:
|
||||
try:
|
||||
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
|
||||
57
backend/app/core/earth_geography_policy.py
Normal file
@@ -0,0 +1,57 @@
|
||||
"""Country display ownership for Earth comparisons and cards.
|
||||
|
||||
Keep source-country evidence separate; use this projection only for display
|
||||
grouping so regional source records cannot become competing national keys.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.core.countries import COUNTRY_ENTRIES, normalize_country
|
||||
|
||||
|
||||
_DISPLAY_ISO_A3 = {
|
||||
canonical: next(
|
||||
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
|
||||
None,
|
||||
)
|
||||
for canonical, aliases in COUNTRY_ENTRIES
|
||||
}
|
||||
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
|
||||
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _policy_country_by_code() -> dict[str, str]:
|
||||
try:
|
||||
content = _POLICY_PATH.read_text(encoding="utf-8")
|
||||
except FileNotFoundError as error:
|
||||
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
|
||||
policy = json.loads(content)
|
||||
if policy.get("profile") != "china-pov-v1":
|
||||
raise ValueError("unsupported Earth display-policy profile")
|
||||
mappings = {}
|
||||
for rule in policy["rules"]:
|
||||
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
|
||||
if target not in _COUNTRY_BY_ISO_A3:
|
||||
continue
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
mappings[code] = _COUNTRY_BY_ISO_A3[target]
|
||||
return mappings
|
||||
|
||||
|
||||
def display_country(value: str | None) -> str | None:
|
||||
country = normalize_country(value)
|
||||
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
|
||||
owner = _policy_country_by_code().get(code)
|
||||
if owner:
|
||||
return owner
|
||||
return country
|
||||
|
||||
|
||||
def display_iso_a3(value: str | None) -> str | None:
|
||||
country = display_country(value)
|
||||
return _DISPLAY_ISO_A3.get(country) if country else None
|
||||
@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
|
||||
class ProviderApi(StrEnum):
|
||||
ANTHROPIC_MESSAGES = "anthropic-messages"
|
||||
OPENAI_COMPLETIONS = "openai-completions"
|
||||
OPENAI_RESPONSES = "openai-responses"
|
||||
OLLAMA_GENERATE = "ollama-generate"
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ import redis
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from jose import JWTError, jwt
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings
|
||||
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
|
||||
return None
|
||||
|
||||
|
||||
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
|
||||
try:
|
||||
parsed_id = int(user_id)
|
||||
except (TypeError, ValueError):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
|
||||
result = await db.execute(select(User).where(User.id == parsed_id))
|
||||
user = result.scalar_one_or_none()
|
||||
if user is None or not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
return user
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
@@ -103,27 +118,7 @@ async def get_current_user(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
async def get_current_user_refresh(
|
||||
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
def blacklist_token(token: str) -> None:
|
||||
|
||||
@@ -4,11 +4,14 @@ import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Dict, Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
|
||||
logger = get_logger(__name__, service="websocket")
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
@@ -114,16 +117,22 @@ class DataBroadcaster:
|
||||
return
|
||||
pending = self._pending_vessel_updates
|
||||
self._pending_vessel_updates = {}
|
||||
vessels = []
|
||||
for item in pending.values():
|
||||
vessel = dict(item)
|
||||
source = vessel.pop("_source", None)
|
||||
action = vessel.pop("_action", "upsert")
|
||||
created = vessel.pop("_created", None)
|
||||
vessel["source"] = source
|
||||
vessel["action"] = action
|
||||
vessel["created"] = created
|
||||
vessels.append(vessel)
|
||||
try:
|
||||
vessels = await self._load_current_vessel_updates(list(pending))
|
||||
except Exception:
|
||||
# Preserve newer updates that arrived during the failed database read.
|
||||
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
|
||||
raise
|
||||
if not vessels:
|
||||
return
|
||||
vessels = [
|
||||
{
|
||||
**vessel,
|
||||
"action": vessel.get("action", "upsert"),
|
||||
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
|
||||
}
|
||||
for vessel in vessels
|
||||
]
|
||||
await manager.broadcast_vessels(
|
||||
{
|
||||
"action": "upsert",
|
||||
@@ -133,12 +142,31 @@ class DataBroadcaster:
|
||||
}
|
||||
)
|
||||
|
||||
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
|
||||
|
||||
mmsis = [int(key) for key in keys if key.isdigit()]
|
||||
vessels = []
|
||||
async with async_session_factory() as db:
|
||||
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
|
||||
vessels.extend(await get_current_vessels_by_mmsi(
|
||||
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
|
||||
))
|
||||
present = {int(vessel["mmsi"]) for vessel in vessels}
|
||||
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
|
||||
return vessels
|
||||
|
||||
async def broadcast_vessels_periodically(self):
|
||||
while self.running:
|
||||
try:
|
||||
await self.flush_vessel_updates()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to flush vessel updates",
|
||||
event="vessels.broadcast.failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(self._vessel_flush_interval)
|
||||
|
||||
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
|
||||
|
||||
@@ -63,6 +63,8 @@ class ConnectionManager:
|
||||
|
||||
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
|
||||
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
|
||||
if channel == "vessels":
|
||||
self.vessel_subscriptions.pop(websocket, None)
|
||||
subscribers = self.channel_subscriptions.get(channel)
|
||||
if subscribers is not None:
|
||||
subscribers.discard(websocket)
|
||||
@@ -88,7 +90,8 @@ class ConnectionManager:
|
||||
return subscription
|
||||
|
||||
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
bbox = config.get("bbox")
|
||||
global_scope = config.get("scope") == "global"
|
||||
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
|
||||
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
|
||||
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
|
||||
try:
|
||||
@@ -103,7 +106,7 @@ class ConnectionManager:
|
||||
raise ValueError("bbox longitude values must be between -180 and 180")
|
||||
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
|
||||
raise ValueError("bbox latitude values must be between -90 and 90")
|
||||
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
|
||||
|
||||
zoom = int(config.get("zoom") or 1)
|
||||
@@ -116,10 +119,11 @@ class ConnectionManager:
|
||||
if str(item).strip()
|
||||
}
|
||||
return {
|
||||
"scope": "global" if global_scope else "viewport",
|
||||
"bbox": (lon_min, lat_min, lon_max, lat_max),
|
||||
"zoom": zoom,
|
||||
"limit": limit,
|
||||
"type": vessel_types,
|
||||
"type": sorted(vessel_types),
|
||||
"last_sent_at": None,
|
||||
}
|
||||
|
||||
@@ -152,7 +156,9 @@ class ConnectionManager:
|
||||
vessel
|
||||
for vessel in vessels
|
||||
if self._vessel_matches_subscription(vessel, subscription)
|
||||
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
|
||||
]
|
||||
if subscription.get("scope") != "global":
|
||||
matched = matched[:subscription["limit"]]
|
||||
if not matched:
|
||||
continue
|
||||
subscription["last_sent_at"] = datetime.now(UTC)
|
||||
@@ -162,16 +168,24 @@ class ConnectionManager:
|
||||
"timestamp": subscription["last_sent_at"].isoformat(),
|
||||
"payload": {
|
||||
**data,
|
||||
"vessels": matched,
|
||||
"vessels": [],
|
||||
"subscription": {
|
||||
"bbox": list(subscription["bbox"]),
|
||||
"zoom": subscription["zoom"],
|
||||
"limit": subscription["limit"],
|
||||
"scope": subscription.get("scope", "viewport"),
|
||||
},
|
||||
},
|
||||
}
|
||||
try:
|
||||
await connection.send_json(message)
|
||||
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
|
||||
await connection.send_json({
|
||||
**message,
|
||||
"payload": {
|
||||
**message["payload"],
|
||||
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
|
||||
},
|
||||
})
|
||||
except Exception:
|
||||
self.unsubscribe_all(connection)
|
||||
|
||||
@@ -180,6 +194,8 @@ class ConnectionManager:
|
||||
vessel: dict[str, Any],
|
||||
subscription: dict[str, Any],
|
||||
) -> bool:
|
||||
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
|
||||
return True
|
||||
try:
|
||||
lon = float(vessel.get("lon"))
|
||||
lat = float(vessel.get("lat"))
|
||||
|
||||
94
backend/app/schemas/compute_configuration.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""Validated, source-backed observations for the compute comparison cards."""
|
||||
|
||||
from datetime import date
|
||||
from typing import Annotated, Literal, Self
|
||||
|
||||
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
|
||||
|
||||
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
|
||||
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
|
||||
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
|
||||
Text = Annotated[str, Field(min_length=1, max_length=500)]
|
||||
|
||||
|
||||
class ConfigurationModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
|
||||
class ComputeSource(ConfigurationModel):
|
||||
id: Identifier
|
||||
title: Text
|
||||
url: HttpUrl
|
||||
publisher: Text
|
||||
report_year: Year
|
||||
published_at: date | None = None
|
||||
locator: Text
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def public_reference(cls, value: HttpUrl) -> HttpUrl:
|
||||
if value.username or value.password or len(str(value)) > 2000:
|
||||
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
|
||||
return value
|
||||
|
||||
|
||||
class ComputeObservation(ConfigurationModel):
|
||||
year: Year
|
||||
metric: Literal["global_share", "absolute"]
|
||||
source_id: Identifier
|
||||
values: dict[str, Value] = Field(min_length=1, max_length=250)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def comparable_values(self) -> Self:
|
||||
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
|
||||
raise ValueError(
|
||||
"Use countries from the canonical compute catalog, not regional totals"
|
||||
)
|
||||
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
|
||||
raise ValueError("Global shares for one year must total at most 100 percent")
|
||||
return self
|
||||
|
||||
|
||||
class InnovationObservation(ConfigurationModel):
|
||||
year: Year
|
||||
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
|
||||
source_id: Identifier
|
||||
|
||||
|
||||
class ComputeConfiguration(ConfigurationModel):
|
||||
reviewed_at: date
|
||||
sources: list[ComputeSource] = Field(max_length=200)
|
||||
observations: list[ComputeObservation] = Field(max_length=202)
|
||||
innovation: list[InnovationObservation] = Field(max_length=101)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def consistent_evidence(self) -> Self:
|
||||
ids = [source.id for source in self.sources]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Report source IDs must be unique")
|
||||
keys = [(row.metric, row.year) for row in self.observations]
|
||||
if len(keys) != len(set(keys)):
|
||||
raise ValueError("Each metric/year may have only one observation")
|
||||
years = [row.year for row in self.innovation]
|
||||
if len(years) != len(set(years)):
|
||||
raise ValueError("Innovation observation years must be unique")
|
||||
sources = {source.id: source for source in self.sources}
|
||||
for row in [*self.observations, *self.innovation]:
|
||||
source = sources.get(row.source_id)
|
||||
if source is None:
|
||||
raise ValueError(f"Missing report source: {row.source_id}")
|
||||
if row.year > source.report_year:
|
||||
raise ValueError("Observation year must not be after its report year")
|
||||
if self.reviewed_at > date.today():
|
||||
raise ValueError("Review date must not be in the future")
|
||||
return self
|
||||
|
||||
|
||||
class ComputeConfigurationWrite(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
config: ComputeConfiguration
|
||||
|
||||
|
||||
class ComputeConfigurationReset(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
@@ -36,6 +36,9 @@ class UserInDB(UserBase):
|
||||
|
||||
|
||||
class UserResponse(UserBase):
|
||||
# Bootstrap accounts have reserved .local email addresses. Validate email
|
||||
# on writes, while allowing existing local identities to be serialized.
|
||||
email: str
|
||||
id: int
|
||||
role: str
|
||||
gatekeeper_groups: list[str] = Field(default_factory=list)
|
||||
|
||||
@@ -31,6 +31,7 @@ class BaseCollector(ABC):
|
||||
frequency_hours: int = 4
|
||||
data_type: str = "generic"
|
||||
fail_on_empty: bool = False
|
||||
preserve_snapshot_on_failure: bool = False
|
||||
|
||||
def __init__(self):
|
||||
self._current_task = None
|
||||
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
|
||||
context={"task_id": task_id, "raw_count": len(raw_data)},
|
||||
)
|
||||
data = self.transform(raw_data)
|
||||
if self.fail_on_empty and not data:
|
||||
raise RuntimeError(f"Collector {self.name} produced no valid records")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.transforming.success",
|
||||
"Collector transform phase completed",
|
||||
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
if snapshot_id is not None and self.preserve_snapshot_on_failure:
|
||||
await self._rollback_incomplete_run(
|
||||
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
|
||||
)
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = "failed"
|
||||
task.phase_message = str(e)
|
||||
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.status = SnapshotStatus.FAILED.value
|
||||
if self.preserve_snapshot_on_failure:
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
snapshot.summary = {"error": str(e)}
|
||||
await db.commit()
|
||||
|
||||
@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
ACTIVE_GROUP = "active"
|
||||
FALLBACK_GROUPS = (
|
||||
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
|
||||
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
|
||||
|
||||
|
||||
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
_downloader = ResumableFileDownloader(
|
||||
cache_namespace="celestrak",
|
||||
default_accept="application/json",
|
||||
preflight_head=False,
|
||||
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if not is_final_attempt:
|
||||
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
|
||||
|
||||
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active satellite download failed after retries: {last_error}"
|
||||
)
|
||||
|
||||
async def _fetch_fallback_groups(
|
||||
self,
|
||||
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
body_path = cached_path
|
||||
else:
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' download failed: {exc}"
|
||||
) from exc
|
||||
body_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if body_path is None:
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
commit=True,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_json_file(path: Path) -> bool:
|
||||
@classmethod
|
||||
def _validate_json_file(cls, path: Path) -> bool:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
|
||||
cls._load_active_payload(path)
|
||||
except RuntimeError:
|
||||
return False
|
||||
return isinstance(data, list)
|
||||
return True
|
||||
|
||||
async def _log_parse_failure(self, exc: Exception) -> None:
|
||||
await emit_business_log(
|
||||
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
),
|
||||
)
|
||||
|
||||
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
|
||||
@staticmethod
|
||||
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
else:
|
||||
invalid_count += 1
|
||||
if invalid_count:
|
||||
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
|
||||
)
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak active payload contains no satellite records")
|
||||
return records
|
||||
|
||||
82
backend/app/services/collectors/compute_base.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""Quality gates shared by the two compute sources."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class ComputeCollector(BaseCollector):
|
||||
fail_on_empty = True
|
||||
preserve_snapshot_on_failure = True
|
||||
|
||||
async def _create_snapshot(
|
||||
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
|
||||
) -> int:
|
||||
previous_count = await db.scalar(
|
||||
select(DataSnapshot.record_count)
|
||||
.where(
|
||||
DataSnapshot.source == self.name,
|
||||
DataSnapshot.is_current.is_(True),
|
||||
DataSnapshot.status == "success",
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
if previous_count and len(data) < previous_count * 0.6:
|
||||
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
|
||||
return await super()._create_snapshot(db, task_id, data, started_at)
|
||||
|
||||
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
|
||||
if self.name == "top500":
|
||||
await self._copy_verified_locations(db, data)
|
||||
return await super()._save_data(db, data, **kwargs)
|
||||
|
||||
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
|
||||
"""Keep historical location keys; copy only unambiguous exact identities.
|
||||
|
||||
Never migrate by rank: the machine occupying rank 1 changes over time.
|
||||
"""
|
||||
names = [item["name"] for item in data]
|
||||
locations = (
|
||||
(
|
||||
await db.execute(
|
||||
select(ComputeCenterLocationRecord).where(
|
||||
ComputeCenterLocationRecord.source == self.name,
|
||||
ComputeCenterLocationRecord.name.in_(names),
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
existing = {location.source_id for location in locations}
|
||||
for item in data:
|
||||
if item["source_id"] in existing:
|
||||
continue
|
||||
country = normalize_country(item["country"])
|
||||
if not country:
|
||||
continue
|
||||
matches = [
|
||||
location
|
||||
for location in locations
|
||||
if location.name == item["name"]
|
||||
and normalize_country(location.country) == country
|
||||
and (not location.site or location.site == item.get("metadata", {}).get("site"))
|
||||
and not location.source_id.startswith("top500_system_")
|
||||
]
|
||||
if len(matches) != 1:
|
||||
continue
|
||||
old = matches[0]
|
||||
values = {
|
||||
column.name: getattr(old, column.name)
|
||||
for column in old.__table__.columns
|
||||
if column.name not in {"id", "source_id", "created_at", "updated_at"}
|
||||
}
|
||||
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
|
||||
existing.add(item["source_id"])
|
||||
@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
|
||||
ValidateCallback = Callable[[Path], bool]
|
||||
|
||||
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
|
||||
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
default_accept: str = "*/*",
|
||||
cache_root: Path | None = None,
|
||||
preflight_head: bool = True,
|
||||
cache_ttl_seconds: float = 0,
|
||||
) -> None:
|
||||
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
|
||||
self._user_agent = user_agent
|
||||
self._default_accept = default_accept
|
||||
self._preflight_head = preflight_head
|
||||
self._cache_ttl_seconds = cache_ttl_seconds
|
||||
|
||||
@staticmethod
|
||||
def _cache_key(url: str) -> str:
|
||||
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
|
||||
if not meta_path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
data = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError, OSError:
|
||||
return {}
|
||||
|
||||
@staticmethod
|
||||
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
|
||||
return last_modified == str(meta.get("last_modified") or "").strip()
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _response_info(response: httpx.Response) -> dict[str, Any]:
|
||||
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
|
||||
raw_length = response.headers.get("content-length")
|
||||
encoded = response.headers.get("content-encoding", "identity") != "identity"
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": int(raw_length) if raw_length and not encoded else None,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
|
||||
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
|
||||
try:
|
||||
response = await client.head(url)
|
||||
if response.status_code >= 400:
|
||||
return {}
|
||||
content_length_raw = response.headers.get("content-length")
|
||||
content_length = int(content_length_raw) if content_length_raw else None
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": content_length,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
except (httpx.HTTPError, ValueError):
|
||||
return self._response_info(response)
|
||||
except httpx.HTTPError, ValueError:
|
||||
return {}
|
||||
|
||||
async def download_file(
|
||||
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
|
||||
) -> Path:
|
||||
self._cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
final_path, part_path, meta_path = self._cache_paths(url, extension)
|
||||
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
|
||||
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
|
||||
size = cached_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return cached_path
|
||||
meta = self._load_meta(meta_path)
|
||||
remote = await self.fetch_remote_info(client, url)
|
||||
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
if final_path.exists():
|
||||
local_size = final_path.stat().st_size
|
||||
size_match = expected_size is None or local_size == expected_size
|
||||
if self._validators_match(meta, remote) and size_match:
|
||||
if (
|
||||
(remote.get("etag") or remote.get("last_modified"))
|
||||
and self._validators_match(meta, remote)
|
||||
and size_match
|
||||
):
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
final_path.unlink(missing_ok=True)
|
||||
else:
|
||||
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
|
||||
headers["If-Range"] = str(remote.get("last_modified"))
|
||||
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code == 304 and final_path.exists():
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
if response.status_code == 304:
|
||||
if cached_path is None:
|
||||
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
|
||||
final_path.touch()
|
||||
size = final_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return final_path
|
||||
if response.status_code >= 400:
|
||||
body = (await response.aread()).decode("utf-8", errors="replace")
|
||||
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
|
||||
else:
|
||||
mode = "wb"
|
||||
resume_from = 0
|
||||
remote = self._response_info(response)
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
downloaded = resume_from
|
||||
last_emit_bytes = 0
|
||||
last_emit_time = time.monotonic()
|
||||
min_emit_bytes = (
|
||||
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
|
||||
max(expected_size // 150, 512 * 1024)
|
||||
if expected_size and expected_size > 0
|
||||
else 1024 * 1024
|
||||
)
|
||||
|
||||
with part_path.open(mode) as f:
|
||||
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
|
||||
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
|
||||
)
|
||||
|
||||
# Never replace the last good payload with an HTML error or invalid JSON.
|
||||
if validate_existing and not validate_existing(part_path):
|
||||
part_path.unlink(missing_ok=True)
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
part_path.replace(final_path)
|
||||
self._save_meta(
|
||||
meta_path,
|
||||
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
|
||||
},
|
||||
)
|
||||
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
|
||||
|
||||
@@ -1,119 +1,135 @@
|
||||
"""Epoch AI GPU Clusters Collector
|
||||
|
||||
Collects data from Epoch AI GPU clusters tracking.
|
||||
https://epoch.ai/data/gpu-clusters
|
||||
"""
|
||||
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.countries import normalize_country
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
|
||||
|
||||
|
||||
|
||||
class EpochAIGPUCollector(BaseCollector):
|
||||
class EpochAIGPUCollector(ComputeCollector):
|
||||
name = "epoch_ai_gpu"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
frequency_hours = 6
|
||||
data_type = "gpu_cluster"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
async def fetch(self) -> list[dict]:
|
||||
configured = self._resolved_url or EPOCH_CSV_URL
|
||||
parsed = urlparse(configured)
|
||||
url = (
|
||||
EPOCH_CSV_URL
|
||||
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
|
||||
else configured
|
||||
)
|
||||
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.text)
|
||||
return self.parse_response(response.text)
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
"""Parse Epoch AI webpage to extract GPU cluster data"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
def parse_response(self, text: str) -> list[dict]:
|
||||
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
|
||||
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
|
||||
if not required.issubset(reader.fieldnames or []):
|
||||
raise ValueError(
|
||||
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
|
||||
)
|
||||
entries = []
|
||||
seen = set()
|
||||
anonymous_counts = {}
|
||||
for row in reader:
|
||||
name = (row.get("Name") or "").strip()
|
||||
country = normalize_country(row.get("Country"))
|
||||
if not name:
|
||||
raise ValueError("Epoch row has missing identity")
|
||||
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
|
||||
anonymous = "anonym" in name.lower()
|
||||
if anonymous:
|
||||
# Public anonymous rows have no facility identity. Preserve
|
||||
# multiplicity within snapshots, without guessing names.
|
||||
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
|
||||
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
|
||||
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
|
||||
if source_id in seen:
|
||||
raise ValueError("Epoch identity collision; previous snapshot retained")
|
||||
seen.add(source_id)
|
||||
raw_date = row.get("First Operational Date", "")
|
||||
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
|
||||
value = None
|
||||
raw_value = row.get("16-bit OP/s (log)", "")
|
||||
if raw_value:
|
||||
exponent = float(raw_value)
|
||||
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
|
||||
raise ValueError("Epoch performance exponent out of bounds")
|
||||
value = 10**exponent
|
||||
anonymous = anonymous or (country == "中国" and not row.get("Location"))
|
||||
location = row.get("Location")
|
||||
location_assessment = assess_epoch_location(
|
||||
location,
|
||||
country=country,
|
||||
raw_country=row.get("Country"),
|
||||
)
|
||||
entries.append(
|
||||
{
|
||||
"source_id": source_id,
|
||||
"name": name,
|
||||
"country": country,
|
||||
"value": value,
|
||||
"unit": "FLOP/s",
|
||||
"reference_date": reference_date,
|
||||
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
|
||||
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP16",
|
||||
"measurement": "theoretical",
|
||||
"source_url": EPOCH_CSV_URL,
|
||||
"raw_reference_date": raw_date,
|
||||
"status": row.get("Status"),
|
||||
"certainty": row.get("Certainty"),
|
||||
"operator": row.get("Owner"),
|
||||
"gpu_type": row.get("Chip type (primary)"),
|
||||
"gpu_count": row.get("Chip quantity (primary)"),
|
||||
"location": location,
|
||||
"location_precision": location_assessment.precision,
|
||||
"location_precision_reason": location_assessment.reason,
|
||||
"raw_country": row.get("Country"),
|
||||
"raw_performance": raw_value,
|
||||
"performance_encoding": "log10",
|
||||
"builds_upon": row.get("Builds Upon"),
|
||||
"superseded_by": row.get("Superseded by"),
|
||||
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
|
||||
"include_in_standard_analysis": row.get(
|
||||
"Include in Standard Analysis", ""
|
||||
).lower()
|
||||
== "true",
|
||||
"rounded": country == "中国",
|
||||
"anonymous": anonymous,
|
||||
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
|
||||
"country_unresolved": country is None,
|
||||
"sources": [
|
||||
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
# Try to find data table on the page
|
||||
tables = soup.find_all("table")
|
||||
for table in tables:
|
||||
rows = table.find_all("tr")
|
||||
for row in rows[1:]: # Skip header
|
||||
cells = row.find_all(["td", "th"])
|
||||
if len(cells) >= 5:
|
||||
try:
|
||||
cluster_name = cells[0].get_text(strip=True)
|
||||
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
|
||||
continue
|
||||
|
||||
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
|
||||
country, city = self._parse_location(location_cell)
|
||||
|
||||
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
|
||||
|
||||
entry = {
|
||||
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
|
||||
"name": cluster_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": self._parse_performance(perf_cell),
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"raw_data": perf_cell,
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError):
|
||||
continue
|
||||
|
||||
# If no table found, return sample data
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
|
||||
return data
|
||||
|
||||
def _parse_location(self, location: str) -> tuple:
|
||||
"""Parse location string into country and city"""
|
||||
if not location:
|
||||
return "", ""
|
||||
if "," in location:
|
||||
parts = location.rsplit(",", 1)
|
||||
city = parts[0].strip()
|
||||
country = parts[1].strip() if len(parts) > 1 else ""
|
||||
return country, city
|
||||
return location, ""
|
||||
|
||||
def _parse_performance(self, perf: str) -> str:
|
||||
"""Parse performance string to extract value"""
|
||||
if not perf:
|
||||
return "0"
|
||||
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
match = re.search(r"([\d,.]+)", perf)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
return "0"
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "epoch_sample_1",
|
||||
"name": "Sample GPU Cluster",
|
||||
"country": "United States",
|
||||
"city": "San Francisco, CA",
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": "1000",
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"note": "Sample data - Epoch AI page structure may vary",
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
},
|
||||
]
|
||||
@staticmethod
|
||||
def coordinate(value: str | None, bound: int) -> float | None:
|
||||
if not value:
|
||||
return None
|
||||
number = float(value)
|
||||
if not math.isfinite(number) or abs(number) > bound:
|
||||
raise ValueError("Epoch coordinate out of bounds")
|
||||
return number
|
||||
|
||||
@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
if max_sources > 0 and len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
import os
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
# PeeringDB API key - read from environment variable
|
||||
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class PeeringDBIXPCollector(HTTPCollector):
|
||||
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
"""Space-Track TLE Collector
|
||||
"""Space-Track TLE Collector.
|
||||
|
||||
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
|
||||
API documentation: https://www.space-track.org/documentation
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.logging import get_logger
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class SpaceTrackTLECollector(BaseCollector):
|
||||
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
password = settings.SPACETRACK_PASSWORD
|
||||
|
||||
if not username or not password:
|
||||
print("SPACETRACK: No credentials configured, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track credentials are not configured; using sample data",
|
||||
event="collector.spacetrack.credentials_missing",
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch started",
|
||||
event="collector.spacetrack.fetch.start",
|
||||
)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
logger.info_event(
|
||||
"Space-Track login response received",
|
||||
event="collector.spacetrack.login.response",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
|
||||
if login_response.status_code == 403:
|
||||
print("SPACETRACK: Trying alternate login method...")
|
||||
logger.warning_event(
|
||||
"Space-Track login returned forbidden; trying alternate method",
|
||||
event="collector.spacetrack.login.forbidden",
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
) as alt_client:
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
"password": password,
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
logger.info_event(
|
||||
"Space-Track alternate login response received",
|
||||
event="collector.spacetrack.alt_login.response",
|
||||
context={"status_code": alt_login.status_code},
|
||||
)
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
logger.info_event(
|
||||
"Space-Track alternate query completed",
|
||||
event="collector.spacetrack.alt_query.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track login failed; using sample data",
|
||||
event="collector.spacetrack.login.failed",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
logger.info_event(
|
||||
"Space-Track TLE query response received",
|
||||
event="collector.spacetrack.query.response",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE query failed; using sample data",
|
||||
event="collector.spacetrack.query.failed",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch completed",
|
||||
event="collector.spacetrack.fetch.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE fetch failed; using sample data",
|
||||
event="collector.spacetrack.fetch.failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import re
|
||||
from urllib.parse import urljoin
|
||||
from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
|
||||
|
||||
class TOP500Collector(BaseCollector):
|
||||
class TOP500Collector(ComputeCollector):
|
||||
name = "top500"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
entries = self.parse_response(response.text)
|
||||
entries = self.parse_response(response.text, str(response.url))
|
||||
|
||||
semaphore = asyncio.Semaphore(8)
|
||||
|
||||
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
detail_url = urljoin(str(top500_base_url), link.get("href"))
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
return detail_map
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
|
||||
"""Parse TOP500 HTML response"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
canonical = soup.find("link", rel="canonical")
|
||||
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
|
||||
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
|
||||
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
|
||||
|
||||
# Find the table with TOP500 data
|
||||
table = None
|
||||
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
|
||||
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
|
||||
|
||||
if table:
|
||||
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
|
||||
if not reference_date:
|
||||
raise ValueError("TOP500 list date missing; previous snapshot retained")
|
||||
rows = table.select("tr")
|
||||
for row in rows[1:]: # Skip header row
|
||||
cells = row.find_all(["td", "th"])
|
||||
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
|
||||
cores = cells[2].get_text(strip=True).replace(",", "")
|
||||
|
||||
rmax_text = cells[3].get_text(strip=True)
|
||||
rmax = self._parse_performance(rmax_text)
|
||||
rmax = self._parse_performance(rmax_text, headers[3])
|
||||
|
||||
rpeak_text = cells[4].get_text(strip=True)
|
||||
rpeak = self._parse_performance(rpeak_text)
|
||||
rpeak = self._parse_performance(rpeak_text, headers[4])
|
||||
|
||||
power = ""
|
||||
if len(cells) >= 6:
|
||||
power = cells[5].get_text(strip=True).replace(",", "")
|
||||
|
||||
identity = re.search(r"/system/(\d+)", detail_url)
|
||||
if not identity:
|
||||
raise ValueError("TOP500 stable system ID missing")
|
||||
entry = {
|
||||
"source_id": f"top500_{rank}",
|
||||
"source_id": f"top500_system_{identity[1]}",
|
||||
"name": system_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": 0.0,
|
||||
"longitude": 0.0,
|
||||
"latitude": None,
|
||||
"longitude": None,
|
||||
"value": str(rmax),
|
||||
"unit": "PFlop/s",
|
||||
"unit": "FLOP/s",
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP64", "measurement": "LINPACK Rmax",
|
||||
"source_url": detail_url, "list_url": date_url,
|
||||
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
|
||||
"rank": rank,
|
||||
"cores": cores,
|
||||
"rmax": rmax_text,
|
||||
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
|
||||
"site": site,
|
||||
},
|
||||
"_detail_url": detail_url,
|
||||
"reference_date": "2025-11-01",
|
||||
"reference_date": reference_date,
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError) as e:
|
||||
continue
|
||||
|
||||
# If scraping failed, return sample data for testing
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
|
||||
|
||||
return data
|
||||
|
||||
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
|
||||
return 0.0
|
||||
return 0.0
|
||||
|
||||
def _parse_performance(self, text: str) -> float:
|
||||
def _parse_performance(self, text: str, header: str = "") -> float:
|
||||
"""Parse performance value from text (handles E, P, T suffixes)"""
|
||||
text = text.strip().upper()
|
||||
text = text.strip().upper().replace(",", "")
|
||||
if not re.search(r"[EPTGMK]F", text):
|
||||
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
|
||||
if not unit:
|
||||
raise ValueError("TOP500 performance unit missing")
|
||||
text = f"{text} {unit[1]}FLOP/S"
|
||||
multipliers = {
|
||||
"E": 1e18,
|
||||
"P": 1e15,
|
||||
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
# Try simple float parsing
|
||||
try:
|
||||
return float(text.replace(",", ""))
|
||||
value = float(text.replace(",", ""))
|
||||
if not math.isfinite(value) or value < 0:
|
||||
raise ValueError("Invalid TOP500 performance")
|
||||
return value
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "top500_1",
|
||||
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
|
||||
"country": "United States",
|
||||
"city": "Livermore, CA",
|
||||
"latitude": 37.6819,
|
||||
"longitude": -121.7681,
|
||||
"value": "1742.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 1,
|
||||
"cores": "11039616",
|
||||
"rmax": "1742.00",
|
||||
"rpeak": "2746.38",
|
||||
"power": "29581",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/NNSA/LLNL",
|
||||
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
|
||||
"interconnect": "Slingshot-11",
|
||||
"installation_year": "2025",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_2",
|
||||
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
|
||||
"country": "United States",
|
||||
"city": "Oak Ridge, TN",
|
||||
"latitude": 36.0107,
|
||||
"longitude": -84.2663,
|
||||
"value": "1353.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 2,
|
||||
"cores": "9066176",
|
||||
"rmax": "1353.00",
|
||||
"rpeak": "2055.72",
|
||||
"power": "24607",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/SC/Oak Ridge National Laboratory",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_3",
|
||||
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
|
||||
"country": "United States",
|
||||
"city": "Argonne, IL",
|
||||
"latitude": 41.3784,
|
||||
"longitude": -87.8600,
|
||||
"value": "1012.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 3,
|
||||
"cores": "9264128",
|
||||
"rmax": "1012.00",
|
||||
"rpeak": "1980.01",
|
||||
"power": "38698",
|
||||
"manufacturer": "Intel",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
]
|
||||
raise ValueError("Invalid TOP500 performance") from None
|
||||
|
||||
200
backend/app/services/compute_catalog.py
Normal file
@@ -0,0 +1,200 @@
|
||||
"""Canonical country and mutually exclusive regional compute participants."""
|
||||
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
|
||||
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
|
||||
from app.core.earth_geography_policy import display_country
|
||||
|
||||
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
|
||||
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
|
||||
|
||||
# Explicit membership prevents a regional total from double-counting a country.
|
||||
# APAC deliberately excludes China.
|
||||
COMPARISON_GROUPS = (
|
||||
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
|
||||
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
|
||||
(
|
||||
"EUROPE",
|
||||
"欧洲",
|
||||
"Europe",
|
||||
(
|
||||
"GB",
|
||||
"DE",
|
||||
"FR",
|
||||
"CH",
|
||||
"ES",
|
||||
"FI",
|
||||
"HU",
|
||||
"IE",
|
||||
"IT",
|
||||
"LU",
|
||||
"MD",
|
||||
"NL",
|
||||
"PL",
|
||||
"RO",
|
||||
"RU",
|
||||
"UA",
|
||||
),
|
||||
{"latitude": 54.0, "longitude": 20.0},
|
||||
),
|
||||
(
|
||||
"APAC",
|
||||
"亚太(不含中国)",
|
||||
"Asia Pacific (excluding China)",
|
||||
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
|
||||
{"latitude": 36.0, "longitude": 130.5},
|
||||
),
|
||||
(
|
||||
"OTHER",
|
||||
"其他地区",
|
||||
"Other",
|
||||
(
|
||||
"CA",
|
||||
"AO",
|
||||
"EG",
|
||||
"KE",
|
||||
"MZ",
|
||||
"NG",
|
||||
"UG",
|
||||
"ZA",
|
||||
"BR",
|
||||
"CO",
|
||||
"CR",
|
||||
"DO",
|
||||
"MX",
|
||||
"AE",
|
||||
"SY",
|
||||
"TR",
|
||||
),
|
||||
{"latitude": 12.0, "longitude": 10.0},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _country_code(canonical: str, aliases: list[str]) -> str | None:
|
||||
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
|
||||
return "GB" if canonical == "英国" else code
|
||||
|
||||
|
||||
def build_country_catalog() -> list[dict]:
|
||||
"""Build the policy-projected canonical country drilldown catalog."""
|
||||
countries = []
|
||||
for canonical, aliases in COUNTRY_ENTRIES:
|
||||
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
|
||||
continue
|
||||
code = _country_code(canonical, aliases)
|
||||
if code:
|
||||
countries.append(
|
||||
{
|
||||
"code": code,
|
||||
"name": canonical,
|
||||
"name_en": aliases[0],
|
||||
"kind": "country",
|
||||
"members": [code],
|
||||
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
return countries
|
||||
|
||||
|
||||
COUNTRY_CATALOG = build_country_catalog()
|
||||
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
|
||||
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
|
||||
|
||||
|
||||
def _validate_group_members() -> None:
|
||||
members = [
|
||||
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
|
||||
]
|
||||
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
|
||||
raise ValueError("compute comparison groups must cover country catalog exactly once")
|
||||
|
||||
|
||||
_validate_group_members()
|
||||
GROUP_BY_MEMBER = {
|
||||
member: code
|
||||
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
}
|
||||
|
||||
|
||||
def build_catalog() -> list[dict]:
|
||||
"""Return the five default participants before country drilldown entries."""
|
||||
catalog: list[dict] = []
|
||||
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
|
||||
entry = (
|
||||
dict(COUNTRY_PARTICIPANTS[code])
|
||||
if code in COUNTRY_PARTICIPANTS
|
||||
else {
|
||||
"code": code,
|
||||
"name": name,
|
||||
"name_en": name_en,
|
||||
"kind": "region",
|
||||
"members": list(members),
|
||||
"anchor": dict(anchor),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
entry.update(
|
||||
{
|
||||
"members": list(members),
|
||||
"parent_group": name,
|
||||
"parent_group_code": code,
|
||||
"order": index,
|
||||
"default_selected": True,
|
||||
"color": PALETTE[index % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(entry)
|
||||
|
||||
for entry in COUNTRY_CATALOG:
|
||||
if entry["code"] in DEFAULT_PARTICIPANTS:
|
||||
continue
|
||||
country = dict(entry)
|
||||
group_code = GROUP_BY_MEMBER[country["code"]]
|
||||
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
|
||||
country.update(
|
||||
{
|
||||
"parent_group": group_name,
|
||||
"parent_group_code": group_code,
|
||||
"order": len(catalog),
|
||||
"default_selected": False,
|
||||
"color": PALETTE[len(catalog) % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(country)
|
||||
return catalog
|
||||
|
||||
|
||||
CATALOG = build_catalog()
|
||||
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
|
||||
|
||||
|
||||
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
|
||||
"""Resolve country or region participant codes to canonical country codes."""
|
||||
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
|
||||
|
||||
|
||||
def source_country_names(codes: Iterable[str]) -> list[str]:
|
||||
"""Return unique canonical source labels, respecting China and Serbia ownership."""
|
||||
selected_members = {
|
||||
member for members in participant_members(codes).values() for member in members
|
||||
}
|
||||
selected_names = {COUNTRIES[code] for code in selected_members}
|
||||
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
|
||||
|
||||
|
||||
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
|
||||
"""Map stored source-country labels to their requested participant code."""
|
||||
owner_by_country = {
|
||||
member: participant
|
||||
for participant, members in participant_members(codes).items()
|
||||
for member in members
|
||||
}
|
||||
code_by_name = {name: code for code, name in COUNTRIES.items()}
|
||||
return {
|
||||
name: owner_by_country[code_by_name[display_country(name)]]
|
||||
for name in source_country_names(codes)
|
||||
}
|
||||
@@ -3,7 +3,12 @@
|
||||
This module is a thin domain wrapper that wires up
|
||||
:mod:`app.services.location` for compute centers:
|
||||
|
||||
SourceCoordinates
|
||||
Confirmed stored location -> trustworthy source coordinates -> exact
|
||||
official facility registration -> source city/site coordinates.
|
||||
|
||||
Epoch AI coordinates are admitted only at the grain asserted by its official
|
||||
``Location`` column. Country/region/empty locations remain unresolved even
|
||||
when the CSV happens to contain a latitude/longitude pair.
|
||||
|
||||
The online Nominatim step is intentionally reserved for the user-triggered
|
||||
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
|
||||
@@ -31,6 +36,8 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
from app.services.location import (
|
||||
LocationCandidate,
|
||||
@@ -166,8 +173,13 @@ def set_compute_center_location_cache(
|
||||
|
||||
async def refresh_compute_center_location_cache(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
as_of: datetime | None = None,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
result = await session.execute(select(ComputeCenterLocationRecord))
|
||||
query = select(ComputeCenterLocationRecord)
|
||||
if as_of is not None:
|
||||
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
|
||||
result = await session.execute(query)
|
||||
records = result.scalars().all()
|
||||
cache = {}
|
||||
for record in records:
|
||||
@@ -350,6 +362,58 @@ class ROROrganizationResolver:
|
||||
)
|
||||
|
||||
|
||||
class ComputeCenterSourceCoordinatesResolver:
|
||||
"""Use source coordinates only at the grain asserted by the source text."""
|
||||
|
||||
name = "source_coordinates"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._default = SourceCoordinatesResolver()
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
if extra.get("source") != "epoch_ai_gpu":
|
||||
return self._default.resolve(query)
|
||||
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput(
|
||||
attempted_queries=(f"source_coordinates:{assessment.precision}",)
|
||||
)
|
||||
output = self._default.resolve(query)
|
||||
if not output.candidates:
|
||||
return output
|
||||
candidate = output.candidates[0]
|
||||
source_location = extra.get("source_location")
|
||||
source_location_text = (
|
||||
source_location.strip() if isinstance(source_location, str) else ""
|
||||
)
|
||||
source_note = assessment.reason
|
||||
if source_location_text:
|
||||
source_note = f"{source_note}; Epoch Location: {source_location_text}"
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
**{
|
||||
**candidate.__dict__,
|
||||
"precision": assessment.precision,
|
||||
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
|
||||
"source_note": source_note,
|
||||
"needs_confirmation": assessment.precision != "precise",
|
||||
"matched_location_name": source_location_text
|
||||
or candidate.matched_location_name,
|
||||
}
|
||||
),
|
||||
),
|
||||
attempted_queries=output.attempted_queries,
|
||||
)
|
||||
|
||||
|
||||
class StoredComputeCenterLocationResolver:
|
||||
"""Resolve a compute center through the DB-backed current-location cache."""
|
||||
|
||||
@@ -367,13 +431,26 @@ class StoredComputeCenterLocationResolver:
|
||||
longitude = parse_float(stored.get("longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
precision = stored.get("precision") or "city"
|
||||
if (
|
||||
extra.get("source") == "epoch_ai_gpu"
|
||||
and stored.get("location_source") == "source_coordinates"
|
||||
):
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput()
|
||||
precision = assessment.precision
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
display_name=stored.get("name") or query.name or "Compute center",
|
||||
precision=stored.get("precision") or "city",
|
||||
precision=precision,
|
||||
confidence=float(stored.get("confidence") or 0.85),
|
||||
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
|
||||
source=self.name,
|
||||
@@ -386,6 +463,60 @@ class StoredComputeCenterLocationResolver:
|
||||
matched_location_name=stored.get("site") or stored.get("name") or query.name,
|
||||
location_verified_at=stored.get("verified_at"),
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"stored_location_source": stored.get("location_source"),
|
||||
"verification_status": stored.get("verification_status"),
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class VerifiedComputeSiteResolver:
|
||||
"""Resolve an exact source identity to an evidence-backed facility site."""
|
||||
|
||||
name = "official_facility_registry"
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
source = coerce_str(extra.get("source"))
|
||||
source_id = coerce_str(extra.get("source_id"))
|
||||
if not source or not source_id:
|
||||
return ResolverOutput()
|
||||
site = get_verified_compute_site(source, source_id)
|
||||
if not site:
|
||||
return ResolverOutput()
|
||||
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
|
||||
source_note = coerce_str(site.get("source_note"))
|
||||
if source_urls:
|
||||
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=float(site["latitude"]),
|
||||
longitude=float(site["longitude"]),
|
||||
display_name=coerce_str(site.get("matched_location_name"))
|
||||
or query.name
|
||||
or "Compute center",
|
||||
precision=coerce_str(site.get("precision")) or "site",
|
||||
confidence=1.0,
|
||||
query=f"official_facility_registry::{source}:{source_id}",
|
||||
source=self.name,
|
||||
source_note=source_note or None,
|
||||
matched_fields=("source", "source_id"),
|
||||
needs_confirmation=False,
|
||||
city=coerce_str(site.get("city")) or None,
|
||||
region=None,
|
||||
country=coerce_str(site.get("country")) or query.country,
|
||||
matched_location_name=coerce_str(site.get("matched_location_name"))
|
||||
or None,
|
||||
location_verified_at=coerce_str(site.get("verified_at")) or None,
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"address": site.get("address"),
|
||||
"coordinate_basis": site.get("coordinate_basis"),
|
||||
"sources": source_urls,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
@@ -400,7 +531,7 @@ def _short_system_name(name: Any) -> str:
|
||||
return head or text
|
||||
|
||||
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
|
||||
name = coerce_str(getattr(record, "name", None))
|
||||
return {
|
||||
"source": coerce_str(getattr(record, "source", None)),
|
||||
@@ -417,11 +548,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
or metadata.get("manufacturer")
|
||||
),
|
||||
"organization": coerce_str(metadata.get("organization")),
|
||||
"source_location": metadata.get("location"),
|
||||
"source_raw_country": metadata.get("raw_country"),
|
||||
}
|
||||
|
||||
|
||||
def _context_to_query(
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
source_lat: float | None = None,
|
||||
source_lon: float | None = None,
|
||||
@@ -444,6 +577,8 @@ def _context_to_query(
|
||||
"site": context.get("site") or "",
|
||||
"operator": context.get("operator") or "",
|
||||
"organization": context.get("organization") or "",
|
||||
"source_location": context.get("source_location"),
|
||||
"source_raw_country": context.get("source_raw_country"),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -497,11 +632,36 @@ def _compute_center_query_plan(
|
||||
return plan
|
||||
|
||||
|
||||
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
|
||||
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
|
||||
stored_confirmed = (
|
||||
candidate.source == "stored_compute_center_location"
|
||||
and not candidate.needs_confirmation
|
||||
and stored_origin != "source_coordinates"
|
||||
)
|
||||
if stored_confirmed:
|
||||
source_rank = 0
|
||||
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
|
||||
source_rank = 1
|
||||
elif candidate.source == "official_facility_registry":
|
||||
source_rank = 2
|
||||
elif candidate.source == "source_coordinates":
|
||||
source_rank = 3
|
||||
elif candidate.source == "stored_compute_center_location":
|
||||
source_rank = 4
|
||||
else:
|
||||
source_rank = 5
|
||||
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
|
||||
return (source_rank, precision_rank, -float(candidate.confidence or 0))
|
||||
|
||||
|
||||
COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
StoredComputeCenterLocationResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
" or stored compute-center location."
|
||||
@@ -510,7 +670,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
|
||||
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
ROROrganizationResolver(),
|
||||
NominatimResolver(
|
||||
query_plan_builder=_compute_center_query_plan,
|
||||
@@ -518,6 +679,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
geocoder=lambda q: _geocode_online(q),
|
||||
),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
", ROR organization lookup, or online geocoding."
|
||||
@@ -531,6 +693,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
_GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
"source_coordinates": "source_coordinates",
|
||||
"stored_compute_center_location": "stored_compute_center_location",
|
||||
"official_facility_registry": "official_facility_registry",
|
||||
"ror_organization_registry": "ror_organization",
|
||||
"nominatim_online_geocode": "online_geocode",
|
||||
}
|
||||
@@ -539,7 +702,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
def _candidate_to_location(
|
||||
candidate: LocationCandidate,
|
||||
*,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
) -> ComputeCenterLocation:
|
||||
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
|
||||
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
|
||||
@@ -594,7 +757,7 @@ def _candidate_to_location(
|
||||
|
||||
def _diagnostic_for(
|
||||
record: Any,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
failure_reason: str,
|
||||
attempted_queries: tuple[str, ...] = (),
|
||||
@@ -738,7 +901,7 @@ def build_compute_center_location_query(
|
||||
organization: str | None = None,
|
||||
) -> LocationQuery:
|
||||
name_value = coerce_str(name)
|
||||
context: dict[str, str] = {
|
||||
context: dict[str, Any] = {
|
||||
"source": coerce_str(source),
|
||||
"source_id": coerce_str(source_id),
|
||||
"name": name_value,
|
||||
@@ -783,6 +946,23 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
longitude = parse_float(get_record_field(record, "longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
precision = "precise"
|
||||
confidence = 1.0
|
||||
needs_confirmation = False
|
||||
source_note = "Seeded from source-provided compute-center coordinates"
|
||||
if source_value == "epoch_ai_gpu":
|
||||
assessment = assess_epoch_location(
|
||||
metadata.get("location"),
|
||||
country=get_record_field(record, "country"),
|
||||
raw_country=metadata.get("raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
continue
|
||||
precision = assessment.precision
|
||||
confidence = 1.0 if precision == "precise" else 0.9
|
||||
needs_confirmation = precision != "precise"
|
||||
source_note = assessment.reason
|
||||
existing = await session.scalar(
|
||||
select(ComputeCenterLocationRecord)
|
||||
.where(ComputeCenterLocationRecord.source == source_value)
|
||||
@@ -790,7 +970,6 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
)
|
||||
if existing:
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
session.add(
|
||||
ComputeCenterLocationRecord(
|
||||
source=source_value,
|
||||
@@ -802,16 +981,16 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
country=coerce_str(get_record_field(record, "country")) or None,
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
precision="precise",
|
||||
confidence=1.0,
|
||||
precision=precision,
|
||||
confidence=confidence,
|
||||
location_source="source_coordinates",
|
||||
source_note="Seeded from source-provided compute-center coordinates",
|
||||
source_note=source_note,
|
||||
raw_payload={
|
||||
"record_id": getattr(record, "id", None),
|
||||
"source": source_value,
|
||||
"source_id": source_id,
|
||||
},
|
||||
needs_confirmation=False,
|
||||
needs_confirmation=needs_confirmation,
|
||||
verification_status="source_provided",
|
||||
verified_at=None,
|
||||
)
|
||||
|
||||
44
backend/app/services/compute_center_verified_sites.json
Normal file
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"version": 1,
|
||||
"sites": [
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
|
||||
"latitude": 35.380924,
|
||||
"longitude": 139.625008,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "日本",
|
||||
"city": "Yokohama",
|
||||
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
|
||||
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
|
||||
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
|
||||
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
|
||||
"latitude": 41.718109383287235,
|
||||
"longitude": -87.984498648496,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "美国",
|
||||
"city": "Argonne",
|
||||
"matched_location_name": "Argonne Leadership Computing Facility",
|
||||
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
|
||||
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.alcf.anl.gov/systems/sophia",
|
||||
"https://docs.alcf.anl.gov/sophia/",
|
||||
"https://www.alcf.anl.gov/about/visiting-alcf"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
26
backend/app/services/compute_center_verified_sites.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
|
||||
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _site_index() -> dict[tuple[str, str], dict]:
|
||||
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
|
||||
index = {}
|
||||
for site in payload["sites"]:
|
||||
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
|
||||
raise ValueError("Invalid verified compute facility coordinate")
|
||||
for source_id in site["source_ids"]:
|
||||
key = (site["source"], source_id)
|
||||
if key in index:
|
||||
raise ValueError("Duplicate verified compute facility identity")
|
||||
index[key] = site
|
||||
return index
|
||||
|
||||
|
||||
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
|
||||
site = _site_index().get((source, source_id))
|
||||
return deepcopy(site) if site else None
|
||||
99
backend/app/services/compute_configuration.py
Normal file
@@ -0,0 +1,99 @@
|
||||
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import COUNTRY_CATALOG
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
CATEGORY = "compute_comparison"
|
||||
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
|
||||
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
|
||||
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def default_envelope() -> dict:
|
||||
return {
|
||||
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
|
||||
"revision": DEFAULT_REVISION,
|
||||
"is_default": True,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return dict(record.payload) if record else default_envelope()
|
||||
|
||||
|
||||
def configuration_response(envelope: dict) -> dict:
|
||||
return {
|
||||
**envelope,
|
||||
"countries": [
|
||||
{key: country[key] for key in ("code", "name", "name_en")}
|
||||
for country in COUNTRY_CATALOG
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
revision: str,
|
||||
config: ComputeConfiguration,
|
||||
actor: User,
|
||||
reset: bool = False,
|
||||
) -> dict:
|
||||
# The unique category plus row lock also serializes concurrent first writes.
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=default_envelope())
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if record.payload["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
envelope = (
|
||||
default_envelope()
|
||||
if reset
|
||||
else {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"is_default": reset,
|
||||
}
|
||||
)
|
||||
before = record.payload
|
||||
if reset:
|
||||
await db.delete(record)
|
||||
else:
|
||||
record.payload = envelope
|
||||
# Audit and configuration commit together: a published edit must be attributable.
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="compute_comparison.reset" if reset else "compute_comparison.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": envelope},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return envelope
|
||||
122
backend/app/services/compute_history.py
Normal file
@@ -0,0 +1,122 @@
|
||||
"""Versioned, source-backed annual observations for compute comparisons."""
|
||||
|
||||
from copy import deepcopy
|
||||
from datetime import date
|
||||
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
DEFAULT_PARTICIPANTS,
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
)
|
||||
from app.services.compute_configuration import DEFAULT_CONFIGURATION
|
||||
|
||||
|
||||
def _participant_value(
|
||||
members: tuple[str, ...], observations: dict[str, float | int], mode: str
|
||||
) -> tuple[float | int | None, dict]:
|
||||
covered = [member for member in members if member in observations]
|
||||
missing = [member for member in members if member not in observations]
|
||||
coverage = {
|
||||
"status": "complete" if not missing else "incomplete",
|
||||
"required_members": list(members),
|
||||
"covered_members": covered,
|
||||
"missing_members": missing,
|
||||
}
|
||||
if missing:
|
||||
return None, coverage
|
||||
if mode == "bilateral_share":
|
||||
if (
|
||||
not set(members).issubset({"CN", "US"})
|
||||
or not {"CN", "US"}.issubset(observations)
|
||||
or observations["US"] + observations["CN"] <= 0
|
||||
):
|
||||
coverage.update(
|
||||
{
|
||||
"status": "incompatible",
|
||||
"reason": "bilateral_share_requires_CN_and_US_observations",
|
||||
}
|
||||
)
|
||||
return None, coverage
|
||||
return (
|
||||
round(
|
||||
sum(observations[member] for member in members)
|
||||
/ (observations["US"] + observations["CN"])
|
||||
* 100,
|
||||
2,
|
||||
),
|
||||
coverage,
|
||||
)
|
||||
return sum(observations[member] for member in members), coverage
|
||||
|
||||
|
||||
def compute_history(
|
||||
countries: list[str],
|
||||
mode: str = "global_share",
|
||||
config: ComputeConfiguration | None = None,
|
||||
) -> dict:
|
||||
"""Aggregate only fully covered, same-year compatible observations."""
|
||||
config = config if config is not None else DEFAULT_CONFIGURATION
|
||||
metric = "global_share" if mode == "bilateral_share" else mode
|
||||
annual = {row.year: row for row in config.observations if row.metric == metric}
|
||||
gaps = {row.year: row for row in config.innovation}
|
||||
observed_years = [row.year for row in config.observations] + list(gaps)
|
||||
years = tuple(
|
||||
range(
|
||||
min(observed_years, default=date.today().year),
|
||||
max([date.today().year, *observed_years]) + 1,
|
||||
)
|
||||
)
|
||||
members_by_participant = participant_members(countries)
|
||||
rows = []
|
||||
for year in years:
|
||||
record = annual.get(year)
|
||||
observations = record.values if record else {}
|
||||
values, coverage = {}, {}
|
||||
for code, members in members_by_participant.items():
|
||||
values[code], coverage[code] = _participant_value(members, observations, mode)
|
||||
evidence_id = record.source_id if record else None
|
||||
rows.append(
|
||||
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
|
||||
)
|
||||
|
||||
selected_members = {member for members in members_by_participant.values() for member in members}
|
||||
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
|
||||
return {
|
||||
"version": f"compute-history-{config.reviewed_at}",
|
||||
"provenance": "curated_reports",
|
||||
"reviewed_at": config.reviewed_at.isoformat(),
|
||||
"time_basis": "retrospective",
|
||||
"years": years,
|
||||
"mode": mode,
|
||||
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
|
||||
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
|
||||
mode
|
||||
],
|
||||
"catalog": deepcopy(CATALOG),
|
||||
"default_participants": list(DEFAULT_PARTICIPANTS),
|
||||
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
|
||||
"compute": rows,
|
||||
"innovation": [
|
||||
{
|
||||
"year": year,
|
||||
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
|
||||
"evidence_id": gaps[year].source_id
|
||||
if year in gaps and has_innovation_scope
|
||||
else None,
|
||||
}
|
||||
for year in years
|
||||
],
|
||||
"evidence": {
|
||||
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
|
||||
},
|
||||
"innovation_scope": ["US", "CN"],
|
||||
"limitations": [
|
||||
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
|
||||
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
|
||||
"全球份额不从中美或其他局部成员反推剩余地区。",
|
||||
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
|
||||
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
|
||||
],
|
||||
}
|
||||
125
backend/app/services/compute_history_defaults.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"reviewed_at": "2026-09-22",
|
||||
"sources": [
|
||||
{
|
||||
"id": "caict-2021",
|
||||
"title": "中国算力发展指数白皮书(2021)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2021,
|
||||
"published_at": null,
|
||||
"locator": "正文 11 页图 4(2020 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2022",
|
||||
"title": "中国算力发展指数白皮书(2022)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2022,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2021 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2023",
|
||||
"title": "中国算力发展指数白皮书(2023)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2023,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2024",
|
||||
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2024,
|
||||
"published_at": null,
|
||||
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "innovation-2026",
|
||||
"title": "全球人工智能创新指数报告 2026:科技日报报道",
|
||||
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
|
||||
"publisher": "科技日报(新浪转载)",
|
||||
"report_year": 2026,
|
||||
"published_at": "2026-07-17",
|
||||
"locator": "2023–2025 年美国减中国的指数分差"
|
||||
}
|
||||
],
|
||||
"observations": [
|
||||
{
|
||||
"year": 2020,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2021",
|
||||
"values": {
|
||||
"US": 36,
|
||||
"CN": 31,
|
||||
"JP": 6
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2021,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2022",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"US": 41,
|
||||
"CN": 31,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"CN": 302
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"CN": 435
|
||||
}
|
||||
}
|
||||
],
|
||||
"innovation": [
|
||||
{
|
||||
"year": 2023,
|
||||
"value": 22.02,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2024,
|
||||
"value": 19.96,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2025,
|
||||
"value": 17.95,
|
||||
"source_id": "innovation-2026"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -45,7 +45,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
|
||||
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
|
||||
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
|
||||
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
|
||||
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
|
||||
@@ -56,7 +56,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
|
||||
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
|
||||
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
|
||||
|
||||
@@ -6,8 +6,10 @@ import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
|
||||
PMTILES_ARTIFACT_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
|
||||
)
|
||||
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
|
||||
POLICY_FALLBACK_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
)
|
||||
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
|
||||
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
|
||||
|
||||
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
|
||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@lru_cache(maxsize=16)
|
||||
def _read_versioned_asset_cached(
|
||||
path: Path, mtime_ns: int, size_bytes: int
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
payload = _read_json(path)
|
||||
return payload, _stable_json_hash(payload)
|
||||
|
||||
|
||||
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
|
||||
if not path.exists():
|
||||
return {}, ""
|
||||
stat = path.stat()
|
||||
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
|
||||
|
||||
|
||||
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
|
||||
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
|
||||
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
|
||||
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
|
||||
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
|
||||
|
||||
|
||||
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
|
||||
path = REPO_ROOT / source["path"]
|
||||
payload = _read_json(path)
|
||||
features = payload.get("features") if isinstance(payload, dict) else None
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": features if isinstance(features, list) else [],
|
||||
}
|
||||
|
||||
|
||||
def _write_high_precision_geojson_manifest(
|
||||
sources: list[dict[str, Any]],
|
||||
build_input_hash: str,
|
||||
missing_tools: list[str],
|
||||
) -> dict[str, Any]:
|
||||
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
|
||||
coastline = next(source for source in sources if source["kind"] == "coastline")
|
||||
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
|
||||
|
||||
admin0_payload = _load_source_feature_collection(admin0)
|
||||
coastline_payload = _load_source_feature_collection(coastline)
|
||||
claim_payload = _load_source_feature_collection(claim_lines)
|
||||
for feature in coastline_payload["features"]:
|
||||
props = feature.setdefault("properties", {})
|
||||
if isinstance(props, dict):
|
||||
props["PLANET_LAYER"] = "coastline"
|
||||
|
||||
base_payload = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [*admin0_payload["features"], *coastline_payload["features"]],
|
||||
}
|
||||
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
|
||||
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
|
||||
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
|
||||
_write_json(base_path, base_payload)
|
||||
_write_json(hover_path, admin0_payload)
|
||||
_write_json(claim_path, claim_payload)
|
||||
|
||||
manifest = {
|
||||
"version": "natural-earth-v1",
|
||||
"builtAt": _utc_now_iso(),
|
||||
"tileProvider": "geojson-high-precision",
|
||||
"format": "geojson-directory",
|
||||
"buildInputHash": build_input_hash,
|
||||
"base": base_path.name,
|
||||
"hoverIndex": hover_path.name,
|
||||
"claimLine": claim_path.name,
|
||||
"sourceFeatureCount": {
|
||||
"admin0": len(admin0_payload["features"]),
|
||||
"coastline": len(coastline_payload["features"]),
|
||||
"claimLines": len(claim_payload["features"]),
|
||||
},
|
||||
"pmtiles": None,
|
||||
"missingTools": missing_tools,
|
||||
}
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
|
||||
"--admin0-source", admin0["path"],
|
||||
"--coastline-source", coastline["path"],
|
||||
"--claims-source", claim_lines["path"],
|
||||
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
|
||||
"--build-input-hash", build_input_hash,
|
||||
"--pov-policy", _relative(POV_POLICY_PATH),
|
||||
"--pov-geometry", str(POV_GEOMETRY_PATH),
|
||||
],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode:
|
||||
raise EarthBoundaryBuildError(
|
||||
result.stderr.strip() or "Earth boundary policy build failed",
|
||||
code="policy_build_failed",
|
||||
)
|
||||
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
manifest["missingTools"] = missing_tools
|
||||
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
|
||||
return manifest
|
||||
|
||||
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
|
||||
for source in source_manifest.get("sources", [])
|
||||
],
|
||||
"pov_policy": source_manifest.get("povPolicy"),
|
||||
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
|
||||
"build_config": BUILD_CONFIG,
|
||||
}
|
||||
)
|
||||
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
|
||||
bool(boundary_manifest)
|
||||
and boundary_manifest.get("buildInputHash") == build_input_hash
|
||||
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and _policy_verified(boundary_manifest)
|
||||
and PMTILES_ARTIFACT_PATH.exists()
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
|
||||
|
||||
def _policy_verified(manifest: dict[str, Any]) -> bool:
|
||||
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
|
||||
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
|
||||
coverage = manifest.get("povPolicy") or {}
|
||||
return (
|
||||
policy.get("profile") == "china-pov-v1"
|
||||
and coverage.get("profile") == policy["profile"]
|
||||
and coverage.get("sha256") == policy_hash
|
||||
and coverage.get("reviewedGeometrySourceCommit")
|
||||
== geometry.get("sourceCommit")
|
||||
and coverage.get("geometrySha256") == geometry_hash
|
||||
and coverage.get("applied") is True
|
||||
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
|
||||
<= set(coverage.get("appliedRuleIds") or [])
|
||||
)
|
||||
|
||||
|
||||
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
|
||||
paths = [manifest.get("base"), manifest.get("hoverIndex")]
|
||||
if manifest.get("chinaClaims", {}).get("available"):
|
||||
paths.append(manifest.get("claimLine"))
|
||||
for asset in paths:
|
||||
if not isinstance(asset, str) or Path(asset).name != asset:
|
||||
return False
|
||||
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
|
||||
hover = manifest.get("hoverIndex")
|
||||
return (
|
||||
isinstance(hover, str)
|
||||
and Path(hover).name == hover
|
||||
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
|
||||
)
|
||||
|
||||
|
||||
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
|
||||
high_precision_ready = (
|
||||
manifest_exists
|
||||
and _policy_verified(boundary_manifest)
|
||||
and (
|
||||
(
|
||||
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and pmtiles_exists
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
or (
|
||||
boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
and _geojson_artifacts_exist(boundary_manifest)
|
||||
)
|
||||
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
)
|
||||
)
|
||||
legacy_exists = LEGACY_GEOJSON_PATH.exists()
|
||||
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
|
||||
_versioned_asset(POLICY_FALLBACK_PATH)[0]
|
||||
)
|
||||
provider = (
|
||||
boundary_manifest.get("tileProvider")
|
||||
if high_precision_ready
|
||||
else "legacy-geojson" if legacy_exists else "missing"
|
||||
else "policy-geojson-fallback" if legacy_exists else "missing"
|
||||
)
|
||||
return {
|
||||
"provider": provider,
|
||||
"high_precision_ready": high_precision_ready,
|
||||
"fallback_available": legacy_exists,
|
||||
"policy_publishable": bool(
|
||||
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
|
||||
),
|
||||
"config_source": config_source,
|
||||
"config_path": _relative(LOCAL_CONFIG_PATH),
|
||||
"config_exists": LOCAL_CONFIG_PATH.exists(),
|
||||
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"tileProvider": boundary_manifest.get("tileProvider"),
|
||||
"buildInputHash": boundary_manifest.get("buildInputHash"),
|
||||
"builtAt": boundary_manifest.get("builtAt"),
|
||||
"povPolicy": boundary_manifest.get("povPolicy"),
|
||||
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
|
||||
},
|
||||
"pmtiles": {
|
||||
"path": _relative(PMTILES_ARTIFACT_PATH),
|
||||
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
|
||||
},
|
||||
"legacy": {
|
||||
"path": _relative(LEGACY_GEOJSON_PATH),
|
||||
"path": _relative(POLICY_FALLBACK_PATH),
|
||||
"exists": legacy_exists,
|
||||
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
|
||||
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
|
||||
},
|
||||
"source_manifest": {
|
||||
"path": _relative(SOURCE_MANIFEST_PATH),
|
||||
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
|
||||
build_input_hash,
|
||||
"--pov-policy",
|
||||
_relative(POV_POLICY_PATH),
|
||||
"--pov-geometry",
|
||||
str(POV_GEOMETRY_PATH),
|
||||
]
|
||||
)
|
||||
if progress_callback:
|
||||
|
||||
@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
refresh_strategy = "reload"
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
self.refresh_strategy = "reload"
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
|
||||
if not update:
|
||||
return False
|
||||
|
||||
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
|
||||
keys = payload.get("entity_keys")
|
||||
if not isinstance(keys, list):
|
||||
keys = [payload.get("entity_key")]
|
||||
broadcaster.enqueue_vessel_update({
|
||||
"action": "remove",
|
||||
"vessels": [{"mmsi": key} for key in keys if key is not None],
|
||||
})
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
|
||||
@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import codecs
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from email.utils import parsedate_to_datetime
|
||||
@@ -865,9 +866,9 @@ def _get_locale_text(
|
||||
if isinstance(fallback, dict):
|
||||
value = _coerce_str(fallback.get(key))
|
||||
if value:
|
||||
if locale == "en-US" and _contains_cjk_text(value):
|
||||
return ""
|
||||
return value
|
||||
if locale == "en-US" and _is_chinese_language(item.content_language):
|
||||
return item.title if key == "title" else item.summary
|
||||
return ""
|
||||
|
||||
|
||||
@@ -1800,6 +1801,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _decode_feed_response(response: httpx.Response) -> str:
|
||||
content = response.content
|
||||
for marker, encoding in (
|
||||
(codecs.BOM_UTF32_LE, "utf-32"),
|
||||
(codecs.BOM_UTF32_BE, "utf-32"),
|
||||
(codecs.BOM_UTF16_LE, "utf-16"),
|
||||
(codecs.BOM_UTF16_BE, "utf-16"),
|
||||
(codecs.BOM_UTF8, "utf-8-sig"),
|
||||
):
|
||||
if content.startswith(marker):
|
||||
return content.decode(encoding)
|
||||
|
||||
declaration = re.match(
|
||||
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
|
||||
content,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
encoding = response.charset_encoding or (
|
||||
declaration.group(1).decode("ascii") if declaration else "utf-8"
|
||||
)
|
||||
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
|
||||
return content.decode(encoding)
|
||||
|
||||
|
||||
def _parse_feed_entries(
|
||||
xml_text: str,
|
||||
source: NewsFeedSource,
|
||||
@@ -2307,7 +2332,10 @@ def _diversify_news_items_for_locale(
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for item in ranked:
|
||||
key = _news_item_source_id(item) or item.source or item.feed_name or item.id
|
||||
if active_region == "global":
|
||||
key = item.feed_region or "global"
|
||||
else:
|
||||
key = _news_item_source_id(item) or item.source or item.feed_name or item.id
|
||||
if key not in buckets:
|
||||
buckets[key] = []
|
||||
order.append(key)
|
||||
@@ -2450,7 +2478,9 @@ async def _fetch_single_feed_url(
|
||||
feed=feed,
|
||||
)
|
||||
try:
|
||||
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
|
||||
items = _parse_feed_entries(
|
||||
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
|
||||
)
|
||||
except Exception as exc:
|
||||
error, status = _format_source_fetch_error(
|
||||
status_code=status_code,
|
||||
@@ -2458,7 +2488,7 @@ async def _fetch_single_feed_url(
|
||||
body=response.text,
|
||||
parsed_count=0,
|
||||
)
|
||||
if error is None:
|
||||
if status == "empty":
|
||||
error = str(exc)
|
||||
status = "format_error"
|
||||
return source, [], error, _source_health_result(
|
||||
@@ -2751,6 +2781,7 @@ async def get_earth_news_payload(
|
||||
)
|
||||
await record_earth_news_sources_health(db, health_by_source)
|
||||
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
await _enqueue_unverified_locations(ranked_fetched_items)
|
||||
await upsert_earth_news_items(db, ranked_fetched_items)
|
||||
|
||||
items = await _call_store_list_items(
|
||||
@@ -2802,7 +2833,8 @@ async def get_earth_news_payload(
|
||||
)
|
||||
else:
|
||||
cruise_items = items
|
||||
await _enqueue_unverified_locations(items)
|
||||
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
|
||||
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
|
||||
stale = bool(errors and items)
|
||||
|
||||
return _build_payload(
|
||||
|
||||
@@ -14,10 +14,14 @@ from app.core.logging import get_logger
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
|
||||
TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority"
|
||||
TARGET_LOCATION_GROUP = "earth_news_target_location"
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
|
||||
TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000
|
||||
TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1
|
||||
TARGET_LOCATION_MAX_ATTEMPTS = 3
|
||||
|
||||
_redis_client: redis.Redis | None = None
|
||||
@@ -28,6 +32,7 @@ class NewsTargetLocationMessage:
|
||||
message_id: str
|
||||
item_id: str
|
||||
payload: dict[str, Any]
|
||||
stream_name: str = TARGET_LOCATION_STREAM
|
||||
attempts: int = 0
|
||||
|
||||
|
||||
@@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol):
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
...
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
...
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
@@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:queued:{item_id}"
|
||||
|
||||
|
||||
def _priority_queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:priority_queued:{item_id}"
|
||||
|
||||
|
||||
class RedisStreamsNewsTargetLocationQueue:
|
||||
def __init__(self, client: redis.Redis | None = None) -> None:
|
||||
self.client = client or _get_redis_client()
|
||||
@@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
async def _ensure_group(self) -> None:
|
||||
if self._group_ready:
|
||||
return
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
TARGET_LOCATION_STREAM,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM):
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
self._group_ready = True
|
||||
|
||||
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
|
||||
await self._ensure_group()
|
||||
if force:
|
||||
await self.client.delete(_result_key(item_id), _queued_key(item_id))
|
||||
await self.client.delete(_result_key(item_id))
|
||||
queued_key = _priority_queued_key(item_id)
|
||||
elif await self.client.exists(_result_key(item_id)):
|
||||
return False
|
||||
else:
|
||||
queued_key = _queued_key(item_id)
|
||||
dedup_ttl = (
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS
|
||||
if force
|
||||
else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS
|
||||
)
|
||||
queued = await self.client.set(
|
||||
_queued_key(item_id),
|
||||
queued_key,
|
||||
"1",
|
||||
nx=True,
|
||||
ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
|
||||
ex=dedup_ttl,
|
||||
)
|
||||
if not queued:
|
||||
return bool(await self.client.exists(_queued_key(item_id)))
|
||||
return bool(await self.client.exists(queued_key))
|
||||
stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
stream_name,
|
||||
{
|
||||
"item_id": item_id,
|
||||
"attempts": "0",
|
||||
@@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
block_ms: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
await self._ensure_group()
|
||||
streams = await self.client.xreadgroup(
|
||||
streams = []
|
||||
priority_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_PRIORITY_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if priority_claimed:
|
||||
return priority_claimed
|
||||
|
||||
priority_messages = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
{TARGET_LOCATION_PRIORITY_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS,
|
||||
)
|
||||
if priority_messages:
|
||||
streams = priority_messages
|
||||
else:
|
||||
regular_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if regular_claimed:
|
||||
return regular_claimed
|
||||
streams = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
)
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for _stream_name, stream_messages in streams:
|
||||
for stream_name, stream_messages in streams:
|
||||
for message_id, fields in stream_messages:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
messages.append(
|
||||
NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
attempts=attempts,
|
||||
)
|
||||
)
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
|
||||
async def _claim_stale_messages(
|
||||
self,
|
||||
*,
|
||||
stream_name: str,
|
||||
consumer_name: str,
|
||||
count: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
try:
|
||||
_next_id, claimed, _deleted = await self.client.xautoclaim(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS,
|
||||
start_id="0-0",
|
||||
count=count,
|
||||
)
|
||||
except ResponseError:
|
||||
return []
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for message_id, fields in claimed:
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def _message_from_fields(
|
||||
self,
|
||||
stream_name: str,
|
||||
message_id: str,
|
||||
fields: dict[str, str],
|
||||
) -> NewsTargetLocationMessage | None:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
return NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
stream_name=stream_name,
|
||||
attempts=attempts,
|
||||
)
|
||||
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id)
|
||||
await self.client.xdel(message.stream_name, message.message_id)
|
||||
|
||||
async def _discard_message(self, stream_name: str, message_id: str) -> None:
|
||||
await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id)
|
||||
await self.client.xdel(stream_name, message_id)
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
self,
|
||||
@@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
*,
|
||||
error: str,
|
||||
) -> None:
|
||||
await self.ack(message.message_id)
|
||||
await self.ack(message)
|
||||
if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM,
|
||||
@@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
)
|
||||
return
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
message.stream_name,
|
||||
{
|
||||
"item_id": message.item_id,
|
||||
"attempts": str(message.attempts + 1),
|
||||
@@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS,
|
||||
json.dumps(patch, ensure_ascii=False),
|
||||
)
|
||||
await client.delete(_queued_key(item_id))
|
||||
await client.delete(_queued_key(item_id), _priority_queued_key(item_id))
|
||||
|
||||
@@ -19,6 +19,17 @@ from app.services.earth_news_classification import (
|
||||
normalize_breaking_scope,
|
||||
)
|
||||
|
||||
CRUISE_REGION_ORDER = (
|
||||
"americas",
|
||||
"europe",
|
||||
"middle-east-africa",
|
||||
"asia-pacific",
|
||||
"global",
|
||||
)
|
||||
CRUISE_REGION_QUERY_MULTIPLIER = 12
|
||||
CRUISE_REGION_QUERY_MIN_LIMIT = 240
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT = 1000
|
||||
|
||||
|
||||
def _coerce_datetime(value: datetime | None) -> datetime | None:
|
||||
if value is None:
|
||||
@@ -106,6 +117,41 @@ def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str)
|
||||
)
|
||||
|
||||
|
||||
def _diversify_parsed_news_items_by_region(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
if limit <= 0:
|
||||
return []
|
||||
sorted_items = _sort_parsed_news_items(items, active_region="global")
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
for item in sorted_items:
|
||||
region = item.feed_region or "global"
|
||||
buckets.setdefault(region, []).append(item)
|
||||
|
||||
ordered_regions = [
|
||||
*[region for region in CRUISE_REGION_ORDER if buckets.get(region)],
|
||||
*sorted(region for region in buckets if region not in CRUISE_REGION_ORDER),
|
||||
]
|
||||
diversified: list[ParsedNewsItem] = []
|
||||
cursor = 0
|
||||
while len(diversified) < limit:
|
||||
added = False
|
||||
for region in ordered_regions:
|
||||
bucket = buckets.get(region) or []
|
||||
if cursor >= len(bucket):
|
||||
continue
|
||||
diversified.append(bucket[cursor])
|
||||
added = True
|
||||
if len(diversified) >= limit:
|
||||
break
|
||||
if not added:
|
||||
break
|
||||
cursor += 1
|
||||
return diversified
|
||||
|
||||
|
||||
def _query_sort_key(active_region: str):
|
||||
if active_region == "global":
|
||||
return (
|
||||
@@ -167,6 +213,8 @@ async def list_earth_news_items(
|
||||
[record_to_parsed_news_item(record) for record in records],
|
||||
active_region=active_region,
|
||||
)
|
||||
if active_region == "global" and not source_ids:
|
||||
return _diversify_parsed_news_items_by_region(items, limit=limit)
|
||||
return items[:limit]
|
||||
|
||||
|
||||
@@ -177,15 +225,20 @@ async def list_earth_news_cruise_items(
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
query_limit = min(
|
||||
max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT),
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT,
|
||||
)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.last_seen_at.desc(),
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
.limit(min(max(limit * 4, limit), 200))
|
||||
.limit(query_limit)
|
||||
)
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
@@ -194,11 +247,10 @@ async def list_earth_news_cruise_items(
|
||||
if source_clause is not None:
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
items = _sort_parsed_news_items(
|
||||
return _diversify_parsed_news_items_by_region(
|
||||
[record_to_parsed_news_item(record) for record in result.scalars().all()],
|
||||
active_region="global",
|
||||
limit=limit,
|
||||
)
|
||||
return items[:limit]
|
||||
|
||||
|
||||
async def get_earth_news_freshness(
|
||||
|
||||
@@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news")
|
||||
WORKER_BATCH_SIZE = 4
|
||||
WORKER_BLOCK_MS = 5000
|
||||
WORKER_BACKOFF_SECONDS = 5.0
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0
|
||||
WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0
|
||||
|
||||
_worker_task: asyncio.Task | None = None
|
||||
|
||||
@@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None:
|
||||
if not messages:
|
||||
continue
|
||||
provider_client = await _build_provider_client()
|
||||
for message in messages:
|
||||
job_timeout = _get_worker_job_timeout(provider_client)
|
||||
|
||||
async def handle_message(message: NewsTargetLocationMessage) -> None:
|
||||
try:
|
||||
await process_target_location_message(message, provider_client=provider_client)
|
||||
await queue.ack(message.message_id)
|
||||
await asyncio.wait_for(
|
||||
process_target_location_message(message, provider_client=provider_client),
|
||||
timeout=job_timeout,
|
||||
)
|
||||
await queue.ack(message)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except TimeoutError as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job timed out",
|
||||
event="earth_news.target_location.worker_job_timeout",
|
||||
context={"item_id": message.item_id, "timeout_seconds": job_timeout},
|
||||
)
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out")
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job failed",
|
||||
@@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None:
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc))
|
||||
|
||||
await asyncio.gather(*(handle_message(message) for message in messages))
|
||||
|
||||
|
||||
def start_earth_news_target_worker() -> None:
|
||||
global _worker_task
|
||||
@@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None:
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
_worker_task = None
|
||||
|
||||
|
||||
def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float:
|
||||
timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS)
|
||||
retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1)
|
||||
return min(
|
||||
max(
|
||||
timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS,
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS,
|
||||
),
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS,
|
||||
)
|
||||
|
||||
198
backend/app/services/epoch_location.py
Normal file
@@ -0,0 +1,198 @@
|
||||
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
|
||||
|
||||
_COUNTRY_LABEL_OVERRIDES = {
|
||||
"korea republic of": "韩国",
|
||||
}
|
||||
_REGION_ONLY_LABELS = {
|
||||
"europe",
|
||||
"iowa",
|
||||
"maryland",
|
||||
"new mexico",
|
||||
"texas",
|
||||
"the nordics",
|
||||
"virginia",
|
||||
"washington",
|
||||
}
|
||||
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
|
||||
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
|
||||
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
|
||||
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
|
||||
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
|
||||
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
|
||||
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
|
||||
)
|
||||
_NUMBERED_STREET_PATTERN = re.compile(
|
||||
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
|
||||
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
|
||||
r"highway|hwy|way|parkway|pkwy|chome)\b|"
|
||||
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_POSTAL_CODE_PATTERN = re.compile(
|
||||
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
|
||||
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_SITE_CONTEXT_PATTERN = re.compile(
|
||||
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
|
||||
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_COORDINATE_CUE_PATTERN = re.compile(
|
||||
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EpochLocationAssessment:
|
||||
precision: str
|
||||
reason: str
|
||||
|
||||
@property
|
||||
def is_renderable(self) -> bool:
|
||||
return self.precision in {"precise", "site", "city"}
|
||||
|
||||
|
||||
def _normalized_label(value: Any) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
|
||||
|
||||
|
||||
def _country_from_label(value: Any) -> str | None:
|
||||
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
|
||||
_normalized_label(value)
|
||||
)
|
||||
|
||||
|
||||
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
|
||||
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
|
||||
if isinstance(raw, bool):
|
||||
return None
|
||||
try:
|
||||
number = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if math.isfinite(number) and abs(number) <= bound else None
|
||||
|
||||
|
||||
def _has_explicit_coordinate_pair(text: str) -> bool:
|
||||
hemispheres = {}
|
||||
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
|
||||
hemisphere = match.group("hemisphere").upper()
|
||||
degrees = int(match.group("degrees"))
|
||||
minutes = int(match.group("minutes"))
|
||||
seconds = float(match.group("seconds") or 0)
|
||||
bound = 90 if hemisphere in {"N", "S"} else 180
|
||||
if degrees <= bound and minutes < 60 and seconds < 60:
|
||||
hemispheres[hemisphere] = degrees
|
||||
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
|
||||
return True
|
||||
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
|
||||
if labeled_pair:
|
||||
latitude = float(labeled_pair.group("latitude"))
|
||||
longitude = float(labeled_pair.group("longitude"))
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
|
||||
if not decimal_pair:
|
||||
return False
|
||||
latitude, longitude = map(float, decimal_pair.groups())
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
|
||||
|
||||
def _has_address_evidence(text: str) -> bool:
|
||||
if _NUMBERED_STREET_PATTERN.search(text):
|
||||
return True
|
||||
return bool(
|
||||
_POSTAL_CODE_PATTERN.search(text)
|
||||
and ("\n" in text or text.count(",") >= 2)
|
||||
and _SITE_CONTEXT_PATTERN.search(text)
|
||||
)
|
||||
|
||||
|
||||
def assess_epoch_location(
|
||||
location: Any,
|
||||
*,
|
||||
country: Any = None,
|
||||
raw_country: Any = None,
|
||||
) -> EpochLocationAssessment:
|
||||
"""Classify source coordinates without upgrading their source text.
|
||||
|
||||
Epoch's CSV supplies coordinates even when ``Location`` contains only a
|
||||
country. Those coordinates are useful source evidence, but are not a
|
||||
facility point and must not enter the renderable location pipeline.
|
||||
"""
|
||||
if isinstance(location, dict):
|
||||
latitude = _coordinate(location, ("lat", "latitude"), 90)
|
||||
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
|
||||
has_coordinates = latitude is not None and longitude is not None
|
||||
return EpochLocationAssessment(
|
||||
"precise" if has_coordinates else "unknown",
|
||||
"complete explicit coordinate object"
|
||||
if has_coordinates
|
||||
else "coordinate object lacks a complete valid latitude/longitude pair",
|
||||
)
|
||||
|
||||
text = str(location or "").strip()
|
||||
if not text:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is empty")
|
||||
|
||||
normalized = _normalized_label(text)
|
||||
normalized_country = normalize_country(country)
|
||||
source_country = normalize_country(raw_country)
|
||||
location_country = _country_from_label(text)
|
||||
if location_country or (
|
||||
raw_country and normalized == _normalized_label(raw_country)
|
||||
):
|
||||
matched = location_country or source_country or normalized_country
|
||||
return EpochLocationAssessment(
|
||||
"country",
|
||||
f"Epoch Location identifies country only: {matched or text}",
|
||||
)
|
||||
if normalized in _REGION_ONLY_LABELS:
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
|
||||
if (
|
||||
len(comma_parts) == 2
|
||||
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
|
||||
and _country_from_label(comma_parts[1])
|
||||
):
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
if normalized in _UNLOCATABLE_LABELS:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
|
||||
if normalized.isdigit():
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
|
||||
if _has_explicit_coordinate_pair(text):
|
||||
return EpochLocationAssessment(
|
||||
"precise", "Epoch Location contains a complete valid coordinate pair"
|
||||
)
|
||||
if _COORDINATE_CUE_PATTERN.search(text):
|
||||
return EpochLocationAssessment(
|
||||
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
|
||||
)
|
||||
if _has_address_evidence(text):
|
||||
return EpochLocationAssessment("precise", "Epoch Location contains an address")
|
||||
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
|
||||
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
|
||||
return EpochLocationAssessment("city", "Epoch Location identifies a locality")
|
||||
186
backend/app/services/llm_model_catalog.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Authenticated model discovery shared by refresh and connection checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
CATALOG_TIMEOUT_SECONDS = 30
|
||||
CATALOG_MAX_PAGES = 100
|
||||
CATALOG_PAGE_SIZE = 100
|
||||
CATALOG_REQUEST_ATTEMPTS = 2
|
||||
CATALOG_RETRY_DELAY_SECONDS = 0.2
|
||||
SUPPORTED_PROVIDER_APIS = {
|
||||
"openai-completions",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"ollama-generate",
|
||||
}
|
||||
|
||||
|
||||
class LLMProviderCatalogError(RuntimeError):
|
||||
"""A safe, user-facing catalog failure with no upstream response or credentials."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCatalog:
|
||||
url: str
|
||||
models: list[str]
|
||||
|
||||
|
||||
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
if parts.scheme not in {"http", "https"} or not parts.hostname:
|
||||
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
|
||||
if parts.username or parts.password or parts.query or parts.fragment:
|
||||
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
|
||||
if provider_api not in SUPPORTED_PROVIDER_APIS:
|
||||
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
|
||||
path = parts.path.rstrip("/")
|
||||
if provider_api == "ollama-generate":
|
||||
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
|
||||
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
|
||||
path = "/api/v1/models"
|
||||
else:
|
||||
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
|
||||
path += "/v1"
|
||||
path += "/models"
|
||||
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
|
||||
|
||||
|
||||
def catalog_error_message(exc: Exception) -> str:
|
||||
if isinstance(exc, LLMProviderCatalogError):
|
||||
return str(exc)
|
||||
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
|
||||
return "模型目录请求超时,请检查网络后重试。"
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
code = exc.response.status_code
|
||||
messages = {
|
||||
401: "API Key 验证失败,请检查当前供应商的凭证。",
|
||||
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
|
||||
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
|
||||
429: "供应商请求限流,请稍后重试。",
|
||||
}
|
||||
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return "无法连接模型目录,请检查基础地址和网络。"
|
||||
return "模型目录响应无效,请稍后重试。"
|
||||
|
||||
|
||||
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
envelope = payload.get("output", payload)
|
||||
if not isinstance(envelope, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
rows = envelope.get("data", envelope.get("models"))
|
||||
if not isinstance(rows, list):
|
||||
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
|
||||
return rows, envelope
|
||||
|
||||
|
||||
def _model_id(row: dict[str, object]) -> str:
|
||||
value = row.get("id") or row.get("model") or row.get("name")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _model_date(row: dict[str, object]) -> float:
|
||||
value = row.get("created_at") or row.get("published_time") or row.get("created")
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
async def _get_catalog_page(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
headers: dict[str, str],
|
||||
params: dict[str, str | int],
|
||||
) -> object:
|
||||
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
|
||||
try:
|
||||
response = await client.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if attempt or exc.response.status_code not in {502, 503, 504}:
|
||||
raise
|
||||
except httpx.TransportError:
|
||||
if attempt:
|
||||
raise
|
||||
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
|
||||
raise LLMProviderCatalogError("模型目录请求失败。")
|
||||
|
||||
|
||||
async def fetch_model_catalog(
|
||||
provider: str,
|
||||
base_url: str,
|
||||
provider_api: str,
|
||||
api_key: str = "",
|
||||
anthropic_version: str = "2023-06-01",
|
||||
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
|
||||
) -> ModelCatalog:
|
||||
url = model_catalog_url(provider, base_url, provider_api)
|
||||
public_catalog = provider in {"opencode-go", "openrouter"}
|
||||
if not api_key and provider_api != "ollama-generate" and not public_catalog:
|
||||
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
|
||||
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
|
||||
if provider_api == "anthropic-messages" and provider not in {
|
||||
"opencode-go",
|
||||
"openrouter",
|
||||
"alibaba",
|
||||
"moonshotai",
|
||||
}:
|
||||
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
|
||||
elif api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
|
||||
params: dict[str, str | int] = {}
|
||||
if native_dashscope:
|
||||
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
|
||||
rows_by_id: dict[str, dict[str, object]] = {}
|
||||
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
|
||||
# Bound the complete pagination/retry cycle, not just each individual request.
|
||||
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
|
||||
for page in range(CATALOG_MAX_PAGES):
|
||||
payload = await _get_catalog_page(client, url, headers, params)
|
||||
rows, envelope = _model_rows(payload)
|
||||
previous_count = len(rows_by_id)
|
||||
for row in rows:
|
||||
rows_by_id[_model_id(row)] = row
|
||||
has_more = envelope.get("has_more") is True
|
||||
if native_dashscope:
|
||||
total = envelope.get("total")
|
||||
if not isinstance(total, int) or total < 0:
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
has_more = len(rows_by_id) < total
|
||||
params["page_no"] = page + 2
|
||||
elif has_more:
|
||||
cursor = envelope.get("last_id")
|
||||
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
params["after_id"] = cursor
|
||||
if not has_more:
|
||||
models = sorted(
|
||||
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
|
||||
)
|
||||
# An empty Ollama catalog is valid: no models have been installed yet.
|
||||
if not models and provider_api != "ollama-generate":
|
||||
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
|
||||
return ModelCatalog(url=url, models=models)
|
||||
if len(rows_by_id) == previous_count:
|
||||
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
|
||||
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")
|
||||
@@ -3,17 +3,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
from app.services.llm_model_catalog import fetch_model_catalog
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m3": "anthropic-messages",
|
||||
"qwen3.8-max": "anthropic-messages",
|
||||
"qwen3.8-flash": "anthropic-messages",
|
||||
"qwen3.7-max": "anthropic-messages",
|
||||
"qwen3.7-plus": "anthropic-messages",
|
||||
"qwen3.6-plus": "anthropic-messages",
|
||||
"grok-4.6": "openai-responses",
|
||||
"gpt-5.6-luna": "openai-responses",
|
||||
"muse-spark-1.3-contributor": "openai-responses",
|
||||
"muse-spark-1.2-contributor": "openai-responses",
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m3",
|
||||
"kimi-k3",
|
||||
"glm-5.3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
|
||||
]
|
||||
|
||||
|
||||
OPENAI_MODEL_PROVIDER_APIS = {
|
||||
model: "openai-responses"
|
||||
for model in [
|
||||
"gpt-6-astra",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.1",
|
||||
"gpt-5.1-codex",
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"model": "MiniMax-M3",
|
||||
"models": [
|
||||
"MiniMax-M3",
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
"MiniMax-M2.1",
|
||||
"MiniMax-M2.1-highspeed",
|
||||
"MiniMax-M2",
|
||||
],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": "openai-responses",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"model": "gpt-6-astra",
|
||||
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
|
||||
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"model": "claude-opus-5",
|
||||
"models": ["claude-opus-5", "claude-sonnet-4-6"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"model": "deepseek-flash",
|
||||
"models": ["deepseek-flash", "deepseek-v4-pro"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"model": "qwen3.8-max",
|
||||
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"model": "kimi-k3",
|
||||
"models": ["kimi-k3", "kimi-k2.5"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
async def refresh_llm_provider_preset(
|
||||
provider: str,
|
||||
api_key: str | None = None,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
provider_api: str | None = None,
|
||||
anthropic_version: str = "2023-06-01",
|
||||
) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
resolved_base_url = base_url or fallback["base_url"]
|
||||
resolved_api = provider_api or fallback["provider_api"]
|
||||
catalog = await fetch_model_catalog(
|
||||
fallback["provider"],
|
||||
resolved_base_url,
|
||||
resolved_api,
|
||||
api_key or "",
|
||||
anthropic_version,
|
||||
)
|
||||
model_provider_apis = (
|
||||
_opencode_go_model_provider_apis(catalog.models)
|
||||
if fallback["provider"] == "opencode-go"
|
||||
else fallback.get("model_provider_apis", {})
|
||||
)
|
||||
if (
|
||||
fallback["provider"] == "openai"
|
||||
and urlsplit(resolved_base_url).hostname != "api.openai.com"
|
||||
):
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
"base_url": resolved_base_url,
|
||||
"provider_api": resolved_api,
|
||||
"model": catalog.models[0] if catalog.models else "",
|
||||
"models": catalog.models,
|
||||
"model_provider_apis": model_provider_apis,
|
||||
"source": catalog.url,
|
||||
}
|
||||
return refreshed
|
||||
|
||||
@@ -635,6 +635,7 @@ async def _run_assistant_message(
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"session_id": session_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
|
||||
148
backend/app/services/tv_catalog.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""Search and paginate the public live TV catalog at the database boundary."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Select, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.services.tv_streams import (
|
||||
TV_LIVE_SOURCE_COLLECTOR,
|
||||
TV_LIVE_SOURCE_DATA_TYPE,
|
||||
_build_collected_tv_source,
|
||||
build_public_tv_payload,
|
||||
get_tv_settings_payload,
|
||||
)
|
||||
|
||||
|
||||
def _source_key():
|
||||
return func.coalesce(
|
||||
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
|
||||
func.nullif(CollectedData.source_id, ""),
|
||||
CollectedData.entity_key,
|
||||
)
|
||||
|
||||
|
||||
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
source_id = _source_key()
|
||||
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
|
||||
ranked = (
|
||||
select(
|
||||
CollectedData.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=source_id,
|
||||
order_by=CollectedData.id.desc(),
|
||||
)
|
||||
.label("source_rank"),
|
||||
)
|
||||
.where(
|
||||
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
|
||||
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
|
||||
CollectedData.is_current.is_(True),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.deleted_at.is_(None),
|
||||
enabled.notin_(("false", "0", "no", "off")),
|
||||
source_id.notin_(configured_ids),
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
return (
|
||||
select(CollectedData)
|
||||
.join(ranked, ranked.c.id == CollectedData.id)
|
||||
.where(ranked.c.source_rank == 1)
|
||||
)
|
||||
|
||||
|
||||
def _filter_catalog_query(
|
||||
query: Select[tuple[CollectedData]], terms: list[str]
|
||||
) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
searchable = func.lower(
|
||||
func.concat_ws(
|
||||
" ",
|
||||
CollectedData.name,
|
||||
CollectedData.title,
|
||||
CollectedData.source_id,
|
||||
metadata["name"].as_string(),
|
||||
metadata["provider"].as_string(),
|
||||
metadata["region"].as_string(),
|
||||
metadata["country"].as_string(),
|
||||
metadata["language"].as_string(),
|
||||
)
|
||||
)
|
||||
for term in terms:
|
||||
query = query.where(searchable.contains(term, autoescape=True))
|
||||
return query
|
||||
|
||||
|
||||
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
|
||||
searchable = " ".join(
|
||||
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
|
||||
).lower()
|
||||
return all(term in searchable for term in terms)
|
||||
|
||||
|
||||
async def get_tv_catalog_page(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
q: str = "",
|
||||
selected_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
settings = await get_tv_settings_payload(db)
|
||||
payload = build_public_tv_payload(settings, [])
|
||||
configured = payload["sources"]
|
||||
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
|
||||
if selected_id:
|
||||
selected = next((source for source in configured if source["id"] == selected_id), None)
|
||||
if selected is None:
|
||||
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
|
||||
selected = _build_collected_tv_source(record, 0) if record else None
|
||||
if selected:
|
||||
payload["selected_source"] = selected
|
||||
summary = (
|
||||
await db.execute(
|
||||
select(func.count(), func.max(CollectedData.collected_at))
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
|
||||
)
|
||||
).one()
|
||||
total_collected, latest_update = summary
|
||||
terms = q.lower().split()
|
||||
matched_configured = [source for source in configured if _matches_source(source, terms)]
|
||||
filtered_query = _filter_catalog_query(query, terms)
|
||||
matched_collected = (
|
||||
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
|
||||
if terms
|
||||
else total_collected
|
||||
)
|
||||
sources = matched_configured[offset : offset + limit]
|
||||
remaining = limit - len(sources)
|
||||
if remaining:
|
||||
rows = await db.scalars(
|
||||
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
|
||||
.offset(max(0, offset - len(matched_configured)))
|
||||
.limit(remaining)
|
||||
)
|
||||
sources.extend(
|
||||
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
|
||||
)
|
||||
total = len(matched_configured) + matched_collected
|
||||
next_offset = offset + len(sources)
|
||||
return {
|
||||
**payload,
|
||||
"sources": sources,
|
||||
"source_count": len(configured) + total_collected,
|
||||
"latest_updated_at": (
|
||||
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
|
||||
),
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
|
||||
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
|
||||
UNSUPPORTED_CCTV4_STREAM = (
|
||||
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
|
||||
)
|
||||
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"name": "CCTV-4 中文国际(美洲版)",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"embed_url": "",
|
||||
"stream_url": CCTV4_STREAM,
|
||||
"use_proxy": False,
|
||||
"homepage_url": CCTV4_HOMEPAGE,
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
"notes": CCTV4_PLAYBACK_NOTE,
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
|
||||
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
|
||||
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
|
||||
payload.get("stream_url")
|
||||
) == UNSUPPORTED_CCTV4_STREAM:
|
||||
payload.update(
|
||||
name="CCTV-4 中文国际(美洲版)",
|
||||
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
|
||||
use_proxy=False,
|
||||
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
|
||||
)
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
default_source = next(
|
||||
source for source in DEFAULT_TV_SETTINGS["sources"]
|
||||
if source["id"] == DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
normalize_tv_source(default_source, index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
|
||||
@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
|
||||
return [item.to_dict() for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_current_vessels_by_mmsi(
|
||||
db: AsyncSession, mmsis: list[int]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read canonical render state for the vessels changed by a stream flush."""
|
||||
if not mmsis:
|
||||
return []
|
||||
result = await db.execute(
|
||||
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
|
||||
)
|
||||
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def aggregate_vessel_observations(
|
||||
db: AsyncSession,
|
||||
observations: Iterable[AISRawObservation],
|
||||
|
||||
72
backend/tests/test_auth_identity.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.auth import router
|
||||
from app.core import security
|
||||
from app.db.session import DEFAULT_LOGIN_USERS, get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.user import UserRegister
|
||||
|
||||
|
||||
async def identity_request(monkeypatch, user, token=None):
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = user
|
||||
db = AsyncMock()
|
||||
db.execute.return_value = result
|
||||
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/auth")
|
||||
|
||||
async def override_db():
|
||||
yield db
|
||||
|
||||
app.dependency_overrides[get_db] = override_db
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
return await client.get(
|
||||
"/auth/me",
|
||||
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
|
||||
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
|
||||
created = datetime(2026, 9, 21, tzinfo=UTC)
|
||||
user = User(
|
||||
id=1, username=default["username"], email=default["email"],
|
||||
role=default["role"], password_hash="not-exposed", is_active=True,
|
||||
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
|
||||
)
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["email"] == default["email"]
|
||||
assert payload["email_verified"] is True
|
||||
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
|
||||
assert payload["gatekeeper_groups"] == ["ops"]
|
||||
assert "password_hash" not in payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
|
||||
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
|
||||
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_registration_still_validates_email():
|
||||
with pytest.raises(ValidationError):
|
||||
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")
|
||||
218
backend/tests/test_collector_downloads.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""Request-level regressions for quota-limited satellite downloads."""
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
|
||||
from app.services.collectors.downloads import ResumableFileDownloader
|
||||
|
||||
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
|
||||
|
||||
|
||||
def setup_collector(monkeypatch, tmp_path, handler):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/gp.php"
|
||||
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
|
||||
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
original_client = httpx.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
"app.services.collectors.celestrak.httpx.AsyncClient",
|
||||
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
|
||||
)
|
||||
return collector
|
||||
|
||||
|
||||
def expire(path):
|
||||
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
|
||||
os.utime(path, (old, old))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
|
||||
return httpx.Response(
|
||||
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert len(requests) == 1
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
|
||||
assert requests[1].headers["if-none-match"] == '"v1"'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
|
||||
assert requests == ["GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
|
||||
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
with pytest.raises(RuntimeError, match="validation failed"):
|
||||
await collector.fetch()
|
||||
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path, _, meta = collector._downloader._cache_paths(URL, ".json")
|
||||
path.write_text("[]")
|
||||
meta.write_text('{"etag":"old"}')
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
|
||||
assert "if-none-match" not in requests[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
|
||||
)
|
||||
path, _, meta = downloader._cache_paths(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
expire(path)
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(304)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
for _ in range(2):
|
||||
await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert len(requests) == 1
|
||||
assert requests[0].headers["if-none-match"] == "v1"
|
||||
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
|
||||
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
|
||||
body = gzip.compress(raw)
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200,
|
||||
content=body,
|
||||
headers={"content-encoding": "gzip", "content-length": str(len(body))},
|
||||
)
|
||||
)
|
||||
) as client:
|
||||
path = await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert path.read_bytes() == raw
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
final, part, meta = downloader._cache_paths(URL, ".bin")
|
||||
part.parent.mkdir(parents=True)
|
||||
part.write_bytes(b"abc")
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.method == "HEAD":
|
||||
return httpx.Response(
|
||||
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
|
||||
)
|
||||
assert request.headers["range"] == "bytes=3-"
|
||||
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".bin")
|
||||
assert [r.method for r in requests] == ["HEAD", "GET"]
|
||||
assert final.read_bytes() == b"abcdef"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.url.params["GROUP"] == "active":
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
|
||||
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
|
||||
assert [r.method for r in requests] == ["GET", "GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generic_cache_without_validators_must_refresh(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
path = downloader.cached_file_path(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text("[]")
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".json")
|
||||
assert requests == ["HEAD", "GET"]
|
||||
assert path.read_text() == "[1]"
|
||||
@@ -79,7 +79,8 @@ class TestTOP500Collector:
|
||||
"""Test parsing empty response"""
|
||||
collector = TOP500Collector()
|
||||
result = collector.parse_response("<html><body><table></table></body></html>")
|
||||
assert len(result) > 0
|
||||
assert result == []
|
||||
assert collector.fail_on_empty
|
||||
|
||||
def test_parse_response_single_item(self):
|
||||
"""Test parsing single item response"""
|
||||
@@ -97,9 +98,9 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["source_id"] == "top500_1"
|
||||
assert result[0]["source_id"] == "top500_system_1"
|
||||
assert result[0]["name"] == "Test Supercomputer"
|
||||
assert result[0]["country"] == "USA"
|
||||
assert result[0]["metadata"]["rank"] == 1
|
||||
@@ -113,7 +114,7 @@ class TestTOP500Collector:
|
||||
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
|
||||
<tr>
|
||||
<td>1</td>
|
||||
<td>Valid\nVendor\nSite\nUSA</td>
|
||||
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
|
||||
<td>1000</td>
|
||||
<td>10 PFLOP/s</td>
|
||||
<td>12 PFLOP/s</td>
|
||||
@@ -129,7 +130,7 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["name"] == "Valid"
|
||||
|
||||
|
||||
222
backend/tests/test_compute_configuration.py
Normal file
@@ -0,0 +1,222 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
default_envelope,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
|
||||
def default_config():
|
||||
return DEFAULT_CONFIGURATION.model_dump(mode="json")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation",
|
||||
[
|
||||
lambda value: value["observations"][0]["values"].update(CN=101),
|
||||
lambda value: value["observations"][0]["values"].update(CN=-1),
|
||||
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
|
||||
lambda value: value["observations"][0]["values"].update(CN=True),
|
||||
lambda value: value["observations"][0]["values"].update(EUROPE=10),
|
||||
lambda value: value["observations"][0].update(source_id="unknown"),
|
||||
lambda value: value["observations"][0].update(year=2026),
|
||||
lambda value: value["observations"].append(value["observations"][0]),
|
||||
lambda value: value["sources"].append(value["sources"][0]),
|
||||
lambda value: value["innovation"].append(value["innovation"][0]),
|
||||
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
|
||||
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_or_ambiguous_evidence(mutation):
|
||||
payload = default_config()
|
||||
mutation(payload)
|
||||
with pytest.raises(ValidationError):
|
||||
ComputeConfiguration.model_validate(payload)
|
||||
|
||||
|
||||
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
|
||||
payload = default_config()
|
||||
payload["sources"].append(
|
||||
{
|
||||
"id": "new-report",
|
||||
"title": "New report",
|
||||
"url": "https://example.com/report",
|
||||
"publisher": "Research group",
|
||||
"report_year": 2026,
|
||||
"locator": "Figure 2",
|
||||
}
|
||||
)
|
||||
payload["observations"].append(
|
||||
{
|
||||
"year": 2025,
|
||||
"metric": "global_share",
|
||||
"source_id": "new-report",
|
||||
"values": {"CN": 25, "US": 50, "JP": 0},
|
||||
}
|
||||
)
|
||||
config = ComputeConfiguration.model_validate(payload)
|
||||
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
|
||||
row = next(row for row in result["compute"] if row["year"] == 2025)
|
||||
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
|
||||
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
|
||||
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
|
||||
|
||||
|
||||
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
|
||||
payload = default_config()
|
||||
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
|
||||
result = compute_history(
|
||||
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
|
||||
)
|
||||
assert result["compute"][0]["values"] == {"CN": None, "US": None}
|
||||
payload.update(sources=[], observations=[], innovation=[])
|
||||
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
|
||||
assert all(row["values"]["CN"] is None for row in result["compute"])
|
||||
assert result["evidence"] == {}
|
||||
|
||||
|
||||
def fake_db(record=None):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = record
|
||||
result.scalar_one.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_install_uses_defaults_without_writing_database():
|
||||
db = fake_db()
|
||||
assert await read_configuration(db) == default_envelope()
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_series_reads_persisted_override_and_disables_http_cache():
|
||||
envelope = default_envelope()
|
||||
envelope["is_default"] = False
|
||||
envelope["revision"] = "custom"
|
||||
envelope["config"]["observations"][3]["values"]["CN"] = 30
|
||||
response = Response()
|
||||
data = await get_series(
|
||||
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
|
||||
)
|
||||
assert data["compute"][3]["values"]["CN"] == 30
|
||||
assert data["version"] == "custom"
|
||||
assert data["provenance"] == "admin_curated_reports"
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
actor = SimpleNamespace(id=1, username="admin")
|
||||
old = deepcopy(record.payload)
|
||||
result = await publish_configuration(
|
||||
db,
|
||||
revision=old["revision"],
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=actor,
|
||||
)
|
||||
assert result["revision"] != old["revision"]
|
||||
audit = db.add.call_args.args[0]
|
||||
assert isinstance(audit, AuditLog)
|
||||
assert audit.details == {"before": old, "after": result}
|
||||
assert audit.actor_id == actor.id
|
||||
db.commit.assert_awaited_once()
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await publish_configuration(
|
||||
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
|
||||
)
|
||||
assert failure.value.status_code == 409
|
||||
assert record.payload == result
|
||||
db.rollback.assert_awaited_once()
|
||||
db.commit.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reset_changes_revision_and_restores_bundled_data():
|
||||
record = SimpleNamespace(
|
||||
payload={**default_envelope(), "revision": "custom", "is_default": False}
|
||||
)
|
||||
result = await publish_configuration(
|
||||
fake_db(record),
|
||||
revision="custom",
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=SimpleNamespace(id=1, username="admin"),
|
||||
reset=True,
|
||||
)
|
||||
assert result["is_default"] is True
|
||||
assert result["config"] == default_config()
|
||||
assert result["revision"] != "custom"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
|
||||
async def test_non_admin_cannot_configure(role):
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await configuration_admin(SimpleNamespace(role=role))
|
||||
assert failure.value.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_http_configuration_auth_validation_and_public_series():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
assert (await client.get("/series")).status_code == 200
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
id=1,
|
||||
username="admin",
|
||||
role="admin",
|
||||
)
|
||||
loaded = (await client.get("/configuration")).json()
|
||||
invalid = deepcopy(loaded["config"])
|
||||
invalid["observations"][0]["values"]["CN"] = 200
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": invalid,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
db.commit.assert_not_awaited()
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": loaded["config"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["revision"] != loaded["revision"]
|
||||
assert (
|
||||
await client.post(
|
||||
"/configuration/reset",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
},
|
||||
)
|
||||
).status_code == 409
|
||||
268
backend/tests/test_compute_intelligence.py
Normal file
@@ -0,0 +1,268 @@
|
||||
import csv
|
||||
import io
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from app.api.v1.compute_intelligence import sample_query, selected_countries
|
||||
from app.services.collectors.epoch_ai import EpochAIGPUCollector
|
||||
from app.services.collectors.top500 import TOP500Collector
|
||||
from app.services.compute_history import compute_history
|
||||
|
||||
|
||||
def epoch_csv(**overrides):
|
||||
row = {
|
||||
"Name": "Cluster Alpha",
|
||||
"Country": "United States of America",
|
||||
"Status": "Existing",
|
||||
"First Operational Date": "2023-05-02",
|
||||
"16-bit OP/s (log)": "18",
|
||||
"Location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"latitude": "35",
|
||||
"longitude": "-90",
|
||||
**overrides,
|
||||
}
|
||||
output = io.StringIO()
|
||||
writer = csv.DictWriter(output, fieldnames=list(row))
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def test_epoch_csv_preserves_precision_date_and_missing_values():
|
||||
collector = EpochAIGPUCollector()
|
||||
row = collector.parse_response(epoch_csv())[0]
|
||||
assert row["value"] == 1e18
|
||||
assert row["unit"] == "FLOP/s"
|
||||
assert row["metadata"]["precision"] == "FP16"
|
||||
assert row["metadata"]["location_precision"] == "precise"
|
||||
assert row["metadata"]["raw_country"] == "United States of America"
|
||||
assert row["reference_date"] == "2023-05-02"
|
||||
missing = collector.parse_response(
|
||||
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
|
||||
)[0]
|
||||
assert missing["value"] is None
|
||||
assert missing["reference_date"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "country", "expected"),
|
||||
[
|
||||
("United States of America", "United States of America", "country"),
|
||||
("Japan", "Japan", "country"),
|
||||
("Korea (Republic of)", "South Korea", "country"),
|
||||
("Texas", "United States of America", "region"),
|
||||
("Iowa, USA", "United States of America", "region"),
|
||||
("Seoul", "South Korea", "city"),
|
||||
("Paris, France", "France", "city"),
|
||||
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
|
||||
("Cluster 2, Texas", "United States of America", "site"),
|
||||
("Campus 7, Tokyo", "Japan", "site"),
|
||||
("Building 240, Argonne", "United States of America", "site"),
|
||||
("7", "United States of America", "unknown"),
|
||||
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
|
||||
("35.380924,139.625008", "Japan", "precise"),
|
||||
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
|
||||
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
|
||||
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
|
||||
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
|
||||
],
|
||||
)
|
||||
def test_epoch_location_precision_follows_location_field(location, country, expected):
|
||||
row = EpochAIGPUCollector().parse_response(
|
||||
epoch_csv(Location=location, Country=country)
|
||||
)[0]
|
||||
|
||||
assert row["metadata"]["location_precision"] == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "expected"),
|
||||
[
|
||||
({"lat": 35.0, "lng": 139.0}, "precise"),
|
||||
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
|
||||
({"lat": 35.0}, "unknown"),
|
||||
({"lng": 139.0}, "unknown"),
|
||||
({"lat": "invalid", "lng": 139.0}, "unknown"),
|
||||
],
|
||||
)
|
||||
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
assert assess_epoch_location(location).precision == expected
|
||||
|
||||
|
||||
def test_epoch_anonymous_rows_are_not_geocoded():
|
||||
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
|
||||
assert row["latitude"] is None and row["longitude"] is None
|
||||
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
|
||||
)
|
||||
def test_epoch_invalid_payload_fails_closed(payload):
|
||||
with pytest.raises(ValueError):
|
||||
EpochAIGPUCollector().parse_response(payload)
|
||||
|
||||
|
||||
def test_epoch_duplicate_identity_rejected():
|
||||
payload = epoch_csv()
|
||||
with pytest.raises(ValueError, match="identity collision"):
|
||||
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
|
||||
|
||||
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
|
||||
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
|
||||
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
assert len(rows) == 2
|
||||
assert rows[0]["source_id"] != rows[1]["source_id"]
|
||||
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
|
||||
|
||||
|
||||
def test_top500_units_and_stable_identity():
|
||||
collector = TOP500Collector()
|
||||
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
|
||||
first = collector.parse_response(
|
||||
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
|
||||
)[0]
|
||||
second = collector.parse_response(
|
||||
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
|
||||
)[0]
|
||||
assert first["source_id"] == second["source_id"] == "top500_system_123"
|
||||
assert float(first["value"]) == 1.5e15
|
||||
assert first["metadata"]["rpeak_flops"] == 2e15
|
||||
assert first["reference_date"] == "2025-11-01"
|
||||
assert second["reference_date"] == "2026-06-01"
|
||||
with pytest.raises(ValueError, match="date missing"):
|
||||
collector.parse_response(html.format(rank=1))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_large_source_loss_does_not_create_snapshot():
|
||||
db = AsyncMock()
|
||||
db.scalar.return_value = 500
|
||||
db.add = MagicMock()
|
||||
with pytest.raises(ValueError, match="40%"):
|
||||
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
def test_history_missing_years_and_denominators():
|
||||
global_data = compute_history(["CN", "US"])
|
||||
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
|
||||
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
|
||||
bilateral = compute_history(["CN", "US"], "bilateral_share")
|
||||
assert sum(bilateral["compute"][3]["values"].values()) == 100
|
||||
assert bilateral["denominator"] == "US+CN"
|
||||
absolute = compute_history(["US", "CN"], "absolute")
|
||||
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
|
||||
assert global_data["innovation"][-2]["value"] == 17.95
|
||||
assert global_data["innovation"][-1]["value"] is None
|
||||
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_snapshot_query_uses_observation_cutoff_not_location():
|
||||
query = await sample_query(AsyncMock(), ["CN"], 2023)
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "2024-01-01" in sql
|
||||
assert "row_number() OVER (PARTITION BY" in sql
|
||||
assert "compute_contract" in sql and "中国" in sql
|
||||
assert "latitude" not in sql and "longitude" not in sql
|
||||
|
||||
|
||||
def test_country_input_is_bounded_and_deduplicated():
|
||||
assert selected_countries("CN,CN,US") == ["CN", "US"]
|
||||
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
|
||||
with pytest.raises(HTTPException):
|
||||
selected_countries("invalid")
|
||||
with pytest.raises(HTTPException, match="overlapping"):
|
||||
selected_countries("EUROPE,DE")
|
||||
|
||||
|
||||
def test_catalog_supports_global_participants_without_inventing_history():
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
COMPARISON_GROUPS,
|
||||
COUNTRIES,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
|
||||
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
|
||||
{entry["code"] for entry in CATALOG}
|
||||
)
|
||||
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
|
||||
owners = source_country_names(["CN"])
|
||||
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
|
||||
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
|
||||
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
assert [entry["kind"] for entry in CATALOG[:5]] == [
|
||||
"country",
|
||||
"country",
|
||||
"region",
|
||||
"region",
|
||||
"region",
|
||||
]
|
||||
group_members = [
|
||||
member
|
||||
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
]
|
||||
assert len(group_members) == len(set(group_members))
|
||||
assert set(group_members) == set(COUNTRIES)
|
||||
data = compute_history(["JP", "KR"])
|
||||
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
|
||||
assert all(row["values"]["KR"] is None for row in data["compute"])
|
||||
assert all(row["value"] is None for row in data["innovation"])
|
||||
assert len(data["catalog"]) > 30
|
||||
|
||||
|
||||
def test_region_history_requires_complete_same_year_member_coverage():
|
||||
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
|
||||
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
|
||||
assert row_2023["values"]["CN"] == 31
|
||||
assert row_2023["values"]["US"] == 41
|
||||
assert row_2023["values"]["EUROPE"] is None
|
||||
assert row_2023["values"]["APAC"] is None
|
||||
assert row_2023["values"]["OTHER"] is None
|
||||
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
|
||||
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
|
||||
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
|
||||
|
||||
def test_anonymous_records_never_enter_the_location_repair_queue():
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
record = CollectedData(
|
||||
source="epoch_ai_gpu",
|
||||
source_id="anonymous",
|
||||
name="Anonymous",
|
||||
extra_data={"compute_contract": "v1", "anonymous": True},
|
||||
)
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert result["features"] == []
|
||||
assert result["unresolved"] == []
|
||||
assert result["withheld_count"] == 1
|
||||
|
||||
|
||||
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
|
||||
absolute = compute_history(["CN"], "absolute")
|
||||
observed = {row["year"]: row for row in absolute["compute"]}
|
||||
assert observed[2022]["values"]["CN"] == 302
|
||||
assert observed[2022]["evidence_id"] == "caict-2023"
|
||||
assert observed[2023]["evidence_id"] == "caict-2024"
|
||||
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
|
||||
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
|
||||
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
|
||||
for row in absolute["compute"]:
|
||||
if row["evidence_id"]:
|
||||
report = absolute["evidence"][row["evidence_id"]]
|
||||
assert report["url"].startswith("https://www.caict.ac.cn/")
|
||||
18
backend/tests/test_compute_verified_sites.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
|
||||
|
||||
def test_registry_requires_exact_facility_identity_not_operator_name():
|
||||
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
|
||||
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
|
||||
|
||||
|
||||
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
|
||||
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
|
||||
assert site["city"] == "Yokohama"
|
||||
assert site["precision"] == "site"
|
||||
assert site["coordinate_basis"] == "official_access_map_center"
|
||||
assert site["sources"]
|
||||
site["latitude"] = 0
|
||||
assert get_verified_compute_site(
|
||||
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
)["latitude"] != 0
|
||||
@@ -1,4 +1,5 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
|
||||
|
||||
|
||||
def write_geojson(path, name="Test"):
|
||||
names = [name]
|
||||
if name == "earth_admin0_boundaries":
|
||||
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
|
||||
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {"name": name},
|
||||
"properties": {
|
||||
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
|
||||
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
|
||||
},
|
||||
}
|
||||
for item in names
|
||||
],
|
||||
}
|
||||
),
|
||||
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
config = repo / "config/earth-boundary-sources.local.json"
|
||||
example = repo / "config/earth-boundary-sources.example.json"
|
||||
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
|
||||
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
geometry.parent.mkdir(parents=True, exist_ok=True)
|
||||
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
geometry.write_bytes(source_geometry.read_bytes())
|
||||
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
|
||||
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
|
||||
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
|
||||
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
|
||||
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
|
||||
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
|
||||
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
|
||||
return {
|
||||
"repo": repo,
|
||||
"config": config,
|
||||
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
}
|
||||
|
||||
|
||||
def write_fallback(path, policy_path, geometry_path):
|
||||
policy = earth_boundaries._read_json(policy_path)
|
||||
geometry = earth_boundaries._read_json(geometry_path)
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(policy),
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
|
||||
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
}
|
||||
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
|
||||
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "legacy-geojson"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
assert status["provider"] == "missing"
|
||||
assert status["fallback_available"] is False
|
||||
|
||||
|
||||
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
|
||||
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
|
||||
}
|
||||
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "pmtiles-mvt"
|
||||
assert status["high_precision_ready"] is True
|
||||
assert status["policy_publishable"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
|
||||
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
|
||||
assert status["high_precision_ready"] is True
|
||||
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
|
||||
assert paths["manifest"].exists()
|
||||
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
|
||||
assert manifest["povPolicy"]["applied"] is True
|
||||
assert manifest["povPolicy"]["coverage"] == "partial"
|
||||
assert manifest["povPolicy"]["publicationReady"] is False
|
||||
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
|
||||
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
|
||||
by_rule = {
|
||||
feature["properties"].get("PLANET_POLICY_RULE"): feature
|
||||
for feature in hover["features"]
|
||||
}
|
||||
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert all(
|
||||
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
|
||||
for feature in hover["features"]
|
||||
)
|
||||
assert any(phase == "download" for _progress, phase, _message in progress_events)
|
||||
|
||||
|
||||
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
|
||||
assert removed.isdisjoint(DEFAULT_DATASOURCES)
|
||||
for source in removed:
|
||||
assert collector_registry.get(source) is None
|
||||
|
||||
|
||||
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
matches = [
|
||||
feature for feature in payload["features"]
|
||||
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
|
||||
]
|
||||
|
||||
assert len(matches) == 1
|
||||
jeju = matches[0]
|
||||
assert jeju["properties"]["ISO_A3"] == "KOR"
|
||||
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
|
||||
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
|
||||
ring = jeju["geometry"]["coordinates"][0]
|
||||
assert len(ring) == 61
|
||||
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
|
||||
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)
|
||||
|
||||
@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
assert vessel_update["refresh_strategy"] == "reload"
|
||||
|
||||
|
||||
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
|
||||
from app.services.earth_db_change_listener import PendingEarthDbChange
|
||||
|
||||
for source in ("aisstream_vessels", "barentswatch_vessels"):
|
||||
update = build_earth_update_from_db_payload({
|
||||
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
|
||||
})
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
|
||||
pending.add({"operation": "UPDATE"})
|
||||
pending.add({"operation": "DELETE"})
|
||||
assert pending.refresh_strategy == "reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
|
||||
41
backend/tests/test_earth_geography_policy.py
Normal file
@@ -0,0 +1,41 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.earth_geography_policy import display_country, display_iso_a3
|
||||
|
||||
|
||||
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
|
||||
from app.core import earth_geography_policy as policy
|
||||
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
|
||||
try:
|
||||
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
|
||||
policy.display_country("China")
|
||||
finally:
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
|
||||
|
||||
def test_display_country_groups_china_regions_and_kosovo() -> None:
|
||||
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
|
||||
assert display_country(source) == "中国"
|
||||
assert display_iso_a3(source) == "CHN"
|
||||
for source in ("Kosovo", "XK", "SRB"):
|
||||
assert display_country(source) == "塞尔维亚"
|
||||
assert display_iso_a3(source) == "SRB"
|
||||
|
||||
|
||||
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
|
||||
assert display_country("Japan") == "日本"
|
||||
assert display_iso_a3("Japan") == "JPN"
|
||||
assert display_country("unexpected non-country") is None
|
||||
|
||||
|
||||
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
|
||||
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy = json.loads(path.read_text(encoding="utf-8"))
|
||||
for rule in policy["rules"]:
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))
|
||||
@@ -24,6 +24,7 @@ from app.services.earth_news import (
|
||||
from app.services.earth_news_queue import NewsTargetLocationMessage
|
||||
from app.services.earth_news_worker import process_target_location_message
|
||||
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
|
||||
from app.services.earth_news_store import _diversify_parsed_news_items_by_region
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
@@ -172,6 +173,68 @@ def test_diversify_news_items_prefers_display_ready_content_across_sources():
|
||||
assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"]
|
||||
|
||||
|
||||
def test_cruise_news_diversity_keeps_regions_from_being_starved():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(40)],
|
||||
make_item("europe", 1),
|
||||
make_item("middle-east-africa", 1),
|
||||
make_item("americas", 1),
|
||||
make_item("global", 1),
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=8)
|
||||
regions = [item.feed_region for item in result]
|
||||
|
||||
assert "europe" in regions
|
||||
assert "middle-east-africa" in regions
|
||||
assert "americas" in regions
|
||||
assert regions.count("asia-pacific") < len(regions)
|
||||
|
||||
|
||||
def test_global_news_diversity_uses_same_region_balance():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:global:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/global/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(24)],
|
||||
*[make_item("europe", index) for index in range(2)],
|
||||
*[make_item("middle-east-africa", index) for index in range(2)],
|
||||
*[make_item("americas", index) for index in range(2)],
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=6)
|
||||
regions = {item.feed_region for item in result}
|
||||
|
||||
assert {"europe", "middle-east-africa", "americas"}.issubset(regions)
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
@@ -321,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
|
||||
assert items[0].content_language == "zh-CN"
|
||||
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == ""
|
||||
|
||||
items[0].localizations["en-US"] = {
|
||||
"title": "Chinese e-commerce platform reports quarterly growth",
|
||||
"summary": "The platform said cross-border orders rose year over year.",
|
||||
}
|
||||
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
|
||||
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
|
||||
|
||||
|
||||
def test_default_news_sources_include_business_and_ecommerce_sources():
|
||||
|
||||
86
backend/tests/test_earth_news_encoding.py
Normal file
@@ -0,0 +1,86 @@
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
ParsedNewsItem,
|
||||
_fetch_single_feed_url,
|
||||
_serialize_item,
|
||||
)
|
||||
|
||||
|
||||
async def fetch_feed(
|
||||
content: bytes, content_type: str = "text/xml"
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
|
||||
source = NewsFeedSource(
|
||||
id="ebrun",
|
||||
name="亿邦动力",
|
||||
region="asia-pacific",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
homepage_url="https://example.com/",
|
||||
)
|
||||
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
|
||||
transport = httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200, content=content, headers={"content-type": content_type}
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
return await _fetch_single_feed_url(client, source, feed)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("encoding", "declaration", "content_type"),
|
||||
[
|
||||
("gbk", 'encoding="GBK"', "text/xml"),
|
||||
("gb2312", "encoding='GB2312'", "application/rss+xml"),
|
||||
("gb18030", 'encoding="GB18030"', "text/xml"),
|
||||
("gbk", "", "text/xml; charset=gbk"),
|
||||
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
|
||||
("utf-8", "", "text/xml"),
|
||||
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
|
||||
("utf-16", 'encoding="UTF-16"', "text/xml"),
|
||||
("utf-32", 'encoding="UTF-32"', "text/xml"),
|
||||
],
|
||||
)
|
||||
async def test_feed_encoding_preserves_chinese_in_display(
|
||||
encoding: str, declaration: str, content_type: str
|
||||
):
|
||||
title = "618电商战报:零售额增长266%"
|
||||
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
|
||||
xml = f'''<?xml version="1.0" {declaration}?>
|
||||
<rss version="2.0"><channel><item>
|
||||
<title><![CDATA[{title}]]></title>
|
||||
<description><![CDATA[<p>{summary}</p>]]></description>
|
||||
<link>https://example.com/story</link>
|
||||
</item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
|
||||
|
||||
assert error is None
|
||||
assert health["ok"] is True
|
||||
assert len(items) == 1
|
||||
assert items[0].title == title
|
||||
assert items[0].summary == summary
|
||||
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
|
||||
assert payload["display_title"] == title
|
||||
assert payload["display_summary"] == summary
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
|
||||
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
|
||||
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
|
||||
<rss><channel><item><title>电商新闻</title>
|
||||
<link>https://example.com/story</link></item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode("gbk"))
|
||||
|
||||
assert items == []
|
||||
assert error
|
||||
assert health["ok"] is False
|
||||
assert health["status"] == "format_error"
|
||||
219
backend/tests/test_llm_provider_catalog.py
Normal file
@@ -0,0 +1,219 @@
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services import llm_provider_catalog as catalog
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
def mock_http(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"provider,path,auth",
|
||||
[
|
||||
("minimax", "/anthropic/v1/models", "x-api-key"),
|
||||
("anthropic", "/v1/models", "x-api-key"),
|
||||
("openai", "/v1/models", "authorization"),
|
||||
("deepseek", "/v1/models", "authorization"),
|
||||
("alibaba", "/api/v1/models", "authorization"),
|
||||
("moonshotai", "/v1/models", "authorization"),
|
||||
("openrouter", "/api/v1/models", "authorization"),
|
||||
("opencode-go", "/zen/go/v1/models", "authorization"),
|
||||
("ollama", "/api/tags", "authorization"),
|
||||
],
|
||||
)
|
||||
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
|
||||
preset = catalog.get_fallback_llm_provider_preset(provider)
|
||||
|
||||
def handle(request):
|
||||
assert request.url.path == path
|
||||
assert request.url.host == httpx.URL(preset["base_url"]).host
|
||||
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
|
||||
if provider == "alibaba":
|
||||
assert request.url.params["capabilities"] == "TG"
|
||||
return httpx.Response(
|
||||
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
|
||||
)
|
||||
if provider == "ollama":
|
||||
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
|
||||
return httpx.Response(200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
|
||||
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
|
||||
assert result["base_url"] == preset["base_url"]
|
||||
assert "test-key" not in str(result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,base,api,path",
|
||||
[
|
||||
(
|
||||
"minimax",
|
||||
"https://api.minimax.io/anthropic/v1/",
|
||||
"anthropic-messages",
|
||||
"/anthropic/v1/models",
|
||||
),
|
||||
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
|
||||
(
|
||||
"alibaba",
|
||||
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
(
|
||||
"alibaba",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
|
||||
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
|
||||
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
|
||||
],
|
||||
)
|
||||
def test_urls_preserve_region_and_gateway(provider, base, api, path):
|
||||
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
|
||||
assert url.host == httpx.URL(base).host
|
||||
assert url.path == path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_pagination_and_release_order(monkeypatch):
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request.url.params.get("after_id"))
|
||||
if len(seen) == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
|
||||
"has_more": True,
|
||||
"last_id": "old",
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
|
||||
assert seen == [None, "old"]
|
||||
assert result["models"] == ["new", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashscope_pagination(monkeypatch):
|
||||
def handle(request):
|
||||
page = int(request.url.params["page_no"])
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"output": {
|
||||
"total": 2,
|
||||
"models": [
|
||||
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
|
||||
assert result["models"] == ["page-2", "page-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{},
|
||||
{"data": []},
|
||||
{"data": [None]},
|
||||
{"data": [{}]},
|
||||
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
|
||||
],
|
||||
)
|
||||
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
with pytest.raises(discovery.LLMProviderCatalogError):
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
|
||||
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
|
||||
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
|
||||
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
|
||||
assert len(result["models"]) == 140
|
||||
assert result["models"][0] == "model-139"
|
||||
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_ollama_catalog_is_valid(monkeypatch):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
|
||||
result = await catalog.refresh_llm_provider_preset("ollama")
|
||||
assert result["models"] == []
|
||||
assert result["model"] == ""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opencode_documented_protocols(monkeypatch):
|
||||
models = [
|
||||
"minimax-m3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"grok-4.6",
|
||||
"muse-spark-1.3-contributor",
|
||||
"kimi-k3",
|
||||
]
|
||||
mock_http(
|
||||
monkeypatch,
|
||||
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
|
||||
)
|
||||
result = await catalog.refresh_llm_provider_preset("opencode-go")
|
||||
assert list(result["model_provider_apis"].values()) == [
|
||||
"anthropic-messages",
|
||||
"anthropic-messages",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-completions",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_transient_error_only(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def handle(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 2
|
||||
calls.clear()
|
||||
|
||||
def unauthorized(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(401, text="secret-upstream-text")
|
||||
|
||||
mock_http(monkeypatch, unauthorized)
|
||||
with pytest.raises(httpx.HTTPStatusError) as error:
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 1
|
||||
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)
|
||||
165
backend/tests/test_llm_provider_settings.py
Normal file
@@ -0,0 +1,165 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.v1 import settings as api
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings_db(monkeypatch, tmp_path):
|
||||
# Real SQL reads, commits and reloads; do not mock the settings helpers.
|
||||
engine = create_engine("sqlite://")
|
||||
SystemSetting.__table__.create(engine)
|
||||
session = Session(engine)
|
||||
db = SimpleNamespace(
|
||||
execute=AsyncMock(side_effect=session.execute),
|
||||
add=session.add,
|
||||
commit=AsyncMock(side_effect=session.commit),
|
||||
refresh=AsyncMock(side_effect=session.refresh),
|
||||
)
|
||||
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
|
||||
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
|
||||
yield db
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def use_upstream(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_lists_all_builtin_presets(settings_db):
|
||||
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
|
||||
assert len(result["data"]) == 9
|
||||
assert all(row["models"] for row in result["data"])
|
||||
assert "MiniMax-M3" in result["data"][0]["models"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
|
||||
runtime = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"service_token": "private-service-token",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"base_url": "https://saved.test/anthropic",
|
||||
"model": "old",
|
||||
"api_key": "saved-secret",
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
|
||||
before = await api.get_setting_payload(settings_db, "external_integrations")
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request)
|
||||
assert str(request.url) == "https://draft.test/anthropic/v1/models"
|
||||
assert request.headers["x-api-key"] == "draft-secret"
|
||||
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
|
||||
|
||||
use_upstream(monkeypatch, handle)
|
||||
draft = api.AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
base_url="https://draft.test/anthropic",
|
||||
model="unsaved-model",
|
||||
api_key="draft-secret",
|
||||
service_token="unsaved-service-token",
|
||||
)
|
||||
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
|
||||
result = await api.get_ai_provider_presets(None, settings_db)
|
||||
preset = next(row for row in result["data"] if row["provider"] == "minimax")
|
||||
assert len(seen) == 1
|
||||
assert preset["models"] == ["MiniMax-M3", "old"]
|
||||
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
|
||||
assert await api.get_setting_payload(settings_db, "external_integrations") == before
|
||||
assert "secret" not in str(result)
|
||||
assert "private-service-token" not in str(result)
|
||||
assert draft.model == "unsaved-model"
|
||||
stored = await settings_db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
|
||||
)
|
||||
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
|
||||
settings_db, monkeypatch
|
||||
):
|
||||
await api.save_setting_payload(
|
||||
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
|
||||
)
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
|
||||
commits = settings_db.commit.await_count
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
|
||||
assert error.value.status_code == 502
|
||||
assert "private-upstream-secret" not in error.value.detail
|
||||
assert settings_db.commit.await_count == commits
|
||||
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
|
||||
"models"
|
||||
] == ["saved-model"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
|
||||
def fail_request(request):
|
||||
pytest.fail("must not query a private catalog without credentials")
|
||||
|
||||
use_upstream(monkeypatch, fail_request)
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("minimax", None, settings_db)
|
||||
assert error.value.status_code == 400
|
||||
assert "API Key" in error.value.detail
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
|
||||
)
|
||||
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
|
||||
)
|
||||
assert result["success"] is False
|
||||
assert result["connected"] is False
|
||||
assert "private-upstream-secret" not in result["message"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
|
||||
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
|
||||
use_upstream(
|
||||
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
|
||||
)
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "model": "future-model", "api_key": "test-key"}, 5
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["url"].endswith("/anthropic/v1/models")
|
||||
141
backend/tests/test_provider_protocols.py
Normal file
@@ -0,0 +1,141 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
with patch(
|
||||
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
|
||||
return_value={},
|
||||
):
|
||||
from aiprovider.provider_service import ProviderService
|
||||
from aiprovider.schemas import SituationalAnalysisRequest
|
||||
from app.api.v1.settings import _runtime_config_from_ai_payload
|
||||
|
||||
|
||||
def test_custom_openai_gateway_keeps_its_configured_protocol():
|
||||
config = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "openai",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"base_url": "https://gateway.test/v1",
|
||||
"provider_api": "openai-completions",
|
||||
"model": "gpt-6-astra",
|
||||
},
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService({**config, "api_key": "test-key"})
|
||||
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"model,path",
|
||||
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
|
||||
)
|
||||
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
|
||||
llm = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "opencode-go",
|
||||
"providers": {
|
||||
"opencode-go": {
|
||||
"model": model,
|
||||
"api_key": "test-key",
|
||||
"model_provider_apis": {model: "openai-completions"},
|
||||
}
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService(llm)
|
||||
seen = []
|
||||
|
||||
async def post(**request):
|
||||
seen.append(request)
|
||||
assert request["path"] == path
|
||||
assert request["request_body"]["model"] == model
|
||||
if path == "/responses":
|
||||
assert request["request_body"]["store"] is False
|
||||
assert "max_tokens" not in request["request_body"]
|
||||
return {
|
||||
"output": [
|
||||
{"type": "reasoning", "summary": [{"text": "reason"}]},
|
||||
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
|
||||
]
|
||||
}
|
||||
return {"content": [{"type": "text", "text": "OK"}]}
|
||||
|
||||
monkeypatch.setattr(service, "_post", post)
|
||||
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
|
||||
assert len(seen) == 1
|
||||
assert result.content == "OK"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
seen.append(str(request.url))
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": base,
|
||||
"api_key": "test-key",
|
||||
"model": "MiniMax-M3",
|
||||
}
|
||||
)
|
||||
result = await service.analyze(
|
||||
SituationalAnalysisRequest(title="test", objective="reply OK")
|
||||
)
|
||||
assert result.content == "OK"
|
||||
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
import json
|
||||
|
||||
seen.append(request)
|
||||
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
|
||||
assert request.headers["user-agent"] == "Planet/1.0"
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"api_key": "test-key",
|
||||
"model": "minimax-m3",
|
||||
}
|
||||
)
|
||||
await service.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="test",
|
||||
objective="reply OK",
|
||||
thinking={"type": "enabled"},
|
||||
context={"session_id": session_id},
|
||||
)
|
||||
)
|
||||
session_headers = [request.headers["x-opencode-session"] for request in seen]
|
||||
assert session_headers[0] == session_headers[1]
|
||||
assert session_headers[0] != session_headers[2]
|
||||
160
backend/tests/test_tv_catalog.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import normalize_tv_settings
|
||||
|
||||
|
||||
class CatalogSession:
|
||||
"""Execute the real catalog queries against an isolated SQLite database."""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
async def execute(self, query):
|
||||
return self.session.execute(query)
|
||||
|
||||
async def scalar(self, query):
|
||||
return self.session.scalar(query)
|
||||
|
||||
async def scalars(self, query):
|
||||
return self.session.scalars(query)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def catalog_db():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def register_functions(connection, _record):
|
||||
connection.create_function(
|
||||
"concat_ws",
|
||||
-1,
|
||||
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
|
||||
)
|
||||
|
||||
Base.metadata.create_all(
|
||||
engine,
|
||||
tables=[
|
||||
CollectionTask.__table__,
|
||||
DataSnapshot.__table__,
|
||||
CollectedData.__table__,
|
||||
SystemSetting.__table__,
|
||||
],
|
||||
)
|
||||
with Session(engine) as session:
|
||||
for index in range(135):
|
||||
session.add(
|
||||
CollectedData(
|
||||
source="news_live_streams",
|
||||
source_id=f"channel-{index:03}",
|
||||
data_type="news_live_stream",
|
||||
name=f"Channel {index:03}",
|
||||
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
extra_data={
|
||||
"stream_url": "https://example.invalid/live.m3u8",
|
||||
"region": "Canada",
|
||||
},
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
yield CatalogSession(session)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
|
||||
first = await get_tv_catalog_page(catalog_db, limit=50)
|
||||
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
|
||||
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
|
||||
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
|
||||
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
|
||||
assert len(set(ids)) == first["source_count"] == 145
|
||||
assert ids[-1] == "channel-134"
|
||||
assert third["next_offset"] is None and not third["has_more"]
|
||||
beyond = await get_tv_catalog_page(catalog_db, offset=200)
|
||||
assert beyond["sources"] == [] and not beyond["has_more"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
assert payload["total"] == 1 and payload["source_count"] == 145
|
||||
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
|
||||
assert payload["selected_source"]["id"] == "channel-134"
|
||||
assert "channel-134" not in [source["id"] for source in payload["sources"]]
|
||||
assert payload["default_source_id"] == "aljazeera-mubasher"
|
||||
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
|
||||
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
|
||||
for name, values in [
|
||||
("Disabled", {"extra_data": {"is_enabled": False}}),
|
||||
("Historical", {"is_current": False}),
|
||||
("Invalid", {"is_valid": 0}),
|
||||
("Deleted", {"deleted_at": datetime.now(UTC)}),
|
||||
("Replacement", {"source_id": "channel-134"}),
|
||||
]:
|
||||
record = dict(
|
||||
source="news_live_streams",
|
||||
source_id=name,
|
||||
name=name,
|
||||
data_type="news_live_stream",
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
)
|
||||
catalog_db.session.add(CollectedData(**{**record, **values}))
|
||||
catalog_db.session.flush()
|
||||
payload = await get_tv_catalog_page(catalog_db, q="replacement")
|
||||
assert payload["source_count"] == 145
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
|
||||
|
||||
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
|
||||
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
|
||||
)
|
||||
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
|
||||
monkeypatch, config, expected
|
||||
):
|
||||
collector = NewsLiveStreamsCollector()
|
||||
channels = [
|
||||
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
|
||||
]
|
||||
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
|
||||
streams = [
|
||||
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
|
||||
for channel in channels
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
|
||||
)
|
||||
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
|
||||
assert len(records) == expected
|
||||
assert all(record["source_id"] != "sport" for record in records)
|
||||
37
backend/tests/test_tv_playback.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from app.services.tv_streams import (
|
||||
CCTV4_HOMEPAGE,
|
||||
CCTV4_STREAM,
|
||||
UNSUPPORTED_CCTV4_STREAM,
|
||||
normalize_tv_settings,
|
||||
normalize_tv_source,
|
||||
)
|
||||
|
||||
|
||||
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
|
||||
settings = normalize_tv_settings(None)
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert cctv["source_type"] == "hls"
|
||||
assert cctv["stream_url"] == CCTV4_STREAM
|
||||
assert "美洲版" in cctv["name"]
|
||||
assert cctv["use_proxy"] is False
|
||||
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
|
||||
|
||||
|
||||
def test_saved_legacy_default_is_corrected_without_mutating_input():
|
||||
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
|
||||
source = normalize_tv_source(saved)
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["stream_url"] == CCTV4_STREAM
|
||||
assert source["use_proxy"] is False
|
||||
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
|
||||
|
||||
|
||||
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
|
||||
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
|
||||
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
|
||||
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert source["stream_url"] == custom["stream_url"]
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["use_proxy"] is True
|
||||
assert settings["auto_fallback"] is False
|
||||
@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_epoch_country_level_coordinates_are_unresolved():
|
||||
record = _build_record(
|
||||
record_id=33,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Meta 100k",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=39.381266,
|
||||
longitude=-97.922211,
|
||||
metadata={
|
||||
"location": "United States of America",
|
||||
"raw_country": "United States of America",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
assert payload["features"] == []
|
||||
assert len(payload["unresolved"]) == 1
|
||||
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
|
||||
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
|
||||
|
||||
|
||||
def test_epoch_city_coordinates_are_not_labeled_precise():
|
||||
record = _build_record(
|
||||
record_id=34,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Seoul Cluster",
|
||||
country="South Korea",
|
||||
city="",
|
||||
latitude=37.5665,
|
||||
longitude=126.978,
|
||||
metadata={"location": "Seoul", "raw_country": "South Korea"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.is_resolved
|
||||
assert result.location.location_precision == "city"
|
||||
assert result.location.needs_confirmation is True
|
||||
assert result.location.is_estimated is True
|
||||
assert result.location.city is None
|
||||
assert result.location.matched_location_name == "Seoul"
|
||||
assert "Epoch Location: Seoul" in result.location.location_source_note
|
||||
|
||||
|
||||
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
|
||||
record = _build_record(
|
||||
record_id=37,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Petrobras Fênix",
|
||||
country="Brazil",
|
||||
city="",
|
||||
latitude=-22.909534,
|
||||
longitude=-43.209934,
|
||||
metadata={
|
||||
"location": "Rio de Janeiro, Brazil",
|
||||
"raw_country": "Brazil",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
feature = payload["features"][0]
|
||||
assert feature["properties"]["location_precision"] == "city"
|
||||
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
|
||||
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
|
||||
"location_source_note"
|
||||
]
|
||||
|
||||
|
||||
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-35": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-35",
|
||||
"name": "Confirmed Site",
|
||||
"latitude": 35.0,
|
||||
"longitude": 139.0,
|
||||
"precision": "site",
|
||||
"confidence": 1.0,
|
||||
"location_source": "official_facility_registry",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=35,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Japan Cluster",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={
|
||||
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
|
||||
"raw_country": "Japan",
|
||||
},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "stored_compute_center_location"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-36": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-36",
|
||||
"name": "Legacy Country Point",
|
||||
"latitude": 36.386493,
|
||||
"longitude": 138.59223,
|
||||
"precision": "precise",
|
||||
"confidence": 1.0,
|
||||
"location_source": "source_coordinates",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=36,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Legacy Japan Cluster",
|
||||
country="Japan",
|
||||
city="",
|
||||
latitude=36.386493,
|
||||
longitude=138.59223,
|
||||
metadata={"location": "Japan", "raw_country": "Japan"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location is None
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_verified_sites_replace_epoch_country_center_coordinates():
|
||||
specifications = [
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_0_gyoukou",
|
||||
"Gyoukou 2.0",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_2_gyoukou",
|
||||
"Gyoukou 2.2",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_argonne_nl_sophia__formerly_theta_",
|
||||
"Sophia",
|
||||
"United States of America",
|
||||
39.381266,
|
||||
-97.922211,
|
||||
[-87.984498648496, 41.718109383287235],
|
||||
),
|
||||
]
|
||||
records = []
|
||||
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
|
||||
specifications, start=40
|
||||
):
|
||||
record = _build_record(
|
||||
record_id=index,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name=name,
|
||||
country=country,
|
||||
city="",
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
metadata={"location": country, "raw_country": country},
|
||||
)
|
||||
record.source_id = source_id
|
||||
records.append(record)
|
||||
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
|
||||
assert payload["unresolved"] == []
|
||||
assert len(payload["features"]) == 3
|
||||
for feature, specification in zip(payload["features"], specifications):
|
||||
assert feature["geometry"]["coordinates"] == specification[-1]
|
||||
assert feature["properties"]["location_precision"] == "site"
|
||||
assert feature["properties"]["location_source"] == "official_facility_registry"
|
||||
assert "https://" in feature["properties"]["location_source_note"]
|
||||
|
||||
|
||||
def test_verified_site_beats_epoch_city_level_source_coordinates():
|
||||
record = _build_record(
|
||||
record_id=44,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Gyoukou 2.2",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
|
||||
)
|
||||
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "official_facility_registry"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (
|
||||
35.380924,
|
||||
139.625008,
|
||||
)
|
||||
|
||||
|
||||
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-32": {
|
||||
@@ -637,7 +873,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -741,7 +981,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import pytest
|
||||
import importlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.websocket.manager import ConnectionManager
|
||||
from app.core.websocket.broadcaster import DataBroadcaster
|
||||
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
|
||||
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
|
||||
broadcaster = DataBroadcaster()
|
||||
async def load_current(keys):
|
||||
assert keys == ["1"]
|
||||
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
|
||||
broadcaster.enqueue_vessel_update(
|
||||
{
|
||||
"source": "aisstream_vessels",
|
||||
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
assert sent[0]["vessels"] == [
|
||||
{
|
||||
"mmsi": 1,
|
||||
"lat": 59.1,
|
||||
"lon": 10.1,
|
||||
"source": "aisstream_vessels",
|
||||
"lat": 60.0,
|
||||
"lon": 11.0,
|
||||
"source": "barentswatch_vessels",
|
||||
"action": "upsert",
|
||||
"created": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
json.dumps(config)
|
||||
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
|
||||
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
|
||||
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
|
||||
manager.unsubscribe(socket, ["vessels"])
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
assert len(socket.sent) == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_removal_does_not_require_coordinates():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
|
||||
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
|
||||
|
||||
|
||||
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
|
||||
websocket_module = importlib.import_module("app.api.v1.websocket")
|
||||
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
|
||||
app = FastAPI()
|
||||
app.include_router(websocket_module.router)
|
||||
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
|
||||
assert socket.receive_json()["type"] == "connection_established"
|
||||
socket.send_json({
|
||||
"type": "subscribe",
|
||||
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
|
||||
})
|
||||
response = socket.receive_json()
|
||||
assert response["type"] == "subscription_confirmed"
|
||||
assert response["data"]["vessels"]["scope"] == "global"
|
||||
assert response["data"]["vessels"]["type"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
|
||||
broadcaster = DataBroadcaster()
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
|
||||
|
||||
async def fail_read(_keys):
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
|
||||
with pytest.raises(RuntimeError, match="database unavailable"):
|
||||
await broadcaster.flush_vessel_updates()
|
||||
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61
|
||||
|
||||
@@ -25,6 +25,25 @@
|
||||
{
|
||||
"id": "china-taiwan-penghu",
|
||||
"name": "Taiwan and Penghu",
|
||||
"sourceIsoA3": ["TWN"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-hong-kong",
|
||||
"name": "Hong Kong",
|
||||
"sourceIsoA3": ["HKG"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-macao",
|
||||
"name": "Macao",
|
||||
"sourceIsoA3": ["MAC"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
@@ -57,6 +76,7 @@
|
||||
{
|
||||
"id": "kosovo",
|
||||
"name": "Kosovo",
|
||||
"sourceIsoA3": ["KOS", "XKS"],
|
||||
"action": "render_as_disputed_with_parent",
|
||||
"parentIsoA3": "SRB",
|
||||
"hoverIsoA3": "SRB",
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
@@ -32,7 +30,7 @@ services:
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: planet_db
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
@@ -45,7 +43,7 @@ services:
|
||||
image: redis:7-alpine
|
||||
container_name: planet_redis
|
||||
ports:
|
||||
- "6379:6379"
|
||||
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
|
||||
@@ -8,6 +8,275 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.75.4] — 2026-09-29
|
||||
|
||||
Released: 2026-09-29
|
||||
|
||||
### Highlights
|
||||
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
|
||||
|
||||
### Improved / Fixed
|
||||
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
|
||||
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
|
||||
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
|
||||
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
|
||||
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.3] — 2026-09-27
|
||||
|
||||
Released: 2026-09-27
|
||||
|
||||
### Highlights
|
||||
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
|
||||
|
||||
### Improved / Fixed
|
||||
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
|
||||
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
|
||||
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
|
||||
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
|
||||
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.2] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
|
||||
|
||||
### Fixed
|
||||
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
|
||||
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
|
||||
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.1] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
|
||||
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
|
||||
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
|
||||
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
|
||||
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.0] — 2026-09-23
|
||||
|
||||
Released: 2026-09-23
|
||||
|
||||
### Highlights
|
||||
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
|
||||
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
|
||||
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
|
||||
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
|
||||
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
|
||||
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.7] — 2026-09-17
|
||||
|
||||
Released: 2026-09-17
|
||||
|
||||
### Highlights
|
||||
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
|
||||
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
|
||||
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
|
||||
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
|
||||
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
|
||||
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.6] — 2026-09-16
|
||||
|
||||
Released: 2026-09-16
|
||||
|
||||
### Highlights
|
||||
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
|
||||
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
|
||||
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
|
||||
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
|
||||
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
|
||||
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.5] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
|
||||
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
|
||||
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
|
||||
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
|
||||
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
|
||||
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.4] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
|
||||
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
|
||||
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
|
||||
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
|
||||
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
|
||||
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.3] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
|
||||
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
|
||||
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
|
||||
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
|
||||
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
|
||||
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.2] — 2026-07-01
|
||||
|
||||
Released: 2026-07-01
|
||||
|
||||
### Highlights
|
||||
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`,删除重复维护的旧 Claude command 入口。
|
||||
- 强化视觉证据规则:截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置,而不是跳过后猜测。
|
||||
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback,同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容,保留最新单一入口和 harness 验证说明。
|
||||
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate,补齐路径解析、访问失败报告和 OCR fallback 边界。
|
||||
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口,并更新文档受众计划中的旧路径引用。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.1] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内,移除旧的全局“选择资产/上传”工具栏。
|
||||
- 新增字段级图片拖拽反馈,拖到对应字段时直接提示将图片复制为 Logo 或标题图。
|
||||
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式,并同步中英文使用手册、快速开始和控制台上下文文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
|
||||
- `FieldGrid` 支持按字段注入自定义输入控件,同时复用统一草稿提交路径。
|
||||
- 品牌上传拖拽态改为低饱和 tactile 配色,上传按钮保持蓝色轻立体样式,容器内上下/右侧留白对齐为 3px。
|
||||
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.0] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口,减少英文界面中文残留。
|
||||
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示,避免英文态裁切或错位。
|
||||
- 将容易遗漏的 i18n 入口和视觉回归加入 harness,让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 Earth runtime i18n 入口,统一高清材质、启动状态、错误提示和图层状态文案来源。
|
||||
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
|
||||
- 更新控制台与 Earth 布局规则,保留 brand 尺寸语义,同时让标题、副标题和通知胶囊按内容完整显示。
|
||||
- 扩展 frontend smoke 与 harness 文档,固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
|
||||
- 更新 Earth 新闻本地化服务与测试,确保英文界面新闻内容不再回退中文 UI 文案。
|
||||
|
||||
---
|
||||
|
||||
## [0.73.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 新增前端统一 i18n 基础设施,让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
|
||||
- 控制台侧边栏偏好面板接入语言与主题切换,并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
|
||||
- 扩展 harness 与 smoke 覆盖,确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `frontend/src/i18n/`,用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
|
||||
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
|
||||
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译,并覆盖动态计数字符串。
|
||||
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership,避免首屏溢出和状态词裁切。
|
||||
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则,记录语言迁移边界、状态指示器布局约束和一屏验证要求。
|
||||
|
||||
---
|
||||
|
||||
## [0.72.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 将 agent 入口收敛到单一 `AGENTS.md`,并让 harness 明确阻止小写入口再次分叉。
|
||||
- 新增完整本地 harness 验证层,覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
|
||||
- 扩展 Earth News 与控制台 smoke,确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本,并将未跟踪 harness 设施纳入发布。
|
||||
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出,改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
|
||||
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
|
||||
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档,明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
|
||||
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.1] — 2026-06-26
|
||||
|
||||
Released: 2026-06-26
|
||||
|
||||
### Highlights
|
||||
- 修复 Earth 欧洲、美洲、中东与非洲等区域新闻被亚太来源和旧来源过滤饿死的问题,滚动条、面板和巡航重新回到同一批区域 payload。
|
||||
- 将当前可见新闻和巡航新闻提升到目标位置/翻译优先队列,避免历史普通 Redis backlog 阻塞用户正在看的新闻精修。
|
||||
- 新增 agent harness 入口、代码地图、验证脚本与双语技术说明,让后续维护能按现有 uv/Bun/Gitea 工作流检查而不替代项目规则。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- EarthFeed 在全局和巡航队列中按区域轮转候选新闻,保留区域视图的“当前区域 + global”规则,并补充回归测试。
|
||||
- `news.js` 按区域、类型、来源和数量隔离并发刷新请求,丢弃旧区域响应;跨区域时不再复用旧来源筛选。
|
||||
- 新闻展示在中文本地化未完成时回退原始标题和摘要,避免出现有内容却显示“新闻汉化中”的卡片。
|
||||
- 新闻目标位置 worker 新增优先 stream、pending reclaim、任务超时和并发处理;无效消息会确认并删除,减少队列堆积。
|
||||
- 补充 Earth 新闻源、Earth 前端结构、harness 和版本历史文档,并移除控制台 auth store 的调试日志。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.0] — 2026-06-11
|
||||
|
||||
Released: 2026-06-11
|
||||
|
||||
335
docs/HARNESS.md
Normal file
@@ -0,0 +1,335 @@
|
||||
# Agent Harness
|
||||
|
||||
This harness improves discoverability, repeatability, and agent safety for the
|
||||
existing Planet project. It does not replace current project rules, scripts, CI,
|
||||
or release workflows.
|
||||
|
||||
## Authority And Conflicts
|
||||
|
||||
Existing project rules are authoritative:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. Current implementation docs under `docs/technical/`
|
||||
4. Existing scripts, especially `planet.sh`
|
||||
5. Existing Gitea workflow files under `.gitea/workflows/`
|
||||
|
||||
When harness guidance conflicts with any of the above, keep the existing rule,
|
||||
do not overwrite the existing workflow, and add a compatibility note here or in
|
||||
`docs/harness-audit.md`.
|
||||
|
||||
For frontend or documentation audits, also read the Rules Coverage Evidence
|
||||
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
|
||||
static checks, Playwright smoke coverage, and remaining manual review areas, so
|
||||
an agent can distinguish a proved harness pass from a rule that still needs
|
||||
human-quality inspection.
|
||||
|
||||
When the user describes work with product words rather than module names, use
|
||||
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
|
||||
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
|
||||
`地球`, `模型供应商`, and `发版` to the required rule modules.
|
||||
|
||||
## Starting Work
|
||||
|
||||
Recommended startup flow:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Then read only the relevant implementation docs:
|
||||
|
||||
- Backend/API/data work: `docs/technical/zh/backend-*.md` and matching English
|
||||
docs when public docs are affected.
|
||||
- Frontend/admin work: `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
- Earth work: `docs/technical/zh/earth-frontend-context.md`,
|
||||
`docs/technical/zh/earth-render-layer-order.md`, and style docs when visual
|
||||
semantics change.
|
||||
- Operations work: `docs/technical/zh/ops-runbook.md` and
|
||||
`docs/technical/zh/ops-planet-sh-startup.md`.
|
||||
- AI Provider work: `docs/technical/zh/agents-aiprovider.md`.
|
||||
- Documentation work: `docs/documentation-coverage-rules.md`.
|
||||
|
||||
Use focused inspection commands before broad reads:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Purpose | Command |
|
||||
| --- | --- |
|
||||
| First setup | `./planet.sh init` |
|
||||
| Start local stack | `./planet.sh start` |
|
||||
| Start with LAN access | `./planet.sh start --allow-lan` |
|
||||
| Restart all services | `./planet.sh restart` |
|
||||
| Restart one area | `./planet.sh restart -b`, `-f`, `-a`, or `-d` |
|
||||
| Health check | `./planet.sh health` |
|
||||
| Logs | `./planet.sh log`, `./planet.sh log -b`, `-f`, `-a`, or `-m` |
|
||||
| Create local user | `./planet.sh createuser` |
|
||||
| Destructive local reset | `./planet.sh destroy` |
|
||||
| Backend smoke tests | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` |
|
||||
| Mock AIS WebSocket | `bun run mock:ais-ws` |
|
||||
|
||||
## Harness Commands
|
||||
|
||||
Windows desktop launcher tests run separately on Windows with
|
||||
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
|
||||
They use temporary desktop folders and mocked restart commands; no UAC prompt
|
||||
or real service restart is triggered. The WSL bridge regression suite,
|
||||
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
|
||||
disposable command substitutes. For a real entry-point check without restarting,
|
||||
run the repository `planet.cmd --install-only`, then the generated desktop
|
||||
launcher with the same flag. Installation and mocked checks do not prove a live
|
||||
elevated restart; report that separately when it has not been exercised.
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
Docker image smoke builds are expensive and are off by default:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Frontend Playwright smoke runs by default in full validation after the frontend
|
||||
build. It starts a local Vite preview and checks the `/` to Earth redirect,
|
||||
public pages, unknown-route login fallback, protected admin route login
|
||||
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
|
||||
mocked API content, Docs detail page
|
||||
language/theme/search interactions, every Docs catalog slug exposed by the
|
||||
frontend/backend metadata, the Earth iframe entry point, login error handling,
|
||||
register + email verification, password reset, standalone email verification,
|
||||
and authenticated `super_admin` rendering for every admin route plus core
|
||||
`section` deep links derived from the actual admin route and section config.
|
||||
Authenticated admin
|
||||
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
|
||||
mobile passes also fail on global horizontal overflow so table/detail panels
|
||||
must keep overflow ownership inside their own scroll regions. To enforce the
|
||||
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
|
||||
hard rendered check: the shell must resolve to the viewport height through the
|
||||
root 100% height chain, `#root`/document/body must not gain vertical overflow,
|
||||
and the desktop sidebar account/preferences area must remain inside the first
|
||||
viewport while the nav owns any excess scrolling. The smoke also
|
||||
derives the sidebar menu from the actual admin route manifest and clicks every
|
||||
visible `super_admin` menu entry on both desktop and mobile viewports, then
|
||||
exercises safe interaction paths for admin search, section tabs, the AI settings
|
||||
shortcut, logs view switching, user dialog opening, and data distribution toggles.
|
||||
It also exercises Earth News source testing, add/cancel source draft behavior,
|
||||
and manual news group creation against mocked `/earth/news-*` APIs.
|
||||
Documented AI and collector
|
||||
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
|
||||
`/collection-management?section=collector_credentials` are part of the rendered
|
||||
smoke surface:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
|
||||
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
### Visual Evidence And Style Consistency
|
||||
|
||||
- Treat user-provided screenshots and images as primary visual evidence. If a
|
||||
screenshot contradicts written text, inspect the image first and explicitly
|
||||
call out the mismatch before deciding what to change.
|
||||
- Path resolution is part of visual evidence handling. If a referenced
|
||||
screenshot path cannot be opened, try reasonable local equivalents first:
|
||||
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
|
||||
thread attachments, repository files, and obvious local attachment/download
|
||||
locations.
|
||||
- If the image still cannot be found or opened, report the exact path/access
|
||||
blocker instead of guessing. Do not infer image content from the filename, alt
|
||||
text, surrounding prose, logs, or memory.
|
||||
- OCR is acceptable evidence for text-only questions or non-multimodal
|
||||
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
|
||||
and rendering issues still require a real visual inspection or an explicit
|
||||
"could not verify visually" note.
|
||||
- Same-category UI surfaces must use one visual system per product area. Badges,
|
||||
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
|
||||
controls should reuse the shared component, shared token, or established CSS
|
||||
owner for that area instead of introducing a page-local lookalike.
|
||||
- `scripts/harness/frontend-rules-check.sh` warns when semantic
|
||||
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
|
||||
approved React and Earth CSS owner files. A warning means the reviewer should
|
||||
either move the style into the shared owner or document why this is a genuinely
|
||||
new visual family.
|
||||
|
||||
### Earth I18n Harness Rules
|
||||
|
||||
Earth i18n work must validate rendered behavior, not only static text lookup.
|
||||
Agents often miss dynamic strings that are created after initial page load, so
|
||||
the smoke treats these as first-class i18n surfaces:
|
||||
|
||||
- **Visible text and attributes**: translated checks must include `innerText`
|
||||
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
|
||||
buttons are user-facing copy, not implementation details.
|
||||
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
|
||||
BGP markers, compute centers, vessels, and news must render field labels,
|
||||
status values, source tags, action buttons, and disabled/tooltips in the active
|
||||
language.
|
||||
- **English content safety**: English mode must not fall back to Chinese news
|
||||
titles, summaries, feed names, measure words, or generic status labels. If no
|
||||
English localization exists, hide the item or use a neutral English fallback.
|
||||
- **Brand assets**: locale switching must update both text and image assets.
|
||||
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
|
||||
for English while keeping the same top-left layout and logo position.
|
||||
- **Controls and state**: switch/segmented-control visuals must follow the real
|
||||
checked/pressed state after both direct clicks and programmatic panel changes.
|
||||
A control is not valid if the state changes but the thumb, active pill, or
|
||||
`aria-*` state stays stale.
|
||||
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
|
||||
must enter through `earthMessage(...)` plus the centralized
|
||||
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
|
||||
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
|
||||
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
|
||||
or `earth:status` events.
|
||||
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
|
||||
overflow their panel. Prefer a slightly wider owning panel for important
|
||||
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
|
||||
translated tooltip.
|
||||
|
||||
Current frontend smoke explicitly covers the Earth English locale flow: brand
|
||||
image swap, settings language controls, panel switch visual sync, English news
|
||||
filtering, English detail-card text and tooltips, and TV default/source labels.
|
||||
|
||||
## Environment Requirements
|
||||
|
||||
Required for normal development:
|
||||
|
||||
- `zsh` for `planet.sh`
|
||||
- `uv` for Python dependency and test execution
|
||||
- `bun` for frontend dependency and build execution
|
||||
- Python resolved by `uv` from the root `pyproject.toml`
|
||||
|
||||
Harness command lookup first checks the current non-interactive `PATH`. If a
|
||||
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
|
||||
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
|
||||
path. This avoids hardcoding a dotfile while still covering agent environments
|
||||
that do not inherit the user's normal shell setup.
|
||||
|
||||
Required for full local stack operation:
|
||||
|
||||
- Docker and Docker Compose
|
||||
- PostgreSQL and Redis containers started by `planet.sh`
|
||||
|
||||
Optional for delivery smoke:
|
||||
|
||||
- Docker daemon for image builds
|
||||
- Helm for chart lint/template checks
|
||||
|
||||
For routine harness validation, do not install missing system software
|
||||
automatically. Report the gap and point to the explicit bootstrap entry points.
|
||||
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
|
||||
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
|
||||
This bootstrap behavior is intentional; do not invoke it merely to make harness
|
||||
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
|
||||
|
||||
Docker bootstrap regression checks use isolated command stubs and never install
|
||||
packages or modify the host daemon:
|
||||
|
||||
```bash
|
||||
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
|
||||
uv run --frozen --project . python scripts/harness/test_database_startup.py
|
||||
```
|
||||
|
||||
Database startup regressions also run in quick-check. They cover Compose
|
||||
reconciliation of existing containers, visible startup errors, published-port
|
||||
checks, bounded recreation that preserves volumes, and the backend connection
|
||||
gate before schema initialization. Their command stubs and driver mocks do not
|
||||
modify the host Docker environment.
|
||||
|
||||
## What Agents Must Not Change Automatically
|
||||
|
||||
- Do not replace Bun with npm, pnpm, or yarn.
|
||||
- Do not migrate CI from `.gitea/workflows/` to `.github/workflows/`.
|
||||
- Do not rewrite `planet.sh` lifecycle behavior as a parallel script.
|
||||
- Do not run `./planet.sh destroy` unless explicitly requested.
|
||||
- Do not commit `.env`, secrets, private keys, logs, or generated build output.
|
||||
- Do not add external integrations, hooks, or new dependency managers just to
|
||||
satisfy harness structure.
|
||||
- Do not publish internal harness docs into the product Docs UI unless a
|
||||
maintainer explicitly asks for it.
|
||||
|
||||
## Hooks And Reminders
|
||||
|
||||
No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
- Run `scripts/harness/quick-check.sh` before handing off small changes.
|
||||
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
|
||||
- Run `scripts/harness/security-check.sh` after touching config, auth,
|
||||
credentials, docs examples, or generated fixtures.
|
||||
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
|
||||
catch direct stdout/debugger calls before they reach runtime logs.
|
||||
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
|
||||
route, package-manager, debug-output, and UI rule warnings.
|
||||
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
|
||||
route changes.
|
||||
- Add focused tests before modifying backend service behavior or frontend
|
||||
workflows.
|
||||
- For docs changes, run the checks listed in
|
||||
`docs/documentation-coverage-rules.md`.
|
||||
|
||||
## Reusable Workflows
|
||||
|
||||
### Feature Work
|
||||
|
||||
1. Read `rules.md` modules for the touched area.
|
||||
2. Check `CODEMAP.md` for entry points and ownership boundaries.
|
||||
3. Inspect existing tests and docs before editing.
|
||||
4. Make the smallest behavior-preserving or feature-scoped change.
|
||||
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
|
||||
6. Update relevant docs when behavior, workflow, or operations change.
|
||||
7. For rendered frontend changes, verify the affected route with Playwright or
|
||||
the full harness smoke, because `bun run build` alone does not prove page
|
||||
usability.
|
||||
|
||||
### Bug Fix
|
||||
|
||||
1. Reproduce with a focused test or command.
|
||||
2. Patch the owning module, not a caller-side workaround.
|
||||
3. Run the focused regression test.
|
||||
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
|
||||
locally.
|
||||
|
||||
### Documentation Change
|
||||
|
||||
1. Read `docs/documentation-coverage-rules.md`.
|
||||
2. Route docs by audience: UI users, operations, or second-party developers.
|
||||
3. Keep Chinese and English technical docs paired by filename; public Docs also
|
||||
need matching frontend/backend metadata when exposed in the product Docs UI.
|
||||
4. Run the repository-specific docs checks that match the changed files.
|
||||
|
||||
### Release Or Delivery Change
|
||||
|
||||
Use the existing release skill/workflow and `.gitea/workflows/` files. Harness
|
||||
validation can smoke-check Helm and Docker locally, but it must not replace the
|
||||
release process.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `docs/harness-audit.md` records the discovery pass that led to this harness.
|
||||
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
|
||||
`agents.md` entry has been merged into it and should remain absent.
|
||||
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
|
||||
`docs/technical/{zh,en}/`.
|
||||
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
|
||||
with mocked API data. It proves route shells, auth guards, and primary admin
|
||||
sections render, but it is not a replacement for feature-specific browser QA
|
||||
against a real backend.
|
||||
- Frontend smoke prints phase-level progress by default. Use
|
||||
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
|
||||
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
|
||||
progress lines.
|
||||
161
docs/harness-audit.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Harness Audit
|
||||
|
||||
Last audited: 2026-06-26
|
||||
|
||||
This audit records the repository state used to add the agent harness. It is a
|
||||
compatibility note, not a replacement for existing rules or architecture docs.
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Area | Existing Command | Notes |
|
||||
| --- | --- | --- |
|
||||
| Bootstrap | `./planet.sh init` | Syncs uv/Bun dependencies, creates missing env files, starts data services, seeds defaults. |
|
||||
| Start | `./planet.sh start` | Starts backend, frontend, AI Provider, PostgreSQL/Redis, and Motion Agent when available. |
|
||||
| LAN start | `./planet.sh start --allow-lan` | Opens frontend/backend/AI Provider ports and requests Windows firewall/port cleanup when needed. |
|
||||
| Restart | `./planet.sh restart` | Supports scoped restart flags for backend, frontend, AI Provider, database, and Motion Agent. |
|
||||
| Health | `./planet.sh health` | Checks containers, backend `/health`, AI Provider `/health`, frontend, and Motion Agent state. |
|
||||
| Logs | `./planet.sh log` | Supports backend, frontend, AI Provider, and Motion Agent log views. |
|
||||
| User fallback | `./planet.sh createuser` | Interactive emergency/local account creation. |
|
||||
| Destructive reset | `./planet.sh destroy` | Requires confirmation and removes Planet-owned Docker/build/runtime state. Not a validation command. |
|
||||
| Backend CI smoke | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | Mirrors `.gitea/workflows/ci.yaml`. |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | Bun-only workflow. |
|
||||
| Root helper | `bun run mock:ais-ws` | Runs `scripts/mock-ais-ws-server.ts` from the root package. |
|
||||
|
||||
## Existing Agent Instructions
|
||||
|
||||
| File | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
|
||||
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
|
||||
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
|
||||
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
|
||||
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
|
||||
|
||||
## Existing CI Gates
|
||||
|
||||
The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
| Workflow | Gate |
|
||||
| --- | --- |
|
||||
| `.gitea/workflows/ci.yaml` | Backend smoke tests, frontend Bun build, Docker build smoke, Helm lint/template. |
|
||||
| `.gitea/workflows/release.yaml` | Builds and pushes frontend, backend, and AI Provider images on main/tag/manual release events. |
|
||||
| `.gitea/workflows/deploy-staging.yaml` | Deploys Helm release to staging and runs curl smoke tests inside the cluster. |
|
||||
|
||||
## Existing Docs And Architecture Maps
|
||||
|
||||
| Area | Docs |
|
||||
| --- | --- |
|
||||
| Current architecture and startup | `README.md` |
|
||||
| Technical docs index | `docs/technical/zh/README.md`, `docs/technical/en/README.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Operations | `docs/technical/zh/ops-runbook.md`, `docs/technical/en/ops-runbook.md` |
|
||||
| Startup internals | `docs/technical/zh/ops-planet-sh-startup.md`, `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md`, `docs/technical/en/agents-aiprovider.md` |
|
||||
| Frontend admin | `docs/technical/zh/frontend-admin-frontend-context.md`, `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth rendering | `docs/technical/zh/earth-frontend-context.md`, `docs/technical/zh/earth-render-layer-order.md`, `docs/technical/zh/earth-layer-style-reference.md` |
|
||||
| Plans and history | `docs/plans/README.md`, `docs/deprecated/README.md` |
|
||||
|
||||
## Release And Deploy Process
|
||||
|
||||
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
|
||||
`.claude/commands/release.md`.
|
||||
- Version-bearing files include `VERSION`, `frontend/package.json`,
|
||||
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
|
||||
`docs/version-history.md`.
|
||||
- Delivery automation lives in `.gitea/workflows/release.yaml` and
|
||||
`.gitea/workflows/deploy-staging.yaml`.
|
||||
- Helm chart entry point is `deploy/helm/planet/Chart.yaml`.
|
||||
|
||||
## Missing Or Unclear Areas
|
||||
|
||||
- The older lowercase `agents.md` entry has been merged into uppercase
|
||||
`AGENTS.md` so coding agents and harness tools use one source of truth.
|
||||
- `project_context.md` originally included older roadmap assumptions such as
|
||||
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
|
||||
harness pass updated it to separate active stack facts from future directions;
|
||||
current code and technical docs still remain authoritative when details drift.
|
||||
- No safe automatic hook system was already configured. This phase documents
|
||||
manual reminders instead of adding hooks.
|
||||
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
|
||||
|
||||
## Conflicts And Preserved Rules
|
||||
|
||||
| Conflict Or Tension | Resolution |
|
||||
| --- | --- |
|
||||
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
|
||||
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
|
||||
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
|
||||
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
|
||||
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
|
||||
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
|
||||
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
|
||||
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
|
||||
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
|
||||
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
|
||||
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
|
||||
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
|
||||
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
|
||||
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
|
||||
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
|
||||
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
|
||||
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
|
||||
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
|
||||
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
|
||||
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
|
||||
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
|
||||
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
|
||||
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
|
||||
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
|
||||
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
|
||||
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
|
||||
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
|
||||
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
|
||||
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
|
||||
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
|
||||
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
|
||||
|
||||
## Rules Coverage Evidence
|
||||
|
||||
This matrix records how the current harness checks the `rules.md` modules that
|
||||
matter for this frontend and documentation pass. "Automated" means the listed
|
||||
command fails when the rule regresses. "Smoke" means the rendered product route
|
||||
or interaction is opened with Playwright. "Manual" means the rule is still a
|
||||
judgment call and must be inspected during review.
|
||||
|
||||
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
|
||||
the user describes work with Chinese/product terms instead of module names. The
|
||||
index is the routing layer; this table is the coverage/evidence layer.
|
||||
|
||||
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
|
||||
| --- | --- | --- | --- |
|
||||
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
|
||||
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
|
||||
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
|
||||
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
|
||||
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
|
||||
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
|
||||
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
|
||||
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
|
||||
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
|
||||
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
|
||||
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
|
||||
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
|
||||
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
|
||||
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
|
||||
|
||||
## Harness Files Added
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
|
||||
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
|
||||
| `CODEMAP.md` | High-level codebase map and validation references. |
|
||||
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
|
||||
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
|
||||
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
|
||||
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
|
||||
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
|
||||
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
|
||||
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
|
||||
@@ -22,11 +22,14 @@
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
|
||||
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
|
||||
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
|
||||
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
|
||||
62
docs/plans/admin-console-i18n-plan.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# 控制台 i18n 接入计划
|
||||
|
||||
**状态**:基础设施已落地,大型业务页迁移继续进行
|
||||
**创建日期**:2026-06-29
|
||||
**核心目标**:把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系,让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
|
||||
|
||||
## 背景
|
||||
|
||||
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口,但语言状态只保存在 `docs-lang`,不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主,导致用户切到英文文档后,控制台仍是中文。
|
||||
|
||||
本计划把前端语言偏好收敛到 `planet-locale`,默认 `zh-CN`,支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口,通过前端映射与全局 locale 对齐。
|
||||
|
||||
## 设计决策
|
||||
|
||||
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层,避免长期维护自研插值、hook 和资源加载逻辑。
|
||||
- 前端统一语言枚举为 `zh-CN` / `en-US`;Docs 请求继续转换为 `zh` / `en`,新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
|
||||
- 语言偏好首版只保存在浏览器 `localStorage`,不新增后端用户设置字段。
|
||||
- `docs-lang` 保留为兼容读取和写入项,让已访问过 Docs 的浏览器能平滑迁移。
|
||||
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key;大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
|
||||
|
||||
## 分期
|
||||
|
||||
### P1:统一语言基础设施
|
||||
|
||||
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
|
||||
- 在 `frontend/src/main.tsx` 里初始化 i18n,并同步 `document.documentElement.lang`。
|
||||
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
|
||||
|
||||
### P2:高复用界面迁移
|
||||
|
||||
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
|
||||
- 搜索索引按当前语言展示,同时保留中英文关键词以免降低可发现性。
|
||||
- 用户管理页作为独立业务页示范,迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
|
||||
|
||||
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化,以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
|
||||
|
||||
### P3:大型业务页收敛
|
||||
|
||||
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
|
||||
- 过渡期保留 legacy UI 翻译桥,只处理 admin/auth 容器里的精确静态文本和属性。
|
||||
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
|
||||
|
||||
### P4:移除过渡桥
|
||||
|
||||
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时,删除 legacy UI 翻译桥。
|
||||
- 增加 key 完整性检查,确保 `zh-CN` 和 `en-US` 资源结构一致。
|
||||
|
||||
## 验证
|
||||
|
||||
- `cd frontend && bun run build`
|
||||
- `scripts/harness/frontend-rules-check.sh`
|
||||
- `scripts/harness/docs-consistency-check.sh`
|
||||
- `scripts/harness/quick-check.sh`
|
||||
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `frontend/src/i18n/`:统一 locale、资源和过渡桥。
|
||||
- `frontend/src/pages/Docs/Docs.tsx`:Docs 语言状态改为读取全局 locale。
|
||||
- `frontend/src/admin/components/layout/AdminLayout.tsx`:控制台侧边栏语言切换、导航和搜索文案。
|
||||
- `frontend/src/admin/search/indexers.ts`:Admin 搜索目标本地化。
|
||||
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`:当前实现上下文。
|
||||
602
docs/plans/compute-resource-intelligence-plan.md
Normal file
@@ -0,0 +1,602 @@
|
||||
# 算力与资源态势:展示、采集与 AI 迭代实施计划
|
||||
|
||||
状态:首版空间卡片原型实现中(2026-09-21)。已接入算力专题入口、45 个参与方目录、年份/口径选择、区域数据卡与来源、成功快照设施查询,以及 Epoch/TOP500 的首批质量修复。旧固定双图布局已移除。历史年度数据使用受控版本导入;各国完整指标覆盖、卡片图表、全榜获取、跨源设施合并、持久化指标管理、变化归因与 A4 AI 解读尚未完成,不视为全部 A1—A3 已验收。后续章节保留完整设计目标,当前实现以技术文档为准。2026-09-14 完成的口径核查仍是内容依据。面向产品、开发、数据研究与运维人员,作为此专题后续实现的设计依据。
|
||||
|
||||
## 0. 当前执行方向与范围
|
||||
|
||||
2026-09-21 展示方向修订:UI、地球材质、政治展示规则、地理资源对齐及性能门槛以[Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)为准。下文固定右侧国家面板、底部横贯双图等布局不再作为实施目标。采用从区域锚点引出连接线、卡片围绕地球自适应浮动的方向,先验证空间交互再完善卡片内容。数据来源、口径与证据要求继续有效。
|
||||
|
||||
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚:地图显示资源在哪里,图表说明规模和相对位置怎样变化,证据说明数字从哪里来,AI 帮助阅读变化。
|
||||
|
||||
采用“专题先交付,公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线,不是首版展示的前置条件;第 6 节总分保持实验性,不能为了首屏有大数字而生成不完整排名。
|
||||
|
||||
| 当前要做 | 当前完成形态 | 后续才扩展 |
|
||||
| --- | --- | --- |
|
||||
| 算力数据可信 | 修复既有采集,国家/设施/精度/年份不混用 | 大规模自动来源发现、完整证据图谱 |
|
||||
| 全球分布与国家对比 | 现有算力中心地球 + 可比参与方 + 明确样本覆盖 | 制造、矿产、电力和贸易的跨主题关系 |
|
||||
| 历史展示 | 两张核对后的图、时间选择、已知事件轨道 | 复杂预测、不确定性传播与可训练参数 |
|
||||
| 证据与解释 | 来源、参考年、关键字段;基于数据快照的简短 AI 说明 | 自动补证、模型候选、回测、影子运行与提升 |
|
||||
|
||||
首版验收路径:打开算力专题 → 看全球公开设施分布 → 选中国和美国 → 同时看算力与创新历史 → 点一个变化或设施 → 查看出处及解释。此路径稳定后,才扩大评价维度。
|
||||
|
||||
## 1. 要交付什么
|
||||
|
||||
在智能星球现有算力中心图层上增加“算力与资源态势”模式,并在控制台提供对应的数据管理和模型实验工作区。核心问题是:各方拥有什么资源、资源如何形成可用能力、相对位置如何变化、变化有何证据,以及未来什么约束可能改变格局。
|
||||
|
||||
首批比较中国和美国;数据结构从第一天支持多方。随后纳入欧洲、日本、韩国、印度及其他数据合格的参与方。地区汇总与成员国不得重复计入全球分母。所有国家和地区字段使用现有 `normalize_country()` 和规范字典,不以网页原文或 AI 判断覆盖项目标签。
|
||||
|
||||
交付三个相互连接的产品能力:
|
||||
|
||||
| 能力 | 用户获得的结果 | 首版边界 |
|
||||
| --- | --- | --- |
|
||||
| 历史观测 | 算力绝对规模、全球份额、创新指数与分差、设施时间线 | 使用可验证观测;缺失年份可见,不插值冒充事实 |
|
||||
| 变化解释 | 哪些设施、资源、数据修订造成指标变化,每项贡献和证据 | 数学贡献可复算;原因解释区分事实、假设与未解问题 |
|
||||
| 模型改进 | AI 发现缺口、补证、提出改进,经过验证产生新版本 | 固定工作流先行;不能由大模型直接决定国家分数 |
|
||||
|
||||
第一版先交付可信的事实与回放,不以总分完整作为上线前提。完整创新指数分项、有效算力参数和项目交付数据不足时,继续展示已完成的观测能力,综合评分显示“数据不足”,不生成示意国家排名。
|
||||
|
||||
不在第一版承诺全球所有设施的完整清单、精确的国家真实算力、每日变化的国家创新能力、可验证的国家综合实力真值,或自动预测某年必然追平。
|
||||
|
||||
## 2. 当前项目基础与必须补齐的边界
|
||||
|
||||
以下结论来自当前文件检查,未对生产数据库、凭证可用性或线上采集结果做完整审计。
|
||||
|
||||
| 已有入口 | 已确认能力 | 本专题需要补齐 |
|
||||
| --- | --- | --- |
|
||||
| [采集基类](../../backend/app/services/collectors/base.py)、[数据作业](../../backend/app/services/data_jobs.py) | 采集、转换、任务状态、快照、取消与重试基础 | 文档型数据源、质量门槛、领域规范化、失败保留正式快照 |
|
||||
| [Epoch 采集器](../../backend/app/services/collectors/epoch_ai.py) | GPU 集群采集入口 | 改读公开 CSV;去掉解析失败返回示例记录;单位、精度、状态和时间字段校正 |
|
||||
| [TOP500 采集器](../../backend/app/services/collectors/top500.py) | 榜单与详情解析 | 去掉示例回退、固定参考日期、排名作为身份;按当期表头解析单位 |
|
||||
| [采集记录](../../backend/app/models/collected_data.py)、[快照](../../backend/app/models/data_snapshot.py) | `entity_key`、快照关联、前一记录和变化摘要 | 原始证据版本、发表时间、有效时间、指标注册和可重放的观测集合 |
|
||||
| [算力位置服务](../../backend/app/services/compute_center_locations.py)、[GeoJSON](../../backend/app/api/v1/visualization.py) | 设施定位、位置置信信息、未定位记录 | 国家统计与地图可定位集合分离;匿名设施不能猜坐标 |
|
||||
| [算力中心渲染](../../frontend/public/earth/js/compute-centers.js)、[Earth 壳](../../frontend/src/pages/Earth/Earth.tsx) | Three.js 交互层;Earth 由 iframe 加载独立页面 | 历史时点、国家比较、双图、关系层和解释面板 |
|
||||
| [图层适配器](../../backend/app/services/earth_layer_adapters.py)、[数据库监听](../../backend/app/services/earth_db_change_listener.py) | 通过 PostgreSQL outbox 驱动缓存与 `earth_updates` | 正式指标快照发布事件;草稿与实验结果不改变默认地球 |
|
||||
| [AI Client](../../backend/app/services/ai_client.py)、[提示词注册](../../backend/app/ai_tasks/prompts.py) | 全局 provider 配置、模型调用、任务提示词及运维覆盖 | 算力专题的固定工作流、结构化输出验证、预算与审计 |
|
||||
| [搜索工具](../../backend/app/services/ai_tools/web_search.py)、[网页取证](../../backend/app/services/ai_tools/web_fetch.py) | 搜索和基础 HTML 正文提取 | PDF/表格提取、引用位置、下载时大小限制、URL 与重定向的访问限制 |
|
||||
| [证据辅助函数](../../backend/app/services/ai_tools/evidence_store.py) | 摘要、去重和内容哈希规范化 | 该文件不是持久化证据库,需补建不可变原文存储和证据记录 |
|
||||
|
||||
沿用[数据作业与 Outbox 架构](../technical/zh/data-job-earth-sync-architecture.md)及[轻量 Agent 编排计划](agents-light-orchestrator-websearch-plan.md):PostgreSQL 负责可靠账本,Redis 负责缓存;不另起 Kafka、Celery 或另一个独立调度真相源。`aiprovider` 保持协议适配,业务取证、估计、工具权限和模型发布全部由后端负责。
|
||||
|
||||
当前 `JobType` 只有采集、清理和 Earth 刷新。新增分析、实验工作流需要显式扩展任务类型、worker 分发和取消策略,不能把现有采集队列描述成已经支持通用 Agent。
|
||||
|
||||
## 3. 页面与地球怎样展示
|
||||
|
||||
### 3.1 默认工作区
|
||||
|
||||
保留 `/earth` 主入口,在现有算力中心工具中增加专题模式,避免增加一个与地球无关联的新大屏。进入后使用以下概念布局;这是规划线框,不是现有页面截图。
|
||||
|
||||
```text
|
||||
┌ 时间/参考期 · 中国/美国/其他参与方 · 指标口径 · 观测/情景 ┐
|
||||
│ │
|
||||
│ 3D 地球:设施、资源、已证实关系 国家对比面板 │
|
||||
│ 绝对量/份额 │
|
||||
│ 变化贡献 │
|
||||
│ 证据与缺口 │
|
||||
├────────────── 历史时间轴 / 事件轨道 ────────────────────┤
|
||||
│ 算力规模与份额图 综合创新指数与分差图 │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
桌面默认同时显示两张历史图,用相同国家配色,但各自保留单位和实际观测年份。图表高度优先保证刻度、来源标识与数据点可读,不能通过压缩字体维持布局。较窄屏幕把图表移入可切换的详情区,国家和设施详情使用抽屉;不叠加多个遮挡地图的浮层。
|
||||
|
||||
沿用项目一屏高度链和主题。桌面在 1366×768、1920×1080,移动端在约 390×844 验证,另测 125%/150% 浏览器缩放。页面根与中间 flex 容器分别正确设置高度和 `min-height: 0`,仅详情、长表和证据正文内部滚动。
|
||||
|
||||
### 3.2 视图职责
|
||||
|
||||
| 视图 | 主内容 | 点击或切换后的行为 |
|
||||
| --- | --- | --- |
|
||||
| 国家总览 | 可比算力绝对量、份额、子分、实际参考期;总分合格才出现 | 选中参与方,定位地球并联动两图;不重设全局评分尺度 |
|
||||
| 算力趋势 | 绝对规模、全球份额、双边份额三种独立视图 | 同一来源口径内切换;不把百分比轴和 FLOP/s 轴混合 |
|
||||
| 创新趋势 | 各方指数;只有分差时只显示中美分差 | 切换年度,查看分项与报告版本,不能反推缺失的单国得分 |
|
||||
| 变化账单 | 新投产、扩建、退役、效率变化、资料补录、方法修订 | 点选贡献定位资产;逐步展示输入、计算和证据 |
|
||||
| 设施详情 | 设备、精度、投产状态、功率、运营方、地点精度、历史版本 | 展开相关公告、设备来源及不确定字段 |
|
||||
| 资源关系 | 已证实的芯片供给、所有权、电力接入或网络连接 | 选一类关系再显示,不能让所有关系默认铺满地球 |
|
||||
| 情景工作区 | 并网推迟、交付变化、效率假设及结果区间 | 独立 scenario ID;退出恢复正式快照 |
|
||||
| 控制台研究区 | 数据缺口、待核验证据、采集任务、模型实验和版本 | 使用已有采集与 AI 设置入口;Playground 仍用于调试 |
|
||||
|
||||
### 3.3 两张图进入产品的明确规则
|
||||
|
||||
总算力图以核对后的报告观测为起点:2020 年美国/中国为 36%/31%,2021 年 34%/33%,2022 年 34%/33%,2023 年 41%/31%。原始来源和报告版本见第 15 节。2024—2025 年暂不填值;2026 年 42%/33% 只保存为待核实主张,不能接到正式历史曲线上。
|
||||
|
||||
创新图使用报道明确披露的 2023、2024、2025 年分差 22.02、19.96、17.95 分。报告发表于 2026 年,必须区分报告年份与参考年份。完整分项、年度得分和方法修订仍需采集。图中是综合创新指数,不是纯算力或纯产出。
|
||||
|
||||
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读,不进入计算;不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度,不让刷新网页造成“国家实力实时跳动”。
|
||||
|
||||
### 3.3a 当前数据合同(2026-09-22)
|
||||
|
||||
默认主指标是全球计算设备总规模,拆分基础计算、智能计算和超级计算,观察期为 2020—2023。CAICT 报告核查目前使用官方 PDF 索引文本;本地 PDF 下载未成功,因此不宣称已肉眼核验 PDF 页面。绝对量合同记录 2022 年 302(证据版本 `caict2023`)和 2023 年 435(证据版本 `caict2024`);2024 年报告正式书名为《先进计算暨算力发展指数蓝皮书(2024年)》。官方 URL 已替换为四个镜像来源。
|
||||
|
||||
日本全球份额观测为 2020/2021/2022/2023 = 6%/5%/4%/4%。产品分组 `APAC` 与 `OTHER` 不能直接套用报告中的 Japan 或“其他”;`EUROPE` 成员集合尚未与报告口径对齐,仍返回缺失。双边份额分母固定为 US + China,新加入日本不改变分母。主要指标缺失时可显示最近有报告的年份(当前可到 2023),而不是留空为 2026;用户明确选择 2026 时仍显示缺失,不回填最近值。不可比性尚未确认,不展示未经确认的同比变化;覆盖数据保留在辅助区并带 count。
|
||||
|
||||
数据 API 审计字段为 `reviewed_at=2026-09-22`,版本与该日期一致。独立 Epoch 份额模式尚未实现,用户对该问法的选择待后续确认;当前合同不提前新增该模式或声称完成 dataset 计算。
|
||||
|
||||
默认“历史回顾”允许查看最新证据重述后的过去;“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时,必须显示“最新观测:2023”,或在严格同年模式显示缺失。
|
||||
|
||||
### 3.4 地球编码与交互
|
||||
|
||||
- 节点面积按同口径能力编码;跨度过大时可切换对数档位并明示图例。国家颜色只用于身份,不能把颜色本身当作领先或落后。
|
||||
- 已投产用实心,建设或情景用空心/虚线;置信不足使用独立标记,不能把低置信透明度误读为小算力。
|
||||
- 来源只到国家或区域时,保留在国家/区域统计和未定位列表;不画在首都,不把行政区域中心当作设施坐标。
|
||||
- 国家总量与可见设施合计可以不同,分别显示覆盖范围。不得只统计有坐标的节点,也不得把宏观缺口按现有节点比例摊分。
|
||||
- Epoch 当前公开版本对中国部分集群匿名化并取整;匿名记录只能按允许的粒度使用。新增官方公告可作为独立证据,不能依据匿名数值猜出设施身份或制造地点。
|
||||
- 供应、所有权和网络关系采用不同图例,显示关系来源与有效期;商业供应关系不等同于实际运输路线。
|
||||
- 全国可汇总吞吐与最大单集群能力分别展示;地理分散设备不能合并成一个训练集群。
|
||||
- 时间回放使用稳定实体 ID 更新已有节点。隐藏图层、变更时点或情景时同步清理 hover、locked、tooltip 和 selection;请求过期结果不得覆盖新时点。
|
||||
- 使用现有 Three.js Interactable、图标和实例化能力。新增壳层必须遵循现有深度顺序,在远距约 50% 缩放检查闪烁;不另建悬浮球面掩盖深度问题。
|
||||
|
||||
## 4. 数据需要补充采集什么
|
||||
|
||||
### 4.1 P0:决定首版可信度的数据
|
||||
|
||||
“已核对入口”只表示资料或文档可查,不表示接口凭证、完整历史和生产连通性已经验证。以下频率是 Planet 建议检查频率,不是来源承诺的更新频率。
|
||||
|
||||
| ID / 优先级 | 数据与来源 | 必需字段/粒度 | 采集方式与频率 | 当前缺口与采用条件 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D01 / P0 | 信通院历年算力报告 | 国家×参考年×口径;绝对规模、全球总量/份额、单位、精度、估算方法、版本 | 已取得报告先受控导入;月度检查新版;PDF 表格提取后核验 | 已核对 2020—2023 份额及部分绝对量;补 2024 以后和其他国家;跨版本方法改变时断开序列 |
|
||||
| D02 / P0 | Epoch GPU Clusters | 集群/版本;设备数、型号、16/8/32-bit 性能、所在地、状态、投产/退役时间、功率、扩建关系 | 官方公开 CSV,每日条件抓取,按内容哈希去重 | 改造现有 HTML 解析;处理匿名、取整、缺坐标和覆盖偏差;不能声称完整全国清单 |
|
||||
| D03 / P0 | TOP500 / 可选 Green500 | 稳定系统 ID×榜单期;Rmax/Rpeak、FP64、功率、机构、名次、列表日期 | 每周检查新榜单;按发布期导入历史与详情 | 修复 rank 作为 source ID、固定日期和单位问题;榜单消失不等于退役;不当成 AI 总算力 |
|
||||
| D04 / P0 | 全球 AI 创新指数原报告及发布材料 | 国家×参考年×指标;单国总分、分项、权重、标准化、成员范围、方法版本 | 文档导入+月度更新检查 | 当前只具备部分新闻转述分差;优先获得分项和年度可比性证据;不能用差值合成各国历史得分 |
|
||||
| D05 / P0 | 已有位置维表与公开设施公告 | 实体别名、运营方、国家、地点、精度、采用依据、有效期 | 复用位置管线,新增/变化时触发 | 全国汇总独立于定位成败;不把同址不同分期误合并,也不把扩建当第二个全量集群 |
|
||||
|
||||
首版最低闭环:D01 和 D04 的已验证历史可先支撑双图,D02/D03/D05 支撑设施地图;两者不互相伪装为同一覆盖范围。若 D04 分项不可取得,创新总分仍作为独立观测,O 分与综合 S 不发布。
|
||||
|
||||
首版可以先受控导入已核实的年度数据,不必先完成整套 PDF 自动解析和报告发现。D04 原报告分项继续作为补数任务,不阻塞已经核对的创新分差图;原始摘录、出处、年份和未核实范围必须随数据保存。
|
||||
|
||||
### 4.2 P1/P2:有效能力、增长与资源关系
|
||||
|
||||
| ID / 优先级 | 补充数据与候选来源 | 粒度与核心字段 | 方式/频率 | 对模型的作用及限制 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D06 / P1 | Epoch 芯片销售、所有权、硬件数据;厂商规格 | 芯片型号×时间;规格、销售/交付、持有主体、数量、性能换算、估计区间 | 官方数据文件与规格文档;周查/季度快照 | 校验硬件供给;设计者、所有者、实际所在地与使用权分开;不能把出货和已部署重复加总 |
|
||||
| D07 / P1 | MLPerf Training / Inference、可复现厂商测试 | 配置×任务×软件版本;模型、质量门槛、batch、上下文、时延、吞吐、能耗、可用状态 | 官方结果导入;月查新版本 | 校准特定任务的性能系数;不同 benchmark 版本/质量/负载不直连;实验室成绩不等同生产利用率 |
|
||||
| D08 / P1 | IEA、电力统计机构、EIA、并网与运营方资料 | 国家/地区/项目×时点;发电、用电、接入 MW、PUE、并网日期、可用负荷 | 国家统计月/季;报告年;项目公告日查 | 电力作为设施约束;国家发电量、用电 TWh 不能直接转换机房可用 MW;中国与美国地方数据可得性不同 |
|
||||
| D09 / P1 | 项目运营方、许可部门、官方采购和工程进展 | 项目×阶段;公告、开工、设备交付、并网、验收、部分投产、撤销、金额与范围 | 已注册来源每日检查,重要项目每周核验 | 建立投产与延期标签;最终失败项目也保留,避免只收集成功项目;单纯新闻未更新不判定失败 |
|
||||
| D10 / P1 | 公司投资者关系、SEC EDGAR、境内及其他市场法定披露 | 法人/分部×财季;CapEx、已付款/指引、租赁、合同、地域/业务归属 | 官方 API/报告;日查新披露,按季归档 | 用于资金与项目约束;总 CapEx 不是 AI 投资;合并报表、融资租赁、设备与机房成本须防重复 |
|
||||
| D11 / P1 | 创新报告分项、公开模型评测、OpenAlex;HF 作为辅助 | 机构/国家×年/月;研发成果、质量归一化产出、可比模型能力、应用指标 | 原报告年;评测周;研究元数据月 | 重建不含重复资源项的 O;合作成果使用明确分摊;不从作者姓名推断国籍;下载量不代表实际使用或产业产出 |
|
||||
| D12 / P2 | 晶圆制造、先进封装、HBM、关键设备的正式披露 | 工厂/供应方×工艺/产品×季度;已投产能力、良率范围、交付、客户与依赖 | 公司/监管原件;月查、季度快照 | 建立瓶颈和替代关系;晶圆数不能无依据换成 GPU 数;总行业份额不能归给某集群 |
|
||||
| D13 / P2 | USGS、各国统计、UN Comtrade | 商品×国家×年/月;产量、储量、加工量、贸易流、HS 版本和单位 | 年度报告、月/年贸易数据 | 扩展矿产与资源依赖;储量、开采和加工分开;宽泛服务器税号不能反推高端芯片数量 |
|
||||
| D14 / P2 | 既有海缆、PeeringDB、BGP 与已证实连接资料 | 设施/网络/关系×有效期;容量声明、连接、依赖、故障 | 复用现有产品,按其真实更新节奏 | 提供连通性和韧性背景;BGP 可见性不是带宽,海缆容量不是集群内部互联性能 |
|
||||
|
||||
EIA、SEC、Epoch、MLCommons、IEA、USGS 已核对资料入口;UN Comtrade 门户可见,但本次未验证可调用 API,接入要单独完成认证、额度、字段和历史测试。OpenAlex 需在接入时按最新访问规则配置,不假定无限制匿名访问。所有付费、登录和受限数据只列为可选增强,不作为首版隐藏前提。
|
||||
|
||||
### 4.3 首批采购与补数优先顺序
|
||||
|
||||
1. 先完成现有 Epoch/TOP500 的真实性与时间口径修复,避免新增来源扩大错误。
|
||||
2. 导入信通院历史、创新分差和原始出处;追索创新原报告分项及最新版总算力数据。
|
||||
3. 建立中美重点设施及项目阶段台账,并记录未知量。重点设施用于高影响解释,不冒充全国抽样无偏估计。
|
||||
4. 用官方规格与 MLPerf 建立少量明确工作负载的换算表;其余硬件维持理论值或范围。
|
||||
5. 补项目级并网、交付、资金与退役记录,积累可校准的真实标签。
|
||||
6. 最后扩展制造、HBM、矿产与多方关系;每个新主题必须有对应指标和展示消费者,不只收集而不使用。
|
||||
|
||||
## 5. 采集到正式数据的流水线
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[注册的数据源] --> B[原文与内容哈希]
|
||||
B --> C[确定性解析或 AI 提取候选]
|
||||
C --> D[单位/时间/实体/引用校验]
|
||||
D --> E[版本化有效观测]
|
||||
D --> F[冲突与待核实队列]
|
||||
E --> G[确定性指标计算]
|
||||
G --> H[正式结果快照]
|
||||
H --> I[Outbox 与缓存更新]
|
||||
I --> J[地球/双图/变化账单]
|
||||
```
|
||||
|
||||
### 5.1 数据源合同
|
||||
|
||||
每个数据源保存来源机构、域名与下载入口、文档类型、凭证配置引用、检查频率、预期粒度、单位与精度、更新时间语义、空结果语义、历史范围、使用许可、维护负责人和解析器版本。授权文件或数据保存在运行数据存储,不提交到仓库;公开 UI 只展示允许公开的摘录或链接。
|
||||
|
||||
API/CSV 优先;HTML/PDF 采用固定解析;必须用 AI 时先产生候选,核验后保存确定性映射。扫描件 OCR 必须保留页号、表格区域和原图,数值不能只靠模型复述。下载器在流式读取时限制实际字节量,不能下载完整文件后才截断;按域名限速,遵守来源规则。
|
||||
|
||||
### 5.2 事实键与时间
|
||||
|
||||
观测键至少包括实体、指标、单位、精度、工作负载、地理范围、归属口径、参考期及来源版本。历史至少区分:
|
||||
|
||||
| 时间 | 含义 |
|
||||
| --- | --- |
|
||||
| `reference_start/end` 或 `effective_at` | 数据描述的年度、季度或设施事件发生时间 |
|
||||
| `published_at` | 资料何时首次公开;不知道就保持未知 |
|
||||
| `retrieved_at` | Planet 何时取得该版本 |
|
||||
| `recorded_at` / `superseded_at` | 系统何时采用、替换该观测 |
|
||||
|
||||
抓取日期不代替投产日期;报告年份不代替参考年份;机构估计、实测、公告、模型估计和人工情景分别标识。只有年份时保留年份精度,不能伪造 1 月 1 日投产;回放按时间范围或“当年”事件显示。
|
||||
|
||||
### 5.3 合并与质量门槛
|
||||
|
||||
- 稳定身份优先使用官方系统 ID、项目 ID、法人 ID 和可信来源链接。TOP500 排名只作为观测属性。自动实体合并必须可撤销,保存别名和合并理由。
|
||||
- 同一集群的全量扩建公告、一期/二期、重复转载与另一个数据集的同一设备只计一次。芯片交付和机房投产分别保留,不能合并为两个已投产资产。
|
||||
- 国家级统计、企业所有权统计、设施地理统计是不同视图。国家公司在海外租用算力不得同时作为两国境内部署能力;全球成员集合固定且版本化。
|
||||
- 转换单位前先核验浮点精度、稀疏/稠密、峰值/实测和时间量纲。FLOP 与 FLOP/s、TWh 与 MW 均不得互换。
|
||||
- 解析为空、字段突然缺失、总量异常下降或源结构改变,进入失败/隔离状态,保留上次正式结果。缺榜、缺行或匿名化不直接触发实体退役。
|
||||
- 多条转载归到同一 `source_family`;媒体数量不构成多份独立证据。冲突先检查口径,再保存多个主张及采用规则,不能简单取平均。
|
||||
- 缺坐标不影响已知国家统计;未知国家不强行分配。数据覆盖率可描述已登记实体或必填字段的完整率;无法知道总体时不声称“覆盖全球百分之多少”。
|
||||
- 一位有效数字等取整值保留取整范围;这是观测精度范围,不冒充统计置信区间。
|
||||
|
||||
首批历史导入先双人或规则加人工抽查核验高影响记录,保存一套回归样本。后续同版本确定性解析通过固定门槛可自动采用;新来源、新语义或无法解释的大幅变化进入候选队列。
|
||||
|
||||
## 6. 数学模型如何工作
|
||||
|
||||
### 6.1 分清事实、估计和评价
|
||||
|
||||
模型分三层:观测层保存来源说了什么;估计层回答可用能力、交付时间和不确定性;评价层按明确价值取向组合指标。国家综合实力没有一个可直接取得的标签真值,因此不能声称“AI 找到了客观最优的国家评分权重”。
|
||||
|
||||
保留 v2 的模型草案:
|
||||
|
||||
\[
|
||||
S_{c,t}=0.40H_{c,t}+0.30O_{c,t}+0.20G_{c,t}+0.10R_{c,t}
|
||||
\]
|
||||
|
||||
H 是资源能力,O 是不重复的创新与应用表现,G 是扩张潜力,R 是韧性;四项均为 0—100 的版本化子分。S 称为“综合态势分”,包含未来潜力,不能称作已投产算力、胜率或国家实力百分比。权重是默认设计参数,需通过敏感性分析和评价目的评审后冻结。
|
||||
|
||||
### 6.2 资源能力 H 与两类算力模式
|
||||
|
||||
总算力模式使用 D01 的同口径绝对规模;AI 训练模式使用明确工作负载的有效能力;推理模式另选固定模型、质量、上下文与时延要求。不同模式不共用没有物理依据的换算,也不比较彼此总分。
|
||||
|
||||
\[
|
||||
C^{eff}_{c,t,w}=\sum_{j\in deployed(c,t)} C^{peak}_{j,t,p}\,u_{j,t,w}
|
||||
\]
|
||||
|
||||
`p` 是固定精度,`w` 是参考工作负载;`u` 由可比测试、已知运行约束或经校准的估计支持。没有依据时只发布理论能力,或者明示区间;不能统一假设某国芯片只有另一个国家的固定折扣。
|
||||
|
||||
内存、互联、软件和电力会影响同一个有效利用系数,不得分别随意乘一串折扣重复惩罚。若已知场地供电上限,可用场地总功率除以 PUE 约束 IT 负荷,再按同配置效率估计可运行容量;已在利用系数中计入的限电不再扣一次。总吞吐与可用于单次大训练的最大连续集群另列。
|
||||
|
||||
国家宏观估计和公开集群样本并列校验,不直接把二者相加。若 H 只能覆盖已观测设施,分数标题和参与方排名必须明确限定为这个样本,不外推为全国真实能力。
|
||||
|
||||
### 6.3 创新与应用 O
|
||||
|
||||
完整创新指数 I 单独显示。取得 D04 分项后,建立指标归属表,逐一标注资源、扩张、韧性或创新;只把不重叠的研发质量、模型/科研成果与应用表现用于 O。不是简单执行“创新总分减去算力分”,因为不同分项的权重和标准化可能不同。
|
||||
|
||||
备用指标可来自 D11,但新增指标意味着新定义和新版本,不能偷偷替换原报告。论文数量须去重、处理合作分摊及引用年龄;模型结果要冻结评测版本;应用指标明确平台样本。专利、下载、融资和模型排行榜均不是彼此的等价替代。
|
||||
|
||||
\[
|
||||
D^I_t=I_{US,t}-I_{CN,t},\qquad v^I_t=(D^I_{t-1}-D^I_t)/\Delta years
|
||||
\]
|
||||
|
||||
2023—2025 年披露分差缩小 4.07 分,约为起点分差的 18.5%;不能解释为中国能力提高 18.5%,也不能据此推断算法效率提升。只有差值不能恢复各国绝对得分。跨年方法不一致时分段显示,不能外推追平时间。
|
||||
|
||||
### 6.4 扩张潜力 G
|
||||
|
||||
\[
|
||||
G^{raw}_{c,t,h}=\sum_{j\in pipeline(c,t)}P(T_j\leq t+h\mid X_{j,\leq t})\,\Delta C^{eff}_j
|
||||
\]
|
||||
|
||||
默认 `h=12 个月`。X 只含当时可知的项目状态、设备交付、并网、建设和资金证据。分期投产拆分未交付增量;已投产部分进入 H 后从剩余 G 扣除。公告金额不直接转成已投产能力。
|
||||
|
||||
初期采用明确的保守/基准/乐观情景,没有历史标签就不提供伪精确概率。标签积累后可比较阶段条件概率、校准的逻辑回归或生存分析;处理尚未到期项目的右删失、取消与部分交付,不能把所有未完成记录标成失败。
|
||||
|
||||
### 6.5 韧性 R
|
||||
|
||||
\[
|
||||
R_{c,t}=100\sum_s q_s\,\min(1,C^{eff}_{c,t,s}/C^{eff}_{c,t,base})
|
||||
\]
|
||||
|
||||
s 是对各方一致定义的冲击情景,例如特定供应类别中断或并网延迟;q 为公开、固定的情景权重,除非有概率依据,否则不称为发生概率。基准能力为零或未知时不计算此比值。
|
||||
|
||||
关系图用于识别依赖、替代和共同故障源。没有替代来源证据就标为未知;政策公告通常改变未来供应或可用性假设,不自动删除境内现有芯片存量。
|
||||
|
||||
### 6.6 标准化与发布条件
|
||||
|
||||
数量型正向指标可采用固定基期映射:
|
||||
|
||||
\[
|
||||
N(x;a,b)=100\,clip\left(\frac{\ln(1+x/a)}{\ln(1+b/a)},0,1\right),\quad a,b>0
|
||||
\]
|
||||
|
||||
a、b 与 x 单位一致,来自有覆盖说明的固定参考面板,存入模型版本。成本型、强度型和比例型指标各自定义映射。不得按每日领先国家重定标,也不能因加入一个国家导致全部历史分数被静默重写。
|
||||
|
||||
关键输入缺失时 `S=null`,保留可用子分,不填零或自动重分配权重。O、G 或 R 没有可比输入时综合分不上线;原始图表和证据浏览仍可上线。样本少、匿名取整或来源偏差反映在不确定性和适用范围中,不直接扣国家实力分。
|
||||
|
||||
不确定性计算分开保存观测精度、参数估计和情景假设。只有有依据的输入分布才运行 Monte Carlo;共享芯片规格、同一公告、同一供应链的误差必须相关采样。无法量化的覆盖偏差直接披露,不能靠窄置信区间掩盖。排名用“无法区分”处理不稳健结果,不展示过多小数制造精确感。
|
||||
|
||||
### 6.7 变化与归因
|
||||
|
||||
\[
|
||||
g_{c,t}=C_{c,t}/C_{c,t-1}-1,\quad p_{c,t}=C_{c,t}/\sum_kC_{k,t},\quad b_{CN,t}=C_{CN,t}/(C_{CN,t}+C_{US,t})
|
||||
\]
|
||||
|
||||
全球分母必须完整定义;双边份额不得标为全球份额。按报告披露值,2022—2023 年中国总算力从 302 增至 435 EFlops,约增长 44%,份额从 33% 降至 31%。这是“绝对增长、相对份额下降”,不是存量消失。算力与创新现有历史窗口不同,不能拼成同一时期的因果结论。
|
||||
|
||||
同权重、同版本下,分项账单严格满足:
|
||||
|
||||
\[
|
||||
\Delta S=0.40\Delta H+0.30\Delta O+0.20\Delta G+0.10\Delta R
|
||||
\]
|
||||
|
||||
分项内非线性与交互影响可使用固定分组的 Shapley 分解;必须保存分组、基线和近似误差,不能把任意计算顺序产生的贡献当唯一解释。数学归因只解释模型输出,不证明宏观因果。
|
||||
|
||||
正式发布中始终分开:现实变化、补录旧事实、来源修订、模型/基期修订。版本升级用“同一输入、不同模型”的桥接结果展示,不能把换模型造成的跳变计入国家增长。
|
||||
|
||||
## 7. 项目内 AI 如何参与
|
||||
|
||||
### 7.1 运行方式
|
||||
|
||||
复用控制台已经配置的模型,通过 `AIProviderClient` 调用;不要求新的模型供应商,也不把供应商名称写死在业务代码中。第一阶段采用后端固定步骤工作流,各角色是任务模板,可以由同一个模型执行,不需要首先建设复杂的多 Agent 协商系统。
|
||||
|
||||
在现有提示词注册表中新增以下拟议 task key;提示词版本与运维覆盖的有效内容哈希进入每次运行记录。
|
||||
|
||||
| 任务模板 | 输入 | 允许输出 | 关键约束 |
|
||||
| --- | --- | --- | --- |
|
||||
| `compute.sources.discover` | 指标缺口、许可范围、已注册来源 | 来源候选、适用指标、采集建议 | 先找原始出处;新域名不直接进入正式定时采集 |
|
||||
| `compute.evidence.extract` | 原文分块、页码、表格、目标 schema | 带证据定位的字段候选 | 未出现的数值返回缺失;原文中的指令只是待分析内容 |
|
||||
| `compute.evidence.review` | 候选、对照记录、单位与实体规则 | 冲突类型、支持/反驳证据、待核验项 | 第二个 AI 的同意不是独立事实证明 |
|
||||
| `compute.gaps.prioritize` | 缺失、来源健康、参数敏感性、采集成本 | 下一步补证清单 | 优先可能影响结论且可查证的缺口,不以国家倾向排序 |
|
||||
| `compute.change.explain` | 已计算的贡献账单、事实版本 | 有引用的可读解释 | 不能修改计算结果;事实、估计、假设分别表达 |
|
||||
| `compute.model.propose` | 误差报告、数据质量问题、模型配置 | 有边界的参数/映射/新特征提案 | 必须说明可证伪假设、预期改善、影响范围和回退方案 |
|
||||
| `compute.model.review` | 候选方案、独立评估结果、影响报告 | 发布建议与限制 | 无权改变测试标签、门槛、保留集或自己批准自己 |
|
||||
|
||||
结构化结果由后端 Pydantic schema 校验。当前 AI 返回以文本为主,需要新增结果解析与受限重试;JSON 不合格、引用不匹配、单位冲突时运行失败或保留候选,不能将自然语言当作正式数据。
|
||||
|
||||
### 7.2 工具与权限
|
||||
|
||||
允许工具包括:参数化只读指标查询、已注册来源搜索/抓取、文档段落读取、规则化单位转换、实体候选查询、固定估计器运行、创建缺口任务、保存提案、提交实验任务。
|
||||
|
||||
AI 不直接写正式分数、覆盖原始证据、执行任意 SQL/Python/Shell、修改发布门槛或扩大自己的工具权限。生成的采集映射必须通过回放样本后才保存;新增可执行采集器代码属于普通开发和发布流程,不在生产 Agent 内 `eval`。
|
||||
|
||||
网页抓取需校验协议、主机、解析后的目标地址及每次重定向,阻止访问内网、回环和元数据端点;使用专门外部抓取边界,不能把内部业务 API 和外部取证 URL 混用。AI 只得到所需公开证据或经授权的业务摘要,凭证由后端设置解析,不进入提示词、引用或日志。
|
||||
|
||||
### 7.3 用户能看见的 AI 结果
|
||||
|
||||
每份分析展示三部分:已验证事实、模型解释、仍缺哪些证据。每条数值和关键结论可展开来源。用户可以标记“实体合并错误”“单位/年份错误”“证据不支持”“假设不合理”,形成结构化反馈;收藏、点赞和是否喜欢国家排名不能成为事实标签。
|
||||
|
||||
“AI 发现数据不足”可以是成功结果。模型不可用时,确定性采集、计算、双图和地球继续工作,解释面板保留上次结果并标注版本与时间。
|
||||
|
||||
## 8. 数学模型如何自我迭代
|
||||
|
||||
### 8.1 自动迭代分成三类
|
||||
|
||||
| 层次 | 可以怎样改进 | 自动程度 |
|
||||
| --- | --- | --- |
|
||||
| 证据和数据 | 补充来源、修正字段、发现重复和过期、改进解析映射 | 已核准来源与已验证规则可自动运行;新语义及高影响冲突先进入候选 |
|
||||
| 可验证的估计参数 | 项目延期分布、特定硬件/任务效率、记录错误率 | 通过冻结评估、影子运行和预先配置的边界后,可以自动发布小范围参数更新 |
|
||||
| 评价定义 | H/O/G/R 权重、指标含义、基期、国家范围和价值取向 | 默认生成提案,由模型负责人确认版本;不是机器从不存在的“国家实力真值”中自动学习 |
|
||||
|
||||
此处是产品建议的默认发布策略,不是在本次计划工作中申请执行权限。未来可配置自动发布的参数白名单,但不能用“开启自动迭代”笼统授权所有定义变化。
|
||||
|
||||
### 8.2 迭代闭环
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[新事实/人工纠错/成熟预测结果] --> B[确定性误差与质量评估]
|
||||
B --> C[AI 提出可证伪改进]
|
||||
C --> D[生成候选配置与锁定实验]
|
||||
D --> E[按时间回测及国家分组评估]
|
||||
E --> F{是否通过预设门槛}
|
||||
F -->|否| G[记录失败原因并保留现行版本]
|
||||
F -->|是| H[影子运行:结果不改变正式排名]
|
||||
H --> I[按发布策略评审或受限自动提升]
|
||||
I --> J[发布不可变版本与变动桥接]
|
||||
J --> K[监测退化并可原子回滚]
|
||||
K --> B
|
||||
```
|
||||
|
||||
每次提案只改变一种主要因素,或者明确作为一个联合实验;不能同时换来源、标签、权重和评估方法后宣称单个参数有效。失败提案也保存,避免只展示成功实验。
|
||||
|
||||
提案记录至少包括:问题、证据 ID、基线版本、拟议配置差异、适用实体/任务、可检验目标、锁定数据截止时间、主要评估指标、停止条件和回滚版本。可以提出“将并网阶段纳入交付预测”,不能只提出“让某国得分更合理”。
|
||||
|
||||
### 8.3 真实反馈标签从哪里来
|
||||
|
||||
| 估计对象 | 可观察标签 | 评估方式 |
|
||||
| --- | --- | --- |
|
||||
| 文档字段提取 | 人工核对原文的数值、单位、年份、实体和引用位置 | 字段精确率/召回率、关键错误、引用支持率,按语言/文档类型分组 |
|
||||
| 实体合并 | 官方 ID、项目连续记录或人工确认的同一/不同实体 | 错误合并率和遗漏合并率;高影响错误单列 |
|
||||
| 项目按期投产 | 后续正式验收/投产/取消记录,保留观测截止 | Brier score、可靠性图;右删失与失访不能计作失败 |
|
||||
| 新增容量 | 未来实际确认的设备与容量,具有同口径 | 绝对误差、对数误差;同时报告项目等权与容量加权结果 |
|
||||
| 任务性能系数 | 未参与拟合的可复现实测配置 | 相同任务质量下的预测误差与区间覆盖 |
|
||||
| 区间预测 | 后续实际值是否落入范围及区间宽度 | 覆盖率与区间评分一起评价,避免靠无限放宽范围“提高准确率” |
|
||||
|
||||
不使用 AI 自己生成的摘要作事实标签,不用现行总分拟合下一版总分,不把另一个复合指数直接当国家实力真值。完整创新指数可做外部一致性讨论,但若其包含输入指标或与 O 重叠,就不能作为独立验证集。
|
||||
|
||||
### 8.4 回测与数据泄漏控制
|
||||
|
||||
预测训练、验证和保留集按真实可获得时间滚动切分,同一项目的后续阶段、同一公告转载和同一扩建链必须归组,避免分散进训练和测试。每个特征都要求 `published_at <= forecast_cutoff`;仅今天抓到而无法证明当时可得的资料,不参加“当时可知”回测。
|
||||
|
||||
新模型先与简单基线比较:维持上次状态、沿用公告日期、同阶段历史交付中位数。不得只与一个刻意弱的旧模型比较。参数、标准化锚点和来源质量估计只从训练窗口取得。
|
||||
|
||||
大模型的训练记忆可能知道后来结果,所以历史回测中的预测器必须是只读取冻结特征的确定性估计器;不能让 LLM 凭记忆补旧事实。提案生成与真实保留集隔离,实际前瞻影子运行仍是必要证据。
|
||||
|
||||
按国家、硬件代际、项目规模、来源语言分别评估,并按项目/运营方/时间块进行重采样,不能把高度相关的重复记录当独立样本。小样本报告区间与不足,不为满足仪表盘而宣称显著改进。
|
||||
|
||||
### 8.5 建议发布门槛
|
||||
|
||||
以下是实施阶段的初始门槛草案,需在 M0 用试点数据冻结。它们是最低操作条件,不保证统计有效性,AI 无权自行降低。
|
||||
|
||||
| 门槛 | 初始建议 |
|
||||
| --- | --- |
|
||||
| 硬正确性 | 单位、时间穿越、重复计数、采集失败保留正式值等必测案例全部通过 |
|
||||
| 提取评估 | 至少 200 条人工核验字段、覆盖中英文及主要文档类型;报告精确率/召回率及区间;关键数值/单位错误不得进入自动采用集 |
|
||||
| 预测参数自动提升资格 | 至少 50 个有可验证结果的独立项目、覆盖至少 3 个滚动窗口;每个主要参与方至少 10 个,不足则继续影子运行或人工评估 |
|
||||
| 改善幅度 | 预先指定主要损失相对现行版和简单基线至少改善 5%;按依赖结构重采样的损失差区间须支持改善 |
|
||||
| 分组保护 | 主要参与方损失退化不超过预先约定范围,初始建议 2%;不能以平均改善掩盖一个国家明显恶化 |
|
||||
| 区间质量 | 不仅检查覆盖,还检查宽度与评分;对无法量化的来源覆盖偏差单列说明 |
|
||||
| 影子观察 | 至少连续 4 周稳定运行,并达到所需成熟标签数;12 个月预测不因观察满 4 周就算验证完成 |
|
||||
| 试验节制 | 初始每月最多 3 个主要候选,锁定真实保留集;多次搜索造成的选择偏差要通过新的前瞻样本再次检查 |
|
||||
|
||||
标签暂时不足时,系统可以自动补数、解释、积累预测与结果,但不发布“自我学习后的更准模型”。这不会阻塞第一版事实产品上线。
|
||||
|
||||
### 8.6 一个完整迭代例子
|
||||
|
||||
以下是流程示例,不是已发生的项目事实或评估成绩。
|
||||
|
||||
系统发现一批已订购芯片的项目未如期投产;确定性评估记录预测误差。AI 查询证据后提出:现有交付模型只使用芯片到货时间,遗漏电力接入阶段。它提交一个新增“并网是否完成”特征的候选。
|
||||
|
||||
后端按项目原始发表时间构造训练数据,用固定估计器拟合;锁定保留集检验总体和中美分组误差。即便论文或第二个 AI 都认为这个想法合理,只要结果未通过门槛就不发布。通过后先影子运行;正式发布时展示受影响的 G 分、预计容量和模型版本变化,已投产 H 不因推测而下降。
|
||||
|
||||
若后续发现并网字段提取错误导致退化,回滚活动模型指针,保存新证据和失败实验。不能删除失败记录或改写过去已发布的预测。
|
||||
|
||||
## 9. 后端数据与实现结构
|
||||
|
||||
首版采用最小领域实现:沿用 `CollectedData`、`DataSnapshot` 和位置维表,新增经过 schema 校验的国家指标记录与聚合服务。数值口径、参考期、发表时间、来源链接、原文哈希和证据定位可先作为受约束的元数据保存;必要的原件存放在许可允许的运行数据目录。只有现有表确实无法表达查询或历史约束时才增加专用表。
|
||||
|
||||
先冻结面向专题的 API 契约,地球与图表只消费该契约。将来拆出指标观测、证据、实验等表时,通过服务层迁移,不让前端依赖临时 JSON 或物理表结构。首版不以第 9.1 节全部表建完作为开工条件。
|
||||
|
||||
### 9.1 拟议持久化对象
|
||||
|
||||
下表为待新增的逻辑对象,可按最终数据库设计合并有限的小表;不要与现有采集、位置和任务状态形成并行真相源。
|
||||
|
||||
| 对象 | 保存内容 | 关联和约束 |
|
||||
| --- | --- | --- |
|
||||
| `evidence_documents` | 原文存储引用、原始/提取哈希、来源、发表/抓取时间、许可、提取版本 | 内容不可变;全文不在高频列表接口返回 |
|
||||
| `metric_definitions` | 粒度、单位、精度、工作负载、归属、方向、适用范围和版本 | 一个稳定 metric ID;模型版本引用具体定义版本 |
|
||||
| `metric_observations` | 数值/范围、参考期、知识时间、状态、来源版本、取整和缺失原因 | 链接原采集记录;追加式修订,不复制整份原始 JSON |
|
||||
| `observation_evidence` | 观测与证据的支持/反驳关系、页号、表格/段落、短摘录 | 多对多;来源家族避免转载重复投票 |
|
||||
| `compute_entities` / `resource_relations` | 稳定实体、别名、分期关系、所有权/供应/连接、有效期 | 复用现有位置维表;不另存一套会漂移的正式坐标 |
|
||||
| `model_versions` | 指标集合、权重、锚点、参数、代码版本、情景与发布状态 | 不可变版本;活动指针独立且原子切换 |
|
||||
| `model_runs` / `model_results` | 输入清单哈希、版本、截止时点、随机种子、环境、国家结果、分项和不确定性 | 结果可重放;实验、情景与正式运行分开 |
|
||||
| `model_proposals` / `model_evaluations` | 配置差异、假设、冻结评估方案、损失、分组结果、决定与回退目标 | 发布必须引用通过的评估,保存失败实验 |
|
||||
| `agent_runs` | task key、提示词有效哈希、模型、工具调用、引用、预算、结构化输出、终态 | 链接现有任务账本,Agent 自身不拥有另一套调度状态 |
|
||||
|
||||
重放保留输入观测 ID/版本清单或不可变清单文件,不能只保存一段 SQL 然后查询已被更改的数据。记录计算实现版本、依赖环境和随机种子;LLM 输出保存为证据,不要求重新调用模型才能复算数值。
|
||||
|
||||
历史删除与保留策略需要显式区分:普通缓存可清理;被正式模型版本引用的观测及许可允许保存的证据受保留保护。来源撤回或许可变更时标记可访问状态并保留允许保留的哈希和元数据,不强行公开原文。
|
||||
|
||||
### 9.2 待新增模块与现有边界
|
||||
|
||||
| 拟议模块 | 职责 |
|
||||
| --- | --- |
|
||||
| `backend/app/services/compute_intelligence/observations.py` | 规范化、有效观测选择和数据缺口 |
|
||||
| `.../metrics.py`、`.../scoring.py` | 纯数值计算、单位/指标注册、标准化和总分 |
|
||||
| `.../forecasting.py`、`.../evaluation.py` | 固定估计器、时间切分、基线、分组指标与实验 |
|
||||
| `.../workflows.py`、`.../proposals.py` | 固定 AI 工作流、结构化候选、预算和发布规则 |
|
||||
| `.../publication.py` | 正式快照、变动桥接、活动版本和 outbox |
|
||||
| `backend/app/api/v1/compute_intelligence.py` | 参数化只读查询和后台任务入口 |
|
||||
| `frontend/public/earth/js/compute-intelligence.js` | 专题 UI 状态、图表与现有图层联动,消费后端计算结果 |
|
||||
| 控制台专题页与现有采集/AI 设置页签 | 证据核验、缺口、模型实验、版本回看 |
|
||||
|
||||
具体文件在实现时保持单一职责,优先复用已有服务,不为目录结构创建空文件。新来源继续注册到现有采集器体系及 datasource 配置;新提示词放入现有 `default_prompts.json`,不散落在 renderer 或 `aiprovider`。
|
||||
|
||||
### 9.3 拟议 API
|
||||
|
||||
下列接口尚不存在,名称在 M0 与当前路由约定核对后冻结。
|
||||
|
||||
| 方法与路径(统一前缀 `/api/v1/compute-intelligence`) | 返回内容 |
|
||||
| --- | --- |
|
||||
| `GET /overview` | 所选参与方、口径、参考期与知识截止下的结果、适用范围及覆盖说明 |
|
||||
| `GET /series` | 指定指标的有界历史序列,含缺失、版本、实际观测时间与来源引用 |
|
||||
| `GET /entities/{id}` | 设施、分期、资源关系和历史观测 |
|
||||
| `GET /changes` | 数据或模型贡献账单,数据库侧分页 |
|
||||
| `GET /evidence/{id}` | 根据权限和许可返回证据摘要、定位与允许展示的内容 |
|
||||
| `GET /models`、`GET /runs/{id}` | 模型版本、输入清单、计算结果与评估摘要 |
|
||||
| `POST /recompute`、`POST /agents/run` | 建立任务并立即返回任务 ID,允许取消和查看进度 |
|
||||
| `POST /proposals/{id}/evaluate` | 创建冻结数据与方案的评估任务 |
|
||||
| `POST /models/{id}/promote`、`POST /models/{id}/rollback` | 按角色及发布策略原子切换活动版本,记录理由和影响 |
|
||||
|
||||
查询参数固定包含 `metric_profile`、`entities`、`reference_at`、`knowledge_cutoff`、`model_version` 和可选 `scenario_id`。前端不接收任意表达式或 SQL,不自行补算另一套总分。
|
||||
|
||||
### 9.4 发布与性能
|
||||
|
||||
后台完整计算并通过质量门槛后,在事务中发布结果快照和活动指针,通过既有 outbox 唤醒刷新。新 adapter 显式声明国家面板/历史图的刷新范围;设施变化才触发相应 `computeCenters` 更新。草稿证据、失败实验和影子运行不改变默认地球。
|
||||
|
||||
缓存键包含口径、实体集合、参考期、知识截止、数据清单哈希、模型与情景版本。WebSocket 只通知结果版本与必要增量,客户端拒绝较旧响应;不能每次国家 hover 都重算模型或执行 LLM。
|
||||
|
||||
聚合、过滤、排序和分页在数据库完成;预计算国家年度/月度可用序列,地图按分辨率聚合节点。时间滑动去抖并复用已取快照,播放使用相邻状态差量。建议试点验收目标为缓存查询 P95 小于 1 秒、切换已缓存时点小于 500 毫秒,声明基准机器、数据规模和网络条件;未实测前不作为已达到性能。
|
||||
|
||||
## 10. 刷新、成本和失败处理
|
||||
|
||||
| 工作 | 建议节奏 | 触发条件与成本控制 |
|
||||
| --- | --- | --- |
|
||||
| CSV/API 检查 | 日或来源允许的更慢频率 | ETag/Last-Modified/哈希去重,无内容变化不调用 AI |
|
||||
| 年报与指数发现 | 月度,发布季可提高 | 找到新版本再下载和提取,不把月检写成月度指标 |
|
||||
| 重点项目证据 | 日查、周复核 | 去重后仅高影响变化进入提取队列 |
|
||||
| 指标重算 | 有效观测或模型版本改变时 | 只重算受影响实体/时点/指标依赖 |
|
||||
| AI 解释 | 有意义的结果变化或用户请求 | 相同输入哈希复用,不因页面打开重复生成 |
|
||||
| 数据缺口排序 | 每周 | 敏感性、覆盖风险、可获得性和预计采集成本共同决定 |
|
||||
| 参数候选实验 | 月度或成熟标签达到门槛 | 限制候选数量;无足够标签不强制产出升级 |
|
||||
|
||||
试点默认每工作流最多 3 次搜索、8 个页面抓取、4 次 LLM 调用、1 次结构化修复重试,最多同时运行 2 个 AI 工作流;均为可配置预算,不是第三方接口能力承诺。按全局模型配置记录 token/费用,设置日预算和单任务截止时间,超额进入延后或停止状态,不无限自递归。
|
||||
|
||||
失败处理:采集失败保留上次正式值和新鲜度;AI 不可用仍提供确定性结果;引用失效保留许可允许的原件和哈希;长期资料缺口显示缺失并降低结论覆盖范围;新源冲突进入待核验;模型退化恢复前一活动版本。不能用重试或回滚删除已提交有效证据。
|
||||
|
||||
## 11. 分期实施与依赖
|
||||
|
||||
工作量是规划估算,按一名熟悉项目的全栈开发者、持续可用的数据核验支持和现有环境可用计算;不是交付承诺。开发时间不包括等待付费授权、原始报告或 12 个月预测结果成熟的时间。
|
||||
|
||||
### 11.1 当前优先交付:展示专题
|
||||
|
||||
| 顺序 | 修改方向 | 验收内容 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| A1 修复数据与最小合同 | Epoch CSV、TOP500 日期/身份/单位、空结果保护;保存历史指标和来源 | 能得到真实当前快照与核验后的年度序列,不要求宏观指标全到最新年 | 2—3 |
|
||||
| A2 国家对比与历史接口 | 国家汇总、公开样本与宏观统计分离、双图数据、证据摘要 | 绝对量/份额/创新分差有各自定义;缺失与时点可见 | 2—3 |
|
||||
| A3 地球与图表展示 | 专题模式、国家面板、双图、时间联动、设施/证据详情 | 完成第 0 节完整浏览路径,并通过桌面/移动/缩放验证 | 3—5 |
|
||||
| A4 可选 AI 阅读辅助 | 复用现有 client 和任务提示词,针对冻结快照生成带引用说明 | 事实与推断分开;可关闭;失败不影响地球与图表 | 1—2 |
|
||||
|
||||
A1—A3 是当前必须交付的闭环,约 7—11 个开发工作日;A4 可在闭环稳定后增加。实际数据源不通、现有采集回归问题或主题布局复杂度会改变估算。实现时首先验证 A1,而不是先开发通用 Agent 框架或完整综合评分。
|
||||
|
||||
### 11.2 后续平台能力路线
|
||||
|
||||
下表是完整能力建设的工作包估算,与 A 路线存在复用和重叠,不应重复相加。A 路线完成后先盘点已具备的 M0—M2 能力,只实施剩余部分。
|
||||
|
||||
| 阶段 | 工作包 | 交付物与通过条件 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| M0 合同与试点 | 固定口径、成员、来源许可、历史时间语义、角色权限、质量/评估门槛 | 指标注册表、数据源接入清单、黄金核验样本;所有未知明确登记 | 3—5 |
|
||||
| M1 可信数据 | 修复 Epoch/TOP500;证据与指标观测;导入两组历史;稳定身份与失败保护 | 可查询、可追溯、可重放的历史 API;无示例数据进入正式结果 | 7—10 |
|
||||
| M2 可见产品 | 地球专题、双图、国家对比、变化账单、证据抽屉和控制台缺口页 | 中美历史回放闭环;移动/缩放和旧图层回归通过;综合分可保持未就绪 | 7—10 |
|
||||
| M3 资源与潜力 | 芯片、实测、项目、电力/资金台账,场景与分项估计 | H 与可用 G/R 子项具备来源;O 仅在分项齐备后发布;总分门槛满足才开放 | 8—12 |
|
||||
| M4 AI 工作流 | 固定提取/核验/补证/解释流程、反馈和预算;扩展可靠任务类型 | AI 只能产生有引用候选与解释,失败不会污染正式数据 | 7—10 |
|
||||
| M5 模型迭代 | 真实标签、时间回测、候选评估、影子运行、发布桥接与回滚 | 能证明某个可观察估计目标改善;不要求每轮必有新模型 | 8—12 |
|
||||
|
||||
从零完整建设上述广义平台能力约 40—59 个开发工作日,包含比 A 路线更完整的证据和实验治理;它不是当前展示专题的工期。影子期至少四周,但不成熟的预测标签会延长自动提升资格等待期。
|
||||
|
||||
依赖顺序:M0 → M1 → M2;M3 的项目台账应尽早开始积累标签;M4 依赖 M1 的可信证据结构;M5 依赖 M3 的标签和 M4 的审计工作流。多方、矿产和复杂供应链在 M2 之后按数据质量逐个扩展,不等全部主题齐备再上线。
|
||||
|
||||
## 12. 验收与测试矩阵
|
||||
|
||||
| 范围 | 必测案例 | 通过标准 |
|
||||
| --- | --- | --- |
|
||||
| 来源真实性 | HTML 改版、空 CSV、404、部分下载、示例字符串 | 正式数据不被示例或空结果替换;错误原因可见 |
|
||||
| 单位与精度 | T/P/E 换算,FP64/FP16/FP8,稀疏/稠密,MW/TWh,FLOP/FLOP/s | 不同口径拒绝合并;可比转换有可复算结果 |
|
||||
| 时间与版本 | 固定抓取日误作投产日、2026 报告回填 2023、未知发布日期 | 严格截止查询无未来信息;历史回顾明确重述 |
|
||||
| 实体与去重 | 排名换位、扩建、名称变化、匿名化、重复转载、跨源同集群 | 身份稳定;不重复计数;匿名或缺榜不自动退役 |
|
||||
| 统计与地理 | 缺坐标、未知国家、欧盟与成员国、跨境所有权 | 国家统计不依赖坐标;分母不重叠;归属视图不混用 |
|
||||
| 两张历史图 | 已核验年份值、缺失年份、2026 待核实主张 | 只展示有证据的观测;不能伪造平直历史或默认预测 |
|
||||
| 综合评分 | 缺 O/G/R、权重和不为 1、标准化越界、国家集合变化 | 非法配置拒绝;缺失总分为 null;固定版本不静默漂移 |
|
||||
| 变化账单 | 新投产、补录、源修订、换模型 | 分项贡献之和满足计算容差;类型和版本差异可解释 |
|
||||
| AI 输出 | 格式错误、虚构引用、原文含指令、模型超时、重复任务 | 输出隔离;工具边界有效;任务幂等并能取消 |
|
||||
| 模型评估 | 标签穿越、同项目跨集、过拟合小样本、分组退化 | 不通过门槛不发布;保留失败实验 |
|
||||
| 发布与回滚 | 两个并发发布、计算中断、旧 WS/HTTP 响应 | 原子版本;半成品不可见;客户端不回滚到旧时点 |
|
||||
| 地球渲染 | 远距/近距、时间播放、层隐藏、节点锁定、场景退出 | 无闪烁和悬空 tooltip;实例复用,不每帧重新建纹理 |
|
||||
| UI | 桌面、移动、125%/150% 缩放、键盘、无数据/低置信/加载状态 | 一屏工作区无意外全局滚动;关键标签不靠颜色区分 |
|
||||
|
||||
实现阶段按变更执行后端精确测试、`scripts/harness/quick-check.sh`、Bun 前端构建和渲染 smoke;地球改变不能只凭构建通过。保持现有公共页面、鉴权、管理路由、安全导航及其他地球图层的回归证据。性能阈值用记录过的基准规模验证,不能把开发空数据测试称为生产验收。
|
||||
|
||||
## 13. 文档、发布与完成定义
|
||||
|
||||
每阶段 PR 和部署使用已有项目流程,功能旗标分别控制专题 UI、估计分项、AI 工作流与自动参数提升。先开放读取与事实浏览,再启用模型实验;关闭新旗标应回到原有算力图层而不丢历史数据。
|
||||
|
||||
发生真实用户工作流变化时更新中英文手册与快速入门;采集、AI、作业与位置变化更新相应后端技术文档;控制台、Earth、图层顺序和样式变化更新对应 context 与规范。新增公共技术文档需要同步中英文和 Docs 注册;本文件是内部中文计划,不注册为已实现的公共手册。
|
||||
|
||||
当前展示专题完成定义:用户完成第 0 节浏览路径,可以从国家指标变化回到出处和所用计算版本;重新计算得到同一数值;知道哪些资料尚缺。两张图应和国家、设施、时间、证据联动,不能只是贴在地球旁的静态图片。
|
||||
|
||||
后续决策辅助与模型迭代完成定义:情景的假设与结果可复算;AI 的一次成功改进有独立验证目标、评估记录和可回滚版本。只有一段 AI 分析或未经校验的总分,不代表完成决策辅助能力。
|
||||
|
||||
## 14. 风险与预先决定的退路
|
||||
|
||||
| 风险 | 产品和实施处理 |
|
||||
| --- | --- |
|
||||
| 无法取得创新原报告分项 | 保留完整指数独立观测;O 和总分不发布,不伪造拆分 |
|
||||
| 中国或其他地区公开设施更少 | 显示宏观统计与设施样本两个覆盖视图,扩充本地语言原始来源,不把保密或匿名化当能力下降 |
|
||||
| 硬件真实利用率不可知 | 理论量先行,明确任务和区间;不把实验室 benchmark 当生产实测 |
|
||||
| 来源改版、匿名化或许可证变化 | 版本化下载、质量隔离、许可范围内保存证据,必要时停更单一来源 |
|
||||
| 资金、电力、芯片相互重复计分 | 固定指标归属与约束链;不把投入、存量、产出和事件各加一次 |
|
||||
| 训练标签不足或结果多年后才成熟 | 先提供情景与证据改进,延后参数自动提升资格 |
|
||||
| 模型迎合既定国家排名 | 预先定义目标和门槛,评价权重单独评审,不以用户喜好或另一个 AI 打分训练 |
|
||||
| 已有采集/定位状态被新服务覆盖 | 复用当前真相源,使用明确字段所有者、发布快照和回归测试 |
|
||||
|
||||
## 15. 来源与相关文档
|
||||
|
||||
以下链接支持数据口径、来源可得性和架构选择;除明确标注的历史值外,本计划的字段、频率、权重、工期和发布门槛均是拟议设计。
|
||||
|
||||
- [信通院 2021 年版算力白皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202109271518811781_1.pdf):2020 年总算力份额。
|
||||
- [信通院 2022 年版白皮书(镜像)](https://13115299.s21i.faiusr.com/61/1/ABUIABA9GAAgvq2fmwYozLrrpwU.pdf):正文第 13—14 页,2021 年份额与口径。
|
||||
- [信通院 2023 年版白皮书(镜像)](https://www.ahchanye.com/wp-content/uploads/2023/09/2023092815342684.pdf):正文第 13 页的 2022 年份额,以及 302 EFlops 总规模。
|
||||
- [信通院 2024 年版蓝皮书(镜像)](https://pdf.dfcfw.com/pdf/H3_AP202502051642799257_1.pdf):正文第 10 页的 2023 年份额及后续规模说明。
|
||||
- [科技日报关于创新指数报告的报道(新浪转载)](https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml):分差与综合指标维度;未替代原报告分项。
|
||||
- [2026 年 42%/33% 候选出处](https://www.mornai.cn/news/gpu/ai-computing-power-never-sleeps/):待核实商业文章,不作为正式统计。
|
||||
- [Epoch GPU 集群公开数据](https://epoch.ai/data/gpu-clusters)、[字段与下载说明](https://epoch.ai/data/gpu-clusters-documentation):CSV、地理字段、覆盖与公开版本限制。
|
||||
- [Epoch 芯片所有权](https://epoch.ai/data/ai-chip-owners):所有权不等于使用权或所在地,交付与投产有时间差。
|
||||
- [MLPerf Training](https://mlcommons.org/benchmarks/training/)、[MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/):固定任务、质量及测试配置。
|
||||
- [IEA Energy and AI](https://www.iea.org/reports/energy-and-ai)、[EIA 开放数据](https://www.eia.gov/opendata/):能源资料入口,不能直接替代设施并网证据。
|
||||
- [SEC EDGAR API 说明](https://www.sec.gov/search-filings/edgar-application-programming-interfaces):公司提交与财务数据访问;需注意财季与标签口径。
|
||||
- [USGS 矿产概要](https://www.usgs.gov/centers/national-minerals-information-center/mineral-commodity-summaries)、[UN Comtrade 门户](https://comtradeplus.un.org/):矿产与贸易候选数据源。
|
||||
- [OpenAlex 帮助与数据说明](https://help.openalex.org/):研究元数据候选来源;接入时核验访问与归属规则。
|
||||
- [业务架构与数据流转](../technical/zh/platform-data-flows.md)、[后端采集器](../technical/zh/backend-collectors.md)、[AI Provider 边界](../technical/zh/agents-aiprovider.md)。
|
||||
- [智能星球前端上下文](../technical/zh/earth-frontend-context.md)、[渲染图层顺序](../technical/zh/earth-render-layer-order.md)、[图层视觉规范](../technical/zh/earth-layer-style-reference.md)。
|
||||
- [轻量 Agent 编排](agents-light-orchestrator-websearch-plan.md)、[态势感知基础计划](agents-situational-awareness-foundation-plan.md):复用总体方向,不重复建设 provider 内业务编排。
|
||||
154
docs/plans/development-agent-model-allocation-plan.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# Planet 开发 Agent 模型分配与 Token 控制计划
|
||||
|
||||
日期:2026-09-21。状态:已用于本轮地理与空间卡片开发;未改变全局或项目默认模型配置。适用于已授权的开发任务,不代替用户对任务范围的决定。
|
||||
|
||||
目标:在固定政治展示规则、现有 UI 逻辑和性能门槛下,降低完成一个合格任务的总成本与返工次数。不是让所有工作都使用低价模型,也不是让最强模型从头包办。
|
||||
|
||||
当前重点任务见 [Earth 地理资源与空间卡片计划](earth-spatial-presentation-and-geographic-assets-plan.md)。该计划的功能与验收要求优先;本文件只决定如何组织开发。
|
||||
|
||||
## 1. 模型角色
|
||||
|
||||
模型定位参考 OpenAI Docs;下表的具体分工是针对 Planet 的工程建议,需要结合实际一次通过率和返工记录调整,不是模型能力保证。
|
||||
|
||||
| 模型 | 精确模型 ID | 默认推理强度 | 分配的工作 | 不作为默认用途 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Astra | `gpt-6-astra` | `high` | 跨系统难题、空间展示方案、复杂坐标/深度/性能诊断、关键设计与独立审查 | 日常文件搜索、机械编辑、反复运行已确定命令 |
|
||||
| Sol | `gpt-5.6-sol` | `medium`;复杂实现用 `high` | 执行主控、核心代码、共享组件整合、Three.js 状态与生命周期、最终集成 | 逐条整理大批重复记录、重复复述整个项目背景 |
|
||||
| Terra | `gpt-5.6-terra` | `medium` | 接口已明确的常规实现、局部 UI 修复、数据处理脚本、测试编写、浏览器回归 | 独自决定空间交互方向、坐标模型或跨模块架构 |
|
||||
| Luna | `gpt-5.6-luna` | `low`;明确的小修改可用 `medium` | 定向检索、清单比对、日志归类、说明文档初稿、固定格式整理 | 核心渲染、数据迁移、几何归属修改、最终审美或性能放行 |
|
||||
|
||||
日常推荐组合是 Sol 主控加一个 Terra worker;Luna 和 Astra 按需进入,不要求每个阶段凑齐四种模型。已经完成的资源准备不因采用这份方案重新做一遍。
|
||||
|
||||
当前会话的主模型不会因为写入计划自动变成 Sol。真正执行时,主会话模型由实际任务设置决定,子 Agent 在派发时明确指定模型和推理强度,并核对返回的运行配置;不能把角色名称当作已经切换模型的证据。
|
||||
|
||||
## 2. 本项目任务分配表
|
||||
|
||||
| 工作包 | 主执行 | 独立验证/审查 | 交付和升级条件 |
|
||||
| --- | --- | --- | --- |
|
||||
| 现有窗口、连线、巡航、滚动条入口盘点 | Terra | Sol 检查引用与遗漏 | 文件/符号清单、现有行为证据;Luna 可辅助定向搜索,无需读取全仓库 |
|
||||
| 贴图归档、尺寸、哈希、配准元数据检查 | 直接运行脚本;脚本需新增时 Terra | Sol 抽查实际输出 | S0 已完成,后续只做有变化的核对,不委托模型重复计算哈希 |
|
||||
| 固定政治规则的技术合同与几何覆盖边界 | Sol | Astra 审查规则是否贯穿各数据路径 | 权属口径已由项目确定,任何模型均不得重新决定;标识不明确先核对,不能猜测几何 |
|
||||
| 贴图、海岸线、拾取错位定位 | Sol `high` | Astra 在根因不清或涉及多个坐标层时介入 | 用明确经纬点复现,区分错误数据、回退、投影、UV 与半像素偏移 |
|
||||
| 规则已明确的映射、来源清单、局部转换脚本 | Terra | Sol;关键几何结果由 Astra 抽查 | 只落实批准合同;发现新的几何语义不得自行扩大规则 |
|
||||
| 多区域空间卡片的第一版运动设计 | Astra `high` | Sol 检查能否复用当前实现 | 明确锚点、投影、避让、换侧、背面及阅读稳定性;产出小合同,不产出整套 UI 大改 |
|
||||
| 2 张卡片的连接、弹性移动、生命周期原型 | Sol `high` | Terra 实测;Astra 看关键动效证据 | 优先实现最小完整循环;使用参考图和实际运动证据,不用文字自评代替 |
|
||||
| 扩至 4/6 卡、增删区域和事件整合 | Sol | Terra 跑交互/竞态测试 | 复用原型合同;`main.js` 等共享入口只由指定 owner 写入 |
|
||||
| 共享窗口行为、标准字体、移动抽屉及深色滚动条 | Terra | Sol 看渲染与旧面板回归 | 修改公共滚动所有者,验证图层等所有相关页;涉及共享状态重构时升级给 Sol |
|
||||
| 地球柔光、材质层次、节点与背景关系 | Sol | Astra 对照 UE/PPT/海报审查 | 第一轮只改少量确定参数并给实际截图;性能退化交 Sol 诊断,难题交 Astra |
|
||||
| 卡片内指标、普通图表、加载/缺失/错误状态 | Terra | Sol | 数据合同先冻结;不复活固定全屏大面板,不编造缺失年份 |
|
||||
| 国家/区域接口和前后端联动 | Terra;跨模块变更由 Sol | Sol | 不硬编码中美为唯一对象;范围筛选、比较对象和组合去重必须一致 |
|
||||
| 性能基准工具和回归测试 | Terra | Sol 检查方法;Astra 审查疑难退化 | 固定设备、数据与相机操作;日志含帧时间和内存证据,不能只报平均 FPS |
|
||||
| 单位、日期、缓存、旧请求覆盖等复杂数据故障 | Sol | Astra 仅处理跨系统未解问题 | 先给最小复现和失败测试,避免多个模型同时试不同修复 |
|
||||
| 中英文手册、资源清单、计划状态同步 | Luna 初稿;技术性较强时 Terra | Sol 对照实际 diff | 已实现与待实施分开;Luna 不能自行宣告功能完成 |
|
||||
| 最终集成与验收 | Sol | Astra 对关键风险做一次集中审查 | 相关测试、渲染、政治规则及性能证据齐全;普通修订不逐项再次走 Astra |
|
||||
|
||||
## 3. 顺序与并行安排
|
||||
|
||||
### 第一轮:证据与边界
|
||||
|
||||
Sol 读取现有计划、当前工作区差异和必要规则,整理短任务清单。Terra 盘点实际运行的数据提供者、组件和性能基线;独立且批量的文档/引用整理才交 Luna。先读已有检查结果,避免重新下载贴图、重新研究 World Monitor 或重复跑已经通过的检查。
|
||||
|
||||
Sol 将确定的组件接口、规则约束、实际截图与未解问题整理成一个短包,Astra 只审查高返工风险的空间方案和坐标问题。Astra 返回决定、理由和验证点,不再从零扫描整个仓库。
|
||||
|
||||
### 第二轮:最小空间原型与独立 UI 修复
|
||||
|
||||
Sol 实现 2 张卡片的锚点、连线和有阻尼移动;Terra 可以同时处理公共滚动条或写回归测试,前提是文件所有权不冲突。先完成最小可演示运动,再扩展内容和卡片数量。若两者都要改 `controls.js`、`hud.css` 等共享文件,则按顺序交接,不并行写入。
|
||||
|
||||
Astra 对原型的实际截图/录屏做一次聚焦审查,检查是否符合“从地球四周延伸”的方向。空间原型尚未确认时,不展开完整指标卡片和全量国家 UI,以免重演方向错误后大规模返工。
|
||||
|
||||
### 第三轮:内容、全球扩展与性能
|
||||
|
||||
合同稳定后,Terra 接入指标与状态,Sol 负责与渲染/生命周期的整合。性能测试由脚本执行,Terra 整理差异;可复现的退化先交 Sol。只有不能局部解释的 GPU/CPU、坐标、异步问题才交 Astra。
|
||||
|
||||
### 第四轮:一次集成验收
|
||||
|
||||
Sol 整合并运行所需检查;Luna 根据最终 diff 更新文档初稿。Astra 集中审查关键几何、政治规则完整性、卡片运动与性能结论。已通过范围不反复重测,除非新修改影响该范围。
|
||||
|
||||
## 4. 并发上限与文件所有权
|
||||
|
||||
当前环境总并发槽位为 4,含主 Agent;执行时仍以实际可用额度为准。默认使用主 Agent 加 1 个 worker。只有存在两项以上真正独立的工作,且主 Agent 同时有有用工作可做时,才扩大到主 Agent 加 2 个 worker;第 4 个槽位只给临时审查或独立批处理。
|
||||
|
||||
每项任务指定可写文件范围。`main.js`、`controls.js`、共享 HUD/InfoCard 与跨层状态接口同一时间只有一个写入 owner。其他 Agent 只读检查或提交补丁建议,由 owner 集成。不得并行启动多个浏览器重测相同场景,也不得共享有冲突的登录/本地存储测试状态。
|
||||
|
||||
浏览器验证优先使用当前可用的 Browser 控制能力,包括统一 CUA 入口。不能仅因技能目录中缺少单独的 Browser 条目就判定工具不存在;应继续检查已暴露工具并多次重试连接。只有多次确认 Browser 不可用后,才考虑独立 Playwright,并说明具体阻塞。
|
||||
|
||||
Agent 完成有边界的任务后停止。只有同一任务需要补充验证时才续用;不能让 worker 在空闲时自动寻找新功能或自行扩大范围。创建用户侧新任务不作为内部拆工方式。
|
||||
|
||||
## 5. 节省 Token 的具体做法
|
||||
|
||||
1. 派发窄任务,不默认继承整个历史对话。使用简短任务卡加源文件路径;必要历史按最小范围提供。短上下文不能省略政治规则、性能要求和相关视觉证据。
|
||||
2. 主控保存一份简洁任务账本:当前合同、文件 owner、已完成证据、失败原因、下一步。各 Agent 引用同一事实,不各写一份长计划。
|
||||
3. 先 `rg` 定位,再读函数和局部 diff;不重复读取巨大 JSON、整仓日志或全部历史文档。大数据留在文件里,由脚本给出数量、哈希、差异和异常样例。
|
||||
4. 图片、截图和录屏用实际文件查看。不能为了省 token 把视觉任务变成只读文件名或文字概括。
|
||||
5. 相同来源只研究一次,保存来源、日期和关键结论;只有数据变化、结论不足或任务扩大时重查。
|
||||
6. 子任务首次任务卡建议控制在约 500–1500 token,结果摘要约 300–800 token;这是上下文组织目标,不是截断必要证据的硬上限。
|
||||
7. 返回“改了什么、哪些文件、验证证据、剩余阻塞”即可。完整日志和截图放到文件,用路径引用;失败时返回相关片段,不整段倒入主控上下文。
|
||||
8. 哈希、解压、下载、构建、测试和统计交给确定性工具。不要为了运行一条已知命令另开一个 Agent。
|
||||
9. 不重复安排多个模型完整实现同一功能。关键风险做一次独立审查,普通修改由测试与 owner 检查。
|
||||
10. 默认不开 `max` / `ultra` 推理。提高模型或推理强度只针对具体难题,并记录升级理由。
|
||||
|
||||
## 6. 升级与停止条件
|
||||
|
||||
| 情况 | 处理 |
|
||||
| --- | --- |
|
||||
| Luna 遇到不明确需求、需要理解跨文件状态或开始猜测行为 | 交 Terra;若涉及核心状态/几何,直接交 Sol |
|
||||
| Terra 的任务超出明确合同,或同一实质失败在一次有针对性的修复后仍未解决 | 给 Sol 最小复现、尝试记录和失败证据,不继续盲试 |
|
||||
| Sol 遇到跨坐标层、性能来源不明或与现有交互语义冲突的问题 | 给 Astra 聚焦诊断任务;可以由 Astra 完成必要关键补丁,不限于口头审查 |
|
||||
| 已知一开始就是复杂的空间布局或高风险几何设计 | 直接用 Sol/Astra,不为了经过低模型梯级而增加一次无效尝试 |
|
||||
| 网络、凭证、工具缺失、原始资料不存在 | 不通过升模型解决环境问题;检查环境或记录缺口,继续独立工作 |
|
||||
| 失败来自缺失用户决定或无法识别的地名/视觉目标 | 提出具体问题,不让多个模型猜不同版本 |
|
||||
|
||||
默认允许 worker 一次有证据的修复;是否继续由主控根据新信息决定。超时或 token 消耗增加但没有新增证据时停止扩展,整理已知结果后重新分配,不能无限自递归。
|
||||
|
||||
## 7. 审查门槛
|
||||
|
||||
Astra 重点进入三个位置:最初高风险空间/几何合同、首次运动原型、最终关键风险验收;中间疑难问题按需增加。它不需要审阅每一处 CSS、每一段翻译或每个测试日志。
|
||||
|
||||
Sol 对各 worker 的交付负责。不能以“另一模型说通过”代替运行结果。任何模型都无权改变项目政治口径、降低性能门槛、把未知数据填成确定值,或自行部署和发版。
|
||||
|
||||
涉及政治几何和性能结论,使用与实现过程分离的检查证据;普通文档格式或固定映射使用针对性检查即可。视觉方向仍由已确认的用户要求和参考图决定,Astra 的审美判断不替代用户目标。
|
||||
|
||||
## 8. 单任务派发模板
|
||||
|
||||
```text
|
||||
任务:一个可独立验收的目标
|
||||
模型与推理:明确 model ID + reasoning effort
|
||||
必要约束:相关 rules 模块、政治口径、性能、共享组件、视觉参考
|
||||
输入:最少文件/符号、当前合同、已有验证结果和图片路径
|
||||
可写范围:明确文件;共享文件由指定 owner 修改
|
||||
验收:必须出现的行为、必要测试/渲染/性能证据
|
||||
停止/升级:哪些条件不能继续猜测或扩展
|
||||
返回:结果摘要、改动文件、检查结果、证据路径、遗留问题
|
||||
```
|
||||
|
||||
派发有意使用较小上下文时,必须把不可省略的约束写进任务卡。执行任务的 Agent 自行按规则加载相应模块,不因主控已读过就假定子 Agent 也知道。
|
||||
|
||||
## 9. 如何验证分配确实更高效
|
||||
|
||||
每个工作包记录模型、推理强度、输入/输出 token(环境可提供时)、耗时、工具耗时、修复轮数、一次验收是否通过和最终证据。若读不到准确 token,就标记未知,不用消息长度伪造 token 统计。
|
||||
|
||||
比较每个已验收任务的总消耗,包含主控、worker、审查和返工,而不是仅比较某次调用。先完成 3–5 个有代表性的工作包,再调整默认路由;这个数量是试点取样建议,不代表统计显著性或保证节省比例。
|
||||
|
||||
API 单价、订阅额度和任务级成本不是同一个指标。本计划不预先承诺固定节省百分比;模型可用性和实际费用以执行时配置为准。低成本模型导致反复修复时应提前升级;高能力模型能明显减少整体返工时可以直接采用。
|
||||
|
||||
## 10. 官方依据与执行边界
|
||||
|
||||
OpenAI Docs 将 Astra 定位为最具能力、适合困难端到端任务的模型,Sol 面向复杂专业工作,Terra 平衡能力与成本,Luna 面向成本敏感任务;它们并未规定 Planet 各文件应由哪个模型处理。本计划按任务风险给出项目级分工。
|
||||
|
||||
- [OpenAI 模型说明](https://developers.openai.com/api/docs/models)
|
||||
- [Codex 模型选择](https://learn.chatgpt.com/docs/models)
|
||||
- [子 Agent 配置与模型选择](https://learn.chatgpt.com/docs/agent-configuration/subagents)
|
||||
|
||||
只在用户已授权相应开发与委派、且运行环境支持时执行这套拆工。仅制定本计划不会切换当前会话模型、修改默认配置或自动启动 Agent。执行时若某个精确模型不可用,先说明并选择可用替代,不把替代模型冒充原分配。
|
||||
|
||||
## 11. 本轮实际执行账本
|
||||
|
||||
| 工作包 | 实际分配 | 结果与升级 |
|
||||
| --- | --- | --- |
|
||||
| 地理政策、源证据、几何构建与回退 | Sol `high`,`geography_policy` | 版本化规则与校验已实现;如实保留未核验几何,未用模型猜测填补 |
|
||||
| 空间连接线、卡片生命周期与主入口 | Sol `high`,`spatial_callouts` | 完成原型;六卡避让多次渲染仍有冲突后,交主控接手约束求解,worker 停止同文件写入 |
|
||||
| 同机性能基线、对照工具、公共滚动条 | Terra `medium`,`performance_baseline` | 保存桌面/模拟手机各 5 次基线;共享滚动容器修复和对照工具已完成 |
|
||||
| 复杂避让、全球目录/API、视觉参数、集成验证 | 主会话 | 主控整合并审查;性能脚本的场景同步多次失败后也由主控接手修复。独占布局文件,浏览器/构建按顺序执行,避免测试相互干扰 |
|
||||
| Luna 辅助 | 未启用 | 没有值得额外传递上下文的独立机械任务,不为凑齐模型而派工 |
|
||||
|
||||
本轮精确 token 与费用不可得,记为未知;不报告未经测量的节省比例。空间避让存在返工,不能把功能实现或子任务数等同于效率提升。性能问题应在首次稳定复现后采集 CPU/布局证据,再修改热点;本轮最终定位到共享 SVG 长度读取造成的同步布局,单纯调整避让参数不足以解释该成本。性能、运动观感及地理缺项以功能计划的验收状态为准。
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
**状态**:待实施
|
||||
**创建日期**:2026-05-12
|
||||
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.claude/commands/docs.md`,让以后写文档时自动按受众归档。
|
||||
**校正日期**:2026-06-26,控制台深链已从旧 tab 查询口径更新为当前 `?section=` 口径。
|
||||
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.codex/skills/docs/SKILL.md`,让以后写文档时自动按受众归档。
|
||||
|
||||
## 背景
|
||||
|
||||
@@ -31,12 +32,12 @@
|
||||
3. **登录与找回密码** — 登录页、忘记密码流程
|
||||
4. **账户设置** — 修改密码、修改邮箱(需重新验证)、查看权限组、登出
|
||||
5. **Console 总览** — 左侧菜单结构、各路由用途
|
||||
6. **配置数据采集器** — `/collection-management?tab=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?tab=providers`:默认 provider、模型、Base URL、API Key、本地代理;工具 tab(WebSearch、OCR)
|
||||
6. **配置数据采集器** — `/collection-management?section=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?section=integrations`:默认 provider、模型、Base URL、API Key、本地代理;工具 section(WebSearch、OCR)位于 `/ai?section=tools`
|
||||
8. **系统设置** — `/settings` 其他子 tab(系统设置、电视直播源、SMTP 邮件)
|
||||
9. **用户管理(管理员)** — `/users`:创建、删除、改角色、Gatekeeper 权限组
|
||||
10. **数据探索** — `/datasources`、`/data`、`/bgp`、`/alerts/*`
|
||||
11. **AI 测试台** — `/ai?tab=playground`
|
||||
11. **AI 测试台** — `/ai?section=playground`
|
||||
12. **Earth 公开页面** — 现 manual.md 的 Earth 章节原样保留(图层、图例、搜索、位置候选、设置、视角、动捕、巡航、移动端)
|
||||
13. **Docs 文档站** — 当前 Docs 章节保留(权限组说明)
|
||||
|
||||
@@ -48,7 +49,7 @@
|
||||
|
||||
- 打开管理员给你的 URL
|
||||
- 注册账号 + 邮箱验证
|
||||
- 登录后第一次做什么(建议先到 `/collection-management?tab=collector_credentials` 配一个 collector,再到 `/ai` 配模型)
|
||||
- 登录后第一次做什么(建议先到 `/collection-management?section=collector_credentials` 配一个 collector,再到 `/ai?section=integrations` 配模型)
|
||||
- 看 Earth
|
||||
|
||||
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节,**不**再单独出 `ops-quickstart.md`,避免新增维护点。
|
||||
@@ -79,7 +80,7 @@
|
||||
> - 新增客户可见 UI 流 → 同时更新 `manual.md` zh+en 与 `docs-content.ts`
|
||||
> - 新增 ops 命令或脚本 → 只更新 `ops-runbook.md` zh+en
|
||||
|
||||
## .claude/commands/docs.md 增量
|
||||
## `.codex/skills/docs/SKILL.md` 增量
|
||||
|
||||
在 "Step 2 — Decide Scope" 后插一段:
|
||||
|
||||
@@ -99,7 +100,7 @@
|
||||
- `docs/technical/zh/quickstart.md` & `en/quickstart.md` — 重写
|
||||
- `docs/technical/zh/ops-runbook.md` & `en/ops-runbook.md` *(新)*
|
||||
- `docs/documentation-coverage-rules.md` — 加受众分层段
|
||||
- `.claude/commands/docs.md` — 加 Document Audience Routing 段
|
||||
- `.codex/skills/docs/SKILL.md` — 加 Document Audience Routing 段
|
||||
- `frontend/src/pages/Docs/docs-content.ts` — 注册 `ops-runbook` 到 `DOCS_METADATA`(`docs_admin` 组)
|
||||
|
||||
## 依赖
|
||||
|
||||
@@ -1,715 +0,0 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求:
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备:
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
|
||||
|
||||
这意味着:
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计:
|
||||
|
||||
### 1. 世界层(world-space celestial layer)
|
||||
|
||||
用于放置:
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层(earth-fixed layer)
|
||||
|
||||
继续保持当前结构:
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下,不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造,会同时影响:
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做:
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐:
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因:
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
|
||||
|
||||
不作为主选的库:
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因:
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合:
|
||||
|
||||
- 天球:内翻球壳 + 星图纹理
|
||||
- 太阳:`THREE.Sprite`
|
||||
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光:`THREE.DirectionalLight`
|
||||
|
||||
参考:
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求:
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰,适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块:
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口:
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数:
|
||||
|
||||
- 半径:`600 ~ 900`
|
||||
- 材质:`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2:太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数:
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算,建议:
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3:太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为:
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿,避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求:
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4:月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层,不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增:
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
1. `init()` 中:
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中:
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中:
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中:
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后,不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含:
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为:
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议:
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点,快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流,否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”,再求“月相精致”
|
||||
|
||||
月相属于第二步优化,不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth,最稳妥的方案是:
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。
|
||||
@@ -1,106 +0,0 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
|
||||
## Satellite Footprint Follow-Up Items
|
||||
|
||||
从技术文档迁出的 footprint 后续项,作为卫星覆盖能力的计划 backlog:
|
||||
|
||||
1. 为 `iridium-next` 新建独立 footprint adapter。
|
||||
2. 在 UI 上补一个只读提示,让用户知道当前卫星是否支持 footprint。
|
||||
3. 如果未来拿到 GEO beam contour / operator metadata,再为 GEO 开 operator-specific footprint。
|
||||
@@ -0,0 +1,237 @@
|
||||
# Earth 地理资源与空间卡片展示实施计划
|
||||
|
||||
日期:2026-09-21。状态:本地可运行原型。贴图已替换并归档旧图;固定规则、部分核验几何、全球参与方目录、空间卡片、共享移动滚动条与轻量材质调整已实现。完整政治几何、逐段海岸对齐、视觉认可及性能门槛尚未全部验收,不代表可发布。
|
||||
|
||||
本计划以已确认的展示方向为准,替代算力专题旧方案中的固定右侧栏、横贯底部双图和隐藏其他面板的布局。空间卡片与算力目录已有本地可运行原型,但仍需完成最终 QA;算力数据采集、指标证据和长期模型设计继续参见[算力与资源态势计划](compute-resource-intelligence-plan.md)。
|
||||
|
||||
## 1. 目标与工作边界
|
||||
|
||||
地球是空间信息的主体。打开区域对比后,多个区域分别引出连接线,牵引周围的浮动信息卡。地球转动时,地理锚点跟随球体,卡片保持正向可读,通过平滑、有阻尼的运动调整位置并互相避让,形成信息从地球四周延伸出来的感觉。
|
||||
|
||||
地球质感参考 UE 3D 截图在易拉宝与 PPT 中的表现:球面有层次、边缘与明暗交界柔和,背景退后,数据与球体构成同一空间。保持当前操作流畅度是硬验收条件。
|
||||
|
||||
已授权实施范围包括资源、空间卡片、共享 UI 与性能验证。继续使用现有 Three.js 引擎和资源加载机制。资源版本与实际转换参数以[地球资源说明](../../resources/earth/README.md)和清单为准。
|
||||
|
||||
### 当前实现与未完成项
|
||||
|
||||
本轮已实现卡片运动、配置控制、内容承载及中性地球材质调整。配置窗口与卡片显示开关已分离,关闭配置不再停用卡片。卡片默认常显基础信息:36px 主数值、年份/指标/口径说明、设施样本与 Epoch 来源、趋势首末年份及单位、证据入口和区域成员数;长列表仅在需要时展开完整图表。桌面卡片约宽 268px,高度随内容自适应,国家约 214px、区域约 223px;展开卡片支持用户调整至最高 480px,位置/尺寸可保存并恢复自动布局。手机保留同等摘要信息。高清 overlay 使用白色 MeshBasicMaterial、opacity 1、toneMapped:false;r128/r160 按 active API 使用对应 sRGB 读写路径。没有新增 GPU draw pass 或高分辨率纹理,但新增并预加载 45 个本地 SVG 旗帜资源。
|
||||
|
||||
旧版底边换侧的连续跳动已定位并由路径规划与稳定换侧策略处理。新实现由 `spatial-callout-controller.js` 管理,布局采用事件驱动并保留有效旧矩形;投影连线持续跟随地球,另一侧候选需持续 1800ms 才换侧,12 秒阅读间隔用于抑制不必要的替换。悬停、焦点和详情阅读会锁定卡片,用户拖动复用 `hud-drag.js`,现有 controls 也消费该共享模块。多卡布局通过实际空闲矩形安排最终位置,优先避开已有功能窗;局部 UI QA 已完成,全球性能与最终发布回归仍按后续门槛验证。下方性能数字和旧尺寸的回归记录属于此前原型,不能直接作为当前大卡片的性能验收结论。
|
||||
|
||||
- 底图默认“原始底图”(菜单首项),显示菜单可切换并保存;Natural Earth I 保留,NASA 21K 仅作源归档。运行时统一 8K,按需只保留一张 GPU 纹理;成功后释放旧图,失败保留旧图,过期响应不能覆盖当前图源,不新增渲染层。星空逐字节恢复 commit `83a10a6c`,不再校准密度和颜色。高清 overlay 使用白色 MeshBasicMaterial;r128/r160 按 active API 选择对应 sRGB 读写。没有新增长驻高分辨率纹理、后处理或 GPU draw pass;算力卡另新增并预加载 45 个本地 SVG 旗帜资源。Natural Earth 原图偏浅,不把不存在的细节写成恢复结果。
|
||||
- 地理政策由版本化清单贯穿规范编码、后台聚合、高精几何及低精回退。中国、台湾、港澳、科索沃等已有回归;已核验几何及济州岛补全见资源说明。高精技术就绪与政策可发布分开:`high_precision_ready=true`,`policy_publishable=false`。东沙、中沙、曾母暗沙点注记及经核验海上主张线仍有缺项。
|
||||
- 规范目录覆盖 45 个国家编码,界面分组为中国、美国、欧洲、不含中国的亚太和其他;分组不当作统计合计。当前空间原型最多同时显示 6 张,后端最多接受 12 个参与方。没有历史观测的地区返回 `null` 历史与覆盖信息;设施样本数永远不作为计算出的国家总量。
|
||||
- 桌面卡片复用共享 HUD 和连接线,保留稳定 DOM 身份;选择器和来源窗使用共享拖动。背面卡片保留内容并隐去连接线。旋转通过原动画循环更新,投影连线持续更新;布局按事件分帧处理并保留有效旧矩形。`main.js` 的 `isDragging`、`activeTouchPoints`、`hasActiveGlobeInertia` 接入运动模式:手动/惯性快速模式以 72ms/12px 为响应预算,在当前帧完成受限布局,停止后一次 settle;slow auto 保留 1.8 秒换侧和 12 秒阅读保持,用户拖动放置优先。
|
||||
- 手机复用公共抽屉与深色滚动所有者,使用适合手机阅读的字号;目前是抽屉内紧凑卡片列表,尚未实现手机环绕空间卡片。桌面原型没有强制隐藏新闻、媒体或统计窗口。
|
||||
- 2/4/6 卡片、背面投影、共享窗口避让、饱和纵列与共享连线动画已有 11 项单元测试。浏览器已检查 1366×768、1920×1080、390×844 及 125%/150% 对应的有效 CSS 视口;单元测试不替代运动观感验收。
|
||||
|
||||
### 本轮空间卡片 UI 定向验收(2026-09-21)
|
||||
|
||||
本计划保留此前的历史验证记录;它们不代表当前地图源切换或全球 Earth 验收已经完成。当前实现说明只记录所需资源生命周期与 UI 约束,Browser/全球性能结论仍按后续门槛单独复核。
|
||||
|
||||
以上是本轮卡片 UI 与局部资产的定向证据。全球地理政策、逐段海岸对齐、完整 Earth 渲染性能、真实设备性能及发布级回归仍按本计划后续门槛单独验收。
|
||||
|
||||
## 2. 优先级与不可退让的要求
|
||||
|
||||
| 优先级 | 要求 | 执行原则 |
|
||||
| --- | --- | --- |
|
||||
| P0 | 固定政治展示规则 | 外部数据、更新、回退及 AI 均不能覆盖项目规则 |
|
||||
| P0 | 性能不退化 | 在同设备、同数据、同场景下验证;可复现的退化不能用视觉提升抵消 |
|
||||
| P1 | 地理对应一致 | 贴图定位、海岸线、区域归属、拾取和卡片锚点一致 |
|
||||
| P1 | 地球关联的空间卡片 | 使用现有连接线与展示机制扩展,不恢复固定大屏布局 |
|
||||
| P1 | 共享 UI 与移动端一致 | 共用字体、标题栏、按钮、滚动容器及窗口行为 |
|
||||
| P2 | 卡片内图表和内容完善 | 先验收空间关系与运动,再充实图表和解释 |
|
||||
|
||||
## 3. 固定政治展示规则
|
||||
|
||||
项目展示采用固定中国视角:中国相关领土及争议区域按项目既定中国视角处理,台湾按中国处理;科索沃按塞尔维亚处理。该口径先于数据源选择,不由数据源默认分类、模型输出或换源结果决定。
|
||||
|
||||
实施时建立单一、版本化的规则清单与几何修正包。名称映射只是其中一部分,必须覆盖国家编码、Polygon/MultiPolygon 归属、边界绘制、区域填色、鼠标拾取、高亮、统计及卡片归属。处理后的结果才可以成为运行资源;未经规则校验的新来源不得替换正式资源,旧资源回退也必须通过同一规则。
|
||||
|
||||
规则清单应包含:稳定规则 ID、目标区域的明确标识、目标归属、显示名称、几何来源、合并/裁切操作、版本及回归样例。不能只依赖模糊地名字符串替换。语音中未能确认的具体岛礁名称必须在修正清单中明确定位,不能猜测名字后直接移动多边形。
|
||||
|
||||
海岸线、陆地行政区域和海上主张线是不同几何层。项目口径在各层一致执行;海上主张线不能作为陆地面直接填充,也不能代替具体岛礁的几何和归属规则。
|
||||
|
||||
必须具备的失败测试:外部源恢复原始分类、别名变化、同区域不同编码、几何修改后拾取仍用旧数据、低精度回退覆盖规则。任何一项失败,都不发布该资源版本。
|
||||
|
||||
## 4. 已查明的现状
|
||||
|
||||
### 4.1 贴图与国界
|
||||
|
||||
- 原运行贴图为 `frontend/public/earth/assets/8k_earth_daymap.jpg`,8192×4096;代码直接加载 JPG,未读取该文件配套的地理定位元数据。
|
||||
- `country-boundaries.js` 使用独立 GeoJSON/PMTiles 几何绘线与拾取;高精度是一个选择状态,存在退回低精度的路径,按钮名称不等于实际正在使用的文件。
|
||||
- 本机此前检查到 `natural-earth-v1`、`geojson-high-precision` 构建清单,以及独立 base、hover、claim 文件;该构建不能直接等同于完整应用了项目政治几何规则。实施前需要重新读取实际运行版本。
|
||||
- 已提供的两张问题截图中,一张出现海岸折线与影像陆地错位;另一张有可见岛屿但没有对应线框。截图只能证明现象,尚不能断言根因一定是 UV、投影、低精度回退或缺岛。
|
||||
- 精度检查必须同时检查“看得到轮廓”和“能选中对应区域”,二者不能各用一套不一致的几何。
|
||||
|
||||
### 4.2 World Monitor 对照
|
||||
|
||||
本机 World Monitor `463aec0f` 使用独立地表图片与 `countries.geojson`;经纬度归属由点落入多边形的计算决定,二维底图和自带的交互区域也分开加载。
|
||||
|
||||
本次文件统计:World Monitor 的中国、日本、韩国分别为 200、60、14 个坐标点;Planet 的低精度分别为 240、65、19;本机高精度文件分别为 14,118、6,952、3,060。点数不等于测绘准确度,但表明不能靠直接换用 World Monitor 的简化国家文件解决近景海岸问题。它的原始归属处理也不能代替本项目规则。
|
||||
|
||||
### 4.3 已有共享机制
|
||||
|
||||
| 入口 | 当前作用 | 计划复用方式 |
|
||||
| --- | --- | --- |
|
||||
| `frontend/public/earth/js/callout-connector.js` | 连接线、元素锚点与路径 | 扩展为多个稳定区域锚点与卡片配对 |
|
||||
| `frontend/public/earth/js/presentation-controller.js` | 展示生命周期与连接器更新 | 在单项展示的基础上设计受控的多项展示,不破坏现有模式 |
|
||||
| `frontend/public/earth/js/cruise-sequencer.js` | 巡航时序 | 保留当前巡航行为,区域对比明确自己的生命周期 |
|
||||
| `frontend/public/earth/js/info-card.js` | 公用详情与移动详情 | 抽出/复用内容渲染能力,不复制成另一套卡片组件 |
|
||||
| `frontend/public/earth/js/hud-panels.js`、`controls.js` | 面板收起、显示、拖动与约束 | 工具窗继续遵循这些规则;明确 createHUDPanel 本身不包办所有拖动/缩放 |
|
||||
| `frontend/public/earth/css/hud.css`、`info-panel.css` | 公用外观、字号、滚动与移动抽屉 | 去掉专题私设的大字号、独立控件和白色滚动条 |
|
||||
|
||||
## 5. 地理资源:先采用配套贴图的简单方案
|
||||
|
||||
### 5.1 选择与准备
|
||||
|
||||
默认运行底图为“原始底图”(菜单首项),显示菜单可切换并保存底图源;Natural Earth I 保留可用。NASA 21K 仅保留为源归档,运行包统一使用 8192×4096 的单张 GPU 贴图。官方源版本与许可信息仍以资源清单为准,不能把归档源误写成运行资源。
|
||||
|
||||
采用不带政治标注的物理地表图。按需只保留一张运行 GPU 贴图;新源成功加载后释放旧图,失败保留旧图,过期响应不得覆盖当前图源。源包保留 TIFF、世界文件和说明,运行资源统一为 8192×4096 的 Web 图。转换只做尺寸与格式适配,不用生成式图像重绘海岸、岛屿或国界,不随意裁切、旋转或拉伸经纬范围;底图切换不新增渲染层。
|
||||
|
||||
下载的原始大文件保存在本地 `downloads/earth/`,不纳入前端发布包。可追溯清单、配准信息与旧贴图归档放在 `resources/earth/`;运行图放在现有 `frontend/public/earth/assets/`。旧图必须先按字节完整归档并核对 SHA256,之后才能替换运行路径。
|
||||
|
||||
### 5.2 对齐验证
|
||||
|
||||
1. 读取 TIFF/TFW 的像元大小、原点、行列方向、地理范围和 CRS;同时记录 JPEG 派生后的像元中心定位,避免半像素偏移。
|
||||
2. 明确运行中加载的贴图、边界文件、资源版本与是否发生回退。
|
||||
3. 使用同一投影和球面经纬转换叠加成套矢量,检查赤道、零经线、日期变更线和高纬区域。
|
||||
4. 在截图对应海岸、近海岛屿、台湾周边和其他不同经度的典型海岸逐点验证,区分整体偏移、局部概化和缺失几何。
|
||||
5. 政治修正后的同一份区域几何用于标注、高亮和拾取。所有需要烘焙的线也从这份数据导出,不能在图片和矢量上分别手工描线。
|
||||
6. 小岛、岛礁或海岸不满足要求时,先补充对应区域数据并重新配准;仅当这条简化路线确实不够用时,再考虑局部细节或分级资源。
|
||||
|
||||
成套资源声明配准不等于本项目已经逐像素验收。不能承诺卫星影像每一处潮线都与概化海岸线完全一致;验收需保存具体偏差和缺失清单,不能靠强行拉伸边界迎合图片。
|
||||
|
||||
### 5.3 常态渲染与交互
|
||||
|
||||
优先预合成常态显示所需的边界,以减少额外线段绘制。需要独立国界开关时,评估复用现有覆盖层或同定位的透明边界层;不默认再常驻一张 8K 纹理。矢量数据继续承担区域判定和选中高亮,不在每帧重新解释整份 GeoJSON。
|
||||
|
||||
本阶段不以替换 Three.js、引入新 GIS 平台或建设全球在线瓦片服务作为前置条件。
|
||||
|
||||
## 6. 整体视觉方向
|
||||
|
||||
参照资产:`Planet-Handoff-20260916` 中的完整易拉宝效果稿和卡片展示 PPT,尤其 PPT 内来自 UE 3D 的原始星球画面。海报中的线点含概念叠加,不能拿来充当精确地理数据。
|
||||
|
||||
- 地球保持正圆与正确地理位置,视觉上成为主体;不把纹理精细度等同于画面品质。
|
||||
- 球面采用柔和、连续的明暗层次,边缘有薄而自然的大气过渡;地理细节仍然可读。
|
||||
- 降低背景星点对比与密度造成的干扰,让背景退后;变化以参考图与实际渲染对照决定。
|
||||
- 数据节点与连接线有轻重层级,避免强霓虹、油亮塑料感、浓重太空雾和粗硬轮廓。
|
||||
- 优先在现有材质、光照与透明度范围内调整;新增后处理、辉光或高分辨率透明层须先证明成本。
|
||||
- 用同一位置、距离、光照条件对比前后效果,近景与远景都检查边缘、深度冲突和细节。
|
||||
|
||||
## 7. 区域卡片:从地球四周延伸出来
|
||||
|
||||
### 7.1 行为模型
|
||||
|
||||
一个区域对应稳定的地理锚点、连接线和卡片 ID。锚点随球体旋转;卡片处在屏幕空间,保持正向、稳定大小和内容身份。转动时变化的是屏幕位置与线的路径,不重建卡片、不交换身份、不清空正在阅读的内容。
|
||||
|
||||
卡片根据区域投影分布到球体周围,保留疏密和空间关系,不固定平均分成八个方向,不横排占满屏幕。使用有限的避让和有阻尼的缓动,最终能够安定;不持续抖动或为了追踪锚点逐帧跳位。
|
||||
|
||||
### 7.2 布局与生命周期
|
||||
|
||||
- 沿用现有连线的起点/终点接口和动画语言,先验证 2 张、4 张、6 张卡片;6 张是原型测试档位,不是已确定的产品上限。
|
||||
- 保持地域锚点与区域绑定;算法可以移动卡片,但不能移动真实地理坐标来换取好看的布局。
|
||||
- 优先避让其他卡片和已有工具窗,不自动隐藏新闻、直播、统计、图层面板。
|
||||
- 对卡片换侧设置稳定区间,避免锚点越过某条中心线就立即左右互换。
|
||||
- 背面区域不画成穿过地球的前景锚点。建议保留卡片身份,用边缘提示表达背面方位;完整隐藏、收束或边缘提示的最终表现用短动画原型比较后决定。
|
||||
- 用户正在操作的卡片短暂锁定其屏幕位置;是否允许长期固定/拖动偏移,先以现有交互能力做原型验证,不增加未经确认的复杂模式。
|
||||
- 增删区域、切换年份、退出对比、隐藏相关图层和请求失败都要有明确的清理与恢复逻辑;旧响应不能复活已关闭卡片。
|
||||
- 区域名称锚点用于对比展示;不能把区域中心点画成设施位置。
|
||||
|
||||
### 7.3 卡片内容与参与方
|
||||
|
||||
先验证空间展示,再完善图表。首轮卡片可以只包含名称、一个有来源的关键指标、参考期及来源入口;固定侧栏和横贯底部双图的旧设计不再继续扩展,普通浮窗加分页的方案也不是最终展示方向。
|
||||
|
||||
参与方必须支持全球扩展,不能在接口和 UI 中永久硬编码成中美两项。日本、韩国等可以作为独立对象;亚太、欧洲可用于选择菜单的分组或区域筛选。具体默认组合与区域汇总口径还需在完整 UI 原型中确定,不把早期讨论的菜单建议当成已确认设计。组合区域与成员国不能重复计入同一分母。
|
||||
|
||||
缺少同口径历史数据的参与方仍可展示公开设施覆盖及明确的缺失状态,不伪造总量、得分或排名。全国总算力、公开 AI 集群样本、TOP500 FP64 和创新指数继续分开。2026 年未核验值不得加入正式历史。A4 AI 解读不阻塞空间展示验收。
|
||||
|
||||
## 8. 共享窗口与移动端
|
||||
|
||||
设置、来源、图层等操作窗口继续复用已有标题栏、按钮、拖动和按能力启用的缩放逻辑。不得用单独字号和固定网格重新构建一套视觉体系。数据更新和语言切换应保留窗口位置、尺寸、选中对象和滚动进度。
|
||||
|
||||
移动端复用原底部抽屉与详情展示。区域卡片数量受可用空间控制,保证地球与主要交互可见;可收束为可切换的关联卡片,但具体排列需要实机原型确认。不能把桌面全部卡片原尺寸堆到手机上。
|
||||
|
||||
白色滚动条问题从 `.earth-mobile-drawer-content` 及实际滚动所有者核对,统一复用 Earth 的深色滚动条样式与滚动容器。覆盖图层、搜索、态势、算力、新闻和设置页;底部导航保持固定,避免嵌套重复滚动,不仅对算力页补一条局部 CSS。
|
||||
|
||||
## 9. 性能基线与硬验收
|
||||
|
||||
### 9.1 基线
|
||||
|
||||
资源替换前的画面可通过归档旧贴图复现。保存设备、浏览器、GPU、视口、DPR、图层开关、数据量、相机路径与资源版本。在相同条件下重复测试,至少记录 5 次稳定运行,并区分冷启动、热启动和稳定旋转。
|
||||
|
||||
观察:首个可操作画面时间、首个有效地球画面时间、帧时间 p50/p95/p99、长帧/长任务、交互延迟、JS 堆与纹理内存估算、draw calls、纹理/对象数、网络字节及请求次数。只比较平均 FPS 不足以证明没有卡顿。
|
||||
|
||||
### 9.2 实现预算
|
||||
|
||||
- 运行贴图先维持 8192×4096。相同格式下纹理像素量不增加;RGBA8 粗估基础约 128 MiB,完整 mipmap 约 171 MiB,不能把 JPEG 文件变小当成显存同比下降。
|
||||
- 保持已有 mesh、图标实例与纹理缓存;旋转不重新解码图片或构造国家几何。
|
||||
- 地理投影和轻量位置更新与渲染帧同步;碰撞布局只在锚点明显变化、卡片变化或窗口尺寸变化时重算,不让每张卡片各起一套动画循环。
|
||||
- 卡片 DOM 保持稳定,图表仅在数据/尺寸变化时更新;避免每帧测量布局后又写布局造成同步回流。
|
||||
- 网络、缓存、监听器、纹理及卡片数量均有边界,关闭/重开多轮不得持续增长。
|
||||
- 默认不增加新的高分辨率常驻覆盖层、昂贵后处理或全部区域的最高精度常驻数据。
|
||||
|
||||
### 9.3 判定
|
||||
|
||||
同条件下不能出现可复现的流畅度、加载、交互或资源占用退化。测量噪声必须记录并重复验证,不能事先允许固定比例退化来降低要求。超出预算先调整效果、卡片密度或实现,再复测。桌面与移动分别验收;没有真实手机数据时只能说明模拟检查通过,不能宣布手机性能达标。
|
||||
|
||||
### 9.4 内容卡扩容前的原型验证记录(2026-09-21)
|
||||
|
||||
同机 Headless Edge、桌面 1440×900 / 手机尺寸模拟 390×844、DPR 1、自动旋转,每种尺寸 5 次;预热 1 秒、帧时间采样 2 秒。该短采样用于定位,不能代替真实设备长时交互验收。原始基线标识 `2026-09-20T18-30-56-184Z`,最新对照标识 `2026-09-20T21-07-40-495Z`(UTC);完整 JSON 留在本机临时验证目录 `planet-earth-performance-baseline`。
|
||||
|
||||
| 项目 | 原基线 | 本轮结果 | 判断 |
|
||||
| --- | --- | --- | --- |
|
||||
| 桌面无卡 p95 帧时间,5 次中位数 | 7.09 ms | 7.10 ms | 接近;不能据此宣称完整性能通过 |
|
||||
| 手机尺寸模拟无卡 p95 | 7.00 ms | 7.20 ms | 未做真实手机测量 |
|
||||
| 桌面 2 / 4 / 6 卡 p95,5 次中位数 | 无同场景旧实现对照 | 7.10 / 7.10 / 14.00 ms | 六卡比本轮无卡场景有明确成本,仍需优化 |
|
||||
| 六卡采样窗口长任务 | — | 5 次均为 0 | 没有 50 ms 长任务不等于没有掉帧 |
|
||||
| 桌面 DOMContentLoaded 中位数 | 1.262 s | 1.627 s | 已移除 Google Fonts 的约 10 秒阻塞;仍不宣称加载性能与基线完全持平 |
|
||||
| 桌面基础 JS 堆中位数 | 约 50.2 MiB | 约 53.2 MiB | 不是显存;未替代纹理/绘制统计 |
|
||||
|
||||
每轮另做 5 次专题开关并强制 GC 检查,关闭后的保留堆约 25 MB,首末差约 0.28–0.33 MB;这不构成长期无泄漏证明。浏览器单独核对关闭后的卡片悬停/焦点监听器为 0,来源窗与拖出容器的选择器均能关闭。
|
||||
|
||||
最终 Bun 构建、前端规则检查、文档一致性检查与 `git diff --check` 通过。既有前端 smoke(使用其 API mocks)通过 321 项路由/交互检查,包括公共页面、鉴权、管理导航、移动布局和管理页缩放;Earth 空间卡片另用真实本地接口检查,不能混同两种证据。
|
||||
|
||||
布局求解已分帧执行、减少重复候选评分、优先前景区域,在合格布局找到后停止重复试排;工具栏按可见按钮而非透明容器避让。CPU 采样发现共享 SVG 的 `getTotalLength()` 同步布局热点,已改为按已知折线点计算长度,保留原绘线动画。11 项布局/投影/连线测试通过。1366×768 浏览器中 4/6 卡、旋转后、稳定后及工具栏展开时未发现与卡片、选择器或现有 HUD 的矩形交叠;1920×1080、1093×614、911×512 同样完成六卡边界检查,后两项是浏览器缩放的等效 CSS 视口,不冒充真实设备缩放测试。46 项相关后端测试通过,高精与回退的台湾、藏南、阿克赛钦、科索沃、济州岛浏览器拾取均符合规则。官方几何缺项、完整海岸对齐、六卡性能与真实手机仍未通过最终验收。
|
||||
|
||||
## 10. 分阶段实施与交付门槛
|
||||
|
||||
开发模型分工、文件所有权、并行限制和上下文控制按[开发 Agent 模型分配计划](development-agent-model-allocation-plan.md)执行:Sol 主控与核心实现,Terra 承担明确子任务,Luna 处理有边界的辅助工作,Astra 负责高风险设计及疑难审查。模型分工不改变本计划的政治规则、视觉方向和性能验收门槛。
|
||||
|
||||
| 阶段 | 工作 | 必须交付的证据 |
|
||||
| --- | --- | --- |
|
||||
| S0 资源准备 | 新贴图、配准信息、旧图归档、来源清单 | 新旧尺寸/哈希、解码与球面渲染检查;本轮范围 |
|
||||
| S1 规则与配准 | 固定政治清单、实际提供者诊断、配套边界对齐 | 规则测试、截图问题点对照、回退测试;不能只报高精度开关已打开 |
|
||||
| S2 空间运动原型 | 复用连接器,多区域锚点、卡片稳定身份与避让 | 转动/缩放/换侧/背面/退出短录屏,2/4/6 卡档位及性能差值 |
|
||||
| S3 视觉与组件 | 柔光质感、背景层次、共享字体与移动滚动 | UE/PPT/海报参照对照、多个视角、桌面/手机及缩放截图 |
|
||||
| S4 内容联动 | 多参与方、真实指标与来源、时间/设施联动 | 可操作路径、缺失/错误状态、来源与口径验证 |
|
||||
| S5 回归与发布准备 | 性能复测、旧图层/巡航/工具窗回归、同步文档 | 完整验收记录;需要发布时另走仓库发布流程 |
|
||||
|
||||
S1 未通过,不能把新图片称为边界问题已解决。S2 的空间交互未确认,不能直接投入完整图表和全部国家卡片。性能或政治规则任何一项未通过,后续阶段不能据此宣布完成。
|
||||
|
||||
## 11. 验收清单
|
||||
|
||||
- 政治:标签、几何、填色、拾取、统计、卡片与回退源口径一致;科索沃等明确规则有专门用例。
|
||||
- 地理:截图所指海岸和小岛、台湾周边、日期变更线、高纬度;记录偏差而非只肉眼说“差不多”。
|
||||
- 视觉:球体不变形,边缘柔和但地理清楚,卡片从区域延伸且不压满地球;远距无闪烁和黑块。
|
||||
- 交互:转动、缩放、选中、背面、卡片增删、触摸、键盘、语言切换、退出、窗口大小变化及旧响应竞态。
|
||||
- 共存:新闻、媒体、统计、图层及原详情卡保持可用;不通过强制隐藏制造布局空间。
|
||||
- 移动:公共抽屉无白色原生滚动条,底部导航固定,只有预期内容区滚动,文字和操作不被截断。
|
||||
- 响应式:1366×768、1920×1080、约 390×844,以及 125%/150% 缩放;模拟与实机结果分别记录。
|
||||
- 回归:公共页面、鉴权、管理路由、导航/搜索/标签页、既有 BGP/新闻/动捕巡航与旧图层。
|
||||
- 性能:与归档基线同条件比较,保存帧时间、内存、网络和交互证据;循环打开关闭验证资源释放。
|
||||
- 文档:更新实际变更的中英文手册、Earth context、图层规则和资源清单;计划与已实现行为不能混写。
|
||||
|
||||
## 12. 来源、恢复与完成定义
|
||||
|
||||
- [Natural Earth 栅格与矢量配准说明](https://www.naturalearthdata.com/downloads/10m-raster-data/)
|
||||
- [Natural Earth I 资源页面](https://www.naturalearthdata.com/downloads/10m-raster-data/10m-natural-earth-1/)
|
||||
- [Natural Earth 使用条款](https://www.naturalearthdata.com/about/terms-of-use/)
|
||||
- [现有 Presentation 解耦计划](earth-presentation-decoupled-architecture-plan.md)
|
||||
- [原高精度边界计划及历史说明](earth-high-precision-boundary-tiles-plan.md)
|
||||
- [地球前端上下文](../technical/zh/earth-frontend-context.md)
|
||||
- [渲染层次规范](../technical/zh/earth-render-layer-order.md)
|
||||
|
||||
旧贴图恢复路径和 SHA256 见资源清单。先核对归档哈希,再将归档副本复制回原运行资源路径并重建预览;不以回退图片为由撤销其他用户改动。新资源和旧归档均不得无记录覆盖。
|
||||
|
||||
最终完成意味着:展示规则固定且经验证,地图与区域选择对应,多个卡片确实由地球锚点牵引并稳定运动,操作组件和手机滚动保持一致,实际性能不低于基线。单张效果图、构建通过或“高精度”名称都不能单独证明完成。
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## 当前状态
|
||||
|
||||
最新实现继续保留分桶 `Points` 与全局数据范围,并已接通 `vessels` 全局实时订阅:后端读取确认状态后按 MMSI 拆包推送,前端原位修改缓冲;快照用于首次加载、重连和删除校准。旧快照不能覆盖较新的更新或删除,旋转与缩放仍不触发视口请求。下方 Sprite 问题分析和阶段方案保留为历史背景,当前约束以[地球前端上下文](../technical/zh/earth-frontend-context.md)为准。
|
||||
|
||||
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad,而是更稳的分桶 `THREE.Points` 方案:
|
||||
|
||||
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。
|
||||
|
||||