Compare commits

...

22 Commits

Author SHA1 Message Date
rayd1o
dd41588527 release: bump version to 0.76.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 02:29:49 +08:00
rayd1o
45604a1ebb release: bump version to 0.76.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-10-01 01:25:24 +08:00
rayd1o
06c122bade fix: use whole-anchor occlusion for BGP icons
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:59:21 +08:00
rayd1o
96e9b15fda fix: keep satellite selection glow whole at limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 22:21:22 +08:00
rayd1o
a8713e58fd fix: preserve BGP event halo at globe limb
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-30 18:23:49 +08:00
rayd1o
05377bf5c6 release: bump version to 0.76.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-30 17:49:29 +08:00
rayd1o
67548d8278 release: bump version to 0.75.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-29 15:40:32 +08:00
rayd1o
4b8feeb87e release: bump version to 0.75.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-27 09:30:06 +08:00
rayd1o
a2025dbfe0 release: bump version to 0.75.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-25 02:03:53 +08:00
rayd1o
003b316d42 release: bump version to 0.75.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-25 00:16:30 +08:00
rayd1o
0c40764c58 feat: configure compute comparisons and restore adaptive globe dragging
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-24 22:59:55 +08:00
rayd1o
5bfc790dec release: bump version to 0.75.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-23 21:20:12 +08:00
linkong
60eefb19c9 release: bump version to 0.74.7
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-17 17:30:54 +08:00
rayd1o
83a10a6c34 release: bump version to 0.74.6
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-16 21:05:40 +08:00
rayd1o
cee1996809 release: bump version to 0.74.5
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-09-13 14:04:34 +08:00
rayd1o
58671e7bc3 release: bump version to 0.74.4
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-09-13 10:27:00 +08:00
rayd1o
a54fcdbeed release: bump version to 0.74.3
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / delivery (pull_request) Has been cancelled
2026-09-13 02:17:55 +08:00
rayd1o
1dd2921674 release: bump version to 0.74.2
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
release / images (push) Has been cancelled
ci / delivery (push) Has been cancelled
2026-07-01 23:40:00 +08:00
linkong
d30f7d08c5 release: bump version to 0.74.1
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-30 18:54:34 +08:00
linkong
5bdb55f3f1 release: bump version to 0.74.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-30 13:52:52 +08:00
linkong
fbecf30513 release: bump version to 0.73.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-29 17:04:05 +08:00
linkong
19d5ac0fee release: bump version to 0.72.0
Some checks failed
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / delivery (push) Has been cancelled
release / images (push) Has been cancelled
2026-06-29 14:05:06 +08:00
524 changed files with 41715 additions and 9230 deletions

View File

@@ -1,139 +0,0 @@
---
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
---
# /cleanup — 垃圾代码审查与清理
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
## 检查范围
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
## 节省上下文规则
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
```bash
git diff --name-only HEAD
git diff --unified=0 HEAD -- <path>
git diff --check
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
```
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
## 审查清单
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
### 1. 重复逻辑 (Duplicate Logic)
- 完全相同或高度相似的代码块在多处出现
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
### 2. Magic Numbers / Magic Strings
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
### 3. 命名问题
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
- 命名与实际用途不符
- 同一概念在不同地方用不同名字表达
### 4. 死代码 / 无效代码
- 注释掉的旧代码块(3行以上)
- 声明后从未使用的变量/参数/导入
- 永远不会执行的条件分支
### 5. 代码风格问题
- 尾部空白字符(trailing whitespace)
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
- 空行使用不一致(连续多个空行等)
### 6. 其他常见问题
- 私有辅助函数应被 export 但没有,导致调用方重复实现
- 类型/接口重复定义
- 过于冗长的条件表达式可以简化(不改逻辑)
## 执行步骤
### Step 1 — 获取待检查文件列表
```bash
# 无参数时:获取所有未提交修改
git diff HEAD --name-only
# 有参数时:用 $ARGUMENTS 过滤
```
### Step 2 — 逐文件阅读并分析
先从 focused diff 开始:
```bash
git diff --unified=0 HEAD -- <file>
```
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
### Step 3 — 报告问题清单
在修改前,先以列表形式输出所有发现的问题:
```
发现 N 个问题:
[文件] js/foo.js
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
[文件] js/bar.js
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
...
```
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
### Step 4 — 执行修复
对每个问题,使用 Edit 工具进行**最小化修改**:
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
- **命名问题**:重命名变量,更新所有使用处
- **死代码**:直接删除
- **尾部空白/风格**:修正
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
**修复原则:**
- 只改在审查清单中发现的问题,不做额外优化
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
- 改完后用 `grep` 验证旧的坏代码已消失
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
### Step 5 — 输出总结
```
清理完成:
修复了 N 个问题:
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
...
未修改的问题(需人工确认):
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
```
## 约束
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
- **禁止**添加新功能、新抽象、新参数
- **禁止**修改注释内容(只删除注释掉的死代码)
- **禁止**修改测试文件逻辑
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"

View File

@@ -1,104 +0,0 @@
---
description: Create or update repository documentation from current code changes
argument-hint: Optional: topic to document, or leave empty to infer from git diff
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
---
# /docs — Documentation Workflow
## Goal
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
## Repository Rules
Before deciding scope, check whether the repository has a documentation rules file:
```bash
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
```
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
## Workflow
### Step 1 — Understand The Change
```bash
git diff HEAD --stat
git diff HEAD --name-only
git log --oneline -10
rg --files docs
```
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
```bash
git diff HEAD -- <path>
rg -n "class |def |function |export |router|@router|interface |type " <path>
```
### Step 2 — Decide Scope
- Prefer updating an existing relevant document over creating a duplicate.
- Use one document for one coherent topic.
- Split documents only when the change crosses meaningful domains.
- Keep filenames lowercase and hyphenated.
- Apply the repository-specific rules file before writing.
#### Document Audience Routing (Planet)
In this repository, classify the action's performer before picking a target file:
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
### Step 3 — Write
Explain:
- Background/problem: what was wrong or missing before.
- Core design decisions and rationale.
- Operational or user-facing impact.
- Relevant code paths, only when useful for future maintainers.
Style:
- Follow the repository’s existing language and heading conventions.
- Use fenced code blocks with language tags.
- Prefer tables for comparisons or parameter lists.
- Keep snippets concise and relevant.
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
### Step 4 — Verify
- Read the completed docs once for clarity and stale statements.
- Verify referenced paths exist with `test -e` or `rg --files`.
- Run applicable checks from `docs/documentation-coverage-rules.md`.
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
### Step 5 — Report
Summarize changed docs and verification:
```md
Updated:
- path/to/doc.md — what changed
Verified:
- checks that passed
- checks that could not be run, if any
```
## Hard Constraints
- Do not leave placeholder docs.
- Do not duplicate bilingual files byte-for-byte.
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
- Keep docs maintainable and concise.

View File

@@ -1,93 +0,0 @@
---
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
argument-hint: 建议填写任务目标;若同时给出成功标准更好
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
---
# /goal-driven — 目标驱动执行模式
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
适用场景:
- 长周期实现任务
- 高复杂度工程任务
- 可被明确验收的研究、实现、迁移、验证类工作
不适用场景:
- 纯脑暴
- 无法定义成功标准的模糊任务
- 很小的一次性修改
## 输入要求
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
启动时先输出:
```md
Goal
- ...
Criteria for success
- ...
Plan
1. ...
2. ...
3. ...
Verification
- ...
```
## 执行规则
1. 先把任务固化为两个核心块:
- `Goal`
- `Criteria for success`
2. 成功标准必须尽量客观,可验证,可落地。
优先写成:
- 需要交付什么
- 需要通过哪些测试或验证
- 如何判断结果真的完成
3. 进入持续执行循环:
- 完成一个阶段
- 检查当前结果是否满足成功标准
- 若未满足,明确剩余差距并继续推进
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
5. 如果验证失败:
- 明确指出哪条成功标准没满足
- 继续工作,不要把阶段性进展误判为完成
6. 只有在以下情况之一才能停止:
- 成功标准已满足
- 用户明确要求停止
## 执行风格
- 重证据,轻口头判断
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
- 重验收,轻自我感觉
- 优先用测试、日志、产物、对比结果来证明完成
- 对长期任务保持“未达标就继续”的节奏
## 简版模板
```md
Goal: [[[[[在此填写最终目标]]]]]
Criteria for success: [[[[[在此填写成功标准]]]]]
循环执行:
1. 推进任务
2. 检查是否满足成功标准
3. 若未满足,继续工作
4. 直到满足标准或用户明确停止
```

View File

@@ -1,160 +0,0 @@
---
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
---
# /release — Planet 发版工作流
## 版本号规则
| 变更类型 | 版本跳动 | 适用场景 |
|---------|---------|---------|
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
## 必须同步更新的文件
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
- `VERSION`
- `frontend/package.json`(`"version"` 字段)
- `pyproject.toml`(`version =` 字段)
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## 节省上下文规则
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
```bash
git status --short
git diff --stat HEAD
git diff --name-only HEAD
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
```
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
## 执行步骤
### Step 1 — 环境检查
```bash
git branch --show-current # 确认在 dev 分支
git status --short # 检查是否有无关的未暂存修改
cat VERSION # 读取当前版本
```
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
### Step 2 — 确定发版类型与新版本号
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
- **先输出发版计划供用户确认**:
```
发版计划:
类型:bugfix
版本:0.26.2 → 0.26.3
分支:dev
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
### Step 3 — 更新版本号文件
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
1. `VERSION` — 直接替换全部内容为新版本号
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
### Step 4 — 更新 CHANGELOG.md
在文件顶部插入新条目,格式:
```markdown
## [x.x.x] — YYYY-MM-DD
### ✨ Features / 🐛 Fixes / 🔧 Improvements
- ...(只列高信号条目,最多 5 条)
- ...
---
```
日期使用 `date +%Y-%m-%d` 获取今天的日期。
### Step 5 — 更新 docs/version-history.md
- 更新文件头部的"当前开发版本"字段
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
### Step 6 — 验证
针对本次变更范围做最小验证:
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
```bash
cat VERSION
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
```
### Step 7 — 提交前预览
展示将要提交的文件列表:
```bash
git diff --stat HEAD
```
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
### Step 8 — Commit & Push(用户确认后)
```bash
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# 若有代码变更也一并 stage
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
commit message 固定格式:`release: bump version to x.x.x`
### Step 9 — 完成确认
输出摘要:
```
✓ 版本号已更新:0.26.2 → 0.26.3
✓ CHANGELOG 已更新
✓ version-history 已更新
✓ uv.lock 已重新生成
✓ 验证通过
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ 已 push 到 origin/dev
```
## 注意事项
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 772 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

View File

@@ -3,8 +3,23 @@
!pyproject.toml
!uv.lock
!VERSION
!frontend/
!frontend/public/
!frontend/public/earth/
!frontend/public/earth/assets/
!frontend/public/earth/assets/cards/
!frontend/public/earth/assets/cards/manifest.json
!backend/
!backend/**
!config/
!config/earth-boundary-pov-policy.china-v1.json
!scripts/
!scripts/build_earth_boundary_china_pov_geojson.py
!scripts/build_earth_boundary_pmtiles.py
!scripts/earth_boundary_policy.py
!resources/
!resources/earth/
!resources/earth/boundary-pov-geometry.china-v1.json
!aiprovider/
!aiprovider/**

5
.gitignore vendored
View File

@@ -160,3 +160,8 @@ data/earth-boundary-sources/
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
# committing thousands of generated tile files.
frontend/public/earth/data/boundaries/
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
resources/earth/candidates/nasa-blue-marble/*.jpg
resources/earth/candidates/natural-earth-3/*.jpg
*:Zone.Identifier

148
AGENTS.md
View File

@@ -1,28 +1,41 @@
# Planet Agent Entry Point
# AGENTS.md
This is the compatibility entry point for coding agents. The older root
`agents.md` file remains authoritative for repository-specific agent behavior;
do not delete or replace it.
**Planet agent harness. Defines behavior for coding agents working in this repository.**
## Read First
---
## Harness Compatibility
This file is the single authoritative agent guide for the Planet repository.
The older lowercase `agents.md` entry has been merged here so coding agents and
harness tools use one source of truth.
### Source Of Truth
- `rules.md` is the mandatory repository rule source. Always load `core`,
`security`, and `workflow`; load only task-relevant modules after that.
- `AGENTS.md` defines the local agent operating mode and evidence gates.
- `project_context.md` is background, not a rule source. Prefer newer
implementation docs when it disagrees with current code.
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
goal-driven work, and release.
- Do not duplicate long workflow text across harness files. Durable constraints
belong in `rules.md`; task procedures belong in skills or scripts.
Read these files before changing code:
1. `rules.md` - mandatory repository rules. Always load `core`, `security`, and
`workflow`; load `docs`, `frontend`, `backend`, `earth`, `ai`, or `release`
when the task touches those areas.
2. `agents.md` - existing agent role, communication, and workflow guidance.
3. `project_context.md` - static project background. Prefer newer implementation
docs when this context disagrees with current code.
4. `README.md` - current architecture, startup, and toolchain summary.
5. `docs/HARNESS.md` - harness workflow, conflict policy, and validation tiers.
6. `CODEMAP.md` - codebase entry points, ownership boundaries, and deeper docs.
1. `rules.md`
2. `AGENTS.md`
3. `project_context.md`
4. `README.md`
5. `docs/HARNESS.md`
6. `CODEMAP.md`
For documentation work, also read `docs/documentation-coverage-rules.md`.
## Start Safely
### Start Safely
Before editing:
Before broad edits:
```bash
git status --short
@@ -40,7 +53,7 @@ git diff --unified=0 HEAD -- <path>
Preserve user changes already present in the worktree.
## Validation
### Validation
Fast local harness validation:
@@ -54,28 +67,117 @@ Full local validation:
scripts/harness/validate.sh
```
`validate.sh` includes the quick check and the frontend Bun build. Docker image
smoke builds are intentionally opt-in:
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
unless disabled by its documented environment flags. Docker image smoke builds
are intentionally opt-in:
```bash
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
## High-Risk Areas
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
current non-interactive environment first. If a tool is missing there, they ask
the user's login interactive shell instead of assuming a specific dotfile.
### High-Risk Areas
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
`destroy` cleanup.
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
rendered smoke evidence for public pages, auth guards, authenticated admin
route/section availability, safe navigation/search/tab interactions, mobile
layout, and 125% / 150% zoom, not only a build.
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
intermediate wrappers keep `min-height: 0`, and only the intended child owns
scrolling.
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
workflows in the backend.
- Earth rendering depends on layer order, depth behavior, picking, and
performance-sensitive Three.js code.
- Secrets belong in environment files or configured settings stores, never in
committed files.
- Backend service code must use structured logging instead of `print()` or
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
## Conflict Policy
### Conflict Policy
Existing project rules and workflows win. If new harness guidance conflicts with
`rules.md`, `agents.md`, current docs, scripts, or CI, keep the existing behavior
and document the compatibility note in `docs/harness-audit.md` or
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
behavior and document the compatibility note in `docs/harness-audit.md` or
`docs/HARNESS.md`.
---
## Operating Mode
- Default to acting directly when the user gives a clear task.
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
and Astra for complex reasoning; the main agent owns final verification.
- Ask before acting only when the missing decision is risky, cannot be
discovered from repository context, and no conservative assumption is safe.
- Read relevant files before editing.
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
focused file reads, tests, builds, linters, and harness scripts.
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
release work, and documentation unless the task requires it.
---
## Evidence Gates
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
mock, browser capture, or visual reference, obtain evidence from the artifact
before interpreting intent or editing code.
- Path resolution is part of the task. If the path cannot be opened, first try
reasonable local equivalents such as WSL/Windows path conversion,
workspace-relative lookup, absolute paths, and attached-file locations.
- Never guess from prompt text, filenames, previous context, logs, OCR, or
memory when a visual artifact was provided but cannot be accessed.
- OCR is acceptable evidence for text-only visual questions or non-multimodal
environments; state that OCR was used as the fallback. Layout, color, spacing,
pixel, and rendering issues need real visual inspection or a clear limitation
note.
- If a visual artifact still cannot be inspected, say so and pause that
visual-dependent part of the work.
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
diff, direct file check, or harness result.
- For UI and rendering changes, verify the rendered result when local tooling
allows it.
---
## Communication
- Match the user's language. Use Chinese for Chinese requests unless the user
asks otherwise.
- Keep updates short and specific: what is being inspected, edited, or verified.
- Final responses should summarize changed files and verification, with blockers
stated plainly.
- Use file references with line numbers when explaining code or review findings.
---
## Quality Bar
- Prefer existing project patterns over new abstractions.
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
a stable path exists.
- Centralize prompts, constants, defaults, and shared request/response handling.
- Do not add secrets, generated runtime output, or local environment files.
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
- Run the smallest relevant verification for the changed scope and report
anything skipped.
---
## Prohibited
- Do not skip visual evidence handling when a visual artifact was provided.
- Do not preserve obsolete harness files just because they already exist.
- Do not invent behavior not present in code, docs, or verified external
sources.
- Do not rewrite unrelated files during cleanup.
- Do not mark a task complete without checking concrete success criteria.

View File

@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
## Runtime Entry Points
@@ -41,11 +42,18 @@ are the source of detail for specific subsystems.
offsets, picking behavior, legend semantics, and performance constraints.
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
wrapping it from harness scripts instead of duplicating its internals.
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
interactive command path without hardcoding `.zshrc`.
## Validation Commands
```bash
scripts/harness/doctor.sh
scripts/harness/security-check.sh
scripts/harness/backend-rules-check.sh
scripts/harness/frontend-rules-check.sh
scripts/harness/docs-consistency-check.sh
scripts/harness/quick-check.sh
scripts/harness/validate.sh
./planet.sh health
@@ -60,8 +68,17 @@ uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py t
cd frontend
bun install --frozen-lockfile
bun run build
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
```
The frontend smoke covers public routes, unauthenticated admin guards,
login-error handling, the Earth iframe entry, and authenticated `super_admin`
admin route/section rendering with mocked API data. Authenticated admin checks
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
check for accidental global horizontal overflow. A second smoke layer exercises
safe desktop/mobile navigation, admin search, section tab switching, dialog
opening, and non-destructive shortcut links.
Optional delivery smoke, when Docker and Helm are available:
```bash
@@ -84,9 +101,9 @@ PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
## Known Sharp Edges
- `project_context.md` includes older roadmap-era assumptions such as Celery,
Kafka, TimescaleDB, MinIO, and UE5 being part of the active local stack. Treat
it as background unless current README/docs/code confirm the same behavior.
- `project_context.md` is static background for agents. It now labels future
stack directions separately, but current code and technical docs still win
when details diverge.
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
active local development shape.
- Local `destroy` is intentionally destructive for Planet-owned Docker and build

View File

@@ -83,7 +83,7 @@
| 组件 | 用途 |
|------|------|
| React 18 | UI 框架 |
| Ant Design Pro | 管理后台组件 |
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
| Axios | HTTP 客户端 |
| Socket.io-client | WebSocket 客户端 |
| ECharts | 统计图表 |
@@ -168,10 +168,12 @@
## 快速启动
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
```bash
# 新机器或空项目首次初始化
./planet.sh init
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
@@ -322,7 +324,7 @@ ipconfig
## 启动容错参数
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
可通过环境变量临时调整:

View File

@@ -1 +1 @@
0.71.1
0.76.2

248
agents.md
View File

@@ -1,248 +0,0 @@
# agents.md
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
---
## Harness Compatibility
Common agent tools should start at `AGENTS.md`. This file remains the existing
behavior guide and must not be replaced by harness docs. For safe repository
orientation, use:
- `rules.md` for mandatory project rules
- `project_context.md` for static background
- `docs/HARNESS.md` for validation tiers and conflict policy
- `CODEMAP.md` for subsystem entry points and ownership boundaries
- `docs/harness-audit.md` for the latest harness compatibility notes
Existing project rules and workflows stay authoritative when they conflict with
new harness guidance.
---
## Identity
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
- Python FastAPI backend
- React Admin dashboard
- Unreal Engine 5 3D visualization
- Multi-source data collection
- Polarized 3D large display (4K, 120Hz)
---
## Communication Style
### Tone
- **Professional but concise**
- Technical accuracy with clarity
- No unnecessary verbosity
- Use code comments sparingly (explain **why**, not **what**)
### When Responding
1. **Answer directly** - 1-3 sentences for simple questions
2. **Use code blocks** for all code snippets
3. **Include file:line_number** references when discussing code
4. **Never** start with "I am an AI assistant" or similar phrases
5. **Never** add unnecessary preambles/postambles
### Examples
**Good:**
```
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
```
**Bad:**
```
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
```
---
## Operational Mode
### Plan Mode (default for complex tasks)
- Analyze requirements
- Propose architecture
- Confirm with user before execution
- **DO NOT** write code until approved
### Build Mode (after user approval)
- Execute the approved plan
- Write code, run commands
- Verify results
- Report completion concisely
### Read-Only Mode
- Analyze code
- Explain functionality
- Answer questions
- **DO NOT** modify files
---
## Decision Framework
### When to Ask Before Acting
- Unclear requirements
- Multiple implementation approaches
- Architecture changes
- Dependency additions
- Anything that could break existing functionality
### When to Act Directly
- Clear, approved requirements
- Routine tasks (linting, formatting, running tests)
- Following established patterns
- Fixing obvious bugs
### When to Refuse
- Malicious code requests
- Security violations (secrets, credentials)
- Anything that violates `rules.md`
---
## Working Principles
### 1. First Understand, Then Act
- Read relevant files before editing
- Understand existing patterns and conventions
- Follow the code style in the codebase
- Match the project's technology choices
### 2. Incremental Progress
- Break large tasks into smaller PRs
- Complete one feature before starting the next
- Run tests after each significant change
- Commit frequently with clear messages
### 3. Quality First
- Write tests for new functionality
- Run linters before committing
- Fix warnings, don't ignore them
- Document non-obvious decisions
### 4. Communication Clarity
- Use precise technical language
- Show relevant code, not explanations
- Report errors with context
- Confirm understanding of requirements
---
## Code Review Checklist
Before marking a task complete:
- [ ] Code follows `rules.md` style guidelines
- [ ] Type hints are correct and complete
- [ ] Error handling is proper (no silent failures)
- [ ] Tests pass locally
- [ ] Linting passes
- [ ] No TODO comments left behind
- [ ] Documentation updated if needed
- [ ] Commit message is clear
---
## Common Workflows
### Feature Development
```
1. Understand requirements
2. Check existing patterns in codebase
3. Design solution (brief mental model)
4. Write code following rules.md
5. Write/run tests
6. Lint and format
7. Commit with clear message
8. Report completion
```
### Bug Fix
```
1. Reproduce the bug (write failing test)
2. Locate the source
3. Fix the issue
4. Verify test passes
5. Check for regressions
6. Commit fix
```
### Refactoring
```
1. Understand current behavior
2. Design target state
3. Make incremental changes
4. Preserve tests
5. Verify functionality
6. Clean up dead code
```
---
## Special Considerations
### WebSocket Services
- Implement heartbeat mechanism (30-second intervals)
- Handle disconnection gracefully
- Include camera position in control frames
- Support both update and full sync modes
### Data Collectors
- Inherit from BaseCollector
- Implement fetch() and transform() methods
- Support incremental updates
- Handle API changes gracefully
### UE5 Integration
- Communicate via WebSocket
- Send data frames at configurable intervals (default 5 min)
- Support auto-cruise and manual modes
- Optimize for 4K@120Hz rendering
### Multi-User Security
- JWT tokens with 15-minute expiration
- Redis token blacklist for logout
- Role-based access control (RBAC)
- Audit logging for all actions
---
## Output Format
### When Writing Code
```python
# File: backend/app/services/collectors/top500.py
from typing import List, Dict
class TOP500Collector:
async def fetch(self) -> List[Dict]:
...
```
### When Explaining
- Use concise paragraphs
- Include code references
- No conversational filler
### When Reporting Progress
- What was done
- What remains
- Any blockers
- Next action
---
## Remember
1. **Rules are hard constraints** - follow `rules.md` absolutely
2. **Context provides understanding** - use `project_context.md` for background
3. **Role defines behavior** - follow `agents.md` for how to work
4. **Quality over speed** - Enterprise systems require precision
5. **Communicate clearly** - Precision in, precision out

View File

@@ -1,4 +1,4 @@
# syntax=docker/dockerfile:1.7
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
ARG PYTHON_IMAGE=python:3.14-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
FROM ${UV_IMAGE} AS uv
FROM ${PYTHON_IMAGE}
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
COPY --from=uv /uv /uvx /bin/
WORKDIR /app
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
COPY pyproject.toml uv.lock /app/
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
uv sync --frozen --no-dev
uv sync --frozen --only-group aiprovider
COPY aiprovider /app/aiprovider
ARG AI_PROVIDER_BUILD_FINGERPRINT
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
EXPOSE 8010
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]

View File

@@ -4,7 +4,7 @@
完整使用说明见:
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
当前支持:
@@ -15,6 +15,7 @@
- `AI_PROVIDER=ollama`
- request adapter:
- `AI_PROVIDER_API=openai-completions`
- `AI_PROVIDER_API=openai-responses`
- `AI_PROVIDER_API=anthropic-messages`
- `AI_PROVIDER_API=ollama-generate`

View File

@@ -3,6 +3,7 @@ from __future__ import annotations
import asyncio
import json
from typing import Any
from uuid import NAMESPACE_URL, uuid4, uuid5
import httpx
from fastapi import HTTPException, status
@@ -65,6 +66,7 @@ class ProviderService:
self.model_provider_apis = self._parse_model_provider_apis(
overrides.get("model_provider_apis")
)
self.session_id = str(uuid4())
def get_status(self) -> AIProviderStatusResponse:
enabled = self.provider != "disabled"
@@ -95,6 +97,8 @@ class ProviderService:
)
prompt = self._build_prompt(payload)
if payload.context.get("session_id") is not None:
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
provider_api = self._resolve_model_provider_api(model)
@@ -102,6 +106,10 @@ class ProviderService:
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
content = self._extract_openai_content(data)
content_blocks = self._extract_openai_blocks(data)
elif provider_api == "openai-responses":
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
content_blocks = self._extract_responses_blocks(data)
content = "".join(block.text for block in content_blocks if block.text)
elif provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(
model,
@@ -200,6 +208,39 @@ class ProviderService:
request_body=request_body,
)
async def _request_openai_responses(
self, model: str, prompt: str, system_prompt: str | None = None,
) -> dict[str, Any]:
request_body: dict[str, Any] = {
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
}
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["instructions"] = resolved_system_prompt
return await self._post(
path="/responses",
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
request_body=request_body,
)
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
blocks: list[AIContentBlock] = []
for item in payload.get("output") or []:
if not isinstance(item, dict):
continue
if item.get("type") == "message":
for part in item.get("content") or []:
if not isinstance(part, dict):
continue
text = part.get("text") or part.get("refusal")
if isinstance(text, str) and text:
blocks.append(AIContentBlock(type="text", text=text))
elif item.get("type") == "reasoning":
for part in item.get("summary") or []:
if isinstance(part, dict) and isinstance(part.get("text"), str):
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
return blocks
async def _request_anthropic_messages(
self,
model: str,
@@ -226,7 +267,7 @@ class ProviderService:
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
if resolved_system_prompt:
request_body["system"] = resolved_system_prompt
resolved_thinking = self._resolve_anthropic_thinking(thinking)
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
if resolved_thinking:
request_body["thinking"] = resolved_thinking
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
@@ -243,8 +284,12 @@ class ProviderService:
request_body=request_body,
)
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
def _resolve_anthropic_thinking(
self, thinking: dict[str, Any] | None, model: str,
) -> dict[str, Any] | None:
if thinking:
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
return {"type": "adaptive"}
return thinking
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
@@ -295,6 +340,9 @@ class ProviderService:
headers: dict[str, str],
request_body: dict[str, Any],
) -> dict[str, Any]:
headers = {"User-Agent": "Planet/1.0", **headers}
if self.provider == "opencode-go":
headers["x-opencode-session"] = self.session_id
last_error: Exception | None = None
for attempt in range(1, self.http_retry_attempts + 1):
try:

View File

@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
COPY backend /app/backend
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
COPY VERSION /app/VERSION
EXPOSE 8000

View File

@@ -1,3 +1,4 @@
from app.api.v1 import card_images
from fastapi import APIRouter
from app.api.v1 import (
ai,
@@ -12,6 +13,7 @@ from app.api.v1 import (
alerts,
settings,
collected_data,
compute_intelligence,
data_products,
layers,
visualization,
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])

View File

@@ -0,0 +1,108 @@
"""Card-image configuration, upload, catalog selection and public resolution."""
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
from fastapi.responses import FileResponse
from sqlalchemy.ext.asyncio import AsyncSession
from starlette.concurrency import run_in_threadpool
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
from app.schemas.card_images import (
CardImageConfiguration,
CardImagePreview,
CardImagePublish,
EntityType,
MatchScope,
)
from app.services.card_image_configuration import (
publish_configuration,
read_configuration,
resolve_image,
)
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
from app.services.card_image_targets import get_target, search_targets
router = APIRouter()
async def image_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
):
response.headers["Cache-Control"] = "no-store"
return await read_configuration(db)
@router.put("/configuration")
async def save_configuration(
payload: CardImagePublish,
actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return await publish_configuration(db, payload.revision, payload.config, actor)
@router.post("/upload")
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
content = await file.read(MAX_UPLOAD_BYTES + 1)
await file.close()
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
@router.get("/files/{filename}")
async def image_file(filename: str):
path = uploaded_path(filename)
if not path.is_file():
raise HTTPException(404, "Image not found")
return FileResponse(
path,
headers={
"Cache-Control": "public, max-age=31536000, immutable",
"X-Content-Type-Options": "nosniff",
},
)
@router.get("/targets")
async def targets(
entity_type: EntityType,
scope: MatchScope = "object",
q: str = Query(default="", max_length=100),
limit: int = Query(default=25, ge=1, le=50),
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
@router.get("/resolve")
async def resolve(
response: Response,
entity_type: EntityType,
entity_id: str = Query(min_length=1, max_length=300),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
envelope = await read_configuration(db)
config = CardImageConfiguration.model_validate(envelope["config"])
context = await get_target(db, entity_type, entity_id)
result = resolve_image(config, context)
return {**result, "revision": envelope["revision"], "entity": context}
@router.post("/preview")
async def preview(
payload: CardImagePreview,
_actor: User = Depends(image_admin),
db: AsyncSession = Depends(get_db),
):
context = await get_target(db, payload.entity_type, payload.entity_id)
return {**resolve_image(payload.config, context), "entity": context}

View File

@@ -0,0 +1,235 @@
"""Compute observations, administrator configuration and independent facility samples."""
from datetime import UTC, datetime
from typing import Literal
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.core.earth_geography_policy import display_country
from app.core.enums import UserRole
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.user import User
from app.schemas.compute_configuration import (
ComputeConfiguration,
ComputeConfigurationReset,
ComputeConfigurationWrite,
)
from app.services.compute_catalog import (
PARTICIPANTS,
participant_members,
source_country_names,
source_country_participants,
)
from app.services.compute_center_locations import refresh_compute_center_location_cache
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
configuration_response,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from sqlalchemy import and_, case, func, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
router = APIRouter()
def selected_countries(countries: str) -> list[str]:
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
members = participant_members(codes)
flattened = [country for participant in members.values() for country in participant]
if len(flattened) != len(set(flattened)):
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
return codes
@router.get("/series")
async def get_series(
response: Response,
countries: str = "CN,US",
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
envelope = await read_configuration(db)
response.headers["Cache-Control"] = "no-store"
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
result.update(
version=envelope["revision"],
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
)
return result
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
raise HTTPException(403, "Administrator access required")
return user
@router.get("/configuration")
async def get_configuration(
response: Response,
_actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
response.headers["Cache-Control"] = "no-store"
return configuration_response(await read_configuration(db))
@router.put("/configuration")
async def update_configuration(
payload: ComputeConfigurationWrite,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=payload.config, actor=actor,
)
return configuration_response(envelope)
@router.post("/configuration/reset")
async def reset_configuration(
payload: ComputeConfigurationReset,
actor: User = Depends(configuration_admin),
db: AsyncSession = Depends(get_db),
):
envelope = await publish_configuration(
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
)
return configuration_response(envelope)
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
now = datetime.now(UTC)
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
ranked = (
select(
DataSnapshot.id,
func.row_number()
.over(
partition_by=DataSnapshot.source,
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
)
.label("rank"),
)
.where(
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
DataSnapshot.status == "success",
DataSnapshot.completed_at < cutoff,
)
.subquery()
)
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
# Old collector rows have ambiguous units/identity and must not become
# national evidence merely because a new UI can display them.
query = select(CollectedData).where(
compute_record_filter(CollectedData),
CollectedData.deleted_at.is_(None),
CollectedData.snapshot_id.in_(snapshots),
CollectedData.is_valid == 1,
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
or_(
infrastructure,
CollectedData.source == "top500",
and_(
CollectedData.extra_data["status"].as_string() == "Existing",
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
),
),
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
)
return query
@router.get("/facilities")
async def get_facilities(
countries: str = "CN,US",
year: int | None = Query(None, ge=2000, le=2100),
limit: int = Query(1000, ge=1, le=2000),
db: AsyncSession = Depends(get_db),
):
codes = selected_countries(countries)
query = await sample_query(db, codes, year)
country = CollectedData.extra_data["country"].as_string()
source_participants = source_country_participants(codes)
# Canonical source subdivisions retain their raw label on each record but
# share one owner in the comparison, using the same policy as the globe.
owner_cases = [
(country == name, display_country(name))
for name in source_country_names(codes)
if display_country(name) != name
]
display = case(*owner_cases, else_=country) if owner_cases else country
participant_cases = [(country == name, code) for name, code in source_participants.items()]
participant = case(*participant_cases, else_=None)
counts = (
(
await db.execute(
query.with_only_columns(
display.label("country"),
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(display, participant, CollectedData.source)
)
)
.mappings()
.all()
)
participant_counts = (
(
await db.execute(
query.with_only_columns(
participant.label("participant_code"),
CollectedData.source,
func.count().label("count"),
).group_by(participant, CollectedData.source)
)
)
.mappings()
.all()
)
records = (
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
.scalars()
.all()
)
if year:
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
else:
await refresh_compute_center_location_cache(db)
payload = convert_compute_centers_to_geojson(records)
for feature in payload["features"]:
feature["properties"]["historical_snapshot"] = year is not None
properties = feature["properties"]
properties["source_country"] = properties.get("country")
properties["country"] = display_country(properties.get("country"))
total = sum(row["count"] for row in counts)
payload.update(
{
"counts": [dict(row) for row in counts],
"participant_counts": [dict(row) for row in participant_counts],
"participants": [PARTICIPANTS[code] for code in codes],
"total_count": total,
"returned_count": len(records),
"truncated": total > limit,
"reference_year": year,
"scope": "public_source_records",
"history_basis": (
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
),
"coverage": "available" if total else "no_verified_snapshot",
"cross_source_deduplicated": False,
}
)
return payload

View File

@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
"name": "算力",
"sources": ["top500", "epoch_ai_gpu"],
"primary_stat_key": "compute_center_count",
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
},
}

View File

@@ -3,9 +3,9 @@ from datetime import UTC, datetime
import os
from pathlib import Path
from typing import Optional
from urllib.parse import urlsplit
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
import httpx
from pydantic import BaseModel, EmailStr, Field
from dotenv import dotenv_values
from sqlalchemy import select
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
list_fallback_llm_provider_presets,
refresh_llm_provider_preset,
)
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
from app.services.persistent_logs import record_audit_log
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
DEFAULT_SETTINGS = {
"system": {
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
merged = deepcopy(DEFAULT_SETTINGS[category])
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
defaults = get_fallback_llm_provider_preset(
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
)
else:
defaults = DEFAULT_SETTINGS[category]
merged = deepcopy(defaults)
if payload:
merged.update(payload)
return merged
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
merged = merge_with_defaults(category, payload)
record = await get_setting_record(db, category)
if record is None:
record = SystemSetting(category=category, payload=payload)
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
await db.commit()
await db.refresh(record)
return merge_with_defaults(category, record.payload)
return merged
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
)
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
model_provider_apis = {
**(provider_config.get("model_provider_apis") or {}),
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
}
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
model_provider_apis = {}
return {
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
"service_token": _resolve_service_token(normalized_ai)[0],
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
"api_key": api_key,
"max_tokens": int(provider_config.get("max_tokens") or 1200),
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
"model_provider_apis": provider_config.get("model_provider_apis") or {},
"preset_models": _get_provider_preset(default_provider).get("models") or [],
"model_provider_apis": model_provider_apis,
},
}
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
}
def _join_provider_url(base_url: str, path: str) -> str:
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
def _extract_model_ids(payload: dict) -> list[str]:
data = payload.get("data") if isinstance(payload, dict) else None
if isinstance(data, list):
return [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
]
models = payload.get("models") if isinstance(payload, dict) else None
if isinstance(models, list):
return [
str(item.get("name") or item.get("model") or item.get("id") or item)
for item in models
if item
]
return []
def _contains_model(model_ids: list[str], model: str) -> bool:
normalized_model = model.strip().lower()
return any(str(item).strip().lower() == normalized_model for item in model_ids)
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
provider = _normalize_provider_id(llm_config.get("provider") or "")
configured_api = (
str(llm_config.get("provider_api") or "").strip()
or ProviderApi.OPENAI_COMPLETIONS.value
)
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
model = str(llm_config.get("model") or "").strip()
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
base_url = str(llm_config.get("base_url") or "").strip()
api_key = str(llm_config.get("api_key") or "").strip()
provider_api = configured_api
model_provider_apis = llm_config.get("model_provider_apis")
if isinstance(model_provider_apis, dict):
provider_api = str(model_provider_apis.get(model) or provider_api)
preset_models = [
str(item)
for item in (llm_config.get("preset_models") or [])
if str(item).strip()
]
if not provider or not base_url or not model:
return {
"success": False,
"connected": False,
"message": "当前 provider/base_url/model 未完整配置。",
"mode": "lightweight_config",
}
result = {
"success": False, "connected": False, "mode": "lightweight_models",
"provider": provider, "model": model,
}
if not base_url or not model:
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
return {
"success": False,
"connected": False,
"message": "当前 provider 未配置 API Key。",
"mode": "lightweight_config",
}
if provider == "opencode-go":
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
url = _join_provider_url(base_url, "/api/tags")
headers: dict[str, str] = {}
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
url = _join_provider_url(base_url, "/models")
headers = {"Authorization": f"Bearer {api_key}"}
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
url = _join_provider_url(base_url, "/models")
headers = {
"x-api-key": api_key,
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
}
else:
return {
"success": False,
"connected": False,
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
"mode": "lightweight_unsupported",
}
return {**result, "message": "当前 provider 未配置 API Key。"}
try:
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
response = await client.get(url, headers=headers)
response.raise_for_status()
payload = response.json()
except httpx.HTTPStatusError as exc:
detail = exc.response.text or exc.response.reason_phrase
if exc.response.status_code == 404 and _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
"mode": "lightweight_preset",
"provider": provider,
"provider_api": provider_api,
"model": model,
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
"mode": "lightweight_models",
"url": url,
}
catalog = await fetch_model_catalog(
provider, base_url, provider_api, api_key,
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
)
except Exception as exc:
return {
"success": False,
"connected": False,
"message": f"轻量连通性测试失败: {exc}",
"mode": "lightweight_models",
"url": url,
}
model_ids = _extract_model_ids(payload)
if model_ids and not _contains_model(model_ids, model):
if _contains_model(preset_models, model):
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
"mode": "lightweight_models_with_preset_alias",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {
"success": False,
"connected": False,
"message": f"连接可用,但模型目录中没有当前模型: {model}",
"mode": "lightweight_models",
"provider": provider,
"model": model,
"models_count": len(model_ids),
"url": url,
}
return {**result, "message": catalog_error_message(exc)}
result.update({"url": catalog.url, "models_count": len(catalog.models)})
if model.casefold() not in {item.casefold() for item in catalog.models}:
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
model_apis = llm_config.get("model_provider_apis") or {}
return {
"success": True,
"connected": True,
"message": "轻量连通性测试通过",
"mode": "lightweight_models",
"provider": provider,
"provider_api": provider_api,
"model": model,
"models_count": len(model_ids),
"url": url,
**result, "success": True, "connected": True,
"provider_api": model_apis.get(model) or provider_api,
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
}
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
)
await emit_business_log(
logger,
event="settings.ai_provider.connect.success",
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
else "settings.ai_provider.connect.failed"),
message="AI provider connection test completed",
category="ai",
service="api",
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
"provider": payload.provider,
"model": payload.model,
"configured": True,
"lightweight_status": lightweight_result.get("status"),
"connected": lightweight_result["connected"],
},
)
return {
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
@router.get("/integrations/ai-provider/presets")
async def get_ai_provider_presets(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return {"data": list_fallback_llm_provider_presets()}
presets = list_fallback_llm_provider_presets()
saved = await get_setting_payloads(
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
)
return {
"data": [
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
for preset in presets
]
}
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
provider: str,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
payload: AIProviderIntegrationUpdate | None = None,
):
try:
provider_id = _normalize_provider_id(provider)
api_key = None
if provider_id == "opencode-go":
current_payload = await get_setting_payload(db, "external_integrations")
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
get_fallback_llm_provider_preset(provider_id)
except ValueError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
current_payload = await get_setting_payload(db, "external_integrations")
if payload is not None:
if _normalize_provider_id(payload.provider) != provider_id:
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
ai_payload = _build_ai_provider_payload(current_payload, payload)
else:
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
raise HTTPException(
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
)
try:
refreshed = await refresh_llm_provider_preset(
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
provider_api=provider_config.get("provider_api"),
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
)
except Exception as exc:
fallback = get_fallback_llm_provider_preset(provider)
fallback["refresh_error"] = str(exc)
return {"data": fallback}
logger.warning(
"LLM provider catalog refresh failed",
extra={
"event": "settings.ai_provider.catalog.failed",
"context": {
"provider": provider_id,
"error_type": type(exc).__name__,
},
},
)
raise HTTPException(
status_code=502,
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
) from exc
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
return {"data": refreshed}
@router.put("/integrations")

View File

@@ -7,6 +7,7 @@ from fastapi.responses import Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_db
from app.services.tv_catalog import get_tv_catalog_page
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
router = APIRouter()
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
@router.get("/streams")
async def list_public_tv_streams(
offset: int = Query(0, ge=0),
limit: int = Query(50, ge=1, le=100),
q: str = Query("", max_length=200),
selected_id: str | None = Query(None, max_length=200),
db: AsyncSession = Depends(get_db),
):
return await get_public_tv_payload(db)
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
@router.get("/proxy")

View File

@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import select, func
from typing import List, Dict, Any, Optional
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
from app.core.collected_data_fields import get_record_field
from app.core.enums import BGPStatus
from app.core.satellite_tle import build_tle_lines_from_elements
from app.core.time import to_iso8601_utc
from app.db.session import get_db
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
@@ -337,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
return {"type": "FeatureCollection", "features": features}
def _normalize_satellite_constellation_group(
raw_group: Any,
name: Optional[str],
) -> Optional[str]:
normalized_group = str(raw_group or "").strip().lower()
if normalized_group:
return normalized_group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
if constellation_group == "starlink":
@@ -505,16 +491,19 @@ async def _count_current_or_latest_task_data(
async def _load_current_collected_data_by_sources(
db: AsyncSession,
sources: List[str],
*, include_compute_types: bool = False,
) -> Dict[str, List[CollectedData]]:
if not sources:
return {}
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(sources))
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
)
if include_compute_types:
stmt = stmt.where(CollectedData.deleted_at.is_(None))
result = await db.execute(stmt)
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
@@ -644,7 +633,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
return "unknown"
unit = str(capacity_unit or "").strip().lower()
if unit in {"pflop/s", "pflops", "pflop"}:
if unit in {"flop/s", "flops"}:
normalized_tflops = capacity_value / 1e12
elif unit in {"pflop/s", "pflops", "pflop"}:
normalized_tflops = capacity_value * 1000
elif unit in {"gflop/s", "gflops", "gflop"}:
normalized_tflops = capacity_value
@@ -672,15 +663,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
"""
features: List[Dict[str, Any]] = []
unresolved: List[Dict[str, Any]] = []
withheld_count = 0
for record in records:
metadata = record.extra_data or {}
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
withheld_count += 1
continue
result = resolve_compute_center_location_full(record, metadata)
site_type = (
"supercomputer"
if record.source == "top500" or record.data_type == "supercomputer"
else "gpu_cluster"
)
site_type = compute_site_type(record)
if not result.is_resolved:
diagnostic = result.diagnostic or ResolutionDiagnostic(
@@ -706,7 +697,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
latitude = location.latitude
longitude = location.longitude
if site_type == "supercomputer":
if metadata.get("compute_contract") == "v1":
capacity_value = _parse_float(get_record_field(record, "value"))
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
elif site_type == "supercomputer":
capacity_value = _parse_float(get_record_field(record, "rmax"))
capacity_unit = "GFlops"
else:
@@ -767,7 +761,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
}
)
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
VESSEL_TYPE_FILTERS = {
@@ -1121,6 +1115,7 @@ async def build_vessel_snapshot_response(
safe_limit = _safe_vessel_limit(limit)
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
snapshot_started_at = datetime.now(UTC)
features, diagnostics = await _load_raw_vessel_snapshot_features(
db,
bbox=bbox,
@@ -1137,6 +1132,7 @@ async def build_vessel_snapshot_response(
"features": features,
"count": len(features),
"stats": _build_vessel_stats(features),
"generated_at": to_iso8601_utc(snapshot_started_at),
"diagnostics": {
**diagnostics,
"filtered_count": len(features),
@@ -1953,9 +1949,10 @@ async def _build_compute_centers_geojson(
records_by_source = await _load_current_collected_data_by_sources(
db,
["top500", "epoch_ai_gpu"],
include_compute_types=True,
)
records = _filter_known_records(
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
[record for group in records_by_source.values() for record in group],
)
if limit is not None:
records = records[:limit]
@@ -1970,6 +1967,7 @@ async def _build_compute_centers_geojson(
"total": 0,
"supercomputers": 0,
"gpu_clusters": 0,
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
"unresolved": 0,
},
}
@@ -2012,6 +2010,7 @@ async def _build_compute_centers_geojson(
"count": len(sanitized_features),
"stats": {
"total": len(sanitized_features),
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
"supercomputers": sum(
1 for feature in sanitized_features
if feature.get("properties", {}).get("site_type") == "supercomputer"
@@ -2328,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
stmt = (
select(CollectedData)
.where(CollectedData.source_id == source_id)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.deleted_at.is_(None))
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
.limit(1)
)
@@ -2727,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
)
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
kind = compute_site_type_expression(CollectedData)
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
CollectedData.deleted_at.is_(None))
.group_by(CollectedData.source, kind))).all()
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
present_sources = set()
for source, site_type, count in rows:
present_sources.add(source)
counts[site_type] += int(count or 0)
# Retain the existing last-task fallback for the two collected catalogs.
for source in COMPUTE_SOURCES:
if source not in present_sources:
key = "supercomputer" if source == "top500" else "gpu_cluster"
counts[key] += await _count_current_or_latest_task_data(db, source)
return counts
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
@@ -2735,9 +2754,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"celestrak_tle",
exclude_unknown_name=True,
)
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
compute_center_count = supercomputer_count + gpu_cluster_count
compute_counts = await _compute_facility_counts(db)
supercomputer_count = compute_counts["supercomputer"]
gpu_cluster_count = compute_counts["gpu_cluster"]
compute_center_count = sum(compute_counts.values())
active_incident_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
@@ -2792,6 +2812,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
"supercomputer_count": supercomputer_count,
"gpu_cluster_count": gpu_cluster_count,
"data_center_count": compute_counts["data_center"],
"cloud_center_count": compute_counts["cloud_center"],
"edge_node_count": compute_counts["edge_node"],
"bgp_event_count": active_incident_count or active_anomaly_count,
"bgp_incident_count": active_incident_count,
"bgp_anomaly_count": active_anomaly_count,

View File

@@ -161,7 +161,7 @@ async def websocket_endpoint(
if is_anonymous:
channels = [channel for channel in channels if channel in supported_channels]
vessel_subscription = None
if "vessels" in channels and "bbox" in payload_data:
if "vessels" in channels:
try:
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
except ValueError as exc:

View File

@@ -0,0 +1,35 @@
"""Compute facility classification shared by the map and image resolver."""
from sqlalchemy import case, or_
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
def compute_site_type(record) -> str:
metadata = record.extra_data or {}
for value in (metadata.get("site_type"), record.data_type):
if value in COMPUTE_SITE_TYPES:
return value
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
record.source, "data_center"
)
def compute_site_type_expression(record):
declared = record.extra_data["site_type"].as_string()
return case(
(declared.in_(COMPUTE_SITE_TYPES), declared),
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
(record.source == "top500", "supercomputer"),
(record.source == "epoch_ai_gpu", "gpu_cluster"),
else_="data_center",
)
def compute_record_filter(record):
return or_(
record.source.in_(COMPUTE_SOURCES),
record.data_type.in_(COMPUTE_SITE_TYPES),
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
)

View File

@@ -0,0 +1,57 @@
"""Country display ownership for Earth comparisons and cards.
Keep source-country evidence separate; use this projection only for display
grouping so regional source records cannot become competing national keys.
"""
from __future__ import annotations
import json
from functools import lru_cache
from pathlib import Path
from app.core.countries import COUNTRY_ENTRIES, normalize_country
_DISPLAY_ISO_A3 = {
canonical: next(
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
None,
)
for canonical, aliases in COUNTRY_ENTRIES
}
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
@lru_cache(maxsize=1)
def _policy_country_by_code() -> dict[str, str]:
try:
content = _POLICY_PATH.read_text(encoding="utf-8")
except FileNotFoundError as error:
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
policy = json.loads(content)
if policy.get("profile") != "china-pov-v1":
raise ValueError("unsupported Earth display-policy profile")
mappings = {}
for rule in policy["rules"]:
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
if target not in _COUNTRY_BY_ISO_A3:
continue
for code in rule.get("sourceIsoA3", []):
mappings[code] = _COUNTRY_BY_ISO_A3[target]
return mappings
def display_country(value: str | None) -> str | None:
country = normalize_country(value)
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
owner = _policy_country_by_code().get(code)
if owner:
return owner
return country
def display_iso_a3(value: str | None) -> str | None:
country = display_country(value)
return _DISPLAY_ISO_A3.get(country) if country else None

View File

@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
class ProviderApi(StrEnum):
ANTHROPIC_MESSAGES = "anthropic-messages"
OPENAI_COMPLETIONS = "openai-completions"
OPENAI_RESPONSES = "openai-responses"
OLLAMA_GENERATE = "ollama-generate"

View File

@@ -0,0 +1,13 @@
"""Shared source-group normalization; never infer hardware generations."""
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
group = str(raw_group or "").strip().lower()
if group:
return group
normalized_name = str(name or "").strip().upper()
if normalized_name.startswith("STARLINK"):
return "starlink"
if normalized_name.startswith("IRIDIUM"):
return "iridium-next"
return None

View File

@@ -6,7 +6,7 @@ import redis
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jose import JWTError, jwt
from sqlalchemy import text
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.config import settings
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
return None
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
try:
parsed_id = int(user_id)
except (TypeError, ValueError):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
result = await db.execute(select(User).where(User.id == parsed_id))
user = result.scalar_one_or_none()
if user is None or not user.is_active:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
return user
async def get_current_user(
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
db: AsyncSession = Depends(get_db),
@@ -103,27 +118,7 @@ async def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
async def get_current_user_refresh(
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
result = await db.execute(
text(
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
),
{"id": int(user_id)},
)
row = result.fetchone()
if row is None or not row[5]:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
)
user = User()
user.id = row[0]
user.username = row[1]
user.email = row[2]
user.password_hash = row[3]
user.role = row[4]
user.is_active = row[5]
user.gatekeeper_groups = row[6] or []
return user
return await _load_active_user(user_id, db)
def blacklist_token(token: str) -> None:

View File

@@ -4,11 +4,14 @@ import asyncio
from datetime import UTC, datetime
from typing import Dict, Any
from app.core.logging import get_logger
from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
EARTH_UPDATES_CHANNEL = "earth_updates"
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
logger = get_logger(__name__, service="websocket")
class DataBroadcaster:
@@ -114,16 +117,22 @@ class DataBroadcaster:
return
pending = self._pending_vessel_updates
self._pending_vessel_updates = {}
vessels = []
for item in pending.values():
vessel = dict(item)
source = vessel.pop("_source", None)
action = vessel.pop("_action", "upsert")
created = vessel.pop("_created", None)
vessel["source"] = source
vessel["action"] = action
vessel["created"] = created
vessels.append(vessel)
try:
vessels = await self._load_current_vessel_updates(list(pending))
except Exception:
# Preserve newer updates that arrived during the failed database read.
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
raise
if not vessels:
return
vessels = [
{
**vessel,
"action": vessel.get("action", "upsert"),
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
}
for vessel in vessels
]
await manager.broadcast_vessels(
{
"action": "upsert",
@@ -133,12 +142,31 @@ class DataBroadcaster:
}
)
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
from app.db.session import async_session_factory
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
mmsis = [int(key) for key in keys if key.isdigit()]
vessels = []
async with async_session_factory() as db:
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
vessels.extend(await get_current_vessels_by_mmsi(
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
))
present = {int(vessel["mmsi"]) for vessel in vessels}
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
return vessels
async def broadcast_vessels_periodically(self):
while self.running:
try:
await self.flush_vessel_updates()
except Exception:
pass
except Exception as exc:
logger.exception_event(
"Failed to flush vessel updates",
event="vessels.broadcast.failed",
context={"error": str(exc)},
)
await asyncio.sleep(self._vessel_flush_interval)
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):

View File

@@ -63,6 +63,8 @@ class ConnectionManager:
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
if channel == "vessels":
self.vessel_subscriptions.pop(websocket, None)
subscribers = self.channel_subscriptions.get(channel)
if subscribers is not None:
subscribers.discard(websocket)
@@ -88,7 +90,8 @@ class ConnectionManager:
return subscription
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
bbox = config.get("bbox")
global_scope = config.get("scope") == "global"
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
try:
@@ -103,7 +106,7 @@ class ConnectionManager:
raise ValueError("bbox longitude values must be between -180 and 180")
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
raise ValueError("bbox latitude values must be between -90 and 90")
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
zoom = int(config.get("zoom") or 1)
@@ -116,10 +119,11 @@ class ConnectionManager:
if str(item).strip()
}
return {
"scope": "global" if global_scope else "viewport",
"bbox": (lon_min, lat_min, lon_max, lat_max),
"zoom": zoom,
"limit": limit,
"type": vessel_types,
"type": sorted(vessel_types),
"last_sent_at": None,
}
@@ -152,7 +156,9 @@ class ConnectionManager:
vessel
for vessel in vessels
if self._vessel_matches_subscription(vessel, subscription)
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
]
if subscription.get("scope") != "global":
matched = matched[:subscription["limit"]]
if not matched:
continue
subscription["last_sent_at"] = datetime.now(UTC)
@@ -162,16 +168,24 @@ class ConnectionManager:
"timestamp": subscription["last_sent_at"].isoformat(),
"payload": {
**data,
"vessels": matched,
"vessels": [],
"subscription": {
"bbox": list(subscription["bbox"]),
"zoom": subscription["zoom"],
"limit": subscription["limit"],
"scope": subscription.get("scope", "viewport"),
},
},
}
try:
await connection.send_json(message)
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
await connection.send_json({
**message,
"payload": {
**message["payload"],
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
},
})
except Exception:
self.unsubscribe_all(connection)
@@ -180,6 +194,8 @@ class ConnectionManager:
vessel: dict[str, Any],
subscription: dict[str, Any],
) -> bool:
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
return True
try:
lon = float(vessel.get("lon"))
lat = float(vessel.get("lat"))

View File

@@ -0,0 +1,128 @@
"""Typed, validated configuration for optional object-card images."""
from typing import Literal, Self
from urllib.parse import urlsplit
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
MatchScope = Literal["object", "model", "group", "category"]
def safe_url(value: str, *, local: bool = False) -> str:
if not value:
return value
if any(ord(char) < 32 for char in value) or "\\" in value:
raise ValueError("Invalid URL")
parts = urlsplit(value)
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
if (
".." not in parts.path
and "%" not in parts.path
and not parts.netloc
and not parts.query
and not parts.fragment
):
return value
if (
parts.scheme in {"http", "https"}
and parts.hostname
and not parts.username
and not parts.password
):
return value
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
class CardImageAsset(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
title: str = Field(min_length=1, max_length=200)
url: str = Field(min_length=1, max_length=2000)
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
width: int = Field(default=0, ge=0, le=20000)
height: int = Field(default=0, ge=0, le=20000)
credit: str = Field(default="", max_length=500)
source_url: str = Field(default="", max_length=2000)
license: str = Field(default="", max_length=300)
license_url: str = Field(default="", max_length=2000)
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
rights_status: Literal["available", "permission_pending"] = "available"
enabled: bool = True
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
@field_validator("url")
@classmethod
def validate_image_url(cls, value: str) -> str:
return safe_url(value, local=True)
@field_validator("source_url", "license_url")
@classmethod
def validate_source_url(cls, value: str) -> str:
return safe_url(value)
@field_validator("variants")
@classmethod
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
for width, url in value.items():
if width not in {"640", "1280"}:
raise ValueError("Unsupported image variant")
safe_url(url, local=True)
return value
class CardImageRule(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
entity_type: EntityType
scope: MatchScope
value: str = Field(min_length=1, max_length=300)
label: str = Field(default="", max_length=500)
asset_id: str | None = Field(default=None, max_length=100)
suppress: bool = False
enabled: bool = True
@model_validator(mode="after")
def validate_action(self) -> Self:
if self.scope != "object":
self.value = self.value.casefold()
if self.suppress and (self.scope != "object" or self.asset_id):
raise ValueError("Only an object rule may suppress its image, without an asset")
if not self.suppress and not self.asset_id:
raise ValueError("Choose an image or suppress this object")
return self
def target(self) -> tuple[str, str, str]:
return self.entity_type, self.scope, self.value
class CardImageConfiguration(BaseModel):
model_config = ConfigDict(extra="forbid")
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
@model_validator(mode="after")
def validate_relations(self) -> Self:
assets = {asset.id for asset in self.assets}
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
self.rules
):
raise ValueError("Image and rule IDs must be unique")
targets = [rule.target() for rule in self.rules]
if len(set(targets)) != len(targets):
raise ValueError("Each target may have only one rule")
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
raise ValueError("Remove image associations before deleting the image")
return self
class CardImagePublish(BaseModel):
revision: str = Field(max_length=200)
config: CardImageConfiguration
class CardImagePreview(BaseModel):
entity_type: EntityType
entity_id: str = Field(min_length=1, max_length=300)
config: CardImageConfiguration

View File

@@ -0,0 +1,94 @@
"""Validated, source-backed observations for the compute comparison cards."""
from datetime import date
from typing import Annotated, Literal, Self
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
Text = Annotated[str, Field(min_length=1, max_length=500)]
class ConfigurationModel(BaseModel):
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
class ComputeSource(ConfigurationModel):
id: Identifier
title: Text
url: HttpUrl
publisher: Text
report_year: Year
published_at: date | None = None
locator: Text
@field_validator("url")
@classmethod
def public_reference(cls, value: HttpUrl) -> HttpUrl:
if value.username or value.password or len(str(value)) > 2000:
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
return value
class ComputeObservation(ConfigurationModel):
year: Year
metric: Literal["global_share", "absolute"]
source_id: Identifier
values: dict[str, Value] = Field(min_length=1, max_length=250)
@model_validator(mode="after")
def comparable_values(self) -> Self:
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
raise ValueError(
"Use countries from the canonical compute catalog, not regional totals"
)
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
raise ValueError("Global shares for one year must total at most 100 percent")
return self
class InnovationObservation(ConfigurationModel):
year: Year
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
source_id: Identifier
class ComputeConfiguration(ConfigurationModel):
reviewed_at: date
sources: list[ComputeSource] = Field(max_length=200)
observations: list[ComputeObservation] = Field(max_length=202)
innovation: list[InnovationObservation] = Field(max_length=101)
@model_validator(mode="after")
def consistent_evidence(self) -> Self:
ids = [source.id for source in self.sources]
if len(ids) != len(set(ids)):
raise ValueError("Report source IDs must be unique")
keys = [(row.metric, row.year) for row in self.observations]
if len(keys) != len(set(keys)):
raise ValueError("Each metric/year may have only one observation")
years = [row.year for row in self.innovation]
if len(years) != len(set(years)):
raise ValueError("Innovation observation years must be unique")
sources = {source.id: source for source in self.sources}
for row in [*self.observations, *self.innovation]:
source = sources.get(row.source_id)
if source is None:
raise ValueError(f"Missing report source: {row.source_id}")
if row.year > source.report_year:
raise ValueError("Observation year must not be after its report year")
if self.reviewed_at > date.today():
raise ValueError("Review date must not be in the future")
return self
class ComputeConfigurationWrite(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)
config: ComputeConfiguration
class ComputeConfigurationReset(ConfigurationModel):
revision: str = Field(min_length=1, max_length=80)

View File

@@ -36,6 +36,9 @@ class UserInDB(UserBase):
class UserResponse(UserBase):
# Bootstrap accounts have reserved .local email addresses. Validate email
# on writes, while allowing existing local identities to be serialized.
email: str
id: int
role: str
gatekeeper_groups: list[str] = Field(default_factory=list)

View File

@@ -0,0 +1,174 @@
"""Revision-checked image settings; new bundled assets never replace user edits."""
from uuid import uuid4
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.card_images import CardImageConfiguration
from app.services.card_image_defaults import (
bundled_configuration,
previous_bundled_assets,
previous_bundled_rules,
)
CATEGORY = "earth_card_images"
def effective_envelope(payload: dict | None) -> dict:
defaults, version = bundled_configuration()
config = (
CardImageConfiguration.model_validate(payload["config"])
if payload
else defaults.model_copy(deep=True)
)
if payload:
previous = previous_bundled_assets()
latest = {asset.id: asset for asset in defaults.assets}
for index, asset in enumerate(config.assets):
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
# Replace the old bundled source while preserving fields the user
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
if old is not None and asset.url == old.url and asset.id in latest:
overrides = {
key: value
for key, value in asset.model_dump().items()
if value != getattr(old, key)
}
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
known_assets = set(payload.get("known_assets", []))
asset_ids = {asset.id for asset in config.assets}
config.assets.extend(
asset.model_copy(deep=True)
for asset in defaults.assets
if asset.id not in known_assets and asset.id not in asset_ids
)
prior_rules = previous_bundled_rules()
latest_rules = {rule.id: rule for rule in defaults.rules}
for index, rule in enumerate(config.rules):
old_rule = prior_rules.get(rule.id)
if (
old_rule
and rule.id in latest_rules
and rule.target() == old_rule.target()
and rule.asset_id == old_rule.asset_id
):
overrides = {
key: value
for key, value in rule.model_dump().items()
if value != getattr(old_rule, key)
}
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
known_rules = set(payload.get("known_rules", []))
rule_ids = {rule.id for rule in config.rules}
targets = {rule.target() for rule in config.rules}
available = {asset.id for asset in config.assets}
config.rules.extend(
rule.model_copy(deep=True)
for rule in defaults.rules
if rule.id not in known_rules
and rule.id not in rule_ids
and rule.target() not in targets
and rule.asset_id in available
)
return {
"config": config.model_dump(mode="json"),
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
"is_default": payload is None,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return effective_envelope(record.payload if record else None)
async def publish_configuration(
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
) -> dict:
defaults, _ = bundled_configuration()
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
if any(
asset.id in pending and asset.rights_status != "permission_pending"
for asset in config.assets
):
raise HTTPException(422, "Bundled image permission is still unverified")
initial = {
"config": defaults.model_dump(mode="json"),
"revision": "default",
"known_assets": [],
"known_rules": [],
}
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=initial)
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if effective_envelope(record.payload)["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
before = record.payload
record.payload = {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="card_images.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": record.payload},
)
)
await db.commit()
return effective_envelope(record.payload)
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
assets = {asset.id: asset for asset in config.assets}
for scope in ("object", "model", "group", "category"):
value = context.get("entity_id" if scope == "object" else scope)
values = [value] if value else []
if scope == "category" and value != "*":
values.append("*")
for value in values:
value = value if scope == "object" else value.casefold()
rule = next(
(
rule
for rule in config.rules
if rule.enabled
and rule.entity_type == context["entity_type"]
and rule.scope == scope
and rule.value == value
),
None,
)
if not rule:
continue
match = {"scope": scope, "value": value, "rule_id": rule.id}
if rule.suppress:
return {"image": None, "match": match, "suppressed": True}
asset = assets.get(rule.asset_id)
if asset and asset.enabled:
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
return {"image": None, "match": None, "suppressed": False}

View File

@@ -0,0 +1,108 @@
"""Read immutable bundled assets without inserting machine-specific database rows."""
import hashlib
import json
from functools import lru_cache
from pathlib import Path
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
REPO_ROOT = Path(__file__).resolve().parents[3]
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
def _asset_from_manifest(item: dict) -> CardImageAsset:
pending = item.get("redistribution_status") == "permission_not_verified"
return CardImageAsset(
id=item["id"],
title=item["label"],
url=item["url"],
width=item["width"],
height=item["height"],
credit=item["credit"],
source_url=item.get("source_page") or "",
license=item["license"],
license_url=item.get("license_url") or "",
kind=item["kind"],
rights_status="permission_pending" if pending else "available",
enabled=item.get("enabled", not pending),
focus_x=item.get("focus_x", 50),
focus_y=item.get("focus_y", 50),
variants=item.get("variants", {}),
)
@lru_cache(maxsize=1)
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: [CardImageAsset.model_validate(item) for item in value]
for key, value in manifest.get("previous_defaults", {}).items()
}
@lru_cache(maxsize=1)
def previous_bundled_rules() -> dict[str, CardImageRule]:
manifest = json.loads(MANIFEST_PATH.read_bytes())
return {
key: CardImageRule.model_validate(value)
for key, value in manifest.get("previous_rules", {}).items()
}
@lru_cache(maxsize=1)
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
content = MANIFEST_PATH.read_bytes()
manifest = json.loads(content)
assets, rules = [], []
for item in manifest["assets"]:
asset = _asset_from_manifest(item)
assets.append(asset)
if not asset.enabled:
continue
hints = item.get("binding_hints", {})
for index, target in enumerate(hints.get("defaults", [])):
rules.append(
CardImageRule(
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
entity_type=target["entity_type"],
scope=target["scope"],
value=target["value"],
label=asset.title,
asset_id=asset.id,
)
)
for index, entity_id in enumerate(hints.get("entity_ids", [])):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-entity-{index}",
entity_type=item["entity_type"],
scope="object",
value=entity_id,
label=asset.title,
asset_id=asset.id,
)
)
for norad_id in hints.get("norad_ids", []):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-{norad_id}",
entity_type="satellite",
scope="object",
value=str(norad_id),
label=f"{asset.title} · NORAD {norad_id}",
asset_id=asset.id,
)
)
if group := hints.get("constellation_group"):
rules.append(
CardImageRule(
id=f"builtin-{asset.id}-group",
entity_type="satellite",
scope="group",
value=group,
label=asset.title,
asset_id=asset.id,
)
)
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()

View File

@@ -0,0 +1,95 @@
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
import hashlib
import io
import re
import warnings
from uuid import uuid4
from pathlib import Path
import os
from fastapi import HTTPException
from PIL import Image, ImageOps, UnidentifiedImageError
from app.schemas.card_images import CardImageAsset
from app.services.card_image_defaults import REPO_ROOT
ASSET_DIR = Path(
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
).resolve()
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
MAX_PIXELS = 32_000_000
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
def uploaded_path(filename: str) -> Path:
if not FILE_PATTERN.fullmatch(filename):
raise HTTPException(404, "Image not found")
return ASSET_DIR / filename
def store_image(content: bytes, title: str) -> CardImageAsset:
if len(content) > MAX_UPLOAD_BYTES:
raise HTTPException(413, "Image must be 12 MiB or smaller")
try:
with warnings.catch_warnings():
warnings.simplefilter("error", Image.DecompressionBombWarning)
image = Image.open(io.BytesIO(content))
extension = FORMATS.get(image.format)
if (
not extension
or image.width * image.height > MAX_PIXELS
or getattr(image, "is_animated", False)
):
raise ValueError("Unsupported or oversized raster")
image.load()
image = ImageOps.exif_transpose(image).convert(
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
)
except (
UnidentifiedImageError,
OSError,
ValueError,
Image.DecompressionBombError,
Image.DecompressionBombWarning,
) as error:
raise HTTPException(
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
) from error
digest = hashlib.sha256(content).hexdigest()
ASSET_DIR.mkdir(parents=True, exist_ok=True)
original = f"{digest}-original.{extension}"
original_path = uploaded_path(original)
if not original_path.exists():
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(original_path)
variants = {}
for size, content in render_variants(image).items():
filename = f"{digest}-{size}.webp"
path = uploaded_path(filename)
if not path.exists():
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
temporary.write_bytes(content)
temporary.replace(path)
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
return CardImageAsset(
id=f"upload-{digest[:24]}",
title=title or "Image",
url=f"/api/v1/earth/card-images/files/{original}",
variants=variants,
width=image.width,
height=image.height,
)
def render_variants(image: Image.Image) -> dict[str, bytes]:
result = {}
for size in (640, 1280):
resized = image.copy()
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
buffer = io.BytesIO()
resized.save(buffer, format="WEBP", quality=93, method=4)
result[str(size)] = buffer.getvalue()
return result

View File

@@ -0,0 +1,180 @@
"""Bounded catalog queries and type defaults for every Earth object card."""
from sqlalchemy import String, case, cast, func, literal, or_, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import (
COMPUTE_SITE_TYPES,
compute_site_type_expression,
compute_record_filter,
)
from app.models.bgp_incident import BGPIncident
from app.models.bgp_collector_location import BGPCollectorLocation
from app.models.collected_data import CollectedData
from app.models.earth_interactable import EarthInteractable
from app.models.vessel import VesselCurrentState
from app.schemas.card_images import EntityType, MatchScope
SOURCES = {
"satellite": ("celestrak_tle", "spacetrack_tle"),
"compute_center": ("top500", "epoch_ai_gpu"),
"cable": ("arcgis_cables",),
"landing_point": ("arcgis_landing_points",),
}
# Named constellation families, not inferred hardware models or generations.
SATELLITE_GROUP_PATTERNS = (
("STARLINK%", "starlink"),
("IRIDIUM%", "iridium-next"),
("BEIDOU-3%", "beidou-3"),
("BEIDOU%", "beidou"),
("QIANFAN%", "qianfan"),
("ONEWEB%", "oneweb"),
("KUIPER%", "kuiper"),
)
def collected_query(entity_type: EntityType):
record, metadata = CollectedData, CollectedData.extra_data
model = func.coalesce(
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
)
category = (
compute_site_type_expression(record)
if entity_type == "compute_center"
else literal(entity_type)
)
if entity_type == "satellite":
group = func.coalesce(
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
case(
*[
(func.upper(record.name).like(pattern), value)
for pattern, value in SATELLITE_GROUP_PATTERNS
],
else_="",
),
)
entity_id = metadata["norad_cat_id"].as_string()
else:
group = func.coalesce(
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
)
entity_id = record.source + ":" + record.source_id
return select(
record.name.label("name"),
cast(entity_id, String).label("entity_id"),
model.label("model"),
group.label("group"),
category.label("category"),
).where(
(
compute_record_filter(record)
if entity_type == "compute_center"
else record.source.in_(SOURCES[entity_type])
),
record.is_current.is_(True),
record.deleted_at.is_(None),
metadata["anonymous"].as_boolean().is_not(True),
)
def target_query(entity_type: EntityType):
if entity_type in SOURCES:
return collected_query(entity_type)
if entity_type == "vessel":
model = VesselCurrentState
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
conditions = []
elif entity_type == "bgp":
model = BGPIncident
identity, name, group, category = (
model.incident_key,
model.title,
model.source,
model.incident_type,
)
conditions = []
elif entity_type == "bgp_collector":
model = BGPCollectorLocation
identity, name, group, category = (
model.collector_id,
model.collector_id,
model.operator,
literal("bgp_collector"),
)
conditions = []
else:
model = EarthInteractable
identity, name, group, category = model.id, model.label, model.layer, model.kind
conditions = [model.is_deleted.is_(False)]
return select(
name.label("name"),
cast(identity, String).label("entity_id"),
literal("").label("model"),
func.coalesce(group, "").label("group"),
func.coalesce(category, entity_type).label("category"),
).where(*conditions)
def category_context(entity_type: EntityType) -> dict[str, str]:
return {
"entity_type": entity_type,
"entity_id": "",
"name": "",
"model": "",
"group": "",
"category": entity_type,
}
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
category = entity_id.partition(":")[2]
if category in COMPUTE_SITE_TYPES:
return {**category_context(entity_type), "category": category}
if entity_id == "__category__":
return category_context(entity_type)
if entity_type in SOURCES and entity_type != "satellite":
source, separator, source_id = entity_id.partition(":")
if not separator or not source or not source_id:
return category_context(entity_type)
# Match the indexed identity columns before projecting JSON metadata.
statement = collected_query(entity_type).where(
CollectedData.source == source, CollectedData.source_id == source_id
)
else:
catalog = target_query(entity_type).subquery()
statement = select(catalog).where(catalog.c.entity_id == entity_id)
row = (await db.execute(statement.limit(1))).mappings().first()
if not row:
# Old map snapshots can outlive a catalog refresh. They may use the type
# illustration, but must not claim an unverified exact-object identity.
return category_context(entity_type)
context = {key: str(value or "") for key, value in row.items()}
return {**context, "entity_type": entity_type}
async def search_targets(
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
) -> list[dict[str, str]]:
catalog = target_query(entity_type).subquery()
value = catalog.c["entity_id" if scope == "object" else scope]
label = catalog.c.name if scope == "object" else value
statement = select(value.label("value"), label.label("label")).where(
value.is_not(None), value != ""
)
if query:
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
statement = statement.where(
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
)
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
if scope == "category" and not query:
known = {item["value"] for item in items}
if entity_type == "compute_center":
items.extend(
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
)
items.insert(0, {"value": "*", "label": "*"})
return items

View File

@@ -31,6 +31,7 @@ class BaseCollector(ABC):
frequency_hours: int = 4
data_type: str = "generic"
fail_on_empty: bool = False
preserve_snapshot_on_failure: bool = False
def __init__(self):
self._current_task = None
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
context={"task_id": task_id, "raw_count": len(raw_data)},
)
data = self.transform(raw_data)
if self.fail_on_empty and not data:
raise RuntimeError(f"Collector {self.name} produced no valid records")
await self._log_collection_event(
"collector.phase.transforming.success",
"Collector transform phase completed",
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
raise
except Exception as e:
await db.rollback()
if snapshot_id is not None and self.preserve_snapshot_on_failure:
await self._rollback_incomplete_run(
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
)
task.status = JobStatus.FAILED.value
task.phase = "failed"
task.phase_message = str(e)
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
snapshot.status = SnapshotStatus.FAILED.value
if self.preserve_snapshot_on_failure:
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
snapshot.summary = {"error": str(e)}
await db.commit()

View File

@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
from app.services.collectors.base import BaseCollector
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
logger = get_logger(__name__, service="collector")
ACTIVE_GROUP = "active"
FALLBACK_GROUPS = (
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
)
FETCH_RETRY_ATTEMPTS = 3
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
_downloader = ResumableFileDownloader(
cache_namespace="celestrak",
default_accept="application/json",
preflight_head=False,
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
)
@property
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
if not is_final_attempt:
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
raise RuntimeError(
f"CelesTrak active satellite download failed after retries: {last_error}"
)
async def _fetch_fallback_groups(
self,
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
try:
for group in FALLBACK_GROUPS:
group_url = self._group_url(group)
cached_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if cached_path is not None:
body_path = cached_path
else:
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
try:
body_path = await self._downloader.download_file(
client,
group_url,
extension=".json",
accept="application/json",
validate_existing=self._validate_json_file,
)
except DownloadHTTPStatusError as exc:
if not self._is_not_updated_response(exc):
raise RuntimeError(
f"CelesTrak fallback group '{group}' download failed: {exc}"
) from exc
body_path = self._downloader.get_cached_file(
group_url,
".json",
validate_existing=self._validate_json_file,
)
if body_path is None:
raise RuntimeError(
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
) from exc
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
commit=True,
)
@staticmethod
def _validate_json_file(path: Path) -> bool:
@classmethod
def _validate_json_file(cls, path: Path) -> bool:
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
cls._load_active_payload(path)
except RuntimeError:
return False
return isinstance(data, list)
return True
async def _log_parse_failure(self, exc: Exception) -> None:
await emit_business_log(
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
),
)
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
@staticmethod
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
else:
invalid_count += 1
if invalid_count:
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
raise RuntimeError(
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
)
if not records:
raise RuntimeError("CelesTrak active payload contains no satellite records")
return records

View File

@@ -0,0 +1,82 @@
"""Quality gates shared by the two compute sources."""
from datetime import datetime
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import normalize_country
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.models.data_snapshot import DataSnapshot
from app.services.collectors.base import BaseCollector
class ComputeCollector(BaseCollector):
fail_on_empty = True
preserve_snapshot_on_failure = True
async def _create_snapshot(
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
) -> int:
previous_count = await db.scalar(
select(DataSnapshot.record_count)
.where(
DataSnapshot.source == self.name,
DataSnapshot.is_current.is_(True),
DataSnapshot.status == "success",
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
if previous_count and len(data) < previous_count * 0.6:
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
return await super()._create_snapshot(db, task_id, data, started_at)
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
if self.name == "top500":
await self._copy_verified_locations(db, data)
return await super()._save_data(db, data, **kwargs)
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
"""Keep historical location keys; copy only unambiguous exact identities.
Never migrate by rank: the machine occupying rank 1 changes over time.
"""
names = [item["name"] for item in data]
locations = (
(
await db.execute(
select(ComputeCenterLocationRecord).where(
ComputeCenterLocationRecord.source == self.name,
ComputeCenterLocationRecord.name.in_(names),
)
)
)
.scalars()
.all()
)
existing = {location.source_id for location in locations}
for item in data:
if item["source_id"] in existing:
continue
country = normalize_country(item["country"])
if not country:
continue
matches = [
location
for location in locations
if location.name == item["name"]
and normalize_country(location.country) == country
and (not location.site or location.site == item.get("metadata", {}).get("site"))
and not location.source_id.startswith("top500_system_")
]
if len(matches) != 1:
continue
old = matches[0]
values = {
column.name: getattr(old, column.name)
for column in old.__table__.columns
if column.name not in {"id", "source_id", "created_at", "updated_at"}
}
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
existing.add(item["source_id"])

View File

@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
import httpx
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
ValidateCallback = Callable[[Path], bool]
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
default_accept: str = "*/*",
cache_root: Path | None = None,
preflight_head: bool = True,
cache_ttl_seconds: float = 0,
) -> None:
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
self._user_agent = user_agent
self._default_accept = default_accept
self._preflight_head = preflight_head
self._cache_ttl_seconds = cache_ttl_seconds
@staticmethod
def _cache_key(url: str) -> str:
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
if not meta_path.exists():
return {}
try:
return json.loads(meta_path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
data = json.loads(meta_path.read_text(encoding="utf-8"))
return data if isinstance(data, dict) else {}
except json.JSONDecodeError, OSError:
return {}
@staticmethod
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
return last_modified == str(meta.get("last_modified") or "").strip()
return True
@staticmethod
def _response_info(response: httpx.Response) -> dict[str, Any]:
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
raw_length = response.headers.get("content-length")
encoded = response.headers.get("content-encoding", "identity") != "identity"
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": int(raw_length) if raw_length and not encoded else None,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
try:
response = await client.head(url)
if response.status_code >= 400:
return {}
content_length_raw = response.headers.get("content-length")
content_length = int(content_length_raw) if content_length_raw else None
return {
"etag": response.headers.get("etag"),
"last_modified": response.headers.get("last-modified"),
"content_length": content_length,
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
}
except (httpx.HTTPError, ValueError):
return self._response_info(response)
except httpx.HTTPError, ValueError:
return {}
async def download_file(
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
) -> Path:
self._cache_dir.mkdir(parents=True, exist_ok=True)
final_path, part_path, meta_path = self._cache_paths(url, extension)
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
size = cached_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return cached_path
meta = self._load_meta(meta_path)
remote = await self.fetch_remote_info(client, url)
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
expected_size = remote.get("content_length")
if final_path.exists():
local_size = final_path.stat().st_size
size_match = expected_size is None or local_size == expected_size
if self._validators_match(meta, remote) and size_match:
if (
(remote.get("etag") or remote.get("last_modified"))
and self._validators_match(meta, remote)
and size_match
):
if validate_existing and not validate_existing(final_path):
final_path.unlink(missing_ok=True)
else:
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
headers["If-Range"] = str(remote.get("last_modified"))
async with client.stream("GET", url, headers=headers) as response:
if response.status_code == 304 and final_path.exists():
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)
if response.status_code == 304:
if cached_path is None:
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
final_path.touch()
size = final_path.stat().st_size
if progress_callback:
await progress_callback(size, size)
return final_path
if response.status_code >= 400:
body = (await response.aread()).decode("utf-8", errors="replace")
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
else:
mode = "wb"
resume_from = 0
remote = self._response_info(response)
expected_size = remote.get("content_length")
downloaded = resume_from
last_emit_bytes = 0
last_emit_time = time.monotonic()
min_emit_bytes = (
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
max(expected_size // 150, 512 * 1024)
if expected_size and expected_size > 0
else 1024 * 1024
)
with part_path.open(mode) as f:
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
)
# Never replace the last good payload with an HTML error or invalid JSON.
if validate_existing and not validate_existing(part_path):
part_path.unlink(missing_ok=True)
raise RuntimeError(f"Downloaded file validation failed for {url}")
part_path.replace(final_path)
self._save_meta(
meta_path,
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
},
)
if validate_existing and not validate_existing(final_path):
raise RuntimeError(f"Downloaded file validation failed for {url}")
if progress_callback and expected_size and expected_size > 0:
await progress_callback(expected_size, expected_size)

View File

@@ -1,119 +1,135 @@
"""Epoch AI GPU Clusters Collector
Collects data from Epoch AI GPU clusters tracking.
https://epoch.ai/data/gpu-clusters
"""
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
import csv
import hashlib
import io
import json
import math
import re
from typing import Dict, Any, List
from datetime import UTC, datetime
from bs4 import BeautifulSoup
from urllib.parse import urlparse
import httpx
from app.services.collectors.base import BaseCollector
from app.core.countries import normalize_country
from app.services.collectors.compute_base import ComputeCollector
from app.services.epoch_location import assess_epoch_location
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
class EpochAIGPUCollector(BaseCollector):
class EpochAIGPUCollector(ComputeCollector):
name = "epoch_ai_gpu"
priority = "P0"
module = "L1"
frequency_hours = 6
data_type = "gpu_cluster"
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch Epoch AI GPU clusters data from webpage"""
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
async def fetch(self) -> list[dict]:
configured = self._resolved_url or EPOCH_CSV_URL
parsed = urlparse(configured)
url = (
EPOCH_CSV_URL
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
else configured
)
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
return self.parse_response(response.text)
return self.parse_response(response.text)
def parse_response(self, html: str) -> List[Dict[str, Any]]:
"""Parse Epoch AI webpage to extract GPU cluster data"""
data = []
soup = BeautifulSoup(html, "html.parser")
def parse_response(self, text: str) -> list[dict]:
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
if not required.issubset(reader.fieldnames or []):
raise ValueError(
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
)
entries = []
seen = set()
anonymous_counts = {}
for row in reader:
name = (row.get("Name") or "").strip()
country = normalize_country(row.get("Country"))
if not name:
raise ValueError("Epoch row has missing identity")
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
anonymous = "anonym" in name.lower()
if anonymous:
# Public anonymous rows have no facility identity. Preserve
# multiplicity within snapshots, without guessing names.
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
if source_id in seen:
raise ValueError("Epoch identity collision; previous snapshot retained")
seen.add(source_id)
raw_date = row.get("First Operational Date", "")
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
value = None
raw_value = row.get("16-bit OP/s (log)", "")
if raw_value:
exponent = float(raw_value)
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
raise ValueError("Epoch performance exponent out of bounds")
value = 10**exponent
anonymous = anonymous or (country == "中国" and not row.get("Location"))
location = row.get("Location")
location_assessment = assess_epoch_location(
location,
country=country,
raw_country=row.get("Country"),
)
entries.append(
{
"source_id": source_id,
"name": name,
"country": country,
"value": value,
"unit": "FLOP/s",
"reference_date": reference_date,
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
"metadata": {
"compute_contract": "v1",
"precision": "FP16",
"measurement": "theoretical",
"source_url": EPOCH_CSV_URL,
"raw_reference_date": raw_date,
"status": row.get("Status"),
"certainty": row.get("Certainty"),
"operator": row.get("Owner"),
"gpu_type": row.get("Chip type (primary)"),
"gpu_count": row.get("Chip quantity (primary)"),
"location": location,
"location_precision": location_assessment.precision,
"location_precision_reason": location_assessment.reason,
"raw_country": row.get("Country"),
"raw_performance": raw_value,
"performance_encoding": "log10",
"builds_upon": row.get("Builds Upon"),
"superseded_by": row.get("Superseded by"),
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
"include_in_standard_analysis": row.get(
"Include in Standard Analysis", ""
).lower()
== "true",
"rounded": country == "中国",
"anonymous": anonymous,
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
"country_unresolved": country is None,
"sources": [
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
],
},
}
)
return entries
# Try to find data table on the page
tables = soup.find_all("table")
for table in tables:
rows = table.find_all("tr")
for row in rows[1:]: # Skip header
cells = row.find_all(["td", "th"])
if len(cells) >= 5:
try:
cluster_name = cells[0].get_text(strip=True)
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
continue
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
country, city = self._parse_location(location_cell)
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
entry = {
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
"name": cluster_name,
"country": country,
"city": city,
"latitude": "",
"longitude": "",
"value": self._parse_performance(perf_cell),
"unit": "TFlop/s",
"metadata": {
"raw_data": perf_cell,
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
}
data.append(entry)
except (ValueError, IndexError, AttributeError):
continue
# If no table found, return sample data
if not data:
data = self._get_sample_data()
return data
def _parse_location(self, location: str) -> tuple:
"""Parse location string into country and city"""
if not location:
return "", ""
if "," in location:
parts = location.rsplit(",", 1)
city = parts[0].strip()
country = parts[1].strip() if len(parts) > 1 else ""
return country, city
return location, ""
def _parse_performance(self, perf: str) -> str:
"""Parse performance string to extract value"""
if not perf:
return "0"
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
if match:
return match.group(1).replace(",", "")
match = re.search(r"([\d,.]+)", perf)
if match:
return match.group(1).replace(",", "")
return "0"
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "epoch_sample_1",
"name": "Sample GPU Cluster",
"country": "United States",
"city": "San Francisco, CA",
"latitude": "",
"longitude": "",
"value": "1000",
"unit": "TFlop/s",
"metadata": {
"note": "Sample data - Epoch AI page structure may vary",
},
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
},
]
@staticmethod
def coordinate(value: str | None, bound: int) -> float | None:
if not value:
return None
number = float(value)
if not math.isfinite(number) or abs(number) > bound:
raise ValueError("Epoch coordinate out of bounds")
return number

View File

@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
async def fetch(self) -> list[dict[str, Any]]:
request_url = (self._resolved_url or "").strip()
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
"reference_date": datetime.now(UTC).isoformat(),
}
)
if len(normalized) >= max_sources:
if max_sources > 0 and len(normalized) >= max_sources:
break
return normalized

View File

@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
"""
import asyncio
import os
from typing import Dict, Any, List
from datetime import UTC, datetime
import os
from typing import Any, Dict, List
from urllib.parse import urlencode
import httpx
from urllib.parse import urlencode
from app.core.logging import get_logger
from app.services.collectors.base import HTTPCollector
# PeeringDB API key - read from environment variable
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
logger = get_logger(__name__, service="collector")
class PeeringDBIXPCollector(HTTPCollector):
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
if response.status_code == 429:
# Rate limited - wait and retry with exponential backoff
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
if response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
if response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
delay = base_delay * (2**attempt)
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
logger.warning_event(
"PeeringDB rate limited; retrying after delay",
event="collector.peeringdb.rate_limited",
context={"delay_seconds": delay, "attempt": attempt + 1},
)
await asyncio.sleep(delay)
last_error = "Rate limited"
continue
raise
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
logger.warning_event(
"PeeringDB collection failed after retries",
event="collector.peeringdb.retries_exhausted",
context={"max_retries": max_retries, "last_error": last_error},
)
return {}
async def fetch(self) -> List[Dict[str, Any]]:

View File

@@ -1,17 +1,21 @@
"""Space-Track TLE Collector
"""Space-Track TLE Collector.
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
API documentation: https://www.space-track.org/documentation
"""
import json
from typing import Dict, Any, List
import httpx
from typing import Any, Dict, List
from urllib.parse import urlparse
from app.services.collectors.base import BaseCollector
import httpx
from app.core.data_sources import get_data_sources_config
from app.core.logging import get_logger
from app.core.satellite_tle import build_tle_lines_from_elements
from app.services.collectors.base import BaseCollector
logger = get_logger(__name__, service="collector")
class SpaceTrackTLECollector(BaseCollector):
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
password = settings.SPACETRACK_PASSWORD
if not username or not password:
print("SPACETRACK: No credentials configured, using sample data")
logger.warning_event(
"Space-Track credentials are not configured; using sample data",
event="collector.spacetrack.credentials_missing",
)
return self._get_sample_data()
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
logger.info_event(
"Space-Track TLE fetch started",
event="collector.spacetrack.fetch.start",
)
try:
async with httpx.AsyncClient(
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
"password": password,
},
)
print(f"SPACETRACK: Login response status: {login_response.status_code}")
print(f"SPACETRACK: Login response URL: {login_response.url}")
logger.info_event(
"Space-Track login response received",
event="collector.spacetrack.login.response",
context={"status_code": login_response.status_code},
)
if login_response.status_code == 403:
print("SPACETRACK: Trying alternate login method...")
logger.warning_event(
"Space-Track login returned forbidden; trying alternate method",
event="collector.spacetrack.login.forbidden",
)
async with httpx.AsyncClient(
timeout=120.0,
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
) as alt_client:
await alt_client.get(f"{self.site_root}/")
form_data = {
"username": username,
"password": password,
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
}
alt_login = await alt_client.post(
self.login_url,
data={
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
"password": password,
},
)
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
logger.info_event(
"Space-Track alternate login response received",
event="collector.spacetrack.alt_login.response",
context={"status_code": alt_login.status_code},
)
if alt_login.status_code == 200:
tle_response = await alt_client.get(self.probe_url)
if tle_response.status_code == 200:
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records via alt method")
logger.info_event(
"Space-Track alternate query completed",
event="collector.spacetrack.alt_query.completed",
context={"record_count": len(data)},
)
return data
if login_response.status_code != 200:
print(f"SPACETRACK: Login failed, using sample data")
logger.warning_event(
"Space-Track login failed; using sample data",
event="collector.spacetrack.login.failed",
context={"status_code": login_response.status_code},
)
return self._get_sample_data()
tle_response = await client.get(self.probe_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:
print(f"SPACETRACK: Query failed, using sample data")
return self._get_sample_data()
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records")
return data
except Exception as e:
print(f"SPACETRACK: Error - {e}, using sample data")
return self._get_sample_data()
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
try:
async with httpx.AsyncClient(
timeout=120.0,
follow_redirects=True,
headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "application/json, text/html, */*",
"Accept-Language": "en-US,en;q=0.9",
},
) as client:
# First, visit the main page to get any cookies
await client.get(f"{self.site_root}/")
# Login to get session cookie
login_response = await client.post(
self.login_url,
data={
"identity": username,
"password": password,
},
logger.info_event(
"Space-Track TLE query response received",
event="collector.spacetrack.query.response",
context={"status_code": tle_response.status_code},
)
print(f"SPACETRACK: Login response status: {login_response.status_code}")
print(f"SPACETRACK: Login response URL: {login_response.url}")
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
if login_response.status_code != 200:
print(f"SPACETRACK: Login failed, using sample data")
return self._get_sample_data()
# Query for TLE data (get first 1000 satellites)
tle_response = await client.get(self.query_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:
print(f"SPACETRACK: Query failed, using sample data")
logger.warning_event(
"Space-Track TLE query failed; using sample data",
event="collector.spacetrack.query.failed",
context={"status_code": tle_response.status_code},
)
return self._get_sample_data()
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records")
logger.info_event(
"Space-Track TLE fetch completed",
event="collector.spacetrack.fetch.completed",
context={"record_count": len(data)},
)
return data
except Exception as e:
print(f"SPACETRACK: Error - {e}, using sample data")
logger.warning_event(
"Space-Track TLE fetch failed; using sample data",
event="collector.spacetrack.fetch.failed",
context={"error": str(e)},
)
return self._get_sample_data()
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:

View File

@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
"""
import asyncio
import math
import re
from urllib.parse import urljoin
from typing import Dict, Any, List
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
from app.services.collectors.compute_base import ComputeCollector
class TOP500Collector(BaseCollector):
class TOP500Collector(ComputeCollector):
name = "top500"
priority = "P0"
module = "L1"
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
response.raise_for_status()
entries = self.parse_response(response.text)
entries = self.parse_response(response.text, str(response.url))
semaphore = asyncio.Semaphore(8)
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
detail_url = ""
if link and link.get("href"):
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
detail_url = urljoin(str(top500_base_url), link.get("href"))
manufacturer = ""
if link and link.next_sibling:
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
return detail_map
def parse_response(self, html: str) -> List[Dict[str, Any]]:
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
"""Parse TOP500 HTML response"""
data = []
soup = BeautifulSoup(html, "html.parser")
canonical = soup.find("link", rel="canonical")
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
# Find the table with TOP500 data
table = None
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
if table:
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
if not reference_date:
raise ValueError("TOP500 list date missing; previous snapshot retained")
rows = table.select("tr")
for row in rows[1:]: # Skip header row
cells = row.find_all(["td", "th"])
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
cores = cells[2].get_text(strip=True).replace(",", "")
rmax_text = cells[3].get_text(strip=True)
rmax = self._parse_performance(rmax_text)
rmax = self._parse_performance(rmax_text, headers[3])
rpeak_text = cells[4].get_text(strip=True)
rpeak = self._parse_performance(rpeak_text)
rpeak = self._parse_performance(rpeak_text, headers[4])
power = ""
if len(cells) >= 6:
power = cells[5].get_text(strip=True).replace(",", "")
identity = re.search(r"/system/(\d+)", detail_url)
if not identity:
raise ValueError("TOP500 stable system ID missing")
entry = {
"source_id": f"top500_{rank}",
"source_id": f"top500_system_{identity[1]}",
"name": system_name,
"country": country,
"city": city,
"latitude": 0.0,
"longitude": 0.0,
"latitude": None,
"longitude": None,
"value": str(rmax),
"unit": "PFlop/s",
"unit": "FLOP/s",
"metadata": {
"compute_contract": "v1",
"precision": "FP64", "measurement": "LINPACK Rmax",
"source_url": detail_url, "list_url": date_url,
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
"rank": rank,
"cores": cores,
"rmax": rmax_text,
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
"site": site,
},
"_detail_url": detail_url,
"reference_date": "2025-11-01",
"reference_date": reference_date,
}
data.append(entry)
except (ValueError, IndexError, AttributeError) as e:
continue
# If scraping failed, return sample data for testing
if not data:
data = self._get_sample_data()
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
return data
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
return 0.0
return 0.0
def _parse_performance(self, text: str) -> float:
def _parse_performance(self, text: str, header: str = "") -> float:
"""Parse performance value from text (handles E, P, T suffixes)"""
text = text.strip().upper()
text = text.strip().upper().replace(",", "")
if not re.search(r"[EPTGMK]F", text):
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
if not unit:
raise ValueError("TOP500 performance unit missing")
text = f"{text} {unit[1]}FLOP/S"
multipliers = {
"E": 1e18,
"P": 1e15,
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
# Try simple float parsing
try:
return float(text.replace(",", ""))
value = float(text.replace(",", ""))
if not math.isfinite(value) or value < 0:
raise ValueError("Invalid TOP500 performance")
return value
except ValueError:
return 0.0
def _get_sample_data(self) -> List[Dict[str, Any]]:
"""Return sample data for testing when scraping fails"""
return [
{
"source_id": "top500_1",
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
"country": "United States",
"city": "Livermore, CA",
"latitude": 37.6819,
"longitude": -121.7681,
"value": "1742.00",
"unit": "PFlop/s",
"metadata": {
"rank": 1,
"cores": "11039616",
"rmax": "1742.00",
"rpeak": "2746.38",
"power": "29581",
"manufacturer": "HPE",
"site": "DOE/NNSA/LLNL",
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
"interconnect": "Slingshot-11",
"installation_year": "2025",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_2",
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
"country": "United States",
"city": "Oak Ridge, TN",
"latitude": 36.0107,
"longitude": -84.2663,
"value": "1353.00",
"unit": "PFlop/s",
"metadata": {
"rank": 2,
"cores": "9066176",
"rmax": "1353.00",
"rpeak": "2055.72",
"power": "24607",
"manufacturer": "HPE",
"site": "DOE/SC/Oak Ridge National Laboratory",
},
"reference_date": "2025-11-01",
},
{
"source_id": "top500_3",
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
"country": "United States",
"city": "Argonne, IL",
"latitude": 41.3784,
"longitude": -87.8600,
"value": "1012.00",
"unit": "PFlop/s",
"metadata": {
"rank": 3,
"cores": "9264128",
"rmax": "1012.00",
"rpeak": "1980.01",
"power": "38698",
"manufacturer": "Intel",
},
"reference_date": "2025-11-01",
},
]
raise ValueError("Invalid TOP500 performance") from None

View File

@@ -0,0 +1,200 @@
"""Canonical country and mutually exclusive regional compute participants."""
import re
from collections.abc import Iterable
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
from app.core.earth_geography_policy import display_country
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
# Explicit membership prevents a regional total from double-counting a country.
# APAC deliberately excludes China.
COMPARISON_GROUPS = (
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
(
"EUROPE",
"欧洲",
"Europe",
(
"GB",
"DE",
"FR",
"CH",
"ES",
"FI",
"HU",
"IE",
"IT",
"LU",
"MD",
"NL",
"PL",
"RO",
"RU",
"UA",
),
{"latitude": 54.0, "longitude": 20.0},
),
(
"APAC",
"亚太(不含中国)",
"Asia Pacific (excluding China)",
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
{"latitude": 36.0, "longitude": 130.5},
),
(
"OTHER",
"其他地区",
"Other",
(
"CA",
"AO",
"EG",
"KE",
"MZ",
"NG",
"UG",
"ZA",
"BR",
"CO",
"CR",
"DO",
"MX",
"AE",
"SY",
"TR",
),
{"latitude": 12.0, "longitude": 10.0},
),
)
def _country_code(canonical: str, aliases: list[str]) -> str | None:
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
return "GB" if canonical == "英国" else code
def build_country_catalog() -> list[dict]:
"""Build the policy-projected canonical country drilldown catalog."""
countries = []
for canonical, aliases in COUNTRY_ENTRIES:
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
continue
code = _country_code(canonical, aliases)
if code:
countries.append(
{
"code": code,
"name": canonical,
"name_en": aliases[0],
"kind": "country",
"members": [code],
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
"anchor_kind": "regional_label",
}
)
return countries
COUNTRY_CATALOG = build_country_catalog()
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
def _validate_group_members() -> None:
members = [
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
]
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
raise ValueError("compute comparison groups must cover country catalog exactly once")
_validate_group_members()
GROUP_BY_MEMBER = {
member: code
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
}
def build_catalog() -> list[dict]:
"""Return the five default participants before country drilldown entries."""
catalog: list[dict] = []
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
entry = (
dict(COUNTRY_PARTICIPANTS[code])
if code in COUNTRY_PARTICIPANTS
else {
"code": code,
"name": name,
"name_en": name_en,
"kind": "region",
"members": list(members),
"anchor": dict(anchor),
"anchor_kind": "regional_label",
}
)
entry.update(
{
"members": list(members),
"parent_group": name,
"parent_group_code": code,
"order": index,
"default_selected": True,
"color": PALETTE[index % len(PALETTE)],
}
)
catalog.append(entry)
for entry in COUNTRY_CATALOG:
if entry["code"] in DEFAULT_PARTICIPANTS:
continue
country = dict(entry)
group_code = GROUP_BY_MEMBER[country["code"]]
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
country.update(
{
"parent_group": group_name,
"parent_group_code": group_code,
"order": len(catalog),
"default_selected": False,
"color": PALETTE[len(catalog) % len(PALETTE)],
}
)
catalog.append(country)
return catalog
CATALOG = build_catalog()
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
"""Resolve country or region participant codes to canonical country codes."""
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
def source_country_names(codes: Iterable[str]) -> list[str]:
"""Return unique canonical source labels, respecting China and Serbia ownership."""
selected_members = {
member for members in participant_members(codes).values() for member in members
}
selected_names = {COUNTRIES[code] for code in selected_members}
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
"""Map stored source-country labels to their requested participant code."""
owner_by_country = {
member: participant
for participant, members in participant_members(codes).items()
for member in members
}
code_by_name = {name: code for code, name in COUNTRIES.items()}
return {
name: owner_by_country[code_by_name[display_country(name)]]
for name in source_country_names(codes)
}

View File

@@ -3,7 +3,12 @@
This module is a thin domain wrapper that wires up
:mod:`app.services.location` for compute centers:
SourceCoordinates
Confirmed stored location -> trustworthy source coordinates -> exact
official facility registration -> source city/site coordinates.
Epoch AI coordinates are admitted only at the grain asserted by its official
``Location`` column. Country/region/empty locations remain unresolved even
when the CSV happens to contain a latitude/longitude pair.
The online Nominatim step is intentionally reserved for the user-triggered
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
@@ -28,9 +33,12 @@ import httpx
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.compute_types import compute_record_filter
from app.core.collected_data_fields import get_record_field
from app.models.collected_data import CollectedData
from app.models.compute_center_location import ComputeCenterLocationRecord
from app.services.compute_center_verified_sites import get_verified_compute_site
from app.services.epoch_location import assess_epoch_location
from app.services.location import (
LocationCandidate,
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
async def refresh_compute_center_location_cache(
session: AsyncSession,
*,
as_of: datetime | None = None,
) -> dict[str, dict[str, Any]]:
result = await session.execute(select(ComputeCenterLocationRecord))
query = select(ComputeCenterLocationRecord)
if as_of is not None:
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
result = await session.execute(query)
records = result.scalars().all()
cache = {}
for record in records:
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
)
class ComputeCenterSourceCoordinatesResolver:
"""Use source coordinates only at the grain asserted by the source text."""
name = "source_coordinates"
def __init__(self) -> None:
self._default = SourceCoordinatesResolver()
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
if extra.get("source") != "epoch_ai_gpu":
return self._default.resolve(query)
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
return ResolverOutput()
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput(
attempted_queries=(f"source_coordinates:{assessment.precision}",)
)
output = self._default.resolve(query)
if not output.candidates:
return output
candidate = output.candidates[0]
source_location = extra.get("source_location")
source_location_text = (
source_location.strip() if isinstance(source_location, str) else ""
)
source_note = assessment.reason
if source_location_text:
source_note = f"{source_note}; Epoch Location: {source_location_text}"
return ResolverOutput(
candidates=(
LocationCandidate(
**{
**candidate.__dict__,
"precision": assessment.precision,
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
"source_note": source_note,
"needs_confirmation": assessment.precision != "precise",
"matched_location_name": source_location_text
or candidate.matched_location_name,
}
),
),
attempted_queries=output.attempted_queries,
)
class StoredComputeCenterLocationResolver:
"""Resolve a compute center through the DB-backed current-location cache."""
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
longitude = parse_float(stored.get("longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
return ResolverOutput()
precision = stored.get("precision") or "city"
if (
extra.get("source") == "epoch_ai_gpu"
and stored.get("location_source") == "source_coordinates"
):
assessment = assess_epoch_location(
extra.get("source_location"),
country=query.country,
raw_country=extra.get("source_raw_country"),
)
if not assessment.is_renderable:
return ResolverOutput()
precision = assessment.precision
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=latitude,
longitude=longitude,
display_name=stored.get("name") or query.name or "Compute center",
precision=stored.get("precision") or "city",
precision=precision,
confidence=float(stored.get("confidence") or 0.85),
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
source=self.name,
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
matched_location_name=stored.get("site") or stored.get("name") or query.name,
location_verified_at=stored.get("verified_at"),
suggested_registry_entry=None,
raw_payload={
"stored_location_source": stored.get("location_source"),
"verification_status": stored.get("verification_status"),
},
),
)
)
class VerifiedComputeSiteResolver:
"""Resolve an exact source identity to an evidence-backed facility site."""
name = "official_facility_registry"
def resolve(self, query: LocationQuery) -> ResolverOutput:
extra = query.extra or {}
source = coerce_str(extra.get("source"))
source_id = coerce_str(extra.get("source_id"))
if not source or not source_id:
return ResolverOutput()
site = get_verified_compute_site(source, source_id)
if not site:
return ResolverOutput()
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
source_note = coerce_str(site.get("source_note"))
if source_urls:
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
return ResolverOutput(
candidates=(
LocationCandidate(
latitude=float(site["latitude"]),
longitude=float(site["longitude"]),
display_name=coerce_str(site.get("matched_location_name"))
or query.name
or "Compute center",
precision=coerce_str(site.get("precision")) or "site",
confidence=1.0,
query=f"official_facility_registry::{source}:{source_id}",
source=self.name,
source_note=source_note or None,
matched_fields=("source", "source_id"),
needs_confirmation=False,
city=coerce_str(site.get("city")) or None,
region=None,
country=coerce_str(site.get("country")) or query.country,
matched_location_name=coerce_str(site.get("matched_location_name"))
or None,
location_verified_at=coerce_str(site.get("verified_at")) or None,
suggested_registry_entry=None,
raw_payload={
"address": site.get("address"),
"coordinate_basis": site.get("coordinate_basis"),
"sources": source_urls,
},
),
)
)
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
return head or text
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
name = coerce_str(getattr(record, "name", None))
return {
"source": coerce_str(getattr(record, "source", None)),
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
or metadata.get("manufacturer")
),
"organization": coerce_str(metadata.get("organization")),
"source_location": metadata.get("location"),
"source_raw_country": metadata.get("raw_country"),
}
def _context_to_query(
context: dict[str, str],
context: dict[str, Any],
*,
source_lat: float | None = None,
source_lon: float | None = None,
@@ -444,6 +578,8 @@ def _context_to_query(
"site": context.get("site") or "",
"operator": context.get("operator") or "",
"organization": context.get("organization") or "",
"source_location": context.get("source_location"),
"source_raw_country": context.get("source_raw_country"),
},
)
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
return plan
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
stored_confirmed = (
candidate.source == "stored_compute_center_location"
and not candidate.needs_confirmation
and stored_origin != "source_coordinates"
)
if stored_confirmed:
source_rank = 0
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
source_rank = 1
elif candidate.source == "official_facility_registry":
source_rank = 2
elif candidate.source == "source_coordinates":
source_rank = 3
elif candidate.source == "stored_compute_center_location":
source_rank = 4
else:
source_rank = 5
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
return (source_rank, precision_rank, -float(candidate.confidence or 0))
COMPUTE_CENTER_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
StoredComputeCenterLocationResolver(),
VerifiedComputeSiteResolver(),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
" or stored compute-center location."
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
[
SourceCoordinatesResolver(),
ComputeCenterSourceCoordinatesResolver(),
VerifiedComputeSiteResolver(),
ROROrganizationResolver(),
NominatimResolver(
query_plan_builder=_compute_center_query_plan,
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
geocoder=lambda q: _geocode_online(q),
),
],
sort_key=_compute_center_candidate_sort_key,
failure_reason=(
"Could not resolve to city-level coordinates from source coords"
", ROR organization lookup, or online geocoding."
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
_GEOGRAPHY_MODE_BY_SOURCE = {
"source_coordinates": "source_coordinates",
"stored_compute_center_location": "stored_compute_center_location",
"official_facility_registry": "official_facility_registry",
"ror_organization_registry": "ror_organization",
"nominatim_online_geocode": "online_geocode",
}
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
def _candidate_to_location(
candidate: LocationCandidate,
*,
context: dict[str, str],
context: dict[str, Any],
) -> ComputeCenterLocation:
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
@@ -594,7 +758,7 @@ def _candidate_to_location(
def _diagnostic_for(
record: Any,
context: dict[str, str],
context: dict[str, Any],
*,
failure_reason: str,
attempted_queries: tuple[str, ...] = (),
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
organization: str | None = None,
) -> LocationQuery:
name_value = coerce_str(name)
context: dict[str, str] = {
context: dict[str, Any] = {
"source": coerce_str(source),
"source_id": coerce_str(source_id),
"name": name_value,
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
"""Seed stored compute-center locations only from real source coordinates."""
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
.where(compute_record_filter(CollectedData))
.where(CollectedData.is_current.is_(True))
)
result = await session.execute(stmt)
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
longitude = parse_float(get_record_field(record, "longitude"))
if latitude in (None, 0.0) or longitude in (None, 0.0):
continue
metadata = record.extra_data or {}
precision = "precise"
confidence = 1.0
needs_confirmation = False
source_note = "Seeded from source-provided compute-center coordinates"
if source_value == "epoch_ai_gpu":
assessment = assess_epoch_location(
metadata.get("location"),
country=get_record_field(record, "country"),
raw_country=metadata.get("raw_country"),
)
if not assessment.is_renderable:
continue
precision = assessment.precision
confidence = 1.0 if precision == "precise" else 0.9
needs_confirmation = precision != "precise"
source_note = assessment.reason
existing = await session.scalar(
select(ComputeCenterLocationRecord)
.where(ComputeCenterLocationRecord.source == source_value)
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
)
if existing:
continue
metadata = record.extra_data or {}
session.add(
ComputeCenterLocationRecord(
source=source_value,
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
country=coerce_str(get_record_field(record, "country")) or None,
latitude=latitude,
longitude=longitude,
precision="precise",
confidence=1.0,
precision=precision,
confidence=confidence,
location_source="source_coordinates",
source_note="Seeded from source-provided compute-center coordinates",
source_note=source_note,
raw_payload={
"record_id": getattr(record, "id", None),
"source": source_value,
"source_id": source_id,
},
needs_confirmation=False,
needs_confirmation=needs_confirmation,
verification_status="source_provided",
verified_at=None,
)

View File

@@ -0,0 +1,44 @@
{
"version": 1,
"sites": [
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
"latitude": 35.380924,
"longitude": 139.625008,
"precision": "site",
"location_source": "official_facility_registry",
"country": "日本",
"city": "Yokohama",
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
]
},
{
"source": "epoch_ai_gpu",
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
"latitude": 41.718109383287235,
"longitude": -87.984498648496,
"precision": "site",
"location_source": "official_facility_registry",
"country": "美国",
"city": "Argonne",
"matched_location_name": "Argonne Leadership Computing Facility",
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
"verified_at": "2026-09-21T17:06:31Z",
"coordinate_basis": "official_access_map_center",
"sources": [
"https://www.alcf.anl.gov/systems/sophia",
"https://docs.alcf.anl.gov/sophia/",
"https://www.alcf.anl.gov/about/visiting-alcf"
]
}
]
}

View File

@@ -0,0 +1,26 @@
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
import json
from copy import deepcopy
from functools import lru_cache
from pathlib import Path
@lru_cache(maxsize=1)
def _site_index() -> dict[tuple[str, str], dict]:
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
index = {}
for site in payload["sites"]:
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
raise ValueError("Invalid verified compute facility coordinate")
for source_id in site["source_ids"]:
key = (site["source"], source_id)
if key in index:
raise ValueError("Duplicate verified compute facility identity")
index[key] = site
return index
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
site = _site_index().get((source, source_id))
return deepcopy(site) if site else None

View File

@@ -0,0 +1,99 @@
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
import hashlib
from pathlib import Path
from uuid import uuid4
from app.core.request_context import get_request_id
from app.models.system_log import AuditLog
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import COUNTRY_CATALOG
from fastapi import HTTPException
from sqlalchemy import select
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.ext.asyncio import AsyncSession
CATEGORY = "compute_comparison"
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
def default_envelope() -> dict:
return {
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
"revision": DEFAULT_REVISION,
"is_default": True,
}
async def read_configuration(db: AsyncSession) -> dict:
record = (
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
).scalar_one_or_none()
return dict(record.payload) if record else default_envelope()
def configuration_response(envelope: dict) -> dict:
return {
**envelope,
"countries": [
{key: country[key] for key in ("code", "name", "name_en")}
for country in COUNTRY_CATALOG
],
}
async def publish_configuration(
db: AsyncSession,
*,
revision: str,
config: ComputeConfiguration,
actor: User,
reset: bool = False,
) -> dict:
# The unique category plus row lock also serializes concurrent first writes.
await db.execute(
insert(SystemSetting)
.values(category=CATEGORY, payload=default_envelope())
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
)
record = (
await db.execute(
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
)
).scalar_one()
if record.payload["revision"] != revision:
await db.rollback()
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
envelope = (
default_envelope()
if reset
else {
"config": config.model_dump(mode="json"),
"revision": uuid4().hex,
"is_default": reset,
}
)
before = record.payload
if reset:
await db.delete(record)
else:
record.payload = envelope
# Audit and configuration commit together: a published edit must be attributable.
db.add(
AuditLog(
actor_id=actor.id,
actor_name=actor.username,
action="compute_comparison.reset" if reset else "compute_comparison.update",
target_type="system_setting",
target_id=CATEGORY,
result="success",
request_id=get_request_id(),
details={"before": before, "after": envelope},
)
)
await db.commit()
return envelope

View File

@@ -0,0 +1,122 @@
"""Versioned, source-backed annual observations for compute comparisons."""
from copy import deepcopy
from datetime import date
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_catalog import (
CATALOG,
DEFAULT_PARTICIPANTS,
PARTICIPANTS,
participant_members,
)
from app.services.compute_configuration import DEFAULT_CONFIGURATION
def _participant_value(
members: tuple[str, ...], observations: dict[str, float | int], mode: str
) -> tuple[float | int | None, dict]:
covered = [member for member in members if member in observations]
missing = [member for member in members if member not in observations]
coverage = {
"status": "complete" if not missing else "incomplete",
"required_members": list(members),
"covered_members": covered,
"missing_members": missing,
}
if missing:
return None, coverage
if mode == "bilateral_share":
if (
not set(members).issubset({"CN", "US"})
or not {"CN", "US"}.issubset(observations)
or observations["US"] + observations["CN"] <= 0
):
coverage.update(
{
"status": "incompatible",
"reason": "bilateral_share_requires_CN_and_US_observations",
}
)
return None, coverage
return (
round(
sum(observations[member] for member in members)
/ (observations["US"] + observations["CN"])
* 100,
2,
),
coverage,
)
return sum(observations[member] for member in members), coverage
def compute_history(
countries: list[str],
mode: str = "global_share",
config: ComputeConfiguration | None = None,
) -> dict:
"""Aggregate only fully covered, same-year compatible observations."""
config = config if config is not None else DEFAULT_CONFIGURATION
metric = "global_share" if mode == "bilateral_share" else mode
annual = {row.year: row for row in config.observations if row.metric == metric}
gaps = {row.year: row for row in config.innovation}
observed_years = [row.year for row in config.observations] + list(gaps)
years = tuple(
range(
min(observed_years, default=date.today().year),
max([date.today().year, *observed_years]) + 1,
)
)
members_by_participant = participant_members(countries)
rows = []
for year in years:
record = annual.get(year)
observations = record.values if record else {}
values, coverage = {}, {}
for code, members in members_by_participant.items():
values[code], coverage[code] = _participant_value(members, observations, mode)
evidence_id = record.source_id if record else None
rows.append(
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
)
selected_members = {member for members in members_by_participant.values() for member in members}
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
return {
"version": f"compute-history-{config.reviewed_at}",
"provenance": "curated_reports",
"reviewed_at": config.reviewed_at.isoformat(),
"time_basis": "retrospective",
"years": years,
"mode": mode,
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
mode
],
"catalog": deepcopy(CATALOG),
"default_participants": list(DEFAULT_PARTICIPANTS),
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
"compute": rows,
"innovation": [
{
"year": year,
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
"evidence_id": gaps[year].source_id
if year in gaps and has_innovation_scope
else None,
}
for year in years
],
"evidence": {
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
},
"innovation_scope": ["US", "CN"],
"limitations": [
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
"全球份额不从中美或其他局部成员反推剩余地区。",
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
],
}

View File

@@ -0,0 +1,125 @@
{
"reviewed_at": "2026-09-22",
"sources": [
{
"id": "caict-2021",
"title": "中国算力发展指数白皮书(2021)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2021,
"published_at": null,
"locator": "正文 11 页图 4(2020 年全球份额)"
},
{
"id": "caict-2022",
"title": "中国算力发展指数白皮书(2022)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2022,
"published_at": null,
"locator": "正文 13 页图 5(2021 年全球份额)"
},
{
"id": "caict-2023",
"title": "中国算力发展指数白皮书(2023)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2023,
"published_at": null,
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
},
{
"id": "caict-2024",
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
"publisher": "中国信息通信研究院",
"report_year": 2024,
"published_at": null,
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
},
{
"id": "innovation-2026",
"title": "全球人工智能创新指数报告 2026:科技日报报道",
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
"publisher": "科技日报(新浪转载)",
"report_year": 2026,
"published_at": "2026-07-17",
"locator": "2023–2025 年美国减中国的指数分差"
}
],
"observations": [
{
"year": 2020,
"metric": "global_share",
"source_id": "caict-2021",
"values": {
"US": 36,
"CN": 31,
"JP": 6
}
},
{
"year": 2021,
"metric": "global_share",
"source_id": "caict-2022",
"values": {
"US": 34,
"CN": 33,
"JP": 5
}
},
{
"year": 2022,
"metric": "global_share",
"source_id": "caict-2023",
"values": {
"US": 34,
"CN": 33,
"JP": 4
}
},
{
"year": 2023,
"metric": "global_share",
"source_id": "caict-2024",
"values": {
"US": 41,
"CN": 31,
"JP": 4
}
},
{
"year": 2022,
"metric": "absolute",
"source_id": "caict-2023",
"values": {
"CN": 302
}
},
{
"year": 2023,
"metric": "absolute",
"source_id": "caict-2024",
"values": {
"CN": 435
}
}
],
"innovation": [
{
"year": 2023,
"value": 22.02,
"source_id": "innovation-2026"
},
{
"year": 2024,
"value": 19.96,
"source_id": "innovation-2026"
},
{
"year": 2025,
"value": 17.95,
"source_id": "innovation-2026"
}
]
}

View File

@@ -45,7 +45,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
@@ -56,7 +56,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),

View File

@@ -6,8 +6,10 @@ import asyncio
import hashlib
import json
import shutil
import subprocess
import sys
from datetime import UTC, datetime
from functools import lru_cache
from pathlib import Path
from typing import Any
from uuid import uuid4
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
PMTILES_ARTIFACT_PATH = (
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
)
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
POLICY_FALLBACK_PATH = (
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
)
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
@lru_cache(maxsize=16)
def _read_versioned_asset_cached(
path: Path, mtime_ns: int, size_bytes: int
) -> tuple[dict[str, Any], str]:
payload = _read_json(path)
return payload, _stable_json_hash(payload)
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
if not path.exists():
return {}, ""
stat = path.stat()
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
path = REPO_ROOT / source["path"]
payload = _read_json(path)
features = payload.get("features") if isinstance(payload, dict) else None
return {
"type": "FeatureCollection",
"features": features if isinstance(features, list) else [],
}
def _write_high_precision_geojson_manifest(
sources: list[dict[str, Any]],
build_input_hash: str,
missing_tools: list[str],
) -> dict[str, Any]:
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
coastline = next(source for source in sources if source["kind"] == "coastline")
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
admin0_payload = _load_source_feature_collection(admin0)
coastline_payload = _load_source_feature_collection(coastline)
claim_payload = _load_source_feature_collection(claim_lines)
for feature in coastline_payload["features"]:
props = feature.setdefault("properties", {})
if isinstance(props, dict):
props["PLANET_LAYER"] = "coastline"
base_payload = {
"type": "FeatureCollection",
"features": [*admin0_payload["features"], *coastline_payload["features"]],
}
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
_write_json(base_path, base_payload)
_write_json(hover_path, admin0_payload)
_write_json(claim_path, claim_payload)
manifest = {
"version": "natural-earth-v1",
"builtAt": _utc_now_iso(),
"tileProvider": "geojson-high-precision",
"format": "geojson-directory",
"buildInputHash": build_input_hash,
"base": base_path.name,
"hoverIndex": hover_path.name,
"claimLine": claim_path.name,
"sourceFeatureCount": {
"admin0": len(admin0_payload["features"]),
"coastline": len(coastline_payload["features"]),
"claimLines": len(claim_payload["features"]),
},
"pmtiles": None,
"missingTools": missing_tools,
}
result = subprocess.run(
[
sys.executable,
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
"--admin0-source", admin0["path"],
"--coastline-source", coastline["path"],
"--claims-source", claim_lines["path"],
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
"--build-input-hash", build_input_hash,
"--pov-policy", _relative(POV_POLICY_PATH),
"--pov-geometry", str(POV_GEOMETRY_PATH),
],
cwd=REPO_ROOT,
capture_output=True,
text=True,
)
if result.returncode:
raise EarthBoundaryBuildError(
result.stderr.strip() or "Earth boundary policy build failed",
code="policy_build_failed",
)
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
manifest["missingTools"] = missing_tools
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
return manifest
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
for source in source_manifest.get("sources", [])
],
"pov_policy": source_manifest.get("povPolicy"),
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
"build_config": BUILD_CONFIG,
}
)
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
bool(boundary_manifest)
and boundary_manifest.get("buildInputHash") == build_input_hash
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and _policy_verified(boundary_manifest)
and PMTILES_ARTIFACT_PATH.exists()
and _pmtiles_hover_exists(boundary_manifest)
)
def _policy_verified(manifest: dict[str, Any]) -> bool:
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
coverage = manifest.get("povPolicy") or {}
return (
policy.get("profile") == "china-pov-v1"
and coverage.get("profile") == policy["profile"]
and coverage.get("sha256") == policy_hash
and coverage.get("reviewedGeometrySourceCommit")
== geometry.get("sourceCommit")
and coverage.get("geometrySha256") == geometry_hash
and coverage.get("applied") is True
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
<= set(coverage.get("appliedRuleIds") or [])
)
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
paths = [manifest.get("base"), manifest.get("hoverIndex")]
if manifest.get("chinaClaims", {}).get("available"):
paths.append(manifest.get("claimLine"))
for asset in paths:
if not isinstance(asset, str) or Path(asset).name != asset:
return False
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
return False
return True
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
hover = manifest.get("hoverIndex")
return (
isinstance(hover, str)
and Path(hover).name == hover
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
)
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
high_precision_ready = (
manifest_exists
and _policy_verified(boundary_manifest)
and (
(
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
and pmtiles_exists
and _pmtiles_hover_exists(boundary_manifest)
)
or (
boundary_manifest.get("tileProvider") == "geojson-high-precision"
and _geojson_artifacts_exist(boundary_manifest)
)
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
)
)
legacy_exists = LEGACY_GEOJSON_PATH.exists()
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
_versioned_asset(POLICY_FALLBACK_PATH)[0]
)
provider = (
boundary_manifest.get("tileProvider")
if high_precision_ready
else "legacy-geojson" if legacy_exists else "missing"
else "policy-geojson-fallback" if legacy_exists else "missing"
)
return {
"provider": provider,
"high_precision_ready": high_precision_ready,
"fallback_available": legacy_exists,
"policy_publishable": bool(
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
),
"config_source": config_source,
"config_path": _relative(LOCAL_CONFIG_PATH),
"config_exists": LOCAL_CONFIG_PATH.exists(),
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
"tileProvider": boundary_manifest.get("tileProvider"),
"buildInputHash": boundary_manifest.get("buildInputHash"),
"builtAt": boundary_manifest.get("builtAt"),
"povPolicy": boundary_manifest.get("povPolicy"),
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
},
"pmtiles": {
"path": _relative(PMTILES_ARTIFACT_PATH),
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
},
"legacy": {
"path": _relative(LEGACY_GEOJSON_PATH),
"path": _relative(POLICY_FALLBACK_PATH),
"exists": legacy_exists,
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
},
"source_manifest": {
"path": _relative(SOURCE_MANIFEST_PATH),
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
build_input_hash,
"--pov-policy",
_relative(POV_POLICY_PATH),
"--pov-geometry",
str(POV_GEOMETRY_PATH),
]
)
if progress_callback:

View File

@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
source_has_adapter = bool(get_earth_update_layers_for_source(source))
effective_source = source if source_has_adapter else (table_name if table_name else source)
operation = payload.get("operation")
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
refresh_strategy = "reload"
update: dict[str, Any] = {
"event": "earth.layer.changed",
"action": "database_changed",
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
operation = payload.get("operation")
if operation:
self.operations.add(str(operation))
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
self.refresh_strategy = "reload"
entity_keys = payload.get("entity_keys")
if not isinstance(entity_keys, list):
entity_key = payload.get("entity_key")
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
if not update:
return False
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
from app.core.websocket.broadcaster import broadcaster
keys = payload.get("entity_keys")
if not isinstance(keys, list):
keys = [payload.get("entity_key")]
broadcaster.enqueue_vessel_update({
"action": "remove",
"vessels": [{"mmsi": key} for key in keys if key is not None],
})
source = update["source"]
pending = self._pending.get(source)
if pending is None:

View File

@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
layers=("vessels",),
cache_patterns=("vessels*", "summary*"),
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
refresh_strategy="delta",
),
EarthLayerAdapter(
sources=frozenset(

View File

@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
import codecs
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from email.utils import parsedate_to_datetime
@@ -865,9 +866,9 @@ def _get_locale_text(
if isinstance(fallback, dict):
value = _coerce_str(fallback.get(key))
if value:
if locale == "en-US" and _contains_cjk_text(value):
return ""
return value
if locale == "en-US" and _is_chinese_language(item.content_language):
return item.title if key == "title" else item.summary
return ""
@@ -1800,6 +1801,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
return ""
def _decode_feed_response(response: httpx.Response) -> str:
content = response.content
for marker, encoding in (
(codecs.BOM_UTF32_LE, "utf-32"),
(codecs.BOM_UTF32_BE, "utf-32"),
(codecs.BOM_UTF16_LE, "utf-16"),
(codecs.BOM_UTF16_BE, "utf-16"),
(codecs.BOM_UTF8, "utf-8-sig"),
):
if content.startswith(marker):
return content.decode(encoding)
declaration = re.match(
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
content,
re.IGNORECASE,
)
encoding = response.charset_encoding or (
declaration.group(1).decode("ascii") if declaration else "utf-8"
)
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
return content.decode(encoding)
def _parse_feed_entries(
xml_text: str,
source: NewsFeedSource,
@@ -2453,7 +2478,9 @@ async def _fetch_single_feed_url(
feed=feed,
)
try:
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
items = _parse_feed_entries(
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
)
except Exception as exc:
error, status = _format_source_fetch_error(
status_code=status_code,
@@ -2461,7 +2488,7 @@ async def _fetch_single_feed_url(
body=response.text,
parsed_count=0,
)
if error is None:
if status == "empty":
error = str(exc)
status = "format_error"
return source, [], error, _source_health_result(

View File

@@ -0,0 +1,198 @@
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
from __future__ import annotations
from dataclasses import dataclass
import math
import re
from typing import Any
from app.core.countries import normalize_country
_COUNTRY_LABEL_OVERRIDES = {
"korea republic of": "韩国",
}
_REGION_ONLY_LABELS = {
"europe",
"iowa",
"maryland",
"new mexico",
"texas",
"the nordics",
"virginia",
"washington",
}
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
re.IGNORECASE,
)
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
re.IGNORECASE | re.DOTALL,
)
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
)
_NUMBERED_STREET_PATTERN = re.compile(
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
r"highway|hwy|way|parkway|pkwy|chome)\b|"
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
re.IGNORECASE,
)
_POSTAL_CODE_PATTERN = re.compile(
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
re.IGNORECASE,
)
_SITE_CONTEXT_PATTERN = re.compile(
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
re.IGNORECASE,
)
_COORDINATE_CUE_PATTERN = re.compile(
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
re.IGNORECASE | re.DOTALL,
)
@dataclass(frozen=True)
class EpochLocationAssessment:
precision: str
reason: str
@property
def is_renderable(self) -> bool:
return self.precision in {"precise", "site", "city"}
def _normalized_label(value: Any) -> str:
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
def _country_from_label(value: Any) -> str | None:
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
_normalized_label(value)
)
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
if isinstance(raw, bool):
return None
try:
number = float(raw)
except (TypeError, ValueError):
return None
return number if math.isfinite(number) and abs(number) <= bound else None
def _has_explicit_coordinate_pair(text: str) -> bool:
hemispheres = {}
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
hemisphere = match.group("hemisphere").upper()
degrees = int(match.group("degrees"))
minutes = int(match.group("minutes"))
seconds = float(match.group("seconds") or 0)
bound = 90 if hemisphere in {"N", "S"} else 180
if degrees <= bound and minutes < 60 and seconds < 60:
hemispheres[hemisphere] = degrees
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
return True
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
if labeled_pair:
latitude = float(labeled_pair.group("latitude"))
longitude = float(labeled_pair.group("longitude"))
return abs(latitude) <= 90 and abs(longitude) <= 180
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
if not decimal_pair:
return False
latitude, longitude = map(float, decimal_pair.groups())
return abs(latitude) <= 90 and abs(longitude) <= 180
def _has_address_evidence(text: str) -> bool:
if _NUMBERED_STREET_PATTERN.search(text):
return True
return bool(
_POSTAL_CODE_PATTERN.search(text)
and ("\n" in text or text.count(",") >= 2)
and _SITE_CONTEXT_PATTERN.search(text)
)
def assess_epoch_location(
location: Any,
*,
country: Any = None,
raw_country: Any = None,
) -> EpochLocationAssessment:
"""Classify source coordinates without upgrading their source text.
Epoch's CSV supplies coordinates even when ``Location`` contains only a
country. Those coordinates are useful source evidence, but are not a
facility point and must not enter the renderable location pipeline.
"""
if isinstance(location, dict):
latitude = _coordinate(location, ("lat", "latitude"), 90)
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
has_coordinates = latitude is not None and longitude is not None
return EpochLocationAssessment(
"precise" if has_coordinates else "unknown",
"complete explicit coordinate object"
if has_coordinates
else "coordinate object lacks a complete valid latitude/longitude pair",
)
text = str(location or "").strip()
if not text:
return EpochLocationAssessment("unknown", "Epoch Location is empty")
normalized = _normalized_label(text)
normalized_country = normalize_country(country)
source_country = normalize_country(raw_country)
location_country = _country_from_label(text)
if location_country or (
raw_country and normalized == _normalized_label(raw_country)
):
matched = location_country or source_country or normalized_country
return EpochLocationAssessment(
"country",
f"Epoch Location identifies country only: {matched or text}",
)
if normalized in _REGION_ONLY_LABELS:
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
if (
len(comma_parts) == 2
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
and _country_from_label(comma_parts[1])
):
return EpochLocationAssessment(
"region", f"Epoch Location identifies region/state only: {text}"
)
if normalized in _UNLOCATABLE_LABELS:
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
if normalized.isdigit():
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
if _has_explicit_coordinate_pair(text):
return EpochLocationAssessment(
"precise", "Epoch Location contains a complete valid coordinate pair"
)
if _COORDINATE_CUE_PATTERN.search(text):
return EpochLocationAssessment(
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
)
if _has_address_evidence(text):
return EpochLocationAssessment("precise", "Epoch Location contains an address")
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
return EpochLocationAssessment("city", "Epoch Location identifies a locality")

View File

@@ -0,0 +1,186 @@
"""Authenticated model discovery shared by refresh and connection checks."""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime
from urllib.parse import urlsplit, urlunsplit
import httpx
CATALOG_TIMEOUT_SECONDS = 30
CATALOG_MAX_PAGES = 100
CATALOG_PAGE_SIZE = 100
CATALOG_REQUEST_ATTEMPTS = 2
CATALOG_RETRY_DELAY_SECONDS = 0.2
SUPPORTED_PROVIDER_APIS = {
"openai-completions",
"openai-responses",
"anthropic-messages",
"ollama-generate",
}
class LLMProviderCatalogError(RuntimeError):
"""A safe, user-facing catalog failure with no upstream response or credentials."""
@dataclass(frozen=True)
class ModelCatalog:
url: str
models: list[str]
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
parts = urlsplit(base_url.strip())
if parts.scheme not in {"http", "https"} or not parts.hostname:
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
if parts.username or parts.password or parts.query or parts.fragment:
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
if provider_api not in SUPPORTED_PROVIDER_APIS:
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
path = parts.path.rstrip("/")
if provider_api == "ollama-generate":
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
path = "/api/v1/models"
else:
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
path += "/v1"
path += "/models"
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
def catalog_error_message(exc: Exception) -> str:
if isinstance(exc, LLMProviderCatalogError):
return str(exc)
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
return "模型目录请求超时,请检查网络后重试。"
if isinstance(exc, httpx.HTTPStatusError):
code = exc.response.status_code
messages = {
401: "API Key 验证失败,请检查当前供应商的凭证。",
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
429: "供应商请求限流,请稍后重试。",
}
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
if isinstance(exc, httpx.RequestError):
return "无法连接模型目录,请检查基础地址和网络。"
return "模型目录响应无效,请稍后重试。"
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
if not isinstance(payload, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
envelope = payload.get("output", payload)
if not isinstance(envelope, dict):
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
rows = envelope.get("data", envelope.get("models"))
if not isinstance(rows, list):
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
if any(not isinstance(row, dict) for row in rows):
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
return rows, envelope
def _model_id(row: dict[str, object]) -> str:
value = row.get("id") or row.get("model") or row.get("name")
if not isinstance(value, str) or not value.strip():
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
return value.strip()
def _model_date(row: dict[str, object]) -> float:
value = row.get("created_at") or row.get("published_time") or row.get("created")
if isinstance(value, (int, float)):
return float(value)
if isinstance(value, str):
try:
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
except ValueError:
pass
return 0
async def _get_catalog_page(
client: httpx.AsyncClient,
url: str,
headers: dict[str, str],
params: dict[str, str | int],
) -> object:
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
try:
response = await client.get(url, headers=headers, params=params)
response.raise_for_status()
return response.json()
except httpx.HTTPStatusError as exc:
if attempt or exc.response.status_code not in {502, 503, 504}:
raise
except httpx.TransportError:
if attempt:
raise
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
raise LLMProviderCatalogError("模型目录请求失败。")
async def fetch_model_catalog(
provider: str,
base_url: str,
provider_api: str,
api_key: str = "",
anthropic_version: str = "2023-06-01",
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
) -> ModelCatalog:
url = model_catalog_url(provider, base_url, provider_api)
public_catalog = provider in {"opencode-go", "openrouter"}
if not api_key and provider_api != "ollama-generate" and not public_catalog:
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
if provider_api == "anthropic-messages" and provider not in {
"opencode-go",
"openrouter",
"alibaba",
"moonshotai",
}:
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
elif api_key:
headers["Authorization"] = f"Bearer {api_key}"
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
params: dict[str, str | int] = {}
if native_dashscope:
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
rows_by_id: dict[str, dict[str, object]] = {}
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
# Bound the complete pagination/retry cycle, not just each individual request.
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
for page in range(CATALOG_MAX_PAGES):
payload = await _get_catalog_page(client, url, headers, params)
rows, envelope = _model_rows(payload)
previous_count = len(rows_by_id)
for row in rows:
rows_by_id[_model_id(row)] = row
has_more = envelope.get("has_more") is True
if native_dashscope:
total = envelope.get("total")
if not isinstance(total, int) or total < 0:
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
has_more = len(rows_by_id) < total
params["page_no"] = page + 2
elif has_more:
cursor = envelope.get("last_id")
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
params["after_id"] = cursor
if not has_more:
models = sorted(
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
)
# An empty Ollama catalog is valid: no models have been installed yet.
if not models and provider_api != "ollama-generate":
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
return ModelCatalog(url=url, models=models)
if len(rows_by_id) == previous_count:
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")

View File

@@ -3,17 +3,30 @@
from __future__ import annotations
from typing import Any
from urllib.parse import urlsplit
import httpx
MODELS_DEV_URL = "https://models.dev/api.json"
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
from app.services.llm_model_catalog import fetch_model_catalog
OPENCODE_GO_MODEL_PROVIDER_APIS = {
"minimax-m3": "anthropic-messages",
"qwen3.8-max": "anthropic-messages",
"qwen3.8-flash": "anthropic-messages",
"qwen3.7-max": "anthropic-messages",
"qwen3.7-plus": "anthropic-messages",
"qwen3.6-plus": "anthropic-messages",
"grok-4.6": "openai-responses",
"gpt-5.6-luna": "openai-responses",
"muse-spark-1.3-contributor": "openai-responses",
"muse-spark-1.2-contributor": "openai-responses",
"minimax-m2.7": "anthropic-messages",
"minimax-m2.5": "anthropic-messages",
}
OPENCODE_GO_FALLBACK_MODELS = [
"minimax-m3",
"kimi-k3",
"glm-5.3",
"qwen3.8-max",
"gpt-5.6-luna",
"minimax-m2.7",
"minimax-m2.5",
"kimi-k2.6",
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
]
OPENAI_MODEL_PROVIDER_APIS = {
model: "openai-responses"
for model in [
"gpt-6-astra",
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
"gpt-5.1",
"gpt-5.1-codex",
]
}
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"minimax": {
"provider": "minimax",
"label": "MiniMax",
"provider_api": "anthropic-messages",
"base_url": "https://api.minimaxi.com/anthropic",
"model": "MiniMax-M2.7",
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
"model": "MiniMax-M3",
"models": [
"MiniMax-M3",
"MiniMax-M2.7",
"MiniMax-M2.7-highspeed",
"MiniMax-M2.5",
"MiniMax-M2.5-highspeed",
"MiniMax-M2.1",
"MiniMax-M2.1-highspeed",
"MiniMax-M2",
],
"api_key_env": "MINIMAX_API_KEY",
"source": "fallback",
},
"openai": {
"provider": "openai",
"label": "OpenAI",
"provider_api": "openai-completions",
"provider_api": "openai-responses",
"base_url": "https://api.openai.com/v1",
"model": "gpt-5.1",
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
"model": "gpt-6-astra",
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
"api_key_env": "OPENAI_API_KEY",
"source": "fallback",
},
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Anthropic",
"provider_api": "anthropic-messages",
"base_url": "https://api.anthropic.com/v1",
"model": "claude-sonnet-4-6",
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
"model": "claude-opus-5",
"models": ["claude-opus-5", "claude-sonnet-4-6"],
"api_key_env": "ANTHROPIC_API_KEY",
"source": "fallback",
},
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "DeepSeek",
"provider_api": "openai-completions",
"base_url": "https://api.deepseek.com/v1",
"model": "deepseek-chat",
"models": ["deepseek-chat", "deepseek-reasoner"],
"model": "deepseek-flash",
"models": ["deepseek-flash", "deepseek-v4-pro"],
"api_key_env": "DEEPSEEK_API_KEY",
"source": "fallback",
},
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Alibaba Qwen / DashScope",
"provider_api": "openai-completions",
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"model": "qwen3-max",
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
"model": "qwen3.8-max",
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
"api_key_env": "DASHSCOPE_API_KEY",
"source": "fallback",
},
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
"label": "Moonshot AI / Kimi",
"provider_api": "openai-completions",
"base_url": "https://api.moonshot.ai/v1",
"model": "kimi-k2.5",
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
"model": "kimi-k3",
"models": ["kimi-k3", "kimi-k2.5"],
"api_key_env": "MOONSHOT_API_KEY",
"source": "fallback",
},
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
},
}
MODELS_DEV_PROVIDER_KEYS = {
"minimax": "minimax",
"openai": "openai",
"anthropic": "anthropic",
"deepseek": "deepseek",
"alibaba": "alibaba",
"moonshotai": "moonshotai",
"openrouter": "openrouter",
}
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
}
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
async def refresh_llm_provider_preset(
provider: str,
api_key: str | None = None,
*,
base_url: str | None = None,
provider_api: str | None = None,
anthropic_version: str = "2023-06-01",
) -> dict[str, Any]:
fallback = get_fallback_llm_provider_preset(provider)
if fallback["provider"] == "opencode-go":
headers = {"User-Agent": "Planet/1.0"}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
OPENCODE_GO_MODELS_URL,
headers=headers,
)
response.raise_for_status()
payload = response.json()
data = payload.get("data") if isinstance(payload, dict) else []
model_ids = [
str(item.get("id"))
for item in data
if isinstance(item, dict) and item.get("id")
][:120]
if not model_ids:
model_ids = fallback["models"]
return {
**fallback,
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
"models": model_ids,
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
"source": OPENCODE_GO_MODELS_URL,
}
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
if not models_dev_key:
return fallback
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
MODELS_DEV_URL,
headers={"User-Agent": "Planet/1.0"},
)
response.raise_for_status()
catalog = response.json()
upstream = catalog.get(models_dev_key)
if not isinstance(upstream, dict):
return fallback
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
model_ids = list(upstream_models.keys())[:80]
base_url = upstream.get("api") or fallback["base_url"]
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
base_url = "https://api.deepseek.com/v1"
refreshed = {
resolved_base_url = base_url or fallback["base_url"]
resolved_api = provider_api or fallback["provider_api"]
catalog = await fetch_model_catalog(
fallback["provider"],
resolved_base_url,
resolved_api,
api_key or "",
anthropic_version,
)
model_provider_apis = (
_opencode_go_model_provider_apis(catalog.models)
if fallback["provider"] == "opencode-go"
else fallback.get("model_provider_apis", {})
)
if (
fallback["provider"] == "openai"
and urlsplit(resolved_base_url).hostname != "api.openai.com"
):
model_provider_apis = {}
return {
**fallback,
"label": upstream.get("name") or fallback["label"],
"base_url": base_url,
"model": model_ids[0] if model_ids else fallback["model"],
"models": model_ids or fallback["models"],
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
"source": MODELS_DEV_URL,
"base_url": resolved_base_url,
"provider_api": resolved_api,
"model": catalog.models[0] if catalog.models else "",
"models": catalog.models,
"model_provider_apis": model_provider_apis,
"source": catalog.url,
}
return refreshed

View File

@@ -635,6 +635,7 @@ async def _run_assistant_message(
constraints=_collect_constraints(payload.constraints),
context={
"source": "playground",
"session_id": session_id,
"preset": payload.selected_preset_key,
"conversation_history": conversation_history,
"history_size": len(conversation_history),

View File

@@ -0,0 +1,148 @@
"""Search and paginate the public live TV catalog at the database boundary."""
from typing import Any
from sqlalchemy import Select, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.models.collected_data import CollectedData
from app.services.tv_streams import (
TV_LIVE_SOURCE_COLLECTOR,
TV_LIVE_SOURCE_DATA_TYPE,
_build_collected_tv_source,
build_public_tv_payload,
get_tv_settings_payload,
)
def _source_key():
return func.coalesce(
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
func.nullif(CollectedData.source_id, ""),
CollectedData.entity_key,
)
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
metadata = CollectedData.extra_data
source_id = _source_key()
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
ranked = (
select(
CollectedData.id,
func.row_number()
.over(
partition_by=source_id,
order_by=CollectedData.id.desc(),
)
.label("source_rank"),
)
.where(
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
CollectedData.is_current.is_(True),
CollectedData.is_valid == 1,
CollectedData.deleted_at.is_(None),
enabled.notin_(("false", "0", "no", "off")),
source_id.notin_(configured_ids),
)
.subquery()
)
return (
select(CollectedData)
.join(ranked, ranked.c.id == CollectedData.id)
.where(ranked.c.source_rank == 1)
)
def _filter_catalog_query(
query: Select[tuple[CollectedData]], terms: list[str]
) -> Select[tuple[CollectedData]]:
metadata = CollectedData.extra_data
searchable = func.lower(
func.concat_ws(
" ",
CollectedData.name,
CollectedData.title,
CollectedData.source_id,
metadata["name"].as_string(),
metadata["provider"].as_string(),
metadata["region"].as_string(),
metadata["country"].as_string(),
metadata["language"].as_string(),
)
)
for term in terms:
query = query.where(searchable.contains(term, autoescape=True))
return query
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
searchable = " ".join(
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
).lower()
return all(term in searchable for term in terms)
async def get_tv_catalog_page(
db: AsyncSession,
*,
offset: int = 0,
limit: int = 50,
q: str = "",
selected_id: str | None = None,
) -> dict[str, Any]:
settings = await get_tv_settings_payload(db)
payload = build_public_tv_payload(settings, [])
configured = payload["sources"]
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
if selected_id:
selected = next((source for source in configured if source["id"] == selected_id), None)
if selected is None:
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
selected = _build_collected_tv_source(record, 0) if record else None
if selected:
payload["selected_source"] = selected
summary = (
await db.execute(
select(func.count(), func.max(CollectedData.collected_at))
.select_from(CollectedData)
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
)
).one()
total_collected, latest_update = summary
terms = q.lower().split()
matched_configured = [source for source in configured if _matches_source(source, terms)]
filtered_query = _filter_catalog_query(query, terms)
matched_collected = (
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
if terms
else total_collected
)
sources = matched_configured[offset : offset + limit]
remaining = limit - len(sources)
if remaining:
rows = await db.scalars(
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
.offset(max(0, offset - len(matched_configured)))
.limit(remaining)
)
sources.extend(
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
)
total = len(matched_configured) + matched_collected
next_offset = offset + len(sources)
return {
**payload,
"sources": sources,
"source_count": len(configured) + total_collected,
"latest_updated_at": (
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
),
"total": total,
"offset": offset,
"limit": limit,
"has_more": next_offset < total,
"next_offset": next_offset if next_offset < total else None,
}

View File

@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
from app.models.collected_data import CollectedData
from app.models.system_setting import SystemSetting
DEFAULT_TV_SOURCE_ID = "cgtn-en"
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
TV_SETTINGS_CATEGORY = "tv"
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
UNSUPPORTED_CCTV4_STREAM = (
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
)
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
DEFAULT_TV_SETTINGS = {
"default_source_id": DEFAULT_TV_SOURCE_ID,
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
"sources": [
{
"id": "cctv4",
"name": "CCTV-4 中文国际",
"name": "CCTV-4 中文国际(美洲版)",
"provider": "CCTV",
"region": "China",
"language": "zh-CN",
"source_type": "hls",
"embed_url": "https://tv.cctv.com/live/cctv4/",
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
"homepage_url": "https://tv.cctv.com/live/cctv4/",
"embed_url": "",
"stream_url": CCTV4_STREAM,
"use_proxy": False,
"homepage_url": CCTV4_HOMEPAGE,
"poster_url": "",
"is_enabled": True,
"is_fallback": True,
"sort_order": 10,
"collector_source": None,
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
"notes": CCTV4_PLAYBACK_NOTE,
},
{
"id": "reuters-tv",
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
payload = dict(source or {})
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
payload.get("stream_url")
) == UNSUPPORTED_CCTV4_STREAM:
payload.update(
name="CCTV-4 中文国际(美洲版)",
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
use_proxy=False,
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
)
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
source_type = _clean_text(payload.get("source_type")).lower()
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
"source_type": source_type,
"embed_url": _clean_url(payload.get("embed_url")),
"stream_url": _clean_url(payload.get("stream_url")),
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
"homepage_url": _clean_url(payload.get("homepage_url")),
"poster_url": _clean_url(payload.get("poster_url")),
"youtube_video_id": youtube_video_id,
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
]
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
default_source = next(
source for source in DEFAULT_TV_SETTINGS["sources"]
if source["id"] == DEFAULT_TV_SOURCE_ID
)
normalized_sources.append(
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
normalize_tv_source(default_source, index=len(normalized_sources))
)
default_source_exists = any(

View File

@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
return [item.to_dict() for item in result.scalars().all()]
async def get_current_vessels_by_mmsi(
db: AsyncSession, mmsis: list[int]
) -> list[dict[str, Any]]:
"""Read canonical render state for the vessels changed by a stream flush."""
if not mmsis:
return []
result = await db.execute(
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
)
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
async def aggregate_vessel_observations(
db: AsyncSession,
observations: Iterable[AISRawObservation],

View File

@@ -0,0 +1,72 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from fastapi import FastAPI
from pydantic import ValidationError
from app.api.v1.auth import router
from app.core import security
from app.db.session import DEFAULT_LOGIN_USERS, get_db
from app.models.user import User
from app.schemas.user import UserRegister
async def identity_request(monkeypatch, user, token=None):
result = MagicMock()
result.scalar_one_or_none.return_value = user
db = AsyncMock()
db.execute.return_value = result
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
app = FastAPI()
app.include_router(router, prefix="/auth")
async def override_db():
yield db
app.dependency_overrides[get_db] = override_db
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test"
) as client:
return await client.get(
"/auth/me",
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
)
@pytest.mark.asyncio
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
created = datetime(2026, 9, 21, tzinfo=UTC)
user = User(
id=1, username=default["username"], email=default["email"],
role=default["role"], password_hash="not-exposed", is_active=True,
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
)
response = await identity_request(monkeypatch, user)
assert response.status_code == 200
payload = response.json()
assert payload["email"] == default["email"]
assert payload["email_verified"] is True
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
assert payload["gatekeeper_groups"] == ["ops"]
assert "password_hash" not in payload
@pytest.mark.asyncio
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
response = await identity_request(monkeypatch, user)
assert response.status_code == 401
@pytest.mark.asyncio
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
assert response.status_code == 401
def test_registration_still_validates_email():
with pytest.raises(ValidationError):
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")

View File

@@ -0,0 +1,333 @@
import io
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import FastAPI, HTTPException
from httpx import ASGITransport, AsyncClient
from PIL import Image
from pydantic import ValidationError
from app.api.v1.card_images import router
from app.core.security import get_current_user
from app.db.session import get_db
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
from app.services.card_image_configuration import (
effective_envelope,
publish_configuration,
resolve_image,
)
from app.services.card_image_defaults import bundled_configuration
from app.services import card_image_files
def configuration():
assets = [
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
for key in ["exact", "model", "group", "category"]
]
rules = [
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
for key, scope, value in [
("exact", "object", "123"),
("model", "model", "m1"),
("group", "group", "starlink"),
("category", "category", "satellite"),
]
]
return CardImageConfiguration(assets=assets, rules=rules)
def test_precedence_disabled_fallback_and_explicit_no_image():
config = configuration()
context = dict(
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
)
for expected in ["exact", "model", "group", "category"]:
assert resolve_image(config, context)["image"]["id"] == expected
next(asset for asset in config.assets if asset.id == expected).enabled = False
assert resolve_image(config, context)["image"] is None
config = configuration()
config.rules[0].asset_id = None
config.rules[0].suppress = True
result = resolve_image(config, context)
assert result["suppressed"] and result["image"] is None
def test_bundled_series_images_retain_truthful_source_metadata():
config, _ = bundled_configuration()
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
assert result["image"]["id"] == "starlink-solar-array"
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
assert pending.rights_status == "permission_pending"
assert (
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
== pending.id
)
assert pending.enabled
@pytest.mark.parametrize(
"url",
[
"javascript:alert(1)",
"//evil.example/image.jpg",
"/earth/assets/cards/../secret",
"https://user:pass@example.org/a.jpg",
],
)
def test_unsafe_urls_rejected(url):
with pytest.raises(ValidationError):
CardImageAsset(id="test", title="test", url=url)
def test_duplicate_targets_and_dangling_assets_rejected():
data = configuration().model_dump()
duplicate = {**data["rules"][0], "id": "another"}
data["rules"].append(duplicate)
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
data = configuration().model_dump()
data["assets"] = []
with pytest.raises(ValidationError):
CardImageConfiguration.model_validate(data)
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
defaults, _ = bundled_configuration()
payload = {
"revision": "custom",
"config": {"assets": [], "rules": []},
"known_assets": [asset.id for asset in defaults.assets],
"known_rules": [rule.id for rule in defaults.rules],
}
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
payload["known_assets"] = []
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
output = io.BytesIO()
Image.new("RGB", (1600, 900), "blue").save(output, format)
asset = card_image_files.store_image(output.getvalue(), "Facility")
assert (asset.width, asset.height) == (1600, 900)
assert len(list(tmp_path.iterdir())) == 3
for size, url in asset.variants.items():
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
assert max(image.size) == int(size)
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
with pytest.raises(HTTPException):
card_image_files.store_image(b"<svg></svg>", "bad")
with pytest.raises(HTTPException):
card_image_files.uploaded_path("../private")
def fake_db(payload):
db = AsyncMock()
db.add = MagicMock()
record = SimpleNamespace(payload=payload)
result = MagicMock()
result.scalar_one.return_value = record
result.scalar_one_or_none.return_value = record
db.execute.return_value = result
return db, record
@pytest.mark.asyncio
async def test_publish_revision_and_audit():
defaults, _ = bundled_configuration()
payload = {
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
db, record = fake_db(payload)
revision = effective_envelope(payload)["revision"]
result = await publish_configuration(
db, revision, defaults, SimpleNamespace(id=1, username="admin")
)
assert result["revision"] != revision
assert db.add.call_args.args[0].action == "card_images.update"
before = deepcopy(record.payload)
with pytest.raises(HTTPException) as error:
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
assert error.value.status_code == 409 and record.payload == before
@pytest.mark.asyncio
async def test_configuration_requires_admin_and_validates_http_payload():
app = FastAPI()
app.include_router(router)
defaults, _ = bundled_configuration()
db, _ = fake_db(
{
"revision": "default",
"config": defaults.model_dump(mode="json"),
"known_assets": [],
"known_rules": [],
}
)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (await client.get("/configuration")).status_code == 403
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
assert (await client.get("/configuration")).status_code == 200
assert (
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
).status_code == 422
@pytest.mark.asyncio
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
app = FastAPI()
app.include_router(router)
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
role="admin", id=1, username="admin"
)
buffer = io.BytesIO()
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
response = await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
assert response.status_code == 200
asset = response.json()
filename = asset["variants"]["640"].rsplit("/", 1)[1]
image = await client.get(f"/files/{filename}")
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
assert "immutable" in image.headers["cache-control"]
assert (
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
).status_code == 422
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
assert (
await client.post(
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
)
).status_code == 403
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
from typing import get_args
from app.schemas.card_images import EntityType
config, _ = bundled_configuration()
for entity_type in get_args(EntityType):
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
assert result["image"] and result["match"]["value"] == "*"
config = config.model_copy(deep=True)
config.rules.append(
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
)
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["starlink-solar-array"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
def starlink():
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["focus_x"] = 20
assert starlink()["focus_x"] == 20
assert starlink()["url"].endswith("starlink-orbit.webp")
payload["config"]["assets"][0]["enabled"] = False
assert starlink()["enabled"] is False
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
assert starlink()["url"] == "https://example.org/custom.jpg"
def test_compute_categories_and_bgp_get_separate_user_images():
config, _ = bundled_configuration()
for kind, filename in [
("supercomputer", "supercomputer.png"),
("gpu_cluster", "gpu-cluster.jpg"),
("data_center", "data-center.png"),
("cloud_center", "cloud-center.png"),
("edge_node", "edge-node.png"),
]:
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
assert result["image"]["url"].endswith(filename)
for kind in ("bgp", "bgp_collector"):
result = resolve_image(config, dict(entity_type=kind, category=kind))
assert result["image"]["url"].endswith(
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
)
def test_saved_collector_default_migrates_without_changing_user_bindings():
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
old_asset = previous_bundled_assets()["network-switches"][0]
payload = {
"revision": "old",
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
"known_assets": [old_asset.id],
"known_rules": [old_rule.id],
}
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
"image"
]["url"].endswith("bgp-collector.png")
payload["config"]["rules"][0]["enabled"] = False
result = effective_envelope(payload)["config"]
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
@pytest.mark.asyncio
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
from app.services.card_image_targets import get_target
from sqlalchemy.dialects import postgresql
db = AsyncMock()
result = MagicMock()
result.mappings.return_value.first.return_value = {
"entity_id": "epoch_ai_gpu:site",
"name": "Site",
"category": "cloud_center",
"group": "Operator",
"model": "Model",
}
db.execute.return_value = result
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
query = db.execute.call_args.args[0]
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "collected_data.source = 'epoch_ai_gpu'" in sql
assert "collected_data.source_id = 'site'" in sql
assert context["category"] == "cloud_center" and context["model"] == "Model"
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
from app.services.card_image_defaults import previous_bundled_assets
old = previous_bundled_assets()["compute-gpu-cluster"][0]
payload = {
"revision": "old",
"config": {"assets": [old.model_dump()], "rules": []},
"known_assets": [old.id],
"known_rules": [],
}
result = effective_envelope(payload)["config"]
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
assert upgraded["variants"] == latest.variants
assert upgraded["url"] == latest.url

View File

@@ -0,0 +1,218 @@
"""Request-level regressions for quota-limited satellite downloads."""
import gzip
import json
import os
import time
from unittest.mock import AsyncMock
import httpx
import pytest
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
from app.services.collectors.downloads import ResumableFileDownloader
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
def setup_collector(monkeypatch, tmp_path, handler):
collector = CelesTrakTLECollector()
collector._resolved_url = "https://celestrak.example/gp.php"
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
original_client = httpx.AsyncClient
monkeypatch.setattr(
"app.services.collectors.celestrak.httpx.AsyncClient",
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
)
return collector
def expire(path):
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
os.utime(path, (old, old))
@pytest.mark.asyncio
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
return httpx.Response(
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
)
collector = setup_collector(monkeypatch, tmp_path, handler)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
assert len(requests) == 1
path = collector._downloader.cached_file_path(URL, ".json")
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
assert requests[1].headers["if-none-match"] == '"v1"'
@pytest.mark.asyncio
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
collector = setup_collector(monkeypatch, tmp_path, handler)
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
assert requests == ["GET"]
@pytest.mark.asyncio
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
path = collector._downloader.cached_file_path(URL, ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
with pytest.raises(RuntimeError, match="validation failed"):
await collector.fetch()
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
@pytest.mark.asyncio
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
collector = setup_collector(monkeypatch, tmp_path, handler)
path, _, meta = collector._downloader._cache_paths(URL, ".json")
path.write_text("[]")
meta.write_text('{"etag":"old"}')
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
assert "if-none-match" not in requests[0].headers
@pytest.mark.asyncio
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
)
path, _, meta = downloader._cache_paths(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text('[{"NORAD_CAT_ID":42}]')
meta.write_text('{"etag":"v1"}')
expire(path)
requests = []
def handler(request):
requests.append(request)
return httpx.Response(304)
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
for _ in range(2):
await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert len(requests) == 1
assert requests[0].headers["if-none-match"] == "v1"
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
@pytest.mark.asyncio
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
body = gzip.compress(raw)
downloader = ResumableFileDownloader(
cache_namespace="test", cache_root=tmp_path, preflight_head=False
)
async with httpx.AsyncClient(
transport=httpx.MockTransport(
lambda request: httpx.Response(
200,
content=body,
headers={"content-encoding": "gzip", "content-length": str(len(body))},
)
)
) as client:
path = await downloader.download_file(
client,
URL,
extension=".json",
validate_existing=CelesTrakTLECollector._validate_json_file,
)
assert path.read_bytes() == raw
@pytest.mark.asyncio
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
final, part, meta = downloader._cache_paths(URL, ".bin")
part.parent.mkdir(parents=True)
part.write_bytes(b"abc")
meta.write_text('{"etag":"v1"}')
requests = []
def handler(request):
requests.append(request)
if request.method == "HEAD":
return httpx.Response(
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
)
assert request.headers["range"] == "bytes=3-"
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".bin")
assert [r.method for r in requests] == ["HEAD", "GET"]
assert final.read_bytes() == b"abcdef"
@pytest.mark.asyncio
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
requests = []
def handler(request):
requests.append(request)
if request.url.params["GROUP"] == "active":
return httpx.Response(
403, text="GP data has not updated since your last successful download"
)
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
collector = setup_collector(monkeypatch, tmp_path, handler)
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
path.write_text('[{"NORAD_CAT_ID":42}]')
expire(path)
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
assert [r.method for r in requests] == ["GET", "GET"]
@pytest.mark.asyncio
async def test_generic_cache_without_validators_must_refresh(tmp_path):
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
path = downloader.cached_file_path(URL, ".json")
path.parent.mkdir(parents=True)
path.write_text("[]")
requests = []
def handler(request):
requests.append(request.method)
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
await downloader.download_file(client, URL, extension=".json")
assert requests == ["HEAD", "GET"]
assert path.read_text() == "[1]"

View File

@@ -79,7 +79,8 @@ class TestTOP500Collector:
"""Test parsing empty response"""
collector = TOP500Collector()
result = collector.parse_response("<html><body><table></table></body></html>")
assert len(result) > 0
assert result == []
assert collector.fail_on_empty
def test_parse_response_single_item(self):
"""Test parsing single item response"""
@@ -97,9 +98,9 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["source_id"] == "top500_1"
assert result[0]["source_id"] == "top500_system_1"
assert result[0]["name"] == "Test Supercomputer"
assert result[0]["country"] == "USA"
assert result[0]["metadata"]["rank"] == 1
@@ -113,7 +114,7 @@ class TestTOP500Collector:
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
<tr>
<td>1</td>
<td>Valid\nVendor\nSite\nUSA</td>
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
<td>1000</td>
<td>10 PFLOP/s</td>
<td>12 PFLOP/s</td>
@@ -129,7 +130,7 @@ class TestTOP500Collector:
</tr>
</table>
"""
result = collector.parse_response(response)
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
assert len(result) == 1
assert result[0]["name"] == "Valid"

View File

@@ -0,0 +1,222 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.system_log import AuditLog
from app.schemas.compute_configuration import ComputeConfiguration
from app.services.compute_configuration import (
DEFAULT_CONFIGURATION,
default_envelope,
publish_configuration,
read_configuration,
)
from app.services.compute_history import compute_history
from fastapi import FastAPI, HTTPException, Response
from httpx import ASGITransport, AsyncClient
from pydantic import ValidationError
def default_config():
return DEFAULT_CONFIGURATION.model_dump(mode="json")
@pytest.mark.parametrize(
"mutation",
[
lambda value: value["observations"][0]["values"].update(CN=101),
lambda value: value["observations"][0]["values"].update(CN=-1),
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
lambda value: value["observations"][0]["values"].update(CN=True),
lambda value: value["observations"][0]["values"].update(EUROPE=10),
lambda value: value["observations"][0].update(source_id="unknown"),
lambda value: value["observations"][0].update(year=2026),
lambda value: value["observations"].append(value["observations"][0]),
lambda value: value["sources"].append(value["sources"][0]),
lambda value: value["innovation"].append(value["innovation"][0]),
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
],
)
def test_rejects_invalid_or_ambiguous_evidence(mutation):
payload = default_config()
mutation(payload)
with pytest.raises(ValidationError):
ComputeConfiguration.model_validate(payload)
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
payload = default_config()
payload["sources"].append(
{
"id": "new-report",
"title": "New report",
"url": "https://example.com/report",
"publisher": "Research group",
"report_year": 2026,
"locator": "Figure 2",
}
)
payload["observations"].append(
{
"year": 2025,
"metric": "global_share",
"source_id": "new-report",
"values": {"CN": 25, "US": 50, "JP": 0},
}
)
config = ComputeConfiguration.model_validate(payload)
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
row = next(row for row in result["compute"] if row["year"] == 2025)
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
payload = default_config()
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
result = compute_history(
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
)
assert result["compute"][0]["values"] == {"CN": None, "US": None}
payload.update(sources=[], observations=[], innovation=[])
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
assert all(row["values"]["CN"] is None for row in result["compute"])
assert result["evidence"] == {}
def fake_db(record=None):
db = AsyncMock()
db.add = MagicMock()
result = MagicMock()
result.scalar_one_or_none.return_value = record
result.scalar_one.return_value = record
db.execute.return_value = result
return db
@pytest.mark.asyncio
async def test_fresh_install_uses_defaults_without_writing_database():
db = fake_db()
assert await read_configuration(db) == default_envelope()
db.add.assert_not_called()
db.commit.assert_not_awaited()
@pytest.mark.asyncio
async def test_series_reads_persisted_override_and_disables_http_cache():
envelope = default_envelope()
envelope["is_default"] = False
envelope["revision"] = "custom"
envelope["config"]["observations"][3]["values"]["CN"] = 30
response = Response()
data = await get_series(
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
)
assert data["compute"][3]["values"]["CN"] == 30
assert data["version"] == "custom"
assert data["provenance"] == "admin_curated_reports"
assert response.headers["Cache-Control"] == "no-store"
@pytest.mark.asyncio
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
actor = SimpleNamespace(id=1, username="admin")
old = deepcopy(record.payload)
result = await publish_configuration(
db,
revision=old["revision"],
config=DEFAULT_CONFIGURATION,
actor=actor,
)
assert result["revision"] != old["revision"]
audit = db.add.call_args.args[0]
assert isinstance(audit, AuditLog)
assert audit.details == {"before": old, "after": result}
assert audit.actor_id == actor.id
db.commit.assert_awaited_once()
with pytest.raises(HTTPException) as failure:
await publish_configuration(
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
)
assert failure.value.status_code == 409
assert record.payload == result
db.rollback.assert_awaited_once()
db.commit.assert_awaited_once()
@pytest.mark.asyncio
async def test_reset_changes_revision_and_restores_bundled_data():
record = SimpleNamespace(
payload={**default_envelope(), "revision": "custom", "is_default": False}
)
result = await publish_configuration(
fake_db(record),
revision="custom",
config=DEFAULT_CONFIGURATION,
actor=SimpleNamespace(id=1, username="admin"),
reset=True,
)
assert result["is_default"] is True
assert result["config"] == default_config()
assert result["revision"] != "custom"
@pytest.mark.asyncio
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
async def test_non_admin_cannot_configure(role):
with pytest.raises(HTTPException) as failure:
await configuration_admin(SimpleNamespace(role=role))
assert failure.value.status_code == 403
@pytest.mark.asyncio
async def test_http_configuration_auth_validation_and_public_series():
app = FastAPI()
app.include_router(router)
record = SimpleNamespace(payload=default_envelope())
db = fake_db(record)
app.dependency_overrides[get_db] = lambda: db
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
assert (await client.get("/configuration")).status_code == 401
assert (await client.get("/series")).status_code == 200
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
id=1,
username="admin",
role="admin",
)
loaded = (await client.get("/configuration")).json()
invalid = deepcopy(loaded["config"])
invalid["observations"][0]["values"]["CN"] = 200
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": invalid,
},
)
assert response.status_code == 422
db.commit.assert_not_awaited()
response = await client.put(
"/configuration",
json={
"revision": loaded["revision"],
"config": loaded["config"],
},
)
assert response.status_code == 200
assert response.json()["revision"] != loaded["revision"]
assert (
await client.post(
"/configuration/reset",
json={
"revision": loaded["revision"],
},
)
).status_code == 409

View File

@@ -0,0 +1,268 @@
import csv
import io
from datetime import UTC, datetime
from unittest.mock import AsyncMock, MagicMock
import pytest
from fastapi import HTTPException
from sqlalchemy.dialects import postgresql
from app.api.v1.compute_intelligence import sample_query, selected_countries
from app.services.collectors.epoch_ai import EpochAIGPUCollector
from app.services.collectors.top500 import TOP500Collector
from app.services.compute_history import compute_history
def epoch_csv(**overrides):
row = {
"Name": "Cluster Alpha",
"Country": "United States of America",
"Status": "Existing",
"First Operational Date": "2023-05-02",
"16-bit OP/s (log)": "18",
"Location": "3231 Riverport Rd, Memphis, TN 38109",
"latitude": "35",
"longitude": "-90",
**overrides,
}
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
return output.getvalue()
def test_epoch_csv_preserves_precision_date_and_missing_values():
collector = EpochAIGPUCollector()
row = collector.parse_response(epoch_csv())[0]
assert row["value"] == 1e18
assert row["unit"] == "FLOP/s"
assert row["metadata"]["precision"] == "FP16"
assert row["metadata"]["location_precision"] == "precise"
assert row["metadata"]["raw_country"] == "United States of America"
assert row["reference_date"] == "2023-05-02"
missing = collector.parse_response(
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
)[0]
assert missing["value"] is None
assert missing["reference_date"] is None
@pytest.mark.parametrize(
("location", "country", "expected"),
[
("United States of America", "United States of America", "country"),
("Japan", "Japan", "country"),
("Korea (Republic of)", "South Korea", "country"),
("Texas", "United States of America", "region"),
("Iowa, USA", "United States of America", "region"),
("Seoul", "South Korea", "city"),
("Paris, France", "France", "city"),
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
("Cluster 2, Texas", "United States of America", "site"),
("Campus 7, Tokyo", "Japan", "site"),
("Building 240, Argonne", "United States of America", "site"),
("7", "United States of America", "unknown"),
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
("35.380924,139.625008", "Japan", "precise"),
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
],
)
def test_epoch_location_precision_follows_location_field(location, country, expected):
row = EpochAIGPUCollector().parse_response(
epoch_csv(Location=location, Country=country)
)[0]
assert row["metadata"]["location_precision"] == expected
@pytest.mark.parametrize(
("location", "expected"),
[
({"lat": 35.0, "lng": 139.0}, "precise"),
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
({"lat": 35.0}, "unknown"),
({"lng": 139.0}, "unknown"),
({"lat": "invalid", "lng": 139.0}, "unknown"),
],
)
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
from app.services.epoch_location import assess_epoch_location
assert assess_epoch_location(location).precision == expected
def test_epoch_anonymous_rows_are_not_geocoded():
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
assert row["latitude"] is None and row["longitude"] is None
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
@pytest.mark.parametrize(
"payload",
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
)
def test_epoch_invalid_payload_fails_closed(payload):
with pytest.raises(ValueError):
EpochAIGPUCollector().parse_response(payload)
def test_epoch_duplicate_identity_rejected():
payload = epoch_csv()
with pytest.raises(ValueError, match="identity collision"):
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
assert len(rows) == 2
assert rows[0]["source_id"] != rows[1]["source_id"]
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
def test_top500_units_and_stable_identity():
collector = TOP500Collector()
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
first = collector.parse_response(
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
)[0]
second = collector.parse_response(
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
)[0]
assert first["source_id"] == second["source_id"] == "top500_system_123"
assert float(first["value"]) == 1.5e15
assert first["metadata"]["rpeak_flops"] == 2e15
assert first["reference_date"] == "2025-11-01"
assert second["reference_date"] == "2026-06-01"
with pytest.raises(ValueError, match="date missing"):
collector.parse_response(html.format(rank=1))
@pytest.mark.asyncio
async def test_large_source_loss_does_not_create_snapshot():
db = AsyncMock()
db.scalar.return_value = 500
db.add = MagicMock()
with pytest.raises(ValueError, match="40%"):
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
db.add.assert_not_called()
db.commit.assert_not_awaited()
def test_history_missing_years_and_denominators():
global_data = compute_history(["CN", "US"])
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
bilateral = compute_history(["CN", "US"], "bilateral_share")
assert sum(bilateral["compute"][3]["values"].values()) == 100
assert bilateral["denominator"] == "US+CN"
absolute = compute_history(["US", "CN"], "absolute")
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
assert global_data["innovation"][-2]["value"] == 17.95
assert global_data["innovation"][-1]["value"] is None
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
@pytest.mark.asyncio
async def test_snapshot_query_uses_observation_cutoff_not_location():
query = await sample_query(AsyncMock(), ["CN"], 2023)
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
assert "2024-01-01" in sql
assert "row_number() OVER (PARTITION BY" in sql
assert "compute_contract" in sql and "中国" in sql
assert "latitude" not in sql and "longitude" not in sql
def test_country_input_is_bounded_and_deduplicated():
assert selected_countries("CN,CN,US") == ["CN", "US"]
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
with pytest.raises(HTTPException):
selected_countries("invalid")
with pytest.raises(HTTPException, match="overlapping"):
selected_countries("EUROPE,DE")
def test_catalog_supports_global_participants_without_inventing_history():
from app.services.compute_catalog import (
CATALOG,
COMPARISON_GROUPS,
COUNTRIES,
source_country_names,
source_country_participants,
)
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
{entry["code"] for entry in CATALOG}
)
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
owners = source_country_names(["CN"])
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
assert [entry["kind"] for entry in CATALOG[:5]] == [
"country",
"country",
"region",
"region",
"region",
]
group_members = [
member
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
for member in members
]
assert len(group_members) == len(set(group_members))
assert set(group_members) == set(COUNTRIES)
data = compute_history(["JP", "KR"])
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
assert all(row["values"]["KR"] is None for row in data["compute"])
assert all(row["value"] is None for row in data["innovation"])
assert len(data["catalog"]) > 30
def test_region_history_requires_complete_same_year_member_coverage():
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
assert row_2023["values"]["CN"] == 31
assert row_2023["values"]["US"] == 41
assert row_2023["values"]["EUROPE"] is None
assert row_2023["values"]["APAC"] is None
assert row_2023["values"]["OTHER"] is None
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
def test_anonymous_records_never_enter_the_location_repair_queue():
from app.api.v1.visualization import convert_compute_centers_to_geojson
from app.models.collected_data import CollectedData
record = CollectedData(
source="epoch_ai_gpu",
source_id="anonymous",
name="Anonymous",
extra_data={"compute_contract": "v1", "anonymous": True},
)
result = convert_compute_centers_to_geojson([record])
assert result["features"] == []
assert result["unresolved"] == []
assert result["withheld_count"] == 1
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
absolute = compute_history(["CN"], "absolute")
observed = {row["year"]: row for row in absolute["compute"]}
assert observed[2022]["values"]["CN"] == 302
assert observed[2022]["evidence_id"] == "caict-2023"
assert observed[2023]["evidence_id"] == "caict-2024"
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
for row in absolute["compute"]:
if row["evidence_id"]:
report = absolute["evidence"][row["evidence_id"]]
assert report["url"].startswith("https://www.caict.ac.cn/")

View File

@@ -0,0 +1,18 @@
from app.services.compute_center_verified_sites import get_verified_compute_site
def test_registry_requires_exact_facility_identity_not_operator_name():
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
assert site["city"] == "Yokohama"
assert site["precision"] == "site"
assert site["coordinate_basis"] == "official_access_map_center"
assert site["sources"]
site["latitude"] = 0
assert get_verified_compute_site(
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
)["latitude"] != 0

View File

@@ -1,4 +1,5 @@
import json
from pathlib import Path
import pytest
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
def write_geojson(path, name="Test"):
names = [name]
if name == "earth_admin0_boundaries":
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
path.write_text(
json.dumps(
{
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
"features": [
{
"type": "Feature",
"properties": {"name": name},
"properties": {
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
},
"geometry": {
"type": "Polygon",
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
},
}
for item in names
],
}
),
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
config = repo / "config/earth-boundary-sources.local.json"
example = repo / "config/earth-boundary-sources.example.json"
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
path.mkdir(parents=True, exist_ok=True)
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
geometry.parent.mkdir(parents=True, exist_ok=True)
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
geometry.write_bytes(source_geometry.read_bytes())
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
return {
"repo": repo,
"config": config,
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
}
def write_fallback(path, policy_path, geometry_path):
policy = earth_boundaries._read_json(policy_path)
geometry = earth_boundaries._read_json(geometry_path)
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(policy),
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
}
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "legacy-geojson"
assert status["fallback_available"] is True
assert status["high_precision_ready"] is False
assert status["provider"] == "missing"
assert status["fallback_available"] is False
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_geojson(paths["legacy"])
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
coverage = {
"profile": "china-pov-v1",
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
"applied": True,
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
}
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "pmtiles-mvt"
assert status["high_precision_ready"] is True
assert status["policy_publishable"] is False
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
paths = patch_paths(monkeypatch, tmp_path)
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
paths["pmtiles"].write_bytes(b"pmtiles")
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
status = earth_boundaries.get_boundary_status()
assert status["provider"] == "policy-geojson-fallback"
assert status["high_precision_ready"] is False
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
assert status["high_precision_ready"] is True
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
assert paths["manifest"].exists()
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
assert manifest["povPolicy"]["applied"] is True
assert manifest["povPolicy"]["coverage"] == "partial"
assert manifest["povPolicy"]["publicationReady"] is False
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
by_rule = {
feature["properties"].get("PLANET_POLICY_RULE"): feature
for feature in hover["features"]
}
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
assert all(
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
for feature in hover["features"]
)
assert any(phase == "download" for _progress, phase, _message in progress_events)
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
assert removed.isdisjoint(DEFAULT_DATASOURCES)
for source in removed:
assert collector_registry.get(source) is None
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
repo = Path(__file__).resolve().parents[2]
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
payload = json.loads(path.read_text(encoding="utf-8"))
matches = [
feature for feature in payload["features"]
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
]
assert len(matches) == 1
jeju = matches[0]
assert jeju["properties"]["ISO_A3"] == "KOR"
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
ring = jeju["geometry"]["coordinates"][0]
assert len(ring) == 61
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)

View File

@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
assert vessel_update is not None
assert vessel_update["source"] == "vessel_position"
assert vessel_update["layers"] == ["vessels"]
assert vessel_update["refresh_strategy"] == "reload"
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
from app.services.earth_db_change_listener import PendingEarthDbChange
for source in ("aisstream_vessels", "barentswatch_vessels"):
update = build_earth_update_from_db_payload({
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
})
assert update["refresh_strategy"] == "delta"
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
pending.add({"operation": "UPDATE"})
pending.add({"operation": "DELETE"})
assert pending.refresh_strategy == "reload"
def test_build_earth_update_maps_interactable_delete_to_delta():

View File

@@ -0,0 +1,41 @@
import json
from pathlib import Path
import pytest
from app.core.earth_geography_policy import display_country, display_iso_a3
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
from app.core import earth_geography_policy as policy
policy._policy_country_by_code.cache_clear()
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
try:
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
policy.display_country("China")
finally:
policy._policy_country_by_code.cache_clear()
def test_display_country_groups_china_regions_and_kosovo() -> None:
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
assert display_country(source) == "中国"
assert display_iso_a3(source) == "CHN"
for source in ("Kosovo", "XK", "SRB"):
assert display_country(source) == "塞尔维亚"
assert display_iso_a3(source) == "SRB"
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
assert display_country("Japan") == "日本"
assert display_iso_a3("Japan") == "JPN"
assert display_country("unexpected non-country") is None
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
policy = json.loads(path.read_text(encoding="utf-8"))
for rule in policy["rules"]:
for code in rule.get("sourceIsoA3", []):
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))

View File

@@ -384,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
assert items[0].content_language == "zh-CN"
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
assert payload_en["display_title"] == ""
items[0].localizations["en-US"] = {
"title": "Chinese e-commerce platform reports quarterly growth",
"summary": "The platform said cross-border orders rose year over year.",
}
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
def test_default_news_sources_include_business_and_ecommerce_sources():

View File

@@ -0,0 +1,86 @@
from typing import Any
import httpx
import pytest
from app.services.earth_news import (
NewsFeedEndpoint,
NewsFeedSource,
ParsedNewsItem,
_fetch_single_feed_url,
_serialize_item,
)
async def fetch_feed(
content: bytes, content_type: str = "text/xml"
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
source = NewsFeedSource(
id="ebrun",
name="亿邦动力",
region="asia-pacific",
feed_url="https://example.com/rss.xml",
homepage_url="https://example.com/",
)
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
transport = httpx.MockTransport(
lambda request: httpx.Response(
200, content=content, headers={"content-type": content_type}
)
)
async with httpx.AsyncClient(transport=transport) as client:
return await _fetch_single_feed_url(client, source, feed)
@pytest.mark.asyncio
@pytest.mark.parametrize(
("encoding", "declaration", "content_type"),
[
("gbk", 'encoding="GBK"', "text/xml"),
("gb2312", "encoding='GB2312'", "application/rss+xml"),
("gb18030", 'encoding="GB18030"', "text/xml"),
("gbk", "", "text/xml; charset=gbk"),
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
("utf-8", "", "text/xml"),
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
("utf-16", 'encoding="UTF-16"', "text/xml"),
("utf-32", 'encoding="UTF-32"', "text/xml"),
],
)
async def test_feed_encoding_preserves_chinese_in_display(
encoding: str, declaration: str, content_type: str
):
title = "618电商战报:零售额增长266%"
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
xml = f'''<?xml version="1.0" {declaration}?>
<rss version="2.0"><channel><item>
<title><![CDATA[{title}]]></title>
<description><![CDATA[<p>{summary}</p>]]></description>
<link>https://example.com/story</link>
</item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
assert error is None
assert health["ok"] is True
assert len(items) == 1
assert items[0].title == title
assert items[0].summary == summary
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
assert payload["display_title"] == title
assert payload["display_summary"] == summary
@pytest.mark.asyncio
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
<rss><channel><item><title>电商新闻</title>
<link>https://example.com/story</link></item></channel></rss>'''
_, items, error, health = await fetch_feed(xml.encode("gbk"))
assert items == []
assert error
assert health["ok"] is False
assert health["status"] == "format_error"

View File

@@ -0,0 +1,219 @@
from copy import deepcopy
import httpx
import pytest
from app.services import llm_model_catalog as discovery
from app.services import llm_provider_catalog as catalog
REAL_CLIENT = httpx.AsyncClient
def mock_http(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
@pytest.mark.parametrize(
"provider,path,auth",
[
("minimax", "/anthropic/v1/models", "x-api-key"),
("anthropic", "/v1/models", "x-api-key"),
("openai", "/v1/models", "authorization"),
("deepseek", "/v1/models", "authorization"),
("alibaba", "/api/v1/models", "authorization"),
("moonshotai", "/v1/models", "authorization"),
("openrouter", "/api/v1/models", "authorization"),
("opencode-go", "/zen/go/v1/models", "authorization"),
("ollama", "/api/tags", "authorization"),
],
)
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
preset = catalog.get_fallback_llm_provider_preset(provider)
def handle(request):
assert request.url.path == path
assert request.url.host == httpx.URL(preset["base_url"]).host
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
if provider == "alibaba":
assert request.url.params["capabilities"] == "TG"
return httpx.Response(
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
)
if provider == "ollama":
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
return httpx.Response(200, json={"data": [{"id": "latest"}]})
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
assert result["base_url"] == preset["base_url"]
assert "test-key" not in str(result)
@pytest.mark.parametrize(
"provider,base,api,path",
[
(
"minimax",
"https://api.minimax.io/anthropic/v1/",
"anthropic-messages",
"/anthropic/v1/models",
),
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
(
"alibaba",
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
(
"alibaba",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
"openai-completions",
"/api/v1/models",
),
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
],
)
def test_urls_preserve_region_and_gateway(provider, base, api, path):
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
assert url.host == httpx.URL(base).host
assert url.path == path
@pytest.mark.asyncio
async def test_anthropic_pagination_and_release_order(monkeypatch):
seen = []
def handle(request):
seen.append(request.url.params.get("after_id"))
if len(seen) == 1:
return httpx.Response(
200,
json={
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
"has_more": True,
"last_id": "old",
},
)
return httpx.Response(
200,
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
assert seen == [None, "old"]
assert result["models"] == ["new", "old"]
@pytest.mark.asyncio
async def test_dashscope_pagination(monkeypatch):
def handle(request):
page = int(request.url.params["page_no"])
return httpx.Response(
200,
json={
"output": {
"total": 2,
"models": [
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
],
}
},
)
mock_http(monkeypatch, handle)
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
assert result["models"] == ["page-2", "page-1"]
@pytest.mark.asyncio
@pytest.mark.parametrize(
"payload",
[
{},
{"data": []},
{"data": [None]},
{"data": [{}]},
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
],
)
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
with pytest.raises(discovery.LLMProviderCatalogError):
await catalog.refresh_llm_provider_preset("minimax", "test-key")
@pytest.mark.asyncio
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
assert len(result["models"]) == 140
assert result["models"][0] == "model-139"
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
@pytest.mark.asyncio
async def test_empty_ollama_catalog_is_valid(monkeypatch):
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
result = await catalog.refresh_llm_provider_preset("ollama")
assert result["models"] == []
assert result["model"] == ""
@pytest.mark.asyncio
async def test_opencode_documented_protocols(monkeypatch):
models = [
"minimax-m3",
"qwen3.8-max",
"gpt-5.6-luna",
"grok-4.6",
"muse-spark-1.3-contributor",
"kimi-k3",
]
mock_http(
monkeypatch,
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
)
result = await catalog.refresh_llm_provider_preset("opencode-go")
assert list(result["model_provider_apis"].values()) == [
"anthropic-messages",
"anthropic-messages",
"openai-responses",
"openai-responses",
"openai-responses",
"openai-completions",
]
@pytest.mark.asyncio
async def test_retry_transient_error_only(monkeypatch):
calls = []
def handle(request):
calls.append(request)
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
mock_http(monkeypatch, handle)
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 2
calls.clear()
def unauthorized(request):
calls.append(request)
return httpx.Response(401, text="secret-upstream-text")
mock_http(monkeypatch, unauthorized)
with pytest.raises(httpx.HTTPStatusError) as error:
await catalog.refresh_llm_provider_preset("minimax", "test-key")
assert len(calls) == 1
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)

View File

@@ -0,0 +1,165 @@
from copy import deepcopy
from types import SimpleNamespace
from unittest.mock import AsyncMock
import httpx
import pytest
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from app.api.v1 import settings as api
from app.models.system_setting import SystemSetting
from app.services import llm_model_catalog as discovery
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
REAL_CLIENT = httpx.AsyncClient
@pytest.fixture
def settings_db(monkeypatch, tmp_path):
# Real SQL reads, commits and reloads; do not mock the settings helpers.
engine = create_engine("sqlite://")
SystemSetting.__table__.create(engine)
session = Session(engine)
db = SimpleNamespace(
execute=AsyncMock(side_effect=session.execute),
add=session.add,
commit=AsyncMock(side_effect=session.commit),
refresh=AsyncMock(side_effect=session.refresh),
)
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
yield db
session.close()
engine.dispose()
def use_upstream(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
)
@pytest.mark.asyncio
async def test_fresh_database_lists_all_builtin_presets(settings_db):
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
assert len(result["data"]) == 9
assert all(row["models"] for row in result["data"])
assert "MiniMax-M3" in result["data"][0]["models"]
@pytest.mark.asyncio
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
runtime = {
"ai_provider": {
"default_provider": "minimax",
"service_token": "private-service-token",
"providers": {
"minimax": {
"base_url": "https://saved.test/anthropic",
"model": "old",
"api_key": "saved-secret",
}
},
}
}
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
before = await api.get_setting_payload(settings_db, "external_integrations")
seen = []
def handle(request):
seen.append(request)
assert str(request.url) == "https://draft.test/anthropic/v1/models"
assert request.headers["x-api-key"] == "draft-secret"
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
use_upstream(monkeypatch, handle)
draft = api.AIProviderIntegrationUpdate(
provider="minimax",
base_url="https://draft.test/anthropic",
model="unsaved-model",
api_key="draft-secret",
service_token="unsaved-service-token",
)
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
result = await api.get_ai_provider_presets(None, settings_db)
preset = next(row for row in result["data"] if row["provider"] == "minimax")
assert len(seen) == 1
assert preset["models"] == ["MiniMax-M3", "old"]
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
assert await api.get_setting_payload(settings_db, "external_integrations") == before
assert "secret" not in str(result)
assert "private-service-token" not in str(result)
assert draft.model == "unsaved-model"
stored = await settings_db.execute(
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
)
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
@pytest.mark.asyncio
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
settings_db, monkeypatch
):
await api.save_setting_payload(
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
)
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
commits = settings_db.commit.await_count
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
assert error.value.status_code == 502
assert "private-upstream-secret" not in error.value.detail
assert settings_db.commit.await_count == commits
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
"models"
] == ["saved-model"]
@pytest.mark.asyncio
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
def fail_request(request):
pytest.fail("must not query a private catalog without credentials")
use_upstream(monkeypatch, fail_request)
with pytest.raises(api.HTTPException) as error:
await api.refresh_ai_provider_preset("minimax", None, settings_db)
assert error.value.status_code == 400
assert "API Key" in error.value.detail
@pytest.mark.asyncio
@pytest.mark.parametrize(
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
)
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
result = await api._check_ai_provider_lightweight(
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
)
assert result["success"] is False
assert result["connected"] is False
assert "private-upstream-secret" not in result["message"]
@pytest.mark.asyncio
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
assert result["success"] is False
@pytest.mark.asyncio
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
use_upstream(
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
)
preset = list_fallback_llm_provider_presets()[0]
result = await api._check_ai_provider_lightweight(
{**preset, "model": "future-model", "api_key": "test-key"}, 5
)
assert result["success"] is True
assert result["url"].endswith("/anthropic/v1/models")

View File

@@ -0,0 +1,141 @@
from pathlib import Path
import sys
from unittest.mock import patch
import httpx
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
with patch(
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
return_value={},
):
from aiprovider.provider_service import ProviderService
from aiprovider.schemas import SituationalAnalysisRequest
from app.api.v1.settings import _runtime_config_from_ai_payload
def test_custom_openai_gateway_keeps_its_configured_protocol():
config = _runtime_config_from_ai_payload(
{
"default_provider": "openai",
"providers": {
"openai": {
"base_url": "https://gateway.test/v1",
"provider_api": "openai-completions",
"model": "gpt-6-astra",
},
},
}
)["llm_config"]
service = ProviderService({**config, "api_key": "test-key"})
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
@pytest.mark.asyncio
@pytest.mark.parametrize(
"model,path",
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
)
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
llm = _runtime_config_from_ai_payload(
{
"default_provider": "opencode-go",
"providers": {
"opencode-go": {
"model": model,
"api_key": "test-key",
"model_provider_apis": {model: "openai-completions"},
}
},
}
)["llm_config"]
service = ProviderService(llm)
seen = []
async def post(**request):
seen.append(request)
assert request["path"] == path
assert request["request_body"]["model"] == model
if path == "/responses":
assert request["request_body"]["store"] is False
assert "max_tokens" not in request["request_body"]
return {
"output": [
{"type": "reasoning", "summary": [{"text": "reason"}]},
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
]
}
return {"content": [{"type": "text", "text": "OK"}]}
monkeypatch.setattr(service, "_post", post)
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
assert len(seen) == 1
assert result.content == "OK"
@pytest.mark.asyncio
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
seen.append(str(request.url))
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
service = ProviderService(
{
"provider": "minimax",
"provider_api": "anthropic-messages",
"base_url": base,
"api_key": "test-key",
"model": "MiniMax-M3",
}
)
result = await service.analyze(
SituationalAnalysisRequest(title="test", objective="reply OK")
)
assert result.content == "OK"
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
@pytest.mark.asyncio
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
seen = []
client_type = httpx.AsyncClient
def handle(request):
import json
seen.append(request)
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
assert request.headers["user-agent"] == "Planet/1.0"
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
transport = httpx.MockTransport(handle)
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
service = ProviderService(
{
"provider": "opencode-go",
"provider_api": "anthropic-messages",
"base_url": "https://opencode.ai/zen/go/v1",
"api_key": "test-key",
"model": "minimax-m3",
}
)
await service.analyze(
SituationalAnalysisRequest(
title="test",
objective="reply OK",
thinking={"type": "enabled"},
context={"session_id": session_id},
)
)
session_headers = [request.headers["x-opencode-session"] for request in seen]
assert session_headers[0] == session_headers[1]
assert session_headers[0] != session_headers[2]

View File

@@ -0,0 +1,160 @@
from datetime import UTC, datetime
from unittest.mock import AsyncMock
import pytest
from sqlalchemy import create_engine, event
from sqlalchemy.orm import Session
from app.db.session import Base
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.system_setting import SystemSetting
from app.models.task import CollectionTask
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
from app.services.tv_catalog import get_tv_catalog_page
from app.services.tv_streams import normalize_tv_settings
class CatalogSession:
"""Execute the real catalog queries against an isolated SQLite database."""
def __init__(self, session):
self.session = session
async def execute(self, query):
return self.session.execute(query)
async def scalar(self, query):
return self.session.scalar(query)
async def scalars(self, query):
return self.session.scalars(query)
@pytest.fixture
def catalog_db():
engine = create_engine("sqlite:///:memory:")
@event.listens_for(engine, "connect")
def register_functions(connection, _record):
connection.create_function(
"concat_ws",
-1,
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
)
Base.metadata.create_all(
engine,
tables=[
CollectionTask.__table__,
DataSnapshot.__table__,
CollectedData.__table__,
SystemSetting.__table__,
],
)
with Session(engine) as session:
for index in range(135):
session.add(
CollectedData(
source="news_live_streams",
source_id=f"channel-{index:03}",
data_type="news_live_stream",
name=f"Channel {index:03}",
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
is_current=True,
is_valid=1,
extra_data={
"stream_url": "https://example.invalid/live.m3u8",
"region": "Canada",
},
)
)
session.flush()
yield CatalogSession(session)
engine.dispose()
@pytest.mark.asyncio
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
first = await get_tv_catalog_page(catalog_db, limit=50)
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
assert len(set(ids)) == first["source_count"] == 145
assert ids[-1] == "channel-134"
assert third["next_offset"] is None and not third["has_more"]
beyond = await get_tv_catalog_page(catalog_db, offset=200)
assert beyond["sources"] == [] and not beyond["has_more"]
@pytest.mark.asyncio
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
assert payload["total"] == 1 and payload["source_count"] == 145
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
@pytest.mark.asyncio
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
assert payload["selected_source"]["id"] == "channel-134"
assert "channel-134" not in [source["id"] for source in payload["sources"]]
assert payload["default_source_id"] == "aljazeera-mubasher"
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
@pytest.mark.asyncio
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
for name, values in [
("Disabled", {"extra_data": {"is_enabled": False}}),
("Historical", {"is_current": False}),
("Invalid", {"is_valid": 0}),
("Deleted", {"deleted_at": datetime.now(UTC)}),
("Replacement", {"source_id": "channel-134"}),
]:
record = dict(
source="news_live_streams",
source_id=name,
name=name,
data_type="news_live_stream",
is_current=True,
is_valid=1,
)
catalog_db.session.add(CollectedData(**{**record, **values}))
catalog_db.session.flush()
payload = await get_tv_catalog_page(catalog_db, q="replacement")
assert payload["source_count"] == 145
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
assert settings["default_source_id"] == "aljazeera-mubasher"
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
@pytest.mark.asyncio
@pytest.mark.parametrize(
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
)
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
monkeypatch, config, expected
):
collector = NewsLiveStreamsCollector()
channels = [
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
]
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
streams = [
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
for channel in channels
]
monkeypatch.setattr(
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
)
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
assert len(records) == expected
assert all(record["source_id"] != "sport" for record in records)

View File

@@ -0,0 +1,37 @@
from app.services.tv_streams import (
CCTV4_HOMEPAGE,
CCTV4_STREAM,
UNSUPPORTED_CCTV4_STREAM,
normalize_tv_settings,
normalize_tv_source,
)
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
settings = normalize_tv_settings(None)
assert settings["default_source_id"] == "aljazeera-mubasher"
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert cctv["source_type"] == "hls"
assert cctv["stream_url"] == CCTV4_STREAM
assert "美洲版" in cctv["name"]
assert cctv["use_proxy"] is False
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
def test_saved_legacy_default_is_corrected_without_mutating_input():
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
source = normalize_tv_source(saved)
assert source["source_type"] == "hls"
assert source["stream_url"] == CCTV4_STREAM
assert source["use_proxy"] is False
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
assert source["stream_url"] == custom["stream_url"]
assert source["source_type"] == "hls"
assert source["use_proxy"] is True
assert settings["auto_fallback"] is False

View File

@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
longitude=-90.05,
metadata={
"organization": "xAI",
"location": "3231 Riverport Rd, Memphis, TN 38109",
"gpu_type": "H100",
"gpu_count": 100000,
"value": "20000",
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
compute_center_locations.set_compute_center_location_cache({})
def test_epoch_country_level_coordinates_are_unresolved():
record = _build_record(
record_id=33,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Meta 100k",
country="United States",
city="",
latitude=39.381266,
longitude=-97.922211,
metadata={
"location": "United States of America",
"raw_country": "United States of America",
},
)
payload = convert_compute_centers_to_geojson([record])
assert payload["features"] == []
assert len(payload["unresolved"]) == 1
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
def test_epoch_city_coordinates_are_not_labeled_precise():
record = _build_record(
record_id=34,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Seoul Cluster",
country="South Korea",
city="",
latitude=37.5665,
longitude=126.978,
metadata={"location": "Seoul", "raw_country": "South Korea"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.is_resolved
assert result.location.location_precision == "city"
assert result.location.needs_confirmation is True
assert result.location.is_estimated is True
assert result.location.city is None
assert result.location.matched_location_name == "Seoul"
assert "Epoch Location: Seoul" in result.location.location_source_note
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
record = _build_record(
record_id=37,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Petrobras Fênix",
country="Brazil",
city="",
latitude=-22.909534,
longitude=-43.209934,
metadata={
"location": "Rio de Janeiro, Brazil",
"raw_country": "Brazil",
},
)
payload = convert_compute_centers_to_geojson([record])
feature = payload["features"][0]
assert feature["properties"]["location_precision"] == "city"
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
"location_source_note"
]
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-35": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-35",
"name": "Confirmed Site",
"latitude": 35.0,
"longitude": 139.0,
"precision": "site",
"confidence": 1.0,
"location_source": "official_facility_registry",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=35,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Japan Cluster",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
"raw_country": "Japan",
},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "stored_compute_center_location"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-36": {
"source": "epoch_ai_gpu",
"source_id": "epoch_ai_gpu-36",
"name": "Legacy Country Point",
"latitude": 36.386493,
"longitude": 138.59223,
"precision": "precise",
"confidence": 1.0,
"location_source": "source_coordinates",
"needs_confirmation": False,
}
})
try:
record = _build_record(
record_id=36,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Legacy Japan Cluster",
country="Japan",
city="",
latitude=36.386493,
longitude=138.59223,
metadata={"location": "Japan", "raw_country": "Japan"},
)
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location is None
finally:
compute_center_locations.set_compute_center_location_cache({})
def test_verified_sites_replace_epoch_country_center_coordinates():
specifications = [
(
"epoch_jamstec_zettascaler_2_0_gyoukou",
"Gyoukou 2.0",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_jamstec_zettascaler_2_2_gyoukou",
"Gyoukou 2.2",
"Japan",
36.386493,
138.59223,
[139.625008, 35.380924],
),
(
"epoch_argonne_nl_sophia__formerly_theta_",
"Sophia",
"United States of America",
39.381266,
-97.922211,
[-87.984498648496, 41.718109383287235],
),
]
records = []
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
specifications, start=40
):
record = _build_record(
record_id=index,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name=name,
country=country,
city="",
latitude=latitude,
longitude=longitude,
metadata={"location": country, "raw_country": country},
)
record.source_id = source_id
records.append(record)
payload = convert_compute_centers_to_geojson(records)
assert payload["unresolved"] == []
assert len(payload["features"]) == 3
for feature, specification in zip(payload["features"], specifications):
assert feature["geometry"]["coordinates"] == specification[-1]
assert feature["properties"]["location_precision"] == "site"
assert feature["properties"]["location_source"] == "official_facility_registry"
assert "https://" in feature["properties"]["location_source_note"]
def test_verified_site_beats_epoch_city_level_source_coordinates():
record = _build_record(
record_id=44,
source="epoch_ai_gpu",
data_type="gpu_cluster",
name="Gyoukou 2.2",
country="Japan",
city="Tokyo",
latitude=35.68882,
longitude=139.692526,
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
)
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
result = compute_center_locations.resolve_compute_center_location_full(
record, record.extra_data
)
assert result.location.location_source == "official_facility_registry"
assert result.location.location_precision == "site"
assert (result.location.latitude, result.location.longitude) == (
35.380924,
139.625008,
)
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
compute_center_locations.set_compute_center_location_cache({
"epoch_ai_gpu:epoch_ai_gpu-32": {
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
@pytest.mark.asyncio
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
@pytest.mark.asyncio
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
llm_candidate = compute_center_locations.LocationCandidate(
latitude=45.764,
longitude=4.8357,
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
@pytest.mark.asyncio
async def test_compute_centers_geojson_endpoint_returns_stats():
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
@pytest.mark.asyncio
async def test_visualization_geo_summary_returns_counts(monkeypatch):
async def build_without_shared_cache(**kwargs):
return await kwargs["builder"]()
# Exercise the fixture DB, never the running development server's Redis payload.
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
records = [
_build_record(
record_id=1,
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
city="Memphis",
latitude=35.15,
longitude=-90.05,
metadata={"value": "20000", "unit": "TFlop/s"},
metadata={
"location": "3231 Riverport Rd, Memphis, TN 38109",
"value": "20000",
"unit": "TFlop/s",
},
),
]
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
class _FakeSession:
async def execute(self, query):
query_text = str(query).lower()
if "group by collected_data.source" in query_text:
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
if "bgp_incidents" in query_text:
return _ScalarResult(scalar_value=2)
if "bgp_anomalies" in query_text:
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
def _fake_ror(query):
assert query == "Oak Ridge National Laboratory"
return {
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
@pytest.mark.asyncio
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
assert "FORBIDDEN_PRECISIONS" in text, (
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
)
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
from app.core.compute_types import compute_site_type
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
metadata={"location_precision": "site"})
result = convert_compute_centers_to_geojson([record])
assert compute_site_type(record) == kind
assert result["features"][0]["properties"]["site_type"] == kind
record.source = "epoch_ai_gpu"
record.data_type = "gpu_cluster"
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
assert compute_site_type(record) == kind
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind

View File

@@ -1,5 +1,9 @@
import pytest
import importlib
import json
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from app.core.websocket.manager import ConnectionManager
from app.core.websocket.broadcaster import DataBroadcaster
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
broadcaster = DataBroadcaster()
async def load_current(keys):
assert keys == ["1"]
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
broadcaster.enqueue_vessel_update(
{
"source": "aisstream_vessels",
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
assert sent[0]["vessels"] == [
{
"mmsi": 1,
"lat": 59.1,
"lon": 10.1,
"source": "aisstream_vessels",
"lat": 60.0,
"lon": 11.0,
"source": "barentswatch_vessels",
"action": "upsert",
"created": None,
}
]
@pytest.mark.asyncio
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
manager = ConnectionManager()
socket = FakeWebSocket()
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
json.dumps(config)
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
await manager.broadcast_vessels({"vessels": vessels})
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
manager.unsubscribe(socket, ["vessels"])
await manager.broadcast_vessels({"vessels": vessels})
assert len(socket.sent) == 3
@pytest.mark.asyncio
async def test_global_vessel_removal_does_not_require_coordinates():
manager = ConnectionManager()
socket = FakeWebSocket()
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
websocket_module = importlib.import_module("app.api.v1.websocket")
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
app = FastAPI()
app.include_router(websocket_module.router)
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
assert socket.receive_json()["type"] == "connection_established"
socket.send_json({
"type": "subscribe",
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
})
response = socket.receive_json()
assert response["type"] == "subscription_confirmed"
assert response["data"]["vessels"]["scope"] == "global"
assert response["data"]["vessels"]["type"] == []
@pytest.mark.asyncio
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
broadcaster = DataBroadcaster()
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
async def fail_read(_keys):
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
raise RuntimeError("database unavailable")
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
with pytest.raises(RuntimeError, match="database unavailable"):
await broadcaster.flush_vessel_updates()
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61

View File

@@ -25,6 +25,25 @@
{
"id": "china-taiwan-penghu",
"name": "Taiwan and Penghu",
"sourceIsoA3": ["TWN"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-hong-kong",
"name": "Hong Kong",
"sourceIsoA3": ["HKG"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
"labelPolicy": "show_country_only"
},
{
"id": "china-macao",
"name": "Macao",
"sourceIsoA3": ["MAC"],
"action": "union_to_country",
"targetIsoA3": "CHN",
"hoverIsoA3": "CHN",
@@ -57,6 +76,7 @@
{
"id": "kosovo",
"name": "Kosovo",
"sourceIsoA3": ["KOS", "XKS"],
"action": "render_as_disputed_with_parent",
"parentIsoA3": "SRB",
"hoverIsoA3": "SRB",

View File

@@ -1,5 +1,3 @@
version: '3.8'
services:
aiprovider:
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
@@ -32,7 +30,7 @@ services:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: planet_db
ports:
- "5432:5432"
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
@@ -45,7 +43,7 @@ services:
image: redis:7-alpine
container_name: planet_redis
ports:
- "6379:6379"
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
volumes:
- redis_data:/data
healthcheck:

View File

@@ -8,6 +8,300 @@ This project follows the repository versioning rule:
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.76.2] — 2026-10-01
Released: 2026-10-01
### Highlights
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
### Added / Fixed / Improved
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
---
## [0.76.1] — 2026-10-01
Released: 2026-10-01
### Highlights
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
### Added / Fixed / Improved
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
---
## [0.76.0] — 2026-09-30
Released: 2026-09-30
### Highlights
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
### Added / Fixed / Improved
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
---
## [0.75.4] — 2026-09-29
Released: 2026-09-29
### Highlights
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
### Improved / Fixed
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
---
## [0.75.3] — 2026-09-27
Released: 2026-09-27
### Highlights
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
### Improved / Fixed
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
---
## [0.75.2] — 2026-09-25
Released: 2026-09-25
### Highlights
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
### Fixed
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
---
## [0.75.1] — 2026-09-25
Released: 2026-09-25
### Highlights
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
### Added / Fixed / Improved
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
---
## [0.75.0] — 2026-09-23
Released: 2026-09-23
### Highlights
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
### Added / Fixed / Improved
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
---
## [0.74.7] — 2026-09-17
Released: 2026-09-17
### Highlights
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
### Added / Fixed / Improved
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
---
## [0.74.6] — 2026-09-16
Released: 2026-09-16
### Highlights
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
### Added / Fixed / Improved
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
---
## [0.74.5] — 2026-09-13
Released: 2026-09-13
### Highlights
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
### Added / Fixed / Improved
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
---
## [0.74.4] — 2026-09-13
Released: 2026-09-13
### Highlights
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
### Added / Fixed / Improved
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
---
## [0.74.3] — 2026-09-13
Released: 2026-09-13
### Highlights
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
### Added / Fixed / Improved
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
---
## [0.74.2] — 2026-07-01
Released: 2026-07-01
### Highlights
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`,删除重复维护的旧 Claude command 入口。
- 强化视觉证据规则:截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置,而不是跳过后猜测。
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback,同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
### Added / Fixed / Improved
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容,保留最新单一入口和 harness 验证说明。
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate,补齐路径解析、访问失败报告和 OCR fallback 边界。
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口,并更新文档受众计划中的旧路径引用。
---
## [0.74.1] — 2026-06-30
Released: 2026-06-30
### Highlights
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内,移除旧的全局“选择资产/上传”工具栏。
- 新增字段级图片拖拽反馈,拖到对应字段时直接提示将图片复制为 Logo 或标题图。
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式,并同步中英文使用手册、快速开始和控制台上下文文档。
### Added / Fixed / Improved
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
- `FieldGrid` 支持按字段注入自定义输入控件,同时复用统一草稿提交路径。
- 品牌上传拖拽态改为低饱和 tactile 配色,上传按钮保持蓝色轻立体样式,容器内上下/右侧留白对齐为 3px。
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
---
## [0.74.0] — 2026-06-30
Released: 2026-06-30
### Highlights
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口,减少英文界面中文残留。
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示,避免英文态裁切或错位。
- 将容易遗漏的 i18n 入口和视觉回归加入 harness,让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
### Added / Fixed / Improved
- 新增 Earth runtime i18n 入口,统一高清材质、启动状态、错误提示和图层状态文案来源。
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
- 更新控制台与 Earth 布局规则,保留 brand 尺寸语义,同时让标题、副标题和通知胶囊按内容完整显示。
- 扩展 frontend smoke 与 harness 文档,固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
- 更新 Earth 新闻本地化服务与测试,确保英文界面新闻内容不再回退中文 UI 文案。
---
## [0.73.0] — 2026-06-29
Released: 2026-06-29
### Highlights
- 新增前端统一 i18n 基础设施,让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
- 控制台侧边栏偏好面板接入语言与主题切换,并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
- 扩展 harness 与 smoke 覆盖,确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
### Added / Fixed / Improved
- 新增 `frontend/src/i18n/`,用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译,并覆盖动态计数字符串。
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership,避免首屏溢出和状态词裁切。
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则,记录语言迁移边界、状态指示器布局约束和一屏验证要求。
---
## [0.72.0] — 2026-06-29
Released: 2026-06-29
### Highlights
- 将 agent 入口收敛到单一 `AGENTS.md`,并让 harness 明确阻止小写入口再次分叉。
- 新增完整本地 harness 验证层,覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
- 扩展 Earth News 与控制台 smoke,确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
### Added / Fixed / Improved
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本,并将未跟踪 harness 设施纳入发布。
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出,改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档,明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
---
## [0.71.1] — 2026-06-26
Released: 2026-06-26

View File

@@ -9,7 +9,7 @@ or release workflows.
Existing project rules are authoritative:
1. `rules.md`
2. `agents.md`
2. `AGENTS.md`
3. Current implementation docs under `docs/technical/`
4. Existing scripts, especially `planet.sh`
5. Existing Gitea workflow files under `.gitea/workflows/`
@@ -18,6 +18,17 @@ When harness guidance conflicts with any of the above, keep the existing rule,
do not overwrite the existing workflow, and add a compatibility note here or in
`docs/harness-audit.md`.
For frontend or documentation audits, also read the Rules Coverage Evidence
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
static checks, Playwright smoke coverage, and remaining manual review areas, so
an agent can distinguish a proved harness pass from a rule that still needs
human-quality inspection.
When the user describes work with product words rather than module names, use
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
`地球`, `模型供应商`, and `发版` to the required rule modules.
## Starting Work
Recommended startup flow:
@@ -68,11 +79,25 @@ git diff --unified=0 HEAD -- <path>
## Harness Commands
Windows desktop launcher tests run separately on Windows with
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
They use temporary desktop folders and mocked restart commands; no UAC prompt
or real service restart is triggered. The WSL bridge regression suite,
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
disposable command substitutes. For a real entry-point check without restarting,
run the repository `planet.cmd --install-only`, then the generated desktop
launcher with the same flag. Installation and mocked checks do not prove a live
elevated restart; report that separately when it has not been exercised.
| Tier | Command | What It Does |
| --- | --- | --- |
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, optional Helm checks, and opt-in Docker image smoke builds. |
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
Docker image smoke builds are expensive and are off by default:
@@ -80,6 +105,106 @@ Docker image smoke builds are expensive and are off by default:
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
```
Frontend Playwright smoke runs by default in full validation after the frontend
build. It starts a local Vite preview and checks the `/` to Earth redirect,
public pages, unknown-route login fallback, protected admin route login
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
mocked API content, Docs detail page
language/theme/search interactions, every Docs catalog slug exposed by the
frontend/backend metadata, the Earth iframe entry point, login error handling,
register + email verification, password reset, standalone email verification,
and authenticated `super_admin` rendering for every admin route plus core
`section` deep links derived from the actual admin route and section config.
Authenticated admin
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
mobile passes also fail on global horizontal overflow so table/detail panels
must keep overflow ownership inside their own scroll regions. To enforce the
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
hard rendered check: the shell must resolve to the viewport height through the
root 100% height chain, `#root`/document/body must not gain vertical overflow,
and the desktop sidebar account/preferences area must remain inside the first
viewport while the nav owns any excess scrolling. The smoke also
derives the sidebar menu from the actual admin route manifest and clicks every
visible `super_admin` menu entry on both desktop and mobile viewports, then
exercises safe interaction paths for admin search, section tabs, the AI settings
shortcut, logs view switching, user dialog opening, and data distribution toggles.
It also exercises Earth News source testing, add/cancel source draft behavior,
and manual news group creation against mocked `/earth/news-*` APIs.
Documented AI and collector
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
`/collection-management?section=collector_credentials` are part of the rendered
smoke surface:
```bash
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
```
### Visual Evidence And Style Consistency
- Treat user-provided screenshots and images as primary visual evidence. If a
screenshot contradicts written text, inspect the image first and explicitly
call out the mismatch before deciding what to change.
- Path resolution is part of visual evidence handling. If a referenced
screenshot path cannot be opened, try reasonable local equivalents first:
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
thread attachments, repository files, and obvious local attachment/download
locations.
- If the image still cannot be found or opened, report the exact path/access
blocker instead of guessing. Do not infer image content from the filename, alt
text, surrounding prose, logs, or memory.
- OCR is acceptable evidence for text-only questions or non-multimodal
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
and rendering issues still require a real visual inspection or an explicit
"could not verify visually" note.
- Same-category UI surfaces must use one visual system per product area. Badges,
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
controls should reuse the shared component, shared token, or established CSS
owner for that area instead of introducing a page-local lookalike.
- `scripts/harness/frontend-rules-check.sh` warns when semantic
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
approved React and Earth CSS owner files. A warning means the reviewer should
either move the style into the shared owner or document why this is a genuinely
new visual family.
### Earth I18n Harness Rules
Earth i18n work must validate rendered behavior, not only static text lookup.
Agents often miss dynamic strings that are created after initial page load, so
the smoke treats these as first-class i18n surfaces:
- **Visible text and attributes**: translated checks must include `innerText`
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
buttons are user-facing copy, not implementation details.
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
BGP markers, compute centers, vessels, and news must render field labels,
status values, source tags, action buttons, and disabled/tooltips in the active
language.
- **English content safety**: English mode must not fall back to Chinese news
titles, summaries, feed names, measure words, or generic status labels. If no
English localization exists, hide the item or use a neutral English fallback.
- **Brand assets**: locale switching must update both text and image assets.
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
for English while keeping the same top-left layout and logo position.
- **Controls and state**: switch/segmented-control visuals must follow the real
checked/pressed state after both direct clicks and programmatic panel changes.
A control is not valid if the state changes but the thumb, active pill, or
`aria-*` state stays stale.
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
must enter through `earthMessage(...)` plus the centralized
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
or `earth:status` events.
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
overflow their panel. Prefer a slightly wider owning panel for important
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
translated tooltip.
Current frontend smoke explicitly covers the Earth English locale flow: brand
image swap, settings language controls, panel switch visual sync, English news
filtering, English detail-card text and tooltips, and TV default/source labels.
## Environment Requirements
Required for normal development:
@@ -89,6 +214,12 @@ Required for normal development:
- `bun` for frontend dependency and build execution
- Python resolved by `uv` from the root `pyproject.toml`
Harness command lookup first checks the current non-interactive `PATH`. If a
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
path. This avoids hardcoding a dotfile while still covering agent environments
that do not inherit the user's normal shell setup.
Required for full local stack operation:
- Docker and Docker Compose
@@ -99,9 +230,26 @@ Optional for delivery smoke:
- Docker daemon for image builds
- Helm for chart lint/template checks
If a required local tool is missing, do not install system software
automatically. Report the gap and point to `./planet.sh init` or
`scripts/bootstrap-dev.sh` as the existing bootstrap path.
For routine harness validation, do not install missing system software
automatically. Report the gap and point to the explicit bootstrap entry points.
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
This bootstrap behavior is intentional; do not invoke it merely to make harness
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
Docker bootstrap regression checks use isolated command stubs and never install
packages or modify the host daemon:
```bash
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
uv run --frozen --project . python scripts/harness/test_database_startup.py
```
Database startup regressions also run in quick-check. They cover Compose
reconciliation of existing containers, visible startup errors, published-port
checks, bounded recreation that preserves volumes, and the backend connection
gate before schema initialization. Their command stubs and driver mocks do not
modify the host Docker environment.
## What Agents Must Not Change Automatically
@@ -121,6 +269,14 @@ No automatic hooks are installed in this phase. Manual reminders:
- Run `scripts/harness/quick-check.sh` before handing off small changes.
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
- Run `scripts/harness/security-check.sh` after touching config, auth,
credentials, docs examples, or generated fixtures.
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
catch direct stdout/debugger calls before they reach runtime logs.
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
route, package-manager, debug-output, and UI rule warnings.
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
route changes.
- Add focused tests before modifying backend service behavior or frontend
workflows.
- For docs changes, run the checks listed in
@@ -136,6 +292,9 @@ No automatic hooks are installed in this phase. Manual reminders:
4. Make the smallest behavior-preserving or feature-scoped change.
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
6. Update relevant docs when behavior, workflow, or operations change.
7. For rendered frontend changes, verify the affected route with Playwright or
the full harness smoke, because `bun run build` alone does not prove page
usability.
### Bug Fix
@@ -149,7 +308,8 @@ No automatic hooks are installed in this phase. Manual reminders:
1. Read `docs/documentation-coverage-rules.md`.
2. Route docs by audience: UI users, operations, or second-party developers.
3. Keep Chinese and English public docs consistent when a public doc pair exists.
3. Keep Chinese and English technical docs paired by filename; public Docs also
need matching frontend/backend metadata when exposed in the product Docs UI.
4. Run the repository-specific docs checks that match the changed files.
### Release Or Delivery Change
@@ -161,7 +321,15 @@ release process.
## Implementation Notes
- `docs/harness-audit.md` records the discovery pass that led to this harness.
- `AGENTS.md` is a compatibility entry point for tools that expect the uppercase
filename. The existing `agents.md` file remains in place.
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
`agents.md` entry has been merged into it and should remain absent.
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
`docs/technical/{zh,en}/`.
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
with mocked API data. It proves route shells, auth guards, and primary admin
sections render, but it is not a replacement for feature-specific browser QA
against a real backend.
- Frontend smoke prints phase-level progress by default. Use
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
progress lines.

View File

@@ -25,12 +25,11 @@ compatibility note, not a replacement for existing rules or architecture docs.
| File | Status | Notes |
| --- | --- | --- |
| `agents.md` | Present | Existing root agent behavior guide. It references `rules.md` and `project_context.md`. |
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
| `AGENTS.md` | Added by harness | Compatibility entry point that points to existing rules and harness docs. |
## Existing CI Gates
@@ -69,13 +68,12 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
## Missing Or Unclear Areas
- README previously listed `AGENTS.md` in the project tree while only lowercase
`agents.md` existed. The harness adds uppercase `AGENTS.md` as a compatibility
wrapper and preserves `agents.md`.
- `project_context.md` includes older roadmap assumptions such as Celery, Kafka,
TimescaleDB, MinIO, and UE5 as active stack elements. The current README and
technical docs describe Web Earth, React admin, FastAPI, PostgreSQL/Redis, and
`aiprovider` as the active local development shape.
- The older lowercase `agents.md` entry has been merged into uppercase
`AGENTS.md` so coding agents and harness tools use one source of truth.
- `project_context.md` originally included older roadmap assumptions such as
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
harness pass updated it to separate active stack facts from future directions;
current code and technical docs still remain authoritative when details drift.
- No safe automatic hook system was already configured. This phase documents
manual reminders instead of adding hooks.
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
@@ -84,19 +82,80 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
| Conflict Or Tension | Resolution |
| --- | --- |
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Added a minimal uppercase compatibility entry and preserved the existing lowercase guide. |
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
## Rules Coverage Evidence
This matrix records how the current harness checks the `rules.md` modules that
matter for this frontend and documentation pass. "Automated" means the listed
command fails when the rule regresses. "Smoke" means the rendered product route
or interaction is opened with Playwright. "Manual" means the rule is still a
judgment call and must be inspected during review.
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
the user describes work with Chinese/product terms instead of module names. The
index is the routing layer; this table is the coverage/evidence layer.
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
| --- | --- | --- | --- |
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
## Harness Files Added
| File | Purpose |
| --- | --- |
| `AGENTS.md` | Compatibility agent entry point. |
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
| `CODEMAP.md` | High-level codebase map and validation references. |
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |

View File

@@ -22,11 +22,14 @@
当前重点入口:
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)

View File

@@ -0,0 +1,62 @@
# 控制台 i18n 接入计划
**状态**:基础设施已落地,大型业务页迁移继续进行
**创建日期**:2026-06-29
**核心目标**:把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系,让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
## 背景
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口,但语言状态只保存在 `docs-lang`,不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主,导致用户切到英文文档后,控制台仍是中文。
本计划把前端语言偏好收敛到 `planet-locale`,默认 `zh-CN`,支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口,通过前端映射与全局 locale 对齐。
## 设计决策
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层,避免长期维护自研插值、hook 和资源加载逻辑。
- 前端统一语言枚举为 `zh-CN` / `en-US`;Docs 请求继续转换为 `zh` / `en`,新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
- 语言偏好首版只保存在浏览器 `localStorage`,不新增后端用户设置字段。
- `docs-lang` 保留为兼容读取和写入项,让已访问过 Docs 的浏览器能平滑迁移。
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key;大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
## 分期
### P1:统一语言基础设施
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
- 在 `frontend/src/main.tsx` 里初始化 i18n,并同步 `document.documentElement.lang`。
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
### P2:高复用界面迁移
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
- 搜索索引按当前语言展示,同时保留中英文关键词以免降低可发现性。
- 用户管理页作为独立业务页示范,迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化,以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
### P3:大型业务页收敛
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
- 过渡期保留 legacy UI 翻译桥,只处理 admin/auth 容器里的精确静态文本和属性。
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
### P4:移除过渡桥
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时,删除 legacy UI 翻译桥。
- 增加 key 完整性检查,确保 `zh-CN` 和 `en-US` 资源结构一致。
## 验证
- `cd frontend && bun run build`
- `scripts/harness/frontend-rules-check.sh`
- `scripts/harness/docs-consistency-check.sh`
- `scripts/harness/quick-check.sh`
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
## 相关文件
- `frontend/src/i18n/`:统一 locale、资源和过渡桥。
- `frontend/src/pages/Docs/Docs.tsx`:Docs 语言状态改为读取全局 locale。
- `frontend/src/admin/components/layout/AdminLayout.tsx`:控制台侧边栏语言切换、导航和搜索文案。
- `frontend/src/admin/search/indexers.ts`:Admin 搜索目标本地化。
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`:当前实现上下文。

Some files were not shown because too many files have changed in this diff Show More