Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dd41588527 | ||
|
|
45604a1ebb | ||
|
|
06c122bade | ||
|
|
96e9b15fda | ||
|
|
a8713e58fd | ||
|
|
05377bf5c6 | ||
|
|
67548d8278 | ||
|
|
4b8feeb87e | ||
|
|
a2025dbfe0 | ||
|
|
003b316d42 | ||
|
|
0c40764c58 | ||
|
|
5bfc790dec | ||
|
|
60eefb19c9 | ||
|
|
83a10a6c34 | ||
|
|
cee1996809 | ||
|
|
58671e7bc3 | ||
|
|
a54fcdbeed | ||
|
|
1dd2921674 | ||
|
|
d30f7d08c5 | ||
|
|
5bdb55f3f1 | ||
|
|
fbecf30513 | ||
|
|
19d5ac0fee |
@@ -1,139 +0,0 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
description: Create or update repository documentation from current code changes
|
||||
argument-hint: Optional: topic to document, or leave empty to infer from git diff
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — Documentation Workflow
|
||||
|
||||
## Goal
|
||||
|
||||
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
|
||||
|
||||
## Repository Rules
|
||||
|
||||
Before deciding scope, check whether the repository has a documentation rules file:
|
||||
|
||||
```bash
|
||||
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
|
||||
```
|
||||
|
||||
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Understand The Change
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
rg --files docs
|
||||
```
|
||||
|
||||
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — Decide Scope
|
||||
|
||||
- Prefer updating an existing relevant document over creating a duplicate.
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents only when the change crosses meaningful domains.
|
||||
- Keep filenames lowercase and hyphenated.
|
||||
- Apply the repository-specific rules file before writing.
|
||||
|
||||
#### Document Audience Routing (Planet)
|
||||
|
||||
In this repository, classify the action's performer before picking a target file:
|
||||
|
||||
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
|
||||
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
|
||||
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
|
||||
|
||||
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
|
||||
|
||||
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
|
||||
|
||||
### Step 3 — Write
|
||||
|
||||
Explain:
|
||||
|
||||
- Background/problem: what was wrong or missing before.
|
||||
- Core design decisions and rationale.
|
||||
- Operational or user-facing impact.
|
||||
- Relevant code paths, only when useful for future maintainers.
|
||||
|
||||
Style:
|
||||
|
||||
- Follow the repository’s existing language and heading conventions.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Prefer tables for comparisons or parameter lists.
|
||||
- Keep snippets concise and relevant.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
### Step 4 — Verify
|
||||
|
||||
- Read the completed docs once for clarity and stale statements.
|
||||
- Verify referenced paths exist with `test -e` or `rg --files`.
|
||||
- Run applicable checks from `docs/documentation-coverage-rules.md`.
|
||||
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
|
||||
|
||||
### Step 5 — Report
|
||||
|
||||
Summarize changed docs and verification:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- path/to/doc.md — what changed
|
||||
|
||||
Verified:
|
||||
- checks that passed
|
||||
- checks that could not be run, if any
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- Do not leave placeholder docs.
|
||||
- Do not duplicate bilingual files byte-for-byte.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
|
||||
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
|
||||
- Keep docs maintainable and concise.
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -1,160 +0,0 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
BIN
.codex/screenshots/earth-i18n-current-page.png
Normal file
|
After Width: | Height: | Size: 1.2 MiB |
BIN
.codex/screenshots/earth-i18n-hd-texture-status.png
Normal file
|
After Width: | Height: | Size: 4.4 KiB |
BIN
.codex/screenshots/i18n-admin-data-sidebar.png
Normal file
|
After Width: | Height: | Size: 109 KiB |
BIN
.codex/screenshots/i18n-ai-tool-calls.png
Normal file
|
After Width: | Height: | Size: 124 KiB |
BIN
.codex/screenshots/i18n-earth-brand-config.png
Normal file
|
After Width: | Height: | Size: 170 KiB |
BIN
.codex/screenshots/i18n-earth-hud-brand.png
Normal file
|
After Width: | Height: | Size: 772 KiB |
BIN
.codex/screenshots/i18n-settings-notifications.png
Normal file
|
After Width: | Height: | Size: 95 KiB |
BIN
.codex/screenshots/i18n-settings-system.png
Normal file
|
After Width: | Height: | Size: 96 KiB |
BIN
.codex/screenshots/sidebar-left-align.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
@@ -3,8 +3,23 @@
|
||||
!pyproject.toml
|
||||
!uv.lock
|
||||
!VERSION
|
||||
!frontend/
|
||||
!frontend/public/
|
||||
!frontend/public/earth/
|
||||
!frontend/public/earth/assets/
|
||||
!frontend/public/earth/assets/cards/
|
||||
!frontend/public/earth/assets/cards/manifest.json
|
||||
!backend/
|
||||
!backend/**
|
||||
!config/
|
||||
!config/earth-boundary-pov-policy.china-v1.json
|
||||
!scripts/
|
||||
!scripts/build_earth_boundary_china_pov_geojson.py
|
||||
!scripts/build_earth_boundary_pmtiles.py
|
||||
!scripts/earth_boundary_policy.py
|
||||
!resources/
|
||||
!resources/earth/
|
||||
!resources/earth/boundary-pov-geometry.china-v1.json
|
||||
!aiprovider/
|
||||
!aiprovider/**
|
||||
|
||||
|
||||
5
.gitignore
vendored
@@ -160,3 +160,8 @@ data/earth-boundary-sources/
|
||||
# Git; publish PMTiles/MVT artifacts through release/deploy storage instead of
|
||||
# committing thousands of generated tile files.
|
||||
frontend/public/earth/data/boundaries/
|
||||
|
||||
# Offline comparison originals; runtime 8K maps and provenance remain versioned.
|
||||
resources/earth/candidates/nasa-blue-marble/*.jpg
|
||||
resources/earth/candidates/natural-earth-3/*.jpg
|
||||
*:Zone.Identifier
|
||||
|
||||
148
AGENTS.md
@@ -1,28 +1,41 @@
|
||||
# Planet Agent Entry Point
|
||||
# AGENTS.md
|
||||
|
||||
This is the compatibility entry point for coding agents. The older root
|
||||
`agents.md` file remains authoritative for repository-specific agent behavior;
|
||||
do not delete or replace it.
|
||||
**Planet agent harness. Defines behavior for coding agents working in this repository.**
|
||||
|
||||
## Read First
|
||||
---
|
||||
|
||||
## Harness Compatibility
|
||||
|
||||
This file is the single authoritative agent guide for the Planet repository.
|
||||
The older lowercase `agents.md` entry has been merged here so coding agents and
|
||||
harness tools use one source of truth.
|
||||
|
||||
### Source Of Truth
|
||||
|
||||
- `rules.md` is the mandatory repository rule source. Always load `core`,
|
||||
`security`, and `workflow`; load only task-relevant modules after that.
|
||||
- `AGENTS.md` defines the local agent operating mode and evidence gates.
|
||||
- `project_context.md` is background, not a rule source. Prefer newer
|
||||
implementation docs when it disagrees with current code.
|
||||
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
|
||||
goal-driven work, and release.
|
||||
- Do not duplicate long workflow text across harness files. Durable constraints
|
||||
belong in `rules.md`; task procedures belong in skills or scripts.
|
||||
|
||||
Read these files before changing code:
|
||||
|
||||
1. `rules.md` - mandatory repository rules. Always load `core`, `security`, and
|
||||
`workflow`; load `docs`, `frontend`, `backend`, `earth`, `ai`, or `release`
|
||||
when the task touches those areas.
|
||||
2. `agents.md` - existing agent role, communication, and workflow guidance.
|
||||
3. `project_context.md` - static project background. Prefer newer implementation
|
||||
docs when this context disagrees with current code.
|
||||
4. `README.md` - current architecture, startup, and toolchain summary.
|
||||
5. `docs/HARNESS.md` - harness workflow, conflict policy, and validation tiers.
|
||||
6. `CODEMAP.md` - codebase entry points, ownership boundaries, and deeper docs.
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. `project_context.md`
|
||||
4. `README.md`
|
||||
5. `docs/HARNESS.md`
|
||||
6. `CODEMAP.md`
|
||||
|
||||
For documentation work, also read `docs/documentation-coverage-rules.md`.
|
||||
|
||||
## Start Safely
|
||||
### Start Safely
|
||||
|
||||
Before editing:
|
||||
Before broad edits:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
@@ -40,7 +53,7 @@ git diff --unified=0 HEAD -- <path>
|
||||
|
||||
Preserve user changes already present in the worktree.
|
||||
|
||||
## Validation
|
||||
### Validation
|
||||
|
||||
Fast local harness validation:
|
||||
|
||||
@@ -54,28 +67,117 @@ Full local validation:
|
||||
scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
`validate.sh` includes the quick check and the frontend Bun build. Docker image
|
||||
smoke builds are intentionally opt-in:
|
||||
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
|
||||
unless disabled by its documented environment flags. Docker image smoke builds
|
||||
are intentionally opt-in:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
## High-Risk Areas
|
||||
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
|
||||
current non-interactive environment first. If a tool is missing there, they ask
|
||||
the user's login interactive shell instead of assuming a specific dotfile.
|
||||
|
||||
### High-Risk Areas
|
||||
|
||||
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
|
||||
`destroy` cleanup.
|
||||
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
|
||||
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
|
||||
rendered smoke evidence for public pages, auth guards, authenticated admin
|
||||
route/section availability, safe navigation/search/tab interactions, mobile
|
||||
layout, and 125% / 150% zoom, not only a build.
|
||||
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
|
||||
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
|
||||
intermediate wrappers keep `min-height: 0`, and only the intended child owns
|
||||
scrolling.
|
||||
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
|
||||
workflows in the backend.
|
||||
- Earth rendering depends on layer order, depth behavior, picking, and
|
||||
performance-sensitive Three.js code.
|
||||
- Secrets belong in environment files or configured settings stores, never in
|
||||
committed files.
|
||||
- Backend service code must use structured logging instead of `print()` or
|
||||
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
|
||||
|
||||
## Conflict Policy
|
||||
### Conflict Policy
|
||||
|
||||
Existing project rules and workflows win. If new harness guidance conflicts with
|
||||
`rules.md`, `agents.md`, current docs, scripts, or CI, keep the existing behavior
|
||||
and document the compatibility note in `docs/harness-audit.md` or
|
||||
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
|
||||
behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
`docs/HARNESS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- For authorized multi-agent work, follow the Coding Agent Model Strategy in
|
||||
`rules.md` workflow: Sol for implementation, Luna for bounded supporting work,
|
||||
and Astra for complex reasoning; the main agent owns final verification.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
|
||||
focused file reads, tests, builds, linters, and harness scripts.
|
||||
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
|
||||
release work, and documentation unless the task requires it.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gates
|
||||
|
||||
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
|
||||
mock, browser capture, or visual reference, obtain evidence from the artifact
|
||||
before interpreting intent or editing code.
|
||||
- Path resolution is part of the task. If the path cannot be opened, first try
|
||||
reasonable local equivalents such as WSL/Windows path conversion,
|
||||
workspace-relative lookup, absolute paths, and attached-file locations.
|
||||
- Never guess from prompt text, filenames, previous context, logs, OCR, or
|
||||
memory when a visual artifact was provided but cannot be accessed.
|
||||
- OCR is acceptable evidence for text-only visual questions or non-multimodal
|
||||
environments; state that OCR was used as the fallback. Layout, color, spacing,
|
||||
pixel, and rendering issues need real visual inspection or a clear limitation
|
||||
note.
|
||||
- If a visual artifact still cannot be inspected, say so and pause that
|
||||
visual-dependent part of the work.
|
||||
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
|
||||
diff, direct file check, or harness result.
|
||||
- For UI and rendering changes, verify the rendered result when local tooling
|
||||
allows it.
|
||||
|
||||
---
|
||||
|
||||
## Communication
|
||||
|
||||
- Match the user's language. Use Chinese for Chinese requests unless the user
|
||||
asks otherwise.
|
||||
- Keep updates short and specific: what is being inspected, edited, or verified.
|
||||
- Final responses should summarize changed files and verification, with blockers
|
||||
stated plainly.
|
||||
- Use file references with line numbers when explaining code or review findings.
|
||||
|
||||
---
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Prefer existing project patterns over new abstractions.
|
||||
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
|
||||
a stable path exists.
|
||||
- Centralize prompts, constants, defaults, and shared request/response handling.
|
||||
- Do not add secrets, generated runtime output, or local environment files.
|
||||
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
|
||||
- Run the smallest relevant verification for the changed scope and report
|
||||
anything skipped.
|
||||
|
||||
---
|
||||
|
||||
## Prohibited
|
||||
|
||||
- Do not skip visual evidence handling when a visual artifact was provided.
|
||||
- Do not preserve obsolete harness files just because they already exist.
|
||||
- Do not invent behavior not present in code, docs, or verified external
|
||||
sources.
|
||||
- Do not rewrite unrelated files during cleanup.
|
||||
- Do not mark a task complete without checking concrete success criteria.
|
||||
|
||||
23
CODEMAP.md
@@ -17,6 +17,7 @@ are the source of detail for specific subsystems.
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
| `planet.cmd` and `scripts/windows/` | Portable Windows entry, bound desktop launcher generation, and WSL restart bridge | Delegates service lifecycle to `planet.sh`; Windows and isolated WSL regression tests live under `scripts/harness/`. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
@@ -41,11 +42,18 @@ are the source of detail for specific subsystems.
|
||||
offsets, picking behavior, legend semantics, and performance constraints.
|
||||
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
|
||||
wrapping it from harness scripts instead of duplicating its internals.
|
||||
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
|
||||
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
|
||||
interactive command path without hardcoding `.zshrc`.
|
||||
|
||||
## Validation Commands
|
||||
|
||||
```bash
|
||||
scripts/harness/doctor.sh
|
||||
scripts/harness/security-check.sh
|
||||
scripts/harness/backend-rules-check.sh
|
||||
scripts/harness/frontend-rules-check.sh
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
scripts/harness/quick-check.sh
|
||||
scripts/harness/validate.sh
|
||||
./planet.sh health
|
||||
@@ -60,8 +68,17 @@ uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py t
|
||||
cd frontend
|
||||
bun install --frozen-lockfile
|
||||
bun run build
|
||||
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
|
||||
```
|
||||
|
||||
The frontend smoke covers public routes, unauthenticated admin guards,
|
||||
login-error handling, the Earth iframe entry, and authenticated `super_admin`
|
||||
admin route/section rendering with mocked API data. Authenticated admin checks
|
||||
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
|
||||
check for accidental global horizontal overflow. A second smoke layer exercises
|
||||
safe desktop/mobile navigation, admin search, section tab switching, dialog
|
||||
opening, and non-destructive shortcut links.
|
||||
|
||||
Optional delivery smoke, when Docker and Helm are available:
|
||||
|
||||
```bash
|
||||
@@ -84,9 +101,9 @@ PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
|
||||
## Known Sharp Edges
|
||||
|
||||
- `project_context.md` includes older roadmap-era assumptions such as Celery,
|
||||
Kafka, TimescaleDB, MinIO, and UE5 being part of the active local stack. Treat
|
||||
it as background unless current README/docs/code confirm the same behavior.
|
||||
- `project_context.md` is static background for agents. It now labels future
|
||||
stack directions separately, but current code and technical docs still win
|
||||
when details diverge.
|
||||
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
|
||||
active local development shape.
|
||||
- Local `destroy` is intentionally destructive for Planet-owned Docker and build
|
||||
|
||||
@@ -83,7 +83,7 @@
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| React 18 | UI 框架 |
|
||||
| Ant Design Pro | 管理后台组件 |
|
||||
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
@@ -168,10 +168,12 @@
|
||||
|
||||
## 快速启动
|
||||
|
||||
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
|
||||
|
||||
```bash
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
|
||||
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
|
||||
|
||||
@@ -322,7 +324,7 @@ ipconfig
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
|
||||
248
agents.md
@@ -1,248 +0,0 @@
|
||||
# agents.md
|
||||
|
||||
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
|
||||
|
||||
---
|
||||
|
||||
## Harness Compatibility
|
||||
|
||||
Common agent tools should start at `AGENTS.md`. This file remains the existing
|
||||
behavior guide and must not be replaced by harness docs. For safe repository
|
||||
orientation, use:
|
||||
|
||||
- `rules.md` for mandatory project rules
|
||||
- `project_context.md` for static background
|
||||
- `docs/HARNESS.md` for validation tiers and conflict policy
|
||||
- `CODEMAP.md` for subsystem entry points and ownership boundaries
|
||||
- `docs/harness-audit.md` for the latest harness compatibility notes
|
||||
|
||||
Existing project rules and workflows stay authoritative when they conflict with
|
||||
new harness guidance.
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
|
||||
|
||||
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
|
||||
- Python FastAPI backend
|
||||
- React Admin dashboard
|
||||
- Unreal Engine 5 3D visualization
|
||||
- Multi-source data collection
|
||||
- Polarized 3D large display (4K, 120Hz)
|
||||
|
||||
---
|
||||
|
||||
## Communication Style
|
||||
|
||||
### Tone
|
||||
- **Professional but concise**
|
||||
- Technical accuracy with clarity
|
||||
- No unnecessary verbosity
|
||||
- Use code comments sparingly (explain **why**, not **what**)
|
||||
|
||||
### When Responding
|
||||
1. **Answer directly** - 1-3 sentences for simple questions
|
||||
2. **Use code blocks** for all code snippets
|
||||
3. **Include file:line_number** references when discussing code
|
||||
4. **Never** start with "I am an AI assistant" or similar phrases
|
||||
5. **Never** add unnecessary preambles/postambles
|
||||
|
||||
### Examples
|
||||
|
||||
**Good:**
|
||||
```
|
||||
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
|
||||
```
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operational Mode
|
||||
|
||||
### Plan Mode (default for complex tasks)
|
||||
- Analyze requirements
|
||||
- Propose architecture
|
||||
- Confirm with user before execution
|
||||
- **DO NOT** write code until approved
|
||||
|
||||
### Build Mode (after user approval)
|
||||
- Execute the approved plan
|
||||
- Write code, run commands
|
||||
- Verify results
|
||||
- Report completion concisely
|
||||
|
||||
### Read-Only Mode
|
||||
- Analyze code
|
||||
- Explain functionality
|
||||
- Answer questions
|
||||
- **DO NOT** modify files
|
||||
|
||||
---
|
||||
|
||||
## Decision Framework
|
||||
|
||||
### When to Ask Before Acting
|
||||
- Unclear requirements
|
||||
- Multiple implementation approaches
|
||||
- Architecture changes
|
||||
- Dependency additions
|
||||
- Anything that could break existing functionality
|
||||
|
||||
### When to Act Directly
|
||||
- Clear, approved requirements
|
||||
- Routine tasks (linting, formatting, running tests)
|
||||
- Following established patterns
|
||||
- Fixing obvious bugs
|
||||
|
||||
### When to Refuse
|
||||
- Malicious code requests
|
||||
- Security violations (secrets, credentials)
|
||||
- Anything that violates `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## Working Principles
|
||||
|
||||
### 1. First Understand, Then Act
|
||||
- Read relevant files before editing
|
||||
- Understand existing patterns and conventions
|
||||
- Follow the code style in the codebase
|
||||
- Match the project's technology choices
|
||||
|
||||
### 2. Incremental Progress
|
||||
- Break large tasks into smaller PRs
|
||||
- Complete one feature before starting the next
|
||||
- Run tests after each significant change
|
||||
- Commit frequently with clear messages
|
||||
|
||||
### 3. Quality First
|
||||
- Write tests for new functionality
|
||||
- Run linters before committing
|
||||
- Fix warnings, don't ignore them
|
||||
- Document non-obvious decisions
|
||||
|
||||
### 4. Communication Clarity
|
||||
- Use precise technical language
|
||||
- Show relevant code, not explanations
|
||||
- Report errors with context
|
||||
- Confirm understanding of requirements
|
||||
|
||||
---
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
Before marking a task complete:
|
||||
|
||||
- [ ] Code follows `rules.md` style guidelines
|
||||
- [ ] Type hints are correct and complete
|
||||
- [ ] Error handling is proper (no silent failures)
|
||||
- [ ] Tests pass locally
|
||||
- [ ] Linting passes
|
||||
- [ ] No TODO comments left behind
|
||||
- [ ] Documentation updated if needed
|
||||
- [ ] Commit message is clear
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Feature Development
|
||||
```
|
||||
1. Understand requirements
|
||||
2. Check existing patterns in codebase
|
||||
3. Design solution (brief mental model)
|
||||
4. Write code following rules.md
|
||||
5. Write/run tests
|
||||
6. Lint and format
|
||||
7. Commit with clear message
|
||||
8. Report completion
|
||||
```
|
||||
|
||||
### Bug Fix
|
||||
```
|
||||
1. Reproduce the bug (write failing test)
|
||||
2. Locate the source
|
||||
3. Fix the issue
|
||||
4. Verify test passes
|
||||
5. Check for regressions
|
||||
6. Commit fix
|
||||
```
|
||||
|
||||
### Refactoring
|
||||
```
|
||||
1. Understand current behavior
|
||||
2. Design target state
|
||||
3. Make incremental changes
|
||||
4. Preserve tests
|
||||
5. Verify functionality
|
||||
6. Clean up dead code
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Special Considerations
|
||||
|
||||
### WebSocket Services
|
||||
- Implement heartbeat mechanism (30-second intervals)
|
||||
- Handle disconnection gracefully
|
||||
- Include camera position in control frames
|
||||
- Support both update and full sync modes
|
||||
|
||||
### Data Collectors
|
||||
- Inherit from BaseCollector
|
||||
- Implement fetch() and transform() methods
|
||||
- Support incremental updates
|
||||
- Handle API changes gracefully
|
||||
|
||||
### UE5 Integration
|
||||
- Communicate via WebSocket
|
||||
- Send data frames at configurable intervals (default 5 min)
|
||||
- Support auto-cruise and manual modes
|
||||
- Optimize for 4K@120Hz rendering
|
||||
|
||||
### Multi-User Security
|
||||
- JWT tokens with 15-minute expiration
|
||||
- Redis token blacklist for logout
|
||||
- Role-based access control (RBAC)
|
||||
- Audit logging for all actions
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
### When Writing Code
|
||||
```python
|
||||
# File: backend/app/services/collectors/top500.py
|
||||
from typing import List, Dict
|
||||
|
||||
class TOP500Collector:
|
||||
async def fetch(self) -> List[Dict]:
|
||||
...
|
||||
```
|
||||
|
||||
### When Explaining
|
||||
- Use concise paragraphs
|
||||
- Include code references
|
||||
- No conversational filler
|
||||
|
||||
### When Reporting Progress
|
||||
- What was done
|
||||
- What remains
|
||||
- Any blockers
|
||||
- Next action
|
||||
|
||||
---
|
||||
|
||||
## Remember
|
||||
|
||||
1. **Rules are hard constraints** - follow `rules.md` absolutely
|
||||
2. **Context provides understanding** - use `project_context.md` for background
|
||||
3. **Role defines behavior** - follow `agents.md` for how to work
|
||||
4. **Quality over speed** - Enterprise systems require precision
|
||||
5. **Communicate clearly** - Precision in, precision out
|
||||
@@ -1,4 +1,4 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=openai-responses`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import NAMESPACE_URL, uuid4, uuid5
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
@@ -65,6 +66,7 @@ class ProviderService:
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
self.session_id = str(uuid4())
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -95,6 +97,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
if payload.context.get("session_id") is not None:
|
||||
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
|
||||
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
@@ -102,6 +106,10 @@ class ProviderService:
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif provider_api == "openai-responses":
|
||||
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
|
||||
content_blocks = self._extract_responses_blocks(data)
|
||||
content = "".join(block.text for block in content_blocks if block.text)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
@@ -200,6 +208,39 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_openai_responses(
|
||||
self, model: str, prompt: str, system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body: dict[str, Any] = {
|
||||
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["instructions"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/responses",
|
||||
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in payload.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "message":
|
||||
for part in item.get("content") or []:
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
text = part.get("text") or part.get("refusal")
|
||||
if isinstance(text, str) and text:
|
||||
blocks.append(AIContentBlock(type="text", text=text))
|
||||
elif item.get("type") == "reasoning":
|
||||
for part in item.get("summary") or []:
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
|
||||
return blocks
|
||||
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
@@ -226,7 +267,7 @@ class ProviderService:
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
@@ -243,8 +284,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
def _resolve_anthropic_thinking(
|
||||
self, thinking: dict[str, Any] | None, model: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
|
||||
return {"type": "adaptive"}
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
@@ -295,6 +340,9 @@ class ProviderService:
|
||||
headers: dict[str, str],
|
||||
request_body: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
headers = {"User-Agent": "Planet/1.0", **headers}
|
||||
if self.provider == "opencode-go":
|
||||
headers["x-opencode-session"] = self.session_id
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.http_retry_attempts + 1):
|
||||
try:
|
||||
|
||||
@@ -28,6 +28,12 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
|
||||
COPY backend /app/backend
|
||||
COPY config/earth-boundary-pov-policy.china-v1.json /app/config/earth-boundary-pov-policy.china-v1.json
|
||||
COPY scripts/build_earth_boundary_china_pov_geojson.py /app/scripts/build_earth_boundary_china_pov_geojson.py
|
||||
COPY scripts/build_earth_boundary_pmtiles.py /app/scripts/build_earth_boundary_pmtiles.py
|
||||
COPY scripts/earth_boundary_policy.py /app/scripts/earth_boundary_policy.py
|
||||
COPY resources/earth/boundary-pov-geometry.china-v1.json /app/resources/earth/boundary-pov-geometry.china-v1.json
|
||||
COPY frontend/public/earth/assets/cards/manifest.json /app/frontend/public/earth/assets/cards/manifest.json
|
||||
COPY VERSION /app/VERSION
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
from app.api.v1 import card_images
|
||||
from fastapi import APIRouter
|
||||
from app.api.v1 import (
|
||||
ai,
|
||||
@@ -12,6 +13,7 @@ from app.api.v1 import (
|
||||
alerts,
|
||||
settings,
|
||||
collected_data,
|
||||
compute_intelligence,
|
||||
data_products,
|
||||
layers,
|
||||
visualization,
|
||||
@@ -43,6 +45,7 @@ api_router.include_router(alerts.router, prefix="/alerts", tags=["alerts"])
|
||||
api_router.include_router(settings.router, prefix="/settings", tags=["settings"])
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(data_products.router, prefix="/data-products", tags=["data-products"])
|
||||
api_router.include_router(compute_intelligence.router, prefix="/data-products/compute", tags=["compute"])
|
||||
api_router.include_router(layers.router, prefix="/layers", tags=["layers"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(
|
||||
@@ -56,3 +59,5 @@ api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
api_router.include_router(interactables.router, prefix="/interactables", tags=["interactables"])
|
||||
api_router.include_router(realtime_sources.router, prefix="/realtime-sources", tags=["realtime-sources"])
|
||||
|
||||
api_router.include_router(card_images.router, prefix="/earth/card-images", tags=["card-images"])
|
||||
|
||||
108
backend/app/api/v1/card_images.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Card-image configuration, upload, catalog selection and public resolution."""
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Query, Response, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import (
|
||||
CardImageConfiguration,
|
||||
CardImagePreview,
|
||||
CardImagePublish,
|
||||
EntityType,
|
||||
MatchScope,
|
||||
)
|
||||
from app.services.card_image_configuration import (
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_files import MAX_UPLOAD_BYTES, store_image, uploaded_path
|
||||
from app.services.card_image_targets import get_target, search_targets
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
async def image_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response, _actor: User = Depends(image_admin), db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return await read_configuration(db)
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def save_configuration(
|
||||
payload: CardImagePublish,
|
||||
actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await publish_configuration(db, payload.revision, payload.config, actor)
|
||||
|
||||
|
||||
@router.post("/upload")
|
||||
async def upload_image(file: UploadFile = File(...), _actor: User = Depends(image_admin)):
|
||||
content = await file.read(MAX_UPLOAD_BYTES + 1)
|
||||
await file.close()
|
||||
return await run_in_threadpool(store_image, content, (file.filename or "Image")[:200])
|
||||
|
||||
|
||||
@router.get("/files/{filename}")
|
||||
async def image_file(filename: str):
|
||||
path = uploaded_path(filename)
|
||||
if not path.is_file():
|
||||
raise HTTPException(404, "Image not found")
|
||||
return FileResponse(
|
||||
path,
|
||||
headers={
|
||||
"Cache-Control": "public, max-age=31536000, immutable",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/targets")
|
||||
async def targets(
|
||||
entity_type: EntityType,
|
||||
scope: MatchScope = "object",
|
||||
q: str = Query(default="", max_length=100),
|
||||
limit: int = Query(default=25, ge=1, le=50),
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"items": await search_targets(db, entity_type, scope, q.strip(), limit)}
|
||||
|
||||
|
||||
@router.get("/resolve")
|
||||
async def resolve(
|
||||
response: Response,
|
||||
entity_type: EntityType,
|
||||
entity_id: str = Query(min_length=1, max_length=300),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
envelope = await read_configuration(db)
|
||||
config = CardImageConfiguration.model_validate(envelope["config"])
|
||||
context = await get_target(db, entity_type, entity_id)
|
||||
result = resolve_image(config, context)
|
||||
return {**result, "revision": envelope["revision"], "entity": context}
|
||||
|
||||
|
||||
@router.post("/preview")
|
||||
async def preview(
|
||||
payload: CardImagePreview,
|
||||
_actor: User = Depends(image_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
context = await get_target(db, payload.entity_type, payload.entity_id)
|
||||
return {**resolve_image(payload.config, context), "entity": context}
|
||||
235
backend/app/api/v1/compute_intelligence.py
Normal file
@@ -0,0 +1,235 @@
|
||||
"""Compute observations, administrator configuration and independent facility samples."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, compute_record_filter, compute_site_type_expression
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.core.earth_geography_policy import display_country
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import (
|
||||
ComputeConfiguration,
|
||||
ComputeConfigurationReset,
|
||||
ComputeConfigurationWrite,
|
||||
)
|
||||
from app.services.compute_catalog import (
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
from app.services.compute_center_locations import refresh_compute_center_location_cache
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
configuration_response,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Response
|
||||
from sqlalchemy import and_, case, func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def selected_countries(countries: str) -> list[str]:
|
||||
codes = list(dict.fromkeys(code.strip().upper() for code in countries.split(",")))
|
||||
if not codes or len(codes) > 12 or any(code not in PARTICIPANTS for code in codes):
|
||||
raise HTTPException(422, "Select 1–12 participants from the compute catalog")
|
||||
members = participant_members(codes)
|
||||
flattened = [country for participant in members.values() for country in participant]
|
||||
if len(flattened) != len(set(flattened)):
|
||||
raise HTTPException(422, "Comparison participants cannot have overlapping country members")
|
||||
return codes
|
||||
|
||||
|
||||
@router.get("/series")
|
||||
async def get_series(
|
||||
response: Response,
|
||||
countries: str = "CN,US",
|
||||
mode: Literal["global_share", "bilateral_share", "absolute"] = "global_share",
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
envelope = await read_configuration(db)
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
result = compute_history(codes, mode, ComputeConfiguration.model_validate(envelope["config"]))
|
||||
result.update(
|
||||
version=envelope["revision"],
|
||||
provenance="bundled_reports" if envelope["is_default"] else "admin_curated_reports",
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
async def configuration_admin(user: User = Depends(get_current_user)) -> User:
|
||||
if user.role not in {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}:
|
||||
raise HTTPException(403, "Administrator access required")
|
||||
return user
|
||||
|
||||
|
||||
@router.get("/configuration")
|
||||
async def get_configuration(
|
||||
response: Response,
|
||||
_actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
response.headers["Cache-Control"] = "no-store"
|
||||
return configuration_response(await read_configuration(db))
|
||||
|
||||
|
||||
@router.put("/configuration")
|
||||
async def update_configuration(
|
||||
payload: ComputeConfigurationWrite,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=payload.config, actor=actor,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
@router.post("/configuration/reset")
|
||||
async def reset_configuration(
|
||||
payload: ComputeConfigurationReset,
|
||||
actor: User = Depends(configuration_admin),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
envelope = await publish_configuration(
|
||||
db, revision=payload.revision, config=DEFAULT_CONFIGURATION, actor=actor, reset=True,
|
||||
)
|
||||
return configuration_response(envelope)
|
||||
|
||||
|
||||
async def sample_query(db: AsyncSession, countries: list[str], year: int | None):
|
||||
now = datetime.now(UTC)
|
||||
cutoff = min(datetime(year + 1, 1, 1, tzinfo=UTC), now) if year else now
|
||||
ranked = (
|
||||
select(
|
||||
DataSnapshot.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=DataSnapshot.source,
|
||||
order_by=(DataSnapshot.completed_at.desc(), DataSnapshot.id.desc()),
|
||||
)
|
||||
.label("rank"),
|
||||
)
|
||||
.where(
|
||||
DataSnapshot.source.in_(select(CollectedData.source).where(compute_record_filter(CollectedData)).distinct()),
|
||||
DataSnapshot.status == "success",
|
||||
DataSnapshot.completed_at < cutoff,
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
snapshots = select(ranked.c.id).where(ranked.c.rank == 1)
|
||||
infrastructure = compute_site_type_expression(CollectedData).in_(COMPUTE_SITE_TYPES[2:])
|
||||
# Old collector rows have ambiguous units/identity and must not become
|
||||
# national evidence merely because a new UI can display them.
|
||||
query = select(CollectedData).where(
|
||||
compute_record_filter(CollectedData),
|
||||
CollectedData.deleted_at.is_(None),
|
||||
CollectedData.snapshot_id.in_(snapshots),
|
||||
CollectedData.is_valid == 1,
|
||||
or_(CollectedData.extra_data["compute_contract"].as_string() == "v1", infrastructure),
|
||||
or_(
|
||||
infrastructure,
|
||||
CollectedData.source == "top500",
|
||||
and_(
|
||||
CollectedData.extra_data["status"].as_string() == "Existing",
|
||||
func.coalesce(CollectedData.extra_data["superseded_by"].as_string(), "") == "",
|
||||
CollectedData.extra_data["possible_duplicate"].as_boolean().is_not(True),
|
||||
),
|
||||
),
|
||||
CollectedData.extra_data["country"].as_string().in_(source_country_names(countries)),
|
||||
)
|
||||
return query
|
||||
|
||||
|
||||
@router.get("/facilities")
|
||||
async def get_facilities(
|
||||
countries: str = "CN,US",
|
||||
year: int | None = Query(None, ge=2000, le=2100),
|
||||
limit: int = Query(1000, ge=1, le=2000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
codes = selected_countries(countries)
|
||||
query = await sample_query(db, codes, year)
|
||||
country = CollectedData.extra_data["country"].as_string()
|
||||
source_participants = source_country_participants(codes)
|
||||
# Canonical source subdivisions retain their raw label on each record but
|
||||
# share one owner in the comparison, using the same policy as the globe.
|
||||
owner_cases = [
|
||||
(country == name, display_country(name))
|
||||
for name in source_country_names(codes)
|
||||
if display_country(name) != name
|
||||
]
|
||||
display = case(*owner_cases, else_=country) if owner_cases else country
|
||||
participant_cases = [(country == name, code) for name, code in source_participants.items()]
|
||||
participant = case(*participant_cases, else_=None)
|
||||
counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
display.label("country"),
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(display, participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
participant_counts = (
|
||||
(
|
||||
await db.execute(
|
||||
query.with_only_columns(
|
||||
participant.label("participant_code"),
|
||||
CollectedData.source,
|
||||
func.count().label("count"),
|
||||
).group_by(participant, CollectedData.source)
|
||||
)
|
||||
)
|
||||
.mappings()
|
||||
.all()
|
||||
)
|
||||
records = (
|
||||
(await db.execute(query.order_by(CollectedData.name, CollectedData.id).limit(limit)))
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
if year:
|
||||
await refresh_compute_center_location_cache(db, as_of=datetime(year + 1, 1, 1, tzinfo=UTC))
|
||||
else:
|
||||
await refresh_compute_center_location_cache(db)
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
for feature in payload["features"]:
|
||||
feature["properties"]["historical_snapshot"] = year is not None
|
||||
properties = feature["properties"]
|
||||
properties["source_country"] = properties.get("country")
|
||||
properties["country"] = display_country(properties.get("country"))
|
||||
total = sum(row["count"] for row in counts)
|
||||
payload.update(
|
||||
{
|
||||
"counts": [dict(row) for row in counts],
|
||||
"participant_counts": [dict(row) for row in participant_counts],
|
||||
"participants": [PARTICIPANTS[code] for code in codes],
|
||||
"total_count": total,
|
||||
"returned_count": len(records),
|
||||
"truncated": total > limit,
|
||||
"reference_year": year,
|
||||
"scope": "public_source_records",
|
||||
"history_basis": (
|
||||
"snapshots_collected_by_year_end" if year else "latest_successful_snapshot"
|
||||
),
|
||||
"coverage": "available" if total else "no_verified_snapshot",
|
||||
"cross_source_deduplicated": False,
|
||||
}
|
||||
)
|
||||
return payload
|
||||
@@ -52,7 +52,7 @@ PRODUCT_DEFINITIONS: dict[str, dict] = {
|
||||
"name": "算力",
|
||||
"sources": ["top500", "epoch_ai_gpu"],
|
||||
"primary_stat_key": "compute_center_count",
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count"],
|
||||
"stat_keys": ["compute_center_count", "supercomputer_count", "gpu_cluster_count", "data_center_count", "cloud_center_count", "edge_node_count"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@ from datetime import UTC, datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
|
||||
defaults = get_fallback_llm_provider_preset(
|
||||
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
|
||||
)
|
||||
else:
|
||||
defaults = DEFAULT_SETTINGS[category]
|
||||
merged = deepcopy(defaults)
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
|
||||
|
||||
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
|
||||
merged = merge_with_defaults(category, payload)
|
||||
record = await get_setting_record(db, category)
|
||||
if record is None:
|
||||
record = SystemSetting(category=category, payload=payload)
|
||||
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return merge_with_defaults(category, record.payload)
|
||||
return merged
|
||||
|
||||
|
||||
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
|
||||
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
|
||||
)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
|
||||
model_provider_apis = {
|
||||
**(provider_config.get("model_provider_apis") or {}),
|
||||
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
|
||||
}
|
||||
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": _resolve_service_token(normalized_ai)[0],
|
||||
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
"model_provider_apis": model_provider_apis,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = (
|
||||
str(llm_config.get("provider_api") or "").strip()
|
||||
or ProviderApi.OPENAI_COMPLETIONS.value
|
||||
)
|
||||
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
base_url = str(llm_config.get("base_url") or "").strip()
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
result = {
|
||||
"success": False, "connected": False, "mode": "lightweight_models",
|
||||
"provider": provider, "model": model,
|
||||
}
|
||||
if not base_url or not model:
|
||||
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
|
||||
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
return {**result, "message": "当前 provider 未配置 API Key。"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
catalog = await fetch_model_catalog(
|
||||
provider, base_url, provider_api, api_key,
|
||||
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用,但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {**result, "message": catalog_error_message(exc)}
|
||||
result.update({"url": catalog.url, "models_count": len(catalog.models)})
|
||||
if model.casefold() not in {item.casefold() for item in catalog.models}:
|
||||
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
|
||||
model_apis = llm_config.get("model_provider_apis") or {}
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
**result, "success": True, "connected": True,
|
||||
"provider_api": model_apis.get(model) or provider_api,
|
||||
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
|
||||
}
|
||||
|
||||
|
||||
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
|
||||
else "settings.ai_provider.connect.failed"),
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
"connected": lightweight_result["connected"],
|
||||
},
|
||||
)
|
||||
return {
|
||||
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
presets = list_fallback_llm_provider_presets()
|
||||
saved = await get_setting_payloads(
|
||||
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
|
||||
)
|
||||
return {
|
||||
"data": [
|
||||
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
|
||||
for preset in presets
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
payload: AIProviderIntegrationUpdate | None = None,
|
||||
):
|
||||
try:
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
get_fallback_llm_provider_preset(provider_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
if payload is not None:
|
||||
if _normalize_provider_id(payload.provider) != provider_id:
|
||||
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
|
||||
ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
else:
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
|
||||
)
|
||||
try:
|
||||
refreshed = await refresh_llm_provider_preset(
|
||||
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
|
||||
provider_api=provider_config.get("provider_api"),
|
||||
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
|
||||
)
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
logger.warning(
|
||||
"LLM provider catalog refresh failed",
|
||||
extra={
|
||||
"event": "settings.ai_provider.catalog.failed",
|
||||
"context": {
|
||||
"provider": provider_id,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
|
||||
) from exc
|
||||
|
||||
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
|
||||
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
|
||||
return {"data": refreshed}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
|
||||
@@ -7,6 +7,7 @@ from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=100),
|
||||
q: str = Query("", max_length=200),
|
||||
selected_id: str | None = Query(None, max_length=200),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
|
||||
@@ -17,11 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.compute_types import COMPUTE_SITE_TYPES, COMPUTE_SOURCES, compute_site_type, compute_record_filter, compute_site_type_expression
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
from app.core.satellite_presentation import satellite_constellation_group as _normalize_satellite_constellation_group
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
@@ -337,22 +339,6 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _normalize_satellite_constellation_group(
|
||||
raw_group: Any,
|
||||
name: Optional[str],
|
||||
) -> Optional[str]:
|
||||
normalized_group = str(raw_group or "").strip().lower()
|
||||
if normalized_group:
|
||||
return normalized_group
|
||||
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _get_satellite_footprint_policy(constellation_group: Optional[str]) -> str:
|
||||
if constellation_group == "starlink":
|
||||
@@ -505,16 +491,19 @@ async def _count_current_or_latest_task_data(
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
*, include_compute_types: bool = False,
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(compute_record_filter(CollectedData) if include_compute_types else CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
if include_compute_types:
|
||||
stmt = stmt.where(CollectedData.deleted_at.is_(None))
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
@@ -644,7 +633,9 @@ def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
if unit in {"flop/s", "flops"}:
|
||||
normalized_tflops = capacity_value / 1e12
|
||||
elif unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value
|
||||
@@ -672,15 +663,15 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
"""
|
||||
features: List[Dict[str, Any]] = []
|
||||
unresolved: List[Dict[str, Any]] = []
|
||||
withheld_count = 0
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
if metadata.get("compute_contract") == "v1" and metadata.get("anonymous"):
|
||||
withheld_count += 1
|
||||
continue
|
||||
result = resolve_compute_center_location_full(record, metadata)
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
site_type = compute_site_type(record)
|
||||
|
||||
if not result.is_resolved:
|
||||
diagnostic = result.diagnostic or ResolutionDiagnostic(
|
||||
@@ -706,7 +697,10 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
latitude = location.latitude
|
||||
longitude = location.longitude
|
||||
|
||||
if site_type == "supercomputer":
|
||||
if metadata.get("compute_contract") == "v1":
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "FLOP/s")
|
||||
elif site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
@@ -767,7 +761,7 @@ def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved}
|
||||
return {"type": "FeatureCollection", "features": features, "unresolved": unresolved, "withheld_count": withheld_count}
|
||||
|
||||
|
||||
VESSEL_TYPE_FILTERS = {
|
||||
@@ -1121,6 +1115,7 @@ async def build_vessel_snapshot_response(
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
|
||||
snapshot_started_at = datetime.now(UTC)
|
||||
features, diagnostics = await _load_raw_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
@@ -1137,6 +1132,7 @@ async def build_vessel_snapshot_response(
|
||||
"features": features,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
"generated_at": to_iso8601_utc(snapshot_started_at),
|
||||
"diagnostics": {
|
||||
**diagnostics,
|
||||
"filtered_count": len(features),
|
||||
@@ -1953,9 +1949,10 @@ async def _build_compute_centers_geojson(
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
include_compute_types=True,
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
[record for group in records_by_source.values() for record in group],
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
@@ -1970,6 +1967,7 @@ async def _build_compute_centers_geojson(
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
"by_type": dict.fromkeys(COMPUTE_SITE_TYPES, 0),
|
||||
"unresolved": 0,
|
||||
},
|
||||
}
|
||||
@@ -2012,6 +2010,7 @@ async def _build_compute_centers_geojson(
|
||||
"count": len(sanitized_features),
|
||||
"stats": {
|
||||
"total": len(sanitized_features),
|
||||
"by_type": {kind: sum(feature["properties"]["site_type"] == kind for feature in sanitized_features) for kind in COMPUTE_SITE_TYPES},
|
||||
"supercomputers": sum(
|
||||
1 for feature in sanitized_features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
@@ -2328,7 +2327,8 @@ async def _load_compute_center_record(db: AsyncSession, source_id: str) -> Colle
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source_id == source_id)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.deleted_at.is_(None))
|
||||
.order_by(CollectedData.is_current.desc(), CollectedData.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
@@ -2727,6 +2727,25 @@ async def get_visualization_geo_summary(db: AsyncSession = Depends(get_db), resp
|
||||
)
|
||||
|
||||
|
||||
async def _compute_facility_counts(db: AsyncSession) -> dict[str, int]:
|
||||
kind = compute_site_type_expression(CollectedData)
|
||||
rows = (await db.execute(select(CollectedData.source, kind, func.count(CollectedData.id))
|
||||
.where(compute_record_filter(CollectedData), CollectedData.is_current.is_(True),
|
||||
CollectedData.deleted_at.is_(None))
|
||||
.group_by(CollectedData.source, kind))).all()
|
||||
counts = dict.fromkeys(COMPUTE_SITE_TYPES, 0)
|
||||
present_sources = set()
|
||||
for source, site_type, count in rows:
|
||||
present_sources.add(source)
|
||||
counts[site_type] += int(count or 0)
|
||||
# Retain the existing last-task fallback for the two collected catalogs.
|
||||
for source in COMPUTE_SOURCES:
|
||||
if source not in present_sources:
|
||||
key = "supercomputer" if source == "top500" else "gpu_cluster"
|
||||
counts[key] += await _count_current_or_latest_task_data(db, source)
|
||||
return counts
|
||||
|
||||
|
||||
async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
cable_count = await _count_current_or_latest_task_data(db, "arcgis_cables")
|
||||
landing_point_count = await _count_current_or_latest_task_data(db, "arcgis_landing_points")
|
||||
@@ -2735,9 +2754,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
)
|
||||
supercomputer_count = await _count_current_or_latest_task_data(db, "top500")
|
||||
gpu_cluster_count = await _count_current_or_latest_task_data(db, "epoch_ai_gpu")
|
||||
compute_center_count = supercomputer_count + gpu_cluster_count
|
||||
compute_counts = await _compute_facility_counts(db)
|
||||
supercomputer_count = compute_counts["supercomputer"]
|
||||
gpu_cluster_count = compute_counts["gpu_cluster"]
|
||||
compute_center_count = sum(compute_counts.values())
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
|
||||
@@ -2792,6 +2812,9 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"aisstream_lag_seconds": aisstream_health.lag_seconds if aisstream_health else None,
|
||||
"supercomputer_count": supercomputer_count,
|
||||
"gpu_cluster_count": gpu_cluster_count,
|
||||
"data_center_count": compute_counts["data_center"],
|
||||
"cloud_center_count": compute_counts["cloud_center"],
|
||||
"edge_node_count": compute_counts["edge_node"],
|
||||
"bgp_event_count": active_incident_count or active_anomaly_count,
|
||||
"bgp_incident_count": active_incident_count,
|
||||
"bgp_anomaly_count": active_anomaly_count,
|
||||
|
||||
@@ -161,7 +161,7 @@ async def websocket_endpoint(
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
if "vessels" in channels and "bbox" in payload_data:
|
||||
if "vessels" in channels:
|
||||
try:
|
||||
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
|
||||
35
backend/app/core/compute_types.py
Normal file
@@ -0,0 +1,35 @@
|
||||
"""Compute facility classification shared by the map and image resolver."""
|
||||
|
||||
from sqlalchemy import case, or_
|
||||
|
||||
COMPUTE_SITE_TYPES = ("supercomputer", "gpu_cluster", "data_center", "cloud_center", "edge_node")
|
||||
COMPUTE_SOURCES = ("top500", "epoch_ai_gpu")
|
||||
|
||||
|
||||
def compute_site_type(record) -> str:
|
||||
metadata = record.extra_data or {}
|
||||
for value in (metadata.get("site_type"), record.data_type):
|
||||
if value in COMPUTE_SITE_TYPES:
|
||||
return value
|
||||
return {"top500": "supercomputer", "epoch_ai_gpu": "gpu_cluster"}.get(
|
||||
record.source, "data_center"
|
||||
)
|
||||
|
||||
|
||||
def compute_site_type_expression(record):
|
||||
declared = record.extra_data["site_type"].as_string()
|
||||
return case(
|
||||
(declared.in_(COMPUTE_SITE_TYPES), declared),
|
||||
(record.data_type.in_(COMPUTE_SITE_TYPES), record.data_type),
|
||||
(record.source == "top500", "supercomputer"),
|
||||
(record.source == "epoch_ai_gpu", "gpu_cluster"),
|
||||
else_="data_center",
|
||||
)
|
||||
|
||||
|
||||
def compute_record_filter(record):
|
||||
return or_(
|
||||
record.source.in_(COMPUTE_SOURCES),
|
||||
record.data_type.in_(COMPUTE_SITE_TYPES),
|
||||
record.extra_data["site_type"].as_string().in_(COMPUTE_SITE_TYPES),
|
||||
)
|
||||
57
backend/app/core/earth_geography_policy.py
Normal file
@@ -0,0 +1,57 @@
|
||||
"""Country display ownership for Earth comparisons and cards.
|
||||
|
||||
Keep source-country evidence separate; use this projection only for display
|
||||
grouping so regional source records cannot become competing national keys.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.core.countries import COUNTRY_ENTRIES, normalize_country
|
||||
|
||||
|
||||
_DISPLAY_ISO_A3 = {
|
||||
canonical: next(
|
||||
(alias for alias in aliases if len(alias) == 3 and alias.isascii() and alias.isupper()),
|
||||
None,
|
||||
)
|
||||
for canonical, aliases in COUNTRY_ENTRIES
|
||||
}
|
||||
_COUNTRY_BY_ISO_A3 = {code: country for country, code in _DISPLAY_ISO_A3.items() if code}
|
||||
_POLICY_PATH = Path(__file__).resolve().parents[3] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _policy_country_by_code() -> dict[str, str]:
|
||||
try:
|
||||
content = _POLICY_PATH.read_text(encoding="utf-8")
|
||||
except FileNotFoundError as error:
|
||||
raise FileNotFoundError(f"EARTH_DISPLAY_POLICY_MISSING: {_POLICY_PATH}") from error
|
||||
policy = json.loads(content)
|
||||
if policy.get("profile") != "china-pov-v1":
|
||||
raise ValueError("unsupported Earth display-policy profile")
|
||||
mappings = {}
|
||||
for rule in policy["rules"]:
|
||||
target = rule.get("targetIsoA3") or rule.get("parentIsoA3")
|
||||
if target not in _COUNTRY_BY_ISO_A3:
|
||||
continue
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
mappings[code] = _COUNTRY_BY_ISO_A3[target]
|
||||
return mappings
|
||||
|
||||
|
||||
def display_country(value: str | None) -> str | None:
|
||||
country = normalize_country(value)
|
||||
code = _DISPLAY_ISO_A3.get(country) if country else str(value or "").strip().upper()
|
||||
owner = _policy_country_by_code().get(code)
|
||||
if owner:
|
||||
return owner
|
||||
return country
|
||||
|
||||
|
||||
def display_iso_a3(value: str | None) -> str | None:
|
||||
country = display_country(value)
|
||||
return _DISPLAY_ISO_A3.get(country) if country else None
|
||||
@@ -138,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
|
||||
class ProviderApi(StrEnum):
|
||||
ANTHROPIC_MESSAGES = "anthropic-messages"
|
||||
OPENAI_COMPLETIONS = "openai-completions"
|
||||
OPENAI_RESPONSES = "openai-responses"
|
||||
OLLAMA_GENERATE = "ollama-generate"
|
||||
|
||||
|
||||
|
||||
13
backend/app/core/satellite_presentation.py
Normal file
@@ -0,0 +1,13 @@
|
||||
"""Shared source-group normalization; never infer hardware generations."""
|
||||
|
||||
|
||||
def satellite_constellation_group(raw_group: object, name: str | None) -> str | None:
|
||||
group = str(raw_group or "").strip().lower()
|
||||
if group:
|
||||
return group
|
||||
normalized_name = str(name or "").strip().upper()
|
||||
if normalized_name.startswith("STARLINK"):
|
||||
return "starlink"
|
||||
if normalized_name.startswith("IRIDIUM"):
|
||||
return "iridium-next"
|
||||
return None
|
||||
@@ -6,7 +6,7 @@ import redis
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from jose import JWTError, jwt
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.config import settings
|
||||
@@ -81,6 +81,21 @@ def decode_token(token: str) -> Optional[dict]:
|
||||
return None
|
||||
|
||||
|
||||
async def _load_active_user(user_id: str, db: AsyncSession) -> User:
|
||||
try:
|
||||
parsed_id = int(user_id)
|
||||
except (TypeError, ValueError):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token")
|
||||
result = await db.execute(select(User).where(User.id == parsed_id))
|
||||
user = result.scalar_one_or_none()
|
||||
if user is None or not user.is_active:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
return user
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
credentials: HTTPAuthorizationCredentials = Depends(oauth2_scheme),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
@@ -103,27 +118,7 @@ async def get_current_user(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
async def get_current_user_refresh(
|
||||
@@ -143,27 +138,7 @@ async def get_current_user_refresh(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid token",
|
||||
)
|
||||
result = await db.execute(
|
||||
text(
|
||||
"SELECT id, username, email, password_hash, role, is_active, gatekeeper_groups FROM users WHERE id = :id"
|
||||
),
|
||||
{"id": int(user_id)},
|
||||
)
|
||||
row = result.fetchone()
|
||||
if row is None or not row[5]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="User not found or inactive",
|
||||
)
|
||||
user = User()
|
||||
user.id = row[0]
|
||||
user.username = row[1]
|
||||
user.email = row[2]
|
||||
user.password_hash = row[3]
|
||||
user.role = row[4]
|
||||
user.is_active = row[5]
|
||||
user.gatekeeper_groups = row[6] or []
|
||||
return user
|
||||
return await _load_active_user(user_id, db)
|
||||
|
||||
|
||||
def blacklist_token(token: str) -> None:
|
||||
|
||||
@@ -4,11 +4,14 @@ import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Dict, Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
|
||||
logger = get_logger(__name__, service="websocket")
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
@@ -114,16 +117,22 @@ class DataBroadcaster:
|
||||
return
|
||||
pending = self._pending_vessel_updates
|
||||
self._pending_vessel_updates = {}
|
||||
vessels = []
|
||||
for item in pending.values():
|
||||
vessel = dict(item)
|
||||
source = vessel.pop("_source", None)
|
||||
action = vessel.pop("_action", "upsert")
|
||||
created = vessel.pop("_created", None)
|
||||
vessel["source"] = source
|
||||
vessel["action"] = action
|
||||
vessel["created"] = created
|
||||
vessels.append(vessel)
|
||||
try:
|
||||
vessels = await self._load_current_vessel_updates(list(pending))
|
||||
except Exception:
|
||||
# Preserve newer updates that arrived during the failed database read.
|
||||
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
|
||||
raise
|
||||
if not vessels:
|
||||
return
|
||||
vessels = [
|
||||
{
|
||||
**vessel,
|
||||
"action": vessel.get("action", "upsert"),
|
||||
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
|
||||
}
|
||||
for vessel in vessels
|
||||
]
|
||||
await manager.broadcast_vessels(
|
||||
{
|
||||
"action": "upsert",
|
||||
@@ -133,12 +142,31 @@ class DataBroadcaster:
|
||||
}
|
||||
)
|
||||
|
||||
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
|
||||
|
||||
mmsis = [int(key) for key in keys if key.isdigit()]
|
||||
vessels = []
|
||||
async with async_session_factory() as db:
|
||||
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
|
||||
vessels.extend(await get_current_vessels_by_mmsi(
|
||||
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
|
||||
))
|
||||
present = {int(vessel["mmsi"]) for vessel in vessels}
|
||||
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
|
||||
return vessels
|
||||
|
||||
async def broadcast_vessels_periodically(self):
|
||||
while self.running:
|
||||
try:
|
||||
await self.flush_vessel_updates()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to flush vessel updates",
|
||||
event="vessels.broadcast.failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(self._vessel_flush_interval)
|
||||
|
||||
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
|
||||
|
||||
@@ -63,6 +63,8 @@ class ConnectionManager:
|
||||
|
||||
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
|
||||
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
|
||||
if channel == "vessels":
|
||||
self.vessel_subscriptions.pop(websocket, None)
|
||||
subscribers = self.channel_subscriptions.get(channel)
|
||||
if subscribers is not None:
|
||||
subscribers.discard(websocket)
|
||||
@@ -88,7 +90,8 @@ class ConnectionManager:
|
||||
return subscription
|
||||
|
||||
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
bbox = config.get("bbox")
|
||||
global_scope = config.get("scope") == "global"
|
||||
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
|
||||
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
|
||||
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
|
||||
try:
|
||||
@@ -103,7 +106,7 @@ class ConnectionManager:
|
||||
raise ValueError("bbox longitude values must be between -180 and 180")
|
||||
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
|
||||
raise ValueError("bbox latitude values must be between -90 and 90")
|
||||
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
|
||||
|
||||
zoom = int(config.get("zoom") or 1)
|
||||
@@ -116,10 +119,11 @@ class ConnectionManager:
|
||||
if str(item).strip()
|
||||
}
|
||||
return {
|
||||
"scope": "global" if global_scope else "viewport",
|
||||
"bbox": (lon_min, lat_min, lon_max, lat_max),
|
||||
"zoom": zoom,
|
||||
"limit": limit,
|
||||
"type": vessel_types,
|
||||
"type": sorted(vessel_types),
|
||||
"last_sent_at": None,
|
||||
}
|
||||
|
||||
@@ -152,7 +156,9 @@ class ConnectionManager:
|
||||
vessel
|
||||
for vessel in vessels
|
||||
if self._vessel_matches_subscription(vessel, subscription)
|
||||
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
|
||||
]
|
||||
if subscription.get("scope") != "global":
|
||||
matched = matched[:subscription["limit"]]
|
||||
if not matched:
|
||||
continue
|
||||
subscription["last_sent_at"] = datetime.now(UTC)
|
||||
@@ -162,16 +168,24 @@ class ConnectionManager:
|
||||
"timestamp": subscription["last_sent_at"].isoformat(),
|
||||
"payload": {
|
||||
**data,
|
||||
"vessels": matched,
|
||||
"vessels": [],
|
||||
"subscription": {
|
||||
"bbox": list(subscription["bbox"]),
|
||||
"zoom": subscription["zoom"],
|
||||
"limit": subscription["limit"],
|
||||
"scope": subscription.get("scope", "viewport"),
|
||||
},
|
||||
},
|
||||
}
|
||||
try:
|
||||
await connection.send_json(message)
|
||||
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
|
||||
await connection.send_json({
|
||||
**message,
|
||||
"payload": {
|
||||
**message["payload"],
|
||||
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
|
||||
},
|
||||
})
|
||||
except Exception:
|
||||
self.unsubscribe_all(connection)
|
||||
|
||||
@@ -180,6 +194,8 @@ class ConnectionManager:
|
||||
vessel: dict[str, Any],
|
||||
subscription: dict[str, Any],
|
||||
) -> bool:
|
||||
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
|
||||
return True
|
||||
try:
|
||||
lon = float(vessel.get("lon"))
|
||||
lat = float(vessel.get("lat"))
|
||||
|
||||
128
backend/app/schemas/card_images.py
Normal file
@@ -0,0 +1,128 @@
|
||||
"""Typed, validated configuration for optional object-card images."""
|
||||
|
||||
from typing import Literal, Self
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
EntityType = Literal["satellite", "compute_center", "cable", "landing_point", "vessel", "bgp", "bgp_collector", "earth_interactable"]
|
||||
MatchScope = Literal["object", "model", "group", "category"]
|
||||
|
||||
|
||||
def safe_url(value: str, *, local: bool = False) -> str:
|
||||
if not value:
|
||||
return value
|
||||
if any(ord(char) < 32 for char in value) or "\\" in value:
|
||||
raise ValueError("Invalid URL")
|
||||
parts = urlsplit(value)
|
||||
if local and value.startswith(("/earth/assets/cards/", "/api/v1/earth/card-images/files/")):
|
||||
if (
|
||||
".." not in parts.path
|
||||
and "%" not in parts.path
|
||||
and not parts.netloc
|
||||
and not parts.query
|
||||
and not parts.fragment
|
||||
):
|
||||
return value
|
||||
if (
|
||||
parts.scheme in {"http", "https"}
|
||||
and parts.hostname
|
||||
and not parts.username
|
||||
and not parts.password
|
||||
):
|
||||
return value
|
||||
raise ValueError("Use an HTTP(S) URL without credentials or a managed image path")
|
||||
|
||||
|
||||
class CardImageAsset(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
title: str = Field(min_length=1, max_length=200)
|
||||
url: str = Field(min_length=1, max_length=2000)
|
||||
variants: dict[str, str] = Field(default_factory=dict, max_length=4)
|
||||
width: int = Field(default=0, ge=0, le=20000)
|
||||
height: int = Field(default=0, ge=0, le=20000)
|
||||
credit: str = Field(default="", max_length=500)
|
||||
source_url: str = Field(default="", max_length=2000)
|
||||
license: str = Field(default="", max_length=300)
|
||||
license_url: str = Field(default="", max_length=2000)
|
||||
kind: Literal["photo", "model_photo", "render", "animation_still"] = "photo"
|
||||
rights_status: Literal["available", "permission_pending"] = "available"
|
||||
enabled: bool = True
|
||||
focus_x: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
focus_y: float = Field(default=50, ge=0, le=100, allow_inf_nan=False)
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def validate_image_url(cls, value: str) -> str:
|
||||
return safe_url(value, local=True)
|
||||
|
||||
@field_validator("source_url", "license_url")
|
||||
@classmethod
|
||||
def validate_source_url(cls, value: str) -> str:
|
||||
return safe_url(value)
|
||||
|
||||
@field_validator("variants")
|
||||
@classmethod
|
||||
def validate_variants(cls, value: dict[str, str]) -> dict[str, str]:
|
||||
for width, url in value.items():
|
||||
if width not in {"640", "1280"}:
|
||||
raise ValueError("Unsupported image variant")
|
||||
safe_url(url, local=True)
|
||||
return value
|
||||
|
||||
|
||||
class CardImageRule(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
id: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z0-9_-]+$")
|
||||
entity_type: EntityType
|
||||
scope: MatchScope
|
||||
value: str = Field(min_length=1, max_length=300)
|
||||
label: str = Field(default="", max_length=500)
|
||||
asset_id: str | None = Field(default=None, max_length=100)
|
||||
suppress: bool = False
|
||||
enabled: bool = True
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_action(self) -> Self:
|
||||
if self.scope != "object":
|
||||
self.value = self.value.casefold()
|
||||
if self.suppress and (self.scope != "object" or self.asset_id):
|
||||
raise ValueError("Only an object rule may suppress its image, without an asset")
|
||||
if not self.suppress and not self.asset_id:
|
||||
raise ValueError("Choose an image or suppress this object")
|
||||
return self
|
||||
|
||||
def target(self) -> tuple[str, str, str]:
|
||||
return self.entity_type, self.scope, self.value
|
||||
|
||||
|
||||
class CardImageConfiguration(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
assets: list[CardImageAsset] = Field(default_factory=list, max_length=1000)
|
||||
rules: list[CardImageRule] = Field(default_factory=list, max_length=3000)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_relations(self) -> Self:
|
||||
assets = {asset.id for asset in self.assets}
|
||||
if len(assets) != len(self.assets) or len({rule.id for rule in self.rules}) != len(
|
||||
self.rules
|
||||
):
|
||||
raise ValueError("Image and rule IDs must be unique")
|
||||
targets = [rule.target() for rule in self.rules]
|
||||
if len(set(targets)) != len(targets):
|
||||
raise ValueError("Each target may have only one rule")
|
||||
if any(rule.asset_id and rule.asset_id not in assets for rule in self.rules):
|
||||
raise ValueError("Remove image associations before deleting the image")
|
||||
return self
|
||||
|
||||
|
||||
class CardImagePublish(BaseModel):
|
||||
revision: str = Field(max_length=200)
|
||||
config: CardImageConfiguration
|
||||
|
||||
|
||||
class CardImagePreview(BaseModel):
|
||||
entity_type: EntityType
|
||||
entity_id: str = Field(min_length=1, max_length=300)
|
||||
config: CardImageConfiguration
|
||||
94
backend/app/schemas/compute_configuration.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""Validated, source-backed observations for the compute comparison cards."""
|
||||
|
||||
from datetime import date
|
||||
from typing import Annotated, Literal, Self
|
||||
|
||||
from app.services.compute_catalog import COUNTRY_PARTICIPANTS
|
||||
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator, model_validator
|
||||
|
||||
Year = Annotated[int, Field(strict=True, ge=2000, le=2100)]
|
||||
Value = Annotated[float, Field(strict=True, allow_inf_nan=False, ge=0, le=1e12)]
|
||||
Identifier = Annotated[str, Field(min_length=1, max_length=80, pattern=r"^[a-zA-Z0-9_-]+$")]
|
||||
Text = Annotated[str, Field(min_length=1, max_length=500)]
|
||||
|
||||
|
||||
class ConfigurationModel(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", str_strip_whitespace=True)
|
||||
|
||||
|
||||
class ComputeSource(ConfigurationModel):
|
||||
id: Identifier
|
||||
title: Text
|
||||
url: HttpUrl
|
||||
publisher: Text
|
||||
report_year: Year
|
||||
published_at: date | None = None
|
||||
locator: Text
|
||||
|
||||
@field_validator("url")
|
||||
@classmethod
|
||||
def public_reference(cls, value: HttpUrl) -> HttpUrl:
|
||||
if value.username or value.password or len(str(value)) > 2000:
|
||||
raise ValueError("Use a report URL without credentials (maximum 2000 characters)")
|
||||
return value
|
||||
|
||||
|
||||
class ComputeObservation(ConfigurationModel):
|
||||
year: Year
|
||||
metric: Literal["global_share", "absolute"]
|
||||
source_id: Identifier
|
||||
values: dict[str, Value] = Field(min_length=1, max_length=250)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def comparable_values(self) -> Self:
|
||||
if any(code not in COUNTRY_PARTICIPANTS for code in self.values):
|
||||
raise ValueError(
|
||||
"Use countries from the canonical compute catalog, not regional totals"
|
||||
)
|
||||
if self.metric == "global_share" and sum(self.values.values()) > 100.000001:
|
||||
raise ValueError("Global shares for one year must total at most 100 percent")
|
||||
return self
|
||||
|
||||
|
||||
class InnovationObservation(ConfigurationModel):
|
||||
year: Year
|
||||
value: Annotated[float, Field(strict=True, allow_inf_nan=False, ge=-100, le=100)]
|
||||
source_id: Identifier
|
||||
|
||||
|
||||
class ComputeConfiguration(ConfigurationModel):
|
||||
reviewed_at: date
|
||||
sources: list[ComputeSource] = Field(max_length=200)
|
||||
observations: list[ComputeObservation] = Field(max_length=202)
|
||||
innovation: list[InnovationObservation] = Field(max_length=101)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def consistent_evidence(self) -> Self:
|
||||
ids = [source.id for source in self.sources]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Report source IDs must be unique")
|
||||
keys = [(row.metric, row.year) for row in self.observations]
|
||||
if len(keys) != len(set(keys)):
|
||||
raise ValueError("Each metric/year may have only one observation")
|
||||
years = [row.year for row in self.innovation]
|
||||
if len(years) != len(set(years)):
|
||||
raise ValueError("Innovation observation years must be unique")
|
||||
sources = {source.id: source for source in self.sources}
|
||||
for row in [*self.observations, *self.innovation]:
|
||||
source = sources.get(row.source_id)
|
||||
if source is None:
|
||||
raise ValueError(f"Missing report source: {row.source_id}")
|
||||
if row.year > source.report_year:
|
||||
raise ValueError("Observation year must not be after its report year")
|
||||
if self.reviewed_at > date.today():
|
||||
raise ValueError("Review date must not be in the future")
|
||||
return self
|
||||
|
||||
|
||||
class ComputeConfigurationWrite(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
config: ComputeConfiguration
|
||||
|
||||
|
||||
class ComputeConfigurationReset(ConfigurationModel):
|
||||
revision: str = Field(min_length=1, max_length=80)
|
||||
@@ -36,6 +36,9 @@ class UserInDB(UserBase):
|
||||
|
||||
|
||||
class UserResponse(UserBase):
|
||||
# Bootstrap accounts have reserved .local email addresses. Validate email
|
||||
# on writes, while allowing existing local identities to be serialized.
|
||||
email: str
|
||||
id: int
|
||||
role: str
|
||||
gatekeeper_groups: list[str] = Field(default_factory=list)
|
||||
|
||||
174
backend/app/services/card_image_configuration.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""Revision-checked image settings; new bundled assets never replace user edits."""
|
||||
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.card_images import CardImageConfiguration
|
||||
from app.services.card_image_defaults import (
|
||||
bundled_configuration,
|
||||
previous_bundled_assets,
|
||||
previous_bundled_rules,
|
||||
)
|
||||
|
||||
CATEGORY = "earth_card_images"
|
||||
|
||||
|
||||
def effective_envelope(payload: dict | None) -> dict:
|
||||
defaults, version = bundled_configuration()
|
||||
config = (
|
||||
CardImageConfiguration.model_validate(payload["config"])
|
||||
if payload
|
||||
else defaults.model_copy(deep=True)
|
||||
)
|
||||
if payload:
|
||||
previous = previous_bundled_assets()
|
||||
latest = {asset.id: asset for asset in defaults.assets}
|
||||
for index, asset in enumerate(config.assets):
|
||||
old = next((item for item in previous.get(asset.id, []) if item.url == asset.url), None)
|
||||
# Replace the old bundled source while preserving fields the user
|
||||
# changed (crop, title, enabled state, etc.). Custom URLs are untouched.
|
||||
if old is not None and asset.url == old.url and asset.id in latest:
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in asset.model_dump().items()
|
||||
if value != getattr(old, key)
|
||||
}
|
||||
config.assets[index] = latest[asset.id].model_copy(update=overrides, deep=True)
|
||||
known_assets = set(payload.get("known_assets", []))
|
||||
asset_ids = {asset.id for asset in config.assets}
|
||||
config.assets.extend(
|
||||
asset.model_copy(deep=True)
|
||||
for asset in defaults.assets
|
||||
if asset.id not in known_assets and asset.id not in asset_ids
|
||||
)
|
||||
prior_rules = previous_bundled_rules()
|
||||
latest_rules = {rule.id: rule for rule in defaults.rules}
|
||||
for index, rule in enumerate(config.rules):
|
||||
old_rule = prior_rules.get(rule.id)
|
||||
if (
|
||||
old_rule
|
||||
and rule.id in latest_rules
|
||||
and rule.target() == old_rule.target()
|
||||
and rule.asset_id == old_rule.asset_id
|
||||
):
|
||||
overrides = {
|
||||
key: value
|
||||
for key, value in rule.model_dump().items()
|
||||
if value != getattr(old_rule, key)
|
||||
}
|
||||
config.rules[index] = latest_rules[rule.id].model_copy(update=overrides, deep=True)
|
||||
known_rules = set(payload.get("known_rules", []))
|
||||
rule_ids = {rule.id for rule in config.rules}
|
||||
targets = {rule.target() for rule in config.rules}
|
||||
available = {asset.id for asset in config.assets}
|
||||
config.rules.extend(
|
||||
rule.model_copy(deep=True)
|
||||
for rule in defaults.rules
|
||||
if rule.id not in known_rules
|
||||
and rule.id not in rule_ids
|
||||
and rule.target() not in targets
|
||||
and rule.asset_id in available
|
||||
)
|
||||
return {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": f"{version}:{payload['revision'] if payload else 'default'}",
|
||||
"is_default": payload is None,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return effective_envelope(record.payload if record else None)
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession, revision: str, config: CardImageConfiguration, actor: User
|
||||
) -> dict:
|
||||
defaults, _ = bundled_configuration()
|
||||
pending = {asset.id for asset in defaults.assets if asset.rights_status == "permission_pending"}
|
||||
if any(
|
||||
asset.id in pending and asset.rights_status != "permission_pending"
|
||||
for asset in config.assets
|
||||
):
|
||||
raise HTTPException(422, "Bundled image permission is still unverified")
|
||||
initial = {
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"revision": "default",
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=initial)
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if effective_envelope(record.payload)["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
before = record.payload
|
||||
record.payload = {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="card_images.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": record.payload},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return effective_envelope(record.payload)
|
||||
|
||||
|
||||
def resolve_image(config: CardImageConfiguration, context: dict[str, str]) -> dict:
|
||||
assets = {asset.id: asset for asset in config.assets}
|
||||
for scope in ("object", "model", "group", "category"):
|
||||
value = context.get("entity_id" if scope == "object" else scope)
|
||||
values = [value] if value else []
|
||||
if scope == "category" and value != "*":
|
||||
values.append("*")
|
||||
for value in values:
|
||||
value = value if scope == "object" else value.casefold()
|
||||
rule = next(
|
||||
(
|
||||
rule
|
||||
for rule in config.rules
|
||||
if rule.enabled
|
||||
and rule.entity_type == context["entity_type"]
|
||||
and rule.scope == scope
|
||||
and rule.value == value
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not rule:
|
||||
continue
|
||||
match = {"scope": scope, "value": value, "rule_id": rule.id}
|
||||
if rule.suppress:
|
||||
return {"image": None, "match": match, "suppressed": True}
|
||||
asset = assets.get(rule.asset_id)
|
||||
if asset and asset.enabled:
|
||||
return {"image": asset.model_dump(mode="json"), "match": match, "suppressed": False}
|
||||
return {"image": None, "match": None, "suppressed": False}
|
||||
108
backend/app/services/card_image_defaults.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""Read immutable bundled assets without inserting machine-specific database rows."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
MANIFEST_PATH = REPO_ROOT / "frontend/public/earth/assets/cards/manifest.json"
|
||||
|
||||
|
||||
def _asset_from_manifest(item: dict) -> CardImageAsset:
|
||||
pending = item.get("redistribution_status") == "permission_not_verified"
|
||||
return CardImageAsset(
|
||||
id=item["id"],
|
||||
title=item["label"],
|
||||
url=item["url"],
|
||||
width=item["width"],
|
||||
height=item["height"],
|
||||
credit=item["credit"],
|
||||
source_url=item.get("source_page") or "",
|
||||
license=item["license"],
|
||||
license_url=item.get("license_url") or "",
|
||||
kind=item["kind"],
|
||||
rights_status="permission_pending" if pending else "available",
|
||||
enabled=item.get("enabled", not pending),
|
||||
focus_x=item.get("focus_x", 50),
|
||||
focus_y=item.get("focus_y", 50),
|
||||
variants=item.get("variants", {}),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_assets() -> dict[str, list[CardImageAsset]]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: [CardImageAsset.model_validate(item) for item in value]
|
||||
for key, value in manifest.get("previous_defaults", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def previous_bundled_rules() -> dict[str, CardImageRule]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_bytes())
|
||||
return {
|
||||
key: CardImageRule.model_validate(value)
|
||||
for key, value in manifest.get("previous_rules", {}).items()
|
||||
}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def bundled_configuration() -> tuple[CardImageConfiguration, str]:
|
||||
content = MANIFEST_PATH.read_bytes()
|
||||
manifest = json.loads(content)
|
||||
assets, rules = [], []
|
||||
for item in manifest["assets"]:
|
||||
asset = _asset_from_manifest(item)
|
||||
assets.append(asset)
|
||||
if not asset.enabled:
|
||||
continue
|
||||
hints = item.get("binding_hints", {})
|
||||
for index, target in enumerate(hints.get("defaults", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=target.get("id", f"builtin-{asset.id}-default-{index}"),
|
||||
entity_type=target["entity_type"],
|
||||
scope=target["scope"],
|
||||
value=target["value"],
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for index, entity_id in enumerate(hints.get("entity_ids", [])):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-entity-{index}",
|
||||
entity_type=item["entity_type"],
|
||||
scope="object",
|
||||
value=entity_id,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
for norad_id in hints.get("norad_ids", []):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-{norad_id}",
|
||||
entity_type="satellite",
|
||||
scope="object",
|
||||
value=str(norad_id),
|
||||
label=f"{asset.title} · NORAD {norad_id}",
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
if group := hints.get("constellation_group"):
|
||||
rules.append(
|
||||
CardImageRule(
|
||||
id=f"builtin-{asset.id}-group",
|
||||
entity_type="satellite",
|
||||
scope="group",
|
||||
value=group,
|
||||
label=asset.title,
|
||||
asset_id=asset.id,
|
||||
)
|
||||
)
|
||||
return CardImageConfiguration(assets=assets, rules=rules), hashlib.sha256(content).hexdigest()
|
||||
95
backend/app/services/card_image_files.py
Normal file
@@ -0,0 +1,95 @@
|
||||
"""Validated raster uploads and content-addressed, bounded card-size derivatives."""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import re
|
||||
import warnings
|
||||
from uuid import uuid4
|
||||
from pathlib import Path
|
||||
import os
|
||||
|
||||
from fastapi import HTTPException
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
from app.schemas.card_images import CardImageAsset
|
||||
from app.services.card_image_defaults import REPO_ROOT
|
||||
|
||||
ASSET_DIR = Path(
|
||||
os.environ.get("PLANET_CARD_IMAGE_DIR", str(REPO_ROOT / "data/earth-card-images"))
|
||||
).resolve()
|
||||
MAX_UPLOAD_BYTES = 12 * 1024 * 1024
|
||||
MAX_PIXELS = 32_000_000
|
||||
FILE_PATTERN = re.compile(r"^[a-f0-9]{64}-(?:original\.(?:jpg|png|webp)|(?:640|1280)\.webp)$")
|
||||
FORMATS = {"JPEG": "jpg", "PNG": "png", "WEBP": "webp"}
|
||||
|
||||
|
||||
def uploaded_path(filename: str) -> Path:
|
||||
if not FILE_PATTERN.fullmatch(filename):
|
||||
raise HTTPException(404, "Image not found")
|
||||
return ASSET_DIR / filename
|
||||
|
||||
|
||||
def store_image(content: bytes, title: str) -> CardImageAsset:
|
||||
if len(content) > MAX_UPLOAD_BYTES:
|
||||
raise HTTPException(413, "Image must be 12 MiB or smaller")
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
||||
image = Image.open(io.BytesIO(content))
|
||||
extension = FORMATS.get(image.format)
|
||||
if (
|
||||
not extension
|
||||
or image.width * image.height > MAX_PIXELS
|
||||
or getattr(image, "is_animated", False)
|
||||
):
|
||||
raise ValueError("Unsupported or oversized raster")
|
||||
image.load()
|
||||
image = ImageOps.exif_transpose(image).convert(
|
||||
"RGBA" if "A" in image.getbands() or "transparency" in image.info else "RGB"
|
||||
)
|
||||
except (
|
||||
UnidentifiedImageError,
|
||||
OSError,
|
||||
ValueError,
|
||||
Image.DecompressionBombError,
|
||||
Image.DecompressionBombWarning,
|
||||
) as error:
|
||||
raise HTTPException(
|
||||
422, "Use a valid, still JPEG, PNG or WebP image below 32 megapixels"
|
||||
) from error
|
||||
digest = hashlib.sha256(content).hexdigest()
|
||||
ASSET_DIR.mkdir(parents=True, exist_ok=True)
|
||||
original = f"{digest}-original.{extension}"
|
||||
original_path = uploaded_path(original)
|
||||
if not original_path.exists():
|
||||
temporary = original_path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(original_path)
|
||||
variants = {}
|
||||
for size, content in render_variants(image).items():
|
||||
filename = f"{digest}-{size}.webp"
|
||||
path = uploaded_path(filename)
|
||||
if not path.exists():
|
||||
temporary = path.with_suffix(f".{uuid4().hex}.tmp")
|
||||
temporary.write_bytes(content)
|
||||
temporary.replace(path)
|
||||
variants[size] = f"/api/v1/earth/card-images/files/{filename}"
|
||||
return CardImageAsset(
|
||||
id=f"upload-{digest[:24]}",
|
||||
title=title or "Image",
|
||||
url=f"/api/v1/earth/card-images/files/{original}",
|
||||
variants=variants,
|
||||
width=image.width,
|
||||
height=image.height,
|
||||
)
|
||||
|
||||
|
||||
def render_variants(image: Image.Image) -> dict[str, bytes]:
|
||||
result = {}
|
||||
for size in (640, 1280):
|
||||
resized = image.copy()
|
||||
resized.thumbnail((size, size), Image.Resampling.LANCZOS)
|
||||
buffer = io.BytesIO()
|
||||
resized.save(buffer, format="WEBP", quality=93, method=4)
|
||||
result[str(size)] = buffer.getvalue()
|
||||
return result
|
||||
180
backend/app/services/card_image_targets.py
Normal file
@@ -0,0 +1,180 @@
|
||||
"""Bounded catalog queries and type defaults for every Earth object card."""
|
||||
|
||||
from sqlalchemy import String, case, cast, func, literal, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import (
|
||||
COMPUTE_SITE_TYPES,
|
||||
compute_site_type_expression,
|
||||
compute_record_filter,
|
||||
)
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_collector_location import BGPCollectorLocation
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.earth_interactable import EarthInteractable
|
||||
from app.models.vessel import VesselCurrentState
|
||||
from app.schemas.card_images import EntityType, MatchScope
|
||||
|
||||
SOURCES = {
|
||||
"satellite": ("celestrak_tle", "spacetrack_tle"),
|
||||
"compute_center": ("top500", "epoch_ai_gpu"),
|
||||
"cable": ("arcgis_cables",),
|
||||
"landing_point": ("arcgis_landing_points",),
|
||||
}
|
||||
# Named constellation families, not inferred hardware models or generations.
|
||||
SATELLITE_GROUP_PATTERNS = (
|
||||
("STARLINK%", "starlink"),
|
||||
("IRIDIUM%", "iridium-next"),
|
||||
("BEIDOU-3%", "beidou-3"),
|
||||
("BEIDOU%", "beidou"),
|
||||
("QIANFAN%", "qianfan"),
|
||||
("ONEWEB%", "oneweb"),
|
||||
("KUIPER%", "kuiper"),
|
||||
)
|
||||
|
||||
|
||||
def collected_query(entity_type: EntityType):
|
||||
record, metadata = CollectedData, CollectedData.extra_data
|
||||
model = func.coalesce(
|
||||
metadata["satellite_model" if entity_type == "satellite" else "model"].as_string(), ""
|
||||
)
|
||||
category = (
|
||||
compute_site_type_expression(record)
|
||||
if entity_type == "compute_center"
|
||||
else literal(entity_type)
|
||||
)
|
||||
if entity_type == "satellite":
|
||||
group = func.coalesce(
|
||||
func.nullif(func.lower(metadata["constellation_group"].as_string()), ""),
|
||||
case(
|
||||
*[
|
||||
(func.upper(record.name).like(pattern), value)
|
||||
for pattern, value in SATELLITE_GROUP_PATTERNS
|
||||
],
|
||||
else_="",
|
||||
),
|
||||
)
|
||||
entity_id = metadata["norad_cat_id"].as_string()
|
||||
else:
|
||||
group = func.coalesce(
|
||||
metadata["operator"].as_string(), metadata["organization"].as_string(), ""
|
||||
)
|
||||
entity_id = record.source + ":" + record.source_id
|
||||
return select(
|
||||
record.name.label("name"),
|
||||
cast(entity_id, String).label("entity_id"),
|
||||
model.label("model"),
|
||||
group.label("group"),
|
||||
category.label("category"),
|
||||
).where(
|
||||
(
|
||||
compute_record_filter(record)
|
||||
if entity_type == "compute_center"
|
||||
else record.source.in_(SOURCES[entity_type])
|
||||
),
|
||||
record.is_current.is_(True),
|
||||
record.deleted_at.is_(None),
|
||||
metadata["anonymous"].as_boolean().is_not(True),
|
||||
)
|
||||
|
||||
|
||||
def target_query(entity_type: EntityType):
|
||||
if entity_type in SOURCES:
|
||||
return collected_query(entity_type)
|
||||
if entity_type == "vessel":
|
||||
model = VesselCurrentState
|
||||
identity, name, group, category = model.mmsi, model.name, model.flag, model.vessel_type_name
|
||||
conditions = []
|
||||
elif entity_type == "bgp":
|
||||
model = BGPIncident
|
||||
identity, name, group, category = (
|
||||
model.incident_key,
|
||||
model.title,
|
||||
model.source,
|
||||
model.incident_type,
|
||||
)
|
||||
conditions = []
|
||||
elif entity_type == "bgp_collector":
|
||||
model = BGPCollectorLocation
|
||||
identity, name, group, category = (
|
||||
model.collector_id,
|
||||
model.collector_id,
|
||||
model.operator,
|
||||
literal("bgp_collector"),
|
||||
)
|
||||
conditions = []
|
||||
else:
|
||||
model = EarthInteractable
|
||||
identity, name, group, category = model.id, model.label, model.layer, model.kind
|
||||
conditions = [model.is_deleted.is_(False)]
|
||||
return select(
|
||||
name.label("name"),
|
||||
cast(identity, String).label("entity_id"),
|
||||
literal("").label("model"),
|
||||
func.coalesce(group, "").label("group"),
|
||||
func.coalesce(category, entity_type).label("category"),
|
||||
).where(*conditions)
|
||||
|
||||
|
||||
def category_context(entity_type: EntityType) -> dict[str, str]:
|
||||
return {
|
||||
"entity_type": entity_type,
|
||||
"entity_id": "",
|
||||
"name": "",
|
||||
"model": "",
|
||||
"group": "",
|
||||
"category": entity_type,
|
||||
}
|
||||
|
||||
|
||||
async def get_target(db: AsyncSession, entity_type: EntityType, entity_id: str) -> dict[str, str]:
|
||||
if entity_type == "compute_center" and entity_id.startswith("__category__:"):
|
||||
category = entity_id.partition(":")[2]
|
||||
if category in COMPUTE_SITE_TYPES:
|
||||
return {**category_context(entity_type), "category": category}
|
||||
if entity_id == "__category__":
|
||||
return category_context(entity_type)
|
||||
if entity_type in SOURCES and entity_type != "satellite":
|
||||
source, separator, source_id = entity_id.partition(":")
|
||||
if not separator or not source or not source_id:
|
||||
return category_context(entity_type)
|
||||
# Match the indexed identity columns before projecting JSON metadata.
|
||||
statement = collected_query(entity_type).where(
|
||||
CollectedData.source == source, CollectedData.source_id == source_id
|
||||
)
|
||||
else:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
statement = select(catalog).where(catalog.c.entity_id == entity_id)
|
||||
row = (await db.execute(statement.limit(1))).mappings().first()
|
||||
if not row:
|
||||
# Old map snapshots can outlive a catalog refresh. They may use the type
|
||||
# illustration, but must not claim an unverified exact-object identity.
|
||||
return category_context(entity_type)
|
||||
context = {key: str(value or "") for key, value in row.items()}
|
||||
return {**context, "entity_type": entity_type}
|
||||
|
||||
|
||||
async def search_targets(
|
||||
db: AsyncSession, entity_type: EntityType, scope: MatchScope, query: str, limit: int
|
||||
) -> list[dict[str, str]]:
|
||||
catalog = target_query(entity_type).subquery()
|
||||
value = catalog.c["entity_id" if scope == "object" else scope]
|
||||
label = catalog.c.name if scope == "object" else value
|
||||
statement = select(value.label("value"), label.label("label")).where(
|
||||
value.is_not(None), value != ""
|
||||
)
|
||||
if query:
|
||||
escaped = query.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
|
||||
statement = statement.where(
|
||||
or_(value.ilike(f"%{escaped}%", escape="\\"), label.ilike(f"%{escaped}%", escape="\\"))
|
||||
)
|
||||
rows = (await db.execute(statement.distinct().order_by(label, value).limit(limit))).mappings()
|
||||
items = [{"value": row["value"], "label": row["label"] or row["value"]} for row in rows]
|
||||
if scope == "category" and not query:
|
||||
known = {item["value"] for item in items}
|
||||
if entity_type == "compute_center":
|
||||
items.extend(
|
||||
{"value": kind, "label": kind} for kind in COMPUTE_SITE_TYPES if kind not in known
|
||||
)
|
||||
items.insert(0, {"value": "*", "label": "*"})
|
||||
return items
|
||||
@@ -31,6 +31,7 @@ class BaseCollector(ABC):
|
||||
frequency_hours: int = 4
|
||||
data_type: str = "generic"
|
||||
fail_on_empty: bool = False
|
||||
preserve_snapshot_on_failure: bool = False
|
||||
|
||||
def __init__(self):
|
||||
self._current_task = None
|
||||
@@ -361,6 +362,8 @@ class BaseCollector(ABC):
|
||||
context={"task_id": task_id, "raw_count": len(raw_data)},
|
||||
)
|
||||
data = self.transform(raw_data)
|
||||
if self.fail_on_empty and not data:
|
||||
raise RuntimeError(f"Collector {self.name} produced no valid records")
|
||||
await self._log_collection_event(
|
||||
"collector.phase.transforming.success",
|
||||
"Collector transform phase completed",
|
||||
@@ -456,6 +459,10 @@ class BaseCollector(ABC):
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
if snapshot_id is not None and self.preserve_snapshot_on_failure:
|
||||
await self._rollback_incomplete_run(
|
||||
db, task_id=task_id, snapshot_id=snapshot_id, reason=str(e)
|
||||
)
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = "failed"
|
||||
task.phase_message = str(e)
|
||||
@@ -465,6 +472,8 @@ class BaseCollector(ABC):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.status = SnapshotStatus.FAILED.value
|
||||
if self.preserve_snapshot_on_failure:
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
snapshot.summary = {"error": str(e)}
|
||||
await db.commit()
|
||||
|
||||
@@ -18,7 +18,6 @@ from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.downloads import DownloadHTTPStatusError, ResumableFileDownloader
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
ACTIVE_GROUP = "active"
|
||||
FALLBACK_GROUPS = (
|
||||
@@ -39,6 +38,7 @@ FALLBACK_GROUPS = (
|
||||
)
|
||||
FETCH_RETRY_ATTEMPTS = 3
|
||||
FETCH_RETRY_BASE_DELAY_SECONDS = 0.8
|
||||
CELESTRAK_CACHE_TTL_SECONDS = 2 * 60 * 60
|
||||
CELESTRAK_NOT_UPDATED_MARKER = "GP data has not updated since your last successful"
|
||||
|
||||
|
||||
@@ -51,6 +51,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
_downloader = ResumableFileDownloader(
|
||||
cache_namespace="celestrak",
|
||||
default_accept="application/json",
|
||||
preflight_head=False,
|
||||
cache_ttl_seconds=CELESTRAK_CACHE_TTL_SECONDS,
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -195,7 +197,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
if not is_final_attempt:
|
||||
await asyncio.sleep(FETCH_RETRY_BASE_DELAY_SECONDS * attempt)
|
||||
|
||||
raise RuntimeError(f"CelesTrak active satellite download failed after retries: {last_error}")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active satellite download failed after retries: {last_error}"
|
||||
)
|
||||
|
||||
async def _fetch_fallback_groups(
|
||||
self,
|
||||
@@ -226,25 +230,25 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
try:
|
||||
for group in FALLBACK_GROUPS:
|
||||
group_url = self._group_url(group)
|
||||
cached_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if cached_path is not None:
|
||||
body_path = cached_path
|
||||
else:
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(f"CelesTrak fallback group '{group}' download failed: {exc}") from exc
|
||||
try:
|
||||
body_path = await self._downloader.download_file(
|
||||
client,
|
||||
group_url,
|
||||
extension=".json",
|
||||
accept="application/json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
except DownloadHTTPStatusError as exc:
|
||||
if not self._is_not_updated_response(exc):
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' download failed: {exc}"
|
||||
) from exc
|
||||
body_path = self._downloader.get_cached_file(
|
||||
group_url,
|
||||
".json",
|
||||
validate_existing=self._validate_json_file,
|
||||
)
|
||||
if body_path is None:
|
||||
raise RuntimeError(
|
||||
f"CelesTrak fallback group '{group}' has not updated and no local cached copy is available"
|
||||
) from exc
|
||||
@@ -348,13 +352,13 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
commit=True,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _validate_json_file(path: Path) -> bool:
|
||||
@classmethod
|
||||
def _validate_json_file(cls, path: Path) -> bool:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError):
|
||||
cls._load_active_payload(path)
|
||||
except RuntimeError:
|
||||
return False
|
||||
return isinstance(data, list)
|
||||
return True
|
||||
|
||||
async def _log_parse_failure(self, exc: Exception) -> None:
|
||||
await emit_business_log(
|
||||
@@ -375,7 +379,8 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
),
|
||||
)
|
||||
|
||||
def _load_active_payload(self, path: Path) -> List[Dict[str, Any]]:
|
||||
@staticmethod
|
||||
def _load_active_payload(path: Path) -> List[Dict[str, Any]]:
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
@@ -391,7 +396,9 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
else:
|
||||
invalid_count += 1
|
||||
if invalid_count:
|
||||
raise RuntimeError(f"CelesTrak active payload contains {invalid_count} invalid record(s)")
|
||||
raise RuntimeError(
|
||||
f"CelesTrak active payload contains {invalid_count} invalid record(s)"
|
||||
)
|
||||
if not records:
|
||||
raise RuntimeError("CelesTrak active payload contains no satellite records")
|
||||
return records
|
||||
|
||||
82
backend/app/services/collectors/compute_base.py
Normal file
@@ -0,0 +1,82 @@
|
||||
"""Quality gates shared by the two compute sources."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class ComputeCollector(BaseCollector):
|
||||
fail_on_empty = True
|
||||
preserve_snapshot_on_failure = True
|
||||
|
||||
async def _create_snapshot(
|
||||
self, db: AsyncSession, task_id: int, data: list[dict], started_at: datetime
|
||||
) -> int:
|
||||
previous_count = await db.scalar(
|
||||
select(DataSnapshot.record_count)
|
||||
.where(
|
||||
DataSnapshot.source == self.name,
|
||||
DataSnapshot.is_current.is_(True),
|
||||
DataSnapshot.status == "success",
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
if previous_count and len(data) < previous_count * 0.6:
|
||||
raise ValueError("Compute source lost over 40% of rows; previous snapshot retained")
|
||||
return await super()._create_snapshot(db, task_id, data, started_at)
|
||||
|
||||
async def _save_data(self, db: AsyncSession, data: list[dict], **kwargs) -> int:
|
||||
if self.name == "top500":
|
||||
await self._copy_verified_locations(db, data)
|
||||
return await super()._save_data(db, data, **kwargs)
|
||||
|
||||
async def _copy_verified_locations(self, db: AsyncSession, data: list[dict]) -> None:
|
||||
"""Keep historical location keys; copy only unambiguous exact identities.
|
||||
|
||||
Never migrate by rank: the machine occupying rank 1 changes over time.
|
||||
"""
|
||||
names = [item["name"] for item in data]
|
||||
locations = (
|
||||
(
|
||||
await db.execute(
|
||||
select(ComputeCenterLocationRecord).where(
|
||||
ComputeCenterLocationRecord.source == self.name,
|
||||
ComputeCenterLocationRecord.name.in_(names),
|
||||
)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
)
|
||||
existing = {location.source_id for location in locations}
|
||||
for item in data:
|
||||
if item["source_id"] in existing:
|
||||
continue
|
||||
country = normalize_country(item["country"])
|
||||
if not country:
|
||||
continue
|
||||
matches = [
|
||||
location
|
||||
for location in locations
|
||||
if location.name == item["name"]
|
||||
and normalize_country(location.country) == country
|
||||
and (not location.site or location.site == item.get("metadata", {}).get("site"))
|
||||
and not location.source_id.startswith("top500_system_")
|
||||
]
|
||||
if len(matches) != 1:
|
||||
continue
|
||||
old = matches[0]
|
||||
values = {
|
||||
column.name: getattr(old, column.name)
|
||||
for column in old.__table__.columns
|
||||
if column.name not in {"id", "source_id", "created_at", "updated_at"}
|
||||
}
|
||||
db.add(ComputeCenterLocationRecord(**values, source_id=item["source_id"]))
|
||||
existing.add(item["source_id"])
|
||||
@@ -12,7 +12,6 @@ from typing import Any, Awaitable, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
ProgressCallback = Callable[[int, int | None], Awaitable[None]]
|
||||
ValidateCallback = Callable[[Path], bool]
|
||||
|
||||
@@ -52,10 +51,14 @@ class ResumableFileDownloader:
|
||||
user_agent: str = "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
default_accept: str = "*/*",
|
||||
cache_root: Path | None = None,
|
||||
preflight_head: bool = True,
|
||||
cache_ttl_seconds: float = 0,
|
||||
) -> None:
|
||||
self._cache_dir = (cache_root or default_download_cache_root()) / cache_namespace
|
||||
self._user_agent = user_agent
|
||||
self._default_accept = default_accept
|
||||
self._preflight_head = preflight_head
|
||||
self._cache_ttl_seconds = cache_ttl_seconds
|
||||
|
||||
@staticmethod
|
||||
def _cache_key(url: str) -> str:
|
||||
@@ -93,8 +96,9 @@ class ResumableFileDownloader:
|
||||
if not meta_path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
data = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except json.JSONDecodeError, OSError:
|
||||
return {}
|
||||
|
||||
@staticmethod
|
||||
@@ -111,20 +115,25 @@ class ResumableFileDownloader:
|
||||
return last_modified == str(meta.get("last_modified") or "").strip()
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _response_info(response: httpx.Response) -> dict[str, Any]:
|
||||
# httpx yields decoded bytes; encoded Content-Length cannot validate them.
|
||||
raw_length = response.headers.get("content-length")
|
||||
encoded = response.headers.get("content-encoding", "identity") != "identity"
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": int(raw_length) if raw_length and not encoded else None,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
|
||||
async def fetch_remote_info(self, client: httpx.AsyncClient, url: str) -> dict[str, Any]:
|
||||
try:
|
||||
response = await client.head(url)
|
||||
if response.status_code >= 400:
|
||||
return {}
|
||||
content_length_raw = response.headers.get("content-length")
|
||||
content_length = int(content_length_raw) if content_length_raw else None
|
||||
return {
|
||||
"etag": response.headers.get("etag"),
|
||||
"last_modified": response.headers.get("last-modified"),
|
||||
"content_length": content_length,
|
||||
"accept_ranges": (response.headers.get("accept-ranges") or "").lower(),
|
||||
}
|
||||
except (httpx.HTTPError, ValueError):
|
||||
return self._response_info(response)
|
||||
except httpx.HTTPError, ValueError:
|
||||
return {}
|
||||
|
||||
async def download_file(
|
||||
@@ -139,14 +148,24 @@ class ResumableFileDownloader:
|
||||
) -> Path:
|
||||
self._cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
final_path, part_path, meta_path = self._cache_paths(url, extension)
|
||||
cached_path = self.get_cached_file(url, extension, validate_existing=validate_existing)
|
||||
if cached_path and 0 <= time.time() - cached_path.stat().st_mtime < self._cache_ttl_seconds:
|
||||
size = cached_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return cached_path
|
||||
meta = self._load_meta(meta_path)
|
||||
remote = await self.fetch_remote_info(client, url)
|
||||
remote = await self.fetch_remote_info(client, url) if self._preflight_head else {}
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
if final_path.exists():
|
||||
local_size = final_path.stat().st_size
|
||||
size_match = expected_size is None or local_size == expected_size
|
||||
if self._validators_match(meta, remote) and size_match:
|
||||
if (
|
||||
(remote.get("etag") or remote.get("last_modified"))
|
||||
and self._validators_match(meta, remote)
|
||||
and size_match
|
||||
):
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
final_path.unlink(missing_ok=True)
|
||||
else:
|
||||
@@ -181,9 +200,13 @@ class ResumableFileDownloader:
|
||||
headers["If-Range"] = str(remote.get("last_modified"))
|
||||
|
||||
async with client.stream("GET", url, headers=headers) as response:
|
||||
if response.status_code == 304 and final_path.exists():
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
if response.status_code == 304:
|
||||
if cached_path is None:
|
||||
raise RuntimeError(f"Server returned 304 without a valid local cache for {url}")
|
||||
final_path.touch()
|
||||
size = final_path.stat().st_size
|
||||
if progress_callback:
|
||||
await progress_callback(size, size)
|
||||
return final_path
|
||||
if response.status_code >= 400:
|
||||
body = (await response.aread()).decode("utf-8", errors="replace")
|
||||
@@ -194,12 +217,16 @@ class ResumableFileDownloader:
|
||||
else:
|
||||
mode = "wb"
|
||||
resume_from = 0
|
||||
remote = self._response_info(response)
|
||||
expected_size = remote.get("content_length")
|
||||
|
||||
downloaded = resume_from
|
||||
last_emit_bytes = 0
|
||||
last_emit_time = time.monotonic()
|
||||
min_emit_bytes = (
|
||||
max(expected_size // 150, 512 * 1024) if expected_size and expected_size > 0 else 1024 * 1024
|
||||
max(expected_size // 150, 512 * 1024)
|
||||
if expected_size and expected_size > 0
|
||||
else 1024 * 1024
|
||||
)
|
||||
|
||||
with part_path.open(mode) as f:
|
||||
@@ -230,6 +257,10 @@ class ResumableFileDownloader:
|
||||
f"Resumable download incomplete for {url}: expected={expected_size}, got={final_size}"
|
||||
)
|
||||
|
||||
# Never replace the last good payload with an HTML error or invalid JSON.
|
||||
if validate_existing and not validate_existing(part_path):
|
||||
part_path.unlink(missing_ok=True)
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
part_path.replace(final_path)
|
||||
self._save_meta(
|
||||
meta_path,
|
||||
@@ -242,9 +273,6 @@ class ResumableFileDownloader:
|
||||
},
|
||||
)
|
||||
|
||||
if validate_existing and not validate_existing(final_path):
|
||||
raise RuntimeError(f"Downloaded file validation failed for {url}")
|
||||
|
||||
if progress_callback and expected_size and expected_size > 0:
|
||||
await progress_callback(expected_size, expected_size)
|
||||
|
||||
|
||||
@@ -1,119 +1,135 @@
|
||||
"""Epoch AI GPU Clusters Collector
|
||||
|
||||
Collects data from Epoch AI GPU clusters tracking.
|
||||
https://epoch.ai/data/gpu-clusters
|
||||
"""
|
||||
"""Epoch's official CSV; preserve precision, source dates and anonymous records."""
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.countries import normalize_country
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
EPOCH_CSV_URL = "https://epoch.ai/data/gpu_clusters.csv"
|
||||
|
||||
|
||||
|
||||
class EpochAIGPUCollector(BaseCollector):
|
||||
class EpochAIGPUCollector(ComputeCollector):
|
||||
name = "epoch_ai_gpu"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
frequency_hours = 6
|
||||
data_type = "gpu_cluster"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
async def fetch(self) -> list[dict]:
|
||||
configured = self._resolved_url or EPOCH_CSV_URL
|
||||
parsed = urlparse(configured)
|
||||
url = (
|
||||
EPOCH_CSV_URL
|
||||
if parsed.hostname == "epoch.ai" and not parsed.path.endswith(".csv")
|
||||
else configured
|
||||
)
|
||||
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.text)
|
||||
return self.parse_response(response.text)
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
"""Parse Epoch AI webpage to extract GPU cluster data"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
def parse_response(self, text: str) -> list[dict]:
|
||||
reader = csv.DictReader(io.StringIO(text.lstrip("\ufeff")))
|
||||
required = {"Name", "Country", "Status", "First Operational Date", "16-bit OP/s (log)"}
|
||||
if not required.issubset(reader.fieldnames or []):
|
||||
raise ValueError(
|
||||
"Epoch CSV schema changed or HTML returned; previous snapshot retained"
|
||||
)
|
||||
entries = []
|
||||
seen = set()
|
||||
anonymous_counts = {}
|
||||
for row in reader:
|
||||
name = (row.get("Name") or "").strip()
|
||||
country = normalize_country(row.get("Country"))
|
||||
if not name:
|
||||
raise ValueError("Epoch row has missing identity")
|
||||
source_id = f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', name.lower())}"
|
||||
anonymous = "anonym" in name.lower()
|
||||
if anonymous:
|
||||
# Public anonymous rows have no facility identity. Preserve
|
||||
# multiplicity within snapshots, without guessing names.
|
||||
digest = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()[:24]
|
||||
anonymous_counts[digest] = anonymous_counts.get(digest, 0) + 1
|
||||
source_id = f"epoch_anonymous_{digest}_{anonymous_counts[digest]}"
|
||||
if source_id in seen:
|
||||
raise ValueError("Epoch identity collision; previous snapshot retained")
|
||||
seen.add(source_id)
|
||||
raw_date = row.get("First Operational Date", "")
|
||||
reference_date = raw_date if re.fullmatch(r"\d{4}-\d{2}-\d{2}", raw_date) else None
|
||||
value = None
|
||||
raw_value = row.get("16-bit OP/s (log)", "")
|
||||
if raw_value:
|
||||
exponent = float(raw_value)
|
||||
if not math.isfinite(exponent) or not 0 <= exponent <= 30:
|
||||
raise ValueError("Epoch performance exponent out of bounds")
|
||||
value = 10**exponent
|
||||
anonymous = anonymous or (country == "中国" and not row.get("Location"))
|
||||
location = row.get("Location")
|
||||
location_assessment = assess_epoch_location(
|
||||
location,
|
||||
country=country,
|
||||
raw_country=row.get("Country"),
|
||||
)
|
||||
entries.append(
|
||||
{
|
||||
"source_id": source_id,
|
||||
"name": name,
|
||||
"country": country,
|
||||
"value": value,
|
||||
"unit": "FLOP/s",
|
||||
"reference_date": reference_date,
|
||||
"latitude": None if anonymous else self.coordinate(row.get("latitude"), 90),
|
||||
"longitude": None if anonymous else self.coordinate(row.get("longitude"), 180),
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP16",
|
||||
"measurement": "theoretical",
|
||||
"source_url": EPOCH_CSV_URL,
|
||||
"raw_reference_date": raw_date,
|
||||
"status": row.get("Status"),
|
||||
"certainty": row.get("Certainty"),
|
||||
"operator": row.get("Owner"),
|
||||
"gpu_type": row.get("Chip type (primary)"),
|
||||
"gpu_count": row.get("Chip quantity (primary)"),
|
||||
"location": location,
|
||||
"location_precision": location_assessment.precision,
|
||||
"location_precision_reason": location_assessment.reason,
|
||||
"raw_country": row.get("Country"),
|
||||
"raw_performance": raw_value,
|
||||
"performance_encoding": "log10",
|
||||
"builds_upon": row.get("Builds Upon"),
|
||||
"superseded_by": row.get("Superseded by"),
|
||||
"possible_duplicate": row.get("Possible Duplicate", "").lower() == "true",
|
||||
"include_in_standard_analysis": row.get(
|
||||
"Include in Standard Analysis", ""
|
||||
).lower()
|
||||
== "true",
|
||||
"rounded": country == "中国",
|
||||
"anonymous": anonymous,
|
||||
"identity_scope": "snapshot_record" if anonymous else "named_cluster",
|
||||
"country_unresolved": country is None,
|
||||
"sources": [
|
||||
row[f"Source {i}"] for i in range(1, 6) if row.get(f"Source {i}")
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
# Try to find data table on the page
|
||||
tables = soup.find_all("table")
|
||||
for table in tables:
|
||||
rows = table.find_all("tr")
|
||||
for row in rows[1:]: # Skip header
|
||||
cells = row.find_all(["td", "th"])
|
||||
if len(cells) >= 5:
|
||||
try:
|
||||
cluster_name = cells[0].get_text(strip=True)
|
||||
if not cluster_name or cluster_name in ["Cluster", "System", "Name"]:
|
||||
continue
|
||||
|
||||
location_cell = cells[1].get_text(strip=True) if len(cells) > 1 else ""
|
||||
country, city = self._parse_location(location_cell)
|
||||
|
||||
perf_cell = cells[2].get_text(strip=True) if len(cells) > 2 else ""
|
||||
|
||||
entry = {
|
||||
"source_id": f"epoch_{re.sub(r'[^a-zA-Z0-9]', '_', cluster_name.lower())}",
|
||||
"name": cluster_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": self._parse_performance(perf_cell),
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"raw_data": perf_cell,
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError):
|
||||
continue
|
||||
|
||||
# If no table found, return sample data
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
|
||||
return data
|
||||
|
||||
def _parse_location(self, location: str) -> tuple:
|
||||
"""Parse location string into country and city"""
|
||||
if not location:
|
||||
return "", ""
|
||||
if "," in location:
|
||||
parts = location.rsplit(",", 1)
|
||||
city = parts[0].strip()
|
||||
country = parts[1].strip() if len(parts) > 1 else ""
|
||||
return country, city
|
||||
return location, ""
|
||||
|
||||
def _parse_performance(self, perf: str) -> str:
|
||||
"""Parse performance string to extract value"""
|
||||
if not perf:
|
||||
return "0"
|
||||
match = re.search(r"([\d,.]+)\s*(TFlop/s|PFlop/s|GFlop/s)?", perf, re.I)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
match = re.search(r"([\d,.]+)", perf)
|
||||
if match:
|
||||
return match.group(1).replace(",", "")
|
||||
return "0"
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "epoch_sample_1",
|
||||
"name": "Sample GPU Cluster",
|
||||
"country": "United States",
|
||||
"city": "San Francisco, CA",
|
||||
"latitude": "",
|
||||
"longitude": "",
|
||||
"value": "1000",
|
||||
"unit": "TFlop/s",
|
||||
"metadata": {
|
||||
"note": "Sample data - Epoch AI page structure may vary",
|
||||
},
|
||||
"reference_date": datetime.now(UTC).strftime("%Y-%m-%d"),
|
||||
},
|
||||
]
|
||||
@staticmethod
|
||||
def coordinate(value: str | None, bound: int) -> float | None:
|
||||
if not value:
|
||||
return None
|
||||
number = float(value)
|
||||
if not math.isfinite(number) or abs(number) > bound:
|
||||
raise ValueError("Epoch coordinate out of bounds")
|
||||
return number
|
||||
|
||||
@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
if max_sources > 0 and len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
import os
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
# PeeringDB API key - read from environment variable
|
||||
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class PeeringDBIXPCollector(HTTPCollector):
|
||||
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
"""Space-Track TLE Collector
|
||||
"""Space-Track TLE Collector.
|
||||
|
||||
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
|
||||
API documentation: https://www.space-track.org/documentation
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.logging import get_logger
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class SpaceTrackTLECollector(BaseCollector):
|
||||
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
password = settings.SPACETRACK_PASSWORD
|
||||
|
||||
if not username or not password:
|
||||
print("SPACETRACK: No credentials configured, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track credentials are not configured; using sample data",
|
||||
event="collector.spacetrack.credentials_missing",
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch started",
|
||||
event="collector.spacetrack.fetch.start",
|
||||
)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
logger.info_event(
|
||||
"Space-Track login response received",
|
||||
event="collector.spacetrack.login.response",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
|
||||
if login_response.status_code == 403:
|
||||
print("SPACETRACK: Trying alternate login method...")
|
||||
logger.warning_event(
|
||||
"Space-Track login returned forbidden; trying alternate method",
|
||||
event="collector.spacetrack.login.forbidden",
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
) as alt_client:
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
"password": password,
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
logger.info_event(
|
||||
"Space-Track alternate login response received",
|
||||
event="collector.spacetrack.alt_login.response",
|
||||
context={"status_code": alt_login.status_code},
|
||||
)
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
logger.info_event(
|
||||
"Space-Track alternate query completed",
|
||||
event="collector.spacetrack.alt_query.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track login failed; using sample data",
|
||||
event="collector.spacetrack.login.failed",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
logger.info_event(
|
||||
"Space-Track TLE query response received",
|
||||
event="collector.spacetrack.query.response",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE query failed; using sample data",
|
||||
event="collector.spacetrack.query.failed",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch completed",
|
||||
event="collector.spacetrack.fetch.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE fetch failed; using sample data",
|
||||
event="collector.spacetrack.fetch.failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -5,16 +5,18 @@ https://top500.org/lists/top500/
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import re
|
||||
from urllib.parse import urljoin
|
||||
from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.services.collectors.compute_base import ComputeCollector
|
||||
|
||||
|
||||
class TOP500Collector(BaseCollector):
|
||||
class TOP500Collector(ComputeCollector):
|
||||
name = "top500"
|
||||
priority = "P0"
|
||||
module = "L1"
|
||||
@@ -28,7 +30,7 @@ class TOP500Collector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
entries = self.parse_response(response.text)
|
||||
entries = self.parse_response(response.text, str(response.url))
|
||||
|
||||
semaphore = asyncio.Semaphore(8)
|
||||
|
||||
@@ -55,7 +57,7 @@ class TOP500Collector(BaseCollector):
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
detail_url = urljoin(str(top500_base_url), link.get("href"))
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
@@ -127,10 +129,14 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
return detail_map
|
||||
|
||||
def parse_response(self, html: str) -> List[Dict[str, Any]]:
|
||||
def parse_response(self, html: str, source_url: str = "") -> List[Dict[str, Any]]:
|
||||
"""Parse TOP500 HTML response"""
|
||||
data = []
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
canonical = soup.find("link", rel="canonical")
|
||||
date_url = source_url or (canonical.get("href", "") if canonical else "") or self._resolved_url or ""
|
||||
date_match = re.search(r"/lists/top500/(\d{4})/(06|11)/", date_url)
|
||||
reference_date = f"{date_match[1]}-{date_match[2]}-01" if date_match else None
|
||||
|
||||
# Find the table with TOP500 data
|
||||
table = None
|
||||
@@ -152,6 +158,9 @@ class TOP500Collector(BaseCollector):
|
||||
table = soup.find("table", {"class": "top500-table"}) or soup.find("table", {"id": "top500"})
|
||||
|
||||
if table:
|
||||
headers = [cell.get_text(" ", strip=True) for cell in table.select("tr")[0].find_all(["th", "td"])]
|
||||
if not reference_date:
|
||||
raise ValueError("TOP500 list date missing; previous snapshot retained")
|
||||
rows = table.select("tr")
|
||||
for row in rows[1:]: # Skip header row
|
||||
cells = row.find_all(["td", "th"])
|
||||
@@ -176,25 +185,32 @@ class TOP500Collector(BaseCollector):
|
||||
cores = cells[2].get_text(strip=True).replace(",", "")
|
||||
|
||||
rmax_text = cells[3].get_text(strip=True)
|
||||
rmax = self._parse_performance(rmax_text)
|
||||
rmax = self._parse_performance(rmax_text, headers[3])
|
||||
|
||||
rpeak_text = cells[4].get_text(strip=True)
|
||||
rpeak = self._parse_performance(rpeak_text)
|
||||
rpeak = self._parse_performance(rpeak_text, headers[4])
|
||||
|
||||
power = ""
|
||||
if len(cells) >= 6:
|
||||
power = cells[5].get_text(strip=True).replace(",", "")
|
||||
|
||||
identity = re.search(r"/system/(\d+)", detail_url)
|
||||
if not identity:
|
||||
raise ValueError("TOP500 stable system ID missing")
|
||||
entry = {
|
||||
"source_id": f"top500_{rank}",
|
||||
"source_id": f"top500_system_{identity[1]}",
|
||||
"name": system_name,
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": 0.0,
|
||||
"longitude": 0.0,
|
||||
"latitude": None,
|
||||
"longitude": None,
|
||||
"value": str(rmax),
|
||||
"unit": "PFlop/s",
|
||||
"unit": "FLOP/s",
|
||||
"metadata": {
|
||||
"compute_contract": "v1",
|
||||
"precision": "FP64", "measurement": "LINPACK Rmax",
|
||||
"source_url": detail_url, "list_url": date_url,
|
||||
"status": "listed", "rmax_flops": rmax, "rpeak_flops": rpeak,
|
||||
"rank": rank,
|
||||
"cores": cores,
|
||||
"rmax": rmax_text,
|
||||
@@ -204,15 +220,11 @@ class TOP500Collector(BaseCollector):
|
||||
"site": site,
|
||||
},
|
||||
"_detail_url": detail_url,
|
||||
"reference_date": "2025-11-01",
|
||||
"reference_date": reference_date,
|
||||
}
|
||||
data.append(entry)
|
||||
except (ValueError, IndexError, AttributeError) as e:
|
||||
continue
|
||||
|
||||
# If scraping failed, return sample data for testing
|
||||
if not data:
|
||||
data = self._get_sample_data()
|
||||
raise ValueError("Invalid TOP500 row; previous snapshot retained") from e
|
||||
|
||||
return data
|
||||
|
||||
@@ -227,9 +239,14 @@ class TOP500Collector(BaseCollector):
|
||||
return 0.0
|
||||
return 0.0
|
||||
|
||||
def _parse_performance(self, text: str) -> float:
|
||||
def _parse_performance(self, text: str, header: str = "") -> float:
|
||||
"""Parse performance value from text (handles E, P, T suffixes)"""
|
||||
text = text.strip().upper()
|
||||
text = text.strip().upper().replace(",", "")
|
||||
if not re.search(r"[EPTGMK]F", text):
|
||||
unit = re.search(r"\b([EPTGMK])FLOP/?S", header.upper())
|
||||
if not unit:
|
||||
raise ValueError("TOP500 performance unit missing")
|
||||
text = f"{text} {unit[1]}FLOP/S"
|
||||
multipliers = {
|
||||
"E": 1e18,
|
||||
"P": 1e15,
|
||||
@@ -249,73 +266,9 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
# Try simple float parsing
|
||||
try:
|
||||
return float(text.replace(",", ""))
|
||||
value = float(text.replace(",", ""))
|
||||
if not math.isfinite(value) or value < 0:
|
||||
raise ValueError("Invalid TOP500 performance")
|
||||
return value
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
def _get_sample_data(self) -> List[Dict[str, Any]]:
|
||||
"""Return sample data for testing when scraping fails"""
|
||||
return [
|
||||
{
|
||||
"source_id": "top500_1",
|
||||
"name": "El Capitan - HPE Cray EX255a, AMD 4th Gen EPYC 24C 1.8GHz, AMD Instinct MI300A",
|
||||
"country": "United States",
|
||||
"city": "Livermore, CA",
|
||||
"latitude": 37.6819,
|
||||
"longitude": -121.7681,
|
||||
"value": "1742.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 1,
|
||||
"cores": "11039616",
|
||||
"rmax": "1742.00",
|
||||
"rpeak": "2746.38",
|
||||
"power": "29581",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/NNSA/LLNL",
|
||||
"processor": "AMD 4th Gen EPYC 24C 1.8GHz",
|
||||
"interconnect": "Slingshot-11",
|
||||
"installation_year": "2025",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_2",
|
||||
"name": "Frontier - HPE Cray EX235a, AMD Optimized 3rd Generation EPYC 64C 2GHz, AMD Instinct MI250X",
|
||||
"country": "United States",
|
||||
"city": "Oak Ridge, TN",
|
||||
"latitude": 36.0107,
|
||||
"longitude": -84.2663,
|
||||
"value": "1353.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 2,
|
||||
"cores": "9066176",
|
||||
"rmax": "1353.00",
|
||||
"rpeak": "2055.72",
|
||||
"power": "24607",
|
||||
"manufacturer": "HPE",
|
||||
"site": "DOE/SC/Oak Ridge National Laboratory",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
{
|
||||
"source_id": "top500_3",
|
||||
"name": "Aurora - HPE Cray EX - Intel Exascale Compute Blade, Xeon CPU Max 9470 52C 2.4GHz, Intel Data Center GPU Max",
|
||||
"country": "United States",
|
||||
"city": "Argonne, IL",
|
||||
"latitude": 41.3784,
|
||||
"longitude": -87.8600,
|
||||
"value": "1012.00",
|
||||
"unit": "PFlop/s",
|
||||
"metadata": {
|
||||
"rank": 3,
|
||||
"cores": "9264128",
|
||||
"rmax": "1012.00",
|
||||
"rpeak": "1980.01",
|
||||
"power": "38698",
|
||||
"manufacturer": "Intel",
|
||||
},
|
||||
"reference_date": "2025-11-01",
|
||||
},
|
||||
]
|
||||
raise ValueError("Invalid TOP500 performance") from None
|
||||
|
||||
200
backend/app/services/compute_catalog.py
Normal file
@@ -0,0 +1,200 @@
|
||||
"""Canonical country and mutually exclusive regional compute participants."""
|
||||
|
||||
import re
|
||||
from collections.abc import Iterable
|
||||
|
||||
from app.core.countries import COUNTRY_CENTROIDS, COUNTRY_ENTRIES
|
||||
from app.core.earth_geography_policy import display_country
|
||||
|
||||
PALETTE = ("#91baff", "#f0c88a", "#89d2c2", "#c5abdf", "#eda9b3", "#b6c99a")
|
||||
DEFAULT_PARTICIPANTS = ("CN", "US", "EUROPE", "APAC", "OTHER")
|
||||
|
||||
# Explicit membership prevents a regional total from double-counting a country.
|
||||
# APAC deliberately excludes China.
|
||||
COMPARISON_GROUPS = (
|
||||
("CN", "中国", "China", ("CN",), {"latitude": 35.8617, "longitude": 104.1954}),
|
||||
("US", "美国", "United States", ("US",), {"latitude": 39.8283, "longitude": -98.5795}),
|
||||
(
|
||||
"EUROPE",
|
||||
"欧洲",
|
||||
"Europe",
|
||||
(
|
||||
"GB",
|
||||
"DE",
|
||||
"FR",
|
||||
"CH",
|
||||
"ES",
|
||||
"FI",
|
||||
"HU",
|
||||
"IE",
|
||||
"IT",
|
||||
"LU",
|
||||
"MD",
|
||||
"NL",
|
||||
"PL",
|
||||
"RO",
|
||||
"RU",
|
||||
"UA",
|
||||
),
|
||||
{"latitude": 54.0, "longitude": 20.0},
|
||||
),
|
||||
(
|
||||
"APAC",
|
||||
"亚太(不含中国)",
|
||||
"Asia Pacific (excluding China)",
|
||||
("JP", "KR", "IN", "AU", "AF", "ID", "NP", "PH", "PK", "SG", "TH"),
|
||||
{"latitude": 36.0, "longitude": 130.5},
|
||||
),
|
||||
(
|
||||
"OTHER",
|
||||
"其他地区",
|
||||
"Other",
|
||||
(
|
||||
"CA",
|
||||
"AO",
|
||||
"EG",
|
||||
"KE",
|
||||
"MZ",
|
||||
"NG",
|
||||
"UG",
|
||||
"ZA",
|
||||
"BR",
|
||||
"CO",
|
||||
"CR",
|
||||
"DO",
|
||||
"MX",
|
||||
"AE",
|
||||
"SY",
|
||||
"TR",
|
||||
),
|
||||
{"latitude": 12.0, "longitude": 10.0},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _country_code(canonical: str, aliases: list[str]) -> str | None:
|
||||
code = next((alias for alias in aliases if re.fullmatch(r"[A-Z]{2}", alias)), None)
|
||||
return "GB" if canonical == "英国" else code
|
||||
|
||||
|
||||
def build_country_catalog() -> list[dict]:
|
||||
"""Build the policy-projected canonical country drilldown catalog."""
|
||||
countries = []
|
||||
for canonical, aliases in COUNTRY_ENTRIES:
|
||||
if display_country(canonical) != canonical or canonical not in COUNTRY_CENTROIDS:
|
||||
continue
|
||||
code = _country_code(canonical, aliases)
|
||||
if code:
|
||||
countries.append(
|
||||
{
|
||||
"code": code,
|
||||
"name": canonical,
|
||||
"name_en": aliases[0],
|
||||
"kind": "country",
|
||||
"members": [code],
|
||||
"anchor": dict(COUNTRY_CENTROIDS[canonical]),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
return countries
|
||||
|
||||
|
||||
COUNTRY_CATALOG = build_country_catalog()
|
||||
COUNTRIES = {entry["code"]: entry["name"] for entry in COUNTRY_CATALOG}
|
||||
COUNTRY_PARTICIPANTS = {entry["code"]: entry for entry in COUNTRY_CATALOG}
|
||||
|
||||
|
||||
def _validate_group_members() -> None:
|
||||
members = [
|
||||
member for _code, _name, _name_en, codes, _anchor in COMPARISON_GROUPS for member in codes
|
||||
]
|
||||
if len(members) != len(set(members)) or set(members) != set(COUNTRIES):
|
||||
raise ValueError("compute comparison groups must cover country catalog exactly once")
|
||||
|
||||
|
||||
_validate_group_members()
|
||||
GROUP_BY_MEMBER = {
|
||||
member: code
|
||||
for code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
}
|
||||
|
||||
|
||||
def build_catalog() -> list[dict]:
|
||||
"""Return the five default participants before country drilldown entries."""
|
||||
catalog: list[dict] = []
|
||||
for index, (code, name, name_en, members, anchor) in enumerate(COMPARISON_GROUPS):
|
||||
entry = (
|
||||
dict(COUNTRY_PARTICIPANTS[code])
|
||||
if code in COUNTRY_PARTICIPANTS
|
||||
else {
|
||||
"code": code,
|
||||
"name": name,
|
||||
"name_en": name_en,
|
||||
"kind": "region",
|
||||
"members": list(members),
|
||||
"anchor": dict(anchor),
|
||||
"anchor_kind": "regional_label",
|
||||
}
|
||||
)
|
||||
entry.update(
|
||||
{
|
||||
"members": list(members),
|
||||
"parent_group": name,
|
||||
"parent_group_code": code,
|
||||
"order": index,
|
||||
"default_selected": True,
|
||||
"color": PALETTE[index % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(entry)
|
||||
|
||||
for entry in COUNTRY_CATALOG:
|
||||
if entry["code"] in DEFAULT_PARTICIPANTS:
|
||||
continue
|
||||
country = dict(entry)
|
||||
group_code = GROUP_BY_MEMBER[country["code"]]
|
||||
group_name = next(group[1] for group in COMPARISON_GROUPS if group[0] == group_code)
|
||||
country.update(
|
||||
{
|
||||
"parent_group": group_name,
|
||||
"parent_group_code": group_code,
|
||||
"order": len(catalog),
|
||||
"default_selected": False,
|
||||
"color": PALETTE[len(catalog) % len(PALETTE)],
|
||||
}
|
||||
)
|
||||
catalog.append(country)
|
||||
return catalog
|
||||
|
||||
|
||||
CATALOG = build_catalog()
|
||||
PARTICIPANTS = {entry["code"]: entry for entry in CATALOG}
|
||||
|
||||
|
||||
def participant_members(codes: Iterable[str]) -> dict[str, tuple[str, ...]]:
|
||||
"""Resolve country or region participant codes to canonical country codes."""
|
||||
return {code: tuple(PARTICIPANTS[code]["members"]) for code in codes}
|
||||
|
||||
|
||||
def source_country_names(codes: Iterable[str]) -> list[str]:
|
||||
"""Return unique canonical source labels, respecting China and Serbia ownership."""
|
||||
selected_members = {
|
||||
member for members in participant_members(codes).values() for member in members
|
||||
}
|
||||
selected_names = {COUNTRIES[code] for code in selected_members}
|
||||
return [name for name, _aliases in COUNTRY_ENTRIES if display_country(name) in selected_names]
|
||||
|
||||
|
||||
def source_country_participants(codes: Iterable[str]) -> dict[str, str]:
|
||||
"""Map stored source-country labels to their requested participant code."""
|
||||
owner_by_country = {
|
||||
member: participant
|
||||
for participant, members in participant_members(codes).items()
|
||||
for member in members
|
||||
}
|
||||
code_by_name = {name: code for code, name in COUNTRIES.items()}
|
||||
return {
|
||||
name: owner_by_country[code_by_name[display_country(name)]]
|
||||
for name in source_country_names(codes)
|
||||
}
|
||||
@@ -3,7 +3,12 @@
|
||||
This module is a thin domain wrapper that wires up
|
||||
:mod:`app.services.location` for compute centers:
|
||||
|
||||
SourceCoordinates
|
||||
Confirmed stored location -> trustworthy source coordinates -> exact
|
||||
official facility registration -> source city/site coordinates.
|
||||
|
||||
Epoch AI coordinates are admitted only at the grain asserted by its official
|
||||
``Location`` column. Country/region/empty locations remain unresolved even
|
||||
when the CSV happens to contain a latitude/longitude pair.
|
||||
|
||||
The online Nominatim step is intentionally reserved for the user-triggered
|
||||
``collect-location`` flow. The regular GeoJSON endpoint runs during Earth
|
||||
@@ -28,9 +33,12 @@ import httpx
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.compute_types import compute_record_filter
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.compute_center_location import ComputeCenterLocationRecord
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
from app.services.location import (
|
||||
LocationCandidate,
|
||||
@@ -166,8 +174,13 @@ def set_compute_center_location_cache(
|
||||
|
||||
async def refresh_compute_center_location_cache(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
as_of: datetime | None = None,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
result = await session.execute(select(ComputeCenterLocationRecord))
|
||||
query = select(ComputeCenterLocationRecord)
|
||||
if as_of is not None:
|
||||
query = query.where(ComputeCenterLocationRecord.verified_at < as_of)
|
||||
result = await session.execute(query)
|
||||
records = result.scalars().all()
|
||||
cache = {}
|
||||
for record in records:
|
||||
@@ -350,6 +363,58 @@ class ROROrganizationResolver:
|
||||
)
|
||||
|
||||
|
||||
class ComputeCenterSourceCoordinatesResolver:
|
||||
"""Use source coordinates only at the grain asserted by the source text."""
|
||||
|
||||
name = "source_coordinates"
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._default = SourceCoordinatesResolver()
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
if extra.get("source") != "epoch_ai_gpu":
|
||||
return self._default.resolve(query)
|
||||
if query.source_latitude in (None, 0.0) or query.source_longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput(
|
||||
attempted_queries=(f"source_coordinates:{assessment.precision}",)
|
||||
)
|
||||
output = self._default.resolve(query)
|
||||
if not output.candidates:
|
||||
return output
|
||||
candidate = output.candidates[0]
|
||||
source_location = extra.get("source_location")
|
||||
source_location_text = (
|
||||
source_location.strip() if isinstance(source_location, str) else ""
|
||||
)
|
||||
source_note = assessment.reason
|
||||
if source_location_text:
|
||||
source_note = f"{source_note}; Epoch Location: {source_location_text}"
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
**{
|
||||
**candidate.__dict__,
|
||||
"precision": assessment.precision,
|
||||
"confidence": 1.0 if assessment.precision == "precise" else 0.9,
|
||||
"source_note": source_note,
|
||||
"needs_confirmation": assessment.precision != "precise",
|
||||
"matched_location_name": source_location_text
|
||||
or candidate.matched_location_name,
|
||||
}
|
||||
),
|
||||
),
|
||||
attempted_queries=output.attempted_queries,
|
||||
)
|
||||
|
||||
|
||||
class StoredComputeCenterLocationResolver:
|
||||
"""Resolve a compute center through the DB-backed current-location cache."""
|
||||
|
||||
@@ -367,13 +432,26 @@ class StoredComputeCenterLocationResolver:
|
||||
longitude = parse_float(stored.get("longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
return ResolverOutput()
|
||||
precision = stored.get("precision") or "city"
|
||||
if (
|
||||
extra.get("source") == "epoch_ai_gpu"
|
||||
and stored.get("location_source") == "source_coordinates"
|
||||
):
|
||||
assessment = assess_epoch_location(
|
||||
extra.get("source_location"),
|
||||
country=query.country,
|
||||
raw_country=extra.get("source_raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
return ResolverOutput()
|
||||
precision = assessment.precision
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
display_name=stored.get("name") or query.name or "Compute center",
|
||||
precision=stored.get("precision") or "city",
|
||||
precision=precision,
|
||||
confidence=float(stored.get("confidence") or 0.85),
|
||||
query=f"stored_compute_center_location::{stored.get('source')}:{stored.get('source_id')}",
|
||||
source=self.name,
|
||||
@@ -386,6 +464,60 @@ class StoredComputeCenterLocationResolver:
|
||||
matched_location_name=stored.get("site") or stored.get("name") or query.name,
|
||||
location_verified_at=stored.get("verified_at"),
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"stored_location_source": stored.get("location_source"),
|
||||
"verification_status": stored.get("verification_status"),
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class VerifiedComputeSiteResolver:
|
||||
"""Resolve an exact source identity to an evidence-backed facility site."""
|
||||
|
||||
name = "official_facility_registry"
|
||||
|
||||
def resolve(self, query: LocationQuery) -> ResolverOutput:
|
||||
extra = query.extra or {}
|
||||
source = coerce_str(extra.get("source"))
|
||||
source_id = coerce_str(extra.get("source_id"))
|
||||
if not source or not source_id:
|
||||
return ResolverOutput()
|
||||
site = get_verified_compute_site(source, source_id)
|
||||
if not site:
|
||||
return ResolverOutput()
|
||||
source_urls = [coerce_str(url) for url in site.get("sources", []) if coerce_str(url)]
|
||||
source_note = coerce_str(site.get("source_note"))
|
||||
if source_urls:
|
||||
source_note = f"{source_note} Sources: {'; '.join(source_urls)}".strip()
|
||||
return ResolverOutput(
|
||||
candidates=(
|
||||
LocationCandidate(
|
||||
latitude=float(site["latitude"]),
|
||||
longitude=float(site["longitude"]),
|
||||
display_name=coerce_str(site.get("matched_location_name"))
|
||||
or query.name
|
||||
or "Compute center",
|
||||
precision=coerce_str(site.get("precision")) or "site",
|
||||
confidence=1.0,
|
||||
query=f"official_facility_registry::{source}:{source_id}",
|
||||
source=self.name,
|
||||
source_note=source_note or None,
|
||||
matched_fields=("source", "source_id"),
|
||||
needs_confirmation=False,
|
||||
city=coerce_str(site.get("city")) or None,
|
||||
region=None,
|
||||
country=coerce_str(site.get("country")) or query.country,
|
||||
matched_location_name=coerce_str(site.get("matched_location_name"))
|
||||
or None,
|
||||
location_verified_at=coerce_str(site.get("verified_at")) or None,
|
||||
suggested_registry_entry=None,
|
||||
raw_payload={
|
||||
"address": site.get("address"),
|
||||
"coordinate_basis": site.get("coordinate_basis"),
|
||||
"sources": source_urls,
|
||||
},
|
||||
),
|
||||
)
|
||||
)
|
||||
@@ -400,7 +532,7 @@ def _short_system_name(name: Any) -> str:
|
||||
return head or text
|
||||
|
||||
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, Any]:
|
||||
name = coerce_str(getattr(record, "name", None))
|
||||
return {
|
||||
"source": coerce_str(getattr(record, "source", None)),
|
||||
@@ -417,11 +549,13 @@ def _record_context(record: Any, metadata: dict[str, Any]) -> dict[str, str]:
|
||||
or metadata.get("manufacturer")
|
||||
),
|
||||
"organization": coerce_str(metadata.get("organization")),
|
||||
"source_location": metadata.get("location"),
|
||||
"source_raw_country": metadata.get("raw_country"),
|
||||
}
|
||||
|
||||
|
||||
def _context_to_query(
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
source_lat: float | None = None,
|
||||
source_lon: float | None = None,
|
||||
@@ -444,6 +578,8 @@ def _context_to_query(
|
||||
"site": context.get("site") or "",
|
||||
"operator": context.get("operator") or "",
|
||||
"organization": context.get("organization") or "",
|
||||
"source_location": context.get("source_location"),
|
||||
"source_raw_country": context.get("source_raw_country"),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -497,11 +633,36 @@ def _compute_center_query_plan(
|
||||
return plan
|
||||
|
||||
|
||||
def _compute_center_candidate_sort_key(candidate: LocationCandidate) -> tuple[int, int, float]:
|
||||
stored_origin = (candidate.raw_payload or {}).get("stored_location_source")
|
||||
stored_confirmed = (
|
||||
candidate.source == "stored_compute_center_location"
|
||||
and not candidate.needs_confirmation
|
||||
and stored_origin != "source_coordinates"
|
||||
)
|
||||
if stored_confirmed:
|
||||
source_rank = 0
|
||||
elif candidate.source == "source_coordinates" and candidate.precision == "precise":
|
||||
source_rank = 1
|
||||
elif candidate.source == "official_facility_registry":
|
||||
source_rank = 2
|
||||
elif candidate.source == "source_coordinates":
|
||||
source_rank = 3
|
||||
elif candidate.source == "stored_compute_center_location":
|
||||
source_rank = 4
|
||||
else:
|
||||
source_rank = 5
|
||||
precision_rank = {"precise": 0, "site": 1, "city": 2}.get(candidate.precision, 9)
|
||||
return (source_rank, precision_rank, -float(candidate.confidence or 0))
|
||||
|
||||
|
||||
COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
StoredComputeCenterLocationResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
" or stored compute-center location."
|
||||
@@ -510,7 +671,8 @@ COMPUTE_CENTER_PIPELINE = LocationPipeline(
|
||||
|
||||
COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
[
|
||||
SourceCoordinatesResolver(),
|
||||
ComputeCenterSourceCoordinatesResolver(),
|
||||
VerifiedComputeSiteResolver(),
|
||||
ROROrganizationResolver(),
|
||||
NominatimResolver(
|
||||
query_plan_builder=_compute_center_query_plan,
|
||||
@@ -518,6 +680,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
geocoder=lambda q: _geocode_online(q),
|
||||
),
|
||||
],
|
||||
sort_key=_compute_center_candidate_sort_key,
|
||||
failure_reason=(
|
||||
"Could not resolve to city-level coordinates from source coords"
|
||||
", ROR organization lookup, or online geocoding."
|
||||
@@ -531,6 +694,7 @@ COMPUTE_CENTER_COLLECTION_PIPELINE = LocationPipeline(
|
||||
_GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
"source_coordinates": "source_coordinates",
|
||||
"stored_compute_center_location": "stored_compute_center_location",
|
||||
"official_facility_registry": "official_facility_registry",
|
||||
"ror_organization_registry": "ror_organization",
|
||||
"nominatim_online_geocode": "online_geocode",
|
||||
}
|
||||
@@ -539,7 +703,7 @@ _GEOGRAPHY_MODE_BY_SOURCE = {
|
||||
def _candidate_to_location(
|
||||
candidate: LocationCandidate,
|
||||
*,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
) -> ComputeCenterLocation:
|
||||
geography_mode = _GEOGRAPHY_MODE_BY_SOURCE.get(candidate.source, "online_geocode")
|
||||
is_estimated = candidate.needs_confirmation or candidate.source.startswith(
|
||||
@@ -594,7 +758,7 @@ def _candidate_to_location(
|
||||
|
||||
def _diagnostic_for(
|
||||
record: Any,
|
||||
context: dict[str, str],
|
||||
context: dict[str, Any],
|
||||
*,
|
||||
failure_reason: str,
|
||||
attempted_queries: tuple[str, ...] = (),
|
||||
@@ -738,7 +902,7 @@ def build_compute_center_location_query(
|
||||
organization: str | None = None,
|
||||
) -> LocationQuery:
|
||||
name_value = coerce_str(name)
|
||||
context: dict[str, str] = {
|
||||
context: dict[str, Any] = {
|
||||
"source": coerce_str(source),
|
||||
"source_id": coerce_str(source_id),
|
||||
"name": name_value,
|
||||
@@ -767,7 +931,7 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
"""Seed stored compute-center locations only from real source coordinates."""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(["top500", "epoch_ai_gpu"]))
|
||||
.where(compute_record_filter(CollectedData))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
)
|
||||
result = await session.execute(stmt)
|
||||
@@ -783,6 +947,23 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
longitude = parse_float(get_record_field(record, "longitude"))
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
precision = "precise"
|
||||
confidence = 1.0
|
||||
needs_confirmation = False
|
||||
source_note = "Seeded from source-provided compute-center coordinates"
|
||||
if source_value == "epoch_ai_gpu":
|
||||
assessment = assess_epoch_location(
|
||||
metadata.get("location"),
|
||||
country=get_record_field(record, "country"),
|
||||
raw_country=metadata.get("raw_country"),
|
||||
)
|
||||
if not assessment.is_renderable:
|
||||
continue
|
||||
precision = assessment.precision
|
||||
confidence = 1.0 if precision == "precise" else 0.9
|
||||
needs_confirmation = precision != "precise"
|
||||
source_note = assessment.reason
|
||||
existing = await session.scalar(
|
||||
select(ComputeCenterLocationRecord)
|
||||
.where(ComputeCenterLocationRecord.source == source_value)
|
||||
@@ -790,7 +971,6 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
)
|
||||
if existing:
|
||||
continue
|
||||
metadata = record.extra_data or {}
|
||||
session.add(
|
||||
ComputeCenterLocationRecord(
|
||||
source=source_value,
|
||||
@@ -802,16 +982,16 @@ async def seed_compute_center_locations_from_source_coords(
|
||||
country=coerce_str(get_record_field(record, "country")) or None,
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
precision="precise",
|
||||
confidence=1.0,
|
||||
precision=precision,
|
||||
confidence=confidence,
|
||||
location_source="source_coordinates",
|
||||
source_note="Seeded from source-provided compute-center coordinates",
|
||||
source_note=source_note,
|
||||
raw_payload={
|
||||
"record_id": getattr(record, "id", None),
|
||||
"source": source_value,
|
||||
"source_id": source_id,
|
||||
},
|
||||
needs_confirmation=False,
|
||||
needs_confirmation=needs_confirmation,
|
||||
verification_status="source_provided",
|
||||
verified_at=None,
|
||||
)
|
||||
|
||||
44
backend/app/services/compute_center_verified_sites.json
Normal file
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"version": 1,
|
||||
"sites": [
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_jamstec_zettascaler_2_0_gyoukou", "epoch_jamstec_zettascaler_2_2_gyoukou"],
|
||||
"latitude": 35.380924,
|
||||
"longitude": 139.625008,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "日本",
|
||||
"city": "Yokohama",
|
||||
"matched_location_name": "JAMSTEC Yokohama Institute for Earth Sciences",
|
||||
"address": "3173-25 Showa-machi, Kanazawa-ku, Yokohama, Kanagawa, Japan",
|
||||
"source_note": "Gyoukou deployment confirmed by JAMSTEC's 2017 release. Campus-level anchor from its official access map center, not a surveyed machine-room or building coordinate.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.jamstec.go.jp/j/about/press_release/archive/2017/20171114.pdf",
|
||||
"https://www.jamstec.go.jp/e/about/access/yokohama.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_ids": ["epoch_argonne_nl_sophia__formerly_theta_"],
|
||||
"latitude": 41.718109383287235,
|
||||
"longitude": -87.984498648496,
|
||||
"precision": "site",
|
||||
"location_source": "official_facility_registry",
|
||||
"country": "美国",
|
||||
"city": "Argonne",
|
||||
"matched_location_name": "Argonne Leadership Computing Facility",
|
||||
"address": "9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA",
|
||||
"source_note": "Sophia is an ALCF system. Campus-level anchor from ALCF's official visitor map center; the published address identifies Building 240 but the coordinate is not a surveyed building or rack location.",
|
||||
"verified_at": "2026-09-21T17:06:31Z",
|
||||
"coordinate_basis": "official_access_map_center",
|
||||
"sources": [
|
||||
"https://www.alcf.anl.gov/systems/sophia",
|
||||
"https://docs.alcf.anl.gov/sophia/",
|
||||
"https://www.alcf.anl.gov/about/visiting-alcf"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
26
backend/app/services/compute_center_verified_sites.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""Evidence-backed facility anchors; never match an operator's headquarters by name."""
|
||||
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _site_index() -> dict[tuple[str, str], dict]:
|
||||
payload = json.loads(Path(__file__).with_suffix(".json").read_text(encoding="utf-8"))
|
||||
index = {}
|
||||
for site in payload["sites"]:
|
||||
if not (-90 <= site["latitude"] <= 90 and -180 <= site["longitude"] <= 180):
|
||||
raise ValueError("Invalid verified compute facility coordinate")
|
||||
for source_id in site["source_ids"]:
|
||||
key = (site["source"], source_id)
|
||||
if key in index:
|
||||
raise ValueError("Duplicate verified compute facility identity")
|
||||
index[key] = site
|
||||
return index
|
||||
|
||||
|
||||
def get_verified_compute_site(source: str, source_id: str) -> dict | None:
|
||||
site = _site_index().get((source, source_id))
|
||||
return deepcopy(site) if site else None
|
||||
99
backend/app/services/compute_configuration.py
Normal file
@@ -0,0 +1,99 @@
|
||||
"""Database overrides with bundled defaults and atomic, revision-checked publishing."""
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.request_context import get_request_id
|
||||
from app.models.system_log import AuditLog
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import COUNTRY_CATALOG
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.dialects.postgresql import insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
CATEGORY = "compute_comparison"
|
||||
DEFAULT_PATH = Path(__file__).with_name("compute_history_defaults.json")
|
||||
DEFAULT_CONFIGURATION = ComputeConfiguration.model_validate_json(DEFAULT_PATH.read_text())
|
||||
DEFAULT_REVISION = hashlib.sha256(DEFAULT_PATH.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def default_envelope() -> dict:
|
||||
return {
|
||||
"config": DEFAULT_CONFIGURATION.model_dump(mode="json"),
|
||||
"revision": DEFAULT_REVISION,
|
||||
"is_default": True,
|
||||
}
|
||||
|
||||
|
||||
async def read_configuration(db: AsyncSession) -> dict:
|
||||
record = (
|
||||
await db.execute(select(SystemSetting).where(SystemSetting.category == CATEGORY))
|
||||
).scalar_one_or_none()
|
||||
return dict(record.payload) if record else default_envelope()
|
||||
|
||||
|
||||
def configuration_response(envelope: dict) -> dict:
|
||||
return {
|
||||
**envelope,
|
||||
"countries": [
|
||||
{key: country[key] for key in ("code", "name", "name_en")}
|
||||
for country in COUNTRY_CATALOG
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def publish_configuration(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
revision: str,
|
||||
config: ComputeConfiguration,
|
||||
actor: User,
|
||||
reset: bool = False,
|
||||
) -> dict:
|
||||
# The unique category plus row lock also serializes concurrent first writes.
|
||||
await db.execute(
|
||||
insert(SystemSetting)
|
||||
.values(category=CATEGORY, payload=default_envelope())
|
||||
.on_conflict_do_nothing(index_elements=[SystemSetting.category])
|
||||
)
|
||||
record = (
|
||||
await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == CATEGORY).with_for_update()
|
||||
)
|
||||
).scalar_one()
|
||||
if record.payload["revision"] != revision:
|
||||
await db.rollback()
|
||||
raise HTTPException(409, "Configuration changed. Reload before saving your edits.")
|
||||
envelope = (
|
||||
default_envelope()
|
||||
if reset
|
||||
else {
|
||||
"config": config.model_dump(mode="json"),
|
||||
"revision": uuid4().hex,
|
||||
"is_default": reset,
|
||||
}
|
||||
)
|
||||
before = record.payload
|
||||
if reset:
|
||||
await db.delete(record)
|
||||
else:
|
||||
record.payload = envelope
|
||||
# Audit and configuration commit together: a published edit must be attributable.
|
||||
db.add(
|
||||
AuditLog(
|
||||
actor_id=actor.id,
|
||||
actor_name=actor.username,
|
||||
action="compute_comparison.reset" if reset else "compute_comparison.update",
|
||||
target_type="system_setting",
|
||||
target_id=CATEGORY,
|
||||
result="success",
|
||||
request_id=get_request_id(),
|
||||
details={"before": before, "after": envelope},
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
return envelope
|
||||
122
backend/app/services/compute_history.py
Normal file
@@ -0,0 +1,122 @@
|
||||
"""Versioned, source-backed annual observations for compute comparisons."""
|
||||
|
||||
from copy import deepcopy
|
||||
from datetime import date
|
||||
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
DEFAULT_PARTICIPANTS,
|
||||
PARTICIPANTS,
|
||||
participant_members,
|
||||
)
|
||||
from app.services.compute_configuration import DEFAULT_CONFIGURATION
|
||||
|
||||
|
||||
def _participant_value(
|
||||
members: tuple[str, ...], observations: dict[str, float | int], mode: str
|
||||
) -> tuple[float | int | None, dict]:
|
||||
covered = [member for member in members if member in observations]
|
||||
missing = [member for member in members if member not in observations]
|
||||
coverage = {
|
||||
"status": "complete" if not missing else "incomplete",
|
||||
"required_members": list(members),
|
||||
"covered_members": covered,
|
||||
"missing_members": missing,
|
||||
}
|
||||
if missing:
|
||||
return None, coverage
|
||||
if mode == "bilateral_share":
|
||||
if (
|
||||
not set(members).issubset({"CN", "US"})
|
||||
or not {"CN", "US"}.issubset(observations)
|
||||
or observations["US"] + observations["CN"] <= 0
|
||||
):
|
||||
coverage.update(
|
||||
{
|
||||
"status": "incompatible",
|
||||
"reason": "bilateral_share_requires_CN_and_US_observations",
|
||||
}
|
||||
)
|
||||
return None, coverage
|
||||
return (
|
||||
round(
|
||||
sum(observations[member] for member in members)
|
||||
/ (observations["US"] + observations["CN"])
|
||||
* 100,
|
||||
2,
|
||||
),
|
||||
coverage,
|
||||
)
|
||||
return sum(observations[member] for member in members), coverage
|
||||
|
||||
|
||||
def compute_history(
|
||||
countries: list[str],
|
||||
mode: str = "global_share",
|
||||
config: ComputeConfiguration | None = None,
|
||||
) -> dict:
|
||||
"""Aggregate only fully covered, same-year compatible observations."""
|
||||
config = config if config is not None else DEFAULT_CONFIGURATION
|
||||
metric = "global_share" if mode == "bilateral_share" else mode
|
||||
annual = {row.year: row for row in config.observations if row.metric == metric}
|
||||
gaps = {row.year: row for row in config.innovation}
|
||||
observed_years = [row.year for row in config.observations] + list(gaps)
|
||||
years = tuple(
|
||||
range(
|
||||
min(observed_years, default=date.today().year),
|
||||
max([date.today().year, *observed_years]) + 1,
|
||||
)
|
||||
)
|
||||
members_by_participant = participant_members(countries)
|
||||
rows = []
|
||||
for year in years:
|
||||
record = annual.get(year)
|
||||
observations = record.values if record else {}
|
||||
values, coverage = {}, {}
|
||||
for code, members in members_by_participant.items():
|
||||
values[code], coverage[code] = _participant_value(members, observations, mode)
|
||||
evidence_id = record.source_id if record else None
|
||||
rows.append(
|
||||
{"year": year, "values": values, "coverage": coverage, "evidence_id": evidence_id}
|
||||
)
|
||||
|
||||
selected_members = {member for members in members_by_participant.values() for member in members}
|
||||
has_innovation_scope = {"CN", "US"}.issubset(selected_members)
|
||||
return {
|
||||
"version": f"compute-history-{config.reviewed_at}",
|
||||
"provenance": "curated_reports",
|
||||
"reviewed_at": config.reviewed_at.isoformat(),
|
||||
"time_basis": "retrospective",
|
||||
"years": years,
|
||||
"mode": mode,
|
||||
"unit": "EFlop/s (FP32)" if mode == "absolute" else "%",
|
||||
"denominator": {"global_share": "global", "bilateral_share": "US+CN", "absolute": None}[
|
||||
mode
|
||||
],
|
||||
"catalog": deepcopy(CATALOG),
|
||||
"default_participants": list(DEFAULT_PARTICIPANTS),
|
||||
"countries": [deepcopy(PARTICIPANTS[code]) for code in countries],
|
||||
"compute": rows,
|
||||
"innovation": [
|
||||
{
|
||||
"year": year,
|
||||
"value": gaps[year].value if year in gaps and has_innovation_scope else None,
|
||||
"evidence_id": gaps[year].source_id
|
||||
if year in gaps and has_innovation_scope
|
||||
else None,
|
||||
}
|
||||
for year in years
|
||||
],
|
||||
"evidence": {
|
||||
source.id: source.model_dump(mode="json", exclude={"id"}) for source in config.sources
|
||||
},
|
||||
"innovation_scope": ["US", "CN"],
|
||||
"limitations": [
|
||||
"公开设施样本不代表全国总算力;FP64、FP16 与 FP32 不可直接相加。",
|
||||
"区域值只在全部成员有同年、同指标、可比较观测时汇总;缺失成员时返回 null。",
|
||||
"全球份额不从中美或其他局部成员反推剩余地区。",
|
||||
"年度资料按已配置报告引用展示,观测年与报告年份分列;跨年方法可比性仍需核验。",
|
||||
"创新分差为美国减中国;只有分差,不反推单国得分,不证明算力与创新的因果关系。",
|
||||
],
|
||||
}
|
||||
125
backend/app/services/compute_history_defaults.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"reviewed_at": "2026-09-22",
|
||||
"sources": [
|
||||
{
|
||||
"id": "caict-2021",
|
||||
"title": "中国算力发展指数白皮书(2021)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202109/P020210918521091309950.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2021,
|
||||
"published_at": null,
|
||||
"locator": "正文 11 页图 4(2020 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2022",
|
||||
"title": "中国算力发展指数白皮书(2022)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202211/P020221105727522653499.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2022,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2021 年全球份额)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2023",
|
||||
"title": "中国算力发展指数白皮书(2023)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202309/P020240326630458153765.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2023,
|
||||
"published_at": null,
|
||||
"locator": "正文 13 页图 5(2022 年全球份额);16 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "caict-2024",
|
||||
"title": "先进计算暨算力发展指数蓝皮书(2024年)",
|
||||
"url": "https://www.caict.ac.cn/kxyj/qwfb/bps/202501/P020250117503621662777.pdf",
|
||||
"publisher": "中国信息通信研究院",
|
||||
"report_year": 2024,
|
||||
"published_at": null,
|
||||
"locator": "正文 10 页图 3(2023 年全球份额);12 页(中国总规模)"
|
||||
},
|
||||
{
|
||||
"id": "innovation-2026",
|
||||
"title": "全球人工智能创新指数报告 2026:科技日报报道",
|
||||
"url": "https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml",
|
||||
"publisher": "科技日报(新浪转载)",
|
||||
"report_year": 2026,
|
||||
"published_at": "2026-07-17",
|
||||
"locator": "2023–2025 年美国减中国的指数分差"
|
||||
}
|
||||
],
|
||||
"observations": [
|
||||
{
|
||||
"year": 2020,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2021",
|
||||
"values": {
|
||||
"US": 36,
|
||||
"CN": 31,
|
||||
"JP": 6
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2021,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2022",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 5
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"US": 34,
|
||||
"CN": 33,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "global_share",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"US": 41,
|
||||
"CN": 31,
|
||||
"JP": 4
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2022,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2023",
|
||||
"values": {
|
||||
"CN": 302
|
||||
}
|
||||
},
|
||||
{
|
||||
"year": 2023,
|
||||
"metric": "absolute",
|
||||
"source_id": "caict-2024",
|
||||
"values": {
|
||||
"CN": 435
|
||||
}
|
||||
}
|
||||
],
|
||||
"innovation": [
|
||||
{
|
||||
"year": 2023,
|
||||
"value": 22.02,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2024,
|
||||
"value": 19.96,
|
||||
"source_id": "innovation-2026"
|
||||
},
|
||||
{
|
||||
"year": 2025,
|
||||
"value": 17.95,
|
||||
"source_id": "innovation-2026"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -45,7 +45,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
|
||||
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
|
||||
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
|
||||
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
|
||||
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
|
||||
@@ -56,7 +56,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
|
||||
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
|
||||
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
|
||||
|
||||
@@ -6,8 +6,10 @@ import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
@@ -25,8 +27,11 @@ BOUNDARY_MANIFEST_PATH = BOUNDARY_OUTPUT_DIR / "manifest.json"
|
||||
PMTILES_ARTIFACT_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/boundaries/earth-boundaries-china-pov-v1.pmtiles"
|
||||
)
|
||||
LEGACY_GEOJSON_PATH = REPO_ROOT / "frontend/public/earth/data/countries-admin0.min.geojson"
|
||||
POLICY_FALLBACK_PATH = (
|
||||
REPO_ROOT / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
)
|
||||
POV_POLICY_PATH = REPO_ROOT / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
POV_GEOMETRY_PATH = REPO_ROOT / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
LOCAL_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.local.json"
|
||||
EXAMPLE_CONFIG_PATH = REPO_ROOT / "config/earth-boundary-sources.example.json"
|
||||
|
||||
@@ -150,6 +155,21 @@ def _stable_json_hash(payload: Any) -> str:
|
||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
@lru_cache(maxsize=16)
|
||||
def _read_versioned_asset_cached(
|
||||
path: Path, mtime_ns: int, size_bytes: int
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
payload = _read_json(path)
|
||||
return payload, _stable_json_hash(payload)
|
||||
|
||||
|
||||
def _versioned_asset(path: Path) -> tuple[dict[str, Any], str]:
|
||||
if not path.exists():
|
||||
return {}, ""
|
||||
stat = path.stat()
|
||||
return _read_versioned_asset_cached(path, stat.st_mtime_ns, stat.st_size)
|
||||
|
||||
|
||||
def _artifact_extension(endpoint: str, content_type: str, payload: bytes) -> str:
|
||||
suffix = Path(endpoint.split("?", 1)[0]).suffix.lower()
|
||||
if suffix in {".json", ".geojson", ".zip", ".pbf"}:
|
||||
@@ -176,62 +196,37 @@ def _directory_stats(path: Path) -> dict[str, int]:
|
||||
return {"file_count": len(files), "size_bytes": sum(item.stat().st_size for item in files)}
|
||||
|
||||
|
||||
def _load_source_feature_collection(source: dict[str, Any]) -> dict[str, Any]:
|
||||
path = REPO_ROOT / source["path"]
|
||||
payload = _read_json(path)
|
||||
features = payload.get("features") if isinstance(payload, dict) else None
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": features if isinstance(features, list) else [],
|
||||
}
|
||||
|
||||
|
||||
def _write_high_precision_geojson_manifest(
|
||||
sources: list[dict[str, Any]],
|
||||
build_input_hash: str,
|
||||
missing_tools: list[str],
|
||||
) -> dict[str, Any]:
|
||||
BOUNDARY_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
admin0 = next(source for source in sources if source["kind"] == "admin0-boundaries")
|
||||
coastline = next(source for source in sources if source["kind"] == "coastline")
|
||||
claim_lines = next(source for source in sources if source["kind"] == "claim-lines")
|
||||
|
||||
admin0_payload = _load_source_feature_collection(admin0)
|
||||
coastline_payload = _load_source_feature_collection(coastline)
|
||||
claim_payload = _load_source_feature_collection(claim_lines)
|
||||
for feature in coastline_payload["features"]:
|
||||
props = feature.setdefault("properties", {})
|
||||
if isinstance(props, dict):
|
||||
props["PLANET_LAYER"] = "coastline"
|
||||
|
||||
base_payload = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [*admin0_payload["features"], *coastline_payload["features"]],
|
||||
}
|
||||
base_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-base.geojson"
|
||||
hover_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-hover.geojson"
|
||||
claim_path = BOUNDARY_OUTPUT_DIR / "earth-boundaries-high-precision-claims.geojson"
|
||||
_write_json(base_path, base_payload)
|
||||
_write_json(hover_path, admin0_payload)
|
||||
_write_json(claim_path, claim_payload)
|
||||
|
||||
manifest = {
|
||||
"version": "natural-earth-v1",
|
||||
"builtAt": _utc_now_iso(),
|
||||
"tileProvider": "geojson-high-precision",
|
||||
"format": "geojson-directory",
|
||||
"buildInputHash": build_input_hash,
|
||||
"base": base_path.name,
|
||||
"hoverIndex": hover_path.name,
|
||||
"claimLine": claim_path.name,
|
||||
"sourceFeatureCount": {
|
||||
"admin0": len(admin0_payload["features"]),
|
||||
"coastline": len(coastline_payload["features"]),
|
||||
"claimLines": len(claim_payload["features"]),
|
||||
},
|
||||
"pmtiles": None,
|
||||
"missingTools": missing_tools,
|
||||
}
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(Path(__file__).resolve().parents[3] / "scripts/build_earth_boundary_china_pov_geojson.py"),
|
||||
"--admin0-source", admin0["path"],
|
||||
"--coastline-source", coastline["path"],
|
||||
"--claims-source", claim_lines["path"],
|
||||
"--output-dir", _relative(BOUNDARY_OUTPUT_DIR),
|
||||
"--build-input-hash", build_input_hash,
|
||||
"--pov-policy", _relative(POV_POLICY_PATH),
|
||||
"--pov-geometry", str(POV_GEOMETRY_PATH),
|
||||
],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if result.returncode:
|
||||
raise EarthBoundaryBuildError(
|
||||
result.stderr.strip() or "Earth boundary policy build failed",
|
||||
code="policy_build_failed",
|
||||
)
|
||||
manifest = _read_json(BOUNDARY_MANIFEST_PATH)
|
||||
manifest["missingTools"] = missing_tools
|
||||
_write_json(BOUNDARY_MANIFEST_PATH, manifest)
|
||||
return manifest
|
||||
|
||||
@@ -291,6 +286,7 @@ def _build_input_hash(source_manifest: dict[str, Any]) -> str:
|
||||
for source in source_manifest.get("sources", [])
|
||||
],
|
||||
"pov_policy": source_manifest.get("povPolicy"),
|
||||
"pov_geometry_sha256": _versioned_asset(POV_GEOMETRY_PATH)[1],
|
||||
"build_config": BUILD_CONFIG,
|
||||
}
|
||||
)
|
||||
@@ -301,7 +297,47 @@ def _has_current_artifacts(boundary_manifest: dict[str, Any], build_input_hash:
|
||||
bool(boundary_manifest)
|
||||
and boundary_manifest.get("buildInputHash") == build_input_hash
|
||||
and boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and _policy_verified(boundary_manifest)
|
||||
and PMTILES_ARTIFACT_PATH.exists()
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
|
||||
|
||||
def _policy_verified(manifest: dict[str, Any]) -> bool:
|
||||
policy, policy_hash = _versioned_asset(POV_POLICY_PATH)
|
||||
geometry, geometry_hash = _versioned_asset(POV_GEOMETRY_PATH)
|
||||
coverage = manifest.get("povPolicy") or {}
|
||||
return (
|
||||
policy.get("profile") == "china-pov-v1"
|
||||
and coverage.get("profile") == policy["profile"]
|
||||
and coverage.get("sha256") == policy_hash
|
||||
and coverage.get("reviewedGeometrySourceCommit")
|
||||
== geometry.get("sourceCommit")
|
||||
and coverage.get("geometrySha256") == geometry_hash
|
||||
and coverage.get("applied") is True
|
||||
and {"china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"}
|
||||
<= set(coverage.get("appliedRuleIds") or [])
|
||||
)
|
||||
|
||||
|
||||
def _geojson_artifacts_exist(manifest: dict[str, Any]) -> bool:
|
||||
paths = [manifest.get("base"), manifest.get("hoverIndex")]
|
||||
if manifest.get("chinaClaims", {}).get("available"):
|
||||
paths.append(manifest.get("claimLine"))
|
||||
for asset in paths:
|
||||
if not isinstance(asset, str) or Path(asset).name != asset:
|
||||
return False
|
||||
if not (BOUNDARY_OUTPUT_DIR / asset).is_file():
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _pmtiles_hover_exists(manifest: dict[str, Any]) -> bool:
|
||||
hover = manifest.get("hoverIndex")
|
||||
return (
|
||||
isinstance(hover, str)
|
||||
and Path(hover).name == hover
|
||||
and (BOUNDARY_OUTPUT_DIR / hover).is_file()
|
||||
)
|
||||
|
||||
|
||||
@@ -313,24 +349,34 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
manifest_exists = BOUNDARY_MANIFEST_PATH.exists()
|
||||
high_precision_ready = (
|
||||
manifest_exists
|
||||
and _policy_verified(boundary_manifest)
|
||||
and (
|
||||
(
|
||||
boundary_manifest.get("tileProvider") == "pmtiles-mvt"
|
||||
and pmtiles_exists
|
||||
and _pmtiles_hover_exists(boundary_manifest)
|
||||
)
|
||||
or (
|
||||
boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
and _geojson_artifacts_exist(boundary_manifest)
|
||||
)
|
||||
or boundary_manifest.get("tileProvider") == "geojson-high-precision"
|
||||
)
|
||||
)
|
||||
legacy_exists = LEGACY_GEOJSON_PATH.exists()
|
||||
legacy_exists = POLICY_FALLBACK_PATH.exists() and _policy_verified(
|
||||
_versioned_asset(POLICY_FALLBACK_PATH)[0]
|
||||
)
|
||||
provider = (
|
||||
boundary_manifest.get("tileProvider")
|
||||
if high_precision_ready
|
||||
else "legacy-geojson" if legacy_exists else "missing"
|
||||
else "policy-geojson-fallback" if legacy_exists else "missing"
|
||||
)
|
||||
return {
|
||||
"provider": provider,
|
||||
"high_precision_ready": high_precision_ready,
|
||||
"fallback_available": legacy_exists,
|
||||
"policy_publishable": bool(
|
||||
high_precision_ready and boundary_manifest.get("povPolicy", {}).get("publicationReady")
|
||||
),
|
||||
"config_source": config_source,
|
||||
"config_path": _relative(LOCAL_CONFIG_PATH),
|
||||
"config_exists": LOCAL_CONFIG_PATH.exists(),
|
||||
@@ -347,6 +393,8 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"tileProvider": boundary_manifest.get("tileProvider"),
|
||||
"buildInputHash": boundary_manifest.get("buildInputHash"),
|
||||
"builtAt": boundary_manifest.get("builtAt"),
|
||||
"povPolicy": boundary_manifest.get("povPolicy"),
|
||||
"publicationReady": boundary_manifest.get("povPolicy", {}).get("publicationReady", False),
|
||||
},
|
||||
"pmtiles": {
|
||||
"path": _relative(PMTILES_ARTIFACT_PATH),
|
||||
@@ -354,9 +402,9 @@ def get_boundary_status() -> dict[str, Any]:
|
||||
"size_bytes": PMTILES_ARTIFACT_PATH.stat().st_size if pmtiles_exists else 0,
|
||||
},
|
||||
"legacy": {
|
||||
"path": _relative(LEGACY_GEOJSON_PATH),
|
||||
"path": _relative(POLICY_FALLBACK_PATH),
|
||||
"exists": legacy_exists,
|
||||
"size_bytes": LEGACY_GEOJSON_PATH.stat().st_size if legacy_exists else 0,
|
||||
"size_bytes": POLICY_FALLBACK_PATH.stat().st_size if legacy_exists else 0,
|
||||
},
|
||||
"source_manifest": {
|
||||
"path": _relative(SOURCE_MANIFEST_PATH),
|
||||
@@ -577,6 +625,8 @@ async def build_boundary_assets(progress_callback: Any = None) -> dict[str, Any]
|
||||
build_input_hash,
|
||||
"--pov-policy",
|
||||
_relative(POV_POLICY_PATH),
|
||||
"--pov-geometry",
|
||||
str(POV_GEOMETRY_PATH),
|
||||
]
|
||||
)
|
||||
if progress_callback:
|
||||
|
||||
@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
refresh_strategy = "reload"
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
self.refresh_strategy = "reload"
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
|
||||
if not update:
|
||||
return False
|
||||
|
||||
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
|
||||
keys = payload.get("entity_keys")
|
||||
if not isinstance(keys, list):
|
||||
keys = [payload.get("entity_key")]
|
||||
broadcaster.enqueue_vessel_update({
|
||||
"action": "remove",
|
||||
"vessels": [{"mmsi": key} for key in keys if key is not None],
|
||||
})
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
|
||||
@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import codecs
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from email.utils import parsedate_to_datetime
|
||||
@@ -865,9 +866,9 @@ def _get_locale_text(
|
||||
if isinstance(fallback, dict):
|
||||
value = _coerce_str(fallback.get(key))
|
||||
if value:
|
||||
if locale == "en-US" and _contains_cjk_text(value):
|
||||
return ""
|
||||
return value
|
||||
if locale == "en-US" and _is_chinese_language(item.content_language):
|
||||
return item.title if key == "title" else item.summary
|
||||
return ""
|
||||
|
||||
|
||||
@@ -1800,6 +1801,30 @@ def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _decode_feed_response(response: httpx.Response) -> str:
|
||||
content = response.content
|
||||
for marker, encoding in (
|
||||
(codecs.BOM_UTF32_LE, "utf-32"),
|
||||
(codecs.BOM_UTF32_BE, "utf-32"),
|
||||
(codecs.BOM_UTF16_LE, "utf-16"),
|
||||
(codecs.BOM_UTF16_BE, "utf-16"),
|
||||
(codecs.BOM_UTF8, "utf-8-sig"),
|
||||
):
|
||||
if content.startswith(marker):
|
||||
return content.decode(encoding)
|
||||
|
||||
declaration = re.match(
|
||||
br"\s*<\?xml\s+[^?]*\bencoding\s*=\s*['\"]([^'\"]+)['\"]",
|
||||
content,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
encoding = response.charset_encoding or (
|
||||
declaration.group(1).decode("ascii") if declaration else "utf-8"
|
||||
)
|
||||
# Decode strictly before parsing: response.text silently replaces non-UTF-8 bytes.
|
||||
return content.decode(encoding)
|
||||
|
||||
|
||||
def _parse_feed_entries(
|
||||
xml_text: str,
|
||||
source: NewsFeedSource,
|
||||
@@ -2453,7 +2478,9 @@ async def _fetch_single_feed_url(
|
||||
feed=feed,
|
||||
)
|
||||
try:
|
||||
items = _parse_feed_entries(response.text, source, feed=feed, config_payload=config_payload)
|
||||
items = _parse_feed_entries(
|
||||
_decode_feed_response(response), source, feed=feed, config_payload=config_payload
|
||||
)
|
||||
except Exception as exc:
|
||||
error, status = _format_source_fetch_error(
|
||||
status_code=status_code,
|
||||
@@ -2461,7 +2488,7 @@ async def _fetch_single_feed_url(
|
||||
body=response.text,
|
||||
parsed_count=0,
|
||||
)
|
||||
if error is None:
|
||||
if status == "empty":
|
||||
error = str(exc)
|
||||
status = "format_error"
|
||||
return source, [], error, _source_health_result(
|
||||
|
||||
198
backend/app/services/epoch_location.py
Normal file
@@ -0,0 +1,198 @@
|
||||
"""Interpret the geographic grain of Epoch AI's official ``Location`` field."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from app.core.countries import normalize_country
|
||||
|
||||
|
||||
_COUNTRY_LABEL_OVERRIDES = {
|
||||
"korea republic of": "韩国",
|
||||
}
|
||||
_REGION_ONLY_LABELS = {
|
||||
"europe",
|
||||
"iowa",
|
||||
"maryland",
|
||||
"new mexico",
|
||||
"texas",
|
||||
"the nordics",
|
||||
"virginia",
|
||||
"washington",
|
||||
}
|
||||
_UNLOCATABLE_LABELS = {'"on prem" for tesla', "on prem for tesla"}
|
||||
_HEMISPHERE_COORDINATE_PATTERN = re.compile(
|
||||
r"(?P<degrees>\d{1,3})(?:[°º]|\s)\s*(?P<minutes>\d{1,2})(?:['′]|\s)"
|
||||
r"(?P<seconds>\d{1,2}(?:\.\d+)?)?[\"″]?\s*(?P<hemisphere>[NSEW])",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LABELED_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"\b(?:lat|latitude)\b\s*[:=]?\s*(?P<latitude>-?\d{1,2}(?:\.\d+)?)"
|
||||
r".{0,40}\b(?:lon|lng|longitude)\b\s*[:=]?\s*"
|
||||
r"(?P<longitude>-?\d{1,3}(?:\.\d+)?)",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_DECIMAL_COORDINATE_PAIR_PATTERN = re.compile(
|
||||
r"^\s*[\[(]?\s*(-?\d{1,2}\.\d+)\s*[,;]\s*"
|
||||
r"(-?\d{1,3}\.\d+)\s*[\])]?\s*$"
|
||||
)
|
||||
_NUMBERED_STREET_PATTERN = re.compile(
|
||||
r"(?:\b(?:no\.?\s*)?\d+[a-z]?\s+(?:[\w.'’\-]+\s+){0,6}"
|
||||
r"(?:road|rd|street|st|avenue|ave|boulevard|blvd|drive|dr|lane|ln|"
|
||||
r"highway|hwy|way|parkway|pkwy|chome)\b|"
|
||||
r"\b(?:via|rue|calle|plaça|strasse|straße)\b[^,\n]{0,80},?\s*\d)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_POSTAL_CODE_PATTERN = re.compile(
|
||||
r"(?:\b\d{5}(?:-\d{4})?\b|\b\d{3}-\d{4}\b|"
|
||||
r"\b[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_SITE_CONTEXT_PATTERN = re.compile(
|
||||
r"\b(?:campus|building|cluster|data\s+cent(?:er|re)|facility|institute|"
|
||||
r"laborator(?:y|ies)|science\s+park|supercomputing\s+cent(?:er|re))\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_COORDINATE_CUE_PATTERN = re.compile(
|
||||
r"(?:\b(?:lat|latitude|lon|lng|longitude)\b|[°º].*[NSEW])",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EpochLocationAssessment:
|
||||
precision: str
|
||||
reason: str
|
||||
|
||||
@property
|
||||
def is_renderable(self) -> bool:
|
||||
return self.precision in {"precise", "site", "city"}
|
||||
|
||||
|
||||
def _normalized_label(value: Any) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", " ", str(value or "").casefold()).strip()
|
||||
|
||||
|
||||
def _country_from_label(value: Any) -> str | None:
|
||||
return normalize_country(value) or _COUNTRY_LABEL_OVERRIDES.get(
|
||||
_normalized_label(value)
|
||||
)
|
||||
|
||||
|
||||
def _coordinate(mapping: dict[str, Any], keys: tuple[str, ...], bound: float) -> float | None:
|
||||
raw = next((mapping.get(key) for key in keys if mapping.get(key) not in (None, "")), None)
|
||||
if isinstance(raw, bool):
|
||||
return None
|
||||
try:
|
||||
number = float(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return number if math.isfinite(number) and abs(number) <= bound else None
|
||||
|
||||
|
||||
def _has_explicit_coordinate_pair(text: str) -> bool:
|
||||
hemispheres = {}
|
||||
for match in _HEMISPHERE_COORDINATE_PATTERN.finditer(text):
|
||||
hemisphere = match.group("hemisphere").upper()
|
||||
degrees = int(match.group("degrees"))
|
||||
minutes = int(match.group("minutes"))
|
||||
seconds = float(match.group("seconds") or 0)
|
||||
bound = 90 if hemisphere in {"N", "S"} else 180
|
||||
if degrees <= bound and minutes < 60 and seconds < 60:
|
||||
hemispheres[hemisphere] = degrees
|
||||
if ({"N", "S"} & hemispheres.keys()) and ({"E", "W"} & hemispheres.keys()):
|
||||
return True
|
||||
labeled_pair = _LABELED_COORDINATE_PAIR_PATTERN.search(text)
|
||||
if labeled_pair:
|
||||
latitude = float(labeled_pair.group("latitude"))
|
||||
longitude = float(labeled_pair.group("longitude"))
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
decimal_pair = _DECIMAL_COORDINATE_PAIR_PATTERN.match(text)
|
||||
if not decimal_pair:
|
||||
return False
|
||||
latitude, longitude = map(float, decimal_pair.groups())
|
||||
return abs(latitude) <= 90 and abs(longitude) <= 180
|
||||
|
||||
|
||||
def _has_address_evidence(text: str) -> bool:
|
||||
if _NUMBERED_STREET_PATTERN.search(text):
|
||||
return True
|
||||
return bool(
|
||||
_POSTAL_CODE_PATTERN.search(text)
|
||||
and ("\n" in text or text.count(",") >= 2)
|
||||
and _SITE_CONTEXT_PATTERN.search(text)
|
||||
)
|
||||
|
||||
|
||||
def assess_epoch_location(
|
||||
location: Any,
|
||||
*,
|
||||
country: Any = None,
|
||||
raw_country: Any = None,
|
||||
) -> EpochLocationAssessment:
|
||||
"""Classify source coordinates without upgrading their source text.
|
||||
|
||||
Epoch's CSV supplies coordinates even when ``Location`` contains only a
|
||||
country. Those coordinates are useful source evidence, but are not a
|
||||
facility point and must not enter the renderable location pipeline.
|
||||
"""
|
||||
if isinstance(location, dict):
|
||||
latitude = _coordinate(location, ("lat", "latitude"), 90)
|
||||
longitude = _coordinate(location, ("lon", "lng", "longitude"), 180)
|
||||
has_coordinates = latitude is not None and longitude is not None
|
||||
return EpochLocationAssessment(
|
||||
"precise" if has_coordinates else "unknown",
|
||||
"complete explicit coordinate object"
|
||||
if has_coordinates
|
||||
else "coordinate object lacks a complete valid latitude/longitude pair",
|
||||
)
|
||||
|
||||
text = str(location or "").strip()
|
||||
if not text:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is empty")
|
||||
|
||||
normalized = _normalized_label(text)
|
||||
normalized_country = normalize_country(country)
|
||||
source_country = normalize_country(raw_country)
|
||||
location_country = _country_from_label(text)
|
||||
if location_country or (
|
||||
raw_country and normalized == _normalized_label(raw_country)
|
||||
):
|
||||
matched = location_country or source_country or normalized_country
|
||||
return EpochLocationAssessment(
|
||||
"country",
|
||||
f"Epoch Location identifies country only: {matched or text}",
|
||||
)
|
||||
if normalized in _REGION_ONLY_LABELS:
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
comma_parts = [part.strip() for part in text.split(",") if part.strip()]
|
||||
if (
|
||||
len(comma_parts) == 2
|
||||
and _normalized_label(comma_parts[0]) in _REGION_ONLY_LABELS
|
||||
and _country_from_label(comma_parts[1])
|
||||
):
|
||||
return EpochLocationAssessment(
|
||||
"region", f"Epoch Location identifies region/state only: {text}"
|
||||
)
|
||||
if normalized in _UNLOCATABLE_LABELS:
|
||||
return EpochLocationAssessment("unknown", "Epoch Location has no geographic place")
|
||||
if normalized.isdigit():
|
||||
return EpochLocationAssessment("unknown", "Epoch Location is only a numeric label")
|
||||
if _has_explicit_coordinate_pair(text):
|
||||
return EpochLocationAssessment(
|
||||
"precise", "Epoch Location contains a complete valid coordinate pair"
|
||||
)
|
||||
if _COORDINATE_CUE_PATTERN.search(text):
|
||||
return EpochLocationAssessment(
|
||||
"unknown", "Epoch Location contains an incomplete or invalid coordinate pair"
|
||||
)
|
||||
if _has_address_evidence(text):
|
||||
return EpochLocationAssessment("precise", "Epoch Location contains an address")
|
||||
if "\n" in text or _SITE_CONTEXT_PATTERN.search(text):
|
||||
return EpochLocationAssessment("site", "Epoch Location names a facility or campus")
|
||||
return EpochLocationAssessment("city", "Epoch Location identifies a locality")
|
||||
186
backend/app/services/llm_model_catalog.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Authenticated model discovery shared by refresh and connection checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
CATALOG_TIMEOUT_SECONDS = 30
|
||||
CATALOG_MAX_PAGES = 100
|
||||
CATALOG_PAGE_SIZE = 100
|
||||
CATALOG_REQUEST_ATTEMPTS = 2
|
||||
CATALOG_RETRY_DELAY_SECONDS = 0.2
|
||||
SUPPORTED_PROVIDER_APIS = {
|
||||
"openai-completions",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"ollama-generate",
|
||||
}
|
||||
|
||||
|
||||
class LLMProviderCatalogError(RuntimeError):
|
||||
"""A safe, user-facing catalog failure with no upstream response or credentials."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCatalog:
|
||||
url: str
|
||||
models: list[str]
|
||||
|
||||
|
||||
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
if parts.scheme not in {"http", "https"} or not parts.hostname:
|
||||
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
|
||||
if parts.username or parts.password or parts.query or parts.fragment:
|
||||
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
|
||||
if provider_api not in SUPPORTED_PROVIDER_APIS:
|
||||
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
|
||||
path = parts.path.rstrip("/")
|
||||
if provider_api == "ollama-generate":
|
||||
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
|
||||
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
|
||||
path = "/api/v1/models"
|
||||
else:
|
||||
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
|
||||
path += "/v1"
|
||||
path += "/models"
|
||||
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
|
||||
|
||||
|
||||
def catalog_error_message(exc: Exception) -> str:
|
||||
if isinstance(exc, LLMProviderCatalogError):
|
||||
return str(exc)
|
||||
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
|
||||
return "模型目录请求超时,请检查网络后重试。"
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
code = exc.response.status_code
|
||||
messages = {
|
||||
401: "API Key 验证失败,请检查当前供应商的凭证。",
|
||||
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
|
||||
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
|
||||
429: "供应商请求限流,请稍后重试。",
|
||||
}
|
||||
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return "无法连接模型目录,请检查基础地址和网络。"
|
||||
return "模型目录响应无效,请稍后重试。"
|
||||
|
||||
|
||||
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
envelope = payload.get("output", payload)
|
||||
if not isinstance(envelope, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
rows = envelope.get("data", envelope.get("models"))
|
||||
if not isinstance(rows, list):
|
||||
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
|
||||
return rows, envelope
|
||||
|
||||
|
||||
def _model_id(row: dict[str, object]) -> str:
|
||||
value = row.get("id") or row.get("model") or row.get("name")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _model_date(row: dict[str, object]) -> float:
|
||||
value = row.get("created_at") or row.get("published_time") or row.get("created")
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
async def _get_catalog_page(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
headers: dict[str, str],
|
||||
params: dict[str, str | int],
|
||||
) -> object:
|
||||
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
|
||||
try:
|
||||
response = await client.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if attempt or exc.response.status_code not in {502, 503, 504}:
|
||||
raise
|
||||
except httpx.TransportError:
|
||||
if attempt:
|
||||
raise
|
||||
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
|
||||
raise LLMProviderCatalogError("模型目录请求失败。")
|
||||
|
||||
|
||||
async def fetch_model_catalog(
|
||||
provider: str,
|
||||
base_url: str,
|
||||
provider_api: str,
|
||||
api_key: str = "",
|
||||
anthropic_version: str = "2023-06-01",
|
||||
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
|
||||
) -> ModelCatalog:
|
||||
url = model_catalog_url(provider, base_url, provider_api)
|
||||
public_catalog = provider in {"opencode-go", "openrouter"}
|
||||
if not api_key and provider_api != "ollama-generate" and not public_catalog:
|
||||
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
|
||||
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
|
||||
if provider_api == "anthropic-messages" and provider not in {
|
||||
"opencode-go",
|
||||
"openrouter",
|
||||
"alibaba",
|
||||
"moonshotai",
|
||||
}:
|
||||
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
|
||||
elif api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
|
||||
params: dict[str, str | int] = {}
|
||||
if native_dashscope:
|
||||
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
|
||||
rows_by_id: dict[str, dict[str, object]] = {}
|
||||
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
|
||||
# Bound the complete pagination/retry cycle, not just each individual request.
|
||||
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
|
||||
for page in range(CATALOG_MAX_PAGES):
|
||||
payload = await _get_catalog_page(client, url, headers, params)
|
||||
rows, envelope = _model_rows(payload)
|
||||
previous_count = len(rows_by_id)
|
||||
for row in rows:
|
||||
rows_by_id[_model_id(row)] = row
|
||||
has_more = envelope.get("has_more") is True
|
||||
if native_dashscope:
|
||||
total = envelope.get("total")
|
||||
if not isinstance(total, int) or total < 0:
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
has_more = len(rows_by_id) < total
|
||||
params["page_no"] = page + 2
|
||||
elif has_more:
|
||||
cursor = envelope.get("last_id")
|
||||
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
params["after_id"] = cursor
|
||||
if not has_more:
|
||||
models = sorted(
|
||||
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
|
||||
)
|
||||
# An empty Ollama catalog is valid: no models have been installed yet.
|
||||
if not models and provider_api != "ollama-generate":
|
||||
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
|
||||
return ModelCatalog(url=url, models=models)
|
||||
if len(rows_by_id) == previous_count:
|
||||
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
|
||||
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")
|
||||
@@ -3,17 +3,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
from app.services.llm_model_catalog import fetch_model_catalog
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m3": "anthropic-messages",
|
||||
"qwen3.8-max": "anthropic-messages",
|
||||
"qwen3.8-flash": "anthropic-messages",
|
||||
"qwen3.7-max": "anthropic-messages",
|
||||
"qwen3.7-plus": "anthropic-messages",
|
||||
"qwen3.6-plus": "anthropic-messages",
|
||||
"grok-4.6": "openai-responses",
|
||||
"gpt-5.6-luna": "openai-responses",
|
||||
"muse-spark-1.3-contributor": "openai-responses",
|
||||
"muse-spark-1.2-contributor": "openai-responses",
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m3",
|
||||
"kimi-k3",
|
||||
"glm-5.3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
|
||||
]
|
||||
|
||||
|
||||
OPENAI_MODEL_PROVIDER_APIS = {
|
||||
model: "openai-responses"
|
||||
for model in [
|
||||
"gpt-6-astra",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.1",
|
||||
"gpt-5.1-codex",
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"model": "MiniMax-M3",
|
||||
"models": [
|
||||
"MiniMax-M3",
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
"MiniMax-M2.1",
|
||||
"MiniMax-M2.1-highspeed",
|
||||
"MiniMax-M2",
|
||||
],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": "openai-responses",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"model": "gpt-6-astra",
|
||||
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
|
||||
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"model": "claude-opus-5",
|
||||
"models": ["claude-opus-5", "claude-sonnet-4-6"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"model": "deepseek-flash",
|
||||
"models": ["deepseek-flash", "deepseek-v4-pro"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"model": "qwen3.8-max",
|
||||
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"model": "kimi-k3",
|
||||
"models": ["kimi-k3", "kimi-k2.5"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
async def refresh_llm_provider_preset(
|
||||
provider: str,
|
||||
api_key: str | None = None,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
provider_api: str | None = None,
|
||||
anthropic_version: str = "2023-06-01",
|
||||
) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
resolved_base_url = base_url or fallback["base_url"]
|
||||
resolved_api = provider_api or fallback["provider_api"]
|
||||
catalog = await fetch_model_catalog(
|
||||
fallback["provider"],
|
||||
resolved_base_url,
|
||||
resolved_api,
|
||||
api_key or "",
|
||||
anthropic_version,
|
||||
)
|
||||
model_provider_apis = (
|
||||
_opencode_go_model_provider_apis(catalog.models)
|
||||
if fallback["provider"] == "opencode-go"
|
||||
else fallback.get("model_provider_apis", {})
|
||||
)
|
||||
if (
|
||||
fallback["provider"] == "openai"
|
||||
and urlsplit(resolved_base_url).hostname != "api.openai.com"
|
||||
):
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
"base_url": resolved_base_url,
|
||||
"provider_api": resolved_api,
|
||||
"model": catalog.models[0] if catalog.models else "",
|
||||
"models": catalog.models,
|
||||
"model_provider_apis": model_provider_apis,
|
||||
"source": catalog.url,
|
||||
}
|
||||
return refreshed
|
||||
|
||||
@@ -635,6 +635,7 @@ async def _run_assistant_message(
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"session_id": session_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
|
||||
148
backend/app/services/tv_catalog.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""Search and paginate the public live TV catalog at the database boundary."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Select, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.services.tv_streams import (
|
||||
TV_LIVE_SOURCE_COLLECTOR,
|
||||
TV_LIVE_SOURCE_DATA_TYPE,
|
||||
_build_collected_tv_source,
|
||||
build_public_tv_payload,
|
||||
get_tv_settings_payload,
|
||||
)
|
||||
|
||||
|
||||
def _source_key():
|
||||
return func.coalesce(
|
||||
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
|
||||
func.nullif(CollectedData.source_id, ""),
|
||||
CollectedData.entity_key,
|
||||
)
|
||||
|
||||
|
||||
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
source_id = _source_key()
|
||||
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
|
||||
ranked = (
|
||||
select(
|
||||
CollectedData.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=source_id,
|
||||
order_by=CollectedData.id.desc(),
|
||||
)
|
||||
.label("source_rank"),
|
||||
)
|
||||
.where(
|
||||
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
|
||||
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
|
||||
CollectedData.is_current.is_(True),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.deleted_at.is_(None),
|
||||
enabled.notin_(("false", "0", "no", "off")),
|
||||
source_id.notin_(configured_ids),
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
return (
|
||||
select(CollectedData)
|
||||
.join(ranked, ranked.c.id == CollectedData.id)
|
||||
.where(ranked.c.source_rank == 1)
|
||||
)
|
||||
|
||||
|
||||
def _filter_catalog_query(
|
||||
query: Select[tuple[CollectedData]], terms: list[str]
|
||||
) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
searchable = func.lower(
|
||||
func.concat_ws(
|
||||
" ",
|
||||
CollectedData.name,
|
||||
CollectedData.title,
|
||||
CollectedData.source_id,
|
||||
metadata["name"].as_string(),
|
||||
metadata["provider"].as_string(),
|
||||
metadata["region"].as_string(),
|
||||
metadata["country"].as_string(),
|
||||
metadata["language"].as_string(),
|
||||
)
|
||||
)
|
||||
for term in terms:
|
||||
query = query.where(searchable.contains(term, autoescape=True))
|
||||
return query
|
||||
|
||||
|
||||
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
|
||||
searchable = " ".join(
|
||||
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
|
||||
).lower()
|
||||
return all(term in searchable for term in terms)
|
||||
|
||||
|
||||
async def get_tv_catalog_page(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
q: str = "",
|
||||
selected_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
settings = await get_tv_settings_payload(db)
|
||||
payload = build_public_tv_payload(settings, [])
|
||||
configured = payload["sources"]
|
||||
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
|
||||
if selected_id:
|
||||
selected = next((source for source in configured if source["id"] == selected_id), None)
|
||||
if selected is None:
|
||||
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
|
||||
selected = _build_collected_tv_source(record, 0) if record else None
|
||||
if selected:
|
||||
payload["selected_source"] = selected
|
||||
summary = (
|
||||
await db.execute(
|
||||
select(func.count(), func.max(CollectedData.collected_at))
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
|
||||
)
|
||||
).one()
|
||||
total_collected, latest_update = summary
|
||||
terms = q.lower().split()
|
||||
matched_configured = [source for source in configured if _matches_source(source, terms)]
|
||||
filtered_query = _filter_catalog_query(query, terms)
|
||||
matched_collected = (
|
||||
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
|
||||
if terms
|
||||
else total_collected
|
||||
)
|
||||
sources = matched_configured[offset : offset + limit]
|
||||
remaining = limit - len(sources)
|
||||
if remaining:
|
||||
rows = await db.scalars(
|
||||
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
|
||||
.offset(max(0, offset - len(matched_configured)))
|
||||
.limit(remaining)
|
||||
)
|
||||
sources.extend(
|
||||
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
|
||||
)
|
||||
total = len(matched_configured) + matched_collected
|
||||
next_offset = offset + len(sources)
|
||||
return {
|
||||
**payload,
|
||||
"sources": sources,
|
||||
"source_count": len(configured) + total_collected,
|
||||
"latest_updated_at": (
|
||||
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
|
||||
),
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
@@ -11,10 +11,16 @@ from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
CCTV4_HOMEPAGE = "https://tv.cctv.com/live/cctv4/"
|
||||
CCTV4_STREAM = "https://h5cul1yar48um3t.wcetv.com/hls/cctv4.m3u8"
|
||||
UNSUPPORTED_CCTV4_STREAM = (
|
||||
"https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8"
|
||||
)
|
||||
CCTV4_PLAYBACK_NOTE = "CCTV-4 美洲版,使用 WCETV HLS 直播源;节目编排可能与亚洲版不同。"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
@@ -22,20 +28,21 @@ DEFAULT_TV_SETTINGS = {
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"name": "CCTV-4 中文国际(美洲版)",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"embed_url": "",
|
||||
"stream_url": CCTV4_STREAM,
|
||||
"use_proxy": False,
|
||||
"homepage_url": CCTV4_HOMEPAGE,
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
"notes": CCTV4_PLAYBACK_NOTE,
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
@@ -237,6 +244,17 @@ def _clean_int(value: Any, *, default: int) -> int:
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
# Retire only the formerly bundled CDRM URL, including saved copies of that default.
|
||||
# HTTP 200 and valid TS framing do not mean this stream is decodable by hls.js.
|
||||
if _clean_text(payload.get("id")) == "cctv4" and _clean_url(
|
||||
payload.get("stream_url")
|
||||
) == UNSUPPORTED_CCTV4_STREAM:
|
||||
payload.update(
|
||||
name="CCTV-4 中文国际(美洲版)",
|
||||
source_type="hls", stream_url=CCTV4_STREAM, embed_url="",
|
||||
use_proxy=False,
|
||||
homepage_url=CCTV4_HOMEPAGE, notes=CCTV4_PLAYBACK_NOTE,
|
||||
)
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
@@ -259,6 +277,7 @@ def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dic
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"use_proxy": _clean_bool(payload.get("use_proxy"), default=True),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
@@ -300,8 +319,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
default_source = next(
|
||||
source for source in DEFAULT_TV_SETTINGS["sources"]
|
||||
if source["id"] == DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
normalize_tv_source(default_source, index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
|
||||
@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
|
||||
return [item.to_dict() for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_current_vessels_by_mmsi(
|
||||
db: AsyncSession, mmsis: list[int]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read canonical render state for the vessels changed by a stream flush."""
|
||||
if not mmsis:
|
||||
return []
|
||||
result = await db.execute(
|
||||
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
|
||||
)
|
||||
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def aggregate_vessel_observations(
|
||||
db: AsyncSession,
|
||||
observations: Iterable[AISRawObservation],
|
||||
|
||||
72
backend/tests/test_auth_identity.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.auth import router
|
||||
from app.core import security
|
||||
from app.db.session import DEFAULT_LOGIN_USERS, get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.user import UserRegister
|
||||
|
||||
|
||||
async def identity_request(monkeypatch, user, token=None):
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = user
|
||||
db = AsyncMock()
|
||||
db.execute.return_value = result
|
||||
monkeypatch.setattr(security.redis_client, "sismember", lambda *_: False)
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/auth")
|
||||
|
||||
async def override_db():
|
||||
yield db
|
||||
|
||||
app.dependency_overrides[get_db] = override_db
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=app), base_url="http://test"
|
||||
) as client:
|
||||
return await client.get(
|
||||
"/auth/me",
|
||||
headers={"Authorization": "Bearer " + (token or security.create_access_token({"sub": 1}))},
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("default", DEFAULT_LOGIN_USERS)
|
||||
async def test_default_identity_preserves_profile_fields(monkeypatch, default):
|
||||
created = datetime(2026, 9, 21, tzinfo=UTC)
|
||||
user = User(
|
||||
id=1, username=default["username"], email=default["email"],
|
||||
role=default["role"], password_hash="not-exposed", is_active=True,
|
||||
email_verified=True, gatekeeper_groups=["ops"], created_at=created,
|
||||
)
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["email"] == default["email"]
|
||||
assert payload["email_verified"] is True
|
||||
assert datetime.fromisoformat(payload["created_at"].replace("Z", "+00:00")) == created
|
||||
assert payload["gatekeeper_groups"] == ["ops"]
|
||||
assert "password_hash" not in payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("user", [None, User(id=1, is_active=False)])
|
||||
async def test_missing_or_inactive_identity_is_rejected(monkeypatch, user):
|
||||
response = await identity_request(monkeypatch, user)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_token_is_not_an_access_identity(monkeypatch):
|
||||
response = await identity_request(monkeypatch, None, security.create_refresh_token({"sub": 1}))
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_registration_still_validates_email():
|
||||
with pytest.raises(ValidationError):
|
||||
UserRegister(username="newuser", email="newuser@planet.local", password="example-password")
|
||||
333
backend/tests/test_card_images.py
Normal file
@@ -0,0 +1,333 @@
|
||||
import io
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from PIL import Image
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.v1.card_images import router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.schemas.card_images import CardImageAsset, CardImageConfiguration, CardImageRule
|
||||
from app.services.card_image_configuration import (
|
||||
effective_envelope,
|
||||
publish_configuration,
|
||||
resolve_image,
|
||||
)
|
||||
from app.services.card_image_defaults import bundled_configuration
|
||||
from app.services import card_image_files
|
||||
|
||||
|
||||
def configuration():
|
||||
assets = [
|
||||
CardImageAsset(id=key, title=key, url=f"/earth/assets/cards/{key}.jpg")
|
||||
for key in ["exact", "model", "group", "category"]
|
||||
]
|
||||
rules = [
|
||||
CardImageRule(id=key, entity_type="satellite", scope=scope, value=value, asset_id=key)
|
||||
for key, scope, value in [
|
||||
("exact", "object", "123"),
|
||||
("model", "model", "m1"),
|
||||
("group", "group", "starlink"),
|
||||
("category", "category", "satellite"),
|
||||
]
|
||||
]
|
||||
return CardImageConfiguration(assets=assets, rules=rules)
|
||||
|
||||
|
||||
def test_precedence_disabled_fallback_and_explicit_no_image():
|
||||
config = configuration()
|
||||
context = dict(
|
||||
entity_type="satellite", entity_id="123", model="M1", group="starlink", category="satellite"
|
||||
)
|
||||
for expected in ["exact", "model", "group", "category"]:
|
||||
assert resolve_image(config, context)["image"]["id"] == expected
|
||||
next(asset for asset in config.assets if asset.id == expected).enabled = False
|
||||
assert resolve_image(config, context)["image"] is None
|
||||
config = configuration()
|
||||
config.rules[0].asset_id = None
|
||||
config.rules[0].suppress = True
|
||||
result = resolve_image(config, context)
|
||||
assert result["suppressed"] and result["image"] is None
|
||||
|
||||
|
||||
def test_bundled_series_images_retain_truthful_source_metadata():
|
||||
config, _ = bundled_configuration()
|
||||
result = resolve_image(config, dict(entity_type="satellite", entity_id="123", group="starlink"))
|
||||
assert result["image"]["id"] == "starlink-solar-array"
|
||||
pending = next(asset for asset in config.assets if asset.id == "qianfan-animation-still")
|
||||
assert pending.rights_status == "permission_pending"
|
||||
assert (
|
||||
resolve_image(config, dict(entity_type="satellite", group="qianfan"))["image"]["id"]
|
||||
== pending.id
|
||||
)
|
||||
assert pending.enabled
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"url",
|
||||
[
|
||||
"javascript:alert(1)",
|
||||
"//evil.example/image.jpg",
|
||||
"/earth/assets/cards/../secret",
|
||||
"https://user:pass@example.org/a.jpg",
|
||||
],
|
||||
)
|
||||
def test_unsafe_urls_rejected(url):
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageAsset(id="test", title="test", url=url)
|
||||
|
||||
|
||||
def test_duplicate_targets_and_dangling_assets_rejected():
|
||||
data = configuration().model_dump()
|
||||
duplicate = {**data["rules"][0], "id": "another"}
|
||||
data["rules"].append(duplicate)
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
data = configuration().model_dump()
|
||||
data["assets"] = []
|
||||
with pytest.raises(ValidationError):
|
||||
CardImageConfiguration.model_validate(data)
|
||||
|
||||
|
||||
def test_new_defaults_do_not_undo_saved_removals_or_manual_edits():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "custom",
|
||||
"config": {"assets": [], "rules": []},
|
||||
"known_assets": [asset.id for asset in defaults.assets],
|
||||
"known_rules": [rule.id for rule in defaults.rules],
|
||||
}
|
||||
assert effective_envelope(payload)["config"] == {"assets": [], "rules": []}
|
||||
payload["known_assets"] = []
|
||||
assert len(effective_envelope(payload)["config"]["assets"]) == len(defaults.assets)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("format", ["JPEG", "PNG", "WEBP"])
|
||||
def test_upload_verifies_raster_and_creates_reusable_variants(tmp_path, monkeypatch, format):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
output = io.BytesIO()
|
||||
Image.new("RGB", (1600, 900), "blue").save(output, format)
|
||||
asset = card_image_files.store_image(output.getvalue(), "Facility")
|
||||
assert (asset.width, asset.height) == (1600, 900)
|
||||
assert len(list(tmp_path.iterdir())) == 3
|
||||
for size, url in asset.variants.items():
|
||||
with Image.open(tmp_path / url.rsplit("/", 1)[1]) as image:
|
||||
assert max(image.size) == int(size)
|
||||
assert card_image_files.store_image(output.getvalue(), "Again").id == asset.id
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.store_image(b"<svg></svg>", "bad")
|
||||
with pytest.raises(HTTPException):
|
||||
card_image_files.uploaded_path("../private")
|
||||
|
||||
|
||||
def fake_db(payload):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
record = SimpleNamespace(payload=payload)
|
||||
result = MagicMock()
|
||||
result.scalar_one.return_value = record
|
||||
result.scalar_one_or_none.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db, record
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_revision_and_audit():
|
||||
defaults, _ = bundled_configuration()
|
||||
payload = {
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
db, record = fake_db(payload)
|
||||
revision = effective_envelope(payload)["revision"]
|
||||
result = await publish_configuration(
|
||||
db, revision, defaults, SimpleNamespace(id=1, username="admin")
|
||||
)
|
||||
assert result["revision"] != revision
|
||||
assert db.add.call_args.args[0].action == "card_images.update"
|
||||
before = deepcopy(record.payload)
|
||||
with pytest.raises(HTTPException) as error:
|
||||
await publish_configuration(db, revision, defaults, SimpleNamespace(id=1, username="admin"))
|
||||
assert error.value.status_code == 409 and record.payload == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configuration_requires_admin_and_validates_http_payload():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
defaults, _ = bundled_configuration()
|
||||
db, _ = fake_db(
|
||||
{
|
||||
"revision": "default",
|
||||
"config": defaults.model_dump(mode="json"),
|
||||
"known_assets": [],
|
||||
"known_rules": [],
|
||||
}
|
||||
)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (await client.get("/configuration")).status_code == 403
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
assert (await client.get("/configuration")).status_code == 200
|
||||
assert (
|
||||
await client.put("/configuration", json={"revision": "x", "config": {"rules": [{}]}})
|
||||
).status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multipart_upload_and_public_file_delivery(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(card_image_files, "ASSET_DIR", tmp_path)
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
role="admin", id=1, username="admin"
|
||||
)
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", (800, 500), "blue").save(buffer, "JPEG")
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
response = await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
asset = response.json()
|
||||
filename = asset["variants"]["640"].rsplit("/", 1)[1]
|
||||
image = await client.get(f"/files/{filename}")
|
||||
assert image.status_code == 200 and image.headers["content-type"] == "image/webp"
|
||||
assert "immutable" in image.headers["cache-control"]
|
||||
assert (
|
||||
await client.post("/upload", files={"file": ("bad.jpg", b"not an image", "image/jpeg")})
|
||||
).status_code == 422
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(role="viewer")
|
||||
assert (
|
||||
await client.post(
|
||||
"/upload", files={"file": ("facility.jpg", buffer.getvalue(), "image/jpeg")}
|
||||
)
|
||||
).status_code == 403
|
||||
|
||||
|
||||
def test_all_object_types_have_a_default_without_overriding_exact_suppression():
|
||||
from typing import get_args
|
||||
from app.schemas.card_images import EntityType
|
||||
|
||||
config, _ = bundled_configuration()
|
||||
for entity_type in get_args(EntityType):
|
||||
result = resolve_image(config, dict(entity_type=entity_type, category="unlisted"))
|
||||
assert result["image"] and result["match"]["value"] == "*"
|
||||
config = config.model_copy(deep=True)
|
||||
config.rules.append(
|
||||
CardImageRule(id="off", entity_type="satellite", scope="object", value="123", suppress=True)
|
||||
)
|
||||
assert resolve_image(config, dict(entity_type="satellite", entity_id="123"))["suppressed"]
|
||||
|
||||
|
||||
def test_upgrade_replaces_old_bundled_source_but_preserves_custom_assets():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["starlink-solar-array"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
|
||||
def starlink():
|
||||
return next(a for a in effective_envelope(payload)["config"]["assets"] if a["id"] == old.id)
|
||||
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["focus_x"] = 20
|
||||
assert starlink()["focus_x"] == 20
|
||||
assert starlink()["url"].endswith("starlink-orbit.webp")
|
||||
payload["config"]["assets"][0]["enabled"] = False
|
||||
assert starlink()["enabled"] is False
|
||||
payload["config"]["assets"][0]["url"] = "https://example.org/custom.jpg"
|
||||
assert starlink()["url"] == "https://example.org/custom.jpg"
|
||||
|
||||
|
||||
def test_compute_categories_and_bgp_get_separate_user_images():
|
||||
config, _ = bundled_configuration()
|
||||
for kind, filename in [
|
||||
("supercomputer", "supercomputer.png"),
|
||||
("gpu_cluster", "gpu-cluster.jpg"),
|
||||
("data_center", "data-center.png"),
|
||||
("cloud_center", "cloud-center.png"),
|
||||
("edge_node", "edge-node.png"),
|
||||
]:
|
||||
result = resolve_image(config, dict(entity_type="compute_center", category=kind))
|
||||
assert result["image"]["url"].endswith(filename)
|
||||
for kind in ("bgp", "bgp_collector"):
|
||||
result = resolve_image(config, dict(entity_type=kind, category=kind))
|
||||
assert result["image"]["url"].endswith(
|
||||
"bgp-event.jpg" if kind == "bgp" else "bgp-collector.png"
|
||||
)
|
||||
|
||||
|
||||
def test_saved_collector_default_migrates_without_changing_user_bindings():
|
||||
from app.services.card_image_defaults import previous_bundled_rules, previous_bundled_assets
|
||||
|
||||
old_rule = previous_bundled_rules()["builtin-network-switches-default-1"]
|
||||
old_asset = previous_bundled_assets()["network-switches"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old_asset.model_dump()], "rules": [old_rule.model_dump()]},
|
||||
"known_assets": [old_asset.id],
|
||||
"known_rules": [old_rule.id],
|
||||
}
|
||||
result = CardImageConfiguration.model_validate(effective_envelope(payload)["config"])
|
||||
assert resolve_image(result, dict(entity_type="bgp_collector", category="bgp_collector"))[
|
||||
"image"
|
||||
]["url"].endswith("bgp-collector.png")
|
||||
payload["config"]["rules"][0]["enabled"] = False
|
||||
result = effective_envelope(payload)["config"]
|
||||
assert not next(rule for rule in result["rules"] if rule["id"] == old_rule.id)["enabled"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_lookup_filters_indexed_identity_and_keeps_actual_category():
|
||||
from app.services.card_image_targets import get_target
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
db = AsyncMock()
|
||||
result = MagicMock()
|
||||
result.mappings.return_value.first.return_value = {
|
||||
"entity_id": "epoch_ai_gpu:site",
|
||||
"name": "Site",
|
||||
"category": "cloud_center",
|
||||
"group": "Operator",
|
||||
"model": "Model",
|
||||
}
|
||||
db.execute.return_value = result
|
||||
context = await get_target(db, "compute_center", "epoch_ai_gpu:site")
|
||||
query = db.execute.call_args.args[0]
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "collected_data.source = 'epoch_ai_gpu'" in sql
|
||||
assert "collected_data.source_id = 'site'" in sql
|
||||
assert context["category"] == "cloud_center" and context["model"] == "Model"
|
||||
|
||||
|
||||
def test_old_gpu_preview_urls_are_replaced_with_latest_variants():
|
||||
from app.services.card_image_defaults import previous_bundled_assets
|
||||
|
||||
old = previous_bundled_assets()["compute-gpu-cluster"][0]
|
||||
payload = {
|
||||
"revision": "old",
|
||||
"config": {"assets": [old.model_dump()], "rules": []},
|
||||
"known_assets": [old.id],
|
||||
"known_rules": [],
|
||||
}
|
||||
result = effective_envelope(payload)["config"]
|
||||
upgraded = next(a for a in result["assets"] if a["id"] == old.id)
|
||||
latest = next(a for a in bundled_configuration()[0].assets if a.id == old.id)
|
||||
assert upgraded["variants"] == latest.variants
|
||||
assert upgraded["url"] == latest.url
|
||||
218
backend/tests/test_collector_downloads.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""Request-level regressions for quota-limited satellite downloads."""
|
||||
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.collectors.celestrak import CelesTrakTLECollector, CELESTRAK_CACHE_TTL_SECONDS
|
||||
from app.services.collectors.downloads import ResumableFileDownloader
|
||||
|
||||
URL = "https://celestrak.example/gp.php?GROUP=active&FORMAT=json"
|
||||
|
||||
|
||||
def setup_collector(monkeypatch, tmp_path, handler):
|
||||
collector = CelesTrakTLECollector()
|
||||
collector._resolved_url = "https://celestrak.example/gp.php"
|
||||
monkeypatch.setattr(collector._downloader, "_cache_dir", tmp_path)
|
||||
monkeypatch.setattr(collector, "update_phase_progress", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.emit_business_log", AsyncMock())
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.asyncio.sleep", AsyncMock())
|
||||
original_client = httpx.AsyncClient
|
||||
monkeypatch.setattr(
|
||||
"app.services.collectors.celestrak.httpx.AsyncClient",
|
||||
lambda **kwargs: original_client(transport=httpx.MockTransport(handler), **kwargs),
|
||||
)
|
||||
return collector
|
||||
|
||||
|
||||
def expire(path):
|
||||
old = time.time() - CELESTRAK_CACHE_TTL_SECONDS - 1
|
||||
os.utime(path, (old, old))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_celestrak_uses_one_get_and_revalidates_after_cache_expiry(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
assert request.method == "GET", "HEAD consumes this upstream's download allowance"
|
||||
return httpx.Response(
|
||||
200, json=[{"NORAD_CAT_ID": len(requests)}], headers={"etag": f'"v{len(requests)}"'}
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 1
|
||||
assert len(requests) == 1
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 2
|
||||
assert requests[1].headers["if-none-match"] == '"v1"'
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_not_updated_reuses_valid_cache_without_retry_storm(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 42
|
||||
assert requests == ["GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bad_download_does_not_replace_last_good_cache(monkeypatch, tmp_path):
|
||||
collector = setup_collector(monkeypatch, tmp_path, lambda request: httpx.Response(200, json=[]))
|
||||
path = collector._downloader.cached_file_path(URL, ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
with pytest.raises(RuntimeError, match="validation failed"):
|
||||
await collector.fetch()
|
||||
assert json.loads(path.read_text()) == [{"NORAD_CAT_ID": 42}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_corrupt_cache_is_not_reused_or_conditionally_requested(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 7}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
path, _, meta = collector._downloader._cache_paths(URL, ".json")
|
||||
path.write_text("[]")
|
||||
meta.write_text('{"etag":"old"}')
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 7
|
||||
assert "if-none-match" not in requests[0].headers
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_conditional_304_keeps_payload_and_refreshes_cache_window(tmp_path):
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False, cache_ttl_seconds=7200
|
||||
)
|
||||
path, _, meta = downloader._cache_paths(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
expire(path)
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
return httpx.Response(304)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
for _ in range(2):
|
||||
await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert len(requests) == 1
|
||||
assert requests[0].headers["if-none-match"] == "v1"
|
||||
assert json.loads(path.read_text())[0]["NORAD_CAT_ID"] == 42
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compressed_get_length_uses_decoded_payload(tmp_path):
|
||||
raw = json.dumps([{"NORAD_CAT_ID": 42}]).encode()
|
||||
body = gzip.compress(raw)
|
||||
downloader = ResumableFileDownloader(
|
||||
cache_namespace="test", cache_root=tmp_path, preflight_head=False
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200,
|
||||
content=body,
|
||||
headers={"content-encoding": "gzip", "content-length": str(len(body))},
|
||||
)
|
||||
)
|
||||
) as client:
|
||||
path = await downloader.download_file(
|
||||
client,
|
||||
URL,
|
||||
extension=".json",
|
||||
validate_existing=CelesTrakTLECollector._validate_json_file,
|
||||
)
|
||||
assert path.read_bytes() == raw
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_collectors_keep_head_and_range_resume(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
final, part, meta = downloader._cache_paths(URL, ".bin")
|
||||
part.parent.mkdir(parents=True)
|
||||
part.write_bytes(b"abc")
|
||||
meta.write_text('{"etag":"v1"}')
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.method == "HEAD":
|
||||
return httpx.Response(
|
||||
200, headers={"etag": "v1", "accept-ranges": "bytes", "content-length": "6"}
|
||||
)
|
||||
assert request.headers["range"] == "bytes=3-"
|
||||
return httpx.Response(206, content=b"def", headers={"content-range": "bytes 3-5/6"})
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".bin")
|
||||
assert [r.method for r in requests] == ["HEAD", "GET"]
|
||||
assert final.read_bytes() == b"abcdef"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fallback_group_refreshes_expired_cache(monkeypatch, tmp_path):
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request)
|
||||
if request.url.params["GROUP"] == "active":
|
||||
return httpx.Response(
|
||||
403, text="GP data has not updated since your last successful download"
|
||||
)
|
||||
return httpx.Response(200, json=[{"NORAD_CAT_ID": 99}])
|
||||
|
||||
collector = setup_collector(monkeypatch, tmp_path, handler)
|
||||
monkeypatch.setattr("app.services.collectors.celestrak.FALLBACK_GROUPS", ("starlink",))
|
||||
path = collector._downloader.cached_file_path(collector._group_url("starlink"), ".json")
|
||||
path.write_text('[{"NORAD_CAT_ID":42}]')
|
||||
expire(path)
|
||||
assert (await collector.fetch())[0]["NORAD_CAT_ID"] == 99
|
||||
assert [r.method for r in requests] == ["GET", "GET"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generic_cache_without_validators_must_refresh(tmp_path):
|
||||
downloader = ResumableFileDownloader(cache_namespace="test", cache_root=tmp_path)
|
||||
path = downloader.cached_file_path(URL, ".json")
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text("[]")
|
||||
requests = []
|
||||
|
||||
def handler(request):
|
||||
requests.append(request.method)
|
||||
return httpx.Response(200, content=b"[1]" if request.method == "GET" else b"")
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
await downloader.download_file(client, URL, extension=".json")
|
||||
assert requests == ["HEAD", "GET"]
|
||||
assert path.read_text() == "[1]"
|
||||
@@ -79,7 +79,8 @@ class TestTOP500Collector:
|
||||
"""Test parsing empty response"""
|
||||
collector = TOP500Collector()
|
||||
result = collector.parse_response("<html><body><table></table></body></html>")
|
||||
assert len(result) > 0
|
||||
assert result == []
|
||||
assert collector.fail_on_empty
|
||||
|
||||
def test_parse_response_single_item(self):
|
||||
"""Test parsing single item response"""
|
||||
@@ -97,9 +98,9 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["source_id"] == "top500_1"
|
||||
assert result[0]["source_id"] == "top500_system_1"
|
||||
assert result[0]["name"] == "Test Supercomputer"
|
||||
assert result[0]["country"] == "USA"
|
||||
assert result[0]["metadata"]["rank"] == 1
|
||||
@@ -113,7 +114,7 @@ class TestTOP500Collector:
|
||||
<tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax</th><th>Rpeak</th><th>Power</th></tr>
|
||||
<tr>
|
||||
<td>1</td>
|
||||
<td>Valid\nVendor\nSite\nUSA</td>
|
||||
<td><a href="/system/2/">Valid</a>\nVendor\nSite\nUSA</td>
|
||||
<td>1000</td>
|
||||
<td>10 PFLOP/s</td>
|
||||
<td>12 PFLOP/s</td>
|
||||
@@ -129,7 +130,7 @@ class TestTOP500Collector:
|
||||
</tr>
|
||||
</table>
|
||||
"""
|
||||
result = collector.parse_response(response)
|
||||
result = collector.parse_response(response, "https://top500.org/lists/top500/2025/11/")
|
||||
assert len(result) == 1
|
||||
assert result[0]["name"] == "Valid"
|
||||
|
||||
|
||||
222
backend/tests/test_compute_configuration.py
Normal file
@@ -0,0 +1,222 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from app.api.v1.compute_intelligence import configuration_admin, get_series, router
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.system_log import AuditLog
|
||||
from app.schemas.compute_configuration import ComputeConfiguration
|
||||
from app.services.compute_configuration import (
|
||||
DEFAULT_CONFIGURATION,
|
||||
default_envelope,
|
||||
publish_configuration,
|
||||
read_configuration,
|
||||
)
|
||||
from app.services.compute_history import compute_history
|
||||
from fastapi import FastAPI, HTTPException, Response
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
|
||||
def default_config():
|
||||
return DEFAULT_CONFIGURATION.model_dump(mode="json")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"mutation",
|
||||
[
|
||||
lambda value: value["observations"][0]["values"].update(CN=101),
|
||||
lambda value: value["observations"][0]["values"].update(CN=-1),
|
||||
lambda value: value["observations"][0]["values"].update(CN=float("nan")),
|
||||
lambda value: value["observations"][0]["values"].update(CN=True),
|
||||
lambda value: value["observations"][0]["values"].update(EUROPE=10),
|
||||
lambda value: value["observations"][0].update(source_id="unknown"),
|
||||
lambda value: value["observations"][0].update(year=2026),
|
||||
lambda value: value["observations"].append(value["observations"][0]),
|
||||
lambda value: value["sources"].append(value["sources"][0]),
|
||||
lambda value: value["innovation"].append(value["innovation"][0]),
|
||||
lambda value: value["sources"][0].update(url="javascript:alert(1)"),
|
||||
lambda value: value["sources"][0].update(url="https://user:pass@example.com/report"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_or_ambiguous_evidence(mutation):
|
||||
payload = default_config()
|
||||
mutation(payload)
|
||||
with pytest.raises(ValidationError):
|
||||
ComputeConfiguration.model_validate(payload)
|
||||
|
||||
|
||||
def test_arbitrary_report_updates_values_sources_and_years_without_code_changes():
|
||||
payload = default_config()
|
||||
payload["sources"].append(
|
||||
{
|
||||
"id": "new-report",
|
||||
"title": "New report",
|
||||
"url": "https://example.com/report",
|
||||
"publisher": "Research group",
|
||||
"report_year": 2026,
|
||||
"locator": "Figure 2",
|
||||
}
|
||||
)
|
||||
payload["observations"].append(
|
||||
{
|
||||
"year": 2025,
|
||||
"metric": "global_share",
|
||||
"source_id": "new-report",
|
||||
"values": {"CN": 25, "US": 50, "JP": 0},
|
||||
}
|
||||
)
|
||||
config = ComputeConfiguration.model_validate(payload)
|
||||
result = compute_history(["CN", "US", "JP", "EUROPE"], config=config)
|
||||
row = next(row for row in result["compute"] if row["year"] == 2025)
|
||||
assert row["values"] == {"CN": 25, "US": 50, "JP": 0, "EUROPE": None}
|
||||
assert result["evidence"][row["evidence_id"]]["title"] == "New report"
|
||||
assert DEFAULT_CONFIGURATION.model_dump(mode="json") == default_config()
|
||||
|
||||
|
||||
def test_zero_bilateral_denominator_is_missing_and_empty_override_stays_empty():
|
||||
payload = default_config()
|
||||
payload["observations"][0]["values"] = {"CN": 0, "US": 0}
|
||||
result = compute_history(
|
||||
["CN", "US"], "bilateral_share", ComputeConfiguration.model_validate(payload)
|
||||
)
|
||||
assert result["compute"][0]["values"] == {"CN": None, "US": None}
|
||||
payload.update(sources=[], observations=[], innovation=[])
|
||||
result = compute_history(["CN"], config=ComputeConfiguration.model_validate(payload))
|
||||
assert all(row["values"]["CN"] is None for row in result["compute"])
|
||||
assert result["evidence"] == {}
|
||||
|
||||
|
||||
def fake_db(record=None):
|
||||
db = AsyncMock()
|
||||
db.add = MagicMock()
|
||||
result = MagicMock()
|
||||
result.scalar_one_or_none.return_value = record
|
||||
result.scalar_one.return_value = record
|
||||
db.execute.return_value = result
|
||||
return db
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_install_uses_defaults_without_writing_database():
|
||||
db = fake_db()
|
||||
assert await read_configuration(db) == default_envelope()
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_series_reads_persisted_override_and_disables_http_cache():
|
||||
envelope = default_envelope()
|
||||
envelope["is_default"] = False
|
||||
envelope["revision"] = "custom"
|
||||
envelope["config"]["observations"][3]["values"]["CN"] = 30
|
||||
response = Response()
|
||||
data = await get_series(
|
||||
response, "CN,US", "global_share", fake_db(SimpleNamespace(payload=envelope))
|
||||
)
|
||||
assert data["compute"][3]["values"]["CN"] == 30
|
||||
assert data["version"] == "custom"
|
||||
assert data["provenance"] == "admin_curated_reports"
|
||||
assert response.headers["Cache-Control"] == "no-store"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_publish_and_audit_are_one_transaction_and_conflicts_do_not_overwrite():
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
actor = SimpleNamespace(id=1, username="admin")
|
||||
old = deepcopy(record.payload)
|
||||
result = await publish_configuration(
|
||||
db,
|
||||
revision=old["revision"],
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=actor,
|
||||
)
|
||||
assert result["revision"] != old["revision"]
|
||||
audit = db.add.call_args.args[0]
|
||||
assert isinstance(audit, AuditLog)
|
||||
assert audit.details == {"before": old, "after": result}
|
||||
assert audit.actor_id == actor.id
|
||||
db.commit.assert_awaited_once()
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await publish_configuration(
|
||||
db, revision=old["revision"], config=DEFAULT_CONFIGURATION, actor=actor
|
||||
)
|
||||
assert failure.value.status_code == 409
|
||||
assert record.payload == result
|
||||
db.rollback.assert_awaited_once()
|
||||
db.commit.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reset_changes_revision_and_restores_bundled_data():
|
||||
record = SimpleNamespace(
|
||||
payload={**default_envelope(), "revision": "custom", "is_default": False}
|
||||
)
|
||||
result = await publish_configuration(
|
||||
fake_db(record),
|
||||
revision="custom",
|
||||
config=DEFAULT_CONFIGURATION,
|
||||
actor=SimpleNamespace(id=1, username="admin"),
|
||||
reset=True,
|
||||
)
|
||||
assert result["is_default"] is True
|
||||
assert result["config"] == default_config()
|
||||
assert result["revision"] != "custom"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("role", ["user", "viewer", "operator"])
|
||||
async def test_non_admin_cannot_configure(role):
|
||||
with pytest.raises(HTTPException) as failure:
|
||||
await configuration_admin(SimpleNamespace(role=role))
|
||||
assert failure.value.status_code == 403
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_http_configuration_auth_validation_and_public_series():
|
||||
app = FastAPI()
|
||||
app.include_router(router)
|
||||
record = SimpleNamespace(payload=default_envelope())
|
||||
db = fake_db(record)
|
||||
app.dependency_overrides[get_db] = lambda: db
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
assert (await client.get("/configuration")).status_code == 401
|
||||
assert (await client.get("/series")).status_code == 200
|
||||
app.dependency_overrides[get_current_user] = lambda: SimpleNamespace(
|
||||
id=1,
|
||||
username="admin",
|
||||
role="admin",
|
||||
)
|
||||
loaded = (await client.get("/configuration")).json()
|
||||
invalid = deepcopy(loaded["config"])
|
||||
invalid["observations"][0]["values"]["CN"] = 200
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": invalid,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
db.commit.assert_not_awaited()
|
||||
response = await client.put(
|
||||
"/configuration",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
"config": loaded["config"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["revision"] != loaded["revision"]
|
||||
assert (
|
||||
await client.post(
|
||||
"/configuration/reset",
|
||||
json={
|
||||
"revision": loaded["revision"],
|
||||
},
|
||||
)
|
||||
).status_code == 409
|
||||
268
backend/tests/test_compute_intelligence.py
Normal file
@@ -0,0 +1,268 @@
|
||||
import csv
|
||||
import io
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from app.api.v1.compute_intelligence import sample_query, selected_countries
|
||||
from app.services.collectors.epoch_ai import EpochAIGPUCollector
|
||||
from app.services.collectors.top500 import TOP500Collector
|
||||
from app.services.compute_history import compute_history
|
||||
|
||||
|
||||
def epoch_csv(**overrides):
|
||||
row = {
|
||||
"Name": "Cluster Alpha",
|
||||
"Country": "United States of America",
|
||||
"Status": "Existing",
|
||||
"First Operational Date": "2023-05-02",
|
||||
"16-bit OP/s (log)": "18",
|
||||
"Location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"latitude": "35",
|
||||
"longitude": "-90",
|
||||
**overrides,
|
||||
}
|
||||
output = io.StringIO()
|
||||
writer = csv.DictWriter(output, fieldnames=list(row))
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
return output.getvalue()
|
||||
|
||||
|
||||
def test_epoch_csv_preserves_precision_date_and_missing_values():
|
||||
collector = EpochAIGPUCollector()
|
||||
row = collector.parse_response(epoch_csv())[0]
|
||||
assert row["value"] == 1e18
|
||||
assert row["unit"] == "FLOP/s"
|
||||
assert row["metadata"]["precision"] == "FP16"
|
||||
assert row["metadata"]["location_precision"] == "precise"
|
||||
assert row["metadata"]["raw_country"] == "United States of America"
|
||||
assert row["reference_date"] == "2023-05-02"
|
||||
missing = collector.parse_response(
|
||||
epoch_csv(**{"16-bit OP/s (log)": "", "First Operational Date": ""})
|
||||
)[0]
|
||||
assert missing["value"] is None
|
||||
assert missing["reference_date"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "country", "expected"),
|
||||
[
|
||||
("United States of America", "United States of America", "country"),
|
||||
("Japan", "Japan", "country"),
|
||||
("Korea (Republic of)", "South Korea", "country"),
|
||||
("Texas", "United States of America", "region"),
|
||||
("Iowa, USA", "United States of America", "region"),
|
||||
("Seoul", "South Korea", "city"),
|
||||
("Paris, France", "France", "city"),
|
||||
("Skolkovo Innovation Center\nMoscow Oblast\nRussia", "Russia", "site"),
|
||||
("Cluster 2, Texas", "United States of America", "site"),
|
||||
("Campus 7, Tokyo", "Japan", "site"),
|
||||
("Building 240, Argonne", "United States of America", "site"),
|
||||
("7", "United States of America", "unknown"),
|
||||
("Jülich Supercomputing Centre\n52428 Jülich, Germany", "Germany", "precise"),
|
||||
("35.380924,139.625008", "Japan", "precise"),
|
||||
('33°24\'28.7"N 112°21\'54.3"W', "United States of America", "precise"),
|
||||
("latitude: 35.0 longitude: 181.0", "United States of America", "unknown"),
|
||||
('99°99\'00"N 200°00\'00"W', "United States of America", "unknown"),
|
||||
("3231 Riverport Rd, Memphis, TN 38109", "United States of America", "precise"),
|
||||
],
|
||||
)
|
||||
def test_epoch_location_precision_follows_location_field(location, country, expected):
|
||||
row = EpochAIGPUCollector().parse_response(
|
||||
epoch_csv(Location=location, Country=country)
|
||||
)[0]
|
||||
|
||||
assert row["metadata"]["location_precision"] == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("location", "expected"),
|
||||
[
|
||||
({"lat": 35.0, "lng": 139.0}, "precise"),
|
||||
({"latitude": 35.0, "longitude": 181.0}, "unknown"),
|
||||
({"lat": 35.0}, "unknown"),
|
||||
({"lng": 139.0}, "unknown"),
|
||||
({"lat": "invalid", "lng": 139.0}, "unknown"),
|
||||
],
|
||||
)
|
||||
def test_epoch_coordinate_object_requires_complete_valid_pair(location, expected):
|
||||
from app.services.epoch_location import assess_epoch_location
|
||||
|
||||
assert assess_epoch_location(location).precision == expected
|
||||
|
||||
|
||||
def test_epoch_anonymous_rows_are_not_geocoded():
|
||||
row = EpochAIGPUCollector().parse_response(epoch_csv(Name="Anonymized 1", Country="China"))[0]
|
||||
assert row["latitude"] is None and row["longitude"] is None
|
||||
assert row["metadata"]["anonymous"] and row["metadata"]["rounded"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
["<html>blocked</html>", "Name,Country\nA,US", epoch_csv(**{"16-bit OP/s (log)": "nan"})],
|
||||
)
|
||||
def test_epoch_invalid_payload_fails_closed(payload):
|
||||
with pytest.raises(ValueError):
|
||||
EpochAIGPUCollector().parse_response(payload)
|
||||
|
||||
|
||||
def test_epoch_duplicate_identity_rejected():
|
||||
payload = epoch_csv()
|
||||
with pytest.raises(ValueError, match="identity collision"):
|
||||
EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
|
||||
|
||||
def test_anonymous_source_rows_preserve_multiplicity_without_claiming_identity():
|
||||
payload = epoch_csv(Name="Anonymized Chinese System", Country="China")
|
||||
rows = EpochAIGPUCollector().parse_response(payload + payload.splitlines()[1] + "\n")
|
||||
assert len(rows) == 2
|
||||
assert rows[0]["source_id"] != rows[1]["source_id"]
|
||||
assert all(row["metadata"]["identity_scope"] == "snapshot_record" for row in rows)
|
||||
|
||||
|
||||
def test_top500_units_and_stable_identity():
|
||||
collector = TOP500Collector()
|
||||
html = '<table id="top500"><tr><th>Rank</th><th>System</th><th>Cores</th><th>Rmax [TFlop/s]</th><th>Rpeak [PFlop/s]</th><th>Power</th></tr><tr><td>{rank}</td><td><a href="/system/123/">Test</a><br>Lab<br>USA</td><td>10</td><td>1,500</td><td>2</td><td>1</td></tr></table>'
|
||||
first = collector.parse_response(
|
||||
html.format(rank=1), "https://top500.org/lists/top500/2025/11/"
|
||||
)[0]
|
||||
second = collector.parse_response(
|
||||
html.format(rank=2), "https://top500.org/lists/top500/2026/06/"
|
||||
)[0]
|
||||
assert first["source_id"] == second["source_id"] == "top500_system_123"
|
||||
assert float(first["value"]) == 1.5e15
|
||||
assert first["metadata"]["rpeak_flops"] == 2e15
|
||||
assert first["reference_date"] == "2025-11-01"
|
||||
assert second["reference_date"] == "2026-06-01"
|
||||
with pytest.raises(ValueError, match="date missing"):
|
||||
collector.parse_response(html.format(rank=1))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_large_source_loss_does_not_create_snapshot():
|
||||
db = AsyncMock()
|
||||
db.scalar.return_value = 500
|
||||
db.add = MagicMock()
|
||||
with pytest.raises(ValueError, match="40%"):
|
||||
await TOP500Collector()._create_snapshot(db, 1, [{}] * 2, datetime.now(UTC))
|
||||
db.add.assert_not_called()
|
||||
db.commit.assert_not_awaited()
|
||||
|
||||
|
||||
def test_history_missing_years_and_denominators():
|
||||
global_data = compute_history(["CN", "US"])
|
||||
assert global_data["compute"][3]["values"] == {"CN": 31, "US": 41}
|
||||
assert global_data["compute"][-1]["values"] == {"CN": None, "US": None}
|
||||
bilateral = compute_history(["CN", "US"], "bilateral_share")
|
||||
assert sum(bilateral["compute"][3]["values"].values()) == 100
|
||||
assert bilateral["denominator"] == "US+CN"
|
||||
absolute = compute_history(["US", "CN"], "absolute")
|
||||
assert absolute["compute"][3]["values"] == {"US": None, "CN": 435}
|
||||
assert global_data["innovation"][-2]["value"] == 17.95
|
||||
assert global_data["innovation"][-1]["value"] is None
|
||||
assert global_data["evidence"]["innovation-2026"]["published_at"] == "2026-07-17"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_snapshot_query_uses_observation_cutoff_not_location():
|
||||
query = await sample_query(AsyncMock(), ["CN"], 2023)
|
||||
sql = str(query.compile(dialect=postgresql.dialect(), compile_kwargs={"literal_binds": True}))
|
||||
assert "2024-01-01" in sql
|
||||
assert "row_number() OVER (PARTITION BY" in sql
|
||||
assert "compute_contract" in sql and "中国" in sql
|
||||
assert "latitude" not in sql and "longitude" not in sql
|
||||
|
||||
|
||||
def test_country_input_is_bounded_and_deduplicated():
|
||||
assert selected_countries("CN,CN,US") == ["CN", "US"]
|
||||
assert selected_countries("CN,US,EUROPE,APAC,OTHER")[-1] == "OTHER"
|
||||
with pytest.raises(HTTPException):
|
||||
selected_countries("invalid")
|
||||
with pytest.raises(HTTPException, match="overlapping"):
|
||||
selected_countries("EUROPE,DE")
|
||||
|
||||
|
||||
def test_catalog_supports_global_participants_without_inventing_history():
|
||||
from app.services.compute_catalog import (
|
||||
CATALOG,
|
||||
COMPARISON_GROUPS,
|
||||
COUNTRIES,
|
||||
source_country_names,
|
||||
source_country_participants,
|
||||
)
|
||||
|
||||
assert {"CN", "US", "JP", "KR", "IN", "DE", "FR", "GB", "AU", "BR", "ZA"}.issubset(
|
||||
{entry["code"] for entry in CATALOG}
|
||||
)
|
||||
assert not {"TW", "HK", "MO", "XK"}.intersection(entry["code"] for entry in CATALOG)
|
||||
owners = source_country_names(["CN"])
|
||||
assert {"中国", "中国(台湾)", "中国(香港)", "中国(澳门)"}.issubset(owners)
|
||||
assert source_country_participants(["CN"])["中国(台湾)"] == "CN"
|
||||
assert [entry["code"] for entry in CATALOG[:5]] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
assert [entry["kind"] for entry in CATALOG[:5]] == [
|
||||
"country",
|
||||
"country",
|
||||
"region",
|
||||
"region",
|
||||
"region",
|
||||
]
|
||||
group_members = [
|
||||
member
|
||||
for _code, _name, _name_en, members, _anchor in COMPARISON_GROUPS
|
||||
for member in members
|
||||
]
|
||||
assert len(group_members) == len(set(group_members))
|
||||
assert set(group_members) == set(COUNTRIES)
|
||||
data = compute_history(["JP", "KR"])
|
||||
assert [row["values"]["JP"] for row in data["compute"][:4]] == [6, 5, 4, 4]
|
||||
assert all(row["values"]["KR"] is None for row in data["compute"])
|
||||
assert all(row["value"] is None for row in data["innovation"])
|
||||
assert len(data["catalog"]) > 30
|
||||
|
||||
|
||||
def test_region_history_requires_complete_same_year_member_coverage():
|
||||
data = compute_history(["CN", "US", "EUROPE", "APAC", "OTHER"])
|
||||
row_2023 = next(row for row in data["compute"] if row["year"] == 2023)
|
||||
assert row_2023["values"]["CN"] == 31
|
||||
assert row_2023["values"]["US"] == 41
|
||||
assert row_2023["values"]["EUROPE"] is None
|
||||
assert row_2023["values"]["APAC"] is None
|
||||
assert row_2023["values"]["OTHER"] is None
|
||||
assert row_2023["coverage"]["EUROPE"]["status"] == "incomplete"
|
||||
assert "DE" in row_2023["coverage"]["EUROPE"]["missing_members"]
|
||||
assert data["default_participants"] == ["CN", "US", "EUROPE", "APAC", "OTHER"]
|
||||
|
||||
|
||||
def test_anonymous_records_never_enter_the_location_repair_queue():
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
record = CollectedData(
|
||||
source="epoch_ai_gpu",
|
||||
source_id="anonymous",
|
||||
name="Anonymous",
|
||||
extra_data={"compute_contract": "v1", "anonymous": True},
|
||||
)
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert result["features"] == []
|
||||
assert result["unresolved"] == []
|
||||
assert result["withheld_count"] == 1
|
||||
|
||||
|
||||
def test_absolute_report_year_mapping_and_bilateral_denominator_ignore_other_countries():
|
||||
absolute = compute_history(["CN"], "absolute")
|
||||
observed = {row["year"]: row for row in absolute["compute"]}
|
||||
assert observed[2022]["values"]["CN"] == 302
|
||||
assert observed[2022]["evidence_id"] == "caict-2023"
|
||||
assert observed[2023]["evidence_id"] == "caict-2024"
|
||||
bilateral = compute_history(["CN", "US", "JP"], "bilateral_share")
|
||||
values = next(row["values"] for row in bilateral["compute"] if row["year"] == 2023)
|
||||
assert values == {"CN": 43.06, "US": 56.94, "JP": None}
|
||||
for row in absolute["compute"]:
|
||||
if row["evidence_id"]:
|
||||
report = absolute["evidence"][row["evidence_id"]]
|
||||
assert report["url"].startswith("https://www.caict.ac.cn/")
|
||||
18
backend/tests/test_compute_verified_sites.py
Normal file
@@ -0,0 +1,18 @@
|
||||
from app.services.compute_center_verified_sites import get_verified_compute_site
|
||||
|
||||
|
||||
def test_registry_requires_exact_facility_identity_not_operator_name():
|
||||
assert get_verified_compute_site("epoch_ai_gpu", "NVIDIA") is None
|
||||
assert get_verified_compute_site("top500", "epoch_argonne_nl_sophia__formerly_theta_") is None
|
||||
|
||||
|
||||
def test_official_campus_anchors_preserve_evidence_and_site_level_precision():
|
||||
site = get_verified_compute_site("epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou")
|
||||
assert site["city"] == "Yokohama"
|
||||
assert site["precision"] == "site"
|
||||
assert site["coordinate_basis"] == "official_access_map_center"
|
||||
assert site["sources"]
|
||||
site["latitude"] = 0
|
||||
assert get_verified_compute_site(
|
||||
"epoch_ai_gpu", "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
)["latitude"] != 0
|
||||
@@ -1,4 +1,5 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -8,6 +9,10 @@ from app.services import earth_boundaries
|
||||
|
||||
|
||||
def write_geojson(path, name="Test"):
|
||||
names = [name]
|
||||
if name == "earth_admin0_boundaries":
|
||||
names = ["China", "India", "Japan", "Palestine", "Taiwan", "Hong Kong", "Macao", "Serbia", "Kosovo"]
|
||||
codes = {"China": "CHN", "India": "IND", "Japan": "JPN", "Palestine": "PSE", "Taiwan": "TWN", "Hong Kong": "HKG", "Macao": "MAC", "Serbia": "SRB", "Kosovo": "KOS"}
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
@@ -15,12 +20,17 @@ def write_geojson(path, name="Test"):
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"properties": {"name": name},
|
||||
"properties": {
|
||||
"NAME": f"source renamed {item}" if item in {"Taiwan", "Kosovo"} else item,
|
||||
"ISO_A3": "-99" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
"ADM0_A3": "XKS" if item == "Kosovo" else codes.get(item, "TEST"),
|
||||
},
|
||||
"geometry": {
|
||||
"type": "Polygon",
|
||||
"coordinates": [[[-1, -1], [1, -1], [1, 1], [-1, 1], [-1, -1]]],
|
||||
},
|
||||
}
|
||||
for item in names
|
||||
],
|
||||
}
|
||||
),
|
||||
@@ -37,9 +47,14 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
config = repo / "config/earth-boundary-sources.local.json"
|
||||
example = repo / "config/earth-boundary-sources.example.json"
|
||||
policy = repo / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
geometry = repo / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
for path in (source_dir, boundary_dir, pmtiles.parent, legacy.parent, config.parent):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
policy.write_text('{"productionTileFormat":"pmtiles+mvt"}\n', encoding="utf-8")
|
||||
source_policy = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy.write_text(source_policy.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
geometry.parent.mkdir(parents=True, exist_ok=True)
|
||||
source_geometry = Path(__file__).resolve().parents[2] / "resources/earth/boundary-pov-geometry.china-v1.json"
|
||||
geometry.write_bytes(source_geometry.read_bytes())
|
||||
example.write_text('{"collectorConfigs":{}}\n', encoding="utf-8")
|
||||
monkeypatch.setattr(earth_boundaries, "REPO_ROOT", repo)
|
||||
monkeypatch.setattr(earth_boundaries, "SOURCE_OUTPUT_DIR", source_dir)
|
||||
@@ -49,10 +64,11 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_OUTPUT_DIR", boundary_dir)
|
||||
monkeypatch.setattr(earth_boundaries, "BOUNDARY_MANIFEST_PATH", boundary_dir / "manifest.json")
|
||||
monkeypatch.setattr(earth_boundaries, "PMTILES_ARTIFACT_PATH", pmtiles)
|
||||
monkeypatch.setattr(earth_boundaries, "LEGACY_GEOJSON_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "POLICY_FALLBACK_PATH", legacy)
|
||||
monkeypatch.setattr(earth_boundaries, "LOCAL_CONFIG_PATH", config)
|
||||
monkeypatch.setattr(earth_boundaries, "EXAMPLE_CONFIG_PATH", example)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_POLICY_PATH", policy)
|
||||
monkeypatch.setattr(earth_boundaries, "POV_GEOMETRY_PATH", geometry)
|
||||
return {
|
||||
"repo": repo,
|
||||
"config": config,
|
||||
@@ -62,27 +78,73 @@ def patch_paths(monkeypatch, tmp_path):
|
||||
}
|
||||
|
||||
|
||||
def write_fallback(path, policy_path, geometry_path):
|
||||
policy = earth_boundaries._read_json(policy_path)
|
||||
geometry = earth_boundaries._read_json(geometry_path)
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(policy),
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(geometry),
|
||||
"reviewedGeometrySourceCommit": geometry["sourceCommit"],
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
}
|
||||
path.write_text(json.dumps({"type": "FeatureCollection", "features": [], "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
|
||||
def test_boundary_status_uses_legacy_provider_when_pmtiles_missing(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_fallback(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "legacy-geojson"
|
||||
assert status["fallback_available"] is True
|
||||
assert status["high_precision_ready"] is False
|
||||
assert status["provider"] == "missing"
|
||||
assert status["fallback_available"] is False
|
||||
|
||||
|
||||
def test_boundary_status_prefers_high_precision_when_manifest_and_pmtiles_exist(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_geojson(paths["legacy"])
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt"}\n', encoding="utf-8")
|
||||
(paths["manifest"].parent / "hover-index.geojson").write_text("{}", encoding="utf-8")
|
||||
coverage = {
|
||||
"profile": "china-pov-v1",
|
||||
"sha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_POLICY_PATH)),
|
||||
"applied": True,
|
||||
"appliedRuleIds": ["china-taiwan-penghu", "china-hong-kong", "china-macao", "kosovo"],
|
||||
"reviewedGeometrySourceCommit": earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)["sourceCommit"],
|
||||
"geometrySha256": earth_boundaries._stable_json_hash(earth_boundaries._read_json(earth_boundaries.POV_GEOMETRY_PATH)),
|
||||
}
|
||||
paths["manifest"].write_text(json.dumps({"tileProvider": "pmtiles-mvt", "hoverIndex": "hover-index.geojson", "povPolicy": coverage}), encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "pmtiles-mvt"
|
||||
assert status["high_precision_ready"] is True
|
||||
assert status["policy_publishable"] is False
|
||||
|
||||
|
||||
def test_boundary_status_rejects_unverified_high_precision(monkeypatch, tmp_path):
|
||||
paths = patch_paths(monkeypatch, tmp_path)
|
||||
write_fallback(paths["legacy"], earth_boundaries.POV_POLICY_PATH, earth_boundaries.POV_GEOMETRY_PATH)
|
||||
paths["pmtiles"].write_bytes(b"pmtiles")
|
||||
paths["manifest"].write_text('{"tileProvider":"pmtiles-mvt","povPolicy":{"applied":false}}', encoding="utf-8")
|
||||
|
||||
status = earth_boundaries.get_boundary_status()
|
||||
|
||||
assert status["provider"] == "policy-geojson-fallback"
|
||||
assert status["high_precision_ready"] is False
|
||||
|
||||
|
||||
def test_save_boundary_config_writes_local_config(monkeypatch, tmp_path):
|
||||
@@ -130,6 +192,22 @@ async def test_build_reports_missing_tools_after_source_artifacts(monkeypatch, t
|
||||
assert status["high_precision_ready"] is True
|
||||
assert (paths["repo"] / "data/earth-boundary-sources/manifest.json").exists()
|
||||
assert paths["manifest"].exists()
|
||||
manifest = json.loads(paths["manifest"].read_text(encoding="utf-8"))
|
||||
assert manifest["povPolicy"]["applied"] is True
|
||||
assert manifest["povPolicy"]["coverage"] == "partial"
|
||||
assert manifest["povPolicy"]["publicationReady"] is False
|
||||
assert "china-south-china-sea-islands" in manifest["povPolicy"]["unresolvedRuleIds"]
|
||||
hover = json.loads((paths["manifest"].parent / manifest["hoverIndex"]).read_text(encoding="utf-8"))
|
||||
by_rule = {
|
||||
feature["properties"].get("PLANET_POLICY_RULE"): feature
|
||||
for feature in hover["features"]
|
||||
}
|
||||
assert by_rule["china-hong-kong"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert by_rule["china-macao"]["properties"]["ISO_A3"] == "CHN"
|
||||
assert all(
|
||||
feature["properties"].get("ADM0_A3") not in {"TWN", "KOS", "XKS"}
|
||||
for feature in hover["features"]
|
||||
)
|
||||
assert any(phase == "download" for _progress, phase, _message in progress_events)
|
||||
|
||||
|
||||
@@ -175,3 +253,23 @@ def test_earth_boundary_collectors_are_not_registered_as_datasources():
|
||||
assert removed.isdisjoint(DEFAULT_DATASOURCES)
|
||||
for source in removed:
|
||||
assert collector_registry.get(source) is None
|
||||
|
||||
|
||||
def test_static_policy_fallback_contains_reviewed_jeju_polygon():
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
path = repo / "frontend/public/earth/data/countries-admin0.china-pov-v1.geojson"
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
matches = [
|
||||
feature for feature in payload["features"]
|
||||
if feature["properties"].get("PLANET_GEOMETRY_SUPPLEMENT") == "jeju-island"
|
||||
]
|
||||
|
||||
assert len(matches) == 1
|
||||
jeju = matches[0]
|
||||
assert jeju["properties"]["ISO_A3"] == "KOR"
|
||||
assert jeju["properties"]["PLANET_SOURCE_NE_ID"] == 1159320985
|
||||
assert payload["povPolicy"]["geometrySupplement"]["jejuComponent"] == 3
|
||||
ring = jeju["geometry"]["coordinates"][0]
|
||||
assert len(ring) == 61
|
||||
assert min(point[0] for point in ring) < 126.53 < max(point[0] for point in ring)
|
||||
assert min(point[1] for point in ring) < 33.36 < max(point[1] for point in ring)
|
||||
|
||||
@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
assert vessel_update["refresh_strategy"] == "reload"
|
||||
|
||||
|
||||
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
|
||||
from app.services.earth_db_change_listener import PendingEarthDbChange
|
||||
|
||||
for source in ("aisstream_vessels", "barentswatch_vessels"):
|
||||
update = build_earth_update_from_db_payload({
|
||||
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
|
||||
})
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
|
||||
pending.add({"operation": "UPDATE"})
|
||||
pending.add({"operation": "DELETE"})
|
||||
assert pending.refresh_strategy == "reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
|
||||
41
backend/tests/test_earth_geography_policy.py
Normal file
@@ -0,0 +1,41 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.earth_geography_policy import display_country, display_iso_a3
|
||||
|
||||
|
||||
def test_missing_deployment_policy_has_an_actionable_error(monkeypatch, tmp_path):
|
||||
from app.core import earth_geography_policy as policy
|
||||
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
monkeypatch.setattr(policy, "_POLICY_PATH", tmp_path / "missing-policy.json")
|
||||
try:
|
||||
with pytest.raises(FileNotFoundError, match="EARTH_DISPLAY_POLICY_MISSING"):
|
||||
policy.display_country("China")
|
||||
finally:
|
||||
policy._policy_country_by_code.cache_clear()
|
||||
|
||||
|
||||
def test_display_country_groups_china_regions_and_kosovo() -> None:
|
||||
for source in ("Taiwan", "TWN", "Hong Kong", "HKG", "Macao", "MAC", "中国(台湾)"):
|
||||
assert display_country(source) == "中国"
|
||||
assert display_iso_a3(source) == "CHN"
|
||||
for source in ("Kosovo", "XK", "SRB"):
|
||||
assert display_country(source) == "塞尔维亚"
|
||||
assert display_iso_a3(source) == "SRB"
|
||||
|
||||
|
||||
def test_display_country_keeps_other_country_and_rejects_unknown() -> None:
|
||||
assert display_country("Japan") == "日本"
|
||||
assert display_iso_a3("Japan") == "JPN"
|
||||
assert display_country("unexpected non-country") is None
|
||||
|
||||
|
||||
def test_country_projection_agrees_with_versioned_boundary_rules() -> None:
|
||||
path = Path(__file__).resolve().parents[2] / "config/earth-boundary-pov-policy.china-v1.json"
|
||||
policy = json.loads(path.read_text(encoding="utf-8"))
|
||||
for rule in policy["rules"]:
|
||||
for code in rule.get("sourceIsoA3", []):
|
||||
assert display_iso_a3(code) == rule.get("targetIsoA3", rule.get("parentIsoA3"))
|
||||
@@ -384,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
|
||||
assert items[0].content_language == "zh-CN"
|
||||
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == ""
|
||||
|
||||
items[0].localizations["en-US"] = {
|
||||
"title": "Chinese e-commerce platform reports quarterly growth",
|
||||
"summary": "The platform said cross-border orders rose year over year.",
|
||||
}
|
||||
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
|
||||
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
|
||||
|
||||
|
||||
def test_default_news_sources_include_business_and_ecommerce_sources():
|
||||
|
||||
86
backend/tests/test_earth_news_encoding.py
Normal file
@@ -0,0 +1,86 @@
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services.earth_news import (
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
ParsedNewsItem,
|
||||
_fetch_single_feed_url,
|
||||
_serialize_item,
|
||||
)
|
||||
|
||||
|
||||
async def fetch_feed(
|
||||
content: bytes, content_type: str = "text/xml"
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None, dict[str, Any]]:
|
||||
source = NewsFeedSource(
|
||||
id="ebrun",
|
||||
name="亿邦动力",
|
||||
region="asia-pacific",
|
||||
feed_url="https://example.com/rss.xml",
|
||||
homepage_url="https://example.com/",
|
||||
)
|
||||
feed = NewsFeedEndpoint(id="b2c", name="B2C", url=source.feed_url)
|
||||
transport = httpx.MockTransport(
|
||||
lambda request: httpx.Response(
|
||||
200, content=content, headers={"content-type": content_type}
|
||||
)
|
||||
)
|
||||
async with httpx.AsyncClient(transport=transport) as client:
|
||||
return await _fetch_single_feed_url(client, source, feed)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("encoding", "declaration", "content_type"),
|
||||
[
|
||||
("gbk", 'encoding="GBK"', "text/xml"),
|
||||
("gb2312", "encoding='GB2312'", "application/rss+xml"),
|
||||
("gb18030", 'encoding="GB18030"', "text/xml"),
|
||||
("gbk", "", "text/xml; charset=gbk"),
|
||||
("utf-8", 'encoding="UTF-8"', "application/rss+xml"),
|
||||
("utf-8", "", "text/xml"),
|
||||
("utf-8-sig", 'encoding="UTF-8"', "text/xml"),
|
||||
("utf-16", 'encoding="UTF-16"', "text/xml"),
|
||||
("utf-32", 'encoding="UTF-32"', "text/xml"),
|
||||
],
|
||||
)
|
||||
async def test_feed_encoding_preserves_chinese_in_display(
|
||||
encoding: str, declaration: str, content_type: str
|
||||
):
|
||||
title = "618电商战报:零售额增长266%"
|
||||
summary = "亿邦动力:跨境电商订单增长,物流效率提升。"
|
||||
xml = f'''<?xml version="1.0" {declaration}?>
|
||||
<rss version="2.0"><channel><item>
|
||||
<title><![CDATA[{title}]]></title>
|
||||
<description><![CDATA[<p>{summary}</p>]]></description>
|
||||
<link>https://example.com/story</link>
|
||||
</item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode(encoding), content_type)
|
||||
|
||||
assert error is None
|
||||
assert health["ok"] is True
|
||||
assert len(items) == 1
|
||||
assert items[0].title == title
|
||||
assert items[0].summary == summary
|
||||
payload = _serialize_item(items[0], active_region="asia-pacific", locale="zh-CN")
|
||||
assert payload["display_title"] == title
|
||||
assert payload["display_summary"] == summary
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("encoding", ["UTF-8", "unknown-encoding"])
|
||||
async def test_invalid_feed_encoding_is_reported_instead_of_storing_garbled_text(encoding: str):
|
||||
xml = f'''<?xml version="1.0" encoding="{encoding}"?>
|
||||
<rss><channel><item><title>电商新闻</title>
|
||||
<link>https://example.com/story</link></item></channel></rss>'''
|
||||
|
||||
_, items, error, health = await fetch_feed(xml.encode("gbk"))
|
||||
|
||||
assert items == []
|
||||
assert error
|
||||
assert health["ok"] is False
|
||||
assert health["status"] == "format_error"
|
||||
219
backend/tests/test_llm_provider_catalog.py
Normal file
@@ -0,0 +1,219 @@
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services import llm_provider_catalog as catalog
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
def mock_http(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"provider,path,auth",
|
||||
[
|
||||
("minimax", "/anthropic/v1/models", "x-api-key"),
|
||||
("anthropic", "/v1/models", "x-api-key"),
|
||||
("openai", "/v1/models", "authorization"),
|
||||
("deepseek", "/v1/models", "authorization"),
|
||||
("alibaba", "/api/v1/models", "authorization"),
|
||||
("moonshotai", "/v1/models", "authorization"),
|
||||
("openrouter", "/api/v1/models", "authorization"),
|
||||
("opencode-go", "/zen/go/v1/models", "authorization"),
|
||||
("ollama", "/api/tags", "authorization"),
|
||||
],
|
||||
)
|
||||
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
|
||||
preset = catalog.get_fallback_llm_provider_preset(provider)
|
||||
|
||||
def handle(request):
|
||||
assert request.url.path == path
|
||||
assert request.url.host == httpx.URL(preset["base_url"]).host
|
||||
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
|
||||
if provider == "alibaba":
|
||||
assert request.url.params["capabilities"] == "TG"
|
||||
return httpx.Response(
|
||||
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
|
||||
)
|
||||
if provider == "ollama":
|
||||
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
|
||||
return httpx.Response(200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
|
||||
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
|
||||
assert result["base_url"] == preset["base_url"]
|
||||
assert "test-key" not in str(result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,base,api,path",
|
||||
[
|
||||
(
|
||||
"minimax",
|
||||
"https://api.minimax.io/anthropic/v1/",
|
||||
"anthropic-messages",
|
||||
"/anthropic/v1/models",
|
||||
),
|
||||
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
|
||||
(
|
||||
"alibaba",
|
||||
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
(
|
||||
"alibaba",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
|
||||
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
|
||||
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
|
||||
],
|
||||
)
|
||||
def test_urls_preserve_region_and_gateway(provider, base, api, path):
|
||||
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
|
||||
assert url.host == httpx.URL(base).host
|
||||
assert url.path == path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_pagination_and_release_order(monkeypatch):
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request.url.params.get("after_id"))
|
||||
if len(seen) == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
|
||||
"has_more": True,
|
||||
"last_id": "old",
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
|
||||
assert seen == [None, "old"]
|
||||
assert result["models"] == ["new", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashscope_pagination(monkeypatch):
|
||||
def handle(request):
|
||||
page = int(request.url.params["page_no"])
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"output": {
|
||||
"total": 2,
|
||||
"models": [
|
||||
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
|
||||
assert result["models"] == ["page-2", "page-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{},
|
||||
{"data": []},
|
||||
{"data": [None]},
|
||||
{"data": [{}]},
|
||||
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
|
||||
],
|
||||
)
|
||||
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
with pytest.raises(discovery.LLMProviderCatalogError):
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
|
||||
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
|
||||
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
|
||||
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
|
||||
assert len(result["models"]) == 140
|
||||
assert result["models"][0] == "model-139"
|
||||
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_ollama_catalog_is_valid(monkeypatch):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
|
||||
result = await catalog.refresh_llm_provider_preset("ollama")
|
||||
assert result["models"] == []
|
||||
assert result["model"] == ""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opencode_documented_protocols(monkeypatch):
|
||||
models = [
|
||||
"minimax-m3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"grok-4.6",
|
||||
"muse-spark-1.3-contributor",
|
||||
"kimi-k3",
|
||||
]
|
||||
mock_http(
|
||||
monkeypatch,
|
||||
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
|
||||
)
|
||||
result = await catalog.refresh_llm_provider_preset("opencode-go")
|
||||
assert list(result["model_provider_apis"].values()) == [
|
||||
"anthropic-messages",
|
||||
"anthropic-messages",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-completions",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_transient_error_only(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def handle(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 2
|
||||
calls.clear()
|
||||
|
||||
def unauthorized(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(401, text="secret-upstream-text")
|
||||
|
||||
mock_http(monkeypatch, unauthorized)
|
||||
with pytest.raises(httpx.HTTPStatusError) as error:
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 1
|
||||
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)
|
||||
165
backend/tests/test_llm_provider_settings.py
Normal file
@@ -0,0 +1,165 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.v1 import settings as api
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings_db(monkeypatch, tmp_path):
|
||||
# Real SQL reads, commits and reloads; do not mock the settings helpers.
|
||||
engine = create_engine("sqlite://")
|
||||
SystemSetting.__table__.create(engine)
|
||||
session = Session(engine)
|
||||
db = SimpleNamespace(
|
||||
execute=AsyncMock(side_effect=session.execute),
|
||||
add=session.add,
|
||||
commit=AsyncMock(side_effect=session.commit),
|
||||
refresh=AsyncMock(side_effect=session.refresh),
|
||||
)
|
||||
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
|
||||
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
|
||||
yield db
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def use_upstream(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_lists_all_builtin_presets(settings_db):
|
||||
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
|
||||
assert len(result["data"]) == 9
|
||||
assert all(row["models"] for row in result["data"])
|
||||
assert "MiniMax-M3" in result["data"][0]["models"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
|
||||
runtime = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"service_token": "private-service-token",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"base_url": "https://saved.test/anthropic",
|
||||
"model": "old",
|
||||
"api_key": "saved-secret",
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
|
||||
before = await api.get_setting_payload(settings_db, "external_integrations")
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request)
|
||||
assert str(request.url) == "https://draft.test/anthropic/v1/models"
|
||||
assert request.headers["x-api-key"] == "draft-secret"
|
||||
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
|
||||
|
||||
use_upstream(monkeypatch, handle)
|
||||
draft = api.AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
base_url="https://draft.test/anthropic",
|
||||
model="unsaved-model",
|
||||
api_key="draft-secret",
|
||||
service_token="unsaved-service-token",
|
||||
)
|
||||
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
|
||||
result = await api.get_ai_provider_presets(None, settings_db)
|
||||
preset = next(row for row in result["data"] if row["provider"] == "minimax")
|
||||
assert len(seen) == 1
|
||||
assert preset["models"] == ["MiniMax-M3", "old"]
|
||||
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
|
||||
assert await api.get_setting_payload(settings_db, "external_integrations") == before
|
||||
assert "secret" not in str(result)
|
||||
assert "private-service-token" not in str(result)
|
||||
assert draft.model == "unsaved-model"
|
||||
stored = await settings_db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
|
||||
)
|
||||
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
|
||||
settings_db, monkeypatch
|
||||
):
|
||||
await api.save_setting_payload(
|
||||
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
|
||||
)
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
|
||||
commits = settings_db.commit.await_count
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
|
||||
assert error.value.status_code == 502
|
||||
assert "private-upstream-secret" not in error.value.detail
|
||||
assert settings_db.commit.await_count == commits
|
||||
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
|
||||
"models"
|
||||
] == ["saved-model"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
|
||||
def fail_request(request):
|
||||
pytest.fail("must not query a private catalog without credentials")
|
||||
|
||||
use_upstream(monkeypatch, fail_request)
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("minimax", None, settings_db)
|
||||
assert error.value.status_code == 400
|
||||
assert "API Key" in error.value.detail
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
|
||||
)
|
||||
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
|
||||
)
|
||||
assert result["success"] is False
|
||||
assert result["connected"] is False
|
||||
assert "private-upstream-secret" not in result["message"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
|
||||
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
|
||||
use_upstream(
|
||||
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
|
||||
)
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "model": "future-model", "api_key": "test-key"}, 5
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["url"].endswith("/anthropic/v1/models")
|
||||
141
backend/tests/test_provider_protocols.py
Normal file
@@ -0,0 +1,141 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
with patch(
|
||||
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
|
||||
return_value={},
|
||||
):
|
||||
from aiprovider.provider_service import ProviderService
|
||||
from aiprovider.schemas import SituationalAnalysisRequest
|
||||
from app.api.v1.settings import _runtime_config_from_ai_payload
|
||||
|
||||
|
||||
def test_custom_openai_gateway_keeps_its_configured_protocol():
|
||||
config = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "openai",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"base_url": "https://gateway.test/v1",
|
||||
"provider_api": "openai-completions",
|
||||
"model": "gpt-6-astra",
|
||||
},
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService({**config, "api_key": "test-key"})
|
||||
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"model,path",
|
||||
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
|
||||
)
|
||||
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
|
||||
llm = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "opencode-go",
|
||||
"providers": {
|
||||
"opencode-go": {
|
||||
"model": model,
|
||||
"api_key": "test-key",
|
||||
"model_provider_apis": {model: "openai-completions"},
|
||||
}
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService(llm)
|
||||
seen = []
|
||||
|
||||
async def post(**request):
|
||||
seen.append(request)
|
||||
assert request["path"] == path
|
||||
assert request["request_body"]["model"] == model
|
||||
if path == "/responses":
|
||||
assert request["request_body"]["store"] is False
|
||||
assert "max_tokens" not in request["request_body"]
|
||||
return {
|
||||
"output": [
|
||||
{"type": "reasoning", "summary": [{"text": "reason"}]},
|
||||
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
|
||||
]
|
||||
}
|
||||
return {"content": [{"type": "text", "text": "OK"}]}
|
||||
|
||||
monkeypatch.setattr(service, "_post", post)
|
||||
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
|
||||
assert len(seen) == 1
|
||||
assert result.content == "OK"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
seen.append(str(request.url))
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": base,
|
||||
"api_key": "test-key",
|
||||
"model": "MiniMax-M3",
|
||||
}
|
||||
)
|
||||
result = await service.analyze(
|
||||
SituationalAnalysisRequest(title="test", objective="reply OK")
|
||||
)
|
||||
assert result.content == "OK"
|
||||
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
import json
|
||||
|
||||
seen.append(request)
|
||||
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
|
||||
assert request.headers["user-agent"] == "Planet/1.0"
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"api_key": "test-key",
|
||||
"model": "minimax-m3",
|
||||
}
|
||||
)
|
||||
await service.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="test",
|
||||
objective="reply OK",
|
||||
thinking={"type": "enabled"},
|
||||
context={"session_id": session_id},
|
||||
)
|
||||
)
|
||||
session_headers = [request.headers["x-opencode-session"] for request in seen]
|
||||
assert session_headers[0] == session_headers[1]
|
||||
assert session_headers[0] != session_headers[2]
|
||||
160
backend/tests/test_tv_catalog.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import normalize_tv_settings
|
||||
|
||||
|
||||
class CatalogSession:
|
||||
"""Execute the real catalog queries against an isolated SQLite database."""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
async def execute(self, query):
|
||||
return self.session.execute(query)
|
||||
|
||||
async def scalar(self, query):
|
||||
return self.session.scalar(query)
|
||||
|
||||
async def scalars(self, query):
|
||||
return self.session.scalars(query)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def catalog_db():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def register_functions(connection, _record):
|
||||
connection.create_function(
|
||||
"concat_ws",
|
||||
-1,
|
||||
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
|
||||
)
|
||||
|
||||
Base.metadata.create_all(
|
||||
engine,
|
||||
tables=[
|
||||
CollectionTask.__table__,
|
||||
DataSnapshot.__table__,
|
||||
CollectedData.__table__,
|
||||
SystemSetting.__table__,
|
||||
],
|
||||
)
|
||||
with Session(engine) as session:
|
||||
for index in range(135):
|
||||
session.add(
|
||||
CollectedData(
|
||||
source="news_live_streams",
|
||||
source_id=f"channel-{index:03}",
|
||||
data_type="news_live_stream",
|
||||
name=f"Channel {index:03}",
|
||||
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
extra_data={
|
||||
"stream_url": "https://example.invalid/live.m3u8",
|
||||
"region": "Canada",
|
||||
},
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
yield CatalogSession(session)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
|
||||
first = await get_tv_catalog_page(catalog_db, limit=50)
|
||||
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
|
||||
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
|
||||
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
|
||||
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
|
||||
assert len(set(ids)) == first["source_count"] == 145
|
||||
assert ids[-1] == "channel-134"
|
||||
assert third["next_offset"] is None and not third["has_more"]
|
||||
beyond = await get_tv_catalog_page(catalog_db, offset=200)
|
||||
assert beyond["sources"] == [] and not beyond["has_more"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
assert payload["total"] == 1 and payload["source_count"] == 145
|
||||
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
|
||||
assert payload["selected_source"]["id"] == "channel-134"
|
||||
assert "channel-134" not in [source["id"] for source in payload["sources"]]
|
||||
assert payload["default_source_id"] == "aljazeera-mubasher"
|
||||
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
|
||||
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
|
||||
for name, values in [
|
||||
("Disabled", {"extra_data": {"is_enabled": False}}),
|
||||
("Historical", {"is_current": False}),
|
||||
("Invalid", {"is_valid": 0}),
|
||||
("Deleted", {"deleted_at": datetime.now(UTC)}),
|
||||
("Replacement", {"source_id": "channel-134"}),
|
||||
]:
|
||||
record = dict(
|
||||
source="news_live_streams",
|
||||
source_id=name,
|
||||
name=name,
|
||||
data_type="news_live_stream",
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
)
|
||||
catalog_db.session.add(CollectedData(**{**record, **values}))
|
||||
catalog_db.session.flush()
|
||||
payload = await get_tv_catalog_page(catalog_db, q="replacement")
|
||||
assert payload["source_count"] == 145
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
|
||||
|
||||
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
|
||||
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
|
||||
)
|
||||
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
|
||||
monkeypatch, config, expected
|
||||
):
|
||||
collector = NewsLiveStreamsCollector()
|
||||
channels = [
|
||||
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
|
||||
]
|
||||
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
|
||||
streams = [
|
||||
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
|
||||
for channel in channels
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
|
||||
)
|
||||
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
|
||||
assert len(records) == expected
|
||||
assert all(record["source_id"] != "sport" for record in records)
|
||||
37
backend/tests/test_tv_playback.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from app.services.tv_streams import (
|
||||
CCTV4_HOMEPAGE,
|
||||
CCTV4_STREAM,
|
||||
UNSUPPORTED_CCTV4_STREAM,
|
||||
normalize_tv_settings,
|
||||
normalize_tv_source,
|
||||
)
|
||||
|
||||
|
||||
def test_default_keeps_aljazeera_and_does_not_offer_cdrm_as_plain_hls():
|
||||
settings = normalize_tv_settings(None)
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
cctv = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert cctv["source_type"] == "hls"
|
||||
assert cctv["stream_url"] == CCTV4_STREAM
|
||||
assert "美洲版" in cctv["name"]
|
||||
assert cctv["use_proxy"] is False
|
||||
assert cctv["homepage_url"] == CCTV4_HOMEPAGE
|
||||
|
||||
|
||||
def test_saved_legacy_default_is_corrected_without_mutating_input():
|
||||
saved = {"id": "cctv4", "source_type": "hls", "stream_url": UNSUPPORTED_CCTV4_STREAM}
|
||||
source = normalize_tv_source(saved)
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["stream_url"] == CCTV4_STREAM
|
||||
assert source["use_proxy"] is False
|
||||
assert saved["stream_url"] == UNSUPPORTED_CCTV4_STREAM
|
||||
|
||||
|
||||
def test_custom_cctv_stream_and_fallback_preference_are_preserved():
|
||||
custom = {"id": "cctv4", "source_type": "hls", "stream_url": "https://example.com/licensed.m3u8"}
|
||||
settings = normalize_tv_settings({"auto_fallback": False, "sources": [custom]})
|
||||
source = next(source for source in settings["sources"] if source["id"] == "cctv4")
|
||||
assert source["stream_url"] == custom["stream_url"]
|
||||
assert source["source_type"] == "hls"
|
||||
assert source["use_proxy"] is True
|
||||
assert settings["auto_fallback"] is False
|
||||
@@ -76,6 +76,7 @@ def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
@@ -168,6 +169,241 @@ def test_compute_center_source_coordinates_win_over_stored_location():
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_epoch_country_level_coordinates_are_unresolved():
|
||||
record = _build_record(
|
||||
record_id=33,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Meta 100k",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=39.381266,
|
||||
longitude=-97.922211,
|
||||
metadata={
|
||||
"location": "United States of America",
|
||||
"raw_country": "United States of America",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
assert payload["features"] == []
|
||||
assert len(payload["unresolved"]) == 1
|
||||
assert payload["unresolved"][0]["source_id"] == "epoch_ai_gpu-33"
|
||||
assert "source_coordinates:country" in payload["unresolved"][0]["attempted_queries"]
|
||||
|
||||
|
||||
def test_epoch_city_coordinates_are_not_labeled_precise():
|
||||
record = _build_record(
|
||||
record_id=34,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Seoul Cluster",
|
||||
country="South Korea",
|
||||
city="",
|
||||
latitude=37.5665,
|
||||
longitude=126.978,
|
||||
metadata={"location": "Seoul", "raw_country": "South Korea"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.is_resolved
|
||||
assert result.location.location_precision == "city"
|
||||
assert result.location.needs_confirmation is True
|
||||
assert result.location.is_estimated is True
|
||||
assert result.location.city is None
|
||||
assert result.location.matched_location_name == "Seoul"
|
||||
assert "Epoch Location: Seoul" in result.location.location_source_note
|
||||
|
||||
|
||||
def test_epoch_city_geojson_preserves_source_location_without_guessing_city():
|
||||
record = _build_record(
|
||||
record_id=37,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Petrobras Fênix",
|
||||
country="Brazil",
|
||||
city="",
|
||||
latitude=-22.909534,
|
||||
longitude=-43.209934,
|
||||
metadata={
|
||||
"location": "Rio de Janeiro, Brazil",
|
||||
"raw_country": "Brazil",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([record])
|
||||
|
||||
feature = payload["features"][0]
|
||||
assert feature["properties"]["location_precision"] == "city"
|
||||
assert feature["properties"]["matched_location_name"] == "Rio de Janeiro, Brazil"
|
||||
assert "Epoch Location: Rio de Janeiro, Brazil" in feature["properties"][
|
||||
"location_source_note"
|
||||
]
|
||||
|
||||
|
||||
def test_confirmed_stored_location_beats_epoch_precise_coordinates():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-35": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-35",
|
||||
"name": "Confirmed Site",
|
||||
"latitude": 35.0,
|
||||
"longitude": 139.0,
|
||||
"precision": "site",
|
||||
"confidence": 1.0,
|
||||
"location_source": "official_facility_registry",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=35,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Japan Cluster",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={
|
||||
"location": "1-1 Verified Street, Tokyo 100-0001, Japan",
|
||||
"raw_country": "Japan",
|
||||
},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "stored_compute_center_location"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (35.0, 139.0)
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_legacy_seeded_country_coordinate_is_ignored_on_read():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-36": {
|
||||
"source": "epoch_ai_gpu",
|
||||
"source_id": "epoch_ai_gpu-36",
|
||||
"name": "Legacy Country Point",
|
||||
"latitude": 36.386493,
|
||||
"longitude": 138.59223,
|
||||
"precision": "precise",
|
||||
"confidence": 1.0,
|
||||
"location_source": "source_coordinates",
|
||||
"needs_confirmation": False,
|
||||
}
|
||||
})
|
||||
try:
|
||||
record = _build_record(
|
||||
record_id=36,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Legacy Japan Cluster",
|
||||
country="Japan",
|
||||
city="",
|
||||
latitude=36.386493,
|
||||
longitude=138.59223,
|
||||
metadata={"location": "Japan", "raw_country": "Japan"},
|
||||
)
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location is None
|
||||
finally:
|
||||
compute_center_locations.set_compute_center_location_cache({})
|
||||
|
||||
|
||||
def test_verified_sites_replace_epoch_country_center_coordinates():
|
||||
specifications = [
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_0_gyoukou",
|
||||
"Gyoukou 2.0",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_jamstec_zettascaler_2_2_gyoukou",
|
||||
"Gyoukou 2.2",
|
||||
"Japan",
|
||||
36.386493,
|
||||
138.59223,
|
||||
[139.625008, 35.380924],
|
||||
),
|
||||
(
|
||||
"epoch_argonne_nl_sophia__formerly_theta_",
|
||||
"Sophia",
|
||||
"United States of America",
|
||||
39.381266,
|
||||
-97.922211,
|
||||
[-87.984498648496, 41.718109383287235],
|
||||
),
|
||||
]
|
||||
records = []
|
||||
for index, (source_id, name, country, latitude, longitude, _expected) in enumerate(
|
||||
specifications, start=40
|
||||
):
|
||||
record = _build_record(
|
||||
record_id=index,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name=name,
|
||||
country=country,
|
||||
city="",
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
metadata={"location": country, "raw_country": country},
|
||||
)
|
||||
record.source_id = source_id
|
||||
records.append(record)
|
||||
|
||||
payload = convert_compute_centers_to_geojson(records)
|
||||
|
||||
assert payload["unresolved"] == []
|
||||
assert len(payload["features"]) == 3
|
||||
for feature, specification in zip(payload["features"], specifications):
|
||||
assert feature["geometry"]["coordinates"] == specification[-1]
|
||||
assert feature["properties"]["location_precision"] == "site"
|
||||
assert feature["properties"]["location_source"] == "official_facility_registry"
|
||||
assert "https://" in feature["properties"]["location_source_note"]
|
||||
|
||||
|
||||
def test_verified_site_beats_epoch_city_level_source_coordinates():
|
||||
record = _build_record(
|
||||
record_id=44,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Gyoukou 2.2",
|
||||
country="Japan",
|
||||
city="Tokyo",
|
||||
latitude=35.68882,
|
||||
longitude=139.692526,
|
||||
metadata={"location": "Tokyo, Japan", "raw_country": "Japan"},
|
||||
)
|
||||
record.source_id = "epoch_jamstec_zettascaler_2_2_gyoukou"
|
||||
|
||||
result = compute_center_locations.resolve_compute_center_location_full(
|
||||
record, record.extra_data
|
||||
)
|
||||
|
||||
assert result.location.location_source == "official_facility_registry"
|
||||
assert result.location.location_precision == "site"
|
||||
assert (result.location.latitude, result.location.longitude) == (
|
||||
35.380924,
|
||||
139.625008,
|
||||
)
|
||||
|
||||
|
||||
def test_compute_center_geojson_uses_stored_location_when_source_coords_missing():
|
||||
compute_center_locations.set_compute_center_location_cache({
|
||||
"epoch_ai_gpu:epoch_ai_gpu-32": {
|
||||
@@ -505,6 +741,7 @@ def test_collect_location_candidates_failure_returns_attempted_queries(monkeypat
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_skips_llm_when_candidates_exist(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -548,6 +785,7 @@ async def test_collect_compute_center_location_skips_llm_when_candidates_exist(m
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_compute_center_location_uses_llm_when_candidates_empty(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
llm_candidate = compute_center_locations.LocationCandidate(
|
||||
latitude=45.764,
|
||||
longitude=4.8357,
|
||||
@@ -615,7 +853,11 @@ async def test_collect_compute_center_location_uses_llm_when_candidates_empty(mo
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -637,7 +879,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -680,6 +926,11 @@ async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
async def build_without_shared_cache(**kwargs):
|
||||
return await kwargs["builder"]()
|
||||
|
||||
# Exercise the fixture DB, never the running development server's Redis payload.
|
||||
monkeypatch.setattr(visualization_api, "get_or_build_layer_payload", build_without_shared_cache)
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
@@ -741,7 +992,11 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
metadata={
|
||||
"location": "3231 Riverport Rd, Memphis, TN 38109",
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
@@ -769,6 +1024,8 @@ async def test_visualization_geo_summary_returns_counts(monkeypatch):
|
||||
class _FakeSession:
|
||||
async def execute(self, query):
|
||||
query_text = str(query).lower()
|
||||
if "group by collected_data.source" in query_text:
|
||||
return _ScalarResult(rows=[("top500", "supercomputer", 1), ("epoch_ai_gpu", "gpu_cluster", 1)])
|
||||
if "bgp_incidents" in query_text:
|
||||
return _ScalarResult(scalar_value=2)
|
||||
if "bgp_anomalies" in query_text:
|
||||
@@ -839,6 +1096,7 @@ async def test_empty_cable_layers_return_empty_feature_collections():
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_candidates_for_known_record(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
def _fake_ror(query):
|
||||
assert query == "Oak Ridge National Laboratory"
|
||||
return {
|
||||
@@ -926,6 +1184,7 @@ async def test_collect_location_endpoint_returns_candidates_for_known_record(mon
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_collect_location_endpoint_returns_failure_reason(monkeypatch):
|
||||
monkeypatch.setattr(visualization_api, "_compute_center_location_web_search_capability", AsyncMock(return_value={"enabled": True}))
|
||||
monkeypatch.setattr(compute_center_locations, "_lookup_ror_organization", lambda _query: None)
|
||||
monkeypatch.setattr(compute_center_locations, "_geocode_online", lambda _query: None)
|
||||
|
||||
@@ -1136,3 +1395,19 @@ def test_repository_has_no_forbidden_precision_tokens():
|
||||
assert "FORBIDDEN_PRECISIONS" in text, (
|
||||
f"{token} appears in {target} outside the FORBIDDEN_PRECISIONS guard"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", ["data_center", "cloud_center", "edge_node"])
|
||||
def test_infrastructure_type_is_preserved_in_geojson_and_image_classification(kind):
|
||||
from app.core.compute_types import compute_site_type
|
||||
record = _build_record(record_id=900, source="facility_catalog", data_type=kind,
|
||||
name="Verified facility", country="Japan", city="Tokyo", latitude=35.68, longitude=139.76,
|
||||
metadata={"location_precision": "site"})
|
||||
result = convert_compute_centers_to_geojson([record])
|
||||
assert compute_site_type(record) == kind
|
||||
assert result["features"][0]["properties"]["site_type"] == kind
|
||||
record.source = "epoch_ai_gpu"
|
||||
record.data_type = "gpu_cluster"
|
||||
record.extra_data = {**record.extra_data, "site_type": kind, "location": "Tokyo, Japan"}
|
||||
assert compute_site_type(record) == kind
|
||||
assert convert_compute_centers_to_geojson([record])["features"][0]["properties"]["site_type"] == kind
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import pytest
|
||||
import importlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.websocket.manager import ConnectionManager
|
||||
from app.core.websocket.broadcaster import DataBroadcaster
|
||||
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
|
||||
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
|
||||
broadcaster = DataBroadcaster()
|
||||
async def load_current(keys):
|
||||
assert keys == ["1"]
|
||||
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
|
||||
broadcaster.enqueue_vessel_update(
|
||||
{
|
||||
"source": "aisstream_vessels",
|
||||
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
assert sent[0]["vessels"] == [
|
||||
{
|
||||
"mmsi": 1,
|
||||
"lat": 59.1,
|
||||
"lon": 10.1,
|
||||
"source": "aisstream_vessels",
|
||||
"lat": 60.0,
|
||||
"lon": 11.0,
|
||||
"source": "barentswatch_vessels",
|
||||
"action": "upsert",
|
||||
"created": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
json.dumps(config)
|
||||
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
|
||||
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
|
||||
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
|
||||
manager.unsubscribe(socket, ["vessels"])
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
assert len(socket.sent) == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_removal_does_not_require_coordinates():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
|
||||
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
|
||||
|
||||
|
||||
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
|
||||
websocket_module = importlib.import_module("app.api.v1.websocket")
|
||||
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
|
||||
app = FastAPI()
|
||||
app.include_router(websocket_module.router)
|
||||
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
|
||||
assert socket.receive_json()["type"] == "connection_established"
|
||||
socket.send_json({
|
||||
"type": "subscribe",
|
||||
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
|
||||
})
|
||||
response = socket.receive_json()
|
||||
assert response["type"] == "subscription_confirmed"
|
||||
assert response["data"]["vessels"]["scope"] == "global"
|
||||
assert response["data"]["vessels"]["type"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
|
||||
broadcaster = DataBroadcaster()
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
|
||||
|
||||
async def fail_read(_keys):
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
|
||||
with pytest.raises(RuntimeError, match="database unavailable"):
|
||||
await broadcaster.flush_vessel_updates()
|
||||
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61
|
||||
|
||||
@@ -25,6 +25,25 @@
|
||||
{
|
||||
"id": "china-taiwan-penghu",
|
||||
"name": "Taiwan and Penghu",
|
||||
"sourceIsoA3": ["TWN"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-hong-kong",
|
||||
"name": "Hong Kong",
|
||||
"sourceIsoA3": ["HKG"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
"labelPolicy": "show_country_only"
|
||||
},
|
||||
{
|
||||
"id": "china-macao",
|
||||
"name": "Macao",
|
||||
"sourceIsoA3": ["MAC"],
|
||||
"action": "union_to_country",
|
||||
"targetIsoA3": "CHN",
|
||||
"hoverIsoA3": "CHN",
|
||||
@@ -57,6 +76,7 @@
|
||||
{
|
||||
"id": "kosovo",
|
||||
"name": "Kosovo",
|
||||
"sourceIsoA3": ["KOS", "XKS"],
|
||||
"action": "render_as_disputed_with_parent",
|
||||
"parentIsoA3": "SRB",
|
||||
"hoverIsoA3": "SRB",
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
aiprovider:
|
||||
image: ${AI_PROVIDER_IMAGE_NAME:-planet-aiprovider:latest}
|
||||
@@ -32,7 +30,7 @@ services:
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: planet_db
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "127.0.0.1:${PLANET_POSTGRES_PORT:-5432}:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
@@ -45,7 +43,7 @@ services:
|
||||
image: redis:7-alpine
|
||||
container_name: planet_redis
|
||||
ports:
|
||||
- "6379:6379"
|
||||
- "127.0.0.1:${PLANET_REDIS_PORT:-6379}:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
|
||||
@@ -8,6 +8,300 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.76.2] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 优化光缆密集区的卫星点拾取,并将点状交互物件优先级泛化到其他密集图层。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 屏幕命中先统一比较卫星、BGP、算力和其他点状对象的距离,海缆线作为兜底候选,保护穿插在密集线缆中的小面积对象。
|
||||
- hover 与 click 共用同一候选排序,避免鼠标提示和最终选中对象不一致。
|
||||
- 补充密集区域拾取回归测试,清理中间态代码并同步渲染层说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.1] — 2026-10-01
|
||||
|
||||
Released: 2026-10-01
|
||||
|
||||
### Highlights
|
||||
- 修复选中 BGP 事件静态光晕在地表边缘被切断或陷入地表的问题。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 抽取通用整体锚点遮挡能力,BGP 事件保持原有图标样式并复用圆角图标的前后显隐规则。
|
||||
- 选中事件光晕与事件图标共享地表锚点、前景层级和前后显隐路径,清理中间态代码与资源。
|
||||
- 补充中英文渲染层说明、回归测试和浏览器验证记录。
|
||||
|
||||
---
|
||||
|
||||
## [0.76.0] — 2026-09-30
|
||||
|
||||
Released: 2026-09-30
|
||||
|
||||
### Highlights
|
||||
- 新增详情卡默认配图与后台配置,缺图时保持完整数据卡片,并清理已替换的旧资源。
|
||||
- 算力中心补齐超算、GPU 集群、数据中心、云计算中心和边缘计算节点类型及配图。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一地表物件身份、拾取优先级、连线目标、聚合展示和选中置顶;跨图层重叠时焦点对象保持在前景。
|
||||
- 详情卡保留全量显示,统一高度、字段顺序、拖拽落位和新闻展开交互;图片解析改用按对象索引的快速路径。
|
||||
- 合并卫星、轨迹、海缆和聚合渲染的重复计算,保留完整数据与现有交互;同步中英文 Earth、后台与运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.4] — 2026-09-29
|
||||
|
||||
Released: 2026-09-29
|
||||
|
||||
### Highlights
|
||||
- 图例统一承载分类筛选,完善卫星微点与性能、圆形菜单 DPI 适配及星空颜色输出,清理废弃代码与旧方案。
|
||||
|
||||
### Improved / Fixed
|
||||
- 海缆、卫星、算力、船只、BGP 和国界图例直接切换显示;同步聚类、拾取、实时更新、选中卡片和覆盖状态,移除图层中的独立筛选入口。
|
||||
- 卫星使用单批硬边 Points,按参考校准远近点径与类别色,保留平滑短拖尾;400% 起显示真实标签,并加入可持久化 FPS 与滚动平均统计。
|
||||
- 缓存轨道插值半径、合并投影误差小于 0.2px 的冗余拖尾段、仅上传活跃缓冲;覆盖网格复用并在 GPU 投影,保留完整数据与网格精度。
|
||||
- 修复扇形菜单及缩放桥接的 DPI/hover 状态、星空局部 sRGB 输出、加载时菱形纬圈、地表标记侧面遮挡与登陆点海缆对齐;保留船只独立箭头并统一电视方角。
|
||||
- RSS/Atom 严格识别 BOM、HTTP/XML 编码,避免中文新闻乱码;删除旧随机星点、背景点、无效图例接口、未引用绘制分支及已完成的旧计划,更新双语说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.3] — 2026-09-27
|
||||
|
||||
Released: 2026-09-27
|
||||
|
||||
### Highlights
|
||||
- 升级地球材质、地图标记与卫星展示,修复拖拽、采集缓存和任务队列,清理废弃渲染路径。
|
||||
|
||||
### Improved / Fixed
|
||||
- 高清地表采用海陆分离 Shader、可读夜侧与单层柔和大气;保留四张底图及真实昼夜,移除旧厚光壳和无用高光配置。
|
||||
- 卫星在真实轨道采样之间平滑移动,类别色在昼夜保持一致,蓝色覆盖与拾取位置同步;预测轨道开放并加入方向流光,消除强制闭合尖角。
|
||||
- 地表设施统一渐变徽标,BGP 保留不规则扁平例外;修复低缩放圆点开关,电视面板与播放器恢复方角。
|
||||
- 鼠标与单指拖出球体仍持续旋转,保留惯性并在极大倍率下限制灵敏度;修复欧拉角换支导致的反向和普通横拖侧歪。
|
||||
- CelesTrak 改为直接 GET、限时有效缓存和校验后原子替换;采集队列按 task ID 保留独立历史,防止失败任务被新任务或迟到消息重新激活。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.2] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- 修复 Windows 启动器固定使用旧版 PowerShell 的问题,优先在 Windows Terminal 新窗口中运行 PowerShell 7。
|
||||
|
||||
### Fixed
|
||||
- 仓库与桌面 CMD 引导优先使用 `pwsh.exe`,缺少新版 PowerShell 或 Terminal 时保留可用环境回退。
|
||||
- 已以管理员运行的入口同样进入首选终端,子窗口校验权限后直接衔接 WSL,避免重复提权和递归开窗。
|
||||
- 补齐终端选择、含空格路径、提权分支回归和中英文说明,保持 WSL 重启参数及默认端口不变。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.1] — 2026-09-25
|
||||
|
||||
Released: 2026-09-25
|
||||
|
||||
### Highlights
|
||||
- Windows 启动器从当前项目位置识别 WSL 发行版与目录,首次生成本机桌面 CMD,后续沿用标准重启流程。
|
||||
- 算力对比数据可由后台维护;小球恢复惯性拖动,大幅放大后保留球面跟手,修复移出球面后的反向旋转。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 桌面入口兼容两种 WSL UNC 路径和 Windows 本地磁盘映射,保留原有文件,支持只安装检查;默认端口仍为 3000 / 8000 / 8010。
|
||||
- 算力报告来源、年度观测与创新分差使用数据库配置,提供数值校验、版本冲突保护、事务审计、恢复默认和地球自动刷新。
|
||||
- 根据相机视锥在按下时固定拖动模式;小球使用旧版位移平滑与惯性,大球不再用球外最近点生成反向旋转。
|
||||
- 更新开发 agent 的 Sol / Luna / Astra 分工规则,补齐中英文运维说明、启动故障编号与隔离回归测试。
|
||||
|
||||
---
|
||||
|
||||
## [0.75.0] — 2026-09-23
|
||||
|
||||
Released: 2026-09-23
|
||||
|
||||
### Highlights
|
||||
- 新增算力与资源区域比较、可拖动的地理卡片和移动端原位展开;以算力占比为主,保留趋势、覆盖与来源信息。
|
||||
- 四张 8K 底图可切换,原始底图设为默认并列于首位;优化高清首帧加载,修复最大缩放穿透和高倍拖动不跟手。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 统一物件卡片的先绘线后显示时序与靠边布局;地球明显转向后成组调整区域卡片,减少交叉并保留阅读时间。
|
||||
- 合并算力区域目录及历史口径,校验设施坐标粒度与公开站点,避免以样本数冒充算力占比或将近似坐标规则排列。
|
||||
- 地表贴图异步解码与按需替换,启动模块打包、数据并行加载;复用共享下拉菜单、HUD 样式与移动端交互。
|
||||
- 国界贴合地表并贯穿固定地理显示规则;修复直播恢复、已选频道回退、本地账户身份响应与相关交互回归。
|
||||
- 补齐后端镜像所需规则、审核几何和构建脚本,记录高精国界重建要求,更新中英文使用及运维文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.7] — 2026-09-17
|
||||
|
||||
Released: 2026-09-17
|
||||
|
||||
### Highlights
|
||||
- 启动进度统一显示在状态副行,成功清除、失败保留具体错误并退出,减少演示前的静默等待和重复报错。
|
||||
- 镜像仓库网络故障支持一次临时备用源回退,校验具体镜像 manifest 并锁定 digest,不改写全局镜像源。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 为公开 Python、uv、PostgreSQL 和 Redis 镜像提供 DaoCloud 回退及可配置前缀;认证、证书、限流和 Dockerfile 依赖安装错误不误触发回退。
|
||||
- 数据库启动前识别其他项目的端口占用,按后端实际配置同步本机映射,保留命名数据卷;修复停止后 TIME_WAIT 被误判为监听占用。
|
||||
- Docker 命令增加有界等待、私有日志和进程组清理;失败副行保留实际原因,避免外层汇总及清理通知覆盖。
|
||||
- 修复 Python 依赖清华源回退前过早冻结错误状态,移除 Compose 失败后的旧手动启动分支。
|
||||
- 同步中英文排障手册,补齐镜像回退、非网络错误、digest 校验、端口复用及失败退出回归,并验证实际备用镜像构建与项目启动。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.6] — 2026-09-16
|
||||
|
||||
Released: 2026-09-16
|
||||
|
||||
### Highlights
|
||||
- AI Provider 构建前自动检测主机代理和直连路径,减少终端网络正常但 Docker 镜像拉取超时的问题。
|
||||
- 运维错误原因与脚本诊断共用一份对照表,统一错误编号、原因和处理建议。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 已有 Compose v2 时不再因构建或启动失败回退 v1;Dockerfile 改用 BuildKit 内置解析器,减少一次外部镜像下载。
|
||||
- 新增 --no-build 启动选项,明确复用本地 AI Provider 镜像,缺少镜像时报错且不改写构建指纹。
|
||||
- 自动代理检测尊重 NO_PROXY,仅修改 Planet 管理的本地 Docker 配置;更新前备份和校验,必要时重启并恢复原有容器,失败时回滚。
|
||||
- 中英文运维表覆盖网络、证书、权限、限流和启动故障;未知错误明确保留未归类状态,并要求新确认原因补表与回归用例。
|
||||
- 修复详细日志模式丢失构建失败退出码的问题,补齐代理切换、Compose 选择、错误分类及文档一致性验证,并收录算力与资源态势实施计划。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.5] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
|
||||
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
|
||||
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
|
||||
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
|
||||
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
|
||||
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.4] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
|
||||
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
|
||||
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
|
||||
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
|
||||
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
|
||||
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.3] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
|
||||
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
|
||||
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
|
||||
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
|
||||
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
|
||||
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.2] — 2026-07-01
|
||||
|
||||
Released: 2026-07-01
|
||||
|
||||
### Highlights
|
||||
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`,删除重复维护的旧 Claude command 入口。
|
||||
- 强化视觉证据规则:截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置,而不是跳过后猜测。
|
||||
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback,同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容,保留最新单一入口和 harness 验证说明。
|
||||
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate,补齐路径解析、访问失败报告和 OCR fallback 边界。
|
||||
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口,并更新文档受众计划中的旧路径引用。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.1] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内,移除旧的全局“选择资产/上传”工具栏。
|
||||
- 新增字段级图片拖拽反馈,拖到对应字段时直接提示将图片复制为 Logo 或标题图。
|
||||
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式,并同步中英文使用手册、快速开始和控制台上下文文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
|
||||
- `FieldGrid` 支持按字段注入自定义输入控件,同时复用统一草稿提交路径。
|
||||
- 品牌上传拖拽态改为低饱和 tactile 配色,上传按钮保持蓝色轻立体样式,容器内上下/右侧留白对齐为 3px。
|
||||
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.0] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口,减少英文界面中文残留。
|
||||
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示,避免英文态裁切或错位。
|
||||
- 将容易遗漏的 i18n 入口和视觉回归加入 harness,让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 Earth runtime i18n 入口,统一高清材质、启动状态、错误提示和图层状态文案来源。
|
||||
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
|
||||
- 更新控制台与 Earth 布局规则,保留 brand 尺寸语义,同时让标题、副标题和通知胶囊按内容完整显示。
|
||||
- 扩展 frontend smoke 与 harness 文档,固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
|
||||
- 更新 Earth 新闻本地化服务与测试,确保英文界面新闻内容不再回退中文 UI 文案。
|
||||
|
||||
---
|
||||
|
||||
## [0.73.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 新增前端统一 i18n 基础设施,让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
|
||||
- 控制台侧边栏偏好面板接入语言与主题切换,并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
|
||||
- 扩展 harness 与 smoke 覆盖,确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `frontend/src/i18n/`,用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
|
||||
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
|
||||
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译,并覆盖动态计数字符串。
|
||||
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership,避免首屏溢出和状态词裁切。
|
||||
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则,记录语言迁移边界、状态指示器布局约束和一屏验证要求。
|
||||
|
||||
---
|
||||
|
||||
## [0.72.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 将 agent 入口收敛到单一 `AGENTS.md`,并让 harness 明确阻止小写入口再次分叉。
|
||||
- 新增完整本地 harness 验证层,覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
|
||||
- 扩展 Earth News 与控制台 smoke,确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本,并将未跟踪 harness 设施纳入发布。
|
||||
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出,改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
|
||||
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
|
||||
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档,明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
|
||||
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.1] — 2026-06-26
|
||||
|
||||
Released: 2026-06-26
|
||||
|
||||
186
docs/HARNESS.md
@@ -9,7 +9,7 @@ or release workflows.
|
||||
Existing project rules are authoritative:
|
||||
|
||||
1. `rules.md`
|
||||
2. `agents.md`
|
||||
2. `AGENTS.md`
|
||||
3. Current implementation docs under `docs/technical/`
|
||||
4. Existing scripts, especially `planet.sh`
|
||||
5. Existing Gitea workflow files under `.gitea/workflows/`
|
||||
@@ -18,6 +18,17 @@ When harness guidance conflicts with any of the above, keep the existing rule,
|
||||
do not overwrite the existing workflow, and add a compatibility note here or in
|
||||
`docs/harness-audit.md`.
|
||||
|
||||
For frontend or documentation audits, also read the Rules Coverage Evidence
|
||||
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
|
||||
static checks, Playwright smoke coverage, and remaining manual review areas, so
|
||||
an agent can distinguish a proved harness pass from a rule that still needs
|
||||
human-quality inspection.
|
||||
|
||||
When the user describes work with product words rather than module names, use
|
||||
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
|
||||
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
|
||||
`地球`, `模型供应商`, and `发版` to the required rule modules.
|
||||
|
||||
## Starting Work
|
||||
|
||||
Recommended startup flow:
|
||||
@@ -68,11 +79,25 @@ git diff --unified=0 HEAD -- <path>
|
||||
|
||||
## Harness Commands
|
||||
|
||||
Windows desktop launcher tests run separately on Windows with
|
||||
`powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\harness\test_windows_launcher.ps1`.
|
||||
They use temporary desktop folders and mocked restart commands; no UAC prompt
|
||||
or real service restart is triggered. The WSL bridge regression suite,
|
||||
`scripts/harness/test_windows_restart.py`, is included in quick-check and uses
|
||||
disposable command substitutes. For a real entry-point check without restarting,
|
||||
run the repository `planet.cmd --install-only`, then the generated desktop
|
||||
launcher with the same flag. Installation and mocked checks do not prove a live
|
||||
elevated restart; report that separately when it has not been exercised.
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
Docker image smoke builds are expensive and are off by default:
|
||||
|
||||
@@ -80,6 +105,106 @@ Docker image smoke builds are expensive and are off by default:
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Frontend Playwright smoke runs by default in full validation after the frontend
|
||||
build. It starts a local Vite preview and checks the `/` to Earth redirect,
|
||||
public pages, unknown-route login fallback, protected admin route login
|
||||
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
|
||||
mocked API content, Docs detail page
|
||||
language/theme/search interactions, every Docs catalog slug exposed by the
|
||||
frontend/backend metadata, the Earth iframe entry point, login error handling,
|
||||
register + email verification, password reset, standalone email verification,
|
||||
and authenticated `super_admin` rendering for every admin route plus core
|
||||
`section` deep links derived from the actual admin route and section config.
|
||||
Authenticated admin
|
||||
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
|
||||
mobile passes also fail on global horizontal overflow so table/detail panels
|
||||
must keep overflow ownership inside their own scroll regions. To enforce the
|
||||
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
|
||||
hard rendered check: the shell must resolve to the viewport height through the
|
||||
root 100% height chain, `#root`/document/body must not gain vertical overflow,
|
||||
and the desktop sidebar account/preferences area must remain inside the first
|
||||
viewport while the nav owns any excess scrolling. The smoke also
|
||||
derives the sidebar menu from the actual admin route manifest and clicks every
|
||||
visible `super_admin` menu entry on both desktop and mobile viewports, then
|
||||
exercises safe interaction paths for admin search, section tabs, the AI settings
|
||||
shortcut, logs view switching, user dialog opening, and data distribution toggles.
|
||||
It also exercises Earth News source testing, add/cancel source draft behavior,
|
||||
and manual news group creation against mocked `/earth/news-*` APIs.
|
||||
Documented AI and collector
|
||||
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
|
||||
`/collection-management?section=collector_credentials` are part of the rendered
|
||||
smoke surface:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
|
||||
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
### Visual Evidence And Style Consistency
|
||||
|
||||
- Treat user-provided screenshots and images as primary visual evidence. If a
|
||||
screenshot contradicts written text, inspect the image first and explicitly
|
||||
call out the mismatch before deciding what to change.
|
||||
- Path resolution is part of visual evidence handling. If a referenced
|
||||
screenshot path cannot be opened, try reasonable local equivalents first:
|
||||
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
|
||||
thread attachments, repository files, and obvious local attachment/download
|
||||
locations.
|
||||
- If the image still cannot be found or opened, report the exact path/access
|
||||
blocker instead of guessing. Do not infer image content from the filename, alt
|
||||
text, surrounding prose, logs, or memory.
|
||||
- OCR is acceptable evidence for text-only questions or non-multimodal
|
||||
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
|
||||
and rendering issues still require a real visual inspection or an explicit
|
||||
"could not verify visually" note.
|
||||
- Same-category UI surfaces must use one visual system per product area. Badges,
|
||||
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
|
||||
controls should reuse the shared component, shared token, or established CSS
|
||||
owner for that area instead of introducing a page-local lookalike.
|
||||
- `scripts/harness/frontend-rules-check.sh` warns when semantic
|
||||
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
|
||||
approved React and Earth CSS owner files. A warning means the reviewer should
|
||||
either move the style into the shared owner or document why this is a genuinely
|
||||
new visual family.
|
||||
|
||||
### Earth I18n Harness Rules
|
||||
|
||||
Earth i18n work must validate rendered behavior, not only static text lookup.
|
||||
Agents often miss dynamic strings that are created after initial page load, so
|
||||
the smoke treats these as first-class i18n surfaces:
|
||||
|
||||
- **Visible text and attributes**: translated checks must include `innerText`
|
||||
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
|
||||
buttons are user-facing copy, not implementation details.
|
||||
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
|
||||
BGP markers, compute centers, vessels, and news must render field labels,
|
||||
status values, source tags, action buttons, and disabled/tooltips in the active
|
||||
language.
|
||||
- **English content safety**: English mode must not fall back to Chinese news
|
||||
titles, summaries, feed names, measure words, or generic status labels. If no
|
||||
English localization exists, hide the item or use a neutral English fallback.
|
||||
- **Brand assets**: locale switching must update both text and image assets.
|
||||
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
|
||||
for English while keeping the same top-left layout and logo position.
|
||||
- **Controls and state**: switch/segmented-control visuals must follow the real
|
||||
checked/pressed state after both direct clicks and programmatic panel changes.
|
||||
A control is not valid if the state changes but the thumb, active pill, or
|
||||
`aria-*` state stays stale.
|
||||
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
|
||||
must enter through `earthMessage(...)` plus the centralized
|
||||
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
|
||||
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
|
||||
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
|
||||
or `earth:status` events.
|
||||
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
|
||||
overflow their panel. Prefer a slightly wider owning panel for important
|
||||
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
|
||||
translated tooltip.
|
||||
|
||||
Current frontend smoke explicitly covers the Earth English locale flow: brand
|
||||
image swap, settings language controls, panel switch visual sync, English news
|
||||
filtering, English detail-card text and tooltips, and TV default/source labels.
|
||||
|
||||
## Environment Requirements
|
||||
|
||||
Required for normal development:
|
||||
@@ -89,6 +214,12 @@ Required for normal development:
|
||||
- `bun` for frontend dependency and build execution
|
||||
- Python resolved by `uv` from the root `pyproject.toml`
|
||||
|
||||
Harness command lookup first checks the current non-interactive `PATH`. If a
|
||||
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
|
||||
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
|
||||
path. This avoids hardcoding a dotfile while still covering agent environments
|
||||
that do not inherit the user's normal shell setup.
|
||||
|
||||
Required for full local stack operation:
|
||||
|
||||
- Docker and Docker Compose
|
||||
@@ -99,9 +230,26 @@ Optional for delivery smoke:
|
||||
- Docker daemon for image builds
|
||||
- Helm for chart lint/template checks
|
||||
|
||||
If a required local tool is missing, do not install system software
|
||||
automatically. Report the gap and point to `./planet.sh init` or
|
||||
`scripts/bootstrap-dev.sh` as the existing bootstrap path.
|
||||
For routine harness validation, do not install missing system software
|
||||
automatically. Report the gap and point to the explicit bootstrap entry points.
|
||||
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
|
||||
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
|
||||
This bootstrap behavior is intentional; do not invoke it merely to make harness
|
||||
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
|
||||
|
||||
Docker bootstrap regression checks use isolated command stubs and never install
|
||||
packages or modify the host daemon:
|
||||
|
||||
```bash
|
||||
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
|
||||
uv run --frozen --project . python scripts/harness/test_database_startup.py
|
||||
```
|
||||
|
||||
Database startup regressions also run in quick-check. They cover Compose
|
||||
reconciliation of existing containers, visible startup errors, published-port
|
||||
checks, bounded recreation that preserves volumes, and the backend connection
|
||||
gate before schema initialization. Their command stubs and driver mocks do not
|
||||
modify the host Docker environment.
|
||||
|
||||
## What Agents Must Not Change Automatically
|
||||
|
||||
@@ -121,6 +269,14 @@ No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
- Run `scripts/harness/quick-check.sh` before handing off small changes.
|
||||
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
|
||||
- Run `scripts/harness/security-check.sh` after touching config, auth,
|
||||
credentials, docs examples, or generated fixtures.
|
||||
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
|
||||
catch direct stdout/debugger calls before they reach runtime logs.
|
||||
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
|
||||
route, package-manager, debug-output, and UI rule warnings.
|
||||
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
|
||||
route changes.
|
||||
- Add focused tests before modifying backend service behavior or frontend
|
||||
workflows.
|
||||
- For docs changes, run the checks listed in
|
||||
@@ -136,6 +292,9 @@ No automatic hooks are installed in this phase. Manual reminders:
|
||||
4. Make the smallest behavior-preserving or feature-scoped change.
|
||||
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
|
||||
6. Update relevant docs when behavior, workflow, or operations change.
|
||||
7. For rendered frontend changes, verify the affected route with Playwright or
|
||||
the full harness smoke, because `bun run build` alone does not prove page
|
||||
usability.
|
||||
|
||||
### Bug Fix
|
||||
|
||||
@@ -149,7 +308,8 @@ No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
1. Read `docs/documentation-coverage-rules.md`.
|
||||
2. Route docs by audience: UI users, operations, or second-party developers.
|
||||
3. Keep Chinese and English public docs consistent when a public doc pair exists.
|
||||
3. Keep Chinese and English technical docs paired by filename; public Docs also
|
||||
need matching frontend/backend metadata when exposed in the product Docs UI.
|
||||
4. Run the repository-specific docs checks that match the changed files.
|
||||
|
||||
### Release Or Delivery Change
|
||||
@@ -161,7 +321,15 @@ release process.
|
||||
## Implementation Notes
|
||||
|
||||
- `docs/harness-audit.md` records the discovery pass that led to this harness.
|
||||
- `AGENTS.md` is a compatibility entry point for tools that expect the uppercase
|
||||
filename. The existing `agents.md` file remains in place.
|
||||
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
|
||||
`agents.md` entry has been merged into it and should remain absent.
|
||||
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
|
||||
`docs/technical/{zh,en}/`.
|
||||
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
|
||||
with mocked API data. It proves route shells, auth guards, and primary admin
|
||||
sections render, but it is not a replacement for feature-specific browser QA
|
||||
against a real backend.
|
||||
- Frontend smoke prints phase-level progress by default. Use
|
||||
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
|
||||
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
|
||||
progress lines.
|
||||
|
||||
@@ -25,12 +25,11 @@ compatibility note, not a replacement for existing rules or architecture docs.
|
||||
|
||||
| File | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `agents.md` | Present | Existing root agent behavior guide. It references `rules.md` and `project_context.md`. |
|
||||
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
|
||||
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
|
||||
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
|
||||
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
|
||||
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
|
||||
| `AGENTS.md` | Added by harness | Compatibility entry point that points to existing rules and harness docs. |
|
||||
|
||||
## Existing CI Gates
|
||||
|
||||
@@ -69,13 +68,12 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
## Missing Or Unclear Areas
|
||||
|
||||
- README previously listed `AGENTS.md` in the project tree while only lowercase
|
||||
`agents.md` existed. The harness adds uppercase `AGENTS.md` as a compatibility
|
||||
wrapper and preserves `agents.md`.
|
||||
- `project_context.md` includes older roadmap assumptions such as Celery, Kafka,
|
||||
TimescaleDB, MinIO, and UE5 as active stack elements. The current README and
|
||||
technical docs describe Web Earth, React admin, FastAPI, PostgreSQL/Redis, and
|
||||
`aiprovider` as the active local development shape.
|
||||
- The older lowercase `agents.md` entry has been merged into uppercase
|
||||
`AGENTS.md` so coding agents and harness tools use one source of truth.
|
||||
- `project_context.md` originally included older roadmap assumptions such as
|
||||
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
|
||||
harness pass updated it to separate active stack facts from future directions;
|
||||
current code and technical docs still remain authoritative when details drift.
|
||||
- No safe automatic hook system was already configured. This phase documents
|
||||
manual reminders instead of adding hooks.
|
||||
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
|
||||
@@ -84,19 +82,80 @@ The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
| Conflict Or Tension | Resolution |
|
||||
| --- | --- |
|
||||
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Added a minimal uppercase compatibility entry and preserved the existing lowercase guide. |
|
||||
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
|
||||
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
|
||||
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
|
||||
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
|
||||
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
|
||||
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
|
||||
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
|
||||
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
|
||||
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
|
||||
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
|
||||
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
|
||||
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
|
||||
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
|
||||
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
|
||||
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
|
||||
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
|
||||
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
|
||||
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
|
||||
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
|
||||
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
|
||||
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
|
||||
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
|
||||
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
|
||||
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
|
||||
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
|
||||
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
|
||||
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
|
||||
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
|
||||
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
|
||||
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
|
||||
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
|
||||
|
||||
## Rules Coverage Evidence
|
||||
|
||||
This matrix records how the current harness checks the `rules.md` modules that
|
||||
matter for this frontend and documentation pass. "Automated" means the listed
|
||||
command fails when the rule regresses. "Smoke" means the rendered product route
|
||||
or interaction is opened with Playwright. "Manual" means the rule is still a
|
||||
judgment call and must be inspected during review.
|
||||
|
||||
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
|
||||
the user describes work with Chinese/product terms instead of module names. The
|
||||
index is the routing layer; this table is the coverage/evidence layer.
|
||||
|
||||
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
|
||||
| --- | --- | --- | --- |
|
||||
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
|
||||
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
|
||||
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
|
||||
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
|
||||
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
|
||||
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
|
||||
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
|
||||
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
|
||||
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
|
||||
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
|
||||
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
|
||||
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
|
||||
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
|
||||
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
|
||||
|
||||
## Harness Files Added
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `AGENTS.md` | Compatibility agent entry point. |
|
||||
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
|
||||
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
|
||||
| `CODEMAP.md` | High-level codebase map and validation references. |
|
||||
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
|
||||
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
|
||||
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
|
||||
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
|
||||
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
|
||||
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
|
||||
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
|
||||
|
||||
@@ -22,11 +22,14 @@
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [开发 Agent 模型分配与 Token 控制计划](development-agent-model-allocation-plan.md)
|
||||
- [Earth 地理资源、空间浮动卡片与性能实施计划](earth-spatial-presentation-and-geographic-assets-plan.md)
|
||||
- [算力与资源态势:展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
|
||||
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [Earth 高精度国界静态瓦片计划](/home/ray/dev/linkong/planet/docs/plans/earth-high-precision-boundary-tiles-plan.md)
|
||||
- [Earth Predicted Orbit Plan](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [Earth WebGL Instancing Satellites Plan](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [Earth Real Terrain Plan](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [Earth News Source Configuration And Collector Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
|
||||
62
docs/plans/admin-console-i18n-plan.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# 控制台 i18n 接入计划
|
||||
|
||||
**状态**:基础设施已落地,大型业务页迁移继续进行
|
||||
**创建日期**:2026-06-29
|
||||
**核心目标**:把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系,让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
|
||||
|
||||
## 背景
|
||||
|
||||
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口,但语言状态只保存在 `docs-lang`,不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主,导致用户切到英文文档后,控制台仍是中文。
|
||||
|
||||
本计划把前端语言偏好收敛到 `planet-locale`,默认 `zh-CN`,支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口,通过前端映射与全局 locale 对齐。
|
||||
|
||||
## 设计决策
|
||||
|
||||
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层,避免长期维护自研插值、hook 和资源加载逻辑。
|
||||
- 前端统一语言枚举为 `zh-CN` / `en-US`;Docs 请求继续转换为 `zh` / `en`,新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
|
||||
- 语言偏好首版只保存在浏览器 `localStorage`,不新增后端用户设置字段。
|
||||
- `docs-lang` 保留为兼容读取和写入项,让已访问过 Docs 的浏览器能平滑迁移。
|
||||
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key;大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
|
||||
|
||||
## 分期
|
||||
|
||||
### P1:统一语言基础设施
|
||||
|
||||
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
|
||||
- 在 `frontend/src/main.tsx` 里初始化 i18n,并同步 `document.documentElement.lang`。
|
||||
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
|
||||
|
||||
### P2:高复用界面迁移
|
||||
|
||||
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
|
||||
- 搜索索引按当前语言展示,同时保留中英文关键词以免降低可发现性。
|
||||
- 用户管理页作为独立业务页示范,迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
|
||||
|
||||
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化,以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
|
||||
|
||||
### P3:大型业务页收敛
|
||||
|
||||
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
|
||||
- 过渡期保留 legacy UI 翻译桥,只处理 admin/auth 容器里的精确静态文本和属性。
|
||||
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
|
||||
|
||||
### P4:移除过渡桥
|
||||
|
||||
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时,删除 legacy UI 翻译桥。
|
||||
- 增加 key 完整性检查,确保 `zh-CN` 和 `en-US` 资源结构一致。
|
||||
|
||||
## 验证
|
||||
|
||||
- `cd frontend && bun run build`
|
||||
- `scripts/harness/frontend-rules-check.sh`
|
||||
- `scripts/harness/docs-consistency-check.sh`
|
||||
- `scripts/harness/quick-check.sh`
|
||||
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `frontend/src/i18n/`:统一 locale、资源和过渡桥。
|
||||
- `frontend/src/pages/Docs/Docs.tsx`:Docs 语言状态改为读取全局 locale。
|
||||
- `frontend/src/admin/components/layout/AdminLayout.tsx`:控制台侧边栏语言切换、导航和搜索文案。
|
||||
- `frontend/src/admin/search/indexers.ts`:Admin 搜索目标本地化。
|
||||
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`:当前实现上下文。
|
||||