Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
83a10a6c34 | ||
|
|
cee1996809 | ||
|
|
58671e7bc3 | ||
|
|
a54fcdbeed | ||
|
|
1dd2921674 | ||
|
|
d30f7d08c5 | ||
|
|
5bdb55f3f1 | ||
|
|
fbecf30513 | ||
|
|
19d5ac0fee | ||
|
|
3265d22af5 | ||
|
|
899e3bce43 | ||
|
|
8c204717cd |
@@ -1,139 +0,0 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码，并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选：指定要检查的文件或目录（默认检查所有未提交修改）
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区（git diff）中的未提交代码，找出并修复常见垃圾代码，**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空，则只检查指定文件/目录；否则检查所有未提交修改（`git diff HEAD`）。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围，不要一上来把完整文件或大 diff 读入上下文：
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时，才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题（只报告在本次 diff 中**新增或修改**的代码里存在的问题）：
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现，已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算（如偏移量、时间、尺寸、阈值），没有命名常量
|
||||
- 硬编码字符串（如 id 名、状态值、URL 片段）散落在逻辑中
|
||||
- 例外：`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量（如 `or_`, `tmp2`, `x2`）
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块（3行以上）
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符（trailing whitespace）
|
||||
- 同一文件内风格不一致（如混用单双引号、缩进不统一）
|
||||
- 空行使用不一致（连续多个空行等）
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有，导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化（不改逻辑）
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时：获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时：用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始：
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单，记录每个问题：文件名、行号、问题类型、建议修复方式。
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前，先以列表形式输出所有发现的问题：
|
||||
|
||||
```
|
||||
发现 N 个问题：
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询，可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量，应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明，应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题，直接输出"未发现垃圾代码，当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题，使用 Edit 工具进行**最小化修改**：
|
||||
|
||||
- **重复逻辑**：提取为共享常量/函数，更新所有调用点
|
||||
- **Magic number**：在文件顶部或逻辑附近声明 `const NAME = value`，替换所有引用
|
||||
- **命名问题**：重命名变量，更新所有使用处
|
||||
- **死代码**：直接删除
|
||||
- **尾部空白/风格**：修正
|
||||
- **未 export 的函数**：添加 `export`，在调用方改为导入（不重复实现）
|
||||
|
||||
**修复原则：**
|
||||
- 只改在审查清单中发现的问题，不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行，保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁；只有仓库已有对应格式化流程时，才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成：
|
||||
|
||||
修复了 N 个问题：
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14（4处引用）
|
||||
✓ controls.js — export updateLayerButtonState，移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题（需人工确认）：
|
||||
! foo.js L45 — 注释代码块较长，建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API（除非问题正是私有函数应被 export）
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容（只删除注释掉的死代码）
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定，**跳过**，在总结中标记为"需人工确认"
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
description: Create or update repository documentation from current code changes
|
||||
argument-hint: Optional: topic to document, or leave empty to infer from git diff
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — Documentation Workflow
|
||||
|
||||
## Goal
|
||||
|
||||
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
|
||||
|
||||
## Repository Rules
|
||||
|
||||
Before deciding scope, check whether the repository has a documentation rules file:
|
||||
|
||||
```bash
|
||||
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
|
||||
```
|
||||
|
||||
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Understand The Change
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
rg --files docs
|
||||
```
|
||||
|
||||
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — Decide Scope
|
||||
|
||||
- Prefer updating an existing relevant document over creating a duplicate.
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents only when the change crosses meaningful domains.
|
||||
- Keep filenames lowercase and hyphenated.
|
||||
- Apply the repository-specific rules file before writing.
|
||||
|
||||
#### Document Audience Routing (Planet)
|
||||
|
||||
In this repository, classify the action's performer before picking a target file:
|
||||
|
||||
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
|
||||
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
|
||||
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
|
||||
|
||||
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
|
||||
|
||||
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
|
||||
|
||||
### Step 3 — Write
|
||||
|
||||
Explain:
|
||||
|
||||
- Background/problem: what was wrong or missing before.
|
||||
- Core design decisions and rationale.
|
||||
- Operational or user-facing impact.
|
||||
- Relevant code paths, only when useful for future maintainers.
|
||||
|
||||
Style:
|
||||
|
||||
- Follow the repository’s existing language and heading conventions.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Prefer tables for comparisons or parameter lists.
|
||||
- Keep snippets concise and relevant.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
### Step 4 — Verify
|
||||
|
||||
- Read the completed docs once for clarity and stale statements.
|
||||
- Verify referenced paths exist with `test -e` or `rg --files`.
|
||||
- Run applicable checks from `docs/documentation-coverage-rules.md`.
|
||||
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
|
||||
|
||||
### Step 5 — Report
|
||||
|
||||
Summarize changed docs and verification:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- path/to/doc.md — what changed
|
||||
|
||||
Verified:
|
||||
- checks that passed
|
||||
- checks that could not be run, if any
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- Do not leave placeholder docs.
|
||||
- Do not duplicate bilingual files byte-for-byte.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
|
||||
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
|
||||
- Keep docs maintainable and concise.
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行，直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标；若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务：先固定目标与成功标准，再持续执行和反复验收，直到标准真正满足。
|
||||
|
||||
适用场景：
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景：
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标，没有成功标准，先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出：
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块：
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观，可验证，可落地。
|
||||
优先写成：
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环：
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足，明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论，都必须经过验证，不能直接接受。
|
||||
|
||||
5. 如果验证失败：
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作，不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止：
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据，轻口头判断
|
||||
- 优先使用确定性工具证据：`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复；保留在工具调用里，回复只总结关键证据
|
||||
- 重验收，轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行：
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足，继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -1,160 +0,0 @@
|
||||
---
|
||||
description: 发版工作流：根据变更类型决定版本号，更新所有版本文件和 changelog，运行验证，commit 并 push
|
||||
argument-hint: 可选：feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能，无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合，或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复，无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版，除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准；bugfix + 小 feature 混合默认判定为 `improvement`（`+0.0.1`）。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录，以下路径均相对于根目录：
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`（`"version"` 字段）
|
||||
- `pyproject.toml`（`version =` 字段）
|
||||
- `uv.lock`（**不要手动编辑**，通过 `uv lock` 重新生成）
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主，优先使用紧凑命令和定点读取：
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版，否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**，停下来告知用户，不要继续。
|
||||
|
||||
若存在无关的未暂存修改，列出并询问用户是否一并提交，或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型（`feature` / `bugfix`），直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 计算新版本号（例：`0.26.2` → bugfix → `0.26.3`）
|
||||
- **先输出发版计划供用户确认**：
|
||||
|
||||
```
|
||||
发版计划：
|
||||
类型：bugfix
|
||||
版本：0.26.2 → 0.26.3
|
||||
分支：dev
|
||||
将更新：VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新（每步用 Edit 工具，精确替换，不要重写整个文件）：
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`（在仓库根目录下执行）
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目，格式：
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...（只列高信号条目，最多 5 条）
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行：`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证：
|
||||
|
||||
- Python 文件有修改：先用 `git diff --name-only HEAD -- '*.py'` 列出，再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改：先用 `git diff --name-only HEAD -- frontend` 判断范围，再运行项目标准检查（若无则跳过并说明）
|
||||
- 版本号一致性检查：用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表：
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中，**不包含**非预期文件（如调试文件、.env 等）。
|
||||
|
||||
### Step 8 — Commit & Push（用户确认后）
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式：`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要：
|
||||
|
||||
```
|
||||
✓ 版本号已更新：0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成，绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码，不混入无关改动
|
||||
- 若环境中 `uv` 不可用，说明原因并跳过 lockfile 更新，提醒用户手动运行
|
||||
BIN
.codex/screenshots/earth-i18n-current-page.png
Normal file
BIN
.codex/screenshots/earth-i18n-current-page.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
BIN
.codex/screenshots/earth-i18n-hd-texture-status.png
Normal file
BIN
.codex/screenshots/earth-i18n-hd-texture-status.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.4 KiB |
BIN
.codex/screenshots/i18n-admin-data-sidebar.png
Normal file
BIN
.codex/screenshots/i18n-admin-data-sidebar.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 109 KiB |
BIN
.codex/screenshots/i18n-ai-tool-calls.png
Normal file
BIN
.codex/screenshots/i18n-ai-tool-calls.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
BIN
.codex/screenshots/i18n-earth-brand-config.png
Normal file
BIN
.codex/screenshots/i18n-earth-brand-config.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 170 KiB |
BIN
.codex/screenshots/i18n-earth-hud-brand.png
Normal file
BIN
.codex/screenshots/i18n-earth-hud-brand.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 772 KiB |
BIN
.codex/screenshots/i18n-settings-notifications.png
Normal file
BIN
.codex/screenshots/i18n-settings-notifications.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 95 KiB |
BIN
.codex/screenshots/i18n-settings-system.png
Normal file
BIN
.codex/screenshots/i18n-settings-system.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 96 KiB |
BIN
.codex/screenshots/sidebar-left-align.png
Normal file
BIN
.codex/screenshots/sidebar-left-align.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
5
.gitignore
vendored
5
.gitignore
vendored
@@ -25,7 +25,10 @@ __pycache__/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
downloads/*
|
||||
!downloads/usbipd-win/
|
||||
downloads/usbipd-win/*
|
||||
!downloads/usbipd-win/usbipd-win-5.3.0.msi
|
||||
eggs/
|
||||
.eggs/
|
||||
/lib/
|
||||
|
||||
180
AGENTS.md
Normal file
180
AGENTS.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# AGENTS.md
|
||||
|
||||
**Planet agent harness. Defines behavior for coding agents working in this repository.**
|
||||
|
||||
---
|
||||
|
||||
## Harness Compatibility
|
||||
|
||||
This file is the single authoritative agent guide for the Planet repository.
|
||||
The older lowercase `agents.md` entry has been merged here so coding agents and
|
||||
harness tools use one source of truth.
|
||||
|
||||
### Source Of Truth
|
||||
|
||||
- `rules.md` is the mandatory repository rule source. Always load `core`,
|
||||
`security`, and `workflow`; load only task-relevant modules after that.
|
||||
- `AGENTS.md` defines the local agent operating mode and evidence gates.
|
||||
- `project_context.md` is background, not a rule source. Prefer newer
|
||||
implementation docs when it disagrees with current code.
|
||||
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
|
||||
goal-driven work, and release.
|
||||
- Do not duplicate long workflow text across harness files. Durable constraints
|
||||
belong in `rules.md`; task procedures belong in skills or scripts.
|
||||
|
||||
Read these files before changing code:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. `project_context.md`
|
||||
4. `README.md`
|
||||
5. `docs/HARNESS.md`
|
||||
6. `CODEMAP.md`
|
||||
|
||||
For documentation work, also read `docs/documentation-coverage-rules.md`.
|
||||
|
||||
### Start Safely
|
||||
|
||||
Before broad edits:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Use focused context commands before reading large files:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
Preserve user changes already present in the worktree.
|
||||
|
||||
### Validation
|
||||
|
||||
Fast local harness validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/quick-check.sh
|
||||
```
|
||||
|
||||
Full local validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
|
||||
unless disabled by its documented environment flags. Docker image smoke builds
|
||||
are intentionally opt-in:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
|
||||
current non-interactive environment first. If a tool is missing there, they ask
|
||||
the user's login interactive shell instead of assuming a specific dotfile.
|
||||
|
||||
### High-Risk Areas
|
||||
|
||||
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
|
||||
`destroy` cleanup.
|
||||
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
|
||||
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
|
||||
rendered smoke evidence for public pages, auth guards, authenticated admin
|
||||
route/section availability, safe navigation/search/tab interactions, mobile
|
||||
layout, and 125% / 150% zoom, not only a build.
|
||||
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
|
||||
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
|
||||
intermediate wrappers keep `min-height: 0`, and only the intended child owns
|
||||
scrolling.
|
||||
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
|
||||
workflows in the backend.
|
||||
- Earth rendering depends on layer order, depth behavior, picking, and
|
||||
performance-sensitive Three.js code.
|
||||
- Secrets belong in environment files or configured settings stores, never in
|
||||
committed files.
|
||||
- Backend service code must use structured logging instead of `print()` or
|
||||
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
|
||||
|
||||
### Conflict Policy
|
||||
|
||||
Existing project rules and workflows win. If new harness guidance conflicts with
|
||||
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
|
||||
behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
`docs/HARNESS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
|
||||
focused file reads, tests, builds, linters, and harness scripts.
|
||||
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
|
||||
release work, and documentation unless the task requires it.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gates
|
||||
|
||||
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
|
||||
mock, browser capture, or visual reference, obtain evidence from the artifact
|
||||
before interpreting intent or editing code.
|
||||
- Path resolution is part of the task. If the path cannot be opened, first try
|
||||
reasonable local equivalents such as WSL/Windows path conversion,
|
||||
workspace-relative lookup, absolute paths, and attached-file locations.
|
||||
- Never guess from prompt text, filenames, previous context, logs, OCR, or
|
||||
memory when a visual artifact was provided but cannot be accessed.
|
||||
- OCR is acceptable evidence for text-only visual questions or non-multimodal
|
||||
environments; state that OCR was used as the fallback. Layout, color, spacing,
|
||||
pixel, and rendering issues need real visual inspection or a clear limitation
|
||||
note.
|
||||
- If a visual artifact still cannot be inspected, say so and pause that
|
||||
visual-dependent part of the work.
|
||||
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
|
||||
diff, direct file check, or harness result.
|
||||
- For UI and rendering changes, verify the rendered result when local tooling
|
||||
allows it.
|
||||
|
||||
---
|
||||
|
||||
## Communication
|
||||
|
||||
- Match the user's language. Use Chinese for Chinese requests unless the user
|
||||
asks otherwise.
|
||||
- Keep updates short and specific: what is being inspected, edited, or verified.
|
||||
- Final responses should summarize changed files and verification, with blockers
|
||||
stated plainly.
|
||||
- Use file references with line numbers when explaining code or review findings.
|
||||
|
||||
---
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Prefer existing project patterns over new abstractions.
|
||||
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
|
||||
a stable path exists.
|
||||
- Centralize prompts, constants, defaults, and shared request/response handling.
|
||||
- Do not add secrets, generated runtime output, or local environment files.
|
||||
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
|
||||
- Run the smallest relevant verification for the changed scope and report
|
||||
anything skipped.
|
||||
|
||||
---
|
||||
|
||||
## Prohibited
|
||||
|
||||
- Do not skip visual evidence handling when a visual artifact was provided.
|
||||
- Do not preserve obsolete harness files just because they already exist.
|
||||
- Do not invent behavior not present in code, docs, or verified external
|
||||
sources.
|
||||
- Do not rewrite unrelated files during cleanup.
|
||||
- Do not mark a task complete without checking concrete success criteria.
|
||||
109
CODEMAP.md
Normal file
109
CODEMAP.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Code Map
|
||||
|
||||
This map gives agents and maintainers a quick orientation without replacing the
|
||||
deeper architecture docs. Current implementation docs under `docs/technical/`
|
||||
are the source of detail for specific subsystems.
|
||||
|
||||
## Top-Level Areas
|
||||
|
||||
| Path | Role | Notes |
|
||||
| --- | --- | --- |
|
||||
| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
|
||||
| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
|
||||
| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
|
||||
| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
|
||||
| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
|
||||
| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
| Runtime | Entry Point | Validation |
|
||||
| --- | --- | --- |
|
||||
| Local full stack | `./planet.sh start` | `./planet.sh health` |
|
||||
| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
|
||||
| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
|
||||
| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
|
||||
| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
|
||||
| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
|
||||
|
||||
## Ownership Boundaries
|
||||
|
||||
- Backend owns business state, auth, evidence collection, prompt selection, AI
|
||||
task orchestration, and database persistence.
|
||||
- `aiprovider` owns provider identity, request adapter style, model gateway
|
||||
retries, and health/status endpoints only.
|
||||
- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
|
||||
and client-side state that mirrors backend truth.
|
||||
- Web Earth rendering changes must preserve documented layer order, altitude
|
||||
offsets, picking behavior, legend semantics, and performance constraints.
|
||||
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
|
||||
wrapping it from harness scripts instead of duplicating its internals.
|
||||
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
|
||||
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
|
||||
interactive command path without hardcoding `.zshrc`.
|
||||
|
||||
## Validation Commands
|
||||
|
||||
```bash
|
||||
scripts/harness/doctor.sh
|
||||
scripts/harness/security-check.sh
|
||||
scripts/harness/backend-rules-check.sh
|
||||
scripts/harness/frontend-rules-check.sh
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
scripts/harness/quick-check.sh
|
||||
scripts/harness/validate.sh
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
CI-equivalent local checks:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
|
||||
|
||||
cd frontend
|
||||
bun install --frozen-lockfile
|
||||
bun run build
|
||||
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
|
||||
```
|
||||
|
||||
The frontend smoke covers public routes, unauthenticated admin guards,
|
||||
login-error handling, the Earth iframe entry, and authenticated `super_admin`
|
||||
admin route/section rendering with mocked API data. Authenticated admin checks
|
||||
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
|
||||
check for accidental global horizontal overflow. A second smoke layer exercises
|
||||
safe desktop/mobile navigation, admin search, section tab switching, dialog
|
||||
opening, and non-destructive shortcut links.
|
||||
|
||||
Optional delivery smoke, when Docker and Helm are available:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
## Deeper Docs
|
||||
|
||||
| Topic | Start Here |
|
||||
| --- | --- |
|
||||
| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
|
||||
| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
|
||||
| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
|
||||
| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
|
||||
| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Harness workflow | `docs/HARNESS.md` |
|
||||
|
||||
## Known Sharp Edges
|
||||
|
||||
- `project_context.md` is static background for agents. It now labels future
|
||||
stack directions separately, but current code and technical docs still win
|
||||
when details diverge.
|
||||
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
|
||||
active local development shape.
|
||||
- Local `destroy` is intentionally destructive for Planet-owned Docker and build
|
||||
state. Never run it as a validation shortcut.
|
||||
@@ -83,7 +83,7 @@
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| React 18 | UI 框架 |
|
||||
| Ant Design Pro | 管理后台组件 |
|
||||
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
@@ -168,10 +168,12 @@
|
||||
|
||||
## 快速启动
|
||||
|
||||
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上，`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx，启动 Docker 服务并配置当前用户的访问权限；需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
|
||||
|
||||
```bash
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun，同步 Python/前端依赖
|
||||
# 会先准备 Docker / Compose / Buildx，再安装/检查 uv、bun 并同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis，并创建表、默认数据源和本地默认用户
|
||||
|
||||
@@ -322,7 +324,7 @@ ipconfig
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试，并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康，再验证后端实际连接；`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待，避免重复消耗健康检查预算。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
|
||||
1
TODO.md
1
TODO.md
@@ -4,6 +4,7 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
|
||||
|
||||
## Earth
|
||||
|
||||
- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
|
||||
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
|
||||
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
|
||||
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
|
||||
|
||||
231
agents.md
231
agents.md
@@ -1,231 +0,0 @@
|
||||
# agents.md
|
||||
|
||||
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
|
||||
|
||||
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
|
||||
- Python FastAPI backend
|
||||
- React Admin dashboard
|
||||
- Unreal Engine 5 3D visualization
|
||||
- Multi-source data collection
|
||||
- Polarized 3D large display (4K, 120Hz)
|
||||
|
||||
---
|
||||
|
||||
## Communication Style
|
||||
|
||||
### Tone
|
||||
- **Professional but concise**
|
||||
- Technical accuracy with clarity
|
||||
- No unnecessary verbosity
|
||||
- Use code comments sparingly (explain **why**, not **what**)
|
||||
|
||||
### When Responding
|
||||
1. **Answer directly** - 1-3 sentences for simple questions
|
||||
2. **Use code blocks** for all code snippets
|
||||
3. **Include file:line_number** references when discussing code
|
||||
4. **Never** start with "I am an AI assistant" or similar phrases
|
||||
5. **Never** add unnecessary preambles/postambles
|
||||
|
||||
### Examples
|
||||
|
||||
**Good:**
|
||||
```
|
||||
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
|
||||
```
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operational Mode
|
||||
|
||||
### Plan Mode (default for complex tasks)
|
||||
- Analyze requirements
|
||||
- Propose architecture
|
||||
- Confirm with user before execution
|
||||
- **DO NOT** write code until approved
|
||||
|
||||
### Build Mode (after user approval)
|
||||
- Execute the approved plan
|
||||
- Write code, run commands
|
||||
- Verify results
|
||||
- Report completion concisely
|
||||
|
||||
### Read-Only Mode
|
||||
- Analyze code
|
||||
- Explain functionality
|
||||
- Answer questions
|
||||
- **DO NOT** modify files
|
||||
|
||||
---
|
||||
|
||||
## Decision Framework
|
||||
|
||||
### When to Ask Before Acting
|
||||
- Unclear requirements
|
||||
- Multiple implementation approaches
|
||||
- Architecture changes
|
||||
- Dependency additions
|
||||
- Anything that could break existing functionality
|
||||
|
||||
### When to Act Directly
|
||||
- Clear, approved requirements
|
||||
- Routine tasks (linting, formatting, running tests)
|
||||
- Following established patterns
|
||||
- Fixing obvious bugs
|
||||
|
||||
### When to Refuse
|
||||
- Malicious code requests
|
||||
- Security violations (secrets, credentials)
|
||||
- Anything that violates `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## Working Principles
|
||||
|
||||
### 1. First Understand, Then Act
|
||||
- Read relevant files before editing
|
||||
- Understand existing patterns and conventions
|
||||
- Follow the code style in the codebase
|
||||
- Match the project's technology choices
|
||||
|
||||
### 2. Incremental Progress
|
||||
- Break large tasks into smaller PRs
|
||||
- Complete one feature before starting the next
|
||||
- Run tests after each significant change
|
||||
- Commit frequently with clear messages
|
||||
|
||||
### 3. Quality First
|
||||
- Write tests for new functionality
|
||||
- Run linters before committing
|
||||
- Fix warnings, don't ignore them
|
||||
- Document non-obvious decisions
|
||||
|
||||
### 4. Communication Clarity
|
||||
- Use precise technical language
|
||||
- Show relevant code, not explanations
|
||||
- Report errors with context
|
||||
- Confirm understanding of requirements
|
||||
|
||||
---
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
Before marking a task complete:
|
||||
|
||||
- [ ] Code follows `rules.md` style guidelines
|
||||
- [ ] Type hints are correct and complete
|
||||
- [ ] Error handling is proper (no silent failures)
|
||||
- [ ] Tests pass locally
|
||||
- [ ] Linting passes
|
||||
- [ ] No TODO comments left behind
|
||||
- [ ] Documentation updated if needed
|
||||
- [ ] Commit message is clear
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Feature Development
|
||||
```
|
||||
1. Understand requirements
|
||||
2. Check existing patterns in codebase
|
||||
3. Design solution (brief mental model)
|
||||
4. Write code following rules.md
|
||||
5. Write/run tests
|
||||
6. Lint and format
|
||||
7. Commit with clear message
|
||||
8. Report completion
|
||||
```
|
||||
|
||||
### Bug Fix
|
||||
```
|
||||
1. Reproduce the bug (write failing test)
|
||||
2. Locate the source
|
||||
3. Fix the issue
|
||||
4. Verify test passes
|
||||
5. Check for regressions
|
||||
6. Commit fix
|
||||
```
|
||||
|
||||
### Refactoring
|
||||
```
|
||||
1. Understand current behavior
|
||||
2. Design target state
|
||||
3. Make incremental changes
|
||||
4. Preserve tests
|
||||
5. Verify functionality
|
||||
6. Clean up dead code
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Special Considerations
|
||||
|
||||
### WebSocket Services
|
||||
- Implement heartbeat mechanism (30-second intervals)
|
||||
- Handle disconnection gracefully
|
||||
- Include camera position in control frames
|
||||
- Support both update and full sync modes
|
||||
|
||||
### Data Collectors
|
||||
- Inherit from BaseCollector
|
||||
- Implement fetch() and transform() methods
|
||||
- Support incremental updates
|
||||
- Handle API changes gracefully
|
||||
|
||||
### UE5 Integration
|
||||
- Communicate via WebSocket
|
||||
- Send data frames at configurable intervals (default 5 min)
|
||||
- Support auto-cruise and manual modes
|
||||
- Optimize for 4K@120Hz rendering
|
||||
|
||||
### Multi-User Security
|
||||
- JWT tokens with 15-minute expiration
|
||||
- Redis token blacklist for logout
|
||||
- Role-based access control (RBAC)
|
||||
- Audit logging for all actions
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
### When Writing Code
|
||||
```python
|
||||
# File: backend/app/services/collectors/top500.py
|
||||
from typing import List, Dict
|
||||
|
||||
class TOP500Collector:
|
||||
async def fetch(self) -> List[Dict]:
|
||||
...
|
||||
```
|
||||
|
||||
### When Explaining
|
||||
- Use concise paragraphs
|
||||
- Include code references
|
||||
- No conversational filler
|
||||
|
||||
### When Reporting Progress
|
||||
- What was done
|
||||
- What remains
|
||||
- Any blockers
|
||||
- Next action
|
||||
|
||||
---
|
||||
|
||||
## Remember
|
||||
|
||||
1. **Rules are hard constraints** - follow `rules.md` absolutely
|
||||
2. **Context provides understanding** - use `project_context.md` for background
|
||||
3. **Role defines behavior** - follow `agents.md` for how to work
|
||||
4. **Quality over speed** - Enterprise systems require precision
|
||||
5. **Communicate clearly** - Precision in, precision out
|
||||
@@ -1,4 +1,4 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# Use BuildKit's bundled frontend to avoid a separate Docker Hub fetch.
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.14-slim
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:latest
|
||||
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=openai-responses`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import NAMESPACE_URL, uuid4, uuid5
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
@@ -65,6 +66,7 @@ class ProviderService:
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
self.session_id = str(uuid4())
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -95,6 +97,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
if payload.context.get("session_id") is not None:
|
||||
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
|
||||
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
@@ -102,6 +106,10 @@ class ProviderService:
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif provider_api == "openai-responses":
|
||||
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
|
||||
content_blocks = self._extract_responses_blocks(data)
|
||||
content = "".join(block.text for block in content_blocks if block.text)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
@@ -200,6 +208,39 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_openai_responses(
|
||||
self, model: str, prompt: str, system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body: dict[str, Any] = {
|
||||
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["instructions"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/responses",
|
||||
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in payload.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "message":
|
||||
for part in item.get("content") or []:
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
text = part.get("text") or part.get("refusal")
|
||||
if isinstance(text, str) and text:
|
||||
blocks.append(AIContentBlock(type="text", text=text))
|
||||
elif item.get("type") == "reasoning":
|
||||
for part in item.get("summary") or []:
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
|
||||
return blocks
|
||||
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
@@ -226,7 +267,7 @@ class ProviderService:
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
@@ -243,8 +284,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
def _resolve_anthropic_thinking(
|
||||
self, thinking: dict[str, Any] | None, model: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
|
||||
return {"type": "adaptive"}
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
@@ -295,6 +340,9 @@ class ProviderService:
|
||||
headers: dict[str, str],
|
||||
request_body: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
headers = {"User-Agent": "Planet/1.0", **headers}
|
||||
if self.provider == "opencode-go":
|
||||
headers["x-opencode-session"] = self.session_id
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.http_retry_attempts + 1):
|
||||
try:
|
||||
|
||||
@@ -4,6 +4,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.enums import OtpPurpose, UserRole
|
||||
from app.core.logging import get_logger
|
||||
from app.core.security import (
|
||||
create_access_token,
|
||||
@@ -170,7 +171,7 @@ async def get_me(current_user: User = Depends(get_current_user)):
|
||||
}
|
||||
|
||||
|
||||
async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: str) -> None:
|
||||
async def _send_code_or_raise(db: AsyncSession, email: str, code: str, purpose: OtpPurpose) -> None:
|
||||
try:
|
||||
await send_verification_email(db, to=email, code=code, purpose=purpose)
|
||||
except EmailNotConfiguredError as exc:
|
||||
@@ -207,7 +208,7 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
|
||||
username=payload.username,
|
||||
email=payload.email,
|
||||
password_hash=get_password_hash(payload.password),
|
||||
role="viewer",
|
||||
role=UserRole.VIEWER.value,
|
||||
is_active=True,
|
||||
email_verified=False,
|
||||
)
|
||||
@@ -215,13 +216,13 @@ async def register(payload: UserRegister, db: AsyncSession = Depends(get_db)):
|
||||
await db.commit()
|
||||
|
||||
try:
|
||||
code = otp.issue_code(payload.email, "register")
|
||||
code = otp.issue_code(payload.email, OtpPurpose.REGISTER)
|
||||
except otp.OtpResendRateLimited as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail={"code": exc.code, "retry_after_seconds": exc.retry_after_seconds},
|
||||
) from exc
|
||||
await _send_code_or_raise(db, payload.email, code, "register")
|
||||
await _send_code_or_raise(db, payload.email, code, OtpPurpose.REGISTER)
|
||||
return {"status": "pending_verification", "email": payload.email}
|
||||
|
||||
|
||||
@@ -266,7 +267,7 @@ async def resend_code(payload: ResendCodeRequest, db: AsyncSession = Depends(get
|
||||
if user is None:
|
||||
# Avoid email enumeration; pretend success.
|
||||
return {"status": "ok"}
|
||||
if payload.purpose == "register" and user.email_verified:
|
||||
if payload.purpose is OtpPurpose.REGISTER and user.email_verified:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail={"code": "ALREADY_VERIFIED"},
|
||||
@@ -289,12 +290,17 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
|
||||
# Don't leak whether an email is registered.
|
||||
return {"status": "ok"}
|
||||
try:
|
||||
code = otp.issue_code(payload.email, "reset_password")
|
||||
code = otp.issue_code(payload.email, OtpPurpose.RESET_PASSWORD)
|
||||
except otp.OtpResendRateLimited:
|
||||
# Silently accept; the user can retry after the cooldown.
|
||||
return {"status": "ok"}
|
||||
try:
|
||||
await send_verification_email(db, to=payload.email, code=code, purpose="reset_password")
|
||||
await send_verification_email(
|
||||
db,
|
||||
to=payload.email,
|
||||
code=code,
|
||||
purpose=OtpPurpose.RESET_PASSWORD,
|
||||
)
|
||||
except EmailNotConfiguredError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
@@ -304,7 +310,11 @@ async def forgot_password(payload: ForgotPasswordRequest, db: AsyncSession = Dep
|
||||
logger.warning_event(
|
||||
"SMTP send failed",
|
||||
event="auth.email.send_failed",
|
||||
context={"email": payload.email, "purpose": "reset_password", "error": str(exc)},
|
||||
context={
|
||||
"email": payload.email,
|
||||
"purpose": OtpPurpose.RESET_PASSWORD.value,
|
||||
"error": str(exc),
|
||||
},
|
||||
)
|
||||
return {"status": "ok"}
|
||||
|
||||
@@ -318,7 +328,7 @@ async def reset_password(payload: ResetPasswordRequest, db: AsyncSession = Depen
|
||||
detail={"code": "OTP_INVALID"},
|
||||
)
|
||||
try:
|
||||
otp.verify_code(payload.email, "reset_password", payload.code)
|
||||
otp.verify_code(payload.email, OtpPurpose.RESET_PASSWORD, payload.code)
|
||||
except otp.OtpExpired as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_410_GONE,
|
||||
|
||||
@@ -13,6 +13,7 @@ import httpx
|
||||
|
||||
from app.core.target_schema_registry import get_target_schema, list_target_schemas
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.core.enums import AuthType, MappingValidationStatus, UserRole
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
@@ -34,7 +35,6 @@ from app.services.datasource_mapping import (
|
||||
)
|
||||
from app.services.custom_datasource_runtime import (
|
||||
CustomDatasourceRuntimeError,
|
||||
fetch_rest_payload,
|
||||
get_custom_stream_status,
|
||||
run_mapped_rest_config,
|
||||
run_mapped_websocket_config,
|
||||
@@ -42,8 +42,6 @@ from app.services.custom_datasource_runtime import (
|
||||
stop_custom_stream,
|
||||
test_websocket_config,
|
||||
)
|
||||
|
||||
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
|
||||
from app.services.datasource_connectivity import (
|
||||
_resolve_aisstream_api_key,
|
||||
_resolve_spacetrack_credentials_with_override,
|
||||
@@ -57,7 +55,8 @@ from app.services.persistent_logs import record_audit_log
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
|
||||
DATASOURCE_MAPPING_PROMPT_KEY = "datasource.mapping"
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
|
||||
|
||||
def _user_role_value(user: User) -> str:
|
||||
@@ -124,7 +123,7 @@ class DataSourceConfigCreate(BaseModel):
|
||||
description: Optional[str] = None
|
||||
source_type: str = Field(..., description="rest, websocket, http, api, database")
|
||||
endpoint: str = Field(..., max_length=500)
|
||||
auth_type: str = Field(default="none", description="none, bearer, api_key, basic")
|
||||
auth_type: AuthType = Field(default=AuthType.NONE, description="none, bearer, api_key, basic")
|
||||
auth_config: dict = Field(default={})
|
||||
headers: dict = Field(default={})
|
||||
config: dict = Field(default={"timeout": 30, "retry": 3})
|
||||
@@ -135,7 +134,7 @@ class DataSourceConfigUpdate(BaseModel):
|
||||
description: Optional[str] = None
|
||||
source_type: Optional[str] = None
|
||||
endpoint: Optional[str] = Field(None, max_length=500)
|
||||
auth_type: Optional[str] = None
|
||||
auth_type: Optional[AuthType] = None
|
||||
auth_config: Optional[dict] = None
|
||||
headers: Optional[dict] = None
|
||||
config: Optional[dict] = None
|
||||
@@ -210,7 +209,7 @@ class MappingTemplateCreate(BaseModel):
|
||||
mapping_json: dict
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: str = Field(default="draft", pattern="^(draft|valid|invalid)$")
|
||||
validation_status: MappingValidationStatus = MappingValidationStatus.DRAFT
|
||||
is_active: bool = False
|
||||
|
||||
|
||||
@@ -219,7 +218,7 @@ class MappingTemplateUpdate(BaseModel):
|
||||
mapping_json: Optional[dict] = None
|
||||
sample_payload: Any | None = None
|
||||
sample_payload_hash: Optional[str] = None
|
||||
validation_status: Optional[str] = Field(default=None, pattern="^(draft|valid|invalid)$")
|
||||
validation_status: Optional[MappingValidationStatus] = None
|
||||
is_active: Optional[bool] = None
|
||||
|
||||
|
||||
@@ -918,6 +917,7 @@ async def get_datasource_target_schemas(
|
||||
@router.post("/mappings/propose")
|
||||
async def propose_datasource_mapping(
|
||||
payload: MappingProposeRequest,
|
||||
db: AsyncSession = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
ai_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
):
|
||||
|
||||
@@ -7,6 +7,7 @@ from sqlalchemy import func, or_, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.enums import JobStatus, SnapshotStatus
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -581,7 +582,7 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.status == "running")
|
||||
.where(CollectionTask.status == JobStatus.RUNNING.value)
|
||||
.order_by(CollectionTask.started_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
@@ -609,8 +610,8 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = JobStatus.FAILED.value
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
await db.commit()
|
||||
@@ -647,7 +648,7 @@ async def rollback_orphaned_running_task(
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.status = SnapshotStatus.CANCELLED.value
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
@@ -670,13 +671,13 @@ async def rollback_orphaned_running_task(
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
running_task.status = "cancelled"
|
||||
running_task.phase = "cancelled"
|
||||
running_task.status = JobStatus.CANCELLED.value
|
||||
running_task.phase = JobStatus.CANCELLED.value
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
|
||||
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_status = JobStatus.CANCELLED.value
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
@@ -711,7 +712,7 @@ async def fail_and_rollback_stale_running_task(
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "failed"
|
||||
snapshot.status = SnapshotStatus.FAILED.value
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
@@ -739,11 +740,11 @@ async def fail_and_rollback_stale_running_task(
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
|
||||
)
|
||||
running_task.status = "failed"
|
||||
running_task.phase = "failed"
|
||||
running_task.status = JobStatus.FAILED.value
|
||||
running_task.phase = JobStatus.FAILED.value
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_status = JobStatus.FAILED.value
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import delete, func, select, text
|
||||
@@ -27,6 +27,20 @@ from app.services.earth_news import (
|
||||
save_earth_news_sources_payload,
|
||||
test_news_source_config,
|
||||
)
|
||||
from app.services.earth_news_manual import (
|
||||
broadcast_manual_news_changed,
|
||||
create_manual_news_group,
|
||||
delete_manual_news_item,
|
||||
get_news_record_or_404,
|
||||
import_manual_news_items,
|
||||
list_news_groups,
|
||||
list_news_records,
|
||||
parse_manual_news_import_upload,
|
||||
rename_manual_news_group,
|
||||
reprocess_manual_news_item,
|
||||
serialize_news_record,
|
||||
upsert_manual_news_item,
|
||||
)
|
||||
from app.services.earth_boundaries import (
|
||||
EarthBoundaryBuildError,
|
||||
get_boundary_build_status,
|
||||
@@ -119,6 +133,26 @@ class EarthNewsSourceTestPayload(BaseModel):
|
||||
source: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class EarthNewsManualItemPayload(BaseModel):
|
||||
title: str = Field(default="", max_length=500)
|
||||
summary: str = Field(default="", max_length=1200)
|
||||
content: str = Field(default="", max_length=12000)
|
||||
url: str = Field(default="", max_length=2000)
|
||||
source: str = Field(default="", max_length=255)
|
||||
region: str = Field(default="global", max_length=80)
|
||||
published_at: str | None = None
|
||||
category: str = Field(default="other", max_length=80)
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
location: dict[str, Any] | None = None
|
||||
homepage_url: str = Field(default="", max_length=2000)
|
||||
content_language: str = Field(default="", max_length=32)
|
||||
group_id: str | None = Field(default=None, max_length=120)
|
||||
|
||||
|
||||
class EarthNewsManualGroupPayload(BaseModel):
|
||||
name: str = Field(default="", max_length=120)
|
||||
|
||||
|
||||
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
|
||||
merged = DEFAULT_EARTH_BRAND.copy()
|
||||
if payload:
|
||||
@@ -375,6 +409,162 @@ async def test_earth_news_source(
|
||||
return await test_news_source_config(payload.source, db=db)
|
||||
|
||||
|
||||
@router.get("/news-groups")
|
||||
async def list_earth_news_groups_admin(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await list_news_groups(db)
|
||||
|
||||
|
||||
@router.post("/news-groups")
|
||||
async def create_earth_news_group_admin(
|
||||
payload: EarthNewsManualGroupPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
group = await create_manual_news_group(db, payload.name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
return {"status": "ok", "group": group}
|
||||
|
||||
|
||||
@router.put("/news-groups/{group_id:path}")
|
||||
async def rename_earth_news_group_admin(
|
||||
group_id: str,
|
||||
payload: EarthNewsManualGroupPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
group = await rename_manual_news_group(db, group_id, payload.name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "group": group}
|
||||
|
||||
|
||||
@router.get("/news-items")
|
||||
async def list_earth_news_items_admin(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(50, ge=1, le=100),
|
||||
source_type: str | None = Query(None),
|
||||
region: str | None = Query(None),
|
||||
category: str | None = Query(None),
|
||||
status_filter: str | None = Query(None, alias="status"),
|
||||
group_id: str | None = Query(None),
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await list_news_records(
|
||||
db,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
source_type=source_type,
|
||||
region=region,
|
||||
category=category,
|
||||
status_filter=status_filter,
|
||||
group_id=group_id,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/news-items")
|
||||
async def create_earth_news_item_admin(
|
||||
payload: EarthNewsManualItemPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
result = await upsert_manual_news_item(db, payload.model_dump())
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
|
||||
|
||||
|
||||
@router.post("/news-items/import")
|
||||
async def import_earth_news_items_admin(
|
||||
file: UploadFile = File(...),
|
||||
group_id: str | None = Form(default=None),
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
payload = await parse_manual_news_import_upload(await file.read())
|
||||
result = await import_manual_news_items(db, payload, group_id=group_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", **result}
|
||||
|
||||
|
||||
@router.put("/news-items/{item_id:path}")
|
||||
async def update_earth_news_item_admin(
|
||||
item_id: str,
|
||||
payload: EarthNewsManualItemPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
existing = await get_news_record_or_404(db, item_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
try:
|
||||
result = await upsert_manual_news_item(
|
||||
db,
|
||||
payload.model_dump(),
|
||||
item_id_override=item_id,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
|
||||
|
||||
|
||||
@router.delete("/news-items/{item_id:path}")
|
||||
async def delete_earth_news_item_admin(
|
||||
item_id: str,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
deleted = await delete_manual_news_item(db, item_id)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "deleted", "id": item_id}
|
||||
|
||||
|
||||
@router.post("/news-items/{item_id:path}/reprocess")
|
||||
async def reprocess_earth_news_item_admin(
|
||||
item_id: str,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
existing = await get_news_record_or_404(db, item_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
try:
|
||||
queued = await reprocess_manual_news_item(db, item_id)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
|
||||
|
||||
|
||||
@router.get("/oobe-status")
|
||||
async def get_earth_oobe_status(
|
||||
current_user: User | None = Depends(_get_optional_current_user),
|
||||
|
||||
@@ -105,7 +105,7 @@ def _parse_layer_bbox(bbox: str) -> tuple[float, float, float, float]:
|
||||
@router.get("/vessels/snapshot")
|
||||
async def get_vessel_layer_snapshot(
|
||||
bbox: str = Query(..., description="lon_min,lat_min,lon_max,lat_max"),
|
||||
zoom: int = Query(..., ge=1, le=20),
|
||||
zoom: float = Query(..., ge=1, le=20),
|
||||
limit: int = Query(DEFAULT_LAYER_LIMIT, ge=1),
|
||||
vessel_type: Optional[str] = Query(None, alias="type"),
|
||||
since_minutes: int = Query(60, ge=1, le=1440),
|
||||
|
||||
@@ -3,15 +3,16 @@ from datetime import UTC, datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.enums import ProviderApi, TVSourceType, UserRole
|
||||
from app.core.security import get_current_user
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.config import settings as app_settings
|
||||
@@ -64,6 +65,7 @@ from app.services.llm_provider_catalog import (
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
@@ -73,7 +75,8 @@ router = APIRouter()
|
||||
logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {"admin", "super_admin"}
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -236,7 +239,7 @@ class TVStreamSourceUpdate(BaseModel):
|
||||
provider: str = Field(default="Unknown", max_length=100)
|
||||
region: str = Field(default="Global", max_length=100)
|
||||
language: str = Field(default="und", max_length=32)
|
||||
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
|
||||
source_type: TVSourceType = TVSourceType.IFRAME
|
||||
embed_url: str = ""
|
||||
stream_url: str = ""
|
||||
homepage_url: str = ""
|
||||
@@ -279,7 +282,7 @@ class AIProviderIntegrationUpdate(BaseModel):
|
||||
service_token: Optional[str] = None
|
||||
default_provider: Optional[str] = None
|
||||
provider: str = Field(default="minimax", max_length=80)
|
||||
provider_api: str = Field(default="anthropic-messages", max_length=80)
|
||||
provider_api: ProviderApi = ProviderApi.ANTHROPIC_MESSAGES
|
||||
base_url: str = Field(default="", max_length=500)
|
||||
model: str = Field(default="", max_length=200)
|
||||
api_key: Optional[str] = None
|
||||
@@ -344,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
|
||||
defaults = get_fallback_llm_provider_preset(
|
||||
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
|
||||
)
|
||||
else:
|
||||
defaults = DEFAULT_SETTINGS[category]
|
||||
merged = deepcopy(defaults)
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -381,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
|
||||
|
||||
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
|
||||
merged = merge_with_defaults(category, payload)
|
||||
record = await get_setting_record(db, category)
|
||||
if record is None:
|
||||
record = SystemSetting(category=category, payload=payload)
|
||||
@@ -390,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return merge_with_defaults(category, record.payload)
|
||||
return merged
|
||||
|
||||
|
||||
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
|
||||
@@ -423,7 +433,7 @@ def _get_provider_preset(provider: str) -> dict:
|
||||
except ValueError:
|
||||
return {
|
||||
"provider": provider,
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": ProviderApi.OPENAI_COMPLETIONS.value,
|
||||
"base_url": "",
|
||||
"model": "",
|
||||
"models": [],
|
||||
@@ -487,12 +497,12 @@ def _provider_defaults(provider: str) -> dict:
|
||||
preset = _get_provider_preset(provider)
|
||||
return {
|
||||
"provider": provider,
|
||||
"provider_api": preset.get("provider_api") or "openai-completions",
|
||||
"provider_api": preset.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
|
||||
"base_url": preset.get("base_url") or "",
|
||||
"model": preset.get("model") or "",
|
||||
"api_key": "",
|
||||
"max_tokens": (
|
||||
1200 if preset.get("provider_api") == "anthropic-messages" else 4096
|
||||
1200 if preset.get("provider_api") == ProviderApi.ANTHROPIC_MESSAGES.value else 4096
|
||||
),
|
||||
"anthropic_version": "2023-06-01",
|
||||
"model_provider_apis": preset.get("model_provider_apis") or {},
|
||||
@@ -654,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
|
||||
)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
|
||||
model_provider_apis = {
|
||||
**(provider_config.get("model_provider_apis") or {}),
|
||||
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
|
||||
}
|
||||
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": _resolve_service_token(normalized_ai)[0],
|
||||
@@ -665,14 +681,13 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
),
|
||||
"llm_config": {
|
||||
"provider": default_provider,
|
||||
"provider_api": provider_config.get("provider_api") or "anthropic-messages",
|
||||
"provider_api": provider_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
|
||||
"base_url": provider_config.get("base_url") or "",
|
||||
"model": provider_config.get("model") or "",
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
"model_provider_apis": model_provider_apis,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -752,156 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = str(llm_config.get("provider_api") or "").strip() or "openai-completions"
|
||||
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
base_url = str(llm_config.get("base_url") or "").strip()
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
if provider_api != "ollama-generate" and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == "ollama-generate":
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == "openai-completions":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == "anthropic-messages":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
result = {
|
||||
"success": False, "connected": False, "mode": "lightweight_models",
|
||||
"provider": provider, "model": model,
|
||||
}
|
||||
if not base_url or not model:
|
||||
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
|
||||
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
|
||||
return {**result, "message": "当前 provider 未配置 API Key。"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过；当前 provider 不提供可用的模型目录，已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
catalog = await fetch_model_catalog(
|
||||
provider, base_url, provider_api, api_key,
|
||||
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过；provider 模型目录未返回当前别名，已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用，但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {**result, "message": catalog_error_message(exc)}
|
||||
result.update({"url": catalog.url, "models_count": len(catalog.models)})
|
||||
if model.casefold() not in {item.casefold() for item in catalog.models}:
|
||||
return {**result, "message": f"模型目录查询成功，但当前模型不在目录中：{model}"}
|
||||
model_apis = llm_config.get("model_provider_apis") or {}
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
**result, "success": True, "connected": True,
|
||||
"provider_api": model_apis.get(model) or provider_api,
|
||||
"message": "模型目录查询成功，当前模型已找到；尚未执行生成调用。",
|
||||
}
|
||||
|
||||
|
||||
@@ -1165,7 +1059,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
|
||||
api_key, api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
providers_payload[provider_id] = {
|
||||
"provider": provider_id,
|
||||
"provider_api": provider_config.get("provider_api") or "openai-completions",
|
||||
"provider_api": provider_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value,
|
||||
"base_url": provider_config.get("base_url") or "",
|
||||
"model": provider_config.get("model") or "",
|
||||
"api_key": _mask_secret(api_key, api_key_source),
|
||||
@@ -1215,7 +1109,7 @@ async def serialize_external_integrations(db: AsyncSession) -> dict:
|
||||
"service_token": _mask_secret(*_resolve_service_token(normalized_ai)),
|
||||
"default_provider": default_provider,
|
||||
"provider": default_provider,
|
||||
"provider_api": display_llm_config.get("provider_api") or "anthropic-messages",
|
||||
"provider_api": display_llm_config.get("provider_api") or ProviderApi.ANTHROPIC_MESSAGES.value,
|
||||
"base_url": display_llm_config.get("base_url") or "https://api.minimaxi.com/anthropic",
|
||||
"model": display_llm_config.get("model") or "MiniMax-M2.7",
|
||||
"api_key": display_llm_config.get("api_key") or _mask_secret(None),
|
||||
@@ -1458,7 +1352,7 @@ async def update_smtp_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if current_user.role not in ("admin", "super_admin"):
|
||||
if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
|
||||
raise HTTPException(status_code=403, detail="Only administrators can change SMTP settings")
|
||||
current = await get_setting_payload(db, "smtp")
|
||||
merged = _build_smtp_payload(current, payload)
|
||||
@@ -1472,7 +1366,7 @@ async def test_smtp_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if current_user.role not in ("admin", "super_admin"):
|
||||
if current_user.role not in (UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value):
|
||||
raise HTTPException(status_code=403, detail="Only administrators can test SMTP settings")
|
||||
from app.services.email import EmailError, send_email
|
||||
|
||||
@@ -1685,7 +1579,8 @@ async def connect_ai_provider_integration(
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
|
||||
else "settings.ai_provider.connect.failed"),
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
@@ -1695,7 +1590,7 @@ async def connect_ai_provider_integration(
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
"connected": lightweight_result["connected"],
|
||||
},
|
||||
)
|
||||
return {
|
||||
@@ -2040,8 +1935,18 @@ async def reset_provider_credential_guide(
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
presets = list_fallback_llm_provider_presets()
|
||||
saved = await get_setting_payloads(
|
||||
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
|
||||
)
|
||||
return {
|
||||
"data": [
|
||||
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
|
||||
for preset in presets
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
@@ -2049,22 +1954,52 @@ async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
payload: AIProviderIntegrationUpdate | None = None,
|
||||
):
|
||||
try:
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
get_fallback_llm_provider_preset(provider_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
if payload is not None:
|
||||
if _normalize_provider_id(payload.provider) != provider_id:
|
||||
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
|
||||
ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
else:
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="请先配置当前供应商的 API Key，再刷新模型列表。",
|
||||
)
|
||||
try:
|
||||
refreshed = await refresh_llm_provider_preset(
|
||||
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
|
||||
provider_api=provider_config.get("provider_api"),
|
||||
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
|
||||
)
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
logger.warning(
|
||||
"LLM provider catalog refresh failed",
|
||||
extra={
|
||||
"event": "settings.ai_provider.catalog.failed",
|
||||
"context": {
|
||||
"provider": provider_id,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
|
||||
) from exc
|
||||
|
||||
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
|
||||
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
|
||||
return {"data": refreshed}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
|
||||
@@ -7,6 +7,7 @@ from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=100),
|
||||
q: str = Query("", max_length=200),
|
||||
selected_id: str | None = Query(None, max_length=200),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
|
||||
@@ -5,6 +5,7 @@ from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import get_current_user, get_password_hash
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
@@ -13,6 +14,8 @@ from app.schemas.user import UserCreate, UserUpdate
|
||||
router = APIRouter()
|
||||
|
||||
VALID_GATEKEEPER_GROUPS = {"docs_user", "docs_developer", "docs_admin"}
|
||||
ADMIN_ROLES = [UserRole.SUPER_ADMIN.value, UserRole.ADMIN.value]
|
||||
SUPER_ADMIN_ROLES = [UserRole.SUPER_ADMIN.value]
|
||||
|
||||
|
||||
def check_permission(current_user: User, required_roles: List[str]) -> bool:
|
||||
@@ -32,7 +35,7 @@ async def list_users(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not check_permission(current_user, ["super_admin", "admin"]):
|
||||
if not check_permission(current_user, ADMIN_ROLES):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Insufficient permissions",
|
||||
@@ -91,7 +94,7 @@ async def get_user(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
|
||||
if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Insufficient permissions",
|
||||
@@ -128,7 +131,7 @@ async def create_user(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not check_permission(current_user, ["super_admin"]):
|
||||
if not check_permission(current_user, SUPER_ADMIN_ROLES):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only super_admin can create users",
|
||||
@@ -196,18 +199,18 @@ async def update_user(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not check_permission(current_user, ["super_admin", "admin"]) and current_user.id != user_id:
|
||||
if not check_permission(current_user, ADMIN_ROLES) and current_user.id != user_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Insufficient permissions",
|
||||
)
|
||||
|
||||
if not check_permission(current_user, ["super_admin"]) and user_data.role is not None:
|
||||
if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.role is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only super_admin can change user role",
|
||||
)
|
||||
if not check_permission(current_user, ["super_admin"]) and user_data.gatekeeper_groups is not None:
|
||||
if not check_permission(current_user, SUPER_ADMIN_ROLES) and user_data.gatekeeper_groups is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only super_admin can change Gatekeeper groups",
|
||||
@@ -260,7 +263,7 @@ async def delete_user(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
if not check_permission(current_user, ["super_admin"]):
|
||||
if not check_permission(current_user, SUPER_ADMIN_ROLES):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Only super_admin can delete users",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Bounded vessel snapshot APIs for viewport-first consumers."""
|
||||
"""Bounded vessel snapshot APIs backed by the latest vessel state table."""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
@@ -14,8 +14,8 @@ router = APIRouter()
|
||||
|
||||
@router.get("/snapshot")
|
||||
async def get_vessel_snapshot(
|
||||
bbox: Optional[str] = Query(None, description="Viewport bbox as lon_min,lat_min,lon_max,lat_max"),
|
||||
zoom: int = Query(..., ge=1, le=20, description="Current map zoom level"),
|
||||
bbox: Optional[str] = Query(None, description="Snapshot bbox as lon_min,lat_min,lon_max,lat_max"),
|
||||
zoom: float = Query(..., ge=1, le=20, description="Current map zoom level"),
|
||||
type: Optional[str] = Query(
|
||||
None,
|
||||
description="Comma-separated vessel types: cargo,tanker,passenger,fishing,military,other",
|
||||
|
||||
@@ -18,6 +18,7 @@ from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
@@ -25,7 +26,7 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.vessel import AISSourceHealth, VesselPosition, VesselStatic
|
||||
from app.models.vessel import AISSourceHealth, VesselCurrentState, VesselPosition, VesselStatic
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.cable_graph import build_graph_from_data, CableGraph, haversine_distance
|
||||
from app.services.compute_center_locations import (
|
||||
@@ -47,11 +48,10 @@ from app.services.location.llm_fallback import (
|
||||
from app.services.persistent_logs import record_system_log
|
||||
from app.services.vessel_ais_aggregation import (
|
||||
build_field_conflict_candidates,
|
||||
count_unique_raw_vessel_mmsi,
|
||||
get_aggregated_vessel,
|
||||
get_aggregated_vessel_track,
|
||||
get_aggregated_vessels,
|
||||
get_aggregated_vessels_snapshot,
|
||||
get_current_vessels_snapshot,
|
||||
get_vessel_conflict_records,
|
||||
get_vessel_raw_observations,
|
||||
MAX_SNAPSHOT_LIMIT,
|
||||
@@ -75,7 +75,6 @@ TERRAIN_TILE_BATCH_MAX_ITEMS = 128
|
||||
TERRAIN_TILE_BATCH_CONCURRENCY = 16
|
||||
_terrain_tile_cache: OrderedDict[tuple[int, int, int], tuple[bytes, str, dict[str, str]]] = OrderedDict()
|
||||
VESSEL_NAME_FALLBACK_PATTERN = re.compile(r"^mmsi\s*\d+$", re.IGNORECASE)
|
||||
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED = True
|
||||
SECONDS_PER_MINUTE = 60
|
||||
BYTES_PER_MIB = 1024 * 1024
|
||||
CABLE_CACHE_FRESH_SECONDS = 6 * 60 * SECONDS_PER_MINUTE
|
||||
@@ -839,9 +838,14 @@ def convert_aggregated_vessels_to_geojson(vessels: List[dict[str, Any]]) -> Dict
|
||||
continue
|
||||
source_summary = {}
|
||||
for source, summary in (vessel.get("source_summary") or {}).items():
|
||||
latest_observed_at = summary.get("latest_observed_at")
|
||||
source_summary[source] = {
|
||||
**summary,
|
||||
"latest_observed_at": to_iso8601_utc(summary.get("latest_observed_at")),
|
||||
"latest_observed_at": (
|
||||
to_iso8601_utc(latest_observed_at)
|
||||
if isinstance(latest_observed_at, datetime)
|
||||
else latest_observed_at
|
||||
),
|
||||
}
|
||||
props = {
|
||||
"mmsi": vessel["mmsi"],
|
||||
@@ -1075,7 +1079,7 @@ async def build_vessel_snapshot_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
bbox: tuple[float, float, float, float] | None,
|
||||
zoom: int | None,
|
||||
zoom: float | None,
|
||||
type_filter: str | None,
|
||||
limit: int | None,
|
||||
since_minutes: int = 60,
|
||||
@@ -1117,6 +1121,7 @@ async def build_vessel_snapshot_response(
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
|
||||
snapshot_started_at = datetime.now(UTC)
|
||||
features, diagnostics = await _load_raw_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
@@ -1133,6 +1138,7 @@ async def build_vessel_snapshot_response(
|
||||
"features": features,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
"generated_at": to_iso8601_utc(snapshot_started_at),
|
||||
"diagnostics": {
|
||||
**diagnostics,
|
||||
"filtered_count": len(features),
|
||||
@@ -2340,58 +2346,35 @@ async def _load_raw_vessel_snapshot_features(
|
||||
observed_since: datetime,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||
if bbox is None:
|
||||
aggregated_vessels = await get_aggregated_vessels(
|
||||
db,
|
||||
limit=limit,
|
||||
observed_since=observed_since,
|
||||
)
|
||||
else:
|
||||
aggregated_vessels = await get_aggregated_vessels_snapshot(
|
||||
db,
|
||||
bbox=bbox,
|
||||
limit=limit,
|
||||
observed_since=observed_since,
|
||||
)
|
||||
raw_geojson = convert_aggregated_vessels_to_geojson(aggregated_vessels)
|
||||
raw_features = raw_geojson.get("features", [])
|
||||
features = raw_features
|
||||
legacy_features: list[dict[str, Any]] = []
|
||||
legacy_fallback_used = False
|
||||
if not raw_features and VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED:
|
||||
legacy_features = await _load_legacy_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
limit=limit,
|
||||
)
|
||||
features, _merge_diagnostics = _merge_vessel_features(raw_features, legacy_features)
|
||||
legacy_fallback_used = bool(legacy_features)
|
||||
|
||||
return [], {
|
||||
"source": "vessel_current_state",
|
||||
"current_state_count": 0,
|
||||
"final_unique_mmsi": 0,
|
||||
}
|
||||
current_vessels = await get_current_vessels_snapshot(
|
||||
db,
|
||||
bbox=bbox,
|
||||
limit=limit,
|
||||
observed_since=observed_since,
|
||||
)
|
||||
features = convert_aggregated_vessels_to_geojson(current_vessels).get("features", [])
|
||||
unique_mmsi = len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in features)
|
||||
if key is not None
|
||||
}
|
||||
)
|
||||
return features, {
|
||||
"raw_feature_count": len(raw_features),
|
||||
"raw_unique_mmsi": len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in raw_features)
|
||||
if key is not None
|
||||
}
|
||||
),
|
||||
"legacy_feature_count": len(legacy_features),
|
||||
"legacy_backfilled_mmsi": len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in legacy_features)
|
||||
if key is not None
|
||||
}
|
||||
),
|
||||
"legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
|
||||
"legacy_fallback_used": legacy_fallback_used,
|
||||
"final_unique_mmsi": len(
|
||||
{
|
||||
key
|
||||
for key in (_feature_mmsi_key(feature) for feature in features)
|
||||
if key is not None
|
||||
}
|
||||
),
|
||||
"source": "vessel_current_state",
|
||||
"current_state_count": len(features),
|
||||
"final_unique_mmsi": unique_mmsi,
|
||||
"raw_feature_count": 0,
|
||||
"raw_unique_mmsi": 0,
|
||||
"legacy_feature_count": 0,
|
||||
"legacy_backfilled_mmsi": 0,
|
||||
"legacy_fallback_enabled": False,
|
||||
"legacy_fallback_used": False,
|
||||
}
|
||||
|
||||
@router.get("/vessels/custom-supplements")
|
||||
@@ -2759,10 +2742,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
compute_center_count = supercomputer_count + gpu_cluster_count
|
||||
|
||||
active_incident_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active"),
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value),
|
||||
)
|
||||
active_anomaly_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active"),
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value),
|
||||
)
|
||||
active_incident_count = int(active_incident_result.scalar() or 0)
|
||||
active_anomaly_count = int(active_anomaly_result.scalar() or 0)
|
||||
@@ -2783,21 +2766,14 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
)
|
||||
else:
|
||||
bgp_collector_count = int(bgp_collector_scalar or 0)
|
||||
raw_unique_window_hours = 24
|
||||
raw_unique_mmsi = await count_unique_raw_vessel_mmsi(
|
||||
db,
|
||||
observed_since=datetime.now(UTC) - timedelta(hours=raw_unique_window_hours),
|
||||
vessel_current_window_minutes = 60
|
||||
vessel_current_result = await db.execute(
|
||||
select(func.count(VesselCurrentState.mmsi)).where(
|
||||
VesselCurrentState.observed_at
|
||||
>= datetime.now(UTC) - timedelta(minutes=vessel_current_window_minutes)
|
||||
)
|
||||
)
|
||||
legacy_unique_result = await db.execute(
|
||||
select(func.count(func.distinct(VesselPosition.mmsi)))
|
||||
)
|
||||
legacy_unique_mmsi = int(legacy_unique_result.scalar() or 0)
|
||||
legacy_fallback_active = (
|
||||
VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED
|
||||
and raw_unique_mmsi == 0
|
||||
and legacy_unique_mmsi > 0
|
||||
)
|
||||
vessel_count = legacy_unique_mmsi if legacy_fallback_active else raw_unique_mmsi
|
||||
vessel_count = int(vessel_current_result.scalar() or 0)
|
||||
aisstream_health = await db.get(AISSourceHealth, "aisstream_vessels")
|
||||
|
||||
return {
|
||||
@@ -2808,11 +2784,10 @@ async def _build_visualization_geo_summary(db: AsyncSession) -> dict[str, Any]:
|
||||
"satellite_count": satellite_count,
|
||||
"compute_center_count": compute_center_count,
|
||||
"vessel_count": vessel_count,
|
||||
"vessel_count_source": "legacy_fallback" if legacy_fallback_active else "raw_recent",
|
||||
"vessel_legacy_fallback_enabled": VESSEL_SNAPSHOT_LEGACY_FALLBACK_ENABLED,
|
||||
"vessel_raw_unique_mmsi": raw_unique_mmsi,
|
||||
"vessel_raw_unique_window_hours": raw_unique_window_hours,
|
||||
"vessel_legacy_unique_mmsi": legacy_unique_mmsi,
|
||||
"vessel_count_source": "vessel_current_state",
|
||||
"vessel_current_window_minutes": vessel_current_window_minutes,
|
||||
"vessel_raw_unique_mmsi": 0,
|
||||
"vessel_legacy_unique_mmsi": 0,
|
||||
"aisstream_connection_state": aisstream_health.connection_state if aisstream_health else None,
|
||||
"aisstream_last_seen_at": to_iso8601_utc(aisstream_health.last_seen_at) if aisstream_health else None,
|
||||
"aisstream_message_rate": aisstream_health.message_rate if aisstream_health else None,
|
||||
|
||||
@@ -9,6 +9,7 @@ from jose import jwt, JWTError
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.enums import UserRole
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
@@ -96,7 +97,7 @@ async def websocket_endpoint(
|
||||
"earth_news",
|
||||
EARTH_UPDATES_CHANNEL,
|
||||
]
|
||||
if user_role == "super_admin":
|
||||
if user_role == UserRole.SUPER_ADMIN.value:
|
||||
supported_channels = [*supported_channels, LOG_TAIL_CHANNEL]
|
||||
await manager.connect(websocket, user_id)
|
||||
|
||||
@@ -138,7 +139,7 @@ async def websocket_endpoint(
|
||||
if channel and channel not in channels:
|
||||
channels = [*channels, channel]
|
||||
if LOG_TAIL_CHANNEL in channels:
|
||||
if user_role != "super_admin":
|
||||
if user_role != UserRole.SUPER_ADMIN.value:
|
||||
await websocket.send_json(
|
||||
{
|
||||
"type": "subscription_error",
|
||||
@@ -160,7 +161,7 @@ async def websocket_endpoint(
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
if "vessels" in channels and "bbox" in payload_data:
|
||||
if "vessels" in channels:
|
||||
try:
|
||||
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
|
||||
227
backend/app/core/enums.py
Normal file
227
backend/app/core/enums.py
Normal file
@@ -0,0 +1,227 @@
|
||||
"""Stable backend protocol enums.
|
||||
|
||||
Database columns and JSON payloads continue to store the enum string values.
|
||||
Configurable identifiers, user-authored values, and open-ended taxonomies do
|
||||
not belong in this module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from enum import StrEnum
|
||||
from typing import TypeVar
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
EnumT = TypeVar("EnumT", bound=StrEnum)
|
||||
|
||||
|
||||
def parse_enum(enum_type: type[EnumT], value: object, default: EnumT) -> EnumT:
|
||||
"""Parse an external value without breaking reads of legacy data."""
|
||||
|
||||
if value is None or str(value).strip() == "":
|
||||
return default
|
||||
if isinstance(value, enum_type):
|
||||
return value
|
||||
try:
|
||||
return enum_type(str(value).strip().lower())
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"Unknown %s value %r; falling back to %s",
|
||||
enum_type.__name__,
|
||||
value,
|
||||
default.value,
|
||||
)
|
||||
return default
|
||||
|
||||
|
||||
class NewsImportanceLevel(StrEnum):
|
||||
LOW = "low"
|
||||
MEDIUM = "medium"
|
||||
HIGH = "high"
|
||||
CRITICAL = "critical"
|
||||
|
||||
|
||||
class BreakingLevel(StrEnum):
|
||||
NONE = "none"
|
||||
WATCH = "watch"
|
||||
BREAKING = "breaking"
|
||||
CRITICAL = "critical"
|
||||
|
||||
|
||||
class BreakingScope(StrEnum):
|
||||
REGIONAL = "regional"
|
||||
GLOBAL = "global"
|
||||
|
||||
|
||||
class BreakingSource(StrEnum):
|
||||
RULES = "rules"
|
||||
AI = "ai"
|
||||
MANUAL = "manual"
|
||||
MULTI_SOURCE = "multi_source"
|
||||
|
||||
|
||||
class NewsSourceType(StrEnum):
|
||||
RSS = "rss"
|
||||
ATOM = "atom"
|
||||
AGGREGATED = "aggregated"
|
||||
REFERENCE = "reference"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class NewsEnrichmentStatus(StrEnum):
|
||||
PENDING = "pending"
|
||||
QUEUED = "queued"
|
||||
ATTEMPTED = "attempted"
|
||||
SUCCESS = "success"
|
||||
CONTENT_ONLY = "content_only"
|
||||
LOCATION_ONLY = "location_only"
|
||||
UNAVAILABLE = "unavailable"
|
||||
PROVIDER_ERROR = "provider_error"
|
||||
PARSE_ERROR = "parse_error"
|
||||
NO_RESULT = "no_result"
|
||||
|
||||
|
||||
class NewsMarketImpact(StrEnum):
|
||||
NONE = "none"
|
||||
SECTOR = "sector"
|
||||
NATIONAL = "national"
|
||||
GLOBAL = "global"
|
||||
|
||||
|
||||
class NewsTaggingSource(StrEnum):
|
||||
RULES = "rules"
|
||||
AI = "ai"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class JobType(StrEnum):
|
||||
COLLECT = "collect"
|
||||
CLEAR_DATA = "clear_data"
|
||||
CLEAR_CACHE = "clear_cache"
|
||||
EARTH_REFRESH = "earth_refresh"
|
||||
|
||||
|
||||
class JobStatus(StrEnum):
|
||||
QUEUED = "queued"
|
||||
RUNNING = "running"
|
||||
CANCELLING = "cancelling"
|
||||
SUCCESS = "success"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
|
||||
|
||||
class RollbackPolicy(StrEnum):
|
||||
KEEP_COMMITTED_BATCHES = "keep_committed_batches"
|
||||
|
||||
|
||||
class MappingValidationStatus(StrEnum):
|
||||
DRAFT = "draft"
|
||||
VALID = "valid"
|
||||
INVALID = "invalid"
|
||||
|
||||
|
||||
class SnapshotStatus(StrEnum):
|
||||
RUNNING = "running"
|
||||
SUCCESS = "success"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
|
||||
|
||||
class DatasourceRunStatus(StrEnum):
|
||||
RUNNING = "running"
|
||||
NOT_RUN = "not_run"
|
||||
COLLECTED = "collected"
|
||||
UNCOLLECTED = "uncollected"
|
||||
|
||||
|
||||
class ProviderApi(StrEnum):
|
||||
ANTHROPIC_MESSAGES = "anthropic-messages"
|
||||
OPENAI_COMPLETIONS = "openai-completions"
|
||||
OPENAI_RESPONSES = "openai-responses"
|
||||
OLLAMA_GENERATE = "ollama-generate"
|
||||
|
||||
|
||||
class PlaygroundMessageRole(StrEnum):
|
||||
SYSTEM = "system"
|
||||
USER = "user"
|
||||
ASSISTANT = "assistant"
|
||||
TOOL = "tool"
|
||||
|
||||
|
||||
class PlaygroundMessageKind(StrEnum):
|
||||
MESSAGE = "message"
|
||||
THINKING = "thinking"
|
||||
ERROR = "error"
|
||||
STATUS = "status"
|
||||
|
||||
|
||||
class PlaygroundMessageStatus(StrEnum):
|
||||
PENDING = "pending"
|
||||
THINKING = "thinking"
|
||||
ANSWERING = "answering"
|
||||
DONE = "done"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
ERROR = "error"
|
||||
STOPPED = "stopped"
|
||||
|
||||
|
||||
class OtpPurpose(StrEnum):
|
||||
REGISTER = "register"
|
||||
VERIFY_EMAIL = "verify_email"
|
||||
RESET_PASSWORD = "reset_password"
|
||||
|
||||
|
||||
class UserRole(StrEnum):
|
||||
VIEWER = "viewer"
|
||||
ADMIN = "admin"
|
||||
SUPER_ADMIN = "super_admin"
|
||||
|
||||
|
||||
class AlertSeverity(StrEnum):
|
||||
CRITICAL = "critical"
|
||||
WARNING = "warning"
|
||||
INFO = "info"
|
||||
|
||||
|
||||
class AlertStatus(StrEnum):
|
||||
ACTIVE = "active"
|
||||
ACKNOWLEDGED = "acknowledged"
|
||||
RESOLVED = "resolved"
|
||||
|
||||
|
||||
class BGPStatus(StrEnum):
|
||||
ACTIVE = "active"
|
||||
ACKNOWLEDGED = "acknowledged"
|
||||
RESOLVED = "resolved"
|
||||
|
||||
|
||||
class LogLevel(StrEnum):
|
||||
ALL = "all"
|
||||
ERROR = "error"
|
||||
WARNING = "warning"
|
||||
INFO = "info"
|
||||
DEBUG = "debug"
|
||||
|
||||
|
||||
class ConnectionState(StrEnum):
|
||||
DISCONNECTED = "disconnected"
|
||||
CONNECTING = "connecting"
|
||||
CONNECTED = "connected"
|
||||
ERROR = "error"
|
||||
|
||||
|
||||
class AuthType(StrEnum):
|
||||
NONE = "none"
|
||||
BEARER = "bearer"
|
||||
API_KEY = "api_key"
|
||||
BASIC = "basic"
|
||||
|
||||
|
||||
class TVSourceType(StrEnum):
|
||||
IFRAME = "iframe"
|
||||
HLS = "hls"
|
||||
VIDEO = "video"
|
||||
EXTERNAL = "external"
|
||||
YOUTUBE = "youtube"
|
||||
@@ -4,11 +4,14 @@ import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Dict, Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
|
||||
logger = get_logger(__name__, service="websocket")
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
@@ -114,16 +117,22 @@ class DataBroadcaster:
|
||||
return
|
||||
pending = self._pending_vessel_updates
|
||||
self._pending_vessel_updates = {}
|
||||
vessels = []
|
||||
for item in pending.values():
|
||||
vessel = dict(item)
|
||||
source = vessel.pop("_source", None)
|
||||
action = vessel.pop("_action", "upsert")
|
||||
created = vessel.pop("_created", None)
|
||||
vessel["source"] = source
|
||||
vessel["action"] = action
|
||||
vessel["created"] = created
|
||||
vessels.append(vessel)
|
||||
try:
|
||||
vessels = await self._load_current_vessel_updates(list(pending))
|
||||
except Exception:
|
||||
# Preserve newer updates that arrived during the failed database read.
|
||||
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
|
||||
raise
|
||||
if not vessels:
|
||||
return
|
||||
vessels = [
|
||||
{
|
||||
**vessel,
|
||||
"action": vessel.get("action", "upsert"),
|
||||
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
|
||||
}
|
||||
for vessel in vessels
|
||||
]
|
||||
await manager.broadcast_vessels(
|
||||
{
|
||||
"action": "upsert",
|
||||
@@ -133,12 +142,31 @@ class DataBroadcaster:
|
||||
}
|
||||
)
|
||||
|
||||
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
|
||||
|
||||
mmsis = [int(key) for key in keys if key.isdigit()]
|
||||
vessels = []
|
||||
async with async_session_factory() as db:
|
||||
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
|
||||
vessels.extend(await get_current_vessels_by_mmsi(
|
||||
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
|
||||
))
|
||||
present = {int(vessel["mmsi"]) for vessel in vessels}
|
||||
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
|
||||
return vessels
|
||||
|
||||
async def broadcast_vessels_periodically(self):
|
||||
while self.running:
|
||||
try:
|
||||
await self.flush_vessel_updates()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to flush vessel updates",
|
||||
event="vessels.broadcast.failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(self._vessel_flush_interval)
|
||||
|
||||
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
|
||||
|
||||
@@ -63,6 +63,8 @@ class ConnectionManager:
|
||||
|
||||
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
|
||||
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
|
||||
if channel == "vessels":
|
||||
self.vessel_subscriptions.pop(websocket, None)
|
||||
subscribers = self.channel_subscriptions.get(channel)
|
||||
if subscribers is not None:
|
||||
subscribers.discard(websocket)
|
||||
@@ -88,7 +90,8 @@ class ConnectionManager:
|
||||
return subscription
|
||||
|
||||
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
bbox = config.get("bbox")
|
||||
global_scope = config.get("scope") == "global"
|
||||
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
|
||||
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
|
||||
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
|
||||
try:
|
||||
@@ -103,7 +106,7 @@ class ConnectionManager:
|
||||
raise ValueError("bbox longitude values must be between -180 and 180")
|
||||
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
|
||||
raise ValueError("bbox latitude values must be between -90 and 90")
|
||||
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
|
||||
|
||||
zoom = int(config.get("zoom") or 1)
|
||||
@@ -116,10 +119,11 @@ class ConnectionManager:
|
||||
if str(item).strip()
|
||||
}
|
||||
return {
|
||||
"scope": "global" if global_scope else "viewport",
|
||||
"bbox": (lon_min, lat_min, lon_max, lat_max),
|
||||
"zoom": zoom,
|
||||
"limit": limit,
|
||||
"type": vessel_types,
|
||||
"type": sorted(vessel_types),
|
||||
"last_sent_at": None,
|
||||
}
|
||||
|
||||
@@ -152,7 +156,9 @@ class ConnectionManager:
|
||||
vessel
|
||||
for vessel in vessels
|
||||
if self._vessel_matches_subscription(vessel, subscription)
|
||||
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
|
||||
]
|
||||
if subscription.get("scope") != "global":
|
||||
matched = matched[:subscription["limit"]]
|
||||
if not matched:
|
||||
continue
|
||||
subscription["last_sent_at"] = datetime.now(UTC)
|
||||
@@ -162,16 +168,24 @@ class ConnectionManager:
|
||||
"timestamp": subscription["last_sent_at"].isoformat(),
|
||||
"payload": {
|
||||
**data,
|
||||
"vessels": matched,
|
||||
"vessels": [],
|
||||
"subscription": {
|
||||
"bbox": list(subscription["bbox"]),
|
||||
"zoom": subscription["zoom"],
|
||||
"limit": subscription["limit"],
|
||||
"scope": subscription.get("scope", "viewport"),
|
||||
},
|
||||
},
|
||||
}
|
||||
try:
|
||||
await connection.send_json(message)
|
||||
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
|
||||
await connection.send_json({
|
||||
**message,
|
||||
"payload": {
|
||||
**message["payload"],
|
||||
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
|
||||
},
|
||||
})
|
||||
except Exception:
|
||||
self.unsubscribe_all(connection)
|
||||
|
||||
@@ -180,6 +194,8 @@ class ConnectionManager:
|
||||
vessel: dict[str, Any],
|
||||
subscription: dict[str, Any],
|
||||
) -> bool:
|
||||
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
|
||||
return True
|
||||
try:
|
||||
lon = float(vessel.get("lon"))
|
||||
lat = float(vessel.get("lat"))
|
||||
|
||||
@@ -626,6 +626,7 @@ async def init_db():
|
||||
"bgp_collector_locations",
|
||||
"vessel_static",
|
||||
"vessel_position",
|
||||
"vessel_current_state",
|
||||
"ais_raw_observations",
|
||||
"ais_source_health",
|
||||
"compute_center_locations",
|
||||
@@ -786,6 +787,22 @@ async def init_db():
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_vessel_current_bbox
|
||||
ON vessel_current_state (lon, lat)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_vessel_current_observed
|
||||
ON vessel_current_state (observed_at DESC)
|
||||
"""
|
||||
)
|
||||
)
|
||||
await conn.execute(
|
||||
text(
|
||||
"""
|
||||
|
||||
@@ -1,26 +1,12 @@
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import Column, Integer, String, DateTime, Text, ForeignKey, Enum as SQLEnum
|
||||
from sqlalchemy.orm import relationship
|
||||
from sqlalchemy import Column, Integer, String, DateTime, Text, Enum as SQLEnum
|
||||
|
||||
from app.core.enums import AlertSeverity, AlertStatus
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class AlertSeverity(str, Enum):
|
||||
CRITICAL = "critical"
|
||||
WARNING = "warning"
|
||||
INFO = "info"
|
||||
|
||||
|
||||
class AlertStatus(str, Enum):
|
||||
ACTIVE = "active"
|
||||
ACKNOWLEDGED = "acknowledged"
|
||||
RESOLVED = "resolved"
|
||||
|
||||
|
||||
class Alert(Base):
|
||||
__tablename__ = "alerts"
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ from datetime import datetime
|
||||
|
||||
from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
|
||||
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
@@ -17,7 +18,7 @@ class BGPAnomaly(Base):
|
||||
source = Column(String(100), nullable=False, index=True)
|
||||
anomaly_type = Column(String(50), nullable=False, index=True)
|
||||
severity = Column(String(20), nullable=False, index=True)
|
||||
status = Column(String(20), nullable=False, default="active", index=True)
|
||||
status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
|
||||
entity_key = Column(String(255), nullable=False, index=True)
|
||||
prefix = Column(String(64), nullable=True, index=True)
|
||||
origin_asn = Column(Integer, nullable=True, index=True)
|
||||
|
||||
@@ -4,6 +4,7 @@ from datetime import datetime
|
||||
|
||||
from sqlalchemy import Column, DateTime, Float, ForeignKey, Index, Integer, JSON, String, Text
|
||||
|
||||
from app.core.enums import BGPStatus
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
@@ -20,7 +21,7 @@ class BGPIncident(Base):
|
||||
title = Column(String(255), nullable=False)
|
||||
summary = Column(Text, nullable=False)
|
||||
severity = Column(String(20), nullable=False, index=True)
|
||||
status = Column(String(20), nullable=False, default="active", index=True)
|
||||
status = Column(String(20), nullable=False, default=BGPStatus.ACTIVE.value, index=True)
|
||||
confidence = Column(Float, nullable=False, default=0.5)
|
||||
started_at = Column(DateTime(timezone=True), nullable=False, default=datetime.utcnow, index=True)
|
||||
ended_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import SnapshotStatus
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
@@ -16,7 +17,7 @@ class DataSnapshot(Base):
|
||||
started_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
completed_at = Column(DateTime(timezone=True), nullable=True)
|
||||
record_count = Column(Integer, default=0)
|
||||
status = Column(String(20), nullable=False, default="running")
|
||||
status = Column(String(20), nullable=False, default=SnapshotStatus.RUNNING.value)
|
||||
is_current = Column(Boolean, default=True, index=True)
|
||||
parent_snapshot_id = Column(Integer, ForeignKey("data_snapshots.id"), nullable=True, index=True)
|
||||
summary = Column(JSON, default={})
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, JSON, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import MappingValidationStatus
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
@@ -19,7 +20,7 @@ class DataSourceMappingTemplate(Base):
|
||||
target_schema = Column(String(80), nullable=False, index=True)
|
||||
mapping_json = Column(JSON, nullable=False, default={})
|
||||
sample_payload_hash = Column(String(64), nullable=True)
|
||||
validation_status = Column(String(30), nullable=False, default="draft")
|
||||
validation_status = Column(String(30), nullable=False, default=MappingValidationStatus.DRAFT.value)
|
||||
version = Column(Integer, nullable=False, default=1)
|
||||
is_active = Column(Boolean, nullable=False, default=False, index=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import PlaygroundMessageKind, PlaygroundMessageStatus
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
@@ -13,8 +14,8 @@ class PlaygroundMessage(Base):
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
|
||||
role = Column(String(20), nullable=False)
|
||||
kind = Column(String(20), nullable=False, default="message")
|
||||
status = Column(String(20), nullable=False, default="done")
|
||||
kind = Column(String(20), nullable=False, default=PlaygroundMessageKind.MESSAGE.value)
|
||||
status = Column(String(20), nullable=False, default=PlaygroundMessageStatus.DONE.value)
|
||||
title = Column(String(255), nullable=True)
|
||||
content = Column(Text, nullable=False, default="")
|
||||
thinking_content = Column(Text, nullable=False, default="")
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Float, Integer, JSON, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import JobStatus, JobType, RollbackPolicy
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
@@ -12,9 +13,9 @@ class CollectionTask(Base):
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
datasource_id = Column(Integer, nullable=False, index=True)
|
||||
source = Column(String(100), nullable=True, index=True)
|
||||
task_type = Column(String(30), nullable=False, default="collect", index=True)
|
||||
task_type = Column(String(30), nullable=False, default=JobType.COLLECT.value, index=True)
|
||||
status = Column(String(20), nullable=False) # queued, running, cancelling, success, failed, cancelled
|
||||
phase = Column(String(30), default="queued")
|
||||
phase = Column(String(30), default=JobStatus.QUEUED.value)
|
||||
phase_progress = Column(Float)
|
||||
phase_message = Column(String(255))
|
||||
phase_current = Column(BigInteger)
|
||||
@@ -27,7 +28,7 @@ class CollectionTask(Base):
|
||||
progress = Column(Float, default=0.0) # Progress percentage (0-100)
|
||||
error_message = Column(Text)
|
||||
payload = Column(JSON, default=dict)
|
||||
rollback_policy = Column(String(40), nullable=False, default="keep_committed_batches")
|
||||
rollback_policy = Column(String(40), nullable=False, default=RollbackPolicy.KEEP_COMMITTED_BATCHES.value)
|
||||
dedupe_key = Column(String(180), nullable=True, index=True)
|
||||
worker_id = Column(String(120), nullable=True, index=True)
|
||||
locked_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from sqlalchemy import Boolean, Column, DateTime, Integer, JSON, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
@@ -11,7 +12,7 @@ class User(Base):
|
||||
username = Column(String(50), unique=True, index=True, nullable=False)
|
||||
email = Column(String(255), unique=True, index=True, nullable=False)
|
||||
password_hash = Column(String(255), nullable=False)
|
||||
role = Column(String(20), default="viewer")
|
||||
role = Column(String(20), default=UserRole.VIEWER.value)
|
||||
gatekeeper_groups = Column(JSON, default=list)
|
||||
is_active = Column(Boolean, default=True)
|
||||
email_verified = Column(Boolean, default=False, nullable=False)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from sqlalchemy import BigInteger, Column, DateTime, Float, Index, Integer, JSON, SmallInteger, String
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.core.enums import ConnectionState
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import Base
|
||||
|
||||
@@ -75,6 +76,67 @@ class VesselPosition(Base):
|
||||
}
|
||||
|
||||
|
||||
class VesselCurrentState(Base):
|
||||
"""Latest renderable state for one vessel, independent from AIS history."""
|
||||
|
||||
__tablename__ = "vessel_current_state"
|
||||
|
||||
mmsi = Column(BigInteger, primary_key=True)
|
||||
lat = Column(Float, nullable=False)
|
||||
lon = Column(Float, nullable=False)
|
||||
sog = Column(Float, nullable=True)
|
||||
cog = Column(Float, nullable=True)
|
||||
heading = Column(SmallInteger, nullable=True)
|
||||
nav_status = Column(SmallInteger, nullable=True, index=True)
|
||||
name = Column(String(128), nullable=True)
|
||||
callsign = Column(String(16), nullable=True)
|
||||
vessel_type = Column(SmallInteger, nullable=True, index=True)
|
||||
vessel_type_name = Column(String(64), nullable=True, index=True)
|
||||
flag = Column(String(4), nullable=True, index=True)
|
||||
length = Column(Float, nullable=True)
|
||||
width = Column(Float, nullable=True)
|
||||
draught = Column(Float, nullable=True)
|
||||
imo = Column(BigInteger, nullable=True)
|
||||
source = Column(String(100), nullable=False, index=True)
|
||||
observed_at = Column(DateTime(timezone=True), nullable=False)
|
||||
field_sources = Column(JSON, default=dict)
|
||||
selected_reasons = Column(JSON, default=dict)
|
||||
source_summary = Column(JSON, default=dict)
|
||||
quality_flags = Column(JSON, default=list)
|
||||
updated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_vessel_current_bbox", "lon", "lat"),
|
||||
Index("idx_vessel_current_observed", "observed_at"),
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"sog": self.sog,
|
||||
"cog": self.cog,
|
||||
"heading": self.heading,
|
||||
"nav_status": self.nav_status,
|
||||
"name": self.name,
|
||||
"callsign": self.callsign,
|
||||
"vessel_type": self.vessel_type,
|
||||
"vessel_type_name": self.vessel_type_name,
|
||||
"flag": self.flag,
|
||||
"length": self.length,
|
||||
"width": self.width,
|
||||
"draught": self.draught,
|
||||
"imo": self.imo,
|
||||
"source": self.source,
|
||||
"received_at": self.observed_at,
|
||||
"field_sources": self.field_sources or {},
|
||||
"selected_reasons": self.selected_reasons or {},
|
||||
"source_summary": self.source_summary or {},
|
||||
"quality_flags": self.quality_flags or [],
|
||||
}
|
||||
|
||||
|
||||
class AISRawObservation(Base):
|
||||
"""Source-level AIS fact before aggregation and conflict resolution."""
|
||||
|
||||
@@ -165,7 +227,7 @@ class AISSourceHealth(Base):
|
||||
__tablename__ = "ais_source_health"
|
||||
|
||||
source = Column(String(100), primary_key=True)
|
||||
connection_state = Column(String(32), nullable=False, default="disconnected", index=True)
|
||||
connection_state = Column(String(32), nullable=False, default=ConnectionState.DISCONNECTED.value, index=True)
|
||||
last_seen_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
last_success_at = Column(DateTime(timezone=True), nullable=True, index=True)
|
||||
last_error = Column(String(500), nullable=True)
|
||||
|
||||
@@ -2,6 +2,7 @@ from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.core.enums import PlaygroundMessageKind, PlaygroundMessageRole, PlaygroundMessageStatus
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
@@ -110,9 +111,9 @@ class PlaygroundSessionUpsertRequest(BaseModel):
|
||||
|
||||
class PlaygroundMessageRecord(BaseModel):
|
||||
id: str
|
||||
role: str
|
||||
kind: str = "message"
|
||||
status: str = "done"
|
||||
role: PlaygroundMessageRole
|
||||
kind: PlaygroundMessageKind = PlaygroundMessageKind.MESSAGE
|
||||
status: PlaygroundMessageStatus = PlaygroundMessageStatus.DONE
|
||||
title: str | None = None
|
||||
content: str = ""
|
||||
thinking_content: str = ""
|
||||
|
||||
@@ -3,6 +3,7 @@ from typing import Optional
|
||||
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
|
||||
from app.core.enums import OtpPurpose, UserRole
|
||||
|
||||
class UserBase(BaseModel):
|
||||
username: str
|
||||
@@ -11,13 +12,13 @@ class UserBase(BaseModel):
|
||||
|
||||
class UserCreate(UserBase):
|
||||
password: str = Field(..., min_length=8)
|
||||
role: str = "viewer"
|
||||
role: UserRole = UserRole.VIEWER
|
||||
gatekeeper_groups: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class UserUpdate(BaseModel):
|
||||
email: Optional[EmailStr] = None
|
||||
role: Optional[str] = None
|
||||
role: Optional[UserRole] = None
|
||||
gatekeeper_groups: Optional[list[str]] = None
|
||||
is_active: Optional[bool] = None
|
||||
|
||||
@@ -59,7 +60,7 @@ class VerifyEmailRequest(BaseModel):
|
||||
|
||||
class ResendCodeRequest(BaseModel):
|
||||
email: EmailStr
|
||||
purpose: str = Field(default="register", pattern="^(register|verify_email|reset_password)$")
|
||||
purpose: OtpPurpose = OtpPurpose.REGISTER
|
||||
|
||||
|
||||
class ForgotPasswordRequest(BaseModel):
|
||||
|
||||
@@ -6,6 +6,7 @@ from collections import Counter, defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from app.core.enums import BGPStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
|
||||
|
||||
@@ -127,7 +128,7 @@ def detect_origin_change_anomalies(
|
||||
source=source,
|
||||
anomaly_type=anomaly_type,
|
||||
severity=severity,
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
entity_key=f"{anomaly_type}:{prefix}:{new_origin}",
|
||||
prefix=prefix,
|
||||
origin_asn=sorted(historic)[0] if historic else None,
|
||||
@@ -197,7 +198,7 @@ def detect_more_specific_burst_anomalies(
|
||||
source=source,
|
||||
anomaly_type="more_specific_burst",
|
||||
severity="high",
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
entity_key=f"more_specific_burst:{root_prefix}:{len(unique_prefixes)}:{len(related_collectors)}",
|
||||
prefix=sample.get("prefix"),
|
||||
origin_asn=sample.get("origin_asn"),
|
||||
@@ -267,7 +268,7 @@ def detect_mass_withdrawal_anomalies(
|
||||
source=source,
|
||||
anomaly_type="mass_withdrawal",
|
||||
severity=severity,
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
entity_key=f"mass_withdrawal:{prefix}:{origin_asn}:{len(related_collectors)}:{count}",
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
@@ -354,7 +355,7 @@ def detect_route_leak_anomalies(
|
||||
source=source,
|
||||
anomaly_type="route_leak_candidate",
|
||||
severity="high" if max_path_length >= dominant_length + 3 else "medium",
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
entity_key=f"route_leak_candidate:{prefix}:{max_path_length}:{len(related_collectors)}",
|
||||
prefix=prefix,
|
||||
origin_asn=sample_metadata.get("origin_asn"),
|
||||
@@ -435,7 +436,7 @@ def detect_path_flap_anomalies(
|
||||
source=source,
|
||||
anomaly_type="path_flap",
|
||||
severity=severity,
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
entity_key=f"path_flap:{prefix}:{transitions}:{len(distinct_paths)}",
|
||||
prefix=prefix,
|
||||
origin_asn=sample_metadata.get("origin_asn"),
|
||||
|
||||
@@ -9,6 +9,7 @@ from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.enums import BGPStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.collected_data import CollectedData
|
||||
@@ -290,7 +291,7 @@ async def create_bgp_incidents_for_anomalies(
|
||||
existing.title = title
|
||||
existing.summary = summary
|
||||
existing.severity = severity
|
||||
existing.status = "active"
|
||||
existing.status = BGPStatus.ACTIVE.value
|
||||
existing.confidence = confidence
|
||||
existing.started_at = primary.started_at or existing.started_at or datetime.now(UTC)
|
||||
existing.ended_at = None
|
||||
@@ -313,7 +314,7 @@ async def create_bgp_incidents_for_anomalies(
|
||||
title=title,
|
||||
summary=summary,
|
||||
severity=severity,
|
||||
status="active",
|
||||
status=BGPStatus.ACTIVE.value,
|
||||
confidence=confidence,
|
||||
started_at=primary.started_at or datetime.now(UTC),
|
||||
affected_prefixes=prefixes,
|
||||
|
||||
@@ -12,11 +12,11 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.collected_data_fields import build_dynamic_metadata, get_record_field
|
||||
from app.core.countries import normalize_country
|
||||
from app.core.enums import JobStatus, SnapshotStatus
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.services.business_logs import emit_business_log, exception_context
|
||||
from app.services.earth_layer_adapters import get_earth_update_layers_for_source
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
@@ -238,7 +238,7 @@ class BaseCollector(ABC):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
parent_snapshot_id = snapshot.parent_snapshot_id
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.status = SnapshotStatus.CANCELLED.value
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
summary = dict(snapshot.summary or {})
|
||||
@@ -308,7 +308,7 @@ class BaseCollector(ABC):
|
||||
task.datasource_id = datasource_id
|
||||
task.source = task.source or self.name
|
||||
task.task_type = task.task_type or "collect"
|
||||
task.status = "running"
|
||||
task.status = JobStatus.RUNNING.value
|
||||
task.phase = "queued"
|
||||
task.started_at = task.started_at or start_time
|
||||
task.completed_at = None
|
||||
@@ -393,7 +393,7 @@ class BaseCollector(ABC):
|
||||
},
|
||||
)
|
||||
|
||||
task.status = "success"
|
||||
task.status = JobStatus.SUCCESS.value
|
||||
task.phase = "completed"
|
||||
task.phase_progress = 100.0
|
||||
task.phase_message = "采集完成"
|
||||
@@ -428,7 +428,7 @@ class BaseCollector(ABC):
|
||||
}
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.status = JobStatus.CANCELLED.value
|
||||
task.phase = "cancelled"
|
||||
task.phase_message = "采集已取消"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
@@ -456,7 +456,7 @@ class BaseCollector(ABC):
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = "failed"
|
||||
task.phase_message = str(e)
|
||||
task.error_message = str(e)
|
||||
@@ -464,7 +464,7 @@ class BaseCollector(ABC):
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.status = "failed"
|
||||
snapshot.status = SnapshotStatus.FAILED.value
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
snapshot.summary = {"error": str(e)}
|
||||
await db.commit()
|
||||
@@ -509,7 +509,7 @@ class BaseCollector(ABC):
|
||||
if snapshot:
|
||||
snapshot.record_count = 0
|
||||
snapshot.summary = {"created": 0, "updated": 0, "unchanged": 0}
|
||||
snapshot.status = "success"
|
||||
snapshot.status = SnapshotStatus.SUCCESS.value
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
await db.commit()
|
||||
return 0
|
||||
@@ -642,7 +642,7 @@ class BaseCollector(ABC):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.record_count = records_added
|
||||
snapshot.status = "success"
|
||||
snapshot.status = SnapshotStatus.SUCCESS.value
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
snapshot.summary = {
|
||||
"created": created_count,
|
||||
|
||||
@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
if max_sources > 0 and len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
import os
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
# PeeringDB API key - read from environment variable
|
||||
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class PeeringDBIXPCollector(HTTPCollector):
|
||||
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
"""Space-Track TLE Collector
|
||||
"""Space-Track TLE Collector.
|
||||
|
||||
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
|
||||
API documentation: https://www.space-track.org/documentation
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.logging import get_logger
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class SpaceTrackTLECollector(BaseCollector):
|
||||
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
password = settings.SPACETRACK_PASSWORD
|
||||
|
||||
if not username or not password:
|
||||
print("SPACETRACK: No credentials configured, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track credentials are not configured; using sample data",
|
||||
event="collector.spacetrack.credentials_missing",
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch started",
|
||||
event="collector.spacetrack.fetch.start",
|
||||
)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
logger.info_event(
|
||||
"Space-Track login response received",
|
||||
event="collector.spacetrack.login.response",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
|
||||
if login_response.status_code == 403:
|
||||
print("SPACETRACK: Trying alternate login method...")
|
||||
logger.warning_event(
|
||||
"Space-Track login returned forbidden; trying alternate method",
|
||||
event="collector.spacetrack.login.forbidden",
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
) as alt_client:
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
"password": password,
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
logger.info_event(
|
||||
"Space-Track alternate login response received",
|
||||
event="collector.spacetrack.alt_login.response",
|
||||
context={"status_code": alt_login.status_code},
|
||||
)
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
logger.info_event(
|
||||
"Space-Track alternate query completed",
|
||||
event="collector.spacetrack.alt_query.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track login failed; using sample data",
|
||||
event="collector.spacetrack.login.failed",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
logger.info_event(
|
||||
"Space-Track TLE query response received",
|
||||
event="collector.spacetrack.query.response",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE query failed; using sample data",
|
||||
event="collector.spacetrack.query.failed",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch completed",
|
||||
event="collector.spacetrack.fetch.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE fetch failed; using sample data",
|
||||
event="collector.spacetrack.fetch.failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -6,6 +6,7 @@ from typing import Any
|
||||
import httpx
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.enums import SnapshotStatus
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.services.barentswatch import (
|
||||
@@ -125,7 +126,7 @@ class VesselAISCollector(BaseCollector):
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
snapshot.record_count = records_added
|
||||
snapshot.status = "success"
|
||||
snapshot.status = SnapshotStatus.SUCCESS.value
|
||||
snapshot.completed_at = now
|
||||
snapshot.summary = {
|
||||
"created": records_added,
|
||||
|
||||
@@ -12,6 +12,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.cache import cache
|
||||
from app.core.config import settings
|
||||
from app.core.enums import JobStatus, JobType, RollbackPolicy
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
@@ -36,17 +37,17 @@ from app.services.scheduler import sync_datasource_job
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
JOB_TYPE_COLLECT = "collect"
|
||||
JOB_TYPE_CLEAR_DATA = "clear_data"
|
||||
JOB_TYPE_CLEAR_CACHE = "clear_cache"
|
||||
JOB_TYPE_EARTH_REFRESH = "earth_refresh"
|
||||
JOB_TYPE_COLLECT = JobType.COLLECT.value
|
||||
JOB_TYPE_CLEAR_DATA = JobType.CLEAR_DATA.value
|
||||
JOB_TYPE_CLEAR_CACHE = JobType.CLEAR_CACHE.value
|
||||
JOB_TYPE_EARTH_REFRESH = JobType.EARTH_REFRESH.value
|
||||
|
||||
JOB_STATUS_QUEUED = "queued"
|
||||
JOB_STATUS_RUNNING = "running"
|
||||
JOB_STATUS_CANCELLING = "cancelling"
|
||||
JOB_STATUS_SUCCESS = "success"
|
||||
JOB_STATUS_FAILED = "failed"
|
||||
JOB_STATUS_CANCELLED = "cancelled"
|
||||
JOB_STATUS_QUEUED = JobStatus.QUEUED.value
|
||||
JOB_STATUS_RUNNING = JobStatus.RUNNING.value
|
||||
JOB_STATUS_CANCELLING = JobStatus.CANCELLING.value
|
||||
JOB_STATUS_SUCCESS = JobStatus.SUCCESS.value
|
||||
JOB_STATUS_FAILED = JobStatus.FAILED.value
|
||||
JOB_STATUS_CANCELLED = JobStatus.CANCELLED.value
|
||||
|
||||
ACTIVE_JOB_STATUSES = (JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, JOB_STATUS_CANCELLING)
|
||||
TERMINAL_JOB_STATUSES = (JOB_STATUS_SUCCESS, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED)
|
||||
@@ -80,7 +81,7 @@ async def enqueue_datasource_job(
|
||||
task_type: str,
|
||||
*,
|
||||
payload: dict[str, Any] | None = None,
|
||||
rollback_policy: str = "keep_committed_batches",
|
||||
rollback_policy: str = RollbackPolicy.KEEP_COMMITTED_BATCHES.value,
|
||||
dedupe_key: str | None = None,
|
||||
) -> CollectionTask:
|
||||
if dedupe_key:
|
||||
|
||||
@@ -13,6 +13,7 @@ from sqlalchemy import func, select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.datasource_defaults import DEFAULT_DATASOURCES
|
||||
from app.core.enums import JobStatus
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
@@ -398,7 +399,7 @@ async def has_collected_data(db, source: str) -> bool:
|
||||
|
||||
datasource_result = await db.execute(select(DataSource).where(DataSource.source == source))
|
||||
datasource = datasource_result.scalar_one_or_none()
|
||||
return bool(datasource and datasource.last_status == "success")
|
||||
return bool(datasource and datasource.last_status == JobStatus.SUCCESS.value)
|
||||
|
||||
|
||||
async def get_builtin_connection_status(
|
||||
|
||||
@@ -6,6 +6,7 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from app.core.enums import UserRole
|
||||
from app.models.user import User
|
||||
|
||||
DocsAccess = Literal["public", "docs_user", "docs_developer", "docs_admin"]
|
||||
@@ -43,7 +44,9 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("earth-satellite-footprint-policy.md", "earth-satellite-footprint-policy", "docs_developer", "Earth", 13, "智能星球卫星覆盖策略", "Intelligent Planet Satellite Footprint Policy"),
|
||||
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
|
||||
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
|
||||
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
|
||||
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
|
||||
DocsMetadata("tactile-ui-components.md", "tactile-ui-components", "docs_developer", "Frontend", 24, "Tactile UI 组件库", "Tactile UI Components"),
|
||||
@@ -52,6 +55,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("datasource-collector-settings-connectivity.md", "datasource-collector-settings-connectivity", "docs_developer", "Backend", 32, "数据源、采集器设置与连接验证", "Datasource Collector Settings and Connectivity"),
|
||||
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
|
||||
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
|
||||
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
@@ -69,9 +73,9 @@ def get_user_gatekeeper_groups(user: User | None) -> set[str]:
|
||||
return set()
|
||||
|
||||
role = user.role.value if hasattr(user.role, "value") else str(user.role or "")
|
||||
if role == "super_admin":
|
||||
if role == UserRole.SUPER_ADMIN.value:
|
||||
return {"docs_user", "docs_developer", "docs_admin"}
|
||||
if role == "admin":
|
||||
if role == UserRole.ADMIN.value:
|
||||
return {"docs_user", "docs_developer", "docs_admin"}
|
||||
|
||||
groups = set()
|
||||
|
||||
@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
refresh_strategy = "reload"
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
self.refresh_strategy = "reload"
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
|
||||
if not update:
|
||||
return False
|
||||
|
||||
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
|
||||
keys = payload.get("entity_keys")
|
||||
if not isinstance(keys, list):
|
||||
keys = [payload.get("entity_key")]
|
||||
broadcaster.enqueue_vessel_update({
|
||||
"action": "remove",
|
||||
"vessels": [{"mmsi": key} for key in keys if key is not None],
|
||||
})
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
|
||||
@@ -20,11 +20,12 @@ class EarthLayerAdapter:
|
||||
|
||||
EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
|
||||
tables=frozenset({"vessel_static", "vessel_position", "ais_raw_observations", "ais_source_health"}),
|
||||
sources=frozenset({"barentswatch_vessels", "aisstream_vessels", "vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
|
||||
tables=frozenset({"vessel_static", "vessel_position", "vessel_current_state", "ais_raw_observations", "ais_source_health"}),
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
|
||||
@@ -20,10 +20,27 @@ from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import COUNTRY_VARIANTS_MAP, get_country_centroid, normalize_country
|
||||
from app.core.enums import (
|
||||
BreakingLevel,
|
||||
BreakingScope,
|
||||
BreakingSource,
|
||||
NewsEnrichmentStatus,
|
||||
NewsImportanceLevel,
|
||||
NewsMarketImpact,
|
||||
NewsSourceType,
|
||||
NewsTaggingSource,
|
||||
)
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.ai_tasks.prompts import EffectiveAIPrompt, get_effective_prompt
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.earth_news_classification import (
|
||||
apply_news_classification as _apply_news_classification,
|
||||
breaking_sort_rank as _breaking_sort_rank,
|
||||
highest_breaking_level as _highest_breaking_level,
|
||||
normalize_breaking_level as _normalize_breaking_level_enum,
|
||||
normalize_breaking_scope as _normalize_breaking_scope_enum,
|
||||
)
|
||||
from app.services.location.resolvers.nominatim import build_default_nominatim_geocoder
|
||||
|
||||
|
||||
@@ -69,7 +86,7 @@ class NewsFeedEndpoint:
|
||||
id: str
|
||||
name: str
|
||||
url: str
|
||||
type: str = "rss"
|
||||
type: str = NewsSourceType.RSS.value
|
||||
region: str = ""
|
||||
enabled: bool = True
|
||||
default_category: str = "other"
|
||||
@@ -85,7 +102,7 @@ class NewsFeedSource:
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
feed_directory_url: str = ""
|
||||
source_type: str = "rss"
|
||||
source_type: str = NewsSourceType.RSS.value
|
||||
feed_urls: tuple[str, ...] = ()
|
||||
feeds: tuple[NewsFeedEndpoint, ...] = ()
|
||||
priority: int = 100
|
||||
@@ -120,7 +137,7 @@ class ParsedNewsItem:
|
||||
published_at: datetime | None
|
||||
content_language: str = "en"
|
||||
localizations: dict[str, dict[str, str]] = field(default_factory=dict)
|
||||
enrichment_status: str = "pending"
|
||||
enrichment_status: str = NewsEnrichmentStatus.PENDING.value
|
||||
enrichment_error: str | None = None
|
||||
enriched_at: datetime | None = None
|
||||
target_location: NewsTargetLocation | None = None
|
||||
@@ -132,16 +149,22 @@ class ParsedNewsItem:
|
||||
location_patch: dict[str, Any] | None = None
|
||||
source_tags: list[str] = field(default_factory=list)
|
||||
feed_id: str = ""
|
||||
feed_type: str = "rss"
|
||||
feed_type: str = NewsSourceType.RSS.value
|
||||
feed_default_category: str = "other"
|
||||
category: str = "other"
|
||||
item_tags: list[str] = field(default_factory=list)
|
||||
tagging_source: str = "rules"
|
||||
tagging_source: str = NewsTaggingSource.RULES.value
|
||||
tagging_confidence: float = 0.0
|
||||
importance_score: int = 0
|
||||
importance_level: str = "low"
|
||||
importance_level: str = NewsImportanceLevel.LOW.value
|
||||
importance_reasons: list[str] = field(default_factory=list)
|
||||
market_impact: str = "none"
|
||||
market_impact: str = NewsMarketImpact.NONE.value
|
||||
breaking_level: str = BreakingLevel.NONE.value
|
||||
breaking_scope: str = BreakingScope.REGIONAL.value
|
||||
breaking_reasons: list[str] = field(default_factory=list)
|
||||
breaking_source: str = BreakingSource.RULES.value
|
||||
breaking_confidence: float = 0.0
|
||||
breaking_expires_at: datetime | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -842,9 +865,9 @@ def _get_locale_text(
|
||||
if isinstance(fallback, dict):
|
||||
value = _coerce_str(fallback.get(key))
|
||||
if value:
|
||||
if locale == "en-US" and _contains_cjk_text(value):
|
||||
return ""
|
||||
return value
|
||||
if locale == "en-US" and _is_chinese_language(item.content_language):
|
||||
return item.title if key == "title" else item.summary
|
||||
return ""
|
||||
|
||||
|
||||
@@ -1863,34 +1886,12 @@ def _parse_feed_entries(
|
||||
return items
|
||||
|
||||
|
||||
def _contains_keyword(text: str, keyword: str) -> bool:
|
||||
keyword_text = str(keyword or "").strip().lower()
|
||||
if not keyword_text:
|
||||
return False
|
||||
if re.search(r"[\u4e00-\u9fff]", keyword_text):
|
||||
return keyword_text in text
|
||||
return re.search(rf"(?<![a-z0-9]){re.escape(keyword_text)}(?![a-z0-9])", text) is not None
|
||||
def _normalize_breaking_level(value: str | None) -> str:
|
||||
return _normalize_breaking_level_enum(value).value
|
||||
|
||||
|
||||
def _score_category(text: str, title_text: str, category: dict[str, Any]) -> int:
|
||||
score = 0
|
||||
keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else []
|
||||
for keyword in keywords:
|
||||
if _contains_keyword(title_text, keyword):
|
||||
score += 3
|
||||
elif _contains_keyword(text, keyword):
|
||||
score += 1
|
||||
return score
|
||||
|
||||
|
||||
def _importance_level(score: int) -> str:
|
||||
if score >= 80:
|
||||
return "critical"
|
||||
if score >= 60:
|
||||
return "high"
|
||||
if score >= 35:
|
||||
return "medium"
|
||||
return "low"
|
||||
def _normalize_breaking_scope(value: str | None) -> str:
|
||||
return _normalize_breaking_scope_enum(value).value
|
||||
|
||||
|
||||
def apply_news_classification(
|
||||
@@ -1901,74 +1902,7 @@ def apply_news_classification(
|
||||
config_payload: dict[str, Any] | None = None,
|
||||
) -> ParsedNewsItem:
|
||||
config = normalize_earth_news_sources_payload(config_payload)
|
||||
title_text = item.title.lower()
|
||||
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
|
||||
|
||||
feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other"
|
||||
best_key = feed_default_category
|
||||
best_score = 0
|
||||
second_score = 0
|
||||
for category in config["categories"]:
|
||||
if not isinstance(category, dict) or category.get("enabled") is False:
|
||||
continue
|
||||
score = _score_category(combined_text, title_text, category)
|
||||
if score > best_score:
|
||||
second_score = best_score
|
||||
best_score = score
|
||||
best_key = str(category.get("key") or "other")
|
||||
elif score > second_score:
|
||||
second_score = score
|
||||
|
||||
item_tags: list[str] = []
|
||||
for rule in config["item_tag_rules"]:
|
||||
if not isinstance(rule, dict):
|
||||
continue
|
||||
keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else []
|
||||
if any(_contains_keyword(combined_text, keyword) for keyword in keywords):
|
||||
tag_key = str(rule.get("key") or "").strip()
|
||||
if tag_key and tag_key not in item_tags:
|
||||
item_tags.append(tag_key)
|
||||
if best_score < 3 and rule.get("category"):
|
||||
best_key = str(rule["category"])
|
||||
best_score = 3
|
||||
|
||||
confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35
|
||||
if best_score < 3 and feed_default_category:
|
||||
best_key = feed_default_category
|
||||
confidence = 0.45
|
||||
|
||||
importance_score = max(0, min(100, 18 + source.importance_weight + best_score * 6))
|
||||
reasons: list[str] = []
|
||||
source_tag_set = set(source.source_tags)
|
||||
if "official_data" in source_tag_set:
|
||||
importance_score += 20
|
||||
reasons.append("官方数据源")
|
||||
if "press_release" in source_tag_set:
|
||||
importance_score = max(0, importance_score - 12)
|
||||
reasons.append("企业公告基础权重较低")
|
||||
ecommerce_terms = ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")
|
||||
if any(_contains_keyword(combined_text, term) for term in ecommerce_terms):
|
||||
importance_score += 25
|
||||
reasons.append("命中电商数据指标")
|
||||
major_platforms = ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")
|
||||
if any(_contains_keyword(combined_text, term) for term in major_platforms):
|
||||
importance_score += 15
|
||||
reasons.append("涉及大型平台")
|
||||
if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")):
|
||||
importance_score += 10
|
||||
reasons.append("包含量化指标")
|
||||
|
||||
importance_score = max(0, min(100, importance_score))
|
||||
item.category = best_key or "other"
|
||||
item.item_tags = item_tags
|
||||
item.tagging_source = "rules"
|
||||
item.tagging_confidence = confidence
|
||||
item.importance_score = importance_score
|
||||
item.importance_level = _importance_level(importance_score)
|
||||
item.importance_reasons = reasons or ["按来源权重和分类规则计算"]
|
||||
item.market_impact = "global" if "global" in source_tag_set else "national" if {"china", "us"} & source_tag_set else "sector"
|
||||
item.source_tags = list(source.source_tags)
|
||||
return item
|
||||
return _apply_news_classification(item, source, feed=feed, config=config)
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
@@ -2020,6 +1954,10 @@ def _serialize_enriched_at(value: datetime | None) -> str | None:
|
||||
return value.isoformat().replace("+00:00", "Z") if value else None
|
||||
|
||||
|
||||
def _serialize_breaking_expires_at(value: datetime | None) -> str | None:
|
||||
return value.isoformat().replace("+00:00", "Z") if value else None
|
||||
|
||||
|
||||
def _content_patch(item: ParsedNewsItem) -> dict[str, Any]:
|
||||
return {
|
||||
"content_language": item.content_language,
|
||||
@@ -2047,6 +1985,12 @@ def _news_meta_patch(item: ParsedNewsItem) -> dict[str, Any]:
|
||||
"importance_level": item.importance_level,
|
||||
"importance_reasons": list(item.importance_reasons or []),
|
||||
"market_impact": item.market_impact,
|
||||
"breaking_level": _normalize_breaking_level(item.breaking_level),
|
||||
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
|
||||
"breaking_reasons": list(item.breaking_reasons or []),
|
||||
"breaking_source": item.breaking_source,
|
||||
"breaking_confidence": item.breaking_confidence,
|
||||
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
|
||||
}
|
||||
|
||||
|
||||
@@ -2142,6 +2086,12 @@ def build_target_location_job_payload(item: ParsedNewsItem) -> dict[str, Any]:
|
||||
"importance_level": item.importance_level,
|
||||
"importance_reasons": list(item.importance_reasons or []),
|
||||
"market_impact": item.market_impact,
|
||||
"breaking_level": _normalize_breaking_level(item.breaking_level),
|
||||
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
|
||||
"breaking_reasons": list(item.breaking_reasons or []),
|
||||
"breaking_source": item.breaking_source,
|
||||
"breaking_confidence": item.breaking_confidence,
|
||||
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
|
||||
}
|
||||
|
||||
|
||||
@@ -2173,6 +2123,12 @@ def parsed_news_item_from_job_payload(payload: dict[str, Any]) -> ParsedNewsItem
|
||||
importance_level=str(payload.get("importance_level") or "low"),
|
||||
importance_reasons=list(payload.get("importance_reasons") or []),
|
||||
market_impact=str(payload.get("market_impact") or "none"),
|
||||
breaking_level=_normalize_breaking_level(str(payload.get("breaking_level") or "none")),
|
||||
breaking_scope=_normalize_breaking_scope(str(payload.get("breaking_scope") or "regional")),
|
||||
breaking_reasons=list(payload.get("breaking_reasons") or []),
|
||||
breaking_source=str(payload.get("breaking_source") or "rules"),
|
||||
breaking_confidence=float(payload.get("breaking_confidence") or 0),
|
||||
breaking_expires_at=_parse_datetime(_coerce_str(payload.get("breaking_expires_at"))),
|
||||
)
|
||||
|
||||
|
||||
@@ -2218,6 +2174,12 @@ def _serialize_item(item: ParsedNewsItem, *, active_region: str, locale: str = D
|
||||
"importance_level": item.importance_level,
|
||||
"importance_reasons": list(item.importance_reasons or []),
|
||||
"market_impact": item.market_impact,
|
||||
"breaking_level": _normalize_breaking_level(item.breaking_level),
|
||||
"breaking_scope": _normalize_breaking_scope(item.breaking_scope),
|
||||
"breaking_reasons": list(item.breaking_reasons or []),
|
||||
"breaking_source": item.breaking_source,
|
||||
"breaking_confidence": item.breaking_confidence,
|
||||
"breaking_expires_at": _serialize_breaking_expires_at(item.breaking_expires_at),
|
||||
}
|
||||
|
||||
|
||||
@@ -2239,6 +2201,9 @@ def _build_payload(
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
visible_items = list(items)
|
||||
cruise_visible_items = list(cruise_items if cruise_items is not None else items)
|
||||
highest_breaking_level = _highest_breaking_level(visible_items + cruise_visible_items)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
@@ -2256,11 +2221,13 @@ def _build_payload(
|
||||
"sources": sorted(source_ids or []),
|
||||
"limit": limit,
|
||||
"locale": locale,
|
||||
"has_breaking": highest_breaking_level != "none",
|
||||
"highest_breaking_level": highest_breaking_level,
|
||||
},
|
||||
"items": [_serialize_item(item, active_region=active_region, locale=locale) for item in items],
|
||||
"items": [_serialize_item(item, active_region=active_region, locale=locale) for item in visible_items],
|
||||
"cruise_items": [
|
||||
_serialize_item(item, active_region=active_region, locale=locale)
|
||||
for item in (cruise_items if cruise_items is not None else items)
|
||||
for item in cruise_visible_items
|
||||
],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
@@ -2282,7 +2249,11 @@ def _rank_and_trim_items(
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
False if active_region == "global" else item.feed_region != active_region,
|
||||
-_breaking_sort_rank(item),
|
||||
False
|
||||
if active_region == "global"
|
||||
or (_breaking_sort_rank(item) > 0 and _normalize_breaking_scope(item.breaking_scope) == "global")
|
||||
else item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
@@ -2311,6 +2282,53 @@ def _filter_news_items_by_source_ids(
|
||||
]
|
||||
|
||||
|
||||
def _news_item_source_id(item: ParsedNewsItem) -> str:
|
||||
return item.id.split(":", 1)[0] if ":" in item.id else ""
|
||||
|
||||
|
||||
def _is_news_item_display_ready(item: ParsedNewsItem, *, locale: str) -> bool:
|
||||
return bool(
|
||||
_get_locale_text(item, "title", locale=locale)
|
||||
and _get_locale_text(item, "summary", locale=locale)
|
||||
)
|
||||
|
||||
|
||||
def _diversify_news_items_for_locale(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
active_region: str,
|
||||
limit: int,
|
||||
locale: str,
|
||||
) -> list[ParsedNewsItem]:
|
||||
ranked = sorted(
|
||||
_rank_and_trim_items(items, active_region=active_region, limit=max(len(items), limit)),
|
||||
key=lambda item: (not _is_news_item_display_ready(item, locale=locale),),
|
||||
)
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for item in ranked:
|
||||
if active_region == "global":
|
||||
key = item.feed_region or "global"
|
||||
else:
|
||||
key = _news_item_source_id(item) or item.source or item.feed_name or item.id
|
||||
if key not in buckets:
|
||||
buckets[key] = []
|
||||
order.append(key)
|
||||
buckets[key].append(item)
|
||||
|
||||
diversified: list[ParsedNewsItem] = []
|
||||
while len(diversified) < limit and order:
|
||||
next_order: list[str] = []
|
||||
for source_id in order:
|
||||
bucket = buckets.get(source_id) or []
|
||||
if bucket and len(diversified) < limit:
|
||||
diversified.append(bucket.pop(0))
|
||||
if bucket:
|
||||
next_order.append(source_id)
|
||||
order = next_order
|
||||
return diversified
|
||||
|
||||
|
||||
async def _call_store_list_items(list_fn, db: AsyncSession, **kwargs):
|
||||
try:
|
||||
return await list_fn(db, **kwargs)
|
||||
@@ -2736,6 +2754,7 @@ async def get_earth_news_payload(
|
||||
)
|
||||
await record_earth_news_sources_health(db, health_by_source)
|
||||
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
await _enqueue_unverified_locations(ranked_fetched_items)
|
||||
await upsert_earth_news_items(db, ranked_fetched_items)
|
||||
|
||||
items = await _call_store_list_items(
|
||||
@@ -2746,6 +2765,37 @@ async def get_earth_news_payload(
|
||||
categories=categories,
|
||||
source_ids=source_ids,
|
||||
)
|
||||
if not source_ids:
|
||||
ready_sources = {
|
||||
_news_item_source_id(item)
|
||||
for item in items
|
||||
if _is_news_item_display_ready(item, locale=locale)
|
||||
}
|
||||
missing_ready_sources = [
|
||||
source.id
|
||||
for source in sources
|
||||
if source.id and source.id not in ready_sources
|
||||
]
|
||||
if missing_ready_sources:
|
||||
extra_items: list[ParsedNewsItem] = []
|
||||
for missing_source_id in missing_ready_sources:
|
||||
extra_items.extend(
|
||||
await _call_store_list_items(
|
||||
list_earth_news_items,
|
||||
db,
|
||||
active_region=active_region,
|
||||
limit=3,
|
||||
categories=categories,
|
||||
source_ids={missing_source_id},
|
||||
)
|
||||
)
|
||||
if extra_items:
|
||||
items = _diversify_news_items_for_locale(
|
||||
[*items, *extra_items],
|
||||
active_region=active_region,
|
||||
limit=limit,
|
||||
locale=locale,
|
||||
)
|
||||
if hasattr(db, "execute"):
|
||||
cruise_items = await _call_store_list_items(
|
||||
list_earth_news_cruise_items,
|
||||
@@ -2756,7 +2806,8 @@ async def get_earth_news_payload(
|
||||
)
|
||||
else:
|
||||
cruise_items = items
|
||||
await _enqueue_unverified_locations(items)
|
||||
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
|
||||
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
|
||||
stale = bool(errors and items)
|
||||
|
||||
return _build_payload(
|
||||
|
||||
258
backend/app/services/earth_news_classification.py
Normal file
258
backend/app/services/earth_news_classification.py
Normal file
@@ -0,0 +1,258 @@
|
||||
"""Classification, importance, and breaking-news policy for Earth news."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import re
|
||||
from typing import Any, Protocol
|
||||
|
||||
from app.core.enums import (
|
||||
BreakingLevel,
|
||||
BreakingScope,
|
||||
BreakingSource,
|
||||
NewsImportanceLevel,
|
||||
NewsMarketImpact,
|
||||
NewsTaggingSource,
|
||||
parse_enum,
|
||||
)
|
||||
|
||||
|
||||
class NewsItemLike(Protocol):
|
||||
title: str
|
||||
summary: str
|
||||
source: str
|
||||
feed_name: str
|
||||
published_at: datetime | None
|
||||
feed_default_category: str
|
||||
category: str
|
||||
item_tags: list[str]
|
||||
tagging_source: str
|
||||
tagging_confidence: float
|
||||
importance_score: int
|
||||
importance_level: str
|
||||
importance_reasons: list[str]
|
||||
market_impact: str
|
||||
source_tags: list[str]
|
||||
breaking_level: str
|
||||
breaking_scope: str
|
||||
breaking_reasons: list[str]
|
||||
breaking_source: str
|
||||
breaking_confidence: float
|
||||
breaking_expires_at: datetime | None
|
||||
|
||||
|
||||
class NewsSourceLike(Protocol):
|
||||
default_category: str
|
||||
importance_weight: int
|
||||
source_tags: tuple[str, ...]
|
||||
|
||||
|
||||
class NewsFeedLike(Protocol):
|
||||
default_category: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BreakingRule:
|
||||
level: BreakingLevel
|
||||
scope: BreakingScope
|
||||
reason: str
|
||||
keywords: tuple[str, ...]
|
||||
|
||||
|
||||
IMPORTANCE_THRESHOLDS: tuple[tuple[int, NewsImportanceLevel], ...] = (
|
||||
(80, NewsImportanceLevel.CRITICAL),
|
||||
(60, NewsImportanceLevel.HIGH),
|
||||
(35, NewsImportanceLevel.MEDIUM),
|
||||
(0, NewsImportanceLevel.LOW),
|
||||
)
|
||||
|
||||
BREAKING_LEVEL_RANK: dict[BreakingLevel, int] = {
|
||||
BreakingLevel.NONE: 0,
|
||||
BreakingLevel.WATCH: 1,
|
||||
BreakingLevel.BREAKING: 2,
|
||||
BreakingLevel.CRITICAL: 3,
|
||||
}
|
||||
|
||||
BREAKING_TTL: dict[BreakingLevel, timedelta] = {
|
||||
BreakingLevel.WATCH: timedelta(hours=6),
|
||||
BreakingLevel.BREAKING: timedelta(hours=12),
|
||||
BreakingLevel.CRITICAL: timedelta(hours=24),
|
||||
}
|
||||
|
||||
BREAKING_RULES: tuple[BreakingRule, ...] = (
|
||||
BreakingRule(BreakingLevel.CRITICAL, BreakingScope.GLOBAL, "核事故或核风险", ("nuclear accident", "nuclear emergency", "radiation leak", "核事故", "核泄漏", "辐射泄漏")),
|
||||
BreakingRule(BreakingLevel.CRITICAL, BreakingScope.REGIONAL, "重大军事冲突升级", ("airstrike", "missile strike", "invasion", "martial law", "空袭", "导弹袭击", "入侵", "戒严")),
|
||||
BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "战争或安全事件", ("war escalates", "terror attack", "coup", "hostage", "战争升级", "恐袭", "政变", "人质")),
|
||||
BreakingRule(BreakingLevel.BREAKING, BreakingScope.REGIONAL, "重大灾害应急", ("major earthquake", "tsunami", "volcanic eruption", "state of emergency", "强震", "海啸", "火山喷发", "紧急状态")),
|
||||
BreakingRule(BreakingLevel.BREAKING, BreakingScope.GLOBAL, "金融市场异常", ("market halt", "trading halt", "flash crash", "bank run", "金融熔断", "交易暂停", "银行挤兑")),
|
||||
BreakingRule(BreakingLevel.WATCH, BreakingScope.GLOBAL, "大规模网络安全事件", ("massive cyberattack", "ransomware attack", "data breach", "大规模网络攻击", "勒索软件", "数据泄露")),
|
||||
BreakingRule(BreakingLevel.WATCH, BreakingScope.REGIONAL, "航天或卫星事故", ("rocket explosion", "satellite collision", "space station emergency", "火箭爆炸", "卫星碰撞", "空间站事故")),
|
||||
)
|
||||
|
||||
|
||||
def contains_keyword(text: str, keyword: str) -> bool:
|
||||
keyword_text = str(keyword or "").strip().lower()
|
||||
if not keyword_text:
|
||||
return False
|
||||
if re.search(r"[\u4e00-\u9fff]", keyword_text):
|
||||
return keyword_text in text
|
||||
return re.search(rf"(?<![a-z0-9]){re.escape(keyword_text)}(?![a-z0-9])", text) is not None
|
||||
|
||||
|
||||
def score_category(text: str, title_text: str, category: dict[str, Any]) -> int:
|
||||
score = 0
|
||||
keywords = category.get("keywords") if isinstance(category.get("keywords"), list) else []
|
||||
for keyword in keywords:
|
||||
if contains_keyword(title_text, keyword):
|
||||
score += 3
|
||||
elif contains_keyword(text, keyword):
|
||||
score += 1
|
||||
return score
|
||||
|
||||
|
||||
def importance_level(score: int) -> NewsImportanceLevel:
|
||||
normalized_score = max(0, min(100, int(score)))
|
||||
for threshold, level in IMPORTANCE_THRESHOLDS:
|
||||
if normalized_score >= threshold:
|
||||
return level
|
||||
return NewsImportanceLevel.LOW
|
||||
|
||||
|
||||
def normalize_breaking_level(value: object) -> BreakingLevel:
|
||||
return parse_enum(BreakingLevel, value, BreakingLevel.NONE)
|
||||
|
||||
|
||||
def normalize_breaking_scope(value: object) -> BreakingScope:
|
||||
return parse_enum(BreakingScope, value, BreakingScope.REGIONAL)
|
||||
|
||||
|
||||
def breaking_expires_at(level: object, published_at: datetime | None) -> datetime | None:
|
||||
normalized = normalize_breaking_level(level)
|
||||
if normalized is BreakingLevel.NONE:
|
||||
return None
|
||||
base = published_at or datetime.now(UTC)
|
||||
base = base.replace(tzinfo=UTC) if base.tzinfo is None else base.astimezone(UTC)
|
||||
return base + BREAKING_TTL[normalized]
|
||||
|
||||
|
||||
def is_breaking_active(item: NewsItemLike, *, now: datetime | None = None) -> bool:
|
||||
if normalize_breaking_level(item.breaking_level) is BreakingLevel.NONE:
|
||||
return False
|
||||
expires_at = item.breaking_expires_at
|
||||
if expires_at is None:
|
||||
return True
|
||||
expires_at = expires_at.replace(tzinfo=UTC) if expires_at.tzinfo is None else expires_at.astimezone(UTC)
|
||||
return expires_at > (now or datetime.now(UTC))
|
||||
|
||||
|
||||
def breaking_sort_rank(item: NewsItemLike) -> int:
|
||||
if not is_breaking_active(item):
|
||||
return 0
|
||||
return BREAKING_LEVEL_RANK[normalize_breaking_level(item.breaking_level)]
|
||||
|
||||
|
||||
def highest_breaking_level(items: list[NewsItemLike]) -> BreakingLevel:
|
||||
active = [normalize_breaking_level(item.breaking_level) for item in items if is_breaking_active(item)]
|
||||
return max(active, key=BREAKING_LEVEL_RANK.get) if active else BreakingLevel.NONE
|
||||
|
||||
|
||||
def apply_breaking_rules(item: NewsItemLike) -> None:
|
||||
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
|
||||
best_level = BreakingLevel.NONE
|
||||
best_scope = BreakingScope.REGIONAL
|
||||
reasons: list[str] = []
|
||||
confidence = 0.0
|
||||
for rule in BREAKING_RULES:
|
||||
if not any(contains_keyword(combined_text, keyword) for keyword in rule.keywords):
|
||||
continue
|
||||
if BREAKING_LEVEL_RANK[rule.level] > BREAKING_LEVEL_RANK[best_level]:
|
||||
best_level = rule.level
|
||||
best_scope = rule.scope
|
||||
if rule.reason not in reasons:
|
||||
reasons.append(rule.reason)
|
||||
confidence = max(confidence, 0.72 if rule.level is BreakingLevel.CRITICAL else 0.64 if rule.level is BreakingLevel.BREAKING else 0.52)
|
||||
|
||||
item.breaking_level = best_level.value
|
||||
item.breaking_scope = (best_scope if best_level is not BreakingLevel.NONE else BreakingScope.REGIONAL).value
|
||||
item.breaking_reasons = reasons
|
||||
item.breaking_source = BreakingSource.RULES.value
|
||||
item.breaking_confidence = round(confidence, 2)
|
||||
item.breaking_expires_at = breaking_expires_at(best_level, item.published_at)
|
||||
|
||||
|
||||
def apply_news_classification(
|
||||
item: NewsItemLike,
|
||||
source: NewsSourceLike,
|
||||
*,
|
||||
feed: NewsFeedLike | None,
|
||||
config: dict[str, Any],
|
||||
) -> NewsItemLike:
|
||||
title_text = item.title.lower()
|
||||
combined_text = f"{item.title} {item.summary} {item.source} {item.feed_name}".lower()
|
||||
feed_default_category = (feed.default_category if feed else item.feed_default_category) or source.default_category or "other"
|
||||
best_key = feed_default_category
|
||||
best_score = second_score = 0
|
||||
for category in config["categories"]:
|
||||
if not isinstance(category, dict) or category.get("enabled") is False:
|
||||
continue
|
||||
score = score_category(combined_text, title_text, category)
|
||||
if score > best_score:
|
||||
second_score, best_score = best_score, score
|
||||
best_key = str(category.get("key") or "other")
|
||||
elif score > second_score:
|
||||
second_score = score
|
||||
|
||||
item_tags: list[str] = []
|
||||
for rule in config["item_tag_rules"]:
|
||||
if not isinstance(rule, dict):
|
||||
continue
|
||||
keywords = rule.get("keywords") if isinstance(rule.get("keywords"), list) else []
|
||||
if any(contains_keyword(combined_text, keyword) for keyword in keywords):
|
||||
tag_key = str(rule.get("key") or "").strip()
|
||||
if tag_key and tag_key not in item_tags:
|
||||
item_tags.append(tag_key)
|
||||
if best_score < 3 and rule.get("category"):
|
||||
best_key, best_score = str(rule["category"]), 3
|
||||
|
||||
confidence = round(best_score / (best_score + second_score + 1), 2) if best_score else 0.35
|
||||
if best_score < 3 and feed_default_category:
|
||||
best_key, confidence = feed_default_category, 0.45
|
||||
|
||||
score = max(0, min(100, 18 + source.importance_weight + best_score * 6))
|
||||
reasons: list[str] = []
|
||||
source_tags = set(source.source_tags)
|
||||
if "official_data" in source_tags:
|
||||
score += 20
|
||||
reasons.append("官方数据源")
|
||||
if "press_release" in source_tags:
|
||||
score = max(0, score - 12)
|
||||
reasons.append("企业公告基础权重较低")
|
||||
if any(contains_keyword(combined_text, term) for term in ("网上零售额", "电商物流指数", "gmv", "订单量", "物流指数", "履约", "直播电商", "跨境电商")):
|
||||
score += 25
|
||||
reasons.append("命中电商数据指标")
|
||||
if any(contains_keyword(combined_text, term) for term in ("amazon", "shopify", "walmart", "alibaba", "jd.com", "pinduoduo", "tiktok shop", "shein", "阿里", "京东", "拼多多", "抖音")):
|
||||
score += 15
|
||||
reasons.append("涉及大型平台")
|
||||
if any(term in combined_text for term in ("同比", "环比", "%", "billion", "million", "增长", "下降")):
|
||||
score += 10
|
||||
reasons.append("包含量化指标")
|
||||
|
||||
score = max(0, min(100, score))
|
||||
item.category = best_key or "other"
|
||||
item.item_tags = item_tags
|
||||
item.tagging_source = NewsTaggingSource.RULES.value
|
||||
item.tagging_confidence = confidence
|
||||
item.importance_score = score
|
||||
item.importance_level = importance_level(score).value
|
||||
item.importance_reasons = reasons or ["按来源权重和分类规则计算"]
|
||||
item.market_impact = (
|
||||
NewsMarketImpact.GLOBAL.value
|
||||
if "global" in source_tags
|
||||
else NewsMarketImpact.NATIONAL.value
|
||||
if {"china", "us"} & source_tags
|
||||
else NewsMarketImpact.SECTOR.value
|
||||
)
|
||||
item.source_tags = list(source.source_tags)
|
||||
apply_breaking_rules(item)
|
||||
return item
|
||||
693
backend/app/services/earth_news_manual.py
Normal file
693
backend/app/services/earth_news_manual.py
Normal file
@@ -0,0 +1,693 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.enums import NewsEnrichmentStatus, NewsSourceType, NewsTaggingSource
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.earth_news import (
|
||||
ALLOWED_NEWS_CATEGORY_KEYS,
|
||||
DEFAULT_NEWS_LOCALE,
|
||||
REGION_ANCHORS,
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
NewsTargetLocation,
|
||||
ParsedNewsItem,
|
||||
apply_news_classification,
|
||||
build_anchor_location_patch,
|
||||
build_target_location_job_payload,
|
||||
build_target_location_patch,
|
||||
_serialize_item,
|
||||
)
|
||||
from app.services.earth_news_queue import enqueue_target_location_job
|
||||
from app.services.earth_news_store import record_to_parsed_news_item
|
||||
|
||||
|
||||
MANUAL_NEWS_SOURCE_ID = "manual"
|
||||
MANUAL_NEWS_SOURCE_LABEL = "手动添加"
|
||||
MANUAL_NEWS_MAX_IMPORT_ITEMS = 500
|
||||
MANUAL_NEWS_MAX_TITLE_LENGTH = 500
|
||||
MANUAL_NEWS_MAX_SUMMARY_LENGTH = 1200
|
||||
MANUAL_NEWS_MAX_CONTENT_LENGTH = 12000
|
||||
EARTH_NEWS_MANUAL_GROUPS_CATEGORY = "earth_news_manual_groups"
|
||||
DEFAULT_MANUAL_NEWS_GROUP_ID = "manual-default"
|
||||
DEFAULT_MANUAL_NEWS_GROUP_NAME = "新建新闻组"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManualNewsWriteResult:
|
||||
item: EarthNewsItem
|
||||
created: bool
|
||||
queued: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManualNewsGroup:
|
||||
id: str
|
||||
name: str
|
||||
sort_order: int = 0
|
||||
|
||||
|
||||
def _clean_text(value: object, *, max_length: int) -> str:
|
||||
raw = "" if value is None else str(value)
|
||||
text = BeautifulSoup(html.unescape(raw), "html.parser").get_text(" ", strip=True)
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
if len(text) > max_length:
|
||||
return text[: max_length - 1].rstrip() + "…"
|
||||
return text
|
||||
|
||||
|
||||
def _parse_datetime(value: object) -> datetime | None:
|
||||
if value is None or str(value).strip() == "":
|
||||
return None
|
||||
if isinstance(value, datetime):
|
||||
parsed = value
|
||||
else:
|
||||
try:
|
||||
parsed = datetime.fromisoformat(str(value).strip().replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError("published_at 必须是 ISO8601 时间。") from exc
|
||||
if parsed.tzinfo is None:
|
||||
return parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
|
||||
|
||||
def _detect_language(*parts: str) -> str:
|
||||
text = " ".join(part for part in parts if part)
|
||||
cjk_count = len(re.findall(r"[\u4e00-\u9fff]", text))
|
||||
latin_count = len(re.findall(r"[A-Za-z]", text))
|
||||
return "zh-CN" if cjk_count >= max(4, latin_count // 3) else "en-US"
|
||||
|
||||
|
||||
def _manual_item_id(*, title: str, published_at: datetime | None, url: str, source: str) -> str:
|
||||
published = published_at.isoformat() if published_at else ""
|
||||
basis = "\n".join([title.strip().lower(), published, url.strip().lower(), source.strip().lower()])
|
||||
return f"manual:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:16]}"
|
||||
|
||||
|
||||
def _manual_group_id(name: str) -> str:
|
||||
basis = f"{name.strip().lower()}\n{datetime.now(UTC).isoformat()}"
|
||||
return f"manual-group:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:10]}"
|
||||
|
||||
|
||||
def _news_meta(record: EarthNewsItem) -> dict[str, Any]:
|
||||
location_meta = record.location_meta if isinstance(record.location_meta, dict) else {}
|
||||
news_meta = location_meta.get("news_meta")
|
||||
return dict(news_meta) if isinstance(news_meta, dict) else {}
|
||||
|
||||
|
||||
def _record_source_type(record: EarthNewsItem) -> str:
|
||||
return str(_news_meta(record).get("feed_type") or _news_meta(record).get("source_type") or "rss")
|
||||
|
||||
|
||||
def _record_manual_group_id(record: EarthNewsItem) -> str:
|
||||
return str(_news_meta(record).get("manual_group_id") or DEFAULT_MANUAL_NEWS_GROUP_ID)
|
||||
|
||||
|
||||
def _rss_group_id(record: EarthNewsItem) -> str:
|
||||
basis = "\n".join(
|
||||
[
|
||||
_record_source_type(record),
|
||||
str(record.feed_name or ""),
|
||||
str(record.source or ""),
|
||||
]
|
||||
)
|
||||
return f"rss:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:12]}"
|
||||
|
||||
|
||||
def _default_manual_group() -> dict[str, Any]:
|
||||
return {
|
||||
"id": DEFAULT_MANUAL_NEWS_GROUP_ID,
|
||||
"name": DEFAULT_MANUAL_NEWS_GROUP_NAME,
|
||||
"sort_order": 0,
|
||||
}
|
||||
|
||||
|
||||
def _normalize_manual_groups_payload(payload: Any) -> list[dict[str, Any]]:
|
||||
raw_groups = payload.get("groups") if isinstance(payload, dict) else None
|
||||
normalized: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for index, item in enumerate(raw_groups if isinstance(raw_groups, list) else []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
group_id = str(item.get("id") or "").strip()
|
||||
name = _clean_text(item.get("name"), max_length=120)
|
||||
if not group_id or not name or group_id in seen:
|
||||
continue
|
||||
normalized.append(
|
||||
{
|
||||
"id": group_id,
|
||||
"name": name,
|
||||
"sort_order": int(item.get("sort_order") or index),
|
||||
}
|
||||
)
|
||||
seen.add(group_id)
|
||||
if DEFAULT_MANUAL_NEWS_GROUP_ID not in seen:
|
||||
normalized.insert(0, _default_manual_group())
|
||||
return sorted(normalized, key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
|
||||
|
||||
|
||||
async def _get_manual_groups_record(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_MANUAL_GROUPS_CATEGORY)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_manual_news_groups(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
record = await _get_manual_groups_record(db)
|
||||
return _normalize_manual_groups_payload(record.payload if record else None)
|
||||
|
||||
|
||||
async def _save_manual_news_groups(db: AsyncSession, groups: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
normalized = _normalize_manual_groups_payload({"groups": groups})
|
||||
record = await _get_manual_groups_record(db)
|
||||
payload = {"groups": normalized}
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=EARTH_NEWS_MANUAL_GROUPS_CATEGORY, payload=payload))
|
||||
else:
|
||||
record.payload = payload
|
||||
await db.flush()
|
||||
return normalized
|
||||
|
||||
|
||||
async def resolve_manual_news_group(db: AsyncSession, group_id: str | None) -> ManualNewsGroup:
|
||||
normalized_id = str(group_id or DEFAULT_MANUAL_NEWS_GROUP_ID).strip() or DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
groups = await get_manual_news_groups(db)
|
||||
match = next((item for item in groups if item.get("id") == normalized_id), None)
|
||||
if match is None and normalized_id != DEFAULT_MANUAL_NEWS_GROUP_ID:
|
||||
raise ValueError(f"手动新闻组不存在：{normalized_id}")
|
||||
match = match or _default_manual_group()
|
||||
return ManualNewsGroup(
|
||||
id=str(match["id"]),
|
||||
name=str(match["name"]),
|
||||
sort_order=int(match.get("sort_order") or 0),
|
||||
)
|
||||
|
||||
|
||||
async def create_manual_news_group(db: AsyncSession, name: str) -> dict[str, Any]:
|
||||
group_name = _clean_text(name, max_length=120)
|
||||
if not group_name:
|
||||
raise ValueError("新闻组名称不能为空。")
|
||||
groups = await get_manual_news_groups(db)
|
||||
group = {"id": _manual_group_id(group_name), "name": group_name, "sort_order": len(groups)}
|
||||
groups.append(group)
|
||||
await _save_manual_news_groups(db, groups)
|
||||
return group
|
||||
|
||||
|
||||
async def rename_manual_news_group(db: AsyncSession, group_id: str, name: str) -> dict[str, Any]:
|
||||
group_name = _clean_text(name, max_length=120)
|
||||
if not group_name:
|
||||
raise ValueError("新闻组名称不能为空。")
|
||||
groups = await get_manual_news_groups(db)
|
||||
match = next((item for item in groups if item.get("id") == group_id), None)
|
||||
if match is None:
|
||||
raise ValueError(f"手动新闻组不存在：{group_id}")
|
||||
match["name"] = group_name
|
||||
await _save_manual_news_groups(db, groups)
|
||||
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem).where(
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == NewsSourceType.MANUAL.value
|
||||
)
|
||||
)
|
||||
for record in result.scalars().all():
|
||||
if _record_manual_group_id(record) != group_id:
|
||||
continue
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = dict(location_meta.get("news_meta") or {})
|
||||
news_meta["manual_group_name"] = group_name
|
||||
location_meta["news_meta"] = news_meta
|
||||
record.location_meta = location_meta
|
||||
await db.flush()
|
||||
return match
|
||||
|
||||
|
||||
def _normalize_region(value: object) -> str:
|
||||
region = str(value or "global").strip().lower() or "global"
|
||||
if region not in REGION_ANCHORS:
|
||||
raise ValueError(f"region 不支持：{region}")
|
||||
return region
|
||||
|
||||
|
||||
def _normalize_tags(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
parts = re.split(r"[,，\n]", value)
|
||||
elif isinstance(value, list):
|
||||
parts = [str(item) for item in value]
|
||||
else:
|
||||
raise ValueError("tags 必须是字符串数组或逗号分隔字符串。")
|
||||
return [item.strip() for item in parts if item.strip()][:20]
|
||||
|
||||
|
||||
def _normalize_location(value: object) -> NewsTargetLocation | None:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("location 必须是对象。")
|
||||
lat = value.get("latitude")
|
||||
lon = value.get("longitude")
|
||||
if lat in (None, "") and lon in (None, ""):
|
||||
return None
|
||||
try:
|
||||
latitude = float(lat)
|
||||
longitude = float(lon)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("location.latitude / longitude 必须是数字。") from exc
|
||||
if not -90 <= latitude <= 90 or not -180 <= longitude <= 180:
|
||||
raise ValueError("location 经纬度超出范围。")
|
||||
label = _clean_text(value.get("label"), max_length=255)
|
||||
if not label:
|
||||
label = f"{latitude:.4f}, {longitude:.4f}"
|
||||
return NewsTargetLocation(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
label=label,
|
||||
source="manual_location",
|
||||
confidence=1.0,
|
||||
country=_clean_text(value.get("country"), max_length=100) or None,
|
||||
city=_clean_text(value.get("city"), max_length=100) or None,
|
||||
)
|
||||
|
||||
|
||||
def _manual_source(source_name: str, *, region: str) -> NewsFeedSource:
|
||||
return NewsFeedSource(
|
||||
id=MANUAL_NEWS_SOURCE_ID,
|
||||
name=source_name or MANUAL_NEWS_SOURCE_LABEL,
|
||||
region=region,
|
||||
feed_url="",
|
||||
homepage_url="",
|
||||
source_type=NewsSourceType.MANUAL.value,
|
||||
default_category="other",
|
||||
source_tags=("manual",),
|
||||
)
|
||||
|
||||
|
||||
def _manual_feed(category: str) -> NewsFeedEndpoint:
|
||||
return NewsFeedEndpoint(
|
||||
id=MANUAL_NEWS_SOURCE_ID,
|
||||
name=MANUAL_NEWS_SOURCE_LABEL,
|
||||
url="",
|
||||
type=NewsSourceType.MANUAL.value,
|
||||
default_category=category or "other",
|
||||
tags=("manual",),
|
||||
priority=1,
|
||||
)
|
||||
|
||||
|
||||
def parsed_manual_news_item(
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
item_id_override: str | None = None,
|
||||
) -> tuple[ParsedNewsItem, NewsTargetLocation | None, str]:
|
||||
title = _clean_text(payload.get("title"), max_length=MANUAL_NEWS_MAX_TITLE_LENGTH)
|
||||
if not title:
|
||||
raise ValueError("title 不能为空。")
|
||||
content = _clean_text(payload.get("content"), max_length=MANUAL_NEWS_MAX_CONTENT_LENGTH)
|
||||
summary = _clean_text(payload.get("summary"), max_length=MANUAL_NEWS_MAX_SUMMARY_LENGTH)
|
||||
if not summary:
|
||||
summary = _clean_text(content, max_length=240) if content else title
|
||||
source = _clean_text(payload.get("source"), max_length=255) or MANUAL_NEWS_SOURCE_LABEL
|
||||
region = _normalize_region(payload.get("region"))
|
||||
published_at = _parse_datetime(payload.get("published_at")) or datetime.now(UTC)
|
||||
url = str(payload.get("url") or "").strip()
|
||||
category = str(payload.get("category") or "other").strip().lower() or "other"
|
||||
if category not in ALLOWED_NEWS_CATEGORY_KEYS:
|
||||
raise ValueError(f"category 不支持：{category}")
|
||||
tags = _normalize_tags(payload.get("tags"))
|
||||
target = _normalize_location(payload.get("location"))
|
||||
language = str(payload.get("content_language") or "").strip() or _detect_language(title, summary, content)
|
||||
localizations = {
|
||||
language: {
|
||||
"title": title,
|
||||
"summary": summary,
|
||||
}
|
||||
}
|
||||
item = ParsedNewsItem(
|
||||
id=item_id_override
|
||||
or _manual_item_id(title=title, published_at=published_at, url=url, source=source),
|
||||
title=title,
|
||||
summary=summary,
|
||||
url=url,
|
||||
source=source,
|
||||
feed_name=MANUAL_NEWS_SOURCE_LABEL,
|
||||
feed_region=region,
|
||||
homepage_url=str(payload.get("homepage_url") or ""),
|
||||
published_at=published_at,
|
||||
content_language=language,
|
||||
localizations=localizations,
|
||||
enrichment_status=NewsEnrichmentStatus.PENDING.value,
|
||||
source_tags=["manual"],
|
||||
feed_id=MANUAL_NEWS_SOURCE_ID,
|
||||
feed_type=NewsSourceType.MANUAL.value,
|
||||
feed_default_category=category,
|
||||
category=category,
|
||||
item_tags=tags,
|
||||
tagging_source=NewsTaggingSource.MANUAL.value if payload.get("category") else NewsTaggingSource.RULES.value,
|
||||
tagging_confidence=0.9 if payload.get("category") else 0.0,
|
||||
)
|
||||
source_config = _manual_source(source, region=region)
|
||||
feed = _manual_feed(category)
|
||||
apply_news_classification(item, source_config, feed=feed)
|
||||
if payload.get("category"):
|
||||
item.category = category
|
||||
item.tagging_source = NewsTaggingSource.MANUAL.value
|
||||
item.tagging_confidence = 0.9
|
||||
if tags:
|
||||
item.item_tags = sorted(set([*item.item_tags, *tags]))
|
||||
return item, target, content
|
||||
|
||||
|
||||
def _manual_editable(record: EarthNewsItem) -> bool:
|
||||
if record.id.startswith("manual:"):
|
||||
return True
|
||||
news_meta = (record.location_meta or {}).get("news_meta") if isinstance(record.location_meta, dict) else None
|
||||
return isinstance(news_meta, dict) and news_meta.get("feed_type") == NewsSourceType.MANUAL.value
|
||||
|
||||
|
||||
async def _broadcast_news_reload() -> None:
|
||||
await broadcaster.broadcast_earth_update(
|
||||
{
|
||||
"action": "database_changed",
|
||||
"source": "earth_news_items",
|
||||
"layers": ["news"],
|
||||
"refresh_strategy": "reload",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
async def upsert_manual_news_item(
|
||||
db: AsyncSession,
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
item_id_override: str | None = None,
|
||||
group_id: str | None = None,
|
||||
) -> ManualNewsWriteResult:
|
||||
item, target, content = parsed_manual_news_item(payload, item_id_override=item_id_override)
|
||||
group = await resolve_manual_news_group(db, group_id or payload.get("group_id"))
|
||||
existing = await db.get(EarthNewsItem, item.id)
|
||||
created = existing is None
|
||||
patch = build_target_location_patch(item, target) if target else build_anchor_location_patch(item)
|
||||
patch_meta = dict(patch.get("location_meta") or {})
|
||||
patch_news_meta = dict(patch_meta.get("news_meta") or {})
|
||||
patch_news_meta["feed_type"] = NewsSourceType.MANUAL.value
|
||||
patch_news_meta["source_type"] = NewsSourceType.MANUAL.value
|
||||
patch_news_meta["manual_group_id"] = group.id
|
||||
patch_news_meta["manual_group_name"] = group.name
|
||||
patch_meta["news_meta"] = patch_news_meta
|
||||
patch["location_meta"] = patch_meta
|
||||
now = datetime.now(UTC)
|
||||
record = existing or EarthNewsItem(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
summary=item.summary,
|
||||
content_language=item.content_language,
|
||||
localizations=dict(item.localizations or {}),
|
||||
url=item.url,
|
||||
source=item.source,
|
||||
feed_name=item.feed_name,
|
||||
region=item.feed_region,
|
||||
homepage_url=item.homepage_url,
|
||||
published_at=item.published_at,
|
||||
latitude=patch["latitude"],
|
||||
longitude=patch["longitude"],
|
||||
location_label=patch["location_label"],
|
||||
location_source=patch["location_source"],
|
||||
verified=patch["verified"],
|
||||
location_meta=patch["location_meta"],
|
||||
first_seen_at=now,
|
||||
last_seen_at=now,
|
||||
resolved_at=now if patch["verified"] else None,
|
||||
enrichment_status=item.enrichment_status,
|
||||
)
|
||||
if existing is None:
|
||||
db.add(record)
|
||||
else:
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口编辑。")
|
||||
record.title = item.title
|
||||
record.summary = item.summary
|
||||
record.content_language = item.content_language
|
||||
record.localizations = dict(item.localizations or {})
|
||||
record.url = item.url
|
||||
record.source = item.source
|
||||
record.feed_name = item.feed_name
|
||||
record.region = item.feed_region
|
||||
record.homepage_url = item.homepage_url
|
||||
record.published_at = item.published_at
|
||||
record.last_seen_at = now
|
||||
if target is None and record.location_source == "manual_location":
|
||||
merged_meta = dict(record.location_meta or {})
|
||||
patch_meta = patch.get("location_meta") if isinstance(patch, dict) else None
|
||||
patch_news_meta = patch_meta.get("news_meta") if isinstance(patch_meta, dict) else None
|
||||
if isinstance(patch_news_meta, dict):
|
||||
merged_meta["news_meta"] = patch_news_meta
|
||||
record.location_meta = merged_meta
|
||||
else:
|
||||
record.location_meta = patch["location_meta"]
|
||||
if target:
|
||||
record.latitude = patch["latitude"]
|
||||
record.longitude = patch["longitude"]
|
||||
record.location_label = patch["location_label"]
|
||||
record.location_source = patch["location_source"]
|
||||
record.verified = patch["verified"]
|
||||
record.resolved_at = now
|
||||
elif record.location_source != "manual_location":
|
||||
record.latitude = patch["latitude"]
|
||||
record.longitude = patch["longitude"]
|
||||
record.location_label = patch["location_label"]
|
||||
record.location_source = patch["location_source"]
|
||||
record.verified = patch["verified"]
|
||||
record.resolved_at = None
|
||||
record.enrichment_status = NewsEnrichmentStatus.PENDING.value
|
||||
record.enrichment_error = None
|
||||
record.enriched_at = None
|
||||
if content:
|
||||
meta = dict(record.location_meta or {})
|
||||
meta["manual_content"] = content
|
||||
record.location_meta = meta
|
||||
await db.flush()
|
||||
|
||||
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
|
||||
if queued:
|
||||
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
|
||||
await db.flush()
|
||||
return ManualNewsWriteResult(item=record, created=created, queued=queued)
|
||||
|
||||
|
||||
async def import_manual_news_items(
|
||||
db: AsyncSession,
|
||||
payload: list[Any],
|
||||
*,
|
||||
group_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if len(payload) > MANUAL_NEWS_MAX_IMPORT_ITEMS:
|
||||
raise ValueError(f"单次最多导入 {MANUAL_NEWS_MAX_IMPORT_ITEMS} 条。")
|
||||
created = 0
|
||||
updated = 0
|
||||
queued = 0
|
||||
errors: list[dict[str, Any]] = []
|
||||
for index, raw_item in enumerate(payload):
|
||||
if not isinstance(raw_item, dict):
|
||||
errors.append({"index": index, "error": "条目必须是 JSON 对象。"})
|
||||
continue
|
||||
try:
|
||||
result = await upsert_manual_news_item(db, raw_item, group_id=group_id)
|
||||
created += 1 if result.created else 0
|
||||
updated += 0 if result.created else 1
|
||||
queued += 1 if result.queued else 0
|
||||
except Exception as exc:
|
||||
errors.append({"index": index, "error": str(exc)})
|
||||
if errors and created == 0 and updated == 0:
|
||||
raise ValueError("导入失败，未写入任何新闻。")
|
||||
return {"created": created, "updated": updated, "queued": queued, "failed": len(errors), "errors": errors}
|
||||
|
||||
|
||||
async def parse_manual_news_import_upload(raw_bytes: bytes) -> list[Any]:
|
||||
try:
|
||||
payload = json.loads(raw_bytes.decode("utf-8-sig"))
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ValueError("JSON 文件必须使用 UTF-8 编码。") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"JSON 解析失败：第 {exc.lineno} 行第 {exc.colno} 列。") from exc
|
||||
if not isinstance(payload, list):
|
||||
raise ValueError("JSON 顶层必须是数组。")
|
||||
return payload
|
||||
|
||||
|
||||
def serialize_news_record(record: EarthNewsItem, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
|
||||
item = record_to_parsed_news_item(record)
|
||||
payload = _serialize_item(item, active_region=item.feed_region, locale=locale)
|
||||
news_meta = _news_meta(record)
|
||||
payload["editable"] = _manual_editable(record)
|
||||
payload["source_type"] = payload.get("feed_type")
|
||||
payload["status"] = record.enrichment_status
|
||||
payload["translated"] = bool((record.localizations or {}).get("zh-CN") and (record.localizations or {}).get("en-US"))
|
||||
payload["manual_content"] = (record.location_meta or {}).get("manual_content") if isinstance(record.location_meta, dict) else None
|
||||
payload["manual_group_id"] = news_meta.get("manual_group_id")
|
||||
payload["manual_group_name"] = news_meta.get("manual_group_name")
|
||||
return payload
|
||||
|
||||
|
||||
def _record_matches_group(record: EarthNewsItem, group_id: str) -> bool:
|
||||
source_type = _record_source_type(record)
|
||||
if source_type == NewsSourceType.MANUAL.value:
|
||||
return _record_manual_group_id(record) == group_id
|
||||
return _rss_group_id(record) == group_id
|
||||
|
||||
|
||||
async def list_news_records(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
page: int,
|
||||
page_size: int,
|
||||
source_type: str | None = None,
|
||||
region: str | None = None,
|
||||
category: str | None = None,
|
||||
status_filter: str | None = None,
|
||||
group_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
page = max(page, 1)
|
||||
page_size = min(max(page_size, 1), 100)
|
||||
query = select(EarthNewsItem)
|
||||
count_query = select(func.count(EarthNewsItem.id))
|
||||
filters = []
|
||||
if region and region != "all":
|
||||
filters.append(EarthNewsItem.region == region)
|
||||
if status_filter and status_filter != "all":
|
||||
filters.append(EarthNewsItem.enrichment_status == status_filter)
|
||||
if source_type and source_type != "all":
|
||||
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == source_type)
|
||||
if category and category != "all":
|
||||
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category") == category)
|
||||
for clause in filters:
|
||||
query = query.where(clause)
|
||||
count_query = count_query.where(clause)
|
||||
ordered_query = query.order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
|
||||
if group_id:
|
||||
result = await db.execute(ordered_query)
|
||||
all_records = [record for record in result.scalars().all() if _record_matches_group(record, group_id)]
|
||||
total = len(all_records)
|
||||
records = all_records[(page - 1) * page_size : page * page_size]
|
||||
else:
|
||||
total_result = await db.execute(count_query)
|
||||
result = await db.execute(
|
||||
ordered_query.offset((page - 1) * page_size).limit(page_size)
|
||||
)
|
||||
records = list(result.scalars().all())
|
||||
total = int(total_result.scalar() or 0)
|
||||
return {
|
||||
"items": [serialize_news_record(record) for record in records],
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
async def list_news_groups(db: AsyncSession, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
|
||||
manual_groups = await get_manual_news_groups(db)
|
||||
manual_by_id: dict[str, dict[str, Any]] = {
|
||||
str(group["id"]): {
|
||||
"id": str(group["id"]),
|
||||
"name": str(group["name"]),
|
||||
"group_type": "manual",
|
||||
"source_type": NewsSourceType.MANUAL.value,
|
||||
"editable": True,
|
||||
"sort_order": int(group.get("sort_order") or 0),
|
||||
"count": 0,
|
||||
"items": [],
|
||||
}
|
||||
for group in manual_groups
|
||||
}
|
||||
rss_by_id: dict[str, dict[str, Any]] = {}
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem).order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
|
||||
)
|
||||
for record in result.scalars().all():
|
||||
serialized = serialize_news_record(record, locale=locale)
|
||||
source_type = _record_source_type(record)
|
||||
if source_type == NewsSourceType.MANUAL.value:
|
||||
group_id = _record_manual_group_id(record)
|
||||
group = manual_by_id.setdefault(
|
||||
group_id,
|
||||
{
|
||||
"id": group_id,
|
||||
"name": str(_news_meta(record).get("manual_group_name") or DEFAULT_MANUAL_NEWS_GROUP_NAME),
|
||||
"group_type": "manual",
|
||||
"source_type": NewsSourceType.MANUAL.value,
|
||||
"editable": True,
|
||||
"sort_order": len(manual_by_id),
|
||||
"count": 0,
|
||||
"items": [],
|
||||
},
|
||||
)
|
||||
else:
|
||||
group_id = _rss_group_id(record)
|
||||
group = rss_by_id.setdefault(
|
||||
group_id,
|
||||
{
|
||||
"id": group_id,
|
||||
"name": record.feed_name or record.source or "RSS 新闻",
|
||||
"group_type": "rss",
|
||||
"source_type": source_type,
|
||||
"editable": False,
|
||||
"region": record.region,
|
||||
"source": record.source,
|
||||
"feed_name": record.feed_name,
|
||||
"count": 0,
|
||||
"items": [],
|
||||
},
|
||||
)
|
||||
group["count"] = int(group.get("count") or 0) + 1
|
||||
group.setdefault("items", []).append(serialized)
|
||||
manual_items = sorted(manual_by_id.values(), key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
|
||||
rss_items = sorted(rss_by_id.values(), key=lambda item: str(item.get("name") or ""))
|
||||
return {"groups": [*manual_items, *rss_items], "manual_groups": manual_items, "rss_groups": rss_items}
|
||||
|
||||
|
||||
async def get_news_record_or_404(db: AsyncSession, item_id: str) -> EarthNewsItem | None:
|
||||
return await db.get(EarthNewsItem, item_id)
|
||||
|
||||
|
||||
async def delete_manual_news_item(db: AsyncSession, item_id: str) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口删除。")
|
||||
await db.execute(delete(EarthNewsItem).where(EarthNewsItem.id == item_id))
|
||||
await db.flush()
|
||||
return True
|
||||
|
||||
|
||||
async def reprocess_manual_news_item(db: AsyncSession, item_id: str) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口重新处理。")
|
||||
item = record_to_parsed_news_item(record)
|
||||
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
|
||||
if queued:
|
||||
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
|
||||
record.enrichment_error = None
|
||||
await db.flush()
|
||||
return queued
|
||||
|
||||
|
||||
async def broadcast_manual_news_changed() -> None:
|
||||
await _broadcast_news_reload()
|
||||
@@ -14,10 +14,14 @@ from app.core.logging import get_logger
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
|
||||
TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority"
|
||||
TARGET_LOCATION_GROUP = "earth_news_target_location"
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
|
||||
TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000
|
||||
TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1
|
||||
TARGET_LOCATION_MAX_ATTEMPTS = 3
|
||||
|
||||
_redis_client: redis.Redis | None = None
|
||||
@@ -28,6 +32,7 @@ class NewsTargetLocationMessage:
|
||||
message_id: str
|
||||
item_id: str
|
||||
payload: dict[str, Any]
|
||||
stream_name: str = TARGET_LOCATION_STREAM
|
||||
attempts: int = 0
|
||||
|
||||
|
||||
@@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol):
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
...
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
...
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
@@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:queued:{item_id}"
|
||||
|
||||
|
||||
def _priority_queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:priority_queued:{item_id}"
|
||||
|
||||
|
||||
class RedisStreamsNewsTargetLocationQueue:
|
||||
def __init__(self, client: redis.Redis | None = None) -> None:
|
||||
self.client = client or _get_redis_client()
|
||||
@@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
async def _ensure_group(self) -> None:
|
||||
if self._group_ready:
|
||||
return
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
TARGET_LOCATION_STREAM,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM):
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
self._group_ready = True
|
||||
|
||||
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
|
||||
await self._ensure_group()
|
||||
if force:
|
||||
await self.client.delete(_result_key(item_id), _queued_key(item_id))
|
||||
await self.client.delete(_result_key(item_id))
|
||||
queued_key = _priority_queued_key(item_id)
|
||||
elif await self.client.exists(_result_key(item_id)):
|
||||
return False
|
||||
else:
|
||||
queued_key = _queued_key(item_id)
|
||||
dedup_ttl = (
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS
|
||||
if force
|
||||
else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS
|
||||
)
|
||||
queued = await self.client.set(
|
||||
_queued_key(item_id),
|
||||
queued_key,
|
||||
"1",
|
||||
nx=True,
|
||||
ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
|
||||
ex=dedup_ttl,
|
||||
)
|
||||
if not queued:
|
||||
return bool(await self.client.exists(_queued_key(item_id)))
|
||||
return bool(await self.client.exists(queued_key))
|
||||
stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
stream_name,
|
||||
{
|
||||
"item_id": item_id,
|
||||
"attempts": "0",
|
||||
@@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
block_ms: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
await self._ensure_group()
|
||||
streams = await self.client.xreadgroup(
|
||||
streams = []
|
||||
priority_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_PRIORITY_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if priority_claimed:
|
||||
return priority_claimed
|
||||
|
||||
priority_messages = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
{TARGET_LOCATION_PRIORITY_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS,
|
||||
)
|
||||
if priority_messages:
|
||||
streams = priority_messages
|
||||
else:
|
||||
regular_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if regular_claimed:
|
||||
return regular_claimed
|
||||
streams = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
)
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for _stream_name, stream_messages in streams:
|
||||
for stream_name, stream_messages in streams:
|
||||
for message_id, fields in stream_messages:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
messages.append(
|
||||
NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
attempts=attempts,
|
||||
)
|
||||
)
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
|
||||
async def _claim_stale_messages(
|
||||
self,
|
||||
*,
|
||||
stream_name: str,
|
||||
consumer_name: str,
|
||||
count: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
try:
|
||||
_next_id, claimed, _deleted = await self.client.xautoclaim(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS,
|
||||
start_id="0-0",
|
||||
count=count,
|
||||
)
|
||||
except ResponseError:
|
||||
return []
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for message_id, fields in claimed:
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def _message_from_fields(
|
||||
self,
|
||||
stream_name: str,
|
||||
message_id: str,
|
||||
fields: dict[str, str],
|
||||
) -> NewsTargetLocationMessage | None:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
return NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
stream_name=stream_name,
|
||||
attempts=attempts,
|
||||
)
|
||||
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id)
|
||||
await self.client.xdel(message.stream_name, message.message_id)
|
||||
|
||||
async def _discard_message(self, stream_name: str, message_id: str) -> None:
|
||||
await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id)
|
||||
await self.client.xdel(stream_name, message_id)
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
self,
|
||||
@@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
*,
|
||||
error: str,
|
||||
) -> None:
|
||||
await self.ack(message.message_id)
|
||||
await self.ack(message)
|
||||
if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM,
|
||||
@@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
)
|
||||
return
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
message.stream_name,
|
||||
{
|
||||
"item_id": message.item_id,
|
||||
"attempts": str(message.attempts + 1),
|
||||
@@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS,
|
||||
json.dumps(patch, ensure_ascii=False),
|
||||
)
|
||||
await client.delete(_queued_key(item_id))
|
||||
await client.delete(_queued_key(item_id), _priority_queued_key(item_id))
|
||||
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
@@ -13,6 +13,22 @@ from app.services.earth_news import (
|
||||
build_anchor_location_patch,
|
||||
_news_meta_patch,
|
||||
)
|
||||
from app.services.earth_news_classification import (
|
||||
breaking_sort_rank,
|
||||
normalize_breaking_level,
|
||||
normalize_breaking_scope,
|
||||
)
|
||||
|
||||
CRUISE_REGION_ORDER = (
|
||||
"americas",
|
||||
"europe",
|
||||
"middle-east-africa",
|
||||
"asia-pacific",
|
||||
"global",
|
||||
)
|
||||
CRUISE_REGION_QUERY_MULTIPLIER = 12
|
||||
CRUISE_REGION_QUERY_MIN_LIMIT = 240
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT = 1000
|
||||
|
||||
|
||||
def _coerce_datetime(value: datetime | None) -> datetime | None:
|
||||
@@ -23,6 +39,17 @@ def _coerce_datetime(value: datetime | None) -> datetime | None:
|
||||
return value.astimezone(UTC)
|
||||
|
||||
|
||||
def _coerce_meta_datetime(value: Any) -> datetime | None:
|
||||
if isinstance(value, datetime):
|
||||
return _coerce_datetime(value)
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
return None
|
||||
try:
|
||||
return _coerce_datetime(datetime.fromisoformat(value.replace("Z", "+00:00")))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _location_patch_from_record(record: EarthNewsItem) -> dict[str, Any]:
|
||||
return {
|
||||
"latitude": record.latitude,
|
||||
@@ -64,10 +91,67 @@ def record_to_parsed_news_item(record: EarthNewsItem) -> ParsedNewsItem:
|
||||
importance_level=str(news_meta.get("importance_level") or "low"),
|
||||
importance_reasons=list(news_meta.get("importance_reasons") or []),
|
||||
market_impact=str(news_meta.get("market_impact") or "none"),
|
||||
breaking_level=normalize_breaking_level(news_meta.get("breaking_level")).value,
|
||||
breaking_scope=normalize_breaking_scope(news_meta.get("breaking_scope")).value,
|
||||
breaking_reasons=list(news_meta.get("breaking_reasons") or []),
|
||||
breaking_source=str(news_meta.get("breaking_source") or "rules"),
|
||||
breaking_confidence=float(news_meta.get("breaking_confidence") or 0),
|
||||
breaking_expires_at=_coerce_meta_datetime(news_meta.get("breaking_expires_at")),
|
||||
)
|
||||
return apply_enrichment_patch_to_item(item, _location_patch_from_record(record))
|
||||
|
||||
|
||||
def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
return sorted(
|
||||
items,
|
||||
key=lambda item: (
|
||||
-breaking_sort_rank(item),
|
||||
False
|
||||
if active_region == "global"
|
||||
or (breaking_sort_rank(item) > 0 and normalize_breaking_scope(item.breaking_scope).value == "global")
|
||||
else item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _diversify_parsed_news_items_by_region(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
if limit <= 0:
|
||||
return []
|
||||
sorted_items = _sort_parsed_news_items(items, active_region="global")
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
for item in sorted_items:
|
||||
region = item.feed_region or "global"
|
||||
buckets.setdefault(region, []).append(item)
|
||||
|
||||
ordered_regions = [
|
||||
*[region for region in CRUISE_REGION_ORDER if buckets.get(region)],
|
||||
*sorted(region for region in buckets if region not in CRUISE_REGION_ORDER),
|
||||
]
|
||||
diversified: list[ParsedNewsItem] = []
|
||||
cursor = 0
|
||||
while len(diversified) < limit:
|
||||
added = False
|
||||
for region in ordered_regions:
|
||||
bucket = buckets.get(region) or []
|
||||
if cursor >= len(bucket):
|
||||
continue
|
||||
diversified.append(bucket[cursor])
|
||||
added = True
|
||||
if len(diversified) >= limit:
|
||||
break
|
||||
if not added:
|
||||
break
|
||||
cursor += 1
|
||||
return diversified
|
||||
|
||||
|
||||
def _query_sort_key(active_region: str):
|
||||
if active_region == "global":
|
||||
return (
|
||||
@@ -95,42 +179,6 @@ def _source_filter_clause(source_ids: set[str] | None):
|
||||
return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids))
|
||||
|
||||
|
||||
def _record_source_id(record: EarthNewsItem) -> str:
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
|
||||
source_id = str(news_meta.get("source_id") or "").strip()
|
||||
if source_id:
|
||||
return source_id
|
||||
if isinstance(record.id, str) and ":" in record.id:
|
||||
return record.id.split(":", 1)[0]
|
||||
return record.feed_name or record.source or record.id
|
||||
|
||||
|
||||
def _diversify_records_by_source(records: list[EarthNewsItem], *, limit: int) -> list[EarthNewsItem]:
|
||||
if limit <= 0 or len(records) <= limit:
|
||||
return records[:limit]
|
||||
buckets: dict[str, list[EarthNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for record in records:
|
||||
source_id = _record_source_id(record)
|
||||
if source_id not in buckets:
|
||||
buckets[source_id] = []
|
||||
order.append(source_id)
|
||||
buckets[source_id].append(record)
|
||||
|
||||
diversified: list[EarthNewsItem] = []
|
||||
while len(diversified) < limit and order:
|
||||
next_order: list[str] = []
|
||||
for source_id in order:
|
||||
bucket = buckets.get(source_id) or []
|
||||
if bucket and len(diversified) < limit:
|
||||
diversified.append(bucket.pop(0))
|
||||
if bucket:
|
||||
next_order.append(source_id)
|
||||
order = next_order
|
||||
return diversified
|
||||
|
||||
|
||||
async def list_earth_news_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -139,14 +187,20 @@ async def list_earth_news_items(
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
query_limit = limit if source_ids else min(max(limit * 4, limit), 100)
|
||||
query_limit = limit if source_ids else min(max(limit * 20, limit), 500)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(*_query_sort_key(active_region))
|
||||
.limit(query_limit)
|
||||
)
|
||||
if active_region != "global":
|
||||
query = query.where(EarthNewsItem.region.in_({"global", active_region}))
|
||||
news_meta = EarthNewsItem.location_meta.op("->")("news_meta")
|
||||
query = query.where(
|
||||
or_(
|
||||
EarthNewsItem.region.in_({"global", active_region}),
|
||||
news_meta.op("->>")("breaking_scope") == "global",
|
||||
)
|
||||
)
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
query = query.where(category_clause)
|
||||
@@ -155,11 +209,13 @@ async def list_earth_news_items(
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
records = list(result.scalars().all())
|
||||
if not source_ids:
|
||||
records = _diversify_records_by_source(records, limit=limit)
|
||||
else:
|
||||
records = records[:limit]
|
||||
return [record_to_parsed_news_item(record) for record in records]
|
||||
items = _sort_parsed_news_items(
|
||||
[record_to_parsed_news_item(record) for record in records],
|
||||
active_region=active_region,
|
||||
)
|
||||
if active_region == "global" and not source_ids:
|
||||
return _diversify_parsed_news_items_by_region(items, limit=limit)
|
||||
return items[:limit]
|
||||
|
||||
|
||||
async def list_earth_news_cruise_items(
|
||||
@@ -169,15 +225,20 @@ async def list_earth_news_cruise_items(
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
query_limit = min(
|
||||
max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT),
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT,
|
||||
)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.last_seen_at.desc(),
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
.limit(limit)
|
||||
.limit(query_limit)
|
||||
)
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
@@ -186,7 +247,10 @@ async def list_earth_news_cruise_items(
|
||||
if source_clause is not None:
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
return [record_to_parsed_news_item(record) for record in result.scalars().all()]
|
||||
return _diversify_parsed_news_items_by_region(
|
||||
[record_to_parsed_news_item(record) for record in result.scalars().all()],
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
|
||||
async def get_earth_news_freshness(
|
||||
@@ -334,13 +398,28 @@ async def update_earth_news_item_enrichment(
|
||||
if record is None:
|
||||
return False
|
||||
if "latitude" in patch:
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = dict(patch.get("location_meta") or {})
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
patch_meta = dict(patch.get("location_meta") or {})
|
||||
if record.location_source == "manual_location":
|
||||
current_meta = dict(record.location_meta or {})
|
||||
patch_news_meta = patch_meta.get("news_meta")
|
||||
if isinstance(patch_news_meta, dict):
|
||||
current_meta["news_meta"] = patch_news_meta
|
||||
current_meta["manual_enrichment"] = {
|
||||
"resolution_stage": patch_meta.get("resolution_stage"),
|
||||
"ai_attempted": patch_meta.get("ai_attempted"),
|
||||
"ai_status": patch_meta.get("ai_status"),
|
||||
"ai_error": patch_meta.get("ai_error"),
|
||||
"debug_note": patch_meta.get("debug_note"),
|
||||
}
|
||||
record.location_meta = current_meta
|
||||
else:
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = patch_meta
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
if "content_language" in patch:
|
||||
record.content_language = str(patch.get("content_language") or "en")
|
||||
if "localizations" in patch:
|
||||
|
||||
@@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news")
|
||||
WORKER_BATCH_SIZE = 4
|
||||
WORKER_BLOCK_MS = 5000
|
||||
WORKER_BACKOFF_SECONDS = 5.0
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0
|
||||
WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0
|
||||
|
||||
_worker_task: asyncio.Task | None = None
|
||||
|
||||
@@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None:
|
||||
if not messages:
|
||||
continue
|
||||
provider_client = await _build_provider_client()
|
||||
for message in messages:
|
||||
job_timeout = _get_worker_job_timeout(provider_client)
|
||||
|
||||
async def handle_message(message: NewsTargetLocationMessage) -> None:
|
||||
try:
|
||||
await process_target_location_message(message, provider_client=provider_client)
|
||||
await queue.ack(message.message_id)
|
||||
await asyncio.wait_for(
|
||||
process_target_location_message(message, provider_client=provider_client),
|
||||
timeout=job_timeout,
|
||||
)
|
||||
await queue.ack(message)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except TimeoutError as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job timed out",
|
||||
event="earth_news.target_location.worker_job_timeout",
|
||||
context={"item_id": message.item_id, "timeout_seconds": job_timeout},
|
||||
)
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out")
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job failed",
|
||||
@@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None:
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc))
|
||||
|
||||
await asyncio.gather(*(handle_message(message) for message in messages))
|
||||
|
||||
|
||||
def start_earth_news_target_worker() -> None:
|
||||
global _worker_task
|
||||
@@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None:
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
_worker_task = None
|
||||
|
||||
|
||||
def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float:
|
||||
timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS)
|
||||
retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1)
|
||||
return min(
|
||||
max(
|
||||
timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS,
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS,
|
||||
),
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS,
|
||||
)
|
||||
|
||||
@@ -9,12 +9,12 @@ yet to keep behavior obvious after settings changes).
|
||||
from __future__ import annotations
|
||||
|
||||
from email.message import EmailMessage
|
||||
from typing import Literal, Optional
|
||||
from typing import Optional
|
||||
|
||||
import aiosmtplib
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
OtpPurpose = Literal["register", "verify_email", "reset_password"]
|
||||
from app.core.enums import OtpPurpose
|
||||
|
||||
|
||||
class EmailError(Exception):
|
||||
@@ -81,15 +81,15 @@ async def send_email(
|
||||
|
||||
|
||||
_SUBJECTS: dict[OtpPurpose, str] = {
|
||||
"register": "Confirm your Planet account",
|
||||
"verify_email": "Verify your Planet email",
|
||||
"reset_password": "Reset your Planet password",
|
||||
OtpPurpose.REGISTER: "Confirm your Planet account",
|
||||
OtpPurpose.VERIFY_EMAIL: "Verify your Planet email",
|
||||
OtpPurpose.RESET_PASSWORD: "Reset your Planet password",
|
||||
}
|
||||
|
||||
_HEADLINES: dict[OtpPurpose, str] = {
|
||||
"register": "Welcome to Planet — confirm your email to activate your account.",
|
||||
"verify_email": "Confirm your new email address to keep your Planet account active.",
|
||||
"reset_password": "Use this code to set a new password for your Planet account.",
|
||||
OtpPurpose.REGISTER: "Welcome to Planet — confirm your email to activate your account.",
|
||||
OtpPurpose.VERIFY_EMAIL: "Confirm your new email address to keep your Planet account active.",
|
||||
OtpPurpose.RESET_PASSWORD: "Use this code to set a new password for your Planet account.",
|
||||
}
|
||||
|
||||
|
||||
|
||||
186
backend/app/services/llm_model_catalog.py
Normal file
186
backend/app/services/llm_model_catalog.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Authenticated model discovery shared by refresh and connection checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
CATALOG_TIMEOUT_SECONDS = 30
|
||||
CATALOG_MAX_PAGES = 100
|
||||
CATALOG_PAGE_SIZE = 100
|
||||
CATALOG_REQUEST_ATTEMPTS = 2
|
||||
CATALOG_RETRY_DELAY_SECONDS = 0.2
|
||||
SUPPORTED_PROVIDER_APIS = {
|
||||
"openai-completions",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"ollama-generate",
|
||||
}
|
||||
|
||||
|
||||
class LLMProviderCatalogError(RuntimeError):
|
||||
"""A safe, user-facing catalog failure with no upstream response or credentials."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCatalog:
|
||||
url: str
|
||||
models: list[str]
|
||||
|
||||
|
||||
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
if parts.scheme not in {"http", "https"} or not parts.hostname:
|
||||
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
|
||||
if parts.username or parts.password or parts.query or parts.fragment:
|
||||
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
|
||||
if provider_api not in SUPPORTED_PROVIDER_APIS:
|
||||
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
|
||||
path = parts.path.rstrip("/")
|
||||
if provider_api == "ollama-generate":
|
||||
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
|
||||
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
|
||||
path = "/api/v1/models"
|
||||
else:
|
||||
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
|
||||
path += "/v1"
|
||||
path += "/models"
|
||||
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
|
||||
|
||||
|
||||
def catalog_error_message(exc: Exception) -> str:
|
||||
if isinstance(exc, LLMProviderCatalogError):
|
||||
return str(exc)
|
||||
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
|
||||
return "模型目录请求超时，请检查网络后重试。"
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
code = exc.response.status_code
|
||||
messages = {
|
||||
401: "API Key 验证失败，请检查当前供应商的凭证。",
|
||||
403: "当前 API Key 无权访问该模型目录，请检查账号权限和服务地域。",
|
||||
404: "模型目录接口不存在，请检查基础地址、地域和接口协议。",
|
||||
429: "供应商请求限流，请稍后重试。",
|
||||
}
|
||||
return messages.get(code, f"供应商模型目录返回 HTTP {code}，请稍后重试。")
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return "无法连接模型目录，请检查基础地址和网络。"
|
||||
return "模型目录响应无效，请稍后重试。"
|
||||
|
||||
|
||||
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
envelope = payload.get("output", payload)
|
||||
if not isinstance(envelope, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
rows = envelope.get("data", envelope.get("models"))
|
||||
if not isinstance(rows, list):
|
||||
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
|
||||
return rows, envelope
|
||||
|
||||
|
||||
def _model_id(row: dict[str, object]) -> str:
|
||||
value = row.get("id") or row.get("model") or row.get("name")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _model_date(row: dict[str, object]) -> float:
|
||||
value = row.get("created_at") or row.get("published_time") or row.get("created")
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
async def _get_catalog_page(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
headers: dict[str, str],
|
||||
params: dict[str, str | int],
|
||||
) -> object:
|
||||
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
|
||||
try:
|
||||
response = await client.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if attempt or exc.response.status_code not in {502, 503, 504}:
|
||||
raise
|
||||
except httpx.TransportError:
|
||||
if attempt:
|
||||
raise
|
||||
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
|
||||
raise LLMProviderCatalogError("模型目录请求失败。")
|
||||
|
||||
|
||||
async def fetch_model_catalog(
|
||||
provider: str,
|
||||
base_url: str,
|
||||
provider_api: str,
|
||||
api_key: str = "",
|
||||
anthropic_version: str = "2023-06-01",
|
||||
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
|
||||
) -> ModelCatalog:
|
||||
url = model_catalog_url(provider, base_url, provider_api)
|
||||
public_catalog = provider in {"opencode-go", "openrouter"}
|
||||
if not api_key and provider_api != "ollama-generate" and not public_catalog:
|
||||
raise LLMProviderCatalogError("请先配置当前供应商的 API Key，再刷新模型列表。")
|
||||
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
|
||||
if provider_api == "anthropic-messages" and provider not in {
|
||||
"opencode-go",
|
||||
"openrouter",
|
||||
"alibaba",
|
||||
"moonshotai",
|
||||
}:
|
||||
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
|
||||
elif api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
|
||||
params: dict[str, str | int] = {}
|
||||
if native_dashscope:
|
||||
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
|
||||
rows_by_id: dict[str, dict[str, object]] = {}
|
||||
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
|
||||
# Bound the complete pagination/retry cycle, not just each individual request.
|
||||
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
|
||||
for page in range(CATALOG_MAX_PAGES):
|
||||
payload = await _get_catalog_page(client, url, headers, params)
|
||||
rows, envelope = _model_rows(payload)
|
||||
previous_count = len(rows_by_id)
|
||||
for row in rows:
|
||||
rows_by_id[_model_id(row)] = row
|
||||
has_more = envelope.get("has_more") is True
|
||||
if native_dashscope:
|
||||
total = envelope.get("total")
|
||||
if not isinstance(total, int) or total < 0:
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
has_more = len(rows_by_id) < total
|
||||
params["page_no"] = page + 2
|
||||
elif has_more:
|
||||
cursor = envelope.get("last_id")
|
||||
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
params["after_id"] = cursor
|
||||
if not has_more:
|
||||
models = sorted(
|
||||
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
|
||||
)
|
||||
# An empty Ollama catalog is valid: no models have been installed yet.
|
||||
if not models and provider_api != "ollama-generate":
|
||||
raise LLMProviderCatalogError("供应商返回了空模型目录，已保留上次模型列表。")
|
||||
return ModelCatalog(url=url, models=models)
|
||||
if len(rows_by_id) == previous_count:
|
||||
raise LLMProviderCatalogError("供应商模型目录分页没有进展，已保留上次模型列表。")
|
||||
raise LLMProviderCatalogError("供应商模型目录分页超过限制，已保留上次模型列表。")
|
||||
@@ -3,17 +3,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
from app.services.llm_model_catalog import fetch_model_catalog
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m3": "anthropic-messages",
|
||||
"qwen3.8-max": "anthropic-messages",
|
||||
"qwen3.8-flash": "anthropic-messages",
|
||||
"qwen3.7-max": "anthropic-messages",
|
||||
"qwen3.7-plus": "anthropic-messages",
|
||||
"qwen3.6-plus": "anthropic-messages",
|
||||
"grok-4.6": "openai-responses",
|
||||
"gpt-5.6-luna": "openai-responses",
|
||||
"muse-spark-1.3-contributor": "openai-responses",
|
||||
"muse-spark-1.2-contributor": "openai-responses",
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m3",
|
||||
"kimi-k3",
|
||||
"glm-5.3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
|
||||
]
|
||||
|
||||
|
||||
OPENAI_MODEL_PROVIDER_APIS = {
|
||||
model: "openai-responses"
|
||||
for model in [
|
||||
"gpt-6-astra",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.1",
|
||||
"gpt-5.1-codex",
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"model": "MiniMax-M3",
|
||||
"models": [
|
||||
"MiniMax-M3",
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
"MiniMax-M2.1",
|
||||
"MiniMax-M2.1-highspeed",
|
||||
"MiniMax-M2",
|
||||
],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": "openai-responses",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"model": "gpt-6-astra",
|
||||
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
|
||||
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"model": "claude-opus-5",
|
||||
"models": ["claude-opus-5", "claude-sonnet-4-6"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"model": "deepseek-flash",
|
||||
"models": ["deepseek-flash", "deepseek-v4-pro"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"model": "qwen3.8-max",
|
||||
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"model": "kimi-k3",
|
||||
"models": ["kimi-k3", "kimi-k2.5"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
async def refresh_llm_provider_preset(
|
||||
provider: str,
|
||||
api_key: str | None = None,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
provider_api: str | None = None,
|
||||
anthropic_version: str = "2023-06-01",
|
||||
) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
resolved_base_url = base_url or fallback["base_url"]
|
||||
resolved_api = provider_api or fallback["provider_api"]
|
||||
catalog = await fetch_model_catalog(
|
||||
fallback["provider"],
|
||||
resolved_base_url,
|
||||
resolved_api,
|
||||
api_key or "",
|
||||
anthropic_version,
|
||||
)
|
||||
model_provider_apis = (
|
||||
_opencode_go_model_provider_apis(catalog.models)
|
||||
if fallback["provider"] == "opencode-go"
|
||||
else fallback.get("model_provider_apis", {})
|
||||
)
|
||||
if (
|
||||
fallback["provider"] == "openai"
|
||||
and urlsplit(resolved_base_url).hostname != "api.openai.com"
|
||||
):
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
"base_url": resolved_base_url,
|
||||
"provider_api": resolved_api,
|
||||
"model": catalog.models[0] if catalog.models else "",
|
||||
"models": catalog.models,
|
||||
"model_provider_apis": model_provider_apis,
|
||||
"source": catalog.url,
|
||||
}
|
||||
return refreshed
|
||||
|
||||
@@ -9,14 +9,12 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import secrets
|
||||
from typing import Literal
|
||||
|
||||
import bcrypt
|
||||
|
||||
from app.core.enums import OtpPurpose
|
||||
from app.core.security import redis_client
|
||||
|
||||
OtpPurpose = Literal["register", "verify_email", "reset_password"]
|
||||
|
||||
CODE_TTL_SECONDS = 600 # 10 minutes
|
||||
RESEND_COOLDOWN_SECONDS = 60
|
||||
MAX_ATTEMPTS = 5
|
||||
|
||||
@@ -7,9 +7,14 @@ from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.enums import (
|
||||
PlaygroundMessageKind,
|
||||
PlaygroundMessageRole,
|
||||
PlaygroundMessageStatus,
|
||||
)
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
@@ -21,7 +26,6 @@ from app.schemas.ai import (
|
||||
PlaygroundMessageRecord,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
@@ -37,6 +41,13 @@ STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
ORPHANED_RUN_MESSAGE = "后台生成任务已中断，请点击上一条用户消息的重试按钮重新生成。"
|
||||
ACTIVE_MESSAGE_STATUSES = frozenset(
|
||||
{
|
||||
PlaygroundMessageStatus.PENDING.value,
|
||||
PlaygroundMessageStatus.THINKING.value,
|
||||
PlaygroundMessageStatus.ANSWERING.value,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class _ActiveRun:
|
||||
@@ -93,7 +104,11 @@ async def _require_visible_message(
|
||||
result = await db.execute(select(PlaygroundMessage).where(*conditions))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
detail = "User message not found" if role == "user" else "Playground message not found"
|
||||
detail = (
|
||||
"User message not found"
|
||||
if role == PlaygroundMessageRole.USER.value
|
||||
else "Playground message not found"
|
||||
)
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
|
||||
return message
|
||||
|
||||
@@ -108,7 +123,7 @@ def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None
|
||||
content=message.content or "",
|
||||
thinking_content=message.thinking_content or "",
|
||||
meta=list(message.meta or []),
|
||||
markdown=message.role != "system",
|
||||
markdown=message.role != PlaygroundMessageRole.SYSTEM.value,
|
||||
provider=message.provider,
|
||||
model=message.model,
|
||||
request_id=message.request_id,
|
||||
@@ -197,11 +212,11 @@ async def _reconcile_orphaned_active_messages(
|
||||
) -> list[PlaygroundMessage]:
|
||||
changed = False
|
||||
for item in messages:
|
||||
if item.status not in {"pending", "thinking", "answering"}:
|
||||
if item.status not in ACTIVE_MESSAGE_STATUSES:
|
||||
continue
|
||||
if item.public_id in _ACTIVE_RUNS:
|
||||
continue
|
||||
item.status = "error"
|
||||
item.status = PlaygroundMessageStatus.ERROR.value
|
||||
item.content = item.content or ORPHANED_RUN_MESSAGE
|
||||
orphan_meta = "错误: 后台任务已中断"
|
||||
if orphan_meta not in (item.meta or []):
|
||||
@@ -330,9 +345,9 @@ async def create_turn(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
role="user",
|
||||
kind="message",
|
||||
status="done",
|
||||
role=PlaygroundMessageRole.USER.value,
|
||||
kind=PlaygroundMessageKind.MESSAGE.value,
|
||||
status=PlaygroundMessageStatus.DONE.value,
|
||||
title=payload.selected_preset_key,
|
||||
content=payload.input,
|
||||
meta=[payload.title],
|
||||
@@ -343,9 +358,9 @@ async def create_turn(
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=None,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
role=PlaygroundMessageRole.ASSISTANT.value,
|
||||
kind=PlaygroundMessageKind.THINKING.value,
|
||||
status=PlaygroundMessageStatus.PENDING.value,
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
@@ -397,9 +412,9 @@ async def _create_assistant_retry_turn(
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=user_message.id,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
role=PlaygroundMessageRole.ASSISTANT.value,
|
||||
kind=PlaygroundMessageKind.THINKING.value,
|
||||
status=PlaygroundMessageStatus.PENDING.value,
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
@@ -439,7 +454,7 @@ async def stop_message(
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
|
||||
|
||||
if message.status not in {"pending", "thinking", "answering"}:
|
||||
if message.status not in ACTIVE_MESSAGE_STATUSES:
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
active_run = _ACTIVE_RUNS.get(message.public_id)
|
||||
@@ -447,7 +462,7 @@ async def stop_message(
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
|
||||
message.status = "stopped"
|
||||
message.status = PlaygroundMessageStatus.STOPPED.value
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
@@ -469,7 +484,7 @@ async def resend_turn(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
role=PlaygroundMessageRole.USER.value,
|
||||
)
|
||||
|
||||
later_messages = await db.execute(
|
||||
@@ -481,7 +496,7 @@ async def resend_turn(
|
||||
)
|
||||
for item in later_messages.scalars().all():
|
||||
item.is_visible = False
|
||||
if item.status in {"pending", "thinking", "answering"}:
|
||||
if item.status in ACTIVE_MESSAGE_STATUSES:
|
||||
active_run = _ACTIVE_RUNS.get(item.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
@@ -520,7 +535,7 @@ async def edit_user_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
role=PlaygroundMessageRole.USER.value,
|
||||
)
|
||||
|
||||
user_message.content = payload.content.strip()
|
||||
@@ -566,12 +581,12 @@ def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_u
|
||||
for item in messages:
|
||||
if item.id >= current_user_message_id:
|
||||
break
|
||||
if item.role == "system":
|
||||
if item.role == PlaygroundMessageRole.SYSTEM.value:
|
||||
continue
|
||||
history.append(
|
||||
{
|
||||
"role": item.role,
|
||||
"kind": item.kind or "message",
|
||||
"kind": item.kind or PlaygroundMessageKind.MESSAGE.value,
|
||||
"title": item.title,
|
||||
"content": item.content or "",
|
||||
}
|
||||
@@ -620,6 +635,7 @@ async def _run_assistant_message(
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"session_id": session_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
@@ -668,7 +684,11 @@ async def _run_assistant_message(
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="thinking" if analysis.thinking_blocks else "answering",
|
||||
status=(
|
||||
PlaygroundMessageStatus.THINKING.value
|
||||
if analysis.thinking_blocks
|
||||
else PlaygroundMessageStatus.ANSWERING.value
|
||||
),
|
||||
title=f"{analysis.provider} / {analysis.model}",
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
@@ -706,7 +726,7 @@ async def _run_assistant_message(
|
||||
await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="answering",
|
||||
status=PlaygroundMessageStatus.ANSWERING.value,
|
||||
content=content[:cursor],
|
||||
)
|
||||
await db.commit()
|
||||
@@ -717,7 +737,7 @@ async def _run_assistant_message(
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="done",
|
||||
status=PlaygroundMessageStatus.DONE.value,
|
||||
content=content,
|
||||
meta=[
|
||||
f"Request ID: {request_id}",
|
||||
@@ -762,8 +782,8 @@ async def _run_assistant_message(
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None and message.status in {"pending", "thinking", "answering"}:
|
||||
message.status = "stopped"
|
||||
if message is not None and message.status in ACTIVE_MESSAGE_STATUSES:
|
||||
message.status = PlaygroundMessageStatus.STOPPED.value
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
@@ -795,7 +815,7 @@ async def _run_assistant_message(
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None:
|
||||
message.status = "error"
|
||||
message.status = PlaygroundMessageStatus.ERROR.value
|
||||
message.content = message.content or f"分析失败：{error_message}"
|
||||
message.meta = [
|
||||
*(message.meta or []),
|
||||
|
||||
@@ -8,6 +8,7 @@ from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
from apscheduler.triggers.interval import IntervalTrigger
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.enums import JobStatus
|
||||
from app.core.logging import get_logger
|
||||
from app.db.session import async_session_factory
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -140,7 +141,7 @@ async def run_collector_task(collector_name: str):
|
||||
select(CollectionTask)
|
||||
.where(
|
||||
CollectionTask.datasource_id == datasource.id,
|
||||
CollectionTask.status == "running",
|
||||
CollectionTask.status == JobStatus.RUNNING.value,
|
||||
)
|
||||
.order_by(CollectionTask.started_at.desc(), CollectionTask.id.desc())
|
||||
.limit(1)
|
||||
@@ -184,7 +185,7 @@ async def run_collector_task(collector_name: str):
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({RUNNING_TASK_GUARD_TIMEOUT_MINUTES}m) in scheduler guard"
|
||||
)
|
||||
existing_running.status = "failed"
|
||||
existing_running.status = JobStatus.FAILED.value
|
||||
existing_running.phase = "failed"
|
||||
existing_running.completed_at = now
|
||||
existing_running.error_message = (
|
||||
@@ -243,7 +244,7 @@ async def run_collector_task(collector_name: str):
|
||||
return
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = task_result.get("status")
|
||||
if datasource.last_status == "success":
|
||||
if datasource.last_status == JobStatus.SUCCESS.value:
|
||||
effective_candidate = await get_builtin_effective_candidate(db, datasource_source)
|
||||
checksum, _credential_context = await build_builtin_connectivity_checksum(
|
||||
datasource_source,
|
||||
@@ -284,7 +285,7 @@ async def run_collector_task(collector_name: str):
|
||||
await db.rollback()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_status = JobStatus.CANCELLED.value
|
||||
await db.commit()
|
||||
logger.warning_event(
|
||||
"Collector cancelled by operator",
|
||||
@@ -306,7 +307,7 @@ async def run_collector_task(collector_name: str):
|
||||
await db.rollback()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_status = JobStatus.FAILED.value
|
||||
await db.commit()
|
||||
logger.exception_event(
|
||||
"Collector failed",
|
||||
@@ -335,7 +336,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(
|
||||
select(CollectionTask).where(
|
||||
CollectionTask.status == "running",
|
||||
CollectionTask.status == JobStatus.RUNNING.value,
|
||||
CollectionTask.started_at.is_not(None),
|
||||
CollectionTask.started_at < cutoff,
|
||||
)
|
||||
@@ -343,7 +344,7 @@ async def cleanup_stale_running_tasks(max_age_hours: int = 2) -> int:
|
||||
stale_tasks = result.scalars().all()
|
||||
|
||||
for task in stale_tasks:
|
||||
task.status = "failed"
|
||||
task.status = JobStatus.FAILED.value
|
||||
task.phase = "failed"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
existing_error = (task.error_message or "").strip()
|
||||
|
||||
@@ -6,6 +6,7 @@ from typing import Any
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.enums import BGPStatus
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
@@ -49,11 +50,11 @@ async def build_situational_alert_brief_request(
|
||||
|
||||
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
active_incidents_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == BGPStatus.ACTIVE.value)
|
||||
)
|
||||
bgp_severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.where(BGPIncident.status == "active")
|
||||
.where(BGPIncident.status == BGPStatus.ACTIVE.value)
|
||||
.group_by(BGPIncident.severity)
|
||||
)
|
||||
bgp_region_counter: Counter[str] = Counter()
|
||||
@@ -65,11 +66,11 @@ async def build_situational_alert_brief_request(
|
||||
|
||||
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
active_anomalies_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
|
||||
)
|
||||
anomaly_type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.where(BGPAnomaly.status == "active")
|
||||
.where(BGPAnomaly.status == BGPStatus.ACTIVE.value)
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
.limit(6)
|
||||
|
||||
@@ -7,6 +7,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.core.enums import UserRole
|
||||
from app.core.security import redis_client
|
||||
|
||||
SYSTEM_TASK_TTL_SECONDS = 24 * 60 * 60
|
||||
@@ -47,7 +48,7 @@ def normalize_user_role(role: Any) -> str:
|
||||
|
||||
|
||||
def require_super_admin(user_role: Any) -> bool:
|
||||
return normalize_user_role(user_role) == "super_admin"
|
||||
return normalize_user_role(user_role) == UserRole.SUPER_ADMIN.value
|
||||
|
||||
|
||||
def build_task_id(prefix: str = "restart") -> str:
|
||||
|
||||
@@ -13,6 +13,7 @@ from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.core.enums import LogLevel
|
||||
from app.core.security import redis_client
|
||||
from app.models.system_log import AuditLog, ObservabilityEvent, ObservabilityEventGroup, SystemLog
|
||||
from sqlalchemy import select
|
||||
@@ -24,11 +25,11 @@ BUFFER_LOG_LIMIT = 1000
|
||||
BUFFER_LOG_TTL_SECONDS = 7 * 24 * 60 * 60
|
||||
LOG_BUFFER_KEY_PREFIX = "planet:system_logs"
|
||||
|
||||
LOG_LEVEL_ERROR = "error"
|
||||
LOG_LEVEL_WARNING = "warning"
|
||||
LOG_LEVEL_INFO = "info"
|
||||
LOG_LEVEL_DEBUG = "debug"
|
||||
LOG_LEVEL_ALL = "all"
|
||||
LOG_LEVEL_ERROR = LogLevel.ERROR.value
|
||||
LOG_LEVEL_WARNING = LogLevel.WARNING.value
|
||||
LOG_LEVEL_INFO = LogLevel.INFO.value
|
||||
LOG_LEVEL_DEBUG = LogLevel.DEBUG.value
|
||||
LOG_LEVEL_ALL = LogLevel.ALL.value
|
||||
|
||||
SUPPORTED_LOG_LEVELS = {
|
||||
LOG_LEVEL_ALL,
|
||||
|
||||
148
backend/app/services/tv_catalog.py
Normal file
148
backend/app/services/tv_catalog.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""Search and paginate the public live TV catalog at the database boundary."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Select, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.services.tv_streams import (
|
||||
TV_LIVE_SOURCE_COLLECTOR,
|
||||
TV_LIVE_SOURCE_DATA_TYPE,
|
||||
_build_collected_tv_source,
|
||||
build_public_tv_payload,
|
||||
get_tv_settings_payload,
|
||||
)
|
||||
|
||||
|
||||
def _source_key():
|
||||
return func.coalesce(
|
||||
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
|
||||
func.nullif(CollectedData.source_id, ""),
|
||||
CollectedData.entity_key,
|
||||
)
|
||||
|
||||
|
||||
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
source_id = _source_key()
|
||||
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
|
||||
ranked = (
|
||||
select(
|
||||
CollectedData.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=source_id,
|
||||
order_by=CollectedData.id.desc(),
|
||||
)
|
||||
.label("source_rank"),
|
||||
)
|
||||
.where(
|
||||
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
|
||||
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
|
||||
CollectedData.is_current.is_(True),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.deleted_at.is_(None),
|
||||
enabled.notin_(("false", "0", "no", "off")),
|
||||
source_id.notin_(configured_ids),
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
return (
|
||||
select(CollectedData)
|
||||
.join(ranked, ranked.c.id == CollectedData.id)
|
||||
.where(ranked.c.source_rank == 1)
|
||||
)
|
||||
|
||||
|
||||
def _filter_catalog_query(
|
||||
query: Select[tuple[CollectedData]], terms: list[str]
|
||||
) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
searchable = func.lower(
|
||||
func.concat_ws(
|
||||
" ",
|
||||
CollectedData.name,
|
||||
CollectedData.title,
|
||||
CollectedData.source_id,
|
||||
metadata["name"].as_string(),
|
||||
metadata["provider"].as_string(),
|
||||
metadata["region"].as_string(),
|
||||
metadata["country"].as_string(),
|
||||
metadata["language"].as_string(),
|
||||
)
|
||||
)
|
||||
for term in terms:
|
||||
query = query.where(searchable.contains(term, autoescape=True))
|
||||
return query
|
||||
|
||||
|
||||
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
|
||||
searchable = " ".join(
|
||||
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
|
||||
).lower()
|
||||
return all(term in searchable for term in terms)
|
||||
|
||||
|
||||
async def get_tv_catalog_page(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
q: str = "",
|
||||
selected_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
settings = await get_tv_settings_payload(db)
|
||||
payload = build_public_tv_payload(settings, [])
|
||||
configured = payload["sources"]
|
||||
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
|
||||
if selected_id:
|
||||
selected = next((source for source in configured if source["id"] == selected_id), None)
|
||||
if selected is None:
|
||||
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
|
||||
selected = _build_collected_tv_source(record, 0) if record else None
|
||||
if selected:
|
||||
payload["selected_source"] = selected
|
||||
summary = (
|
||||
await db.execute(
|
||||
select(func.count(), func.max(CollectedData.collected_at))
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
|
||||
)
|
||||
).one()
|
||||
total_collected, latest_update = summary
|
||||
terms = q.lower().split()
|
||||
matched_configured = [source for source in configured if _matches_source(source, terms)]
|
||||
filtered_query = _filter_catalog_query(query, terms)
|
||||
matched_collected = (
|
||||
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
|
||||
if terms
|
||||
else total_collected
|
||||
)
|
||||
sources = matched_configured[offset : offset + limit]
|
||||
remaining = limit - len(sources)
|
||||
if remaining:
|
||||
rows = await db.scalars(
|
||||
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
|
||||
.offset(max(0, offset - len(matched_configured)))
|
||||
.limit(remaining)
|
||||
)
|
||||
sources.extend(
|
||||
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
|
||||
)
|
||||
total = len(matched_configured) + matched_collected
|
||||
next_offset = offset + len(sources)
|
||||
return {
|
||||
**payload,
|
||||
"sources": sources,
|
||||
"source_count": len(configured) + total_collected,
|
||||
"latest_updated_at": (
|
||||
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
|
||||
),
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
@@ -11,7 +11,7 @@ from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
@@ -300,8 +300,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
default_source = next(
|
||||
source for source in DEFAULT_TV_SETTINGS["sources"]
|
||||
if source["id"] == DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
normalize_tv_source(default_source, index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
|
||||
@@ -9,7 +9,12 @@ from sqlalchemy import select
|
||||
from sqlalchemy import Float
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.vessel import AISConflictRecord, AISRawObservation, AISSourceHealth
|
||||
from app.models.vessel import (
|
||||
AISConflictRecord,
|
||||
AISRawObservation,
|
||||
AISSourceHealth,
|
||||
VesselCurrentState,
|
||||
)
|
||||
from app.services.vessel_aggregation_strategy import (
|
||||
DEFAULT_STRATEGY,
|
||||
load_strategy,
|
||||
@@ -44,6 +49,17 @@ CONFLICT_FIELDS = (
|
||||
"width",
|
||||
"draught",
|
||||
)
|
||||
CURRENT_STATE_STATIC_FIELDS = (
|
||||
"name",
|
||||
"callsign",
|
||||
"vessel_type",
|
||||
"vessel_type_name",
|
||||
"flag",
|
||||
"length",
|
||||
"width",
|
||||
"draught",
|
||||
"imo",
|
||||
)
|
||||
|
||||
|
||||
def _json_default(value: Any) -> Any:
|
||||
@@ -489,9 +505,127 @@ async def record_vessel_ais_observation(
|
||||
quality_flags=quality_flags or [],
|
||||
)
|
||||
db.add(observation)
|
||||
await upsert_vessel_current_state(
|
||||
db,
|
||||
source=source,
|
||||
normalized_payload=normalized_json,
|
||||
observed_at=observed_at,
|
||||
quality_flags=quality_flags or [],
|
||||
)
|
||||
return observation
|
||||
|
||||
|
||||
async def upsert_vessel_current_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
source: str,
|
||||
normalized_payload: dict[str, Any],
|
||||
observed_at: datetime,
|
||||
quality_flags: list[str] | None = None,
|
||||
) -> VesselCurrentState | None:
|
||||
"""Keep one latest renderable row per MMSI while preserving useful static fields."""
|
||||
|
||||
if not _has_valid_position(normalized_payload):
|
||||
return None
|
||||
mmsi = int(normalized_payload["mmsi"])
|
||||
current = await db.get(VesselCurrentState, mmsi)
|
||||
if current is not None and current.observed_at is not None:
|
||||
current_observed_at = _coerce_datetime(current.observed_at)
|
||||
if current_observed_at is not None and observed_at < current_observed_at:
|
||||
return current
|
||||
|
||||
if current is None:
|
||||
current = VesselCurrentState(mmsi=mmsi)
|
||||
db.add(current)
|
||||
|
||||
current.lat = float(normalized_payload["lat"])
|
||||
current.lon = float(normalized_payload["lon"])
|
||||
current.source = source
|
||||
current.observed_at = observed_at
|
||||
current.updated_at = datetime.now(UTC)
|
||||
updated_fields: set[str] = {"lat", "lon"}
|
||||
for field in DYNAMIC_FIELDS:
|
||||
if field in {"lat", "lon"}:
|
||||
continue
|
||||
value = _payload_value(normalized_payload, field)
|
||||
if value is not None:
|
||||
setattr(current, field, value)
|
||||
updated_fields.add(field)
|
||||
field_sources = dict(current.field_sources or {})
|
||||
for field in CURRENT_STATE_STATIC_FIELDS:
|
||||
value = _payload_value(normalized_payload, field)
|
||||
if value is None:
|
||||
continue
|
||||
existing_source = field_sources.get(field)
|
||||
existing_value = getattr(current, field, None)
|
||||
if (
|
||||
existing_value in (None, "")
|
||||
or _strategy_source_rank(source, DEFAULT_STRATEGY)
|
||||
>= _strategy_source_rank(str(existing_source or ""), DEFAULT_STRATEGY)
|
||||
):
|
||||
setattr(current, field, value)
|
||||
updated_fields.add(field)
|
||||
|
||||
current.vessel_type_name = current.vessel_type_name or normalize_vessel_type_name(
|
||||
current.vessel_type
|
||||
)
|
||||
selected_reasons = dict(current.selected_reasons or {})
|
||||
for field in updated_fields:
|
||||
field_sources[field] = source
|
||||
selected_reasons[field] = (
|
||||
"newest_observation" if field in DYNAMIC_FIELDS else "source_priority"
|
||||
)
|
||||
current.field_sources = field_sources
|
||||
current.selected_reasons = selected_reasons
|
||||
current.source_summary = {
|
||||
**dict(current.source_summary or {}),
|
||||
source: {
|
||||
"latest_observed_at": observed_at.isoformat(),
|
||||
},
|
||||
}
|
||||
current.quality_flags = sorted(set((current.quality_flags or []) + (quality_flags or [])))
|
||||
return current
|
||||
|
||||
|
||||
async def get_current_vessels_snapshot(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
bbox: tuple[float, float, float, float],
|
||||
limit: int = 1000,
|
||||
observed_since: datetime,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read the bounded latest-state table used by Earth rendering."""
|
||||
|
||||
safe_limit = min(max(int(limit or 1000), 1), MAX_SNAPSHOT_LIMIT)
|
||||
lon_min, lat_min, lon_max, lat_max = bbox
|
||||
stmt = (
|
||||
select(VesselCurrentState)
|
||||
.where(VesselCurrentState.observed_at >= observed_since)
|
||||
.where(VesselCurrentState.lon >= lon_min)
|
||||
.where(VesselCurrentState.lon <= lon_max)
|
||||
.where(VesselCurrentState.lat >= lat_min)
|
||||
.where(VesselCurrentState.lat <= lat_max)
|
||||
.order_by(VesselCurrentState.observed_at.desc(), VesselCurrentState.mmsi.asc())
|
||||
.limit(safe_limit)
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
if not hasattr(result, "scalars"):
|
||||
return []
|
||||
return [item.to_dict() for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_current_vessels_by_mmsi(
|
||||
db: AsyncSession, mmsis: list[int]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read canonical render state for the vessels changed by a stream flush."""
|
||||
if not mmsis:
|
||||
return []
|
||||
result = await db.execute(
|
||||
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
|
||||
)
|
||||
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def aggregate_vessel_observations(
|
||||
db: AsyncSession,
|
||||
observations: Iterable[AISRawObservation],
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
[pytest]
|
||||
pythonpath = ..
|
||||
asyncio_mode = auto
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
|
||||
@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
assert vessel_update["refresh_strategy"] == "reload"
|
||||
|
||||
|
||||
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
|
||||
from app.services.earth_db_change_listener import PendingEarthDbChange
|
||||
|
||||
for source in ("aisstream_vessels", "barentswatch_vessels"):
|
||||
update = build_earth_update_from_db_payload({
|
||||
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
|
||||
})
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
|
||||
pending.add({"operation": "UPDATE"})
|
||||
pending.add({"operation": "DELETE"})
|
||||
assert pending.refresh_strategy == "reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from datetime import UTC, datetime
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
@@ -12,9 +12,11 @@ from app.services.earth_news import (
|
||||
default_earth_news_sources_payload,
|
||||
normalize_earth_news_sources_payload,
|
||||
_fetch_source,
|
||||
_diversify_news_items_for_locale,
|
||||
_enrich_items_with_target_locations,
|
||||
_extract_target_location_from_text,
|
||||
_parse_feed_entries,
|
||||
_rank_and_trim_items,
|
||||
_serialize_item,
|
||||
get_earth_news_payload,
|
||||
test_news_source_config as run_news_source_config_test,
|
||||
@@ -22,6 +24,7 @@ from app.services.earth_news import (
|
||||
from app.services.earth_news_queue import NewsTargetLocationMessage
|
||||
from app.services.earth_news_worker import process_target_location_message
|
||||
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
|
||||
from app.services.earth_news_store import _diversify_parsed_news_items_by_region
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
@@ -50,6 +53,188 @@ def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
assert payload["published_at"] == "2026-04-23T02:30:00Z"
|
||||
|
||||
|
||||
def test_serialize_item_includes_breaking_fields():
|
||||
item = ParsedNewsItem(
|
||||
id="breaking:test",
|
||||
title="Major market halt",
|
||||
summary="Trading halt after flash crash",
|
||||
url="https://example.com/breaking",
|
||||
source="Example Source",
|
||||
feed_name="Example Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime(2026, 5, 15, 2, 0, tzinfo=UTC),
|
||||
breaking_level="critical",
|
||||
breaking_scope="global",
|
||||
breaking_reasons=["重大金融市场异常"],
|
||||
breaking_source="rules",
|
||||
breaking_confidence=0.72,
|
||||
breaking_expires_at=datetime(2026, 5, 16, 2, 0, tzinfo=UTC),
|
||||
)
|
||||
|
||||
payload = _serialize_item(item, active_region="europe")
|
||||
|
||||
assert payload["breaking_level"] == "critical"
|
||||
assert payload["breaking_scope"] == "global"
|
||||
assert payload["breaking_reasons"] == ["重大金融市场异常"]
|
||||
assert payload["breaking_source"] == "rules"
|
||||
assert payload["breaking_confidence"] == 0.72
|
||||
assert payload["breaking_expires_at"] == "2026-05-16T02:00:00Z"
|
||||
|
||||
|
||||
def test_rank_and_trim_items_prioritizes_active_breaking():
|
||||
older_breaking = ParsedNewsItem(
|
||||
id="global:critical",
|
||||
title="Nuclear accident reported",
|
||||
summary="A nuclear accident has been reported.",
|
||||
url="https://example.com/critical",
|
||||
source="Global Source",
|
||||
feed_name="Global Feed",
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime.now(UTC) - timedelta(hours=2),
|
||||
breaking_level="critical",
|
||||
breaking_scope="global",
|
||||
breaking_expires_at=datetime.now(UTC) + timedelta(hours=6),
|
||||
)
|
||||
newer_regular = ParsedNewsItem(
|
||||
id="europe:regular",
|
||||
title="Regular Europe story",
|
||||
summary="A newer regular story.",
|
||||
url="https://example.com/regular",
|
||||
source="Europe Source",
|
||||
feed_name="Europe Feed",
|
||||
feed_region="europe",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime.now(UTC),
|
||||
)
|
||||
expired_breaking = ParsedNewsItem(
|
||||
id="europe:expired",
|
||||
title="Expired breaking",
|
||||
summary="Expired breaking story.",
|
||||
url="https://example.com/expired",
|
||||
source="Europe Source",
|
||||
feed_name="Europe Feed",
|
||||
feed_region="europe",
|
||||
homepage_url="https://example.com",
|
||||
published_at=datetime.now(UTC) + timedelta(minutes=1),
|
||||
breaking_level="critical",
|
||||
breaking_scope="regional",
|
||||
breaking_expires_at=datetime.now(UTC) - timedelta(minutes=1),
|
||||
)
|
||||
|
||||
ranked = _rank_and_trim_items(
|
||||
[newer_regular, expired_breaking, older_breaking],
|
||||
active_region="europe",
|
||||
limit=3,
|
||||
)
|
||||
|
||||
assert [item.id for item in ranked] == ["global:critical", "europe:expired", "europe:regular"]
|
||||
|
||||
|
||||
def test_diversify_news_items_prefers_display_ready_content_across_sources():
|
||||
published_at = datetime(2026, 6, 11, 3, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(source_id: str, suffix: str, *, zh_ready: bool) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{source_id}:{suffix}",
|
||||
title=f"{source_id} title {suffix}",
|
||||
summary=f"{source_id} summary {suffix}",
|
||||
url=f"https://example.com/{source_id}/{suffix}",
|
||||
source=source_id,
|
||||
feed_name=source_id,
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at,
|
||||
content_language="en",
|
||||
localizations={
|
||||
"zh-CN": {
|
||||
"title": f"{source_id} 中文标题 {suffix}",
|
||||
"summary": f"{source_id} 中文摘要 {suffix}",
|
||||
}
|
||||
} if zh_ready else {},
|
||||
)
|
||||
|
||||
items = [
|
||||
make_item("source-a", "1", zh_ready=False),
|
||||
make_item("source-a", "2", zh_ready=False),
|
||||
make_item("source-a", "3", zh_ready=False),
|
||||
make_item("source-b", "1", zh_ready=True),
|
||||
make_item("source-c", "1", zh_ready=True),
|
||||
]
|
||||
|
||||
result = _diversify_news_items_for_locale(
|
||||
items,
|
||||
active_region="global",
|
||||
limit=3,
|
||||
locale="zh-CN",
|
||||
)
|
||||
|
||||
assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"]
|
||||
|
||||
|
||||
def test_cruise_news_diversity_keeps_regions_from_being_starved():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(40)],
|
||||
make_item("europe", 1),
|
||||
make_item("middle-east-africa", 1),
|
||||
make_item("americas", 1),
|
||||
make_item("global", 1),
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=8)
|
||||
regions = [item.feed_region for item in result]
|
||||
|
||||
assert "europe" in regions
|
||||
assert "middle-east-africa" in regions
|
||||
assert "americas" in regions
|
||||
assert regions.count("asia-pacific") < len(regions)
|
||||
|
||||
|
||||
def test_global_news_diversity_uses_same_region_balance():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:global:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/global/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(24)],
|
||||
*[make_item("europe", index) for index in range(2)],
|
||||
*[make_item("middle-east-africa", index) for index in range(2)],
|
||||
*[make_item("americas", index) for index in range(2)],
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=6)
|
||||
regions = {item.feed_region for item in result}
|
||||
|
||||
assert {"europe", "middle-east-africa", "americas"}.issubset(regions)
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
@@ -199,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
|
||||
assert items[0].content_language == "zh-CN"
|
||||
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == ""
|
||||
|
||||
items[0].localizations["en-US"] = {
|
||||
"title": "Chinese e-commerce platform reports quarterly growth",
|
||||
"summary": "The platform said cross-border orders rose year over year.",
|
||||
}
|
||||
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
|
||||
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
|
||||
|
||||
|
||||
def test_default_news_sources_include_business_and_ecommerce_sources():
|
||||
@@ -881,9 +1073,11 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
assert limit == 12
|
||||
return [item]
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
if source_ids is None:
|
||||
assert limit == 12
|
||||
return [item]
|
||||
return []
|
||||
|
||||
async def fail_fetch(_sources):
|
||||
raise AssertionError("fresh database items should not fetch RSS")
|
||||
@@ -982,8 +1176,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
captured["items_region"] = active_region
|
||||
captured["items_categories"] = categories
|
||||
captured["items_source_ids"] = source_ids
|
||||
return [item]
|
||||
captured.setdefault("items_source_ids", []).append(source_ids)
|
||||
return [item] if source_ids is None else []
|
||||
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None):
|
||||
captured["cruise_categories"] = categories
|
||||
@@ -1010,7 +1204,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
assert captured["freshness_region"] == "europe"
|
||||
assert captured["items_region"] == "europe"
|
||||
assert captured["items_categories"] == {"business", "ecommerce"}
|
||||
assert captured["items_source_ids"] is None
|
||||
assert captured["items_source_ids"][0] is None
|
||||
assert any(source_ids for source_ids in captured["items_source_ids"][1:])
|
||||
assert captured["cruise_categories"] == {"business", "ecommerce"}
|
||||
assert captured["cruise_source_ids"] is None
|
||||
assert payload["filters"] == {
|
||||
@@ -1019,6 +1214,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
"sources": [],
|
||||
"limit": 12,
|
||||
"locale": "zh-CN",
|
||||
"has_breaking": False,
|
||||
"highest_breaking_level": "none",
|
||||
}
|
||||
assert payload["items"][0]["category"] == "business"
|
||||
|
||||
|
||||
252
backend/tests/test_earth_news_manual.py
Normal file
252
backend/tests/test_earth_news_manual.py
Normal file
@@ -0,0 +1,252 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.enums import NewsSourceType
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.earth_news import REGION_ANCHORS
|
||||
from app.services.earth_news_manual import (
|
||||
DEFAULT_MANUAL_NEWS_GROUP_ID,
|
||||
create_manual_news_group,
|
||||
import_manual_news_items,
|
||||
list_news_groups,
|
||||
list_news_records,
|
||||
parse_manual_news_import_upload,
|
||||
rename_manual_news_group,
|
||||
upsert_manual_news_item,
|
||||
)
|
||||
|
||||
|
||||
class _FakeResult:
|
||||
def __init__(self, rows=None, scalar=None):
|
||||
self.rows = rows or []
|
||||
self._scalar = scalar
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._scalar
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar
|
||||
|
||||
def scalars(self):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return self.rows
|
||||
|
||||
|
||||
class _FakeNewsSession:
|
||||
def __init__(self, records=None, setting=None):
|
||||
self.records = dict(records or {})
|
||||
self.setting = setting
|
||||
|
||||
async def get(self, _model, item_id):
|
||||
return self.records.get(item_id)
|
||||
|
||||
def add(self, item):
|
||||
if isinstance(item, SystemSetting):
|
||||
self.setting = item
|
||||
else:
|
||||
self.records[item.id] = item
|
||||
|
||||
async def execute(self, stmt):
|
||||
statement = str(stmt)
|
||||
if "system_settings" in statement:
|
||||
return _FakeResult(scalar=self.setting)
|
||||
if "count" in statement.lower():
|
||||
return _FakeResult(scalar=len(self.records))
|
||||
return _FakeResult(rows=list(self.records.values()))
|
||||
|
||||
async def flush(self):
|
||||
return None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_news_queue(monkeypatch):
|
||||
queued = []
|
||||
|
||||
async def _enqueue(payload, force=False):
|
||||
queued.append({"payload": payload, "force": force})
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_manual.enqueue_target_location_job", _enqueue)
|
||||
return queued
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_upsert_uses_region_anchor_and_manual_metadata(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
result = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "手动添加的新闻",
|
||||
"summary": "一条用于测试的手动新闻。",
|
||||
"source": "人工录入",
|
||||
"region": "europe",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"tags": ["manual", "test"],
|
||||
},
|
||||
)
|
||||
|
||||
anchor = REGION_ANCHORS["europe"]
|
||||
assert result.created is True
|
||||
assert result.queued is True
|
||||
assert result.item.id.startswith("manual:")
|
||||
assert result.item.feed_name == "手动添加"
|
||||
assert result.item.source == "人工录入"
|
||||
assert result.item.latitude == anchor.latitude
|
||||
assert result.item.longitude == anchor.longitude
|
||||
assert result.item.location_source == "region_anchor"
|
||||
assert result.item.verified is False
|
||||
assert result.item.location_meta["news_meta"]["feed_type"] == NewsSourceType.MANUAL.value
|
||||
assert result.item.location_meta["news_meta"]["source_type"] == NewsSourceType.MANUAL.value
|
||||
assert result.item.location_meta["news_meta"]["manual_group_id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
assert len(fake_news_queue) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_duplicate_import_upserts_without_duplicate_rows(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
payload = {
|
||||
"title": "Same manual story",
|
||||
"source": "Manual Desk",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"region": "global",
|
||||
}
|
||||
|
||||
first = await upsert_manual_news_item(db, payload)
|
||||
second = await upsert_manual_news_item(db, {**payload, "summary": "Updated summary"})
|
||||
|
||||
assert first.created is True
|
||||
assert second.created is False
|
||||
assert len(db.records) == 1
|
||||
assert db.records[first.item.id].summary == "Updated summary"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_edit_without_location_preserves_manual_coordinates(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
created = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "Taipei-1 data center update",
|
||||
"summary": "Initial summary.",
|
||||
"region": "asia-pacific",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"location": {"label": "Kaohsiung, Taiwan", "latitude": 22.6273, "longitude": 120.3014},
|
||||
},
|
||||
)
|
||||
|
||||
updated = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "Taipei-1 data center update",
|
||||
"summary": "Edited summary only.",
|
||||
"region": "asia-pacific",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
},
|
||||
item_id_override=created.item.id,
|
||||
)
|
||||
|
||||
assert updated.created is False
|
||||
assert updated.item.latitude == pytest.approx(22.6273)
|
||||
assert updated.item.longitude == pytest.approx(120.3014)
|
||||
assert updated.item.location_source == "manual_location"
|
||||
assert updated.item.verified is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_api_service_rejects_rss_records(fake_news_queue):
|
||||
rss_record = EarthNewsItem(
|
||||
id="bbc-world:example",
|
||||
title="RSS story",
|
||||
summary="RSS summary",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
region="global",
|
||||
latitude=20,
|
||||
longitude=0,
|
||||
location_label="全球",
|
||||
location_source="region_anchor",
|
||||
verified=False,
|
||||
location_meta={"news_meta": {"feed_type": "rss"}},
|
||||
first_seen_at=datetime.now(UTC),
|
||||
last_seen_at=datetime.now(UTC),
|
||||
)
|
||||
db = _FakeNewsSession({rss_record.id: rss_record})
|
||||
|
||||
with pytest.raises(PermissionError):
|
||||
await upsert_manual_news_item(
|
||||
db,
|
||||
{"title": "Edited title", "region": "global"},
|
||||
item_id_override=rss_record.id,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_import_reports_per_item_errors(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
result = await import_manual_news_items(
|
||||
db,
|
||||
[
|
||||
{"title": "Valid manual news", "region": "global"},
|
||||
{"summary": "missing title"},
|
||||
],
|
||||
)
|
||||
|
||||
assert result["created"] == 1
|
||||
assert result["failed"] == 1
|
||||
assert result["errors"][0]["index"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_groups_default_create_and_rename(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
initial = await list_news_groups(db)
|
||||
assert initial["manual_groups"][0]["id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
assert initial["manual_groups"][0]["name"] == "新建新闻组"
|
||||
|
||||
group = await create_manual_news_group(db, "专题组")
|
||||
assert group["name"] == "专题组"
|
||||
assert db.setting is not None
|
||||
|
||||
await upsert_manual_news_item(db, {"title": "Grouped story", "region": "global"}, group_id=group["id"])
|
||||
renamed = await rename_manual_news_group(db, group["id"], "重命名专题")
|
||||
|
||||
record = next(iter(db.records.values()))
|
||||
assert renamed["name"] == "重命名专题"
|
||||
assert record.location_meta["news_meta"]["manual_group_id"] == group["id"]
|
||||
assert record.location_meta["news_meta"]["manual_group_name"] == "重命名专题"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_list_filters_by_group_id(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
group = await create_manual_news_group(db, "导入组")
|
||||
|
||||
await import_manual_news_items(
|
||||
db,
|
||||
[
|
||||
{"title": "In group", "region": "global"},
|
||||
{"title": "Also in group", "region": "global"},
|
||||
],
|
||||
group_id=group["id"],
|
||||
)
|
||||
await upsert_manual_news_item(db, {"title": "Default group", "region": "global"})
|
||||
|
||||
grouped = await list_news_records(db, page=1, page_size=20, group_id=group["id"])
|
||||
default_group = await list_news_records(db, page=1, page_size=20, group_id=DEFAULT_MANUAL_NEWS_GROUP_ID)
|
||||
|
||||
assert grouped["total"] == 2
|
||||
assert {item["manual_group_id"] for item in grouped["items"]} == {group["id"]}
|
||||
assert default_group["total"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_import_parser_requires_json_array():
|
||||
with pytest.raises(ValueError, match="顶层必须是数组"):
|
||||
await parse_manual_news_import_upload(b'{"title":"not an array"}')
|
||||
74
backend/tests/test_enum_contracts.py
Normal file
74
backend/tests/test_enum_contracts.py
Normal file
@@ -0,0 +1,74 @@
|
||||
"""Compatibility contracts for stable backend protocol enums."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
|
||||
from app.core.enums import (
|
||||
BreakingLevel,
|
||||
BreakingScope,
|
||||
JobStatus,
|
||||
NewsImportanceLevel,
|
||||
NewsSourceType,
|
||||
PlaygroundMessageKind,
|
||||
PlaygroundMessageStatus,
|
||||
UserRole,
|
||||
parse_enum,
|
||||
)
|
||||
from app.services.earth_news_classification import (
|
||||
BREAKING_LEVEL_RANK,
|
||||
BREAKING_TTL,
|
||||
breaking_sort_rank,
|
||||
importance_level,
|
||||
)
|
||||
|
||||
|
||||
def test_protocol_enum_values_remain_api_compatible() -> None:
|
||||
assert [item.value for item in NewsImportanceLevel] == ["low", "medium", "high", "critical"]
|
||||
assert [item.value for item in BreakingLevel] == ["none", "watch", "breaking", "critical"]
|
||||
assert [item.value for item in BreakingScope] == ["regional", "global"]
|
||||
assert [item.value for item in NewsSourceType] == ["rss", "atom", "aggregated", "reference", "manual"]
|
||||
assert [item.value for item in UserRole] == ["viewer", "admin", "super_admin"]
|
||||
assert JobStatus.RUNNING.value == "running"
|
||||
assert PlaygroundMessageKind.THINKING.value == "thinking"
|
||||
assert PlaygroundMessageStatus.ERROR.value == "error"
|
||||
assert PlaygroundMessageStatus.STOPPED.value == "stopped"
|
||||
|
||||
|
||||
def test_parse_enum_accepts_legacy_strings_and_safely_falls_back(caplog) -> None:
|
||||
assert parse_enum(JobStatus, "RUNNING", JobStatus.FAILED) is JobStatus.RUNNING
|
||||
assert parse_enum(JobStatus, None, JobStatus.QUEUED) is JobStatus.QUEUED
|
||||
assert parse_enum(JobStatus, "legacy-unknown", JobStatus.FAILED) is JobStatus.FAILED
|
||||
assert "legacy-unknown" in caplog.text
|
||||
|
||||
|
||||
def test_importance_level_boundaries() -> None:
|
||||
expected = {
|
||||
34: NewsImportanceLevel.LOW,
|
||||
35: NewsImportanceLevel.MEDIUM,
|
||||
59: NewsImportanceLevel.MEDIUM,
|
||||
60: NewsImportanceLevel.HIGH,
|
||||
79: NewsImportanceLevel.HIGH,
|
||||
80: NewsImportanceLevel.CRITICAL,
|
||||
}
|
||||
assert {score: importance_level(score) for score in expected} == expected
|
||||
|
||||
|
||||
def test_breaking_rank_and_ttl_contracts() -> None:
|
||||
assert BREAKING_LEVEL_RANK[BreakingLevel.CRITICAL] > BREAKING_LEVEL_RANK[BreakingLevel.BREAKING]
|
||||
assert BREAKING_TTL[BreakingLevel.WATCH] == timedelta(hours=6)
|
||||
assert BREAKING_TTL[BreakingLevel.BREAKING] == timedelta(hours=12)
|
||||
assert BREAKING_TTL[BreakingLevel.CRITICAL] == timedelta(hours=24)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
active = SimpleNamespace(
|
||||
breaking_level=BreakingLevel.BREAKING.value,
|
||||
breaking_expires_at=now + timedelta(minutes=1),
|
||||
)
|
||||
expired = SimpleNamespace(
|
||||
breaking_level=BreakingLevel.CRITICAL.value,
|
||||
breaking_expires_at=now - timedelta(minutes=1),
|
||||
)
|
||||
assert breaking_sort_rank(active) == BREAKING_LEVEL_RANK[BreakingLevel.BREAKING]
|
||||
assert breaking_sort_rank(expired) == 0
|
||||
219
backend/tests/test_llm_provider_catalog.py
Normal file
219
backend/tests/test_llm_provider_catalog.py
Normal file
@@ -0,0 +1,219 @@
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services import llm_provider_catalog as catalog
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
def mock_http(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"provider,path,auth",
|
||||
[
|
||||
("minimax", "/anthropic/v1/models", "x-api-key"),
|
||||
("anthropic", "/v1/models", "x-api-key"),
|
||||
("openai", "/v1/models", "authorization"),
|
||||
("deepseek", "/v1/models", "authorization"),
|
||||
("alibaba", "/api/v1/models", "authorization"),
|
||||
("moonshotai", "/v1/models", "authorization"),
|
||||
("openrouter", "/api/v1/models", "authorization"),
|
||||
("opencode-go", "/zen/go/v1/models", "authorization"),
|
||||
("ollama", "/api/tags", "authorization"),
|
||||
],
|
||||
)
|
||||
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
|
||||
preset = catalog.get_fallback_llm_provider_preset(provider)
|
||||
|
||||
def handle(request):
|
||||
assert request.url.path == path
|
||||
assert request.url.host == httpx.URL(preset["base_url"]).host
|
||||
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
|
||||
if provider == "alibaba":
|
||||
assert request.url.params["capabilities"] == "TG"
|
||||
return httpx.Response(
|
||||
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
|
||||
)
|
||||
if provider == "ollama":
|
||||
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
|
||||
return httpx.Response(200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
|
||||
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
|
||||
assert result["base_url"] == preset["base_url"]
|
||||
assert "test-key" not in str(result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,base,api,path",
|
||||
[
|
||||
(
|
||||
"minimax",
|
||||
"https://api.minimax.io/anthropic/v1/",
|
||||
"anthropic-messages",
|
||||
"/anthropic/v1/models",
|
||||
),
|
||||
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
|
||||
(
|
||||
"alibaba",
|
||||
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
(
|
||||
"alibaba",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
|
||||
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
|
||||
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
|
||||
],
|
||||
)
|
||||
def test_urls_preserve_region_and_gateway(provider, base, api, path):
|
||||
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
|
||||
assert url.host == httpx.URL(base).host
|
||||
assert url.path == path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_pagination_and_release_order(monkeypatch):
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request.url.params.get("after_id"))
|
||||
if len(seen) == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
|
||||
"has_more": True,
|
||||
"last_id": "old",
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
|
||||
assert seen == [None, "old"]
|
||||
assert result["models"] == ["new", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashscope_pagination(monkeypatch):
|
||||
def handle(request):
|
||||
page = int(request.url.params["page_no"])
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"output": {
|
||||
"total": 2,
|
||||
"models": [
|
||||
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
|
||||
assert result["models"] == ["page-2", "page-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{},
|
||||
{"data": []},
|
||||
{"data": [None]},
|
||||
{"data": [{}]},
|
||||
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
|
||||
],
|
||||
)
|
||||
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
with pytest.raises(discovery.LLMProviderCatalogError):
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
|
||||
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
|
||||
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
|
||||
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
|
||||
assert len(result["models"]) == 140
|
||||
assert result["models"][0] == "model-139"
|
||||
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_ollama_catalog_is_valid(monkeypatch):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
|
||||
result = await catalog.refresh_llm_provider_preset("ollama")
|
||||
assert result["models"] == []
|
||||
assert result["model"] == ""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opencode_documented_protocols(monkeypatch):
|
||||
models = [
|
||||
"minimax-m3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"grok-4.6",
|
||||
"muse-spark-1.3-contributor",
|
||||
"kimi-k3",
|
||||
]
|
||||
mock_http(
|
||||
monkeypatch,
|
||||
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
|
||||
)
|
||||
result = await catalog.refresh_llm_provider_preset("opencode-go")
|
||||
assert list(result["model_provider_apis"].values()) == [
|
||||
"anthropic-messages",
|
||||
"anthropic-messages",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-completions",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_transient_error_only(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def handle(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 2
|
||||
calls.clear()
|
||||
|
||||
def unauthorized(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(401, text="secret-upstream-text")
|
||||
|
||||
mock_http(monkeypatch, unauthorized)
|
||||
with pytest.raises(httpx.HTTPStatusError) as error:
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 1
|
||||
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)
|
||||
165
backend/tests/test_llm_provider_settings.py
Normal file
165
backend/tests/test_llm_provider_settings.py
Normal file
@@ -0,0 +1,165 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.v1 import settings as api
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings_db(monkeypatch, tmp_path):
|
||||
# Real SQL reads, commits and reloads; do not mock the settings helpers.
|
||||
engine = create_engine("sqlite://")
|
||||
SystemSetting.__table__.create(engine)
|
||||
session = Session(engine)
|
||||
db = SimpleNamespace(
|
||||
execute=AsyncMock(side_effect=session.execute),
|
||||
add=session.add,
|
||||
commit=AsyncMock(side_effect=session.commit),
|
||||
refresh=AsyncMock(side_effect=session.refresh),
|
||||
)
|
||||
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
|
||||
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
|
||||
yield db
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def use_upstream(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_lists_all_builtin_presets(settings_db):
|
||||
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
|
||||
assert len(result["data"]) == 9
|
||||
assert all(row["models"] for row in result["data"])
|
||||
assert "MiniMax-M3" in result["data"][0]["models"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
|
||||
runtime = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"service_token": "private-service-token",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"base_url": "https://saved.test/anthropic",
|
||||
"model": "old",
|
||||
"api_key": "saved-secret",
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
|
||||
before = await api.get_setting_payload(settings_db, "external_integrations")
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request)
|
||||
assert str(request.url) == "https://draft.test/anthropic/v1/models"
|
||||
assert request.headers["x-api-key"] == "draft-secret"
|
||||
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
|
||||
|
||||
use_upstream(monkeypatch, handle)
|
||||
draft = api.AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
base_url="https://draft.test/anthropic",
|
||||
model="unsaved-model",
|
||||
api_key="draft-secret",
|
||||
service_token="unsaved-service-token",
|
||||
)
|
||||
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
|
||||
result = await api.get_ai_provider_presets(None, settings_db)
|
||||
preset = next(row for row in result["data"] if row["provider"] == "minimax")
|
||||
assert len(seen) == 1
|
||||
assert preset["models"] == ["MiniMax-M3", "old"]
|
||||
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
|
||||
assert await api.get_setting_payload(settings_db, "external_integrations") == before
|
||||
assert "secret" not in str(result)
|
||||
assert "private-service-token" not in str(result)
|
||||
assert draft.model == "unsaved-model"
|
||||
stored = await settings_db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
|
||||
)
|
||||
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
|
||||
settings_db, monkeypatch
|
||||
):
|
||||
await api.save_setting_payload(
|
||||
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
|
||||
)
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
|
||||
commits = settings_db.commit.await_count
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
|
||||
assert error.value.status_code == 502
|
||||
assert "private-upstream-secret" not in error.value.detail
|
||||
assert settings_db.commit.await_count == commits
|
||||
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
|
||||
"models"
|
||||
] == ["saved-model"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
|
||||
def fail_request(request):
|
||||
pytest.fail("must not query a private catalog without credentials")
|
||||
|
||||
use_upstream(monkeypatch, fail_request)
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("minimax", None, settings_db)
|
||||
assert error.value.status_code == 400
|
||||
assert "API Key" in error.value.detail
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
|
||||
)
|
||||
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
|
||||
)
|
||||
assert result["success"] is False
|
||||
assert result["connected"] is False
|
||||
assert "private-upstream-secret" not in result["message"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
|
||||
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
|
||||
use_upstream(
|
||||
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
|
||||
)
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "model": "future-model", "api_key": "test-key"}, 5
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["url"].endswith("/anthropic/v1/models")
|
||||
@@ -40,6 +40,9 @@ def test_gesture_event_serializes_stable_protocol_fields():
|
||||
assert payload["seq"] == 7
|
||||
assert payload["source"] == "motion-agent"
|
||||
assert payload["mode"] == "single"
|
||||
assert payload["protocol_version"] == "motion.v2"
|
||||
assert payload["input_mode"] == "single"
|
||||
assert payload["camera_id"] == "unknown"
|
||||
assert payload["payload"] == {}
|
||||
|
||||
|
||||
@@ -88,8 +91,13 @@ def test_motion_server_status_includes_dry_run_camera_and_heartbeat():
|
||||
assert status["camera_count"] == 1
|
||||
assert status["active_camera_ids"] == ["dry-run:null-camera"]
|
||||
assert status["recognizer"] == "dry-run"
|
||||
assert status["protocol_version"] == "motion.v2"
|
||||
assert status["armed"] is False
|
||||
assert status["paused"] is False
|
||||
assert status["devices_open"] is False
|
||||
assert heartbeat == {
|
||||
"timestamp_ms": 123,
|
||||
"protocol_version": "motion.v2",
|
||||
"source": "motion-agent",
|
||||
"type": "heartbeat",
|
||||
}
|
||||
@@ -109,6 +117,7 @@ def test_skeleton_event_serializes_without_raw_image_fields():
|
||||
payload = json.loads(event.to_json())
|
||||
|
||||
assert payload["type"] == "skeleton"
|
||||
assert payload["protocol_version"] == "motion.v2"
|
||||
assert payload["matched_gesture"] == "rotate_left"
|
||||
assert payload["confidence"] == 0.91
|
||||
assert payload["camera_id"] == "usb:0"
|
||||
@@ -120,6 +129,25 @@ def test_skeleton_event_serializes_without_raw_image_fields():
|
||||
assert "frame" not in payload
|
||||
|
||||
|
||||
def test_v2_gesture_set_accepts_frontend_motion_gestures():
|
||||
state = GestureStateMachine(confidence_threshold=0.7, cooldown_ms=0)
|
||||
|
||||
for gesture in [
|
||||
"rotate_up",
|
||||
"rotate_down",
|
||||
"focus_prev",
|
||||
"focus_next",
|
||||
"layer_prev",
|
||||
"layer_next",
|
||||
]:
|
||||
event = state.accept(
|
||||
GestureObservation(gesture, confidence=0.9, intensity=0.8, timestamp_ms=1000)
|
||||
)
|
||||
|
||||
assert event is not None
|
||||
assert event.gesture == gesture
|
||||
|
||||
|
||||
def test_dry_run_recognizer_produces_debug_skeleton():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
@@ -240,3 +268,92 @@ async def test_motion_agent_cli_reports_dependency_error_without_traceback(monke
|
||||
assert exit_code == 2
|
||||
assert "Motion agent failed: missing cv stack" in captured.err
|
||||
assert "Traceback" not in captured.err
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_motion_agent_command_updates_control_state():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
armed = await server.handle_command(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_armed",
|
||||
"request_id": "req-armed",
|
||||
"payload": {"armed": True},
|
||||
}
|
||||
)
|
||||
)
|
||||
paused = await server.handle_command(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_paused",
|
||||
"request_id": "req-paused",
|
||||
"payload": {"paused": True},
|
||||
}
|
||||
)
|
||||
|
||||
assert armed.ok is True
|
||||
assert armed.request_id == "req-armed"
|
||||
assert armed.status["armed"] is True
|
||||
assert paused.ok is True
|
||||
assert paused.status["paused"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_motion_agent_open_devices_command_accepts_dual_mode():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
try:
|
||||
result = await server.handle_command(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "open_devices",
|
||||
"request_id": "req-open",
|
||||
"payload": {"input_mode": "dual_redundant"},
|
||||
}
|
||||
)
|
||||
|
||||
assert result.ok is True
|
||||
assert result.status["input_mode"] == "dual_redundant"
|
||||
assert result.status["active_camera_ids"] == ("dry-run:null-camera",)
|
||||
assert server._recognition_subprocess is not None
|
||||
assert server._recognition_subprocess.returncode is None
|
||||
finally:
|
||||
await server.stop_recognition_subprocess()
|
||||
|
||||
|
||||
def test_motion_agent_dual_fusion_merges_matching_observations():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
server.state.mode = "dual_redundant"
|
||||
|
||||
selected, fusion = server._fuse_observations(
|
||||
[
|
||||
GestureObservation("zoom_in", confidence=0.82, intensity=0.4, camera_id="usb:0"),
|
||||
GestureObservation("zoom_in", confidence=0.86, intensity=0.8, camera_id="usb:1"),
|
||||
]
|
||||
)
|
||||
|
||||
assert selected.gesture == "zoom_in"
|
||||
assert selected.camera_id == "fusion"
|
||||
assert selected.confidence > 0.86
|
||||
assert fusion == {
|
||||
"source_cameras": ["usb:1", "usb:0"],
|
||||
"window_ms": 120,
|
||||
"reason": "matched_observations",
|
||||
}
|
||||
|
||||
|
||||
def test_motion_agent_dual_fusion_suppresses_close_conflict():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True, confidence_threshold=0.7))
|
||||
|
||||
selected, fusion = server._fuse_observations(
|
||||
[
|
||||
GestureObservation("zoom_in", confidence=0.82, intensity=0.5, camera_id="usb:0"),
|
||||
GestureObservation("zoom_out", confidence=0.78, intensity=0.5, camera_id="usb:1"),
|
||||
]
|
||||
)
|
||||
|
||||
assert selected.gesture == "zoom_in"
|
||||
assert selected.confidence == 0
|
||||
assert fusion["reason"] == "conflict_ignored"
|
||||
|
||||
141
backend/tests/test_provider_protocols.py
Normal file
141
backend/tests/test_provider_protocols.py
Normal file
@@ -0,0 +1,141 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
with patch(
|
||||
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
|
||||
return_value={},
|
||||
):
|
||||
from aiprovider.provider_service import ProviderService
|
||||
from aiprovider.schemas import SituationalAnalysisRequest
|
||||
from app.api.v1.settings import _runtime_config_from_ai_payload
|
||||
|
||||
|
||||
def test_custom_openai_gateway_keeps_its_configured_protocol():
|
||||
config = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "openai",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"base_url": "https://gateway.test/v1",
|
||||
"provider_api": "openai-completions",
|
||||
"model": "gpt-6-astra",
|
||||
},
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService({**config, "api_key": "test-key"})
|
||||
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"model,path",
|
||||
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
|
||||
)
|
||||
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
|
||||
llm = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "opencode-go",
|
||||
"providers": {
|
||||
"opencode-go": {
|
||||
"model": model,
|
||||
"api_key": "test-key",
|
||||
"model_provider_apis": {model: "openai-completions"},
|
||||
}
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService(llm)
|
||||
seen = []
|
||||
|
||||
async def post(**request):
|
||||
seen.append(request)
|
||||
assert request["path"] == path
|
||||
assert request["request_body"]["model"] == model
|
||||
if path == "/responses":
|
||||
assert request["request_body"]["store"] is False
|
||||
assert "max_tokens" not in request["request_body"]
|
||||
return {
|
||||
"output": [
|
||||
{"type": "reasoning", "summary": [{"text": "reason"}]},
|
||||
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
|
||||
]
|
||||
}
|
||||
return {"content": [{"type": "text", "text": "OK"}]}
|
||||
|
||||
monkeypatch.setattr(service, "_post", post)
|
||||
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
|
||||
assert len(seen) == 1
|
||||
assert result.content == "OK"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
seen.append(str(request.url))
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": base,
|
||||
"api_key": "test-key",
|
||||
"model": "MiniMax-M3",
|
||||
}
|
||||
)
|
||||
result = await service.analyze(
|
||||
SituationalAnalysisRequest(title="test", objective="reply OK")
|
||||
)
|
||||
assert result.content == "OK"
|
||||
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
import json
|
||||
|
||||
seen.append(request)
|
||||
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
|
||||
assert request.headers["user-agent"] == "Planet/1.0"
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"api_key": "test-key",
|
||||
"model": "minimax-m3",
|
||||
}
|
||||
)
|
||||
await service.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="test",
|
||||
objective="reply OK",
|
||||
thinking={"type": "enabled"},
|
||||
context={"session_id": session_id},
|
||||
)
|
||||
)
|
||||
session_headers = [request.headers["x-opencode-session"] for request in seen]
|
||||
assert session_headers[0] == session_headers[1]
|
||||
assert session_headers[0] != session_headers[2]
|
||||
160
backend/tests/test_tv_catalog.py
Normal file
160
backend/tests/test_tv_catalog.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import normalize_tv_settings
|
||||
|
||||
|
||||
class CatalogSession:
|
||||
"""Execute the real catalog queries against an isolated SQLite database."""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
async def execute(self, query):
|
||||
return self.session.execute(query)
|
||||
|
||||
async def scalar(self, query):
|
||||
return self.session.scalar(query)
|
||||
|
||||
async def scalars(self, query):
|
||||
return self.session.scalars(query)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def catalog_db():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def register_functions(connection, _record):
|
||||
connection.create_function(
|
||||
"concat_ws",
|
||||
-1,
|
||||
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
|
||||
)
|
||||
|
||||
Base.metadata.create_all(
|
||||
engine,
|
||||
tables=[
|
||||
CollectionTask.__table__,
|
||||
DataSnapshot.__table__,
|
||||
CollectedData.__table__,
|
||||
SystemSetting.__table__,
|
||||
],
|
||||
)
|
||||
with Session(engine) as session:
|
||||
for index in range(135):
|
||||
session.add(
|
||||
CollectedData(
|
||||
source="news_live_streams",
|
||||
source_id=f"channel-{index:03}",
|
||||
data_type="news_live_stream",
|
||||
name=f"Channel {index:03}",
|
||||
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
extra_data={
|
||||
"stream_url": "https://example.invalid/live.m3u8",
|
||||
"region": "Canada",
|
||||
},
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
yield CatalogSession(session)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
|
||||
first = await get_tv_catalog_page(catalog_db, limit=50)
|
||||
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
|
||||
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
|
||||
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
|
||||
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
|
||||
assert len(set(ids)) == first["source_count"] == 145
|
||||
assert ids[-1] == "channel-134"
|
||||
assert third["next_offset"] is None and not third["has_more"]
|
||||
beyond = await get_tv_catalog_page(catalog_db, offset=200)
|
||||
assert beyond["sources"] == [] and not beyond["has_more"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
assert payload["total"] == 1 and payload["source_count"] == 145
|
||||
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
|
||||
assert payload["selected_source"]["id"] == "channel-134"
|
||||
assert "channel-134" not in [source["id"] for source in payload["sources"]]
|
||||
assert payload["default_source_id"] == "aljazeera-mubasher"
|
||||
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
|
||||
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
|
||||
for name, values in [
|
||||
("Disabled", {"extra_data": {"is_enabled": False}}),
|
||||
("Historical", {"is_current": False}),
|
||||
("Invalid", {"is_valid": 0}),
|
||||
("Deleted", {"deleted_at": datetime.now(UTC)}),
|
||||
("Replacement", {"source_id": "channel-134"}),
|
||||
]:
|
||||
record = dict(
|
||||
source="news_live_streams",
|
||||
source_id=name,
|
||||
name=name,
|
||||
data_type="news_live_stream",
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
)
|
||||
catalog_db.session.add(CollectedData(**{**record, **values}))
|
||||
catalog_db.session.flush()
|
||||
payload = await get_tv_catalog_page(catalog_db, q="replacement")
|
||||
assert payload["source_count"] == 145
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
|
||||
|
||||
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
|
||||
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
|
||||
)
|
||||
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
|
||||
monkeypatch, config, expected
|
||||
):
|
||||
collector = NewsLiveStreamsCollector()
|
||||
channels = [
|
||||
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
|
||||
]
|
||||
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
|
||||
streams = [
|
||||
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
|
||||
for channel in channels
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
|
||||
)
|
||||
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
|
||||
assert len(records) == expected
|
||||
assert all(record["source_id"] != "sport" for record in records)
|
||||
@@ -8,7 +8,7 @@ from app.api.v1 import visualization
|
||||
from app.api.v1.visualization import convert_vessels_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.vessel import AISRawObservation, VesselPosition, VesselStatic
|
||||
from app.models.vessel import AISRawObservation, VesselCurrentState, VesselPosition, VesselStatic
|
||||
from app.services import barentswatch
|
||||
from app.services.collectors.aisstream import AISStreamCollector
|
||||
from app.services.collectors.vessel_ais import VesselAISCollector
|
||||
@@ -17,6 +17,7 @@ from app.services.vessel_ais_aggregation import (
|
||||
build_field_conflict_candidates,
|
||||
build_observation_hash,
|
||||
record_vessel_ais_observation,
|
||||
upsert_vessel_current_state,
|
||||
)
|
||||
|
||||
|
||||
@@ -109,6 +110,58 @@ async def test_record_vessel_ais_observation_skips_existing_hash():
|
||||
assert db.added == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_upsert_vessel_current_state_keeps_latest_position_and_static_fields():
|
||||
current = VesselCurrentState(
|
||||
mmsi=257123000,
|
||||
lat=59.91,
|
||||
lon=10.73,
|
||||
name="OSLO TRADER",
|
||||
source="barentswatch_vessels",
|
||||
observed_at=datetime(2026, 4, 30, 12, 0, tzinfo=timezone.utc),
|
||||
field_sources={"name": "aisstream_vessels"},
|
||||
)
|
||||
|
||||
class _Session:
|
||||
async def get(self, _model, _mmsi):
|
||||
return current
|
||||
|
||||
def add(self, _item):
|
||||
raise AssertionError("existing current state should be updated")
|
||||
|
||||
db = _Session()
|
||||
result = await upsert_vessel_current_state(
|
||||
db,
|
||||
source="aisstream_vessels",
|
||||
normalized_payload={"mmsi": 257123000, "lat": 59.92, "lon": 10.74, "sog": 12.4},
|
||||
observed_at=datetime(2026, 4, 30, 12, 1, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
assert result is current
|
||||
assert current.lat == pytest.approx(59.92)
|
||||
assert current.lon == pytest.approx(10.74)
|
||||
assert current.name == "OSLO TRADER"
|
||||
assert current.source == "aisstream_vessels"
|
||||
|
||||
await upsert_vessel_current_state(
|
||||
db,
|
||||
source="barentswatch_vessels",
|
||||
normalized_payload={"mmsi": 257123000, "lat": 59.93, "lon": 10.75, "name": "LOW PRIORITY"},
|
||||
observed_at=datetime(2026, 4, 30, 12, 2, tzinfo=timezone.utc),
|
||||
)
|
||||
assert current.lat == pytest.approx(59.93)
|
||||
assert current.name == "OSLO TRADER"
|
||||
|
||||
await upsert_vessel_current_state(
|
||||
db,
|
||||
source="barentswatch_vessels",
|
||||
normalized_payload={"mmsi": 257123000, "lat": 1, "lon": 2, "name": "OLD"},
|
||||
observed_at=datetime(2026, 4, 30, 11, 59, tzinfo=timezone.utc),
|
||||
)
|
||||
assert current.lat == pytest.approx(59.93)
|
||||
assert current.name == "OSLO TRADER"
|
||||
|
||||
|
||||
def test_build_field_conflict_candidates_from_raw_observations():
|
||||
observations = [
|
||||
AISRawObservation(
|
||||
@@ -527,7 +580,7 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch):
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"get_aggregated_vessels_snapshot",
|
||||
"get_current_vessels_snapshot",
|
||||
AsyncMock(
|
||||
return_value=[
|
||||
{
|
||||
@@ -573,6 +626,36 @@ async def test_vessel_snapshot_filters_type_and_bbox(monkeypatch):
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_snapshot_accepts_fractional_zoom(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"get_current_vessels_snapshot",
|
||||
AsyncMock(return_value=[]),
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield object()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get(
|
||||
"/api/v1/vessels/snapshot",
|
||||
params={
|
||||
"bbox": "-180,-85.05112878,180,85.05112878",
|
||||
"zoom": 1.6,
|
||||
"limit": 3000,
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert response.json()["count"] == 0
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legacy_vessels_geojson_route_is_not_registered():
|
||||
transport = ASGITransport(app=app)
|
||||
@@ -597,7 +680,7 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
captured = {}
|
||||
|
||||
async def fake_get_aggregated_vessels_snapshot(db, *, bbox, limit, observed_since):
|
||||
async def fake_get_current_vessels_snapshot(db, *, bbox, limit, observed_since):
|
||||
captured["bbox"] = bbox
|
||||
captured["limit"] = limit
|
||||
captured["observed_since"] = observed_since
|
||||
@@ -624,8 +707,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
|
||||
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"get_aggregated_vessels_snapshot",
|
||||
fake_get_aggregated_vessels_snapshot,
|
||||
"get_current_vessels_snapshot",
|
||||
fake_get_current_vessels_snapshot,
|
||||
)
|
||||
|
||||
class _Result:
|
||||
@@ -661,6 +744,8 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
|
||||
assert captured["bbox"] == (10.0, 59.0, 11.0, 60.0)
|
||||
assert captured["limit"] == 5000
|
||||
assert data["diagnostics"]["bbox_applied"] is True
|
||||
assert data["diagnostics"]["source"] == "vessel_current_state"
|
||||
assert data["diagnostics"]["current_state_count"] == 2
|
||||
assert data["diagnostics"]["legacy_feature_count"] == 0
|
||||
assert data["diagnostics"]["legacy_backfilled_mmsi"] == 0
|
||||
finally:
|
||||
@@ -668,34 +753,12 @@ async def test_vessel_snapshot_filters_bbox_and_caps_limit(monkeypatch):
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(monkeypatch):
|
||||
now = datetime(2026, 4, 28, 1, 0, tzinfo=timezone.utc)
|
||||
async def test_vessel_snapshot_does_not_fallback_to_history_when_current_state_is_empty(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"get_aggregated_vessels_snapshot",
|
||||
"get_current_vessels_snapshot",
|
||||
AsyncMock(return_value=[]),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
visualization,
|
||||
"_load_legacy_vessel_snapshot_features",
|
||||
AsyncMock(
|
||||
return_value=[
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": 257123000,
|
||||
"geometry": {"type": "Point", "coordinates": [10.73, 59.91]},
|
||||
"properties": {
|
||||
"mmsi": 257123000,
|
||||
"name": "OSLO TRADER",
|
||||
"vessel_type": 70,
|
||||
"vessel_type_name": "Cargo",
|
||||
"received_at": now.isoformat(),
|
||||
},
|
||||
}
|
||||
]
|
||||
),
|
||||
)
|
||||
|
||||
result = await visualization.build_vessel_snapshot_response(
|
||||
object(),
|
||||
bbox=(10.0, 59.0, 11.0, 60.0),
|
||||
@@ -705,12 +768,11 @@ async def test_vessel_snapshot_uses_legacy_fallback_when_raw_window_is_empty(mon
|
||||
since_minutes=60,
|
||||
)
|
||||
|
||||
assert result["count"] == 1
|
||||
assert result["features"][0]["properties"]["name"] == "OSLO TRADER"
|
||||
assert result["count"] == 0
|
||||
assert result["diagnostics"]["raw_feature_count"] == 0
|
||||
assert result["diagnostics"]["legacy_feature_count"] == 1
|
||||
assert result["diagnostics"]["legacy_backfilled_mmsi"] == 1
|
||||
assert result["diagnostics"]["legacy_fallback_used"] is True
|
||||
assert result["diagnostics"]["legacy_feature_count"] == 0
|
||||
assert result["diagnostics"]["legacy_backfilled_mmsi"] == 0
|
||||
assert result["diagnostics"]["legacy_fallback_used"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import pytest
|
||||
import importlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.websocket.manager import ConnectionManager
|
||||
from app.core.websocket.broadcaster import DataBroadcaster
|
||||
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
|
||||
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
|
||||
broadcaster = DataBroadcaster()
|
||||
async def load_current(keys):
|
||||
assert keys == ["1"]
|
||||
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
|
||||
broadcaster.enqueue_vessel_update(
|
||||
{
|
||||
"source": "aisstream_vessels",
|
||||
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
assert sent[0]["vessels"] == [
|
||||
{
|
||||
"mmsi": 1,
|
||||
"lat": 59.1,
|
||||
"lon": 10.1,
|
||||
"source": "aisstream_vessels",
|
||||
"lat": 60.0,
|
||||
"lon": 11.0,
|
||||
"source": "barentswatch_vessels",
|
||||
"action": "upsert",
|
||||
"created": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
json.dumps(config)
|
||||
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
|
||||
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
|
||||
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
|
||||
manager.unsubscribe(socket, ["vessels"])
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
assert len(socket.sent) == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_removal_does_not_require_coordinates():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
|
||||
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
|
||||
|
||||
|
||||
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
|
||||
websocket_module = importlib.import_module("app.api.v1.websocket")
|
||||
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
|
||||
app = FastAPI()
|
||||
app.include_router(websocket_module.router)
|
||||
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
|
||||
assert socket.receive_json()["type"] == "connection_established"
|
||||
socket.send_json({
|
||||
"type": "subscribe",
|
||||
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
|
||||
})
|
||||
response = socket.receive_json()
|
||||
assert response["type"] == "subscription_confirmed"
|
||||
assert response["data"]["vessels"]["scope"] == "global"
|
||||
assert response["data"]["vessels"]["type"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
|
||||
broadcaster = DataBroadcaster()
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
|
||||
|
||||
async def fail_read(_keys):
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
|
||||
with pytest.raises(RuntimeError, match="database unavailable"):
|
||||
await broadcaster.flush_vessel_updates()
|
||||
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61
|
||||
|
||||
@@ -8,6 +8,213 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`（bugfix + 小功能混合）
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.74.6] — 2026-09-16
|
||||
|
||||
Released: 2026-09-16
|
||||
|
||||
### Highlights
|
||||
- AI Provider 构建前自动检测主机代理和直连路径，减少终端网络正常但 Docker 镜像拉取超时的问题。
|
||||
- 运维错误原因与脚本诊断共用一份对照表，统一错误编号、原因和处理建议。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 已有 Compose v2 时不再因构建或启动失败回退 v1；Dockerfile 改用 BuildKit 内置解析器，减少一次外部镜像下载。
|
||||
- 新增 --no-build 启动选项，明确复用本地 AI Provider 镜像，缺少镜像时报错且不改写构建指纹。
|
||||
- 自动代理检测尊重 NO_PROXY，仅修改 Planet 管理的本地 Docker 配置；更新前备份和校验，必要时重启并恢复原有容器，失败时回滚。
|
||||
- 中英文运维表覆盖网络、证书、权限、限流和启动故障；未知错误明确保留未归类状态，并要求新确认原因补表与回归用例。
|
||||
- 修复详细日志模式丢失构建失败退出码的问题，补齐代理切换、Compose 选择、错误分类及文档一致性验证，并收录算力与资源态势实施计划。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.5] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 恢复模型供应商预设的加载与持久化，模型刷新和轻量连接检查统一使用供应商官方目录接口。
|
||||
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由，补齐 Responses 适配。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500，使用真实数据库读写回归验证保存与重载。
|
||||
- 统一九家供应商的目录 URL、鉴权、响应解析与分页，保留服务地域和自定义地址；明确处理限流、空目录和失败，移除 404 白名单误判成功。
|
||||
- 模型刷新读取表单草稿地址和凭证，保留默认模型、未保存修改及失败前目录；连接提示区分目录可用与实际生成成功。
|
||||
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识；已知模型选择同步协议，自定义 OpenAI 网关保留手动协议。
|
||||
- 同步中英文接口说明、操作手册与快速开始，并覆盖模型目录失败、草稿保留和协议适配回归。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.4] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 保留 Earth 全量对象与交互，优化海缆、登陆点和卫星绘制，并让 AIS / BarentsWatch 船只通过确认状态增量更新。
|
||||
- 新闻直播支持完整频道目录搜索和无限滚动；模型目录刷新可保存结果，启动流程减少不必要的依赖安装与等待。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 海缆和登陆点合批绘制，卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU，并缩小动态文本的翻译扫描范围。
|
||||
- 船舶全局订阅按 MMSI 合并、拆包和原位更新，保留选择状态，并处理删除、重连及旧快照覆盖。
|
||||
- 新闻直播取消默认 120 项采集截断，增加数据库分页、完整目录搜索和固定计数栏，默认源改为半岛电视台 HLS。
|
||||
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行；模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
|
||||
- AI Provider 镜像使用独立依赖组，容器直接运行已安装环境；启动提前验证数据库、及时识别后端失败，并保留可复用容器。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.3] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限，减少手工安装步骤。
|
||||
- 数据库初始化先核对容器端口与后端真实连接，连接和认证通过后才创建表和默认数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足，修正未安装 Docker 时误提示启动 socket 的诊断。
|
||||
- 自动补齐缺失的 Docker 依赖并启动本地服务，以原用户身份刷新 Docker 组权限；保留参数和 PATH，不依赖 sg。
|
||||
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置，保留端口冲突等具体错误；端口映射异常时最多保留数据卷重建一次 PostgreSQL。
|
||||
- 新增后端数据库只读连接检查，对认证、库名和网络失败给出不含密码或完整连接串的诊断。
|
||||
- 将 Docker 与数据库启动隔离回归测试接入快速检查，并同步 README、harness 和中英文运维说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.2] — 2026-07-01
|
||||
|
||||
Released: 2026-07-01
|
||||
|
||||
### Highlights
|
||||
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`，删除重复维护的旧 Claude command 入口。
|
||||
- 强化视觉证据规则：截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置，而不是跳过后猜测。
|
||||
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback，同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容，保留最新单一入口和 harness 验证说明。
|
||||
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate，补齐路径解析、访问失败报告和 OCR fallback 边界。
|
||||
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口，并更新文档受众计划中的旧路径引用。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.1] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内，移除旧的全局“选择资产/上传”工具栏。
|
||||
- 新增字段级图片拖拽反馈，拖到对应字段时直接提示将图片复制为 Logo 或标题图。
|
||||
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式，并同步中英文使用手册、快速开始和控制台上下文文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
|
||||
- `FieldGrid` 支持按字段注入自定义输入控件，同时复用统一草稿提交路径。
|
||||
- 品牌上传拖拽态改为低饱和 tactile 配色，上传按钮保持蓝色轻立体样式，容器内上下/右侧留白对齐为 3px。
|
||||
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.0] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口，减少英文界面中文残留。
|
||||
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示，避免英文态裁切或错位。
|
||||
- 将容易遗漏的 i18n 入口和视觉回归加入 harness，让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 Earth runtime i18n 入口，统一高清材质、启动状态、错误提示和图层状态文案来源。
|
||||
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
|
||||
- 更新控制台与 Earth 布局规则，保留 brand 尺寸语义，同时让标题、副标题和通知胶囊按内容完整显示。
|
||||
- 扩展 frontend smoke 与 harness 文档，固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
|
||||
- 更新 Earth 新闻本地化服务与测试，确保英文界面新闻内容不再回退中文 UI 文案。
|
||||
|
||||
---
|
||||
|
||||
## [0.73.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 新增前端统一 i18n 基础设施，让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
|
||||
- 控制台侧边栏偏好面板接入语言与主题切换，并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
|
||||
- 扩展 harness 与 smoke 覆盖，确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `frontend/src/i18n/`，用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
|
||||
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
|
||||
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译，并覆盖动态计数字符串。
|
||||
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership，避免首屏溢出和状态词裁切。
|
||||
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则，记录语言迁移边界、状态指示器布局约束和一屏验证要求。
|
||||
|
||||
---
|
||||
|
||||
## [0.72.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 将 agent 入口收敛到单一 `AGENTS.md`，并让 harness 明确阻止小写入口再次分叉。
|
||||
- 新增完整本地 harness 验证层，覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
|
||||
- 扩展 Earth News 与控制台 smoke，确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本，并将未跟踪 harness 设施纳入发布。
|
||||
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出，改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
|
||||
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
|
||||
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档，明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
|
||||
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.1] — 2026-06-26
|
||||
|
||||
Released: 2026-06-26
|
||||
|
||||
### Highlights
|
||||
- 修复 Earth 欧洲、美洲、中东与非洲等区域新闻被亚太来源和旧来源过滤饿死的问题，滚动条、面板和巡航重新回到同一批区域 payload。
|
||||
- 将当前可见新闻和巡航新闻提升到目标位置/翻译优先队列，避免历史普通 Redis backlog 阻塞用户正在看的新闻精修。
|
||||
- 新增 agent harness 入口、代码地图、验证脚本与双语技术说明，让后续维护能按现有 uv/Bun/Gitea 工作流检查而不替代项目规则。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- EarthFeed 在全局和巡航队列中按区域轮转候选新闻，保留区域视图的“当前区域 + global”规则，并补充回归测试。
|
||||
- `news.js` 按区域、类型、来源和数量隔离并发刷新请求，丢弃旧区域响应；跨区域时不再复用旧来源筛选。
|
||||
- 新闻展示在中文本地化未完成时回退原始标题和摘要，避免出现有内容却显示“新闻汉化中”的卡片。
|
||||
- 新闻目标位置 worker 新增优先 stream、pending reclaim、任务超时和并发处理；无效消息会确认并删除，减少队列堆积。
|
||||
- 补充 Earth 新闻源、Earth 前端结构、harness 和版本历史文档，并移除控制台 auth store 的调试日志。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.0] — 2026-06-11
|
||||
|
||||
Released: 2026-06-11
|
||||
|
||||
### Highlights
|
||||
- 将 Motion Agent 升级为可供 Web/UE 共用的双向控制服务，补齐真实 MediaPipe 识别 worker、设备控制、动作白名单和 WSL 摄像头开箱启动链路。
|
||||
- 新增 Earth 手动新闻内容组、条目、导入与重处理能力，并改进按 locale 和启用来源进行的新闻补充与多样化。
|
||||
- 对齐动捕模式下的 Earth 点击、详情锁定和卫星轨迹交互，同时完善启动脚本、测试 harness 与运维说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Motion Agent 支持命令结果、状态、骨架与手势事件，统一单路/双路输入配置，并随 `planet.sh` 默认启动；仓库内提供 usbipd-win fallback 安装包。
|
||||
- Earth 新闻服务集中处理显示就绪判断和来源多样化，避免存储层与编排层重复筛选；新增手动新闻 API 与回归测试。
|
||||
- 清理 Motion Agent 重复识别执行路径、前端不稳定随机 key 和过时计划描述，补齐 pytest 路径 harness、双语使用手册、快速开始与数据流文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.70.0] — 2026-06-04
|
||||
|
||||
Released: 2026-06-04
|
||||
|
||||
### Highlights
|
||||
- 新增后端枚举契约治理，将稳定协议状态集中到 `app/core/enums.py`，同时保持数据库和 API 的小写字符串兼容。
|
||||
- 改进 Earth 新闻分类、重要度与 Breaking 插队链路，并补齐中英文新闻源与枚举契约文档。
|
||||
- 将 Earth 船只展示改为 `vessel_current_state` 当前状态快照，保留 AIS 原始历史用于轨迹和态势分析。
|
||||
- 清理错误的船只视口刷新/订阅思路，恢复全球船只显示，并让性能优化集中到批量渲染、关闭动态聚类和减少 hover/rebuild 开销。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `earth_news_classification.py` 集中管理新闻类型、重要度和 Breaking 规则，避免抓取编排层重复判断。
|
||||
- `/api/v1/vessels/snapshot` 支持全球当前状态读取和小数 zoom，诊断信息明确返回 `vessel_current_state` 来源。
|
||||
- 船只前端使用全球 bbox + `limit=3000`，不再随相机视口重复请求或建立多视口 WebSocket 订阅。
|
||||
- 中英文技术文档同步更新船只、采集器、数据流、渲染层级、样式参数和历史计划状态。
|
||||
|
||||
---
|
||||
|
||||
## [0.69.0] — 2026-06-03
|
||||
|
||||
Released: 2026-06-03
|
||||
|
||||
325
docs/HARNESS.md
Normal file
325
docs/HARNESS.md
Normal file
@@ -0,0 +1,325 @@
|
||||
# Agent Harness
|
||||
|
||||
This harness improves discoverability, repeatability, and agent safety for the
|
||||
existing Planet project. It does not replace current project rules, scripts, CI,
|
||||
or release workflows.
|
||||
|
||||
## Authority And Conflicts
|
||||
|
||||
Existing project rules are authoritative:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. Current implementation docs under `docs/technical/`
|
||||
4. Existing scripts, especially `planet.sh`
|
||||
5. Existing Gitea workflow files under `.gitea/workflows/`
|
||||
|
||||
When harness guidance conflicts with any of the above, keep the existing rule,
|
||||
do not overwrite the existing workflow, and add a compatibility note here or in
|
||||
`docs/harness-audit.md`.
|
||||
|
||||
For frontend or documentation audits, also read the Rules Coverage Evidence
|
||||
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
|
||||
static checks, Playwright smoke coverage, and remaining manual review areas, so
|
||||
an agent can distinguish a proved harness pass from a rule that still needs
|
||||
human-quality inspection.
|
||||
|
||||
When the user describes work with product words rather than module names, use
|
||||
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
|
||||
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
|
||||
`地球`, `模型供应商`, and `发版` to the required rule modules.
|
||||
|
||||
## Starting Work
|
||||
|
||||
Recommended startup flow:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Then read only the relevant implementation docs:
|
||||
|
||||
- Backend/API/data work: `docs/technical/zh/backend-*.md` and matching English
|
||||
docs when public docs are affected.
|
||||
- Frontend/admin work: `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
- Earth work: `docs/technical/zh/earth-frontend-context.md`,
|
||||
`docs/technical/zh/earth-render-layer-order.md`, and style docs when visual
|
||||
semantics change.
|
||||
- Operations work: `docs/technical/zh/ops-runbook.md` and
|
||||
`docs/technical/zh/ops-planet-sh-startup.md`.
|
||||
- AI Provider work: `docs/technical/zh/agents-aiprovider.md`.
|
||||
- Documentation work: `docs/documentation-coverage-rules.md`.
|
||||
|
||||
Use focused inspection commands before broad reads:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Purpose | Command |
|
||||
| --- | --- |
|
||||
| First setup | `./planet.sh init` |
|
||||
| Start local stack | `./planet.sh start` |
|
||||
| Start with LAN access | `./planet.sh start --allow-lan` |
|
||||
| Restart all services | `./planet.sh restart` |
|
||||
| Restart one area | `./planet.sh restart -b`, `-f`, `-a`, or `-d` |
|
||||
| Health check | `./planet.sh health` |
|
||||
| Logs | `./planet.sh log`, `./planet.sh log -b`, `-f`, `-a`, or `-m` |
|
||||
| Create local user | `./planet.sh createuser` |
|
||||
| Destructive local reset | `./planet.sh destroy` |
|
||||
| Backend smoke tests | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` |
|
||||
| Mock AIS WebSocket | `bun run mock:ais-ws` |
|
||||
|
||||
## Harness Commands
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, isolated Docker bootstrap/proxy and startup tests, error-catalog consistency tests, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
Docker image smoke builds are expensive and are off by default:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Frontend Playwright smoke runs by default in full validation after the frontend
|
||||
build. It starts a local Vite preview and checks the `/` to Earth redirect,
|
||||
public pages, unknown-route login fallback, protected admin route login
|
||||
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
|
||||
mocked API content, Docs detail page
|
||||
language/theme/search interactions, every Docs catalog slug exposed by the
|
||||
frontend/backend metadata, the Earth iframe entry point, login error handling,
|
||||
register + email verification, password reset, standalone email verification,
|
||||
and authenticated `super_admin` rendering for every admin route plus core
|
||||
`section` deep links derived from the actual admin route and section config.
|
||||
Authenticated admin
|
||||
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
|
||||
mobile passes also fail on global horizontal overflow so table/detail panels
|
||||
must keep overflow ownership inside their own scroll regions. To enforce the
|
||||
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
|
||||
hard rendered check: the shell must resolve to the viewport height through the
|
||||
root 100% height chain, `#root`/document/body must not gain vertical overflow,
|
||||
and the desktop sidebar account/preferences area must remain inside the first
|
||||
viewport while the nav owns any excess scrolling. The smoke also
|
||||
derives the sidebar menu from the actual admin route manifest and clicks every
|
||||
visible `super_admin` menu entry on both desktop and mobile viewports, then
|
||||
exercises safe interaction paths for admin search, section tabs, the AI settings
|
||||
shortcut, logs view switching, user dialog opening, and data distribution toggles.
|
||||
It also exercises Earth News source testing, add/cancel source draft behavior,
|
||||
and manual news group creation against mocked `/earth/news-*` APIs.
|
||||
Documented AI and collector
|
||||
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
|
||||
`/collection-management?section=collector_credentials` are part of the rendered
|
||||
smoke surface:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
|
||||
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
### Visual Evidence And Style Consistency
|
||||
|
||||
- Treat user-provided screenshots and images as primary visual evidence. If a
|
||||
screenshot contradicts written text, inspect the image first and explicitly
|
||||
call out the mismatch before deciding what to change.
|
||||
- Path resolution is part of visual evidence handling. If a referenced
|
||||
screenshot path cannot be opened, try reasonable local equivalents first:
|
||||
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
|
||||
thread attachments, repository files, and obvious local attachment/download
|
||||
locations.
|
||||
- If the image still cannot be found or opened, report the exact path/access
|
||||
blocker instead of guessing. Do not infer image content from the filename, alt
|
||||
text, surrounding prose, logs, or memory.
|
||||
- OCR is acceptable evidence for text-only questions or non-multimodal
|
||||
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
|
||||
and rendering issues still require a real visual inspection or an explicit
|
||||
"could not verify visually" note.
|
||||
- Same-category UI surfaces must use one visual system per product area. Badges,
|
||||
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
|
||||
controls should reuse the shared component, shared token, or established CSS
|
||||
owner for that area instead of introducing a page-local lookalike.
|
||||
- `scripts/harness/frontend-rules-check.sh` warns when semantic
|
||||
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
|
||||
approved React and Earth CSS owner files. A warning means the reviewer should
|
||||
either move the style into the shared owner or document why this is a genuinely
|
||||
new visual family.
|
||||
|
||||
### Earth I18n Harness Rules
|
||||
|
||||
Earth i18n work must validate rendered behavior, not only static text lookup.
|
||||
Agents often miss dynamic strings that are created after initial page load, so
|
||||
the smoke treats these as first-class i18n surfaces:
|
||||
|
||||
- **Visible text and attributes**: translated checks must include `innerText`
|
||||
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
|
||||
buttons are user-facing copy, not implementation details.
|
||||
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
|
||||
BGP markers, compute centers, vessels, and news must render field labels,
|
||||
status values, source tags, action buttons, and disabled/tooltips in the active
|
||||
language.
|
||||
- **English content safety**: English mode must not fall back to Chinese news
|
||||
titles, summaries, feed names, measure words, or generic status labels. If no
|
||||
English localization exists, hide the item or use a neutral English fallback.
|
||||
- **Brand assets**: locale switching must update both text and image assets.
|
||||
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
|
||||
for English while keeping the same top-left layout and logo position.
|
||||
- **Controls and state**: switch/segmented-control visuals must follow the real
|
||||
checked/pressed state after both direct clicks and programmatic panel changes.
|
||||
A control is not valid if the state changes but the thumb, active pill, or
|
||||
`aria-*` state stays stale.
|
||||
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
|
||||
must enter through `earthMessage(...)` plus the centralized
|
||||
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
|
||||
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
|
||||
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
|
||||
or `earth:status` events.
|
||||
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
|
||||
overflow their panel. Prefer a slightly wider owning panel for important
|
||||
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
|
||||
translated tooltip.
|
||||
|
||||
Current frontend smoke explicitly covers the Earth English locale flow: brand
|
||||
image swap, settings language controls, panel switch visual sync, English news
|
||||
filtering, English detail-card text and tooltips, and TV default/source labels.
|
||||
|
||||
## Environment Requirements
|
||||
|
||||
Required for normal development:
|
||||
|
||||
- `zsh` for `planet.sh`
|
||||
- `uv` for Python dependency and test execution
|
||||
- `bun` for frontend dependency and build execution
|
||||
- Python resolved by `uv` from the root `pyproject.toml`
|
||||
|
||||
Harness command lookup first checks the current non-interactive `PATH`. If a
|
||||
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
|
||||
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
|
||||
path. This avoids hardcoding a dotfile while still covering agent environments
|
||||
that do not inherit the user's normal shell setup.
|
||||
|
||||
Required for full local stack operation:
|
||||
|
||||
- Docker and Docker Compose
|
||||
- PostgreSQL and Redis containers started by `planet.sh`
|
||||
|
||||
Optional for delivery smoke:
|
||||
|
||||
- Docker daemon for image builds
|
||||
- Helm for chart lint/template checks
|
||||
|
||||
For routine harness validation, do not install missing system software
|
||||
automatically. Report the gap and point to the explicit bootstrap entry points.
|
||||
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
|
||||
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
|
||||
This bootstrap behavior is intentional; do not invoke it merely to make harness
|
||||
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
|
||||
|
||||
Docker bootstrap regression checks use isolated command stubs and never install
|
||||
packages or modify the host daemon:
|
||||
|
||||
```bash
|
||||
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
|
||||
uv run --frozen --project . python scripts/harness/test_database_startup.py
|
||||
```
|
||||
|
||||
Database startup regressions also run in quick-check. They cover Compose
|
||||
reconciliation of existing containers, visible startup errors, published-port
|
||||
checks, bounded recreation that preserves volumes, and the backend connection
|
||||
gate before schema initialization. Their command stubs and driver mocks do not
|
||||
modify the host Docker environment.
|
||||
|
||||
## What Agents Must Not Change Automatically
|
||||
|
||||
- Do not replace Bun with npm, pnpm, or yarn.
|
||||
- Do not migrate CI from `.gitea/workflows/` to `.github/workflows/`.
|
||||
- Do not rewrite `planet.sh` lifecycle behavior as a parallel script.
|
||||
- Do not run `./planet.sh destroy` unless explicitly requested.
|
||||
- Do not commit `.env`, secrets, private keys, logs, or generated build output.
|
||||
- Do not add external integrations, hooks, or new dependency managers just to
|
||||
satisfy harness structure.
|
||||
- Do not publish internal harness docs into the product Docs UI unless a
|
||||
maintainer explicitly asks for it.
|
||||
|
||||
## Hooks And Reminders
|
||||
|
||||
No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
- Run `scripts/harness/quick-check.sh` before handing off small changes.
|
||||
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
|
||||
- Run `scripts/harness/security-check.sh` after touching config, auth,
|
||||
credentials, docs examples, or generated fixtures.
|
||||
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
|
||||
catch direct stdout/debugger calls before they reach runtime logs.
|
||||
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
|
||||
route, package-manager, debug-output, and UI rule warnings.
|
||||
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
|
||||
route changes.
|
||||
- Add focused tests before modifying backend service behavior or frontend
|
||||
workflows.
|
||||
- For docs changes, run the checks listed in
|
||||
`docs/documentation-coverage-rules.md`.
|
||||
|
||||
## Reusable Workflows
|
||||
|
||||
### Feature Work
|
||||
|
||||
1. Read `rules.md` modules for the touched area.
|
||||
2. Check `CODEMAP.md` for entry points and ownership boundaries.
|
||||
3. Inspect existing tests and docs before editing.
|
||||
4. Make the smallest behavior-preserving or feature-scoped change.
|
||||
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
|
||||
6. Update relevant docs when behavior, workflow, or operations change.
|
||||
7. For rendered frontend changes, verify the affected route with Playwright or
|
||||
the full harness smoke, because `bun run build` alone does not prove page
|
||||
usability.
|
||||
|
||||
### Bug Fix
|
||||
|
||||
1. Reproduce with a focused test or command.
|
||||
2. Patch the owning module, not a caller-side workaround.
|
||||
3. Run the focused regression test.
|
||||
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
|
||||
locally.
|
||||
|
||||
### Documentation Change
|
||||
|
||||
1. Read `docs/documentation-coverage-rules.md`.
|
||||
2. Route docs by audience: UI users, operations, or second-party developers.
|
||||
3. Keep Chinese and English technical docs paired by filename; public Docs also
|
||||
need matching frontend/backend metadata when exposed in the product Docs UI.
|
||||
4. Run the repository-specific docs checks that match the changed files.
|
||||
|
||||
### Release Or Delivery Change
|
||||
|
||||
Use the existing release skill/workflow and `.gitea/workflows/` files. Harness
|
||||
validation can smoke-check Helm and Docker locally, but it must not replace the
|
||||
release process.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `docs/harness-audit.md` records the discovery pass that led to this harness.
|
||||
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
|
||||
`agents.md` entry has been merged into it and should remain absent.
|
||||
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
|
||||
`docs/technical/{zh,en}/`.
|
||||
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
|
||||
with mocked API data. It proves route shells, auth guards, and primary admin
|
||||
sections render, but it is not a replacement for feature-specific browser QA
|
||||
against a real backend.
|
||||
- Frontend smoke prints phase-level progress by default. Use
|
||||
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
|
||||
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
|
||||
progress lines.
|
||||
161
docs/harness-audit.md
Normal file
161
docs/harness-audit.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Harness Audit
|
||||
|
||||
Last audited: 2026-06-26
|
||||
|
||||
This audit records the repository state used to add the agent harness. It is a
|
||||
compatibility note, not a replacement for existing rules or architecture docs.
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Area | Existing Command | Notes |
|
||||
| --- | --- | --- |
|
||||
| Bootstrap | `./planet.sh init` | Syncs uv/Bun dependencies, creates missing env files, starts data services, seeds defaults. |
|
||||
| Start | `./planet.sh start` | Starts backend, frontend, AI Provider, PostgreSQL/Redis, and Motion Agent when available. |
|
||||
| LAN start | `./planet.sh start --allow-lan` | Opens frontend/backend/AI Provider ports and requests Windows firewall/port cleanup when needed. |
|
||||
| Restart | `./planet.sh restart` | Supports scoped restart flags for backend, frontend, AI Provider, database, and Motion Agent. |
|
||||
| Health | `./planet.sh health` | Checks containers, backend `/health`, AI Provider `/health`, frontend, and Motion Agent state. |
|
||||
| Logs | `./planet.sh log` | Supports backend, frontend, AI Provider, and Motion Agent log views. |
|
||||
| User fallback | `./planet.sh createuser` | Interactive emergency/local account creation. |
|
||||
| Destructive reset | `./planet.sh destroy` | Requires confirmation and removes Planet-owned Docker/build/runtime state. Not a validation command. |
|
||||
| Backend CI smoke | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | Mirrors `.gitea/workflows/ci.yaml`. |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | Bun-only workflow. |
|
||||
| Root helper | `bun run mock:ais-ws` | Runs `scripts/mock-ais-ws-server.ts` from the root package. |
|
||||
|
||||
## Existing Agent Instructions
|
||||
|
||||
| File | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
|
||||
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
|
||||
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
|
||||
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
|
||||
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
|
||||
|
||||
## Existing CI Gates
|
||||
|
||||
The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
| Workflow | Gate |
|
||||
| --- | --- |
|
||||
| `.gitea/workflows/ci.yaml` | Backend smoke tests, frontend Bun build, Docker build smoke, Helm lint/template. |
|
||||
| `.gitea/workflows/release.yaml` | Builds and pushes frontend, backend, and AI Provider images on main/tag/manual release events. |
|
||||
| `.gitea/workflows/deploy-staging.yaml` | Deploys Helm release to staging and runs curl smoke tests inside the cluster. |
|
||||
|
||||
## Existing Docs And Architecture Maps
|
||||
|
||||
| Area | Docs |
|
||||
| --- | --- |
|
||||
| Current architecture and startup | `README.md` |
|
||||
| Technical docs index | `docs/technical/zh/README.md`, `docs/technical/en/README.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Operations | `docs/technical/zh/ops-runbook.md`, `docs/technical/en/ops-runbook.md` |
|
||||
| Startup internals | `docs/technical/zh/ops-planet-sh-startup.md`, `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md`, `docs/technical/en/agents-aiprovider.md` |
|
||||
| Frontend admin | `docs/technical/zh/frontend-admin-frontend-context.md`, `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth rendering | `docs/technical/zh/earth-frontend-context.md`, `docs/technical/zh/earth-render-layer-order.md`, `docs/technical/zh/earth-layer-style-reference.md` |
|
||||
| Plans and history | `docs/plans/README.md`, `docs/deprecated/README.md` |
|
||||
|
||||
## Release And Deploy Process
|
||||
|
||||
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
|
||||
`.claude/commands/release.md`.
|
||||
- Version-bearing files include `VERSION`, `frontend/package.json`,
|
||||
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
|
||||
`docs/version-history.md`.
|
||||
- Delivery automation lives in `.gitea/workflows/release.yaml` and
|
||||
`.gitea/workflows/deploy-staging.yaml`.
|
||||
- Helm chart entry point is `deploy/helm/planet/Chart.yaml`.
|
||||
|
||||
## Missing Or Unclear Areas
|
||||
|
||||
- The older lowercase `agents.md` entry has been merged into uppercase
|
||||
`AGENTS.md` so coding agents and harness tools use one source of truth.
|
||||
- `project_context.md` originally included older roadmap assumptions such as
|
||||
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
|
||||
harness pass updated it to separate active stack facts from future directions;
|
||||
current code and technical docs still remain authoritative when details drift.
|
||||
- No safe automatic hook system was already configured. This phase documents
|
||||
manual reminders instead of adding hooks.
|
||||
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
|
||||
|
||||
## Conflicts And Preserved Rules
|
||||
|
||||
| Conflict Or Tension | Resolution |
|
||||
| --- | --- |
|
||||
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
|
||||
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
|
||||
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
|
||||
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
|
||||
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
|
||||
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
|
||||
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
|
||||
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
|
||||
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
|
||||
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
|
||||
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
|
||||
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
|
||||
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
|
||||
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
|
||||
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
|
||||
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
|
||||
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
|
||||
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
|
||||
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
|
||||
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
|
||||
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
|
||||
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
|
||||
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
|
||||
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
|
||||
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
|
||||
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
|
||||
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
|
||||
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
|
||||
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
|
||||
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
|
||||
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
|
||||
|
||||
## Rules Coverage Evidence
|
||||
|
||||
This matrix records how the current harness checks the `rules.md` modules that
|
||||
matter for this frontend and documentation pass. "Automated" means the listed
|
||||
command fails when the rule regresses. "Smoke" means the rendered product route
|
||||
or interaction is opened with Playwright. "Manual" means the rule is still a
|
||||
judgment call and must be inspected during review.
|
||||
|
||||
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
|
||||
the user describes work with Chinese/product terms instead of module names. The
|
||||
index is the routing layer; this table is the coverage/evidence layer.
|
||||
|
||||
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
|
||||
| --- | --- | --- | --- |
|
||||
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
|
||||
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
|
||||
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
|
||||
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
|
||||
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
|
||||
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
|
||||
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
|
||||
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
|
||||
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
|
||||
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
|
||||
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
|
||||
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
|
||||
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
|
||||
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
|
||||
|
||||
## Harness Files Added
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
|
||||
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
|
||||
| `CODEMAP.md` | High-level codebase map and validation references. |
|
||||
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
|
||||
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
|
||||
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
|
||||
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
|
||||
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
|
||||
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
|
||||
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
|
||||
@@ -22,6 +22,8 @@
|
||||
|
||||
当前重点入口：
|
||||
|
||||
- [算力与资源态势：展示、采集与 AI 迭代实施计划](compute-resource-intelligence-plan.md)
|
||||
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
@@ -33,6 +35,7 @@
|
||||
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
|
||||
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
|
||||
- [Motion Agent v2 控制协议与 3D 标定路线](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md)
|
||||
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
|
||||
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
|
||||
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
|
||||
|
||||
62
docs/plans/admin-console-i18n-plan.md
Normal file
62
docs/plans/admin-console-i18n-plan.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# 控制台 i18n 接入计划
|
||||
|
||||
**状态**：基础设施已落地，大型业务页迁移继续进行
|
||||
**创建日期**：2026-06-29
|
||||
**核心目标**：把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系，让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
|
||||
|
||||
## 背景
|
||||
|
||||
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口，但语言状态只保存在 `docs-lang`，不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主，导致用户切到英文文档后，控制台仍是中文。
|
||||
|
||||
本计划把前端语言偏好收敛到 `planet-locale`，默认 `zh-CN`，支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口，通过前端映射与全局 locale 对齐。
|
||||
|
||||
## 设计决策
|
||||
|
||||
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层，避免长期维护自研插值、hook 和资源加载逻辑。
|
||||
- 前端统一语言枚举为 `zh-CN` / `en-US`；Docs 请求继续转换为 `zh` / `en`，新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
|
||||
- 语言偏好首版只保存在浏览器 `localStorage`，不新增后端用户设置字段。
|
||||
- `docs-lang` 保留为兼容读取和写入项，让已访问过 Docs 的浏览器能平滑迁移。
|
||||
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key；大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
|
||||
|
||||
## 分期
|
||||
|
||||
### P1：统一语言基础设施
|
||||
|
||||
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
|
||||
- 在 `frontend/src/main.tsx` 里初始化 i18n，并同步 `document.documentElement.lang`。
|
||||
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
|
||||
|
||||
### P2：高复用界面迁移
|
||||
|
||||
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
|
||||
- 搜索索引按当前语言展示，同时保留中英文关键词以免降低可发现性。
|
||||
- 用户管理页作为独立业务页示范，迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
|
||||
|
||||
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化，以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
|
||||
|
||||
### P3：大型业务页收敛
|
||||
|
||||
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
|
||||
- 过渡期保留 legacy UI 翻译桥，只处理 admin/auth 容器里的精确静态文本和属性。
|
||||
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
|
||||
|
||||
### P4：移除过渡桥
|
||||
|
||||
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时，删除 legacy UI 翻译桥。
|
||||
- 增加 key 完整性检查，确保 `zh-CN` 和 `en-US` 资源结构一致。
|
||||
|
||||
## 验证
|
||||
|
||||
- `cd frontend && bun run build`
|
||||
- `scripts/harness/frontend-rules-check.sh`
|
||||
- `scripts/harness/docs-consistency-check.sh`
|
||||
- `scripts/harness/quick-check.sh`
|
||||
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `frontend/src/i18n/`：统一 locale、资源和过渡桥。
|
||||
- `frontend/src/pages/Docs/Docs.tsx`：Docs 语言状态改为读取全局 locale。
|
||||
- `frontend/src/admin/components/layout/AdminLayout.tsx`：控制台侧边栏语言切换、导航和搜索文案。
|
||||
- `frontend/src/admin/search/indexers.ts`：Admin 搜索目标本地化。
|
||||
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`：当前实现上下文。
|
||||
592
docs/plans/compute-resource-intelligence-plan.md
Normal file
592
docs/plans/compute-resource-intelligence-plan.md
Normal file
@@ -0,0 +1,592 @@
|
||||
# 算力与资源态势：展示、采集与 AI 迭代实施计划
|
||||
|
||||
状态：待实施。2026-09-14 已完成当前代码边界、主要公开数据入口和历史指标口径核查；本计划不代表采集器、评分服务或 AI 迭代已经上线。面向产品、开发、数据研究与运维人员，作为此专题后续实现的设计依据。
|
||||
|
||||
## 0. 当前执行方向与范围
|
||||
|
||||
平台长期定位是“信息展示 + 决策辅助”。当前实施重点是把全球算力对比作为第一个成熟专题做清楚：地图显示资源在哪里，图表说明规模和相对位置怎样变化，证据说明数字从哪里来，AI 帮助阅读变化。
|
||||
|
||||
采用“专题先交付，公共能力按复用需要沉淀”的顺序。第 7—9 节的完整 AI 迭代、模型实验与通用持久化是后续路线，不是首版展示的前置条件；第 6 节总分保持实验性，不能为了首屏有大数字而生成不完整排名。
|
||||
|
||||
| 当前要做 | 当前完成形态 | 后续才扩展 |
|
||||
| --- | --- | --- |
|
||||
| 算力数据可信 | 修复既有采集，国家/设施/精度/年份不混用 | 大规模自动来源发现、完整证据图谱 |
|
||||
| 全球分布与国家对比 | 现有算力中心地球 + 可比参与方 + 明确样本覆盖 | 制造、矿产、电力和贸易的跨主题关系 |
|
||||
| 历史展示 | 两张核对后的图、时间选择、已知事件轨道 | 复杂预测、不确定性传播与可训练参数 |
|
||||
| 证据与解释 | 来源、参考年、关键字段；基于数据快照的简短 AI 说明 | 自动补证、模型候选、回测、影子运行与提升 |
|
||||
|
||||
首版验收路径：打开算力专题 → 看全球公开设施分布 → 选中国和美国 → 同时看算力与创新历史 → 点一个变化或设施 → 查看出处及解释。此路径稳定后，才扩大评价维度。
|
||||
|
||||
## 1. 要交付什么
|
||||
|
||||
在智能星球现有算力中心图层上增加“算力与资源态势”模式，并在控制台提供对应的数据管理和模型实验工作区。核心问题是：各方拥有什么资源、资源如何形成可用能力、相对位置如何变化、变化有何证据，以及未来什么约束可能改变格局。
|
||||
|
||||
首批比较中国和美国；数据结构从第一天支持多方。随后纳入欧洲、日本、韩国、印度及其他数据合格的参与方。地区汇总与成员国不得重复计入全球分母。所有国家和地区字段使用现有 `normalize_country()` 和规范字典，不以网页原文或 AI 判断覆盖项目标签。
|
||||
|
||||
交付三个相互连接的产品能力：
|
||||
|
||||
| 能力 | 用户获得的结果 | 首版边界 |
|
||||
| --- | --- | --- |
|
||||
| 历史观测 | 算力绝对规模、全球份额、创新指数与分差、设施时间线 | 使用可验证观测；缺失年份可见，不插值冒充事实 |
|
||||
| 变化解释 | 哪些设施、资源、数据修订造成指标变化，每项贡献和证据 | 数学贡献可复算；原因解释区分事实、假设与未解问题 |
|
||||
| 模型改进 | AI 发现缺口、补证、提出改进，经过验证产生新版本 | 固定工作流先行；不能由大模型直接决定国家分数 |
|
||||
|
||||
第一版先交付可信的事实与回放，不以总分完整作为上线前提。完整创新指数分项、有效算力参数和项目交付数据不足时，继续展示已完成的观测能力，综合评分显示“数据不足”，不生成示意国家排名。
|
||||
|
||||
不在第一版承诺全球所有设施的完整清单、精确的国家真实算力、每日变化的国家创新能力、可验证的国家综合实力真值，或自动预测某年必然追平。
|
||||
|
||||
## 2. 当前项目基础与必须补齐的边界
|
||||
|
||||
以下结论来自当前文件检查，未对生产数据库、凭证可用性或线上采集结果做完整审计。
|
||||
|
||||
| 已有入口 | 已确认能力 | 本专题需要补齐 |
|
||||
| --- | --- | --- |
|
||||
| [采集基类](../../backend/app/services/collectors/base.py)、[数据作业](../../backend/app/services/data_jobs.py) | 采集、转换、任务状态、快照、取消与重试基础 | 文档型数据源、质量门槛、领域规范化、失败保留正式快照 |
|
||||
| [Epoch 采集器](../../backend/app/services/collectors/epoch_ai.py) | GPU 集群采集入口 | 改读公开 CSV；去掉解析失败返回示例记录；单位、精度、状态和时间字段校正 |
|
||||
| [TOP500 采集器](../../backend/app/services/collectors/top500.py) | 榜单与详情解析 | 去掉示例回退、固定参考日期、排名作为身份；按当期表头解析单位 |
|
||||
| [采集记录](../../backend/app/models/collected_data.py)、[快照](../../backend/app/models/data_snapshot.py) | `entity_key`、快照关联、前一记录和变化摘要 | 原始证据版本、发表时间、有效时间、指标注册和可重放的观测集合 |
|
||||
| [算力位置服务](../../backend/app/services/compute_center_locations.py)、[GeoJSON](../../backend/app/api/v1/visualization.py) | 设施定位、位置置信信息、未定位记录 | 国家统计与地图可定位集合分离；匿名设施不能猜坐标 |
|
||||
| [算力中心渲染](../../frontend/public/earth/js/compute-centers.js)、[Earth 壳](../../frontend/src/pages/Earth/Earth.tsx) | Three.js 交互层；Earth 由 iframe 加载独立页面 | 历史时点、国家比较、双图、关系层和解释面板 |
|
||||
| [图层适配器](../../backend/app/services/earth_layer_adapters.py)、[数据库监听](../../backend/app/services/earth_db_change_listener.py) | 通过 PostgreSQL outbox 驱动缓存与 `earth_updates` | 正式指标快照发布事件；草稿与实验结果不改变默认地球 |
|
||||
| [AI Client](../../backend/app/services/ai_client.py)、[提示词注册](../../backend/app/ai_tasks/prompts.py) | 全局 provider 配置、模型调用、任务提示词及运维覆盖 | 算力专题的固定工作流、结构化输出验证、预算与审计 |
|
||||
| [搜索工具](../../backend/app/services/ai_tools/web_search.py)、[网页取证](../../backend/app/services/ai_tools/web_fetch.py) | 搜索和基础 HTML 正文提取 | PDF/表格提取、引用位置、下载时大小限制、URL 与重定向的访问限制 |
|
||||
| [证据辅助函数](../../backend/app/services/ai_tools/evidence_store.py) | 摘要、去重和内容哈希规范化 | 该文件不是持久化证据库，需补建不可变原文存储和证据记录 |
|
||||
|
||||
沿用[数据作业与 Outbox 架构](../technical/zh/data-job-earth-sync-architecture.md)及[轻量 Agent 编排计划](agents-light-orchestrator-websearch-plan.md)：PostgreSQL 负责可靠账本，Redis 负责缓存；不另起 Kafka、Celery 或另一个独立调度真相源。`aiprovider` 保持协议适配，业务取证、估计、工具权限和模型发布全部由后端负责。
|
||||
|
||||
当前 `JobType` 只有采集、清理和 Earth 刷新。新增分析、实验工作流需要显式扩展任务类型、worker 分发和取消策略，不能把现有采集队列描述成已经支持通用 Agent。
|
||||
|
||||
## 3. 页面与地球怎样展示
|
||||
|
||||
### 3.1 默认工作区
|
||||
|
||||
保留 `/earth` 主入口，在现有算力中心工具中增加专题模式，避免增加一个与地球无关联的新大屏。进入后使用以下概念布局；这是规划线框，不是现有页面截图。
|
||||
|
||||
```text
|
||||
┌ 时间/参考期 · 中国/美国/其他参与方 · 指标口径 · 观测/情景 ┐
|
||||
│ │
|
||||
│ 3D 地球：设施、资源、已证实关系 国家对比面板 │
|
||||
│ 绝对量/份额 │
|
||||
│ 变化贡献 │
|
||||
│ 证据与缺口 │
|
||||
├────────────── 历史时间轴 / 事件轨道 ────────────────────┤
|
||||
│ 算力规模与份额图 综合创新指数与分差图 │
|
||||
└───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
桌面默认同时显示两张历史图，用相同国家配色，但各自保留单位和实际观测年份。图表高度优先保证刻度、来源标识与数据点可读，不能通过压缩字体维持布局。较窄屏幕把图表移入可切换的详情区，国家和设施详情使用抽屉；不叠加多个遮挡地图的浮层。
|
||||
|
||||
沿用项目一屏高度链和主题。桌面在 1366×768、1920×1080，移动端在约 390×844 验证，另测 125%/150% 浏览器缩放。页面根与中间 flex 容器分别正确设置高度和 `min-height: 0`，仅详情、长表和证据正文内部滚动。
|
||||
|
||||
### 3.2 视图职责
|
||||
|
||||
| 视图 | 主内容 | 点击或切换后的行为 |
|
||||
| --- | --- | --- |
|
||||
| 国家总览 | 可比算力绝对量、份额、子分、实际参考期；总分合格才出现 | 选中参与方，定位地球并联动两图；不重设全局评分尺度 |
|
||||
| 算力趋势 | 绝对规模、全球份额、双边份额三种独立视图 | 同一来源口径内切换；不把百分比轴和 FLOP/s 轴混合 |
|
||||
| 创新趋势 | 各方指数；只有分差时只显示中美分差 | 切换年度，查看分项与报告版本，不能反推缺失的单国得分 |
|
||||
| 变化账单 | 新投产、扩建、退役、效率变化、资料补录、方法修订 | 点选贡献定位资产；逐步展示输入、计算和证据 |
|
||||
| 设施详情 | 设备、精度、投产状态、功率、运营方、地点精度、历史版本 | 展开相关公告、设备来源及不确定字段 |
|
||||
| 资源关系 | 已证实的芯片供给、所有权、电力接入或网络连接 | 选一类关系再显示，不能让所有关系默认铺满地球 |
|
||||
| 情景工作区 | 并网推迟、交付变化、效率假设及结果区间 | 独立 scenario ID；退出恢复正式快照 |
|
||||
| 控制台研究区 | 数据缺口、待核验证据、采集任务、模型实验和版本 | 使用已有采集与 AI 设置入口；Playground 仍用于调试 |
|
||||
|
||||
### 3.3 两张图进入产品的明确规则
|
||||
|
||||
总算力图以核对后的报告观测为起点：2020 年美国/中国为 36%/31%，2021 年 34%/33%，2022 年 34%/33%，2023 年 41%/31%。原始来源和报告版本见第 15 节。2024—2025 年暂不填值；2026 年 42%/33% 只保存为待核实主张，不能接到正式历史曲线上。
|
||||
|
||||
创新图使用报道明确披露的 2023、2024、2025 年分差 22.02、19.96、17.95 分。报告发表于 2026 年，必须区分报告年份与参考年份。完整分项、年度得分和方法修订仍需采集。图中是综合创新指数，不是纯算力或纯产出。
|
||||
|
||||
原始主张、报告估计、已验证记录和未来情景用文字标签区分。年度点之间的视觉连线只辅助阅读，不进入计算；不能把连线插值写成月度实测。正式指标没有新观测时保留参考期并提示新鲜度，不让刷新网页造成“国家实力实时跳动”。
|
||||
|
||||
默认“历史回顾”允许查看最新证据重述后的过去；“当时可知”只使用在截止时点已经发表且可证明可获得的资料。每张图同时展示参考期与知识截止时间。已选 2025 年而指标最新只到 2023 年时，必须显示“最新观测：2023”，或在严格同年模式显示缺失。
|
||||
|
||||
### 3.4 地球编码与交互
|
||||
|
||||
- 节点面积按同口径能力编码；跨度过大时可切换对数档位并明示图例。国家颜色只用于身份，不能把颜色本身当作领先或落后。
|
||||
- 已投产用实心，建设或情景用空心/虚线；置信不足使用独立标记，不能把低置信透明度误读为小算力。
|
||||
- 来源只到国家或区域时，保留在国家/区域统计和未定位列表；不画在首都，不把行政区域中心当作设施坐标。
|
||||
- 国家总量与可见设施合计可以不同，分别显示覆盖范围。不得只统计有坐标的节点，也不得把宏观缺口按现有节点比例摊分。
|
||||
- Epoch 当前公开版本对中国部分集群匿名化并取整；匿名记录只能按允许的粒度使用。新增官方公告可作为独立证据，不能依据匿名数值猜出设施身份或制造地点。
|
||||
- 供应、所有权和网络关系采用不同图例，显示关系来源与有效期；商业供应关系不等同于实际运输路线。
|
||||
- 全国可汇总吞吐与最大单集群能力分别展示；地理分散设备不能合并成一个训练集群。
|
||||
- 时间回放使用稳定实体 ID 更新已有节点。隐藏图层、变更时点或情景时同步清理 hover、locked、tooltip 和 selection；请求过期结果不得覆盖新时点。
|
||||
- 使用现有 Three.js Interactable、图标和实例化能力。新增壳层必须遵循现有深度顺序，在远距约 50% 缩放检查闪烁；不另建悬浮球面掩盖深度问题。
|
||||
|
||||
## 4. 数据需要补充采集什么
|
||||
|
||||
### 4.1 P0：决定首版可信度的数据
|
||||
|
||||
“已核对入口”只表示资料或文档可查，不表示接口凭证、完整历史和生产连通性已经验证。以下频率是 Planet 建议检查频率，不是来源承诺的更新频率。
|
||||
|
||||
| ID / 优先级 | 数据与来源 | 必需字段/粒度 | 采集方式与频率 | 当前缺口与采用条件 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D01 / P0 | 信通院历年算力报告 | 国家×参考年×口径；绝对规模、全球总量/份额、单位、精度、估算方法、版本 | 已取得报告先受控导入；月度检查新版；PDF 表格提取后核验 | 已核对 2020—2023 份额及部分绝对量；补 2024 以后和其他国家；跨版本方法改变时断开序列 |
|
||||
| D02 / P0 | Epoch GPU Clusters | 集群/版本；设备数、型号、16/8/32-bit 性能、所在地、状态、投产/退役时间、功率、扩建关系 | 官方公开 CSV，每日条件抓取，按内容哈希去重 | 改造现有 HTML 解析；处理匿名、取整、缺坐标和覆盖偏差；不能声称完整全国清单 |
|
||||
| D03 / P0 | TOP500 / 可选 Green500 | 稳定系统 ID×榜单期；Rmax/Rpeak、FP64、功率、机构、名次、列表日期 | 每周检查新榜单；按发布期导入历史与详情 | 修复 rank 作为 source ID、固定日期和单位问题；榜单消失不等于退役；不当成 AI 总算力 |
|
||||
| D04 / P0 | 全球 AI 创新指数原报告及发布材料 | 国家×参考年×指标；单国总分、分项、权重、标准化、成员范围、方法版本 | 文档导入＋月度更新检查 | 当前只具备部分新闻转述分差；优先获得分项和年度可比性证据；不能用差值合成各国历史得分 |
|
||||
| D05 / P0 | 已有位置维表与公开设施公告 | 实体别名、运营方、国家、地点、精度、采用依据、有效期 | 复用位置管线，新增/变化时触发 | 全国汇总独立于定位成败；不把同址不同分期误合并，也不把扩建当第二个全量集群 |
|
||||
|
||||
首版最低闭环：D01 和 D04 的已验证历史可先支撑双图，D02/D03/D05 支撑设施地图；两者不互相伪装为同一覆盖范围。若 D04 分项不可取得，创新总分仍作为独立观测，O 分与综合 S 不发布。
|
||||
|
||||
首版可以先受控导入已核实的年度数据，不必先完成整套 PDF 自动解析和报告发现。D04 原报告分项继续作为补数任务，不阻塞已经核对的创新分差图；原始摘录、出处、年份和未核实范围必须随数据保存。
|
||||
|
||||
### 4.2 P1/P2：有效能力、增长与资源关系
|
||||
|
||||
| ID / 优先级 | 补充数据与候选来源 | 粒度与核心字段 | 方式/频率 | 对模型的作用及限制 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| D06 / P1 | Epoch 芯片销售、所有权、硬件数据；厂商规格 | 芯片型号×时间；规格、销售/交付、持有主体、数量、性能换算、估计区间 | 官方数据文件与规格文档；周查/季度快照 | 校验硬件供给；设计者、所有者、实际所在地与使用权分开；不能把出货和已部署重复加总 |
|
||||
| D07 / P1 | MLPerf Training / Inference、可复现厂商测试 | 配置×任务×软件版本；模型、质量门槛、batch、上下文、时延、吞吐、能耗、可用状态 | 官方结果导入；月查新版本 | 校准特定任务的性能系数；不同 benchmark 版本/质量/负载不直连；实验室成绩不等同生产利用率 |
|
||||
| D08 / P1 | IEA、电力统计机构、EIA、并网与运营方资料 | 国家/地区/项目×时点；发电、用电、接入 MW、PUE、并网日期、可用负荷 | 国家统计月/季；报告年；项目公告日查 | 电力作为设施约束；国家发电量、用电 TWh 不能直接转换机房可用 MW；中国与美国地方数据可得性不同 |
|
||||
| D09 / P1 | 项目运营方、许可部门、官方采购和工程进展 | 项目×阶段；公告、开工、设备交付、并网、验收、部分投产、撤销、金额与范围 | 已注册来源每日检查，重要项目每周核验 | 建立投产与延期标签；最终失败项目也保留，避免只收集成功项目；单纯新闻未更新不判定失败 |
|
||||
| D10 / P1 | 公司投资者关系、SEC EDGAR、境内及其他市场法定披露 | 法人/分部×财季；CapEx、已付款/指引、租赁、合同、地域/业务归属 | 官方 API/报告；日查新披露，按季归档 | 用于资金与项目约束；总 CapEx 不是 AI 投资；合并报表、融资租赁、设备与机房成本须防重复 |
|
||||
| D11 / P1 | 创新报告分项、公开模型评测、OpenAlex；HF 作为辅助 | 机构/国家×年/月；研发成果、质量归一化产出、可比模型能力、应用指标 | 原报告年；评测周；研究元数据月 | 重建不含重复资源项的 O；合作成果使用明确分摊；不从作者姓名推断国籍；下载量不代表实际使用或产业产出 |
|
||||
| D12 / P2 | 晶圆制造、先进封装、HBM、关键设备的正式披露 | 工厂/供应方×工艺/产品×季度；已投产能力、良率范围、交付、客户与依赖 | 公司/监管原件；月查、季度快照 | 建立瓶颈和替代关系；晶圆数不能无依据换成 GPU 数；总行业份额不能归给某集群 |
|
||||
| D13 / P2 | USGS、各国统计、UN Comtrade | 商品×国家×年/月；产量、储量、加工量、贸易流、HS 版本和单位 | 年度报告、月/年贸易数据 | 扩展矿产与资源依赖；储量、开采和加工分开；宽泛服务器税号不能反推高端芯片数量 |
|
||||
| D14 / P2 | 既有海缆、PeeringDB、BGP 与已证实连接资料 | 设施/网络/关系×有效期；容量声明、连接、依赖、故障 | 复用现有产品，按其真实更新节奏 | 提供连通性和韧性背景；BGP 可见性不是带宽，海缆容量不是集群内部互联性能 |
|
||||
|
||||
EIA、SEC、Epoch、MLCommons、IEA、USGS 已核对资料入口；UN Comtrade 门户可见，但本次未验证可调用 API，接入要单独完成认证、额度、字段和历史测试。OpenAlex 需在接入时按最新访问规则配置，不假定无限制匿名访问。所有付费、登录和受限数据只列为可选增强，不作为首版隐藏前提。
|
||||
|
||||
### 4.3 首批采购与补数优先顺序
|
||||
|
||||
1. 先完成现有 Epoch/TOP500 的真实性与时间口径修复，避免新增来源扩大错误。
|
||||
2. 导入信通院历史、创新分差和原始出处；追索创新原报告分项及最新版总算力数据。
|
||||
3. 建立中美重点设施及项目阶段台账，并记录未知量。重点设施用于高影响解释，不冒充全国抽样无偏估计。
|
||||
4. 用官方规格与 MLPerf 建立少量明确工作负载的换算表；其余硬件维持理论值或范围。
|
||||
5. 补项目级并网、交付、资金与退役记录，积累可校准的真实标签。
|
||||
6. 最后扩展制造、HBM、矿产与多方关系；每个新主题必须有对应指标和展示消费者，不只收集而不使用。
|
||||
|
||||
## 5. 采集到正式数据的流水线
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[注册的数据源] --> B[原文与内容哈希]
|
||||
B --> C[确定性解析或 AI 提取候选]
|
||||
C --> D[单位/时间/实体/引用校验]
|
||||
D --> E[版本化有效观测]
|
||||
D --> F[冲突与待核实队列]
|
||||
E --> G[确定性指标计算]
|
||||
G --> H[正式结果快照]
|
||||
H --> I[Outbox 与缓存更新]
|
||||
I --> J[地球/双图/变化账单]
|
||||
```
|
||||
|
||||
### 5.1 数据源合同
|
||||
|
||||
每个数据源保存来源机构、域名与下载入口、文档类型、凭证配置引用、检查频率、预期粒度、单位与精度、更新时间语义、空结果语义、历史范围、使用许可、维护负责人和解析器版本。授权文件或数据保存在运行数据存储，不提交到仓库；公开 UI 只展示允许公开的摘录或链接。
|
||||
|
||||
API/CSV 优先；HTML/PDF 采用固定解析；必须用 AI 时先产生候选，核验后保存确定性映射。扫描件 OCR 必须保留页号、表格区域和原图，数值不能只靠模型复述。下载器在流式读取时限制实际字节量，不能下载完整文件后才截断；按域名限速，遵守来源规则。
|
||||
|
||||
### 5.2 事实键与时间
|
||||
|
||||
观测键至少包括实体、指标、单位、精度、工作负载、地理范围、归属口径、参考期及来源版本。历史至少区分：
|
||||
|
||||
| 时间 | 含义 |
|
||||
| --- | --- |
|
||||
| `reference_start/end` 或 `effective_at` | 数据描述的年度、季度或设施事件发生时间 |
|
||||
| `published_at` | 资料何时首次公开；不知道就保持未知 |
|
||||
| `retrieved_at` | Planet 何时取得该版本 |
|
||||
| `recorded_at` / `superseded_at` | 系统何时采用、替换该观测 |
|
||||
|
||||
抓取日期不代替投产日期；报告年份不代替参考年份；机构估计、实测、公告、模型估计和人工情景分别标识。只有年份时保留年份精度，不能伪造 1 月 1 日投产；回放按时间范围或“当年”事件显示。
|
||||
|
||||
### 5.3 合并与质量门槛
|
||||
|
||||
- 稳定身份优先使用官方系统 ID、项目 ID、法人 ID 和可信来源链接。TOP500 排名只作为观测属性。自动实体合并必须可撤销，保存别名和合并理由。
|
||||
- 同一集群的全量扩建公告、一期/二期、重复转载与另一个数据集的同一设备只计一次。芯片交付和机房投产分别保留，不能合并为两个已投产资产。
|
||||
- 国家级统计、企业所有权统计、设施地理统计是不同视图。国家公司在海外租用算力不得同时作为两国境内部署能力；全球成员集合固定且版本化。
|
||||
- 转换单位前先核验浮点精度、稀疏/稠密、峰值/实测和时间量纲。FLOP 与 FLOP/s、TWh 与 MW 均不得互换。
|
||||
- 解析为空、字段突然缺失、总量异常下降或源结构改变，进入失败/隔离状态，保留上次正式结果。缺榜、缺行或匿名化不直接触发实体退役。
|
||||
- 多条转载归到同一 `source_family`；媒体数量不构成多份独立证据。冲突先检查口径，再保存多个主张及采用规则，不能简单取平均。
|
||||
- 缺坐标不影响已知国家统计；未知国家不强行分配。数据覆盖率可描述已登记实体或必填字段的完整率；无法知道总体时不声称“覆盖全球百分之多少”。
|
||||
- 一位有效数字等取整值保留取整范围；这是观测精度范围，不冒充统计置信区间。
|
||||
|
||||
首批历史导入先双人或规则加人工抽查核验高影响记录，保存一套回归样本。后续同版本确定性解析通过固定门槛可自动采用；新来源、新语义或无法解释的大幅变化进入候选队列。
|
||||
|
||||
## 6. 数学模型如何工作
|
||||
|
||||
### 6.1 分清事实、估计和评价
|
||||
|
||||
模型分三层：观测层保存来源说了什么；估计层回答可用能力、交付时间和不确定性；评价层按明确价值取向组合指标。国家综合实力没有一个可直接取得的标签真值，因此不能声称“AI 找到了客观最优的国家评分权重”。
|
||||
|
||||
保留 v2 的模型草案：
|
||||
|
||||
\[
|
||||
S_{c,t}=0.40H_{c,t}+0.30O_{c,t}+0.20G_{c,t}+0.10R_{c,t}
|
||||
\]
|
||||
|
||||
H 是资源能力，O 是不重复的创新与应用表现，G 是扩张潜力，R 是韧性；四项均为 0—100 的版本化子分。S 称为“综合态势分”，包含未来潜力，不能称作已投产算力、胜率或国家实力百分比。权重是默认设计参数，需通过敏感性分析和评价目的评审后冻结。
|
||||
|
||||
### 6.2 资源能力 H 与两类算力模式
|
||||
|
||||
总算力模式使用 D01 的同口径绝对规模；AI 训练模式使用明确工作负载的有效能力；推理模式另选固定模型、质量、上下文与时延要求。不同模式不共用没有物理依据的换算，也不比较彼此总分。
|
||||
|
||||
\[
|
||||
C^{eff}_{c,t,w}=\sum_{j\in deployed(c,t)} C^{peak}_{j,t,p}\,u_{j,t,w}
|
||||
\]
|
||||
|
||||
`p` 是固定精度，`w` 是参考工作负载；`u` 由可比测试、已知运行约束或经校准的估计支持。没有依据时只发布理论能力，或者明示区间；不能统一假设某国芯片只有另一个国家的固定折扣。
|
||||
|
||||
内存、互联、软件和电力会影响同一个有效利用系数，不得分别随意乘一串折扣重复惩罚。若已知场地供电上限，可用场地总功率除以 PUE 约束 IT 负荷，再按同配置效率估计可运行容量；已在利用系数中计入的限电不再扣一次。总吞吐与可用于单次大训练的最大连续集群另列。
|
||||
|
||||
国家宏观估计和公开集群样本并列校验，不直接把二者相加。若 H 只能覆盖已观测设施，分数标题和参与方排名必须明确限定为这个样本，不外推为全国真实能力。
|
||||
|
||||
### 6.3 创新与应用 O
|
||||
|
||||
完整创新指数 I 单独显示。取得 D04 分项后，建立指标归属表，逐一标注资源、扩张、韧性或创新；只把不重叠的研发质量、模型/科研成果与应用表现用于 O。不是简单执行“创新总分减去算力分”，因为不同分项的权重和标准化可能不同。
|
||||
|
||||
备用指标可来自 D11，但新增指标意味着新定义和新版本，不能偷偷替换原报告。论文数量须去重、处理合作分摊及引用年龄；模型结果要冻结评测版本；应用指标明确平台样本。专利、下载、融资和模型排行榜均不是彼此的等价替代。
|
||||
|
||||
\[
|
||||
D^I_t=I_{US,t}-I_{CN,t},\qquad v^I_t=(D^I_{t-1}-D^I_t)/\Delta years
|
||||
\]
|
||||
|
||||
2023—2025 年披露分差缩小 4.07 分，约为起点分差的 18.5%；不能解释为中国能力提高 18.5%，也不能据此推断算法效率提升。只有差值不能恢复各国绝对得分。跨年方法不一致时分段显示，不能外推追平时间。
|
||||
|
||||
### 6.4 扩张潜力 G
|
||||
|
||||
\[
|
||||
G^{raw}_{c,t,h}=\sum_{j\in pipeline(c,t)}P(T_j\leq t+h\mid X_{j,\leq t})\,\Delta C^{eff}_j
|
||||
\]
|
||||
|
||||
默认 `h=12 个月`。X 只含当时可知的项目状态、设备交付、并网、建设和资金证据。分期投产拆分未交付增量；已投产部分进入 H 后从剩余 G 扣除。公告金额不直接转成已投产能力。
|
||||
|
||||
初期采用明确的保守/基准/乐观情景，没有历史标签就不提供伪精确概率。标签积累后可比较阶段条件概率、校准的逻辑回归或生存分析；处理尚未到期项目的右删失、取消与部分交付，不能把所有未完成记录标成失败。
|
||||
|
||||
### 6.5 韧性 R
|
||||
|
||||
\[
|
||||
R_{c,t}=100\sum_s q_s\,\min(1,C^{eff}_{c,t,s}/C^{eff}_{c,t,base})
|
||||
\]
|
||||
|
||||
s 是对各方一致定义的冲击情景，例如特定供应类别中断或并网延迟；q 为公开、固定的情景权重，除非有概率依据，否则不称为发生概率。基准能力为零或未知时不计算此比值。
|
||||
|
||||
关系图用于识别依赖、替代和共同故障源。没有替代来源证据就标为未知；政策公告通常改变未来供应或可用性假设，不自动删除境内现有芯片存量。
|
||||
|
||||
### 6.6 标准化与发布条件
|
||||
|
||||
数量型正向指标可采用固定基期映射：
|
||||
|
||||
\[
|
||||
N(x;a,b)=100\,clip\left(\frac{\ln(1+x/a)}{\ln(1+b/a)},0,1\right),\quad a,b>0
|
||||
\]
|
||||
|
||||
a、b 与 x 单位一致，来自有覆盖说明的固定参考面板，存入模型版本。成本型、强度型和比例型指标各自定义映射。不得按每日领先国家重定标，也不能因加入一个国家导致全部历史分数被静默重写。
|
||||
|
||||
关键输入缺失时 `S=null`，保留可用子分，不填零或自动重分配权重。O、G 或 R 没有可比输入时综合分不上线；原始图表和证据浏览仍可上线。样本少、匿名取整或来源偏差反映在不确定性和适用范围中，不直接扣国家实力分。
|
||||
|
||||
不确定性计算分开保存观测精度、参数估计和情景假设。只有有依据的输入分布才运行 Monte Carlo；共享芯片规格、同一公告、同一供应链的误差必须相关采样。无法量化的覆盖偏差直接披露，不能靠窄置信区间掩盖。排名用“无法区分”处理不稳健结果，不展示过多小数制造精确感。
|
||||
|
||||
### 6.7 变化与归因
|
||||
|
||||
\[
|
||||
g_{c,t}=C_{c,t}/C_{c,t-1}-1,\quad p_{c,t}=C_{c,t}/\sum_kC_{k,t},\quad b_{CN,t}=C_{CN,t}/(C_{CN,t}+C_{US,t})
|
||||
\]
|
||||
|
||||
全球分母必须完整定义；双边份额不得标为全球份额。按报告披露值，2022—2023 年中国总算力从 302 增至 435 EFlops，约增长 44%，份额从 33% 降至 31%。这是“绝对增长、相对份额下降”，不是存量消失。算力与创新现有历史窗口不同，不能拼成同一时期的因果结论。
|
||||
|
||||
同权重、同版本下，分项账单严格满足：
|
||||
|
||||
\[
|
||||
\Delta S=0.40\Delta H+0.30\Delta O+0.20\Delta G+0.10\Delta R
|
||||
\]
|
||||
|
||||
分项内非线性与交互影响可使用固定分组的 Shapley 分解；必须保存分组、基线和近似误差，不能把任意计算顺序产生的贡献当唯一解释。数学归因只解释模型输出，不证明宏观因果。
|
||||
|
||||
正式发布中始终分开：现实变化、补录旧事实、来源修订、模型/基期修订。版本升级用“同一输入、不同模型”的桥接结果展示，不能把换模型造成的跳变计入国家增长。
|
||||
|
||||
## 7. 项目内 AI 如何参与
|
||||
|
||||
### 7.1 运行方式
|
||||
|
||||
复用控制台已经配置的模型，通过 `AIProviderClient` 调用；不要求新的模型供应商，也不把供应商名称写死在业务代码中。第一阶段采用后端固定步骤工作流，各角色是任务模板，可以由同一个模型执行，不需要首先建设复杂的多 Agent 协商系统。
|
||||
|
||||
在现有提示词注册表中新增以下拟议 task key；提示词版本与运维覆盖的有效内容哈希进入每次运行记录。
|
||||
|
||||
| 任务模板 | 输入 | 允许输出 | 关键约束 |
|
||||
| --- | --- | --- | --- |
|
||||
| `compute.sources.discover` | 指标缺口、许可范围、已注册来源 | 来源候选、适用指标、采集建议 | 先找原始出处；新域名不直接进入正式定时采集 |
|
||||
| `compute.evidence.extract` | 原文分块、页码、表格、目标 schema | 带证据定位的字段候选 | 未出现的数值返回缺失；原文中的指令只是待分析内容 |
|
||||
| `compute.evidence.review` | 候选、对照记录、单位与实体规则 | 冲突类型、支持/反驳证据、待核验项 | 第二个 AI 的同意不是独立事实证明 |
|
||||
| `compute.gaps.prioritize` | 缺失、来源健康、参数敏感性、采集成本 | 下一步补证清单 | 优先可能影响结论且可查证的缺口，不以国家倾向排序 |
|
||||
| `compute.change.explain` | 已计算的贡献账单、事实版本 | 有引用的可读解释 | 不能修改计算结果；事实、估计、假设分别表达 |
|
||||
| `compute.model.propose` | 误差报告、数据质量问题、模型配置 | 有边界的参数/映射/新特征提案 | 必须说明可证伪假设、预期改善、影响范围和回退方案 |
|
||||
| `compute.model.review` | 候选方案、独立评估结果、影响报告 | 发布建议与限制 | 无权改变测试标签、门槛、保留集或自己批准自己 |
|
||||
|
||||
结构化结果由后端 Pydantic schema 校验。当前 AI 返回以文本为主，需要新增结果解析与受限重试；JSON 不合格、引用不匹配、单位冲突时运行失败或保留候选，不能将自然语言当作正式数据。
|
||||
|
||||
### 7.2 工具与权限
|
||||
|
||||
允许工具包括：参数化只读指标查询、已注册来源搜索/抓取、文档段落读取、规则化单位转换、实体候选查询、固定估计器运行、创建缺口任务、保存提案、提交实验任务。
|
||||
|
||||
AI 不直接写正式分数、覆盖原始证据、执行任意 SQL/Python/Shell、修改发布门槛或扩大自己的工具权限。生成的采集映射必须通过回放样本后才保存；新增可执行采集器代码属于普通开发和发布流程，不在生产 Agent 内 `eval`。
|
||||
|
||||
网页抓取需校验协议、主机、解析后的目标地址及每次重定向，阻止访问内网、回环和元数据端点；使用专门外部抓取边界，不能把内部业务 API 和外部取证 URL 混用。AI 只得到所需公开证据或经授权的业务摘要，凭证由后端设置解析，不进入提示词、引用或日志。
|
||||
|
||||
### 7.3 用户能看见的 AI 结果
|
||||
|
||||
每份分析展示三部分：已验证事实、模型解释、仍缺哪些证据。每条数值和关键结论可展开来源。用户可以标记“实体合并错误”“单位/年份错误”“证据不支持”“假设不合理”，形成结构化反馈；收藏、点赞和是否喜欢国家排名不能成为事实标签。
|
||||
|
||||
“AI 发现数据不足”可以是成功结果。模型不可用时，确定性采集、计算、双图和地球继续工作，解释面板保留上次结果并标注版本与时间。
|
||||
|
||||
## 8. 数学模型如何自我迭代
|
||||
|
||||
### 8.1 自动迭代分成三类
|
||||
|
||||
| 层次 | 可以怎样改进 | 自动程度 |
|
||||
| --- | --- | --- |
|
||||
| 证据和数据 | 补充来源、修正字段、发现重复和过期、改进解析映射 | 已核准来源与已验证规则可自动运行；新语义及高影响冲突先进入候选 |
|
||||
| 可验证的估计参数 | 项目延期分布、特定硬件/任务效率、记录错误率 | 通过冻结评估、影子运行和预先配置的边界后，可以自动发布小范围参数更新 |
|
||||
| 评价定义 | H/O/G/R 权重、指标含义、基期、国家范围和价值取向 | 默认生成提案，由模型负责人确认版本；不是机器从不存在的“国家实力真值”中自动学习 |
|
||||
|
||||
此处是产品建议的默认发布策略，不是在本次计划工作中申请执行权限。未来可配置自动发布的参数白名单，但不能用“开启自动迭代”笼统授权所有定义变化。
|
||||
|
||||
### 8.2 迭代闭环
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[新事实/人工纠错/成熟预测结果] --> B[确定性误差与质量评估]
|
||||
B --> C[AI 提出可证伪改进]
|
||||
C --> D[生成候选配置与锁定实验]
|
||||
D --> E[按时间回测及国家分组评估]
|
||||
E --> F{是否通过预设门槛}
|
||||
F -->|否| G[记录失败原因并保留现行版本]
|
||||
F -->|是| H[影子运行：结果不改变正式排名]
|
||||
H --> I[按发布策略评审或受限自动提升]
|
||||
I --> J[发布不可变版本与变动桥接]
|
||||
J --> K[监测退化并可原子回滚]
|
||||
K --> B
|
||||
```
|
||||
|
||||
每次提案只改变一种主要因素，或者明确作为一个联合实验；不能同时换来源、标签、权重和评估方法后宣称单个参数有效。失败提案也保存，避免只展示成功实验。
|
||||
|
||||
提案记录至少包括：问题、证据 ID、基线版本、拟议配置差异、适用实体/任务、可检验目标、锁定数据截止时间、主要评估指标、停止条件和回滚版本。可以提出“将并网阶段纳入交付预测”，不能只提出“让某国得分更合理”。
|
||||
|
||||
### 8.3 真实反馈标签从哪里来
|
||||
|
||||
| 估计对象 | 可观察标签 | 评估方式 |
|
||||
| --- | --- | --- |
|
||||
| 文档字段提取 | 人工核对原文的数值、单位、年份、实体和引用位置 | 字段精确率/召回率、关键错误、引用支持率，按语言/文档类型分组 |
|
||||
| 实体合并 | 官方 ID、项目连续记录或人工确认的同一/不同实体 | 错误合并率和遗漏合并率；高影响错误单列 |
|
||||
| 项目按期投产 | 后续正式验收/投产/取消记录，保留观测截止 | Brier score、可靠性图；右删失与失访不能计作失败 |
|
||||
| 新增容量 | 未来实际确认的设备与容量，具有同口径 | 绝对误差、对数误差；同时报告项目等权与容量加权结果 |
|
||||
| 任务性能系数 | 未参与拟合的可复现实测配置 | 相同任务质量下的预测误差与区间覆盖 |
|
||||
| 区间预测 | 后续实际值是否落入范围及区间宽度 | 覆盖率与区间评分一起评价，避免靠无限放宽范围“提高准确率” |
|
||||
|
||||
不使用 AI 自己生成的摘要作事实标签，不用现行总分拟合下一版总分，不把另一个复合指数直接当国家实力真值。完整创新指数可做外部一致性讨论，但若其包含输入指标或与 O 重叠，就不能作为独立验证集。
|
||||
|
||||
### 8.4 回测与数据泄漏控制
|
||||
|
||||
预测训练、验证和保留集按真实可获得时间滚动切分，同一项目的后续阶段、同一公告转载和同一扩建链必须归组，避免分散进训练和测试。每个特征都要求 `published_at <= forecast_cutoff`；仅今天抓到而无法证明当时可得的资料，不参加“当时可知”回测。
|
||||
|
||||
新模型先与简单基线比较：维持上次状态、沿用公告日期、同阶段历史交付中位数。不得只与一个刻意弱的旧模型比较。参数、标准化锚点和来源质量估计只从训练窗口取得。
|
||||
|
||||
大模型的训练记忆可能知道后来结果，所以历史回测中的预测器必须是只读取冻结特征的确定性估计器；不能让 LLM 凭记忆补旧事实。提案生成与真实保留集隔离，实际前瞻影子运行仍是必要证据。
|
||||
|
||||
按国家、硬件代际、项目规模、来源语言分别评估，并按项目/运营方/时间块进行重采样，不能把高度相关的重复记录当独立样本。小样本报告区间与不足，不为满足仪表盘而宣称显著改进。
|
||||
|
||||
### 8.5 建议发布门槛
|
||||
|
||||
以下是实施阶段的初始门槛草案，需在 M0 用试点数据冻结。它们是最低操作条件，不保证统计有效性，AI 无权自行降低。
|
||||
|
||||
| 门槛 | 初始建议 |
|
||||
| --- | --- |
|
||||
| 硬正确性 | 单位、时间穿越、重复计数、采集失败保留正式值等必测案例全部通过 |
|
||||
| 提取评估 | 至少 200 条人工核验字段、覆盖中英文及主要文档类型；报告精确率/召回率及区间；关键数值/单位错误不得进入自动采用集 |
|
||||
| 预测参数自动提升资格 | 至少 50 个有可验证结果的独立项目、覆盖至少 3 个滚动窗口；每个主要参与方至少 10 个，不足则继续影子运行或人工评估 |
|
||||
| 改善幅度 | 预先指定主要损失相对现行版和简单基线至少改善 5%；按依赖结构重采样的损失差区间须支持改善 |
|
||||
| 分组保护 | 主要参与方损失退化不超过预先约定范围，初始建议 2%；不能以平均改善掩盖一个国家明显恶化 |
|
||||
| 区间质量 | 不仅检查覆盖，还检查宽度与评分；对无法量化的来源覆盖偏差单列说明 |
|
||||
| 影子观察 | 至少连续 4 周稳定运行，并达到所需成熟标签数；12 个月预测不因观察满 4 周就算验证完成 |
|
||||
| 试验节制 | 初始每月最多 3 个主要候选，锁定真实保留集；多次搜索造成的选择偏差要通过新的前瞻样本再次检查 |
|
||||
|
||||
标签暂时不足时，系统可以自动补数、解释、积累预测与结果，但不发布“自我学习后的更准模型”。这不会阻塞第一版事实产品上线。
|
||||
|
||||
### 8.6 一个完整迭代例子
|
||||
|
||||
以下是流程示例，不是已发生的项目事实或评估成绩。
|
||||
|
||||
系统发现一批已订购芯片的项目未如期投产；确定性评估记录预测误差。AI 查询证据后提出：现有交付模型只使用芯片到货时间，遗漏电力接入阶段。它提交一个新增“并网是否完成”特征的候选。
|
||||
|
||||
后端按项目原始发表时间构造训练数据，用固定估计器拟合；锁定保留集检验总体和中美分组误差。即便论文或第二个 AI 都认为这个想法合理，只要结果未通过门槛就不发布。通过后先影子运行；正式发布时展示受影响的 G 分、预计容量和模型版本变化，已投产 H 不因推测而下降。
|
||||
|
||||
若后续发现并网字段提取错误导致退化，回滚活动模型指针，保存新证据和失败实验。不能删除失败记录或改写过去已发布的预测。
|
||||
|
||||
## 9. 后端数据与实现结构
|
||||
|
||||
首版采用最小领域实现：沿用 `CollectedData`、`DataSnapshot` 和位置维表，新增经过 schema 校验的国家指标记录与聚合服务。数值口径、参考期、发表时间、来源链接、原文哈希和证据定位可先作为受约束的元数据保存；必要的原件存放在许可允许的运行数据目录。只有现有表确实无法表达查询或历史约束时才增加专用表。
|
||||
|
||||
先冻结面向专题的 API 契约，地球与图表只消费该契约。将来拆出指标观测、证据、实验等表时，通过服务层迁移，不让前端依赖临时 JSON 或物理表结构。首版不以第 9.1 节全部表建完作为开工条件。
|
||||
|
||||
### 9.1 拟议持久化对象
|
||||
|
||||
下表为待新增的逻辑对象，可按最终数据库设计合并有限的小表；不要与现有采集、位置和任务状态形成并行真相源。
|
||||
|
||||
| 对象 | 保存内容 | 关联和约束 |
|
||||
| --- | --- | --- |
|
||||
| `evidence_documents` | 原文存储引用、原始/提取哈希、来源、发表/抓取时间、许可、提取版本 | 内容不可变；全文不在高频列表接口返回 |
|
||||
| `metric_definitions` | 粒度、单位、精度、工作负载、归属、方向、适用范围和版本 | 一个稳定 metric ID；模型版本引用具体定义版本 |
|
||||
| `metric_observations` | 数值/范围、参考期、知识时间、状态、来源版本、取整和缺失原因 | 链接原采集记录；追加式修订，不复制整份原始 JSON |
|
||||
| `observation_evidence` | 观测与证据的支持/反驳关系、页号、表格/段落、短摘录 | 多对多；来源家族避免转载重复投票 |
|
||||
| `compute_entities` / `resource_relations` | 稳定实体、别名、分期关系、所有权/供应/连接、有效期 | 复用现有位置维表；不另存一套会漂移的正式坐标 |
|
||||
| `model_versions` | 指标集合、权重、锚点、参数、代码版本、情景与发布状态 | 不可变版本；活动指针独立且原子切换 |
|
||||
| `model_runs` / `model_results` | 输入清单哈希、版本、截止时点、随机种子、环境、国家结果、分项和不确定性 | 结果可重放；实验、情景与正式运行分开 |
|
||||
| `model_proposals` / `model_evaluations` | 配置差异、假设、冻结评估方案、损失、分组结果、决定与回退目标 | 发布必须引用通过的评估，保存失败实验 |
|
||||
| `agent_runs` | task key、提示词有效哈希、模型、工具调用、引用、预算、结构化输出、终态 | 链接现有任务账本，Agent 自身不拥有另一套调度状态 |
|
||||
|
||||
重放保留输入观测 ID/版本清单或不可变清单文件，不能只保存一段 SQL 然后查询已被更改的数据。记录计算实现版本、依赖环境和随机种子；LLM 输出保存为证据，不要求重新调用模型才能复算数值。
|
||||
|
||||
历史删除与保留策略需要显式区分：普通缓存可清理；被正式模型版本引用的观测及许可允许保存的证据受保留保护。来源撤回或许可变更时标记可访问状态并保留允许保留的哈希和元数据，不强行公开原文。
|
||||
|
||||
### 9.2 待新增模块与现有边界
|
||||
|
||||
| 拟议模块 | 职责 |
|
||||
| --- | --- |
|
||||
| `backend/app/services/compute_intelligence/observations.py` | 规范化、有效观测选择和数据缺口 |
|
||||
| `.../metrics.py`、`.../scoring.py` | 纯数值计算、单位/指标注册、标准化和总分 |
|
||||
| `.../forecasting.py`、`.../evaluation.py` | 固定估计器、时间切分、基线、分组指标与实验 |
|
||||
| `.../workflows.py`、`.../proposals.py` | 固定 AI 工作流、结构化候选、预算和发布规则 |
|
||||
| `.../publication.py` | 正式快照、变动桥接、活动版本和 outbox |
|
||||
| `backend/app/api/v1/compute_intelligence.py` | 参数化只读查询和后台任务入口 |
|
||||
| `frontend/public/earth/js/compute-intelligence.js` | 专题 UI 状态、图表与现有图层联动，消费后端计算结果 |
|
||||
| 控制台专题页与现有采集/AI 设置页签 | 证据核验、缺口、模型实验、版本回看 |
|
||||
|
||||
具体文件在实现时保持单一职责，优先复用已有服务，不为目录结构创建空文件。新来源继续注册到现有采集器体系及 datasource 配置；新提示词放入现有 `default_prompts.json`，不散落在 renderer 或 `aiprovider`。
|
||||
|
||||
### 9.3 拟议 API
|
||||
|
||||
下列接口尚不存在，名称在 M0 与当前路由约定核对后冻结。
|
||||
|
||||
| 方法与路径（统一前缀 `/api/v1/compute-intelligence`） | 返回内容 |
|
||||
| --- | --- |
|
||||
| `GET /overview` | 所选参与方、口径、参考期与知识截止下的结果、适用范围及覆盖说明 |
|
||||
| `GET /series` | 指定指标的有界历史序列，含缺失、版本、实际观测时间与来源引用 |
|
||||
| `GET /entities/{id}` | 设施、分期、资源关系和历史观测 |
|
||||
| `GET /changes` | 数据或模型贡献账单，数据库侧分页 |
|
||||
| `GET /evidence/{id}` | 根据权限和许可返回证据摘要、定位与允许展示的内容 |
|
||||
| `GET /models`、`GET /runs/{id}` | 模型版本、输入清单、计算结果与评估摘要 |
|
||||
| `POST /recompute`、`POST /agents/run` | 建立任务并立即返回任务 ID，允许取消和查看进度 |
|
||||
| `POST /proposals/{id}/evaluate` | 创建冻结数据与方案的评估任务 |
|
||||
| `POST /models/{id}/promote`、`POST /models/{id}/rollback` | 按角色及发布策略原子切换活动版本，记录理由和影响 |
|
||||
|
||||
查询参数固定包含 `metric_profile`、`entities`、`reference_at`、`knowledge_cutoff`、`model_version` 和可选 `scenario_id`。前端不接收任意表达式或 SQL，不自行补算另一套总分。
|
||||
|
||||
### 9.4 发布与性能
|
||||
|
||||
后台完整计算并通过质量门槛后，在事务中发布结果快照和活动指针，通过既有 outbox 唤醒刷新。新 adapter 显式声明国家面板/历史图的刷新范围；设施变化才触发相应 `computeCenters` 更新。草稿证据、失败实验和影子运行不改变默认地球。
|
||||
|
||||
缓存键包含口径、实体集合、参考期、知识截止、数据清单哈希、模型与情景版本。WebSocket 只通知结果版本与必要增量，客户端拒绝较旧响应；不能每次国家 hover 都重算模型或执行 LLM。
|
||||
|
||||
聚合、过滤、排序和分页在数据库完成；预计算国家年度/月度可用序列，地图按分辨率聚合节点。时间滑动去抖并复用已取快照，播放使用相邻状态差量。建议试点验收目标为缓存查询 P95 小于 1 秒、切换已缓存时点小于 500 毫秒，声明基准机器、数据规模和网络条件；未实测前不作为已达到性能。
|
||||
|
||||
## 10. 刷新、成本和失败处理
|
||||
|
||||
| 工作 | 建议节奏 | 触发条件与成本控制 |
|
||||
| --- | --- | --- |
|
||||
| CSV/API 检查 | 日或来源允许的更慢频率 | ETag/Last-Modified/哈希去重，无内容变化不调用 AI |
|
||||
| 年报与指数发现 | 月度，发布季可提高 | 找到新版本再下载和提取，不把月检写成月度指标 |
|
||||
| 重点项目证据 | 日查、周复核 | 去重后仅高影响变化进入提取队列 |
|
||||
| 指标重算 | 有效观测或模型版本改变时 | 只重算受影响实体/时点/指标依赖 |
|
||||
| AI 解释 | 有意义的结果变化或用户请求 | 相同输入哈希复用，不因页面打开重复生成 |
|
||||
| 数据缺口排序 | 每周 | 敏感性、覆盖风险、可获得性和预计采集成本共同决定 |
|
||||
| 参数候选实验 | 月度或成熟标签达到门槛 | 限制候选数量；无足够标签不强制产出升级 |
|
||||
|
||||
试点默认每工作流最多 3 次搜索、8 个页面抓取、4 次 LLM 调用、1 次结构化修复重试，最多同时运行 2 个 AI 工作流；均为可配置预算，不是第三方接口能力承诺。按全局模型配置记录 token/费用，设置日预算和单任务截止时间，超额进入延后或停止状态，不无限自递归。
|
||||
|
||||
失败处理：采集失败保留上次正式值和新鲜度；AI 不可用仍提供确定性结果；引用失效保留许可允许的原件和哈希；长期资料缺口显示缺失并降低结论覆盖范围；新源冲突进入待核验；模型退化恢复前一活动版本。不能用重试或回滚删除已提交有效证据。
|
||||
|
||||
## 11. 分期实施与依赖
|
||||
|
||||
工作量是规划估算，按一名熟悉项目的全栈开发者、持续可用的数据核验支持和现有环境可用计算；不是交付承诺。开发时间不包括等待付费授权、原始报告或 12 个月预测结果成熟的时间。
|
||||
|
||||
### 11.1 当前优先交付：展示专题
|
||||
|
||||
| 顺序 | 修改方向 | 验收内容 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| A1 修复数据与最小合同 | Epoch CSV、TOP500 日期/身份/单位、空结果保护；保存历史指标和来源 | 能得到真实当前快照与核验后的年度序列，不要求宏观指标全到最新年 | 2—3 |
|
||||
| A2 国家对比与历史接口 | 国家汇总、公开样本与宏观统计分离、双图数据、证据摘要 | 绝对量/份额/创新分差有各自定义；缺失与时点可见 | 2—3 |
|
||||
| A3 地球与图表展示 | 专题模式、国家面板、双图、时间联动、设施/证据详情 | 完成第 0 节完整浏览路径，并通过桌面/移动/缩放验证 | 3—5 |
|
||||
| A4 可选 AI 阅读辅助 | 复用现有 client 和任务提示词，针对冻结快照生成带引用说明 | 事实与推断分开；可关闭；失败不影响地球与图表 | 1—2 |
|
||||
|
||||
A1—A3 是当前必须交付的闭环，约 7—11 个开发工作日；A4 可在闭环稳定后增加。实际数据源不通、现有采集回归问题或主题布局复杂度会改变估算。实现时首先验证 A1，而不是先开发通用 Agent 框架或完整综合评分。
|
||||
|
||||
### 11.2 后续平台能力路线
|
||||
|
||||
下表是完整能力建设的工作包估算，与 A 路线存在复用和重叠，不应重复相加。A 路线完成后先盘点已具备的 M0—M2 能力，只实施剩余部分。
|
||||
|
||||
| 阶段 | 工作包 | 交付物与通过条件 | 估计开发工作日 |
|
||||
| --- | --- | --- | ---: |
|
||||
| M0 合同与试点 | 固定口径、成员、来源许可、历史时间语义、角色权限、质量/评估门槛 | 指标注册表、数据源接入清单、黄金核验样本；所有未知明确登记 | 3—5 |
|
||||
| M1 可信数据 | 修复 Epoch/TOP500；证据与指标观测；导入两组历史；稳定身份与失败保护 | 可查询、可追溯、可重放的历史 API；无示例数据进入正式结果 | 7—10 |
|
||||
| M2 可见产品 | 地球专题、双图、国家对比、变化账单、证据抽屉和控制台缺口页 | 中美历史回放闭环；移动/缩放和旧图层回归通过；综合分可保持未就绪 | 7—10 |
|
||||
| M3 资源与潜力 | 芯片、实测、项目、电力/资金台账，场景与分项估计 | H 与可用 G/R 子项具备来源；O 仅在分项齐备后发布；总分门槛满足才开放 | 8—12 |
|
||||
| M4 AI 工作流 | 固定提取/核验/补证/解释流程、反馈和预算；扩展可靠任务类型 | AI 只能产生有引用候选与解释，失败不会污染正式数据 | 7—10 |
|
||||
| M5 模型迭代 | 真实标签、时间回测、候选评估、影子运行、发布桥接与回滚 | 能证明某个可观察估计目标改善；不要求每轮必有新模型 | 8—12 |
|
||||
|
||||
从零完整建设上述广义平台能力约 40—59 个开发工作日，包含比 A 路线更完整的证据和实验治理；它不是当前展示专题的工期。影子期至少四周，但不成熟的预测标签会延长自动提升资格等待期。
|
||||
|
||||
依赖顺序：M0 → M1 → M2；M3 的项目台账应尽早开始积累标签；M4 依赖 M1 的可信证据结构；M5 依赖 M3 的标签和 M4 的审计工作流。多方、矿产和复杂供应链在 M2 之后按数据质量逐个扩展，不等全部主题齐备再上线。
|
||||
|
||||
## 12. 验收与测试矩阵
|
||||
|
||||
| 范围 | 必测案例 | 通过标准 |
|
||||
| --- | --- | --- |
|
||||
| 来源真实性 | HTML 改版、空 CSV、404、部分下载、示例字符串 | 正式数据不被示例或空结果替换；错误原因可见 |
|
||||
| 单位与精度 | T/P/E 换算，FP64/FP16/FP8，稀疏/稠密，MW/TWh，FLOP/FLOP/s | 不同口径拒绝合并；可比转换有可复算结果 |
|
||||
| 时间与版本 | 固定抓取日误作投产日、2026 报告回填 2023、未知发布日期 | 严格截止查询无未来信息；历史回顾明确重述 |
|
||||
| 实体与去重 | 排名换位、扩建、名称变化、匿名化、重复转载、跨源同集群 | 身份稳定；不重复计数；匿名或缺榜不自动退役 |
|
||||
| 统计与地理 | 缺坐标、未知国家、欧盟与成员国、跨境所有权 | 国家统计不依赖坐标；分母不重叠；归属视图不混用 |
|
||||
| 两张历史图 | 已核验年份值、缺失年份、2026 待核实主张 | 只展示有证据的观测；不能伪造平直历史或默认预测 |
|
||||
| 综合评分 | 缺 O/G/R、权重和不为 1、标准化越界、国家集合变化 | 非法配置拒绝；缺失总分为 null；固定版本不静默漂移 |
|
||||
| 变化账单 | 新投产、补录、源修订、换模型 | 分项贡献之和满足计算容差；类型和版本差异可解释 |
|
||||
| AI 输出 | 格式错误、虚构引用、原文含指令、模型超时、重复任务 | 输出隔离；工具边界有效；任务幂等并能取消 |
|
||||
| 模型评估 | 标签穿越、同项目跨集、过拟合小样本、分组退化 | 不通过门槛不发布；保留失败实验 |
|
||||
| 发布与回滚 | 两个并发发布、计算中断、旧 WS/HTTP 响应 | 原子版本；半成品不可见；客户端不回滚到旧时点 |
|
||||
| 地球渲染 | 远距/近距、时间播放、层隐藏、节点锁定、场景退出 | 无闪烁和悬空 tooltip；实例复用，不每帧重新建纹理 |
|
||||
| UI | 桌面、移动、125%/150% 缩放、键盘、无数据/低置信/加载状态 | 一屏工作区无意外全局滚动；关键标签不靠颜色区分 |
|
||||
|
||||
实现阶段按变更执行后端精确测试、`scripts/harness/quick-check.sh`、Bun 前端构建和渲染 smoke；地球改变不能只凭构建通过。保持现有公共页面、鉴权、管理路由、安全导航及其他地球图层的回归证据。性能阈值用记录过的基准规模验证，不能把开发空数据测试称为生产验收。
|
||||
|
||||
## 13. 文档、发布与完成定义
|
||||
|
||||
每阶段 PR 和部署使用已有项目流程，功能旗标分别控制专题 UI、估计分项、AI 工作流与自动参数提升。先开放读取与事实浏览，再启用模型实验；关闭新旗标应回到原有算力图层而不丢历史数据。
|
||||
|
||||
发生真实用户工作流变化时更新中英文手册与快速入门；采集、AI、作业与位置变化更新相应后端技术文档；控制台、Earth、图层顺序和样式变化更新对应 context 与规范。新增公共技术文档需要同步中英文和 Docs 注册；本文件是内部中文计划，不注册为已实现的公共手册。
|
||||
|
||||
当前展示专题完成定义：用户完成第 0 节浏览路径，可以从国家指标变化回到出处和所用计算版本；重新计算得到同一数值；知道哪些资料尚缺。两张图应和国家、设施、时间、证据联动，不能只是贴在地球旁的静态图片。
|
||||
|
||||
后续决策辅助与模型迭代完成定义：情景的假设与结果可复算；AI 的一次成功改进有独立验证目标、评估记录和可回滚版本。只有一段 AI 分析或未经校验的总分，不代表完成决策辅助能力。
|
||||
|
||||
## 14. 风险与预先决定的退路
|
||||
|
||||
| 风险 | 产品和实施处理 |
|
||||
| --- | --- |
|
||||
| 无法取得创新原报告分项 | 保留完整指数独立观测；O 和总分不发布，不伪造拆分 |
|
||||
| 中国或其他地区公开设施更少 | 显示宏观统计与设施样本两个覆盖视图，扩充本地语言原始来源，不把保密或匿名化当能力下降 |
|
||||
| 硬件真实利用率不可知 | 理论量先行，明确任务和区间；不把实验室 benchmark 当生产实测 |
|
||||
| 来源改版、匿名化或许可证变化 | 版本化下载、质量隔离、许可范围内保存证据，必要时停更单一来源 |
|
||||
| 资金、电力、芯片相互重复计分 | 固定指标归属与约束链；不把投入、存量、产出和事件各加一次 |
|
||||
| 训练标签不足或结果多年后才成熟 | 先提供情景与证据改进，延后参数自动提升资格 |
|
||||
| 模型迎合既定国家排名 | 预先定义目标和门槛，评价权重单独评审，不以用户喜好或另一个 AI 打分训练 |
|
||||
| 已有采集/定位状态被新服务覆盖 | 复用当前真相源，使用明确字段所有者、发布快照和回归测试 |
|
||||
|
||||
## 15. 来源与相关文档
|
||||
|
||||
以下链接支持数据口径、来源可得性和架构选择；除明确标注的历史值外，本计划的字段、频率、权重、工期和发布门槛均是拟议设计。
|
||||
|
||||
- [信通院 2021 年版算力白皮书（镜像）](https://pdf.dfcfw.com/pdf/H3_AP202109271518811781_1.pdf)：2020 年总算力份额。
|
||||
- [信通院 2022 年版白皮书（镜像）](https://13115299.s21i.faiusr.com/61/1/ABUIABA9GAAgvq2fmwYozLrrpwU.pdf)：正文第 13—14 页，2021 年份额与口径。
|
||||
- [信通院 2023 年版白皮书（镜像）](https://www.ahchanye.com/wp-content/uploads/2023/09/2023092815342684.pdf)：正文第 13 页的 2022 年份额，以及 302 EFlops 总规模。
|
||||
- [信通院 2024 年版蓝皮书（镜像）](https://pdf.dfcfw.com/pdf/H3_AP202502051642799257_1.pdf)：正文第 10 页的 2023 年份额及后续规模说明。
|
||||
- [科技日报关于创新指数报告的报道（新浪转载）](https://finance.sina.com.cn/tech/roll/2026-07-17/doc-iniickaz8224252.shtml)：分差与综合指标维度；未替代原报告分项。
|
||||
- [2026 年 42%/33% 候选出处](https://www.mornai.cn/news/gpu/ai-computing-power-never-sleeps/)：待核实商业文章，不作为正式统计。
|
||||
- [Epoch GPU 集群公开数据](https://epoch.ai/data/gpu-clusters)、[字段与下载说明](https://epoch.ai/data/gpu-clusters-documentation)：CSV、地理字段、覆盖与公开版本限制。
|
||||
- [Epoch 芯片所有权](https://epoch.ai/data/ai-chip-owners)：所有权不等于使用权或所在地，交付与投产有时间差。
|
||||
- [MLPerf Training](https://mlcommons.org/benchmarks/training/)、[MLPerf Inference Datacenter](https://mlcommons.org/benchmarks/inference-datacenter/)：固定任务、质量及测试配置。
|
||||
- [IEA Energy and AI](https://www.iea.org/reports/energy-and-ai)、[EIA 开放数据](https://www.eia.gov/opendata/)：能源资料入口，不能直接替代设施并网证据。
|
||||
- [SEC EDGAR API 说明](https://www.sec.gov/search-filings/edgar-application-programming-interfaces)：公司提交与财务数据访问；需注意财季与标签口径。
|
||||
- [USGS 矿产概要](https://www.usgs.gov/centers/national-minerals-information-center/mineral-commodity-summaries)、[UN Comtrade 门户](https://comtradeplus.un.org/)：矿产与贸易候选数据源。
|
||||
- [OpenAlex 帮助与数据说明](https://help.openalex.org/)：研究元数据候选来源；接入时核验访问与归属规则。
|
||||
- [业务架构与数据流转](../technical/zh/platform-data-flows.md)、[后端采集器](../technical/zh/backend-collectors.md)、[AI Provider 边界](../technical/zh/agents-aiprovider.md)。
|
||||
- [智能星球前端上下文](../technical/zh/earth-frontend-context.md)、[渲染图层顺序](../technical/zh/earth-render-layer-order.md)、[图层视觉规范](../technical/zh/earth-layer-style-reference.md)。
|
||||
- [轻量 Agent 编排](agents-light-orchestrator-websearch-plan.md)、[态势感知基础计划](agents-situational-awareness-foundation-plan.md)：复用总体方向，不重复建设 provider 内业务编排。
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
**状态**：待实施
|
||||
**创建日期**：2026-05-12
|
||||
**核心目标**：把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册，把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`，并把分层规则写进 `documentation-coverage-rules.md` 和 `.claude/commands/docs.md`，让以后写文档时自动按受众归档。
|
||||
**校正日期**：2026-06-26，控制台深链已从旧 tab 查询口径更新为当前 `?section=` 口径。
|
||||
**核心目标**：把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册，把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`，并把分层规则写进 `documentation-coverage-rules.md` 和 `.codex/skills/docs/SKILL.md`，让以后写文档时自动按受众归档。
|
||||
|
||||
## 背景
|
||||
|
||||
@@ -31,12 +32,12 @@
|
||||
3. **登录与找回密码** — 登录页、忘记密码流程
|
||||
4. **账户设置** — 修改密码、修改邮箱（需重新验证）、查看权限组、登出
|
||||
5. **Console 总览** — 左侧菜单结构、各路由用途
|
||||
6. **配置数据采集器** — `/collection-management?tab=collector_credentials`：选择 collector、连接测试、保存凭证；BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?tab=providers`：默认 provider、模型、Base URL、API Key、本地代理；工具 tab（WebSearch、OCR）
|
||||
6. **配置数据采集器** — `/collection-management?section=collector_credentials`：选择 collector、连接测试、保存凭证；BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?section=integrations`：默认 provider、模型、Base URL、API Key、本地代理；工具 section（WebSearch、OCR）位于 `/ai?section=tools`
|
||||
8. **系统设置** — `/settings` 其他子 tab（系统设置、电视直播源、SMTP 邮件）
|
||||
9. **用户管理（管理员）** — `/users`：创建、删除、改角色、Gatekeeper 权限组
|
||||
10. **数据探索** — `/datasources`、`/data`、`/bgp`、`/alerts/*`
|
||||
11. **AI 测试台** — `/ai?tab=playground`
|
||||
11. **AI 测试台** — `/ai?section=playground`
|
||||
12. **Earth 公开页面** — 现 manual.md 的 Earth 章节原样保留（图层、图例、搜索、位置候选、设置、视角、动捕、巡航、移动端）
|
||||
13. **Docs 文档站** — 当前 Docs 章节保留（权限组说明）
|
||||
|
||||
@@ -48,7 +49,7 @@
|
||||
|
||||
- 打开管理员给你的 URL
|
||||
- 注册账号 + 邮箱验证
|
||||
- 登录后第一次做什么（建议先到 `/collection-management?tab=collector_credentials` 配一个 collector，再到 `/ai` 配模型）
|
||||
- 登录后第一次做什么（建议先到 `/collection-management?section=collector_credentials` 配一个 collector，再到 `/ai?section=integrations` 配模型）
|
||||
- 看 Earth
|
||||
|
||||
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节，**不**再单独出 `ops-quickstart.md`，避免新增维护点。
|
||||
@@ -79,7 +80,7 @@
|
||||
> - 新增客户可见 UI 流 → 同时更新 `manual.md` zh+en 与 `docs-content.ts`
|
||||
> - 新增 ops 命令或脚本 → 只更新 `ops-runbook.md` zh+en
|
||||
|
||||
## .claude/commands/docs.md 增量
|
||||
## `.codex/skills/docs/SKILL.md` 增量
|
||||
|
||||
在 "Step 2 — Decide Scope" 后插一段：
|
||||
|
||||
@@ -99,7 +100,7 @@
|
||||
- `docs/technical/zh/quickstart.md` & `en/quickstart.md` — 重写
|
||||
- `docs/technical/zh/ops-runbook.md` & `en/ops-runbook.md` *(新)*
|
||||
- `docs/documentation-coverage-rules.md` — 加受众分层段
|
||||
- `.claude/commands/docs.md` — 加 Document Audience Routing 段
|
||||
- `.codex/skills/docs/SKILL.md` — 加 Document Audience Routing 段
|
||||
- `frontend/src/pages/Docs/docs-content.ts` — 注册 `ops-runbook` 到 `DOCS_METADATA`（`docs_admin` 组）
|
||||
|
||||
## 依赖
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Earth Motion Capture Gesture Control Plan
|
||||
|
||||
> Update: Motion Agent process/device control, UE/Web shared command protocol, dual-camera redundant fusion, and the next 3D calibration route are now tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md). This document remains useful for the original provider split and gesture-control intent.
|
||||
|
||||
## Goal
|
||||
|
||||
为 Planet Earth 大屏和未来 3D 展示增加一套解耦的动作捕捉手势控制能力。实时输入分成两条路线：网页端可直接通过浏览器 `getUserMedia` 在本机识别；高级设备可继续使用本机 Motion Capture Edge Agent。两条路线都只输出轻量语义事件，客户端负责把“手势事件”映射到“具体交互函数”。
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
**状态**：已实现主体交互，并按实测调整。当前浏览器识别保留右手导航、头部切目标、左手上下切动捕图层、双手张开/收拢缩放；双手上举确认暂时关闭。Motion 目标展示已改为 `CruiseSequencer` + `PresentationController` 的 persistent 展示。
|
||||
|
||||
> Update: Agent-side bidirectional commands, UE/Web shared device control, dual-camera redundant fusion, and future calibrated 3D mode are tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
|
||||
|
||||
## Summary
|
||||
|
||||
把动捕从“几个单点手势触发函数”升级为一套更像大屏遥控器的交互层：右手负责地球导航，头部负责候选切换，左手上下切换动捕候选图层，双手负责缩放，调试面板支持“只显示骨骼”和暂停匹配。进入动捕模式后，Earth 自动软选中屏幕中心附近的正面可交互目标；确认动作预留为把目标升级为锁定，并用巡航/引导线式详情打开，不再模拟鼠标点击。
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user