Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cee1996809 | ||
|
|
58671e7bc3 | ||
|
|
a54fcdbeed | ||
|
|
1dd2921674 | ||
|
|
d30f7d08c5 | ||
|
|
5bdb55f3f1 | ||
|
|
fbecf30513 | ||
|
|
19d5ac0fee | ||
|
|
3265d22af5 | ||
|
|
899e3bce43 |
@@ -1,139 +0,0 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
优先用确定性的 CLI 检查缩小范围,不要一上来把完整文件或大 diff 读入上下文:
|
||||
|
||||
```bash
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
git diff --check
|
||||
rg -n "TODO|FIXME|console\.log|debugger|print\(" <changed-paths>
|
||||
```
|
||||
|
||||
只有 focused diff 不足以安全判断或修改时,才读取完整文件。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
先从 focused diff 开始:
|
||||
|
||||
```bash
|
||||
git diff --unified=0 HEAD -- <file>
|
||||
```
|
||||
|
||||
用 `rg`、`git diff --check`、编译器或 linter 输出确认确定性问题。只有需要上下文时才用 Read 读取完整文件。对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式。
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
- 优先做精确补丁;只有仓库已有对应格式化流程时,才运行格式化工具
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
@@ -1,104 +0,0 @@
|
||||
---
|
||||
description: Create or update repository documentation from current code changes
|
||||
argument-hint: Optional: topic to document, or leave empty to infer from git diff
|
||||
allowed-tools: ["Read", "Edit", "Write", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /docs — Documentation Workflow
|
||||
|
||||
## Goal
|
||||
|
||||
Create or update documentation that explains why a change exists, how it behaves, and what maintainers need to know. Keep this command generic. Repository-specific coverage rules live in the repository and must be loaded separately.
|
||||
|
||||
## Repository Rules
|
||||
|
||||
Before deciding scope, check whether the repository has a documentation rules file:
|
||||
|
||||
```bash
|
||||
test -f docs/documentation-coverage-rules.md && sed -n '1,240p' docs/documentation-coverage-rules.md
|
||||
```
|
||||
|
||||
If it exists, apply it as the project-specific coverage checklist. If it does not exist, continue with the generic workflow below.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Understand The Change
|
||||
|
||||
```bash
|
||||
git diff HEAD --stat
|
||||
git diff HEAD --name-only
|
||||
git log --oneline -10
|
||||
rg --files docs
|
||||
```
|
||||
|
||||
If `$ARGUMENTS` specifies a topic, focus on that topic. Otherwise infer the documentation topic from the changed files. Do not read the full repository diff by default; inspect focused files only:
|
||||
|
||||
```bash
|
||||
git diff HEAD -- <path>
|
||||
rg -n "class |def |function |export |router|@router|interface |type " <path>
|
||||
```
|
||||
|
||||
### Step 2 — Decide Scope
|
||||
|
||||
- Prefer updating an existing relevant document over creating a duplicate.
|
||||
- Use one document for one coherent topic.
|
||||
- Split documents only when the change crosses meaningful domains.
|
||||
- Keep filenames lowercase and hyphenated.
|
||||
- Apply the repository-specific rules file before writing.
|
||||
|
||||
#### Document Audience Routing (Planet)
|
||||
|
||||
In this repository, classify the action's performer before picking a target file:
|
||||
|
||||
- Browser/UI end user → `docs/technical/{zh,en}/manual.md` or `quickstart.md`.
|
||||
- Shell / Docker / log paths / `planet.sh` / SMTP fallbacks / port forwarding → `docs/technical/{zh,en}/ops-runbook.md` (or an existing `ops-*.md`).
|
||||
- Second-party developers → existing `*-context.md` / `backend-*.md` / `earth-*.md` files.
|
||||
|
||||
Never put shell commands, log paths, or Docker operations into `manual.md` / `quickstart.md`. Never put UI button labels or screenshots into `ops-*.md`. When the same action has both a UI and a CLI path, write each in its own home and cross-link them with one sentence.
|
||||
|
||||
For ambiguous or large documentation changes, briefly state the intended doc plan before editing. For clear small changes, proceed directly.
|
||||
|
||||
### Step 3 — Write
|
||||
|
||||
Explain:
|
||||
|
||||
- Background/problem: what was wrong or missing before.
|
||||
- Core design decisions and rationale.
|
||||
- Operational or user-facing impact.
|
||||
- Relevant code paths, only when useful for future maintainers.
|
||||
|
||||
Style:
|
||||
|
||||
- Follow the repository’s existing language and heading conventions.
|
||||
- Use fenced code blocks with language tags.
|
||||
- Prefer tables for comparisons or parameter lists.
|
||||
- Keep snippets concise and relevant.
|
||||
- For UI labels, chart labels, feature names, datasource names, and other terms that may become mixed Chinese/English copy, check `docs/technical/{zh,en}/naming-glossary.md` and use the documented display name. If a confusing term is missing, update the glossary in both languages as part of the docs change.
|
||||
|
||||
### Step 4 — Verify
|
||||
|
||||
- Read the completed docs once for clarity and stale statements.
|
||||
- Verify referenced paths exist with `test -e` or `rg --files`.
|
||||
- Run applicable checks from `docs/documentation-coverage-rules.md`.
|
||||
- Check Markdown links use readable user-facing titles unless repository rules allow otherwise.
|
||||
|
||||
### Step 5 — Report
|
||||
|
||||
Summarize changed docs and verification:
|
||||
|
||||
```md
|
||||
Updated:
|
||||
- path/to/doc.md — what changed
|
||||
|
||||
Verified:
|
||||
- checks that passed
|
||||
- checks that could not be run, if any
|
||||
```
|
||||
|
||||
## Hard Constraints
|
||||
|
||||
- Do not leave placeholder docs.
|
||||
- Do not duplicate bilingual files byte-for-byte.
|
||||
- Do not reference PR numbers, issue numbers, or the current conversation unless explicitly requested.
|
||||
- Do not write changelog-style lists without the reasoning and tradeoffs behind the change.
|
||||
- Keep docs maintainable and concise.
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 优先使用确定性工具证据:`rg`、`git diff --stat`、`git diff -- <path>`、测试、构建、lint、`curl`、数据库查询等能直接证明成功标准的方式
|
||||
- 不把大段命令输出粘进回复;保留在工具调用里,回复只总结关键证据
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
@@ -1,160 +0,0 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 节省上下文规则
|
||||
|
||||
发版判断应以确定性 CLI 证据为主,优先使用紧凑命令和定点读取:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
rg -n "version|^## |^Released:|当前开发版本|current" VERSION frontend/package.json pyproject.toml docs/CHANGELOG.md docs/version-history.md
|
||||
```
|
||||
|
||||
除非需要判断某个代码变更是否属于本次发版,否则不要读取完整 diff。
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据 `git diff --stat HEAD`、`git diff --name-only HEAD`、必要的 focused diff 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:先用 `git diff --name-only HEAD -- '*.py'` 列出,再运行 `python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:先用 `git diff --name-only HEAD -- frontend` 判断范围,再运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
cat VERSION
|
||||
rg -n "\"version\":|^version =|version = " frontend/package.json pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
BIN
.codex/screenshots/earth-i18n-current-page.png
Normal file
|
After Width: | Height: | Size: 1.2 MiB |
BIN
.codex/screenshots/earth-i18n-hd-texture-status.png
Normal file
|
After Width: | Height: | Size: 4.4 KiB |
BIN
.codex/screenshots/i18n-admin-data-sidebar.png
Normal file
|
After Width: | Height: | Size: 109 KiB |
BIN
.codex/screenshots/i18n-ai-tool-calls.png
Normal file
|
After Width: | Height: | Size: 124 KiB |
BIN
.codex/screenshots/i18n-earth-brand-config.png
Normal file
|
After Width: | Height: | Size: 170 KiB |
BIN
.codex/screenshots/i18n-earth-hud-brand.png
Normal file
|
After Width: | Height: | Size: 772 KiB |
BIN
.codex/screenshots/i18n-settings-notifications.png
Normal file
|
After Width: | Height: | Size: 95 KiB |
BIN
.codex/screenshots/i18n-settings-system.png
Normal file
|
After Width: | Height: | Size: 96 KiB |
BIN
.codex/screenshots/sidebar-left-align.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
5
.gitignore
vendored
@@ -25,7 +25,10 @@ __pycache__/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
downloads/*
|
||||
!downloads/usbipd-win/
|
||||
downloads/usbipd-win/*
|
||||
!downloads/usbipd-win/usbipd-win-5.3.0.msi
|
||||
eggs/
|
||||
.eggs/
|
||||
/lib/
|
||||
|
||||
180
AGENTS.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# AGENTS.md
|
||||
|
||||
**Planet agent harness. Defines behavior for coding agents working in this repository.**
|
||||
|
||||
---
|
||||
|
||||
## Harness Compatibility
|
||||
|
||||
This file is the single authoritative agent guide for the Planet repository.
|
||||
The older lowercase `agents.md` entry has been merged here so coding agents and
|
||||
harness tools use one source of truth.
|
||||
|
||||
### Source Of Truth
|
||||
|
||||
- `rules.md` is the mandatory repository rule source. Always load `core`,
|
||||
`security`, and `workflow`; load only task-relevant modules after that.
|
||||
- `AGENTS.md` defines the local agent operating mode and evidence gates.
|
||||
- `project_context.md` is background, not a rule source. Prefer newer
|
||||
implementation docs when it disagrees with current code.
|
||||
- `.codex/skills/` is the active specialized workflow layer for cleanup, docs,
|
||||
goal-driven work, and release.
|
||||
- Do not duplicate long workflow text across harness files. Durable constraints
|
||||
belong in `rules.md`; task procedures belong in skills or scripts.
|
||||
|
||||
Read these files before changing code:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. `project_context.md`
|
||||
4. `README.md`
|
||||
5. `docs/HARNESS.md`
|
||||
6. `CODEMAP.md`
|
||||
|
||||
For documentation work, also read `docs/documentation-coverage-rules.md`.
|
||||
|
||||
### Start Safely
|
||||
|
||||
Before broad edits:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Use focused context commands before reading large files:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
Preserve user changes already present in the worktree.
|
||||
|
||||
### Validation
|
||||
|
||||
Fast local harness validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/quick-check.sh
|
||||
```
|
||||
|
||||
Full local validation:
|
||||
|
||||
```bash
|
||||
scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
`validate.sh` includes quick checks, frontend Bun build, and frontend smoke
|
||||
unless disabled by its documented environment flags. Docker image smoke builds
|
||||
are intentionally opt-in:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Harness scripts resolve `bun`, `uv`, and optional delivery tools from the
|
||||
current non-interactive environment first. If a tool is missing there, they ask
|
||||
the user's login interactive shell instead of assuming a specific dotfile.
|
||||
|
||||
### High-Risk Areas
|
||||
|
||||
- `planet.sh` owns local lifecycle, ports, WSL/LAN behavior, and destructive
|
||||
`destroy` cleanup.
|
||||
- Frontend package management is Bun-only. Do not use npm, pnpm, or yarn.
|
||||
- Frontend changes must satisfy `scripts/harness/frontend-rules-check.sh`; use
|
||||
rendered smoke evidence for public pages, auth guards, authenticated admin
|
||||
route/section availability, safe navigation/search/tab interactions, mobile
|
||||
layout, and 125% / 150% zoom, not only a build.
|
||||
- Admin or Docs layout changes must load `rules.md` `uiux` and preserve the
|
||||
one-screen (`一屏` / `首屏`) height chain: route roots use `height: 100%`,
|
||||
intermediate wrappers keep `min-height: 0`, and only the intended child owns
|
||||
scrolling.
|
||||
- `aiprovider` is a protocol/provider adapter; keep business prompts and product
|
||||
workflows in the backend.
|
||||
- Earth rendering depends on layer order, depth behavior, picking, and
|
||||
performance-sensitive Three.js code.
|
||||
- Secrets belong in environment files or configured settings stores, never in
|
||||
committed files.
|
||||
- Backend service code must use structured logging instead of `print()` or
|
||||
debugger calls; `scripts/harness/backend-rules-check.sh` enforces this.
|
||||
|
||||
### Conflict Policy
|
||||
|
||||
Existing project rules and workflows win. If new harness guidance conflicts with
|
||||
`rules.md`, `AGENTS.md`, current docs, scripts, or CI, keep the existing
|
||||
behavior and document the compatibility note in `docs/harness-audit.md` or
|
||||
`docs/HARNESS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Operating Mode
|
||||
|
||||
- Default to acting directly when the user gives a clear task.
|
||||
- Ask before acting only when the missing decision is risky, cannot be
|
||||
discovered from repository context, and no conservative assumption is safe.
|
||||
- Read relevant files before editing.
|
||||
- Prefer focused CLI evidence: `rg`, `git diff --stat`, `git diff --name-only`,
|
||||
focused file reads, tests, builds, linters, and harness scripts.
|
||||
- Keep changes scoped to the requested area. Do not mix cleanup, feature work,
|
||||
release work, and documentation unless the task requires it.
|
||||
|
||||
---
|
||||
|
||||
## Evidence Gates
|
||||
|
||||
- Visual inputs are blocking evidence. If the user provides a screenshot, image,
|
||||
mock, browser capture, or visual reference, obtain evidence from the artifact
|
||||
before interpreting intent or editing code.
|
||||
- Path resolution is part of the task. If the path cannot be opened, first try
|
||||
reasonable local equivalents such as WSL/Windows path conversion,
|
||||
workspace-relative lookup, absolute paths, and attached-file locations.
|
||||
- Never guess from prompt text, filenames, previous context, logs, OCR, or
|
||||
memory when a visual artifact was provided but cannot be accessed.
|
||||
- OCR is acceptable evidence for text-only visual questions or non-multimodal
|
||||
environments; state that OCR was used as the fallback. Layout, color, spacing,
|
||||
pixel, and rendering issues need real visual inspection or a clear limitation
|
||||
note.
|
||||
- If a visual artifact still cannot be inspected, say so and pause that
|
||||
visual-dependent part of the work.
|
||||
- Claims of completion need evidence: a relevant test, build, lint, screenshot,
|
||||
diff, direct file check, or harness result.
|
||||
- For UI and rendering changes, verify the rendered result when local tooling
|
||||
allows it.
|
||||
|
||||
---
|
||||
|
||||
## Communication
|
||||
|
||||
- Match the user's language. Use Chinese for Chinese requests unless the user
|
||||
asks otherwise.
|
||||
- Keep updates short and specific: what is being inspected, edited, or verified.
|
||||
- Final responses should summarize changed files and verification, with blockers
|
||||
stated plainly.
|
||||
- Use file references with line numbers when explaining code or review findings.
|
||||
|
||||
---
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Prefer existing project patterns over new abstractions.
|
||||
- Remove stale branches, mocks, compatibility paths, and duplicated helpers once
|
||||
a stable path exists.
|
||||
- Centralize prompts, constants, defaults, and shared request/response handling.
|
||||
- Do not add secrets, generated runtime output, or local environment files.
|
||||
- Frontend commands use Bun only. Do not use `npm`, `pnpm`, or `yarn`.
|
||||
- Run the smallest relevant verification for the changed scope and report
|
||||
anything skipped.
|
||||
|
||||
---
|
||||
|
||||
## Prohibited
|
||||
|
||||
- Do not skip visual evidence handling when a visual artifact was provided.
|
||||
- Do not preserve obsolete harness files just because they already exist.
|
||||
- Do not invent behavior not present in code, docs, or verified external
|
||||
sources.
|
||||
- Do not rewrite unrelated files during cleanup.
|
||||
- Do not mark a task complete without checking concrete success criteria.
|
||||
109
CODEMAP.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Code Map
|
||||
|
||||
This map gives agents and maintainers a quick orientation without replacing the
|
||||
deeper architecture docs. Current implementation docs under `docs/technical/`
|
||||
are the source of detail for specific subsystems.
|
||||
|
||||
## Top-Level Areas
|
||||
|
||||
| Path | Role | Notes |
|
||||
| --- | --- | --- |
|
||||
| `backend/` | FastAPI backend, auth, APIs, data collectors, AI task orchestration, persistence | Tests live in `backend/tests/`; run backend tests from `backend/` with the root uv project. |
|
||||
| `frontend/` | React admin console, Docs UI, Web Earth shell, Vite build | Use Bun only. Public Earth assets live under `frontend/public/earth/`. |
|
||||
| `aiprovider/` | Model provider/protocol adapter service | Keep it free of product-specific prompts and workflows. |
|
||||
| `motion_agent/` | Motion capture protocol service used by `planet.sh` | Often dry-runs when cameras are unavailable, especially in WSL. |
|
||||
| `scripts/` | Utility scripts and harness wrappers | Harness commands live in `scripts/harness/`. |
|
||||
| `docs/` | Plans, technical docs, changelog, harness docs | Public technical docs are explicitly registered by the frontend Docs catalog. |
|
||||
| `deploy/helm/planet/` | Helm chart for staging/deployment smoke paths | CI runs helm lint/template when delivery checks are available. |
|
||||
| `.gitea/workflows/` | CI, release image build, staging deploy workflows | This repository uses Gitea workflow files, not `.github/workflows/`. |
|
||||
| `planet.sh` | Main local lifecycle script | Owns init/start/restart/stop/health/log/createuser/destroy. |
|
||||
|
||||
## Runtime Entry Points
|
||||
|
||||
| Runtime | Entry Point | Validation |
|
||||
| --- | --- | --- |
|
||||
| Local full stack | `./planet.sh start` | `./planet.sh health` |
|
||||
| Backend API | `backend/app/main.py` | `cd backend && uv run --frozen --group dev --project .. python -m pytest -q` |
|
||||
| Frontend app | `frontend/src/main.tsx` and `frontend/vite.config.mts` | `cd frontend && bun run build` |
|
||||
| AI Provider | `aiprovider/main.py` | `curl http://localhost:8010/health` after startup |
|
||||
| Motion Agent | `python -m motion_agent` via `planet.sh` | `./planet.sh health` or dry-run startup |
|
||||
| Docs UI | `frontend/src/pages/Docs/` | Docs catalog metadata plus frontend build |
|
||||
|
||||
## Ownership Boundaries
|
||||
|
||||
- Backend owns business state, auth, evidence collection, prompt selection, AI
|
||||
task orchestration, and database persistence.
|
||||
- `aiprovider` owns provider identity, request adapter style, model gateway
|
||||
retries, and health/status endpoints only.
|
||||
- Frontend owns operator workflows, Docs presentation, Web Earth orchestration,
|
||||
and client-side state that mirrors backend truth.
|
||||
- Web Earth rendering changes must preserve documented layer order, altitude
|
||||
offsets, picking behavior, legend semantics, and performance constraints.
|
||||
- `planet.sh` owns local environment bootstrap and service lifecycle. Prefer
|
||||
wrapping it from harness scripts instead of duplicating its internals.
|
||||
- Harness scripts source `scripts/harness/lib.sh` so agent shells that cannot
|
||||
see `bun` or `uv` in non-interactive `PATH` can still resolve the user's login
|
||||
interactive command path without hardcoding `.zshrc`.
|
||||
|
||||
## Validation Commands
|
||||
|
||||
```bash
|
||||
scripts/harness/doctor.sh
|
||||
scripts/harness/security-check.sh
|
||||
scripts/harness/backend-rules-check.sh
|
||||
scripts/harness/frontend-rules-check.sh
|
||||
scripts/harness/docs-consistency-check.sh
|
||||
scripts/harness/quick-check.sh
|
||||
scripts/harness/validate.sh
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
CI-equivalent local checks:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q
|
||||
|
||||
cd frontend
|
||||
bun install --frozen-lockfile
|
||||
bun run build
|
||||
PLANET_FRONTEND_SMOKE_URL=http://127.0.0.1:4173 bun ../scripts/harness/frontend-smoke.mjs
|
||||
```
|
||||
|
||||
The frontend smoke covers public routes, unauthenticated admin guards,
|
||||
login-error handling, the Earth iframe entry, and authenticated `super_admin`
|
||||
admin route/section rendering with mocked API data. Authenticated admin checks
|
||||
run on desktop, mobile, and 125% / 150% zoom; desktop and mobile passes also
|
||||
check for accidental global horizontal overflow. A second smoke layer exercises
|
||||
safe desktop/mobile navigation, admin search, section tab switching, dialog
|
||||
opening, and non-destructive shortcut links.
|
||||
|
||||
Optional delivery smoke, when Docker and Helm are available:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
## Deeper Docs
|
||||
|
||||
| Topic | Start Here |
|
||||
| --- | --- |
|
||||
| Data products and flows | `docs/technical/zh/platform-data-flows.md` and `docs/technical/en/platform-data-flows.md` |
|
||||
| Operations and local lifecycle | `docs/technical/zh/ops-runbook.md` and `docs/technical/en/ops-runbook.md` |
|
||||
| `planet.sh` startup behavior | `docs/technical/zh/ops-planet-sh-startup.md` and `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md` and `docs/technical/en/agents-aiprovider.md` |
|
||||
| Admin frontend | `docs/technical/zh/frontend-admin-frontend-context.md` and `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth frontend | `docs/technical/zh/earth-frontend-context.md` and `docs/technical/en/earth-frontend-context.md` |
|
||||
| Earth render order | `docs/technical/zh/earth-render-layer-order.md` and `docs/technical/en/earth-render-layer-order.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Harness workflow | `docs/HARNESS.md` |
|
||||
|
||||
## Known Sharp Edges
|
||||
|
||||
- `project_context.md` is static background for agents. It now labels future
|
||||
stack directions separately, but current code and technical docs still win
|
||||
when details diverge.
|
||||
- README now describes Web Earth, React admin, FastAPI, and `aiprovider` as the
|
||||
active local development shape.
|
||||
- Local `destroy` is intentionally destructive for Planet-owned Docker and build
|
||||
state. Never run it as a validation shortcut.
|
||||
@@ -83,7 +83,7 @@
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| React 18 | UI 框架 |
|
||||
| Ant Design Pro | 管理后台组件 |
|
||||
| Tactile UI / Radix primitives / lucide-react | 管理后台组件、基础交互与图标 |
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
@@ -168,10 +168,12 @@
|
||||
|
||||
## 快速启动
|
||||
|
||||
入口需要先具备 `zsh`、`curl` 和可访问的软件源。Ubuntu / Ubuntu WSL 上,`init` 会自动检测并补装 Docker Engine、Compose v2 和 Buildx,启动 Docker 服务并配置当前用户的访问权限;需要系统权限时会提示输入 sudo 密码。其他系统请先准备可用的 Docker 环境。
|
||||
|
||||
```bash
|
||||
# 新机器或空项目首次初始化
|
||||
./planet.sh init
|
||||
# 会自动安装/检查 uv、bun,同步 Python/前端依赖
|
||||
# 会先准备 Docker / Compose / Buildx,再安装/检查 uv、bun 并同步 Python/前端依赖
|
||||
# 会在缺少时生成 backend/.env、aiprovider/.env、frontend/.env.local
|
||||
# 会启动 PostgreSQL/Redis,并创建表、默认数据源和本地默认用户
|
||||
|
||||
@@ -322,7 +324,7 @@ ipconfig
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
`planet.sh` 为依赖安装、数据库、AI Provider 启动提供有限次重试。数据库先检查容器健康,再验证后端实际连接;`aiprovider` 直接以宿主机 `/health` 就绪为准。后端进程退出或应用初始化失败时立即停止等待,避免重复消耗健康检查预算。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
|
||||
1
TODO.md
@@ -4,6 +4,7 @@ This file is the active backlog only. Completed history belongs in `docs/CHANGEL
|
||||
|
||||
## Earth
|
||||
|
||||
- [ ] Motion Agent v2 hardening: tune the implemented MediaPipe gesture recognizer across camera placements, exercise the UE command/control client, run reconnect and dual-camera soak tests, and continue the v3 calibrated 3D roadmap described in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
|
||||
- [ ] Earth AI command entry: merge natural-language and speech-triggered LLM commands into the existing Earth search panel as described in [Agent Runtime, Earth LLM Command, And Speech Entry Plan](/home/ray/dev/linkong/planet/docs/plans/agents-earth-command-runtime-plan.md).
|
||||
- [ ] Earth action executor: implement safe visualization actions for layer toggles, batch highlights, filters, focus, result panels, and clear-highlight behavior.
|
||||
- [ ] Earth entity matching: support stable entity ids and batch matching for Beidou satellites, mainland China compute centers, BGP, news, vessels, and cables.
|
||||
|
||||
231
agents.md
@@ -1,231 +0,0 @@
|
||||
# agents.md
|
||||
|
||||
**AI Agent 角色设定。定义 AI 如何行为、沟通和工作。**
|
||||
|
||||
---
|
||||
|
||||
## Identity
|
||||
|
||||
You are **opencode**, an AI coding assistant specialized in enterprise-level systems.
|
||||
|
||||
You are working on the **智能星球计划 (Intelligent Planet Plan)** - a situational awareness system for data-centric competition featuring:
|
||||
- Python FastAPI backend
|
||||
- React Admin dashboard
|
||||
- Unreal Engine 5 3D visualization
|
||||
- Multi-source data collection
|
||||
- Polarized 3D large display (4K, 120Hz)
|
||||
|
||||
---
|
||||
|
||||
## Communication Style
|
||||
|
||||
### Tone
|
||||
- **Professional but concise**
|
||||
- Technical accuracy with clarity
|
||||
- No unnecessary verbosity
|
||||
- Use code comments sparingly (explain **why**, not **what**)
|
||||
|
||||
### When Responding
|
||||
1. **Answer directly** - 1-3 sentences for simple questions
|
||||
2. **Use code blocks** for all code snippets
|
||||
3. **Include file:line_number** references when discussing code
|
||||
4. **Never** start with "I am an AI assistant" or similar phrases
|
||||
5. **Never** add unnecessary preambles/postambles
|
||||
|
||||
### Examples
|
||||
|
||||
**Good:**
|
||||
```
|
||||
GPU clusters are stored in `backend/app/services/collectors/top500.py:45`.
|
||||
```
|
||||
|
||||
**Bad:**
|
||||
```
|
||||
Based on the information you provided, I can see that the GPU clusters are stored in the top500.py file at line 45. Let me explain more about this...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operational Mode
|
||||
|
||||
### Plan Mode (default for complex tasks)
|
||||
- Analyze requirements
|
||||
- Propose architecture
|
||||
- Confirm with user before execution
|
||||
- **DO NOT** write code until approved
|
||||
|
||||
### Build Mode (after user approval)
|
||||
- Execute the approved plan
|
||||
- Write code, run commands
|
||||
- Verify results
|
||||
- Report completion concisely
|
||||
|
||||
### Read-Only Mode
|
||||
- Analyze code
|
||||
- Explain functionality
|
||||
- Answer questions
|
||||
- **DO NOT** modify files
|
||||
|
||||
---
|
||||
|
||||
## Decision Framework
|
||||
|
||||
### When to Ask Before Acting
|
||||
- Unclear requirements
|
||||
- Multiple implementation approaches
|
||||
- Architecture changes
|
||||
- Dependency additions
|
||||
- Anything that could break existing functionality
|
||||
|
||||
### When to Act Directly
|
||||
- Clear, approved requirements
|
||||
- Routine tasks (linting, formatting, running tests)
|
||||
- Following established patterns
|
||||
- Fixing obvious bugs
|
||||
|
||||
### When to Refuse
|
||||
- Malicious code requests
|
||||
- Security violations (secrets, credentials)
|
||||
- Anything that violates `rules.md`
|
||||
|
||||
---
|
||||
|
||||
## Working Principles
|
||||
|
||||
### 1. First Understand, Then Act
|
||||
- Read relevant files before editing
|
||||
- Understand existing patterns and conventions
|
||||
- Follow the code style in the codebase
|
||||
- Match the project's technology choices
|
||||
|
||||
### 2. Incremental Progress
|
||||
- Break large tasks into smaller PRs
|
||||
- Complete one feature before starting the next
|
||||
- Run tests after each significant change
|
||||
- Commit frequently with clear messages
|
||||
|
||||
### 3. Quality First
|
||||
- Write tests for new functionality
|
||||
- Run linters before committing
|
||||
- Fix warnings, don't ignore them
|
||||
- Document non-obvious decisions
|
||||
|
||||
### 4. Communication Clarity
|
||||
- Use precise technical language
|
||||
- Show relevant code, not explanations
|
||||
- Report errors with context
|
||||
- Confirm understanding of requirements
|
||||
|
||||
---
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
Before marking a task complete:
|
||||
|
||||
- [ ] Code follows `rules.md` style guidelines
|
||||
- [ ] Type hints are correct and complete
|
||||
- [ ] Error handling is proper (no silent failures)
|
||||
- [ ] Tests pass locally
|
||||
- [ ] Linting passes
|
||||
- [ ] No TODO comments left behind
|
||||
- [ ] Documentation updated if needed
|
||||
- [ ] Commit message is clear
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Feature Development
|
||||
```
|
||||
1. Understand requirements
|
||||
2. Check existing patterns in codebase
|
||||
3. Design solution (brief mental model)
|
||||
4. Write code following rules.md
|
||||
5. Write/run tests
|
||||
6. Lint and format
|
||||
7. Commit with clear message
|
||||
8. Report completion
|
||||
```
|
||||
|
||||
### Bug Fix
|
||||
```
|
||||
1. Reproduce the bug (write failing test)
|
||||
2. Locate the source
|
||||
3. Fix the issue
|
||||
4. Verify test passes
|
||||
5. Check for regressions
|
||||
6. Commit fix
|
||||
```
|
||||
|
||||
### Refactoring
|
||||
```
|
||||
1. Understand current behavior
|
||||
2. Design target state
|
||||
3. Make incremental changes
|
||||
4. Preserve tests
|
||||
5. Verify functionality
|
||||
6. Clean up dead code
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Special Considerations
|
||||
|
||||
### WebSocket Services
|
||||
- Implement heartbeat mechanism (30-second intervals)
|
||||
- Handle disconnection gracefully
|
||||
- Include camera position in control frames
|
||||
- Support both update and full sync modes
|
||||
|
||||
### Data Collectors
|
||||
- Inherit from BaseCollector
|
||||
- Implement fetch() and transform() methods
|
||||
- Support incremental updates
|
||||
- Handle API changes gracefully
|
||||
|
||||
### UE5 Integration
|
||||
- Communicate via WebSocket
|
||||
- Send data frames at configurable intervals (default 5 min)
|
||||
- Support auto-cruise and manual modes
|
||||
- Optimize for 4K@120Hz rendering
|
||||
|
||||
### Multi-User Security
|
||||
- JWT tokens with 15-minute expiration
|
||||
- Redis token blacklist for logout
|
||||
- Role-based access control (RBAC)
|
||||
- Audit logging for all actions
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
### When Writing Code
|
||||
```python
|
||||
# File: backend/app/services/collectors/top500.py
|
||||
from typing import List, Dict
|
||||
|
||||
class TOP500Collector:
|
||||
async def fetch(self) -> List[Dict]:
|
||||
...
|
||||
```
|
||||
|
||||
### When Explaining
|
||||
- Use concise paragraphs
|
||||
- Include code references
|
||||
- No conversational filler
|
||||
|
||||
### When Reporting Progress
|
||||
- What was done
|
||||
- What remains
|
||||
- Any blockers
|
||||
- Next action
|
||||
|
||||
---
|
||||
|
||||
## Remember
|
||||
|
||||
1. **Rules are hard constraints** - follow `rules.md` absolutely
|
||||
2. **Context provides understanding** - use `project_context.md` for background
|
||||
3. **Role defines behavior** - follow `agents.md` for how to work
|
||||
4. **Quality over speed** - Enterprise systems require precision
|
||||
5. **Communicate clearly** - Precision in, precision out
|
||||
@@ -7,9 +7,6 @@ ARG AI_PROVIDER_BUILD_FINGERPRINT=unknown
|
||||
FROM ${UV_IMAGE} AS uv
|
||||
FROM ${PYTHON_IMAGE}
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
COPY --from=uv /uv /uvx /bin/
|
||||
|
||||
WORKDIR /app
|
||||
@@ -28,13 +25,16 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
--mount=type=secret,id=planet_uv_config,target=/root/.config/uv/uv.toml,required=false \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
|
||||
ARG AI_PROVIDER_BUILD_FINGERPRINT
|
||||
LABEL planet.aiprovider.build-fingerprint="${AI_PROVIDER_BUILD_FINGERPRINT}"
|
||||
|
||||
EXPOSE 8010
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:8010/health >/dev/null || exit 1
|
||||
|
||||
CMD ["uv", "run", "--frozen", "--no-dev", "--project", "/app", "python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
CMD ["/app/.venv/bin/python", "-m", "uvicorn", "aiprovider.main:app", "--host", "0.0.0.0", "--port", "8010"]
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
- [AI Provider 指南](../docs/technical/zh/agents-aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=openai-responses`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
from uuid import NAMESPACE_URL, uuid4, uuid5
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException, status
|
||||
@@ -65,6 +66,7 @@ class ProviderService:
|
||||
self.model_provider_apis = self._parse_model_provider_apis(
|
||||
overrides.get("model_provider_apis")
|
||||
)
|
||||
self.session_id = str(uuid4())
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
@@ -95,6 +97,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
if payload.context.get("session_id") is not None:
|
||||
self.session_id = str(uuid5(NAMESPACE_URL, f"planet:{payload.context['session_id']}"))
|
||||
|
||||
provider_api = self._resolve_model_provider_api(model)
|
||||
|
||||
@@ -102,6 +106,10 @@ class ProviderService:
|
||||
data = await self._request_openai_compatible(model, prompt, payload.system_prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif provider_api == "openai-responses":
|
||||
data = await self._request_openai_responses(model, prompt, payload.system_prompt)
|
||||
content_blocks = self._extract_responses_blocks(data)
|
||||
content = "".join(block.text for block in content_blocks if block.text)
|
||||
elif provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(
|
||||
model,
|
||||
@@ -200,6 +208,39 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_openai_responses(
|
||||
self, model: str, prompt: str, system_prompt: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body: dict[str, Any] = {
|
||||
"model": model, "input": prompt, "max_output_tokens": self.max_tokens, "store": False,
|
||||
}
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["instructions"] = resolved_system_prompt
|
||||
return await self._post(
|
||||
path="/responses",
|
||||
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _extract_responses_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in payload.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("type") == "message":
|
||||
for part in item.get("content") or []:
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
text = part.get("text") or part.get("refusal")
|
||||
if isinstance(text, str) and text:
|
||||
blocks.append(AIContentBlock(type="text", text=text))
|
||||
elif item.get("type") == "reasoning":
|
||||
for part in item.get("summary") or []:
|
||||
if isinstance(part, dict) and isinstance(part.get("text"), str):
|
||||
blocks.append(AIContentBlock(type="thinking", thinking=part["text"]))
|
||||
return blocks
|
||||
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
@@ -226,7 +267,7 @@ class ProviderService:
|
||||
resolved_system_prompt = self._resolve_system_prompt(system_prompt)
|
||||
if resolved_system_prompt:
|
||||
request_body["system"] = resolved_system_prompt
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking, model)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
@@ -243,8 +284,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
def _resolve_anthropic_thinking(
|
||||
self, thinking: dict[str, Any] | None, model: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
if model.casefold() == "minimax-m3" and thinking.get("type") == "enabled":
|
||||
return {"type": "adaptive"}
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
@@ -295,6 +340,9 @@ class ProviderService:
|
||||
headers: dict[str, str],
|
||||
request_body: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
headers = {"User-Agent": "Planet/1.0", **headers}
|
||||
if self.provider == "opencode-go":
|
||||
headers["x-opencode-session"] = self.session_id
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, self.http_retry_attempts + 1):
|
||||
try:
|
||||
|
||||
@@ -6,7 +6,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile, status
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import delete, func, select, text
|
||||
@@ -27,6 +27,20 @@ from app.services.earth_news import (
|
||||
save_earth_news_sources_payload,
|
||||
test_news_source_config,
|
||||
)
|
||||
from app.services.earth_news_manual import (
|
||||
broadcast_manual_news_changed,
|
||||
create_manual_news_group,
|
||||
delete_manual_news_item,
|
||||
get_news_record_or_404,
|
||||
import_manual_news_items,
|
||||
list_news_groups,
|
||||
list_news_records,
|
||||
parse_manual_news_import_upload,
|
||||
rename_manual_news_group,
|
||||
reprocess_manual_news_item,
|
||||
serialize_news_record,
|
||||
upsert_manual_news_item,
|
||||
)
|
||||
from app.services.earth_boundaries import (
|
||||
EarthBoundaryBuildError,
|
||||
get_boundary_build_status,
|
||||
@@ -119,6 +133,26 @@ class EarthNewsSourceTestPayload(BaseModel):
|
||||
source: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class EarthNewsManualItemPayload(BaseModel):
|
||||
title: str = Field(default="", max_length=500)
|
||||
summary: str = Field(default="", max_length=1200)
|
||||
content: str = Field(default="", max_length=12000)
|
||||
url: str = Field(default="", max_length=2000)
|
||||
source: str = Field(default="", max_length=255)
|
||||
region: str = Field(default="global", max_length=80)
|
||||
published_at: str | None = None
|
||||
category: str = Field(default="other", max_length=80)
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
location: dict[str, Any] | None = None
|
||||
homepage_url: str = Field(default="", max_length=2000)
|
||||
content_language: str = Field(default="", max_length=32)
|
||||
group_id: str | None = Field(default=None, max_length=120)
|
||||
|
||||
|
||||
class EarthNewsManualGroupPayload(BaseModel):
|
||||
name: str = Field(default="", max_length=120)
|
||||
|
||||
|
||||
def _normalize_earth_brand_payload(payload: dict[str, Any] | None) -> dict[str, str]:
|
||||
merged = DEFAULT_EARTH_BRAND.copy()
|
||||
if payload:
|
||||
@@ -375,6 +409,162 @@ async def test_earth_news_source(
|
||||
return await test_news_source_config(payload.source, db=db)
|
||||
|
||||
|
||||
@router.get("/news-groups")
|
||||
async def list_earth_news_groups_admin(
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await list_news_groups(db)
|
||||
|
||||
|
||||
@router.post("/news-groups")
|
||||
async def create_earth_news_group_admin(
|
||||
payload: EarthNewsManualGroupPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
group = await create_manual_news_group(db, payload.name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
return {"status": "ok", "group": group}
|
||||
|
||||
|
||||
@router.put("/news-groups/{group_id:path}")
|
||||
async def rename_earth_news_group_admin(
|
||||
group_id: str,
|
||||
payload: EarthNewsManualGroupPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
group = await rename_manual_news_group(db, group_id, payload.name)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "group": group}
|
||||
|
||||
|
||||
@router.get("/news-items")
|
||||
async def list_earth_news_items_admin(
|
||||
page: int = Query(1, ge=1),
|
||||
page_size: int = Query(50, ge=1, le=100),
|
||||
source_type: str | None = Query(None),
|
||||
region: str | None = Query(None),
|
||||
category: str | None = Query(None),
|
||||
status_filter: str | None = Query(None, alias="status"),
|
||||
group_id: str | None = Query(None),
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await list_news_records(
|
||||
db,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
source_type=source_type,
|
||||
region=region,
|
||||
category=category,
|
||||
status_filter=status_filter,
|
||||
group_id=group_id,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/news-items")
|
||||
async def create_earth_news_item_admin(
|
||||
payload: EarthNewsManualItemPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
result = await upsert_manual_news_item(db, payload.model_dump())
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
|
||||
|
||||
|
||||
@router.post("/news-items/import")
|
||||
async def import_earth_news_items_admin(
|
||||
file: UploadFile = File(...),
|
||||
group_id: str | None = Form(default=None),
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
payload = await parse_manual_news_import_upload(await file.read())
|
||||
result = await import_manual_news_items(db, payload, group_id=group_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", **result}
|
||||
|
||||
|
||||
@router.put("/news-items/{item_id:path}")
|
||||
async def update_earth_news_item_admin(
|
||||
item_id: str,
|
||||
payload: EarthNewsManualItemPayload,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
existing = await get_news_record_or_404(db, item_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
try:
|
||||
result = await upsert_manual_news_item(
|
||||
db,
|
||||
payload.model_dump(),
|
||||
item_id_override=item_id,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "ok", "created": result.created, "queued": result.queued, "item": serialize_news_record(result.item)}
|
||||
|
||||
|
||||
@router.delete("/news-items/{item_id:path}")
|
||||
async def delete_earth_news_item_admin(
|
||||
item_id: str,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
try:
|
||||
deleted = await delete_manual_news_item(db, item_id)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "deleted", "id": item_id}
|
||||
|
||||
|
||||
@router.post("/news-items/{item_id:path}/reprocess")
|
||||
async def reprocess_earth_news_item_admin(
|
||||
item_id: str,
|
||||
_current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
existing = await get_news_record_or_404(db, item_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="News item not found.")
|
||||
try:
|
||||
queued = await reprocess_manual_news_item(db, item_id)
|
||||
except PermissionError as exc:
|
||||
raise HTTPException(status_code=403, detail=str(exc)) from exc
|
||||
await db.commit()
|
||||
await broadcast_manual_news_changed()
|
||||
return {"status": "queued" if queued else "not_queued", "queued": queued, "id": item_id}
|
||||
|
||||
|
||||
@router.get("/oobe-status")
|
||||
async def get_earth_oobe_status(
|
||||
current_user: User | None = Depends(_get_optional_current_user),
|
||||
|
||||
@@ -3,9 +3,9 @@ from datetime import UTC, datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
|
||||
import httpx
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
from dotenv import dotenv_values
|
||||
from sqlalchemy import select
|
||||
@@ -65,6 +65,7 @@ from app.services.llm_provider_catalog import (
|
||||
list_fallback_llm_provider_presets,
|
||||
refresh_llm_provider_preset,
|
||||
)
|
||||
from app.services.llm_model_catalog import catalog_error_message, fetch_model_catalog
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
from app.services.persistent_logs import record_audit_log
|
||||
@@ -75,6 +76,7 @@ logger = get_logger(__name__, service="api")
|
||||
AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS = 5
|
||||
AI_CONNECTION_TEST_PROMPT_KEY = "ai.connection_test"
|
||||
SECRET_REVEAL_ROLES = {UserRole.ADMIN.value, UserRole.SUPER_ADMIN.value}
|
||||
LLM_PROVIDER_PRESET_CATEGORY_PREFIX = "llm_provider_preset:"
|
||||
|
||||
DEFAULT_SETTINGS = {
|
||||
"system": {
|
||||
@@ -345,7 +347,13 @@ class ExternalIntegrationsUpdate(BaseModel):
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if category.startswith(LLM_PROVIDER_PRESET_CATEGORY_PREFIX):
|
||||
defaults = get_fallback_llm_provider_preset(
|
||||
category.removeprefix(LLM_PROVIDER_PRESET_CATEGORY_PREFIX)
|
||||
)
|
||||
else:
|
||||
defaults = DEFAULT_SETTINGS[category]
|
||||
merged = deepcopy(defaults)
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -382,6 +390,7 @@ async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
|
||||
|
||||
async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -> dict:
|
||||
merged = merge_with_defaults(category, payload)
|
||||
record = await get_setting_record(db, category)
|
||||
if record is None:
|
||||
record = SystemSetting(category=category, payload=payload)
|
||||
@@ -391,7 +400,7 @@ async def save_setting_payload(db: AsyncSession, category: str, payload: dict) -
|
||||
|
||||
await db.commit()
|
||||
await db.refresh(record)
|
||||
return merge_with_defaults(category, record.payload)
|
||||
return merged
|
||||
|
||||
|
||||
AI_PROVIDER_ENV_FILE = Path(__file__).resolve().parents[4] / "aiprovider" / ".env"
|
||||
@@ -655,6 +664,12 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
normalized_ai["providers"].get(default_provider) or _provider_defaults(default_provider)
|
||||
)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(default_provider, provider_config)
|
||||
model_provider_apis = {
|
||||
**(provider_config.get("model_provider_apis") or {}),
|
||||
**(_get_provider_preset(default_provider).get("model_provider_apis") or {}),
|
||||
}
|
||||
if default_provider == "openai" and urlsplit(provider_config.get("base_url") or "").hostname != "api.openai.com":
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
"service_url": normalized_ai.get("service_url") or app_settings.AI_PROVIDER_SERVICE_URL,
|
||||
"service_token": _resolve_service_token(normalized_ai)[0],
|
||||
@@ -672,8 +687,7 @@ def _runtime_config_from_ai_payload(ai_payload: dict) -> dict:
|
||||
"api_key": api_key,
|
||||
"max_tokens": int(provider_config.get("max_tokens") or 1200),
|
||||
"anthropic_version": provider_config.get("anthropic_version") or "2023-06-01",
|
||||
"model_provider_apis": provider_config.get("model_provider_apis") or {},
|
||||
"preset_models": _get_provider_preset(default_provider).get("models") or [],
|
||||
"model_provider_apis": model_provider_apis,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -753,159 +767,35 @@ async def _validate_ai_provider_full_connection(ai_payload: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _join_provider_url(base_url: str, path: str) -> str:
|
||||
return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
|
||||
|
||||
|
||||
def _extract_model_ids(payload: dict) -> list[str]:
|
||||
data = payload.get("data") if isinstance(payload, dict) else None
|
||||
if isinstance(data, list):
|
||||
return [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
]
|
||||
models = payload.get("models") if isinstance(payload, dict) else None
|
||||
if isinstance(models, list):
|
||||
return [
|
||||
str(item.get("name") or item.get("model") or item.get("id") or item)
|
||||
for item in models
|
||||
if item
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_model(model_ids: list[str], model: str) -> bool:
|
||||
normalized_model = model.strip().lower()
|
||||
return any(str(item).strip().lower() == normalized_model for item in model_ids)
|
||||
|
||||
|
||||
async def _check_ai_provider_lightweight(llm_config: dict, timeout_seconds: int) -> dict:
|
||||
provider = _normalize_provider_id(llm_config.get("provider") or "")
|
||||
configured_api = (
|
||||
str(llm_config.get("provider_api") or "").strip()
|
||||
or ProviderApi.OPENAI_COMPLETIONS.value
|
||||
)
|
||||
provider_api = str(llm_config.get("provider_api") or ProviderApi.OPENAI_COMPLETIONS.value)
|
||||
model = str(llm_config.get("model") or "").strip()
|
||||
base_url = str(llm_config.get("base_url") or "").strip().rstrip("/")
|
||||
base_url = str(llm_config.get("base_url") or "").strip()
|
||||
api_key = str(llm_config.get("api_key") or "").strip()
|
||||
provider_api = configured_api
|
||||
model_provider_apis = llm_config.get("model_provider_apis")
|
||||
if isinstance(model_provider_apis, dict):
|
||||
provider_api = str(model_provider_apis.get(model) or provider_api)
|
||||
preset_models = [
|
||||
str(item)
|
||||
for item in (llm_config.get("preset_models") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
if not provider or not base_url or not model:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider/base_url/model 未完整配置。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
result = {
|
||||
"success": False, "connected": False, "mode": "lightweight_models",
|
||||
"provider": provider, "model": model,
|
||||
}
|
||||
if not base_url or not model:
|
||||
return {**result, "message": "当前 provider/base_url/model 未完整配置。"}
|
||||
if provider_api != ProviderApi.OLLAMA_GENERATE.value and not api_key:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": "当前 provider 未配置 API Key。",
|
||||
"mode": "lightweight_config",
|
||||
}
|
||||
|
||||
if provider == "opencode-go":
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.OLLAMA_GENERATE.value:
|
||||
url = _join_provider_url(base_url, "/api/tags")
|
||||
headers: dict[str, str] = {}
|
||||
elif provider_api == ProviderApi.OPENAI_COMPLETIONS.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
elif provider_api == ProviderApi.ANTHROPIC_MESSAGES.value:
|
||||
url = _join_provider_url(base_url, "/models")
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": str(llm_config.get("anthropic_version") or "2023-06-01"),
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"当前 provider_api 不支持轻量连通性测试: {provider_api}",
|
||||
"mode": "lightweight_unsupported",
|
||||
}
|
||||
|
||||
return {**result, "message": "当前 provider 未配置 API Key。"}
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=min(timeout_seconds, AI_PROVIDER_QUICK_CONNECT_TIMEOUT_SECONDS)) as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = exc.response.text or exc.response.reason_phrase
|
||||
if exc.response.status_code == 404 and _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;当前 provider 不提供可用的模型目录,已按内置模型预设确认。",
|
||||
"mode": "lightweight_preset",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: HTTP {exc.response.status_code} {detail}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
catalog = await fetch_model_catalog(
|
||||
provider, base_url, provider_api, api_key,
|
||||
str(llm_config.get("anthropic_version") or "2023-06-01"), timeout_seconds,
|
||||
)
|
||||
except Exception as exc:
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"轻量连通性测试失败: {exc}",
|
||||
"mode": "lightweight_models",
|
||||
"url": url,
|
||||
}
|
||||
|
||||
model_ids = _extract_model_ids(payload)
|
||||
if model_ids and not _contains_model(model_ids, model):
|
||||
if _contains_model(preset_models, model):
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过;provider 模型目录未返回当前别名,已按内置模型预设确认。",
|
||||
"mode": "lightweight_models_with_preset_alias",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"connected": False,
|
||||
"message": f"连接可用,但模型目录中没有当前模型: {model}",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
}
|
||||
|
||||
return {**result, "message": catalog_error_message(exc)}
|
||||
result.update({"url": catalog.url, "models_count": len(catalog.models)})
|
||||
if model.casefold() not in {item.casefold() for item in catalog.models}:
|
||||
return {**result, "message": f"模型目录查询成功,但当前模型不在目录中:{model}"}
|
||||
model_apis = llm_config.get("model_provider_apis") or {}
|
||||
return {
|
||||
"success": True,
|
||||
"connected": True,
|
||||
"message": "轻量连通性测试通过",
|
||||
"mode": "lightweight_models",
|
||||
"provider": provider,
|
||||
"provider_api": provider_api,
|
||||
"model": model,
|
||||
"models_count": len(model_ids),
|
||||
"url": url,
|
||||
**result, "success": True, "connected": True,
|
||||
"provider_api": model_apis.get(model) or provider_api,
|
||||
"message": "模型目录查询成功,当前模型已找到;尚未执行生成调用。",
|
||||
}
|
||||
|
||||
|
||||
@@ -1689,7 +1579,8 @@ async def connect_ai_provider_integration(
|
||||
)
|
||||
await emit_business_log(
|
||||
logger,
|
||||
event="settings.ai_provider.connect.success",
|
||||
event=("settings.ai_provider.connect.success" if lightweight_result["success"]
|
||||
else "settings.ai_provider.connect.failed"),
|
||||
message="AI provider connection test completed",
|
||||
category="ai",
|
||||
service="api",
|
||||
@@ -1699,7 +1590,7 @@ async def connect_ai_provider_integration(
|
||||
"provider": payload.provider,
|
||||
"model": payload.model,
|
||||
"configured": True,
|
||||
"lightweight_status": lightweight_result.get("status"),
|
||||
"connected": lightweight_result["connected"],
|
||||
},
|
||||
)
|
||||
return {
|
||||
@@ -2044,8 +1935,18 @@ async def reset_provider_credential_guide(
|
||||
@router.get("/integrations/ai-provider/presets")
|
||||
async def get_ai_provider_presets(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"data": list_fallback_llm_provider_presets()}
|
||||
presets = list_fallback_llm_provider_presets()
|
||||
saved = await get_setting_payloads(
|
||||
db, [f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}" for preset in presets]
|
||||
)
|
||||
return {
|
||||
"data": [
|
||||
{**preset, **saved[f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{preset['provider']}"]}
|
||||
for preset in presets
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/integrations/ai-provider/presets/{provider}/refresh")
|
||||
@@ -2053,22 +1954,52 @@ async def refresh_ai_provider_preset(
|
||||
provider: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
payload: AIProviderIntegrationUpdate | None = None,
|
||||
):
|
||||
try:
|
||||
provider_id = _normalize_provider_id(provider)
|
||||
api_key = None
|
||||
if provider_id == "opencode-go":
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
return {"data": await refresh_llm_provider_preset(provider_id, api_key=api_key)}
|
||||
get_fallback_llm_provider_preset(provider_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
current_payload = await get_setting_payload(db, "external_integrations")
|
||||
if payload is not None:
|
||||
if _normalize_provider_id(payload.provider) != provider_id:
|
||||
raise HTTPException(status_code=400, detail="刷新供应商与表单供应商不一致。")
|
||||
ai_payload = _build_ai_provider_payload(current_payload, payload)
|
||||
else:
|
||||
ai_payload = _normalize_ai_provider_payload(current_payload.get("ai_provider") or {})
|
||||
provider_config = ai_payload["providers"].get(provider_id) or _provider_defaults(provider_id)
|
||||
api_key, _api_key_source = _resolve_provider_api_key(provider_id, provider_config)
|
||||
if not api_key and provider_id not in {"ollama", "opencode-go", "openrouter"}:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="请先配置当前供应商的 API Key,再刷新模型列表。",
|
||||
)
|
||||
try:
|
||||
refreshed = await refresh_llm_provider_preset(
|
||||
provider_id, api_key=api_key, base_url=provider_config.get("base_url"),
|
||||
provider_api=provider_config.get("provider_api"),
|
||||
anthropic_version=provider_config.get("anthropic_version") or "2023-06-01",
|
||||
)
|
||||
except Exception as exc:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
fallback["refresh_error"] = str(exc)
|
||||
return {"data": fallback}
|
||||
logger.warning(
|
||||
"LLM provider catalog refresh failed",
|
||||
extra={
|
||||
"event": "settings.ai_provider.catalog.failed",
|
||||
"context": {
|
||||
"provider": provider_id,
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"{catalog_error_message(exc)} 已保留上次模型列表。",
|
||||
) from exc
|
||||
|
||||
refreshed["refreshed_at"] = to_iso8601_utc(datetime.now(UTC))
|
||||
await save_setting_payload(db, f"{LLM_PROVIDER_PRESET_CATEGORY_PREFIX}{provider_id}", refreshed)
|
||||
return {"data": refreshed}
|
||||
|
||||
|
||||
@router.put("/integrations")
|
||||
|
||||
@@ -7,6 +7,7 @@ from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
@@ -34,9 +35,13 @@ def _should_strip_hls_metadata_line(line: str) -> bool:
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(50, ge=1, le=100),
|
||||
q: str = Query("", max_length=200),
|
||||
selected_id: str | None = Query(None, max_length=200),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
return await get_tv_catalog_page(db, offset=offset, limit=limit, q=q, selected_id=selected_id)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
|
||||
@@ -1121,6 +1121,7 @@ async def build_vessel_snapshot_response(
|
||||
safe_limit = _safe_vessel_limit(limit)
|
||||
safe_since_minutes = min(max(int(since_minutes or 60), 1), 1440)
|
||||
observed_since = datetime.now(UTC) - timedelta(minutes=safe_since_minutes)
|
||||
snapshot_started_at = datetime.now(UTC)
|
||||
features, diagnostics = await _load_raw_vessel_snapshot_features(
|
||||
db,
|
||||
bbox=bbox,
|
||||
@@ -1137,6 +1138,7 @@ async def build_vessel_snapshot_response(
|
||||
"features": features,
|
||||
"count": len(features),
|
||||
"stats": _build_vessel_stats(features),
|
||||
"generated_at": to_iso8601_utc(snapshot_started_at),
|
||||
"diagnostics": {
|
||||
**diagnostics,
|
||||
"filtered_count": len(features),
|
||||
|
||||
@@ -161,7 +161,7 @@ async def websocket_endpoint(
|
||||
if is_anonymous:
|
||||
channels = [channel for channel in channels if channel in supported_channels]
|
||||
vessel_subscription = None
|
||||
if "vessels" in channels and "bbox" in payload_data:
|
||||
if "vessels" in channels:
|
||||
try:
|
||||
vessel_subscription = manager.subscribe_vessels(websocket, payload_data)
|
||||
except ValueError as exc:
|
||||
|
||||
@@ -66,6 +66,7 @@ class NewsSourceType(StrEnum):
|
||||
ATOM = "atom"
|
||||
AGGREGATED = "aggregated"
|
||||
REFERENCE = "reference"
|
||||
MANUAL = "manual"
|
||||
|
||||
|
||||
class NewsEnrichmentStatus(StrEnum):
|
||||
@@ -137,6 +138,7 @@ class DatasourceRunStatus(StrEnum):
|
||||
class ProviderApi(StrEnum):
|
||||
ANTHROPIC_MESSAGES = "anthropic-messages"
|
||||
OPENAI_COMPLETIONS = "openai-completions"
|
||||
OPENAI_RESPONSES = "openai-responses"
|
||||
OLLAMA_GENERATE = "ollama-generate"
|
||||
|
||||
|
||||
|
||||
@@ -4,11 +4,14 @@ import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Dict, Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.core.websocket.manager import manager
|
||||
|
||||
|
||||
EARTH_UPDATES_CHANNEL = "earth_updates"
|
||||
VESSEL_STATE_QUERY_BATCH_SIZE = 1000
|
||||
logger = get_logger(__name__, service="websocket")
|
||||
|
||||
|
||||
class DataBroadcaster:
|
||||
@@ -114,16 +117,22 @@ class DataBroadcaster:
|
||||
return
|
||||
pending = self._pending_vessel_updates
|
||||
self._pending_vessel_updates = {}
|
||||
vessels = []
|
||||
for item in pending.values():
|
||||
vessel = dict(item)
|
||||
source = vessel.pop("_source", None)
|
||||
action = vessel.pop("_action", "upsert")
|
||||
created = vessel.pop("_created", None)
|
||||
vessel["source"] = source
|
||||
vessel["action"] = action
|
||||
vessel["created"] = created
|
||||
vessels.append(vessel)
|
||||
try:
|
||||
vessels = await self._load_current_vessel_updates(list(pending))
|
||||
except Exception:
|
||||
# Preserve newer updates that arrived during the failed database read.
|
||||
self._pending_vessel_updates = {**pending, **self._pending_vessel_updates}
|
||||
raise
|
||||
if not vessels:
|
||||
return
|
||||
vessels = [
|
||||
{
|
||||
**vessel,
|
||||
"action": vessel.get("action", "upsert"),
|
||||
"created": pending.get(str(vessel["mmsi"]), {}).get("_created"),
|
||||
}
|
||||
for vessel in vessels
|
||||
]
|
||||
await manager.broadcast_vessels(
|
||||
{
|
||||
"action": "upsert",
|
||||
@@ -133,12 +142,31 @@ class DataBroadcaster:
|
||||
}
|
||||
)
|
||||
|
||||
async def _load_current_vessel_updates(self, keys: list[str]) -> list[Dict[str, Any]]:
|
||||
from app.db.session import async_session_factory
|
||||
from app.services.vessel_ais_aggregation import get_current_vessels_by_mmsi
|
||||
|
||||
mmsis = [int(key) for key in keys if key.isdigit()]
|
||||
vessels = []
|
||||
async with async_session_factory() as db:
|
||||
for offset in range(0, len(mmsis), VESSEL_STATE_QUERY_BATCH_SIZE):
|
||||
vessels.extend(await get_current_vessels_by_mmsi(
|
||||
db, mmsis[offset:offset + VESSEL_STATE_QUERY_BATCH_SIZE]
|
||||
))
|
||||
present = {int(vessel["mmsi"]) for vessel in vessels}
|
||||
vessels.extend({"mmsi": mmsi, "action": "remove"} for mmsi in mmsis if mmsi not in present)
|
||||
return vessels
|
||||
|
||||
async def broadcast_vessels_periodically(self):
|
||||
while self.running:
|
||||
try:
|
||||
await self.flush_vessel_updates()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.exception_event(
|
||||
"Failed to flush vessel updates",
|
||||
event="vessels.broadcast.failed",
|
||||
context={"error": str(exc)},
|
||||
)
|
||||
await asyncio.sleep(self._vessel_flush_interval)
|
||||
|
||||
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
|
||||
|
||||
@@ -63,6 +63,8 @@ class ConnectionManager:
|
||||
|
||||
def unsubscribe(self, websocket: WebSocket, channels: list[str]):
|
||||
for channel in {str(channel).strip() for channel in channels if str(channel).strip()}:
|
||||
if channel == "vessels":
|
||||
self.vessel_subscriptions.pop(websocket, None)
|
||||
subscribers = self.channel_subscriptions.get(channel)
|
||||
if subscribers is not None:
|
||||
subscribers.discard(websocket)
|
||||
@@ -88,7 +90,8 @@ class ConnectionManager:
|
||||
return subscription
|
||||
|
||||
def _normalize_vessel_subscription(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
bbox = config.get("bbox")
|
||||
global_scope = config.get("scope") == "global"
|
||||
bbox = [-180, -90, 180, 90] if global_scope else config.get("bbox")
|
||||
if not isinstance(bbox, (list, tuple)) or len(bbox) != 4:
|
||||
raise ValueError("vessels subscription requires bbox=[lon_min,lat_min,lon_max,lat_max]")
|
||||
try:
|
||||
@@ -103,7 +106,7 @@ class ConnectionManager:
|
||||
raise ValueError("bbox longitude values must be between -180 and 180")
|
||||
if not (-90 <= lat_min <= 90 and -90 <= lat_max <= 90):
|
||||
raise ValueError("bbox latitude values must be between -90 and 90")
|
||||
if (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
if not global_scope and (lon_max - lon_min) * (lat_max - lat_min) > MAX_VESSEL_BBOX_AREA:
|
||||
raise ValueError("bbox is too large; zoom in or request a smaller viewport")
|
||||
|
||||
zoom = int(config.get("zoom") or 1)
|
||||
@@ -116,10 +119,11 @@ class ConnectionManager:
|
||||
if str(item).strip()
|
||||
}
|
||||
return {
|
||||
"scope": "global" if global_scope else "viewport",
|
||||
"bbox": (lon_min, lat_min, lon_max, lat_max),
|
||||
"zoom": zoom,
|
||||
"limit": limit,
|
||||
"type": vessel_types,
|
||||
"type": sorted(vessel_types),
|
||||
"last_sent_at": None,
|
||||
}
|
||||
|
||||
@@ -152,7 +156,9 @@ class ConnectionManager:
|
||||
vessel
|
||||
for vessel in vessels
|
||||
if self._vessel_matches_subscription(vessel, subscription)
|
||||
][: min(subscription["limit"], MAX_VESSEL_WS_MESSAGE_ITEMS)]
|
||||
]
|
||||
if subscription.get("scope") != "global":
|
||||
matched = matched[:subscription["limit"]]
|
||||
if not matched:
|
||||
continue
|
||||
subscription["last_sent_at"] = datetime.now(UTC)
|
||||
@@ -162,16 +168,24 @@ class ConnectionManager:
|
||||
"timestamp": subscription["last_sent_at"].isoformat(),
|
||||
"payload": {
|
||||
**data,
|
||||
"vessels": matched,
|
||||
"vessels": [],
|
||||
"subscription": {
|
||||
"bbox": list(subscription["bbox"]),
|
||||
"zoom": subscription["zoom"],
|
||||
"limit": subscription["limit"],
|
||||
"scope": subscription.get("scope", "viewport"),
|
||||
},
|
||||
},
|
||||
}
|
||||
try:
|
||||
await connection.send_json(message)
|
||||
for offset in range(0, len(matched), MAX_VESSEL_WS_MESSAGE_ITEMS):
|
||||
await connection.send_json({
|
||||
**message,
|
||||
"payload": {
|
||||
**message["payload"],
|
||||
"vessels": matched[offset:offset + MAX_VESSEL_WS_MESSAGE_ITEMS],
|
||||
},
|
||||
})
|
||||
except Exception:
|
||||
self.unsubscribe_all(connection)
|
||||
|
||||
@@ -180,6 +194,8 @@ class ConnectionManager:
|
||||
vessel: dict[str, Any],
|
||||
subscription: dict[str, Any],
|
||||
) -> bool:
|
||||
if vessel.get("action") == "remove" and subscription.get("scope") == "global":
|
||||
return True
|
||||
try:
|
||||
lon = float(vessel.get("lon"))
|
||||
lat = float(vessel.get("lat"))
|
||||
|
||||
@@ -35,7 +35,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 0 # Zero keeps the complete matching channel catalog.
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
@@ -445,7 +445,7 @@ class NewsLiveStreamsCollector(BaseCollector):
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
if max_sources > 0 and len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
@@ -11,17 +11,20 @@ To get higher limits, set PEERINGDB_API_KEY environment variable.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
import os
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
# PeeringDB API key - read from environment variable
|
||||
PEERINGDB_API_KEY = os.environ.get("PEERINGDB_API_KEY", "")
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class PeeringDBIXPCollector(HTTPCollector):
|
||||
@@ -39,6 +42,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -61,7 +65,11 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -72,13 +80,21 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -146,6 +162,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -167,7 +184,11 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -178,13 +199,21 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
@@ -254,6 +283,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
@@ -275,7 +305,11 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
@@ -286,13 +320,21 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
print(f"PeeringDB rate limited, waiting {delay}s before retry...")
|
||||
logger.warning_event(
|
||||
"PeeringDB rate limited; retrying after delay",
|
||||
event="collector.peeringdb.rate_limited",
|
||||
context={"delay_seconds": delay, "attempt": attempt + 1},
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
last_error = "Rate limited"
|
||||
continue
|
||||
raise
|
||||
|
||||
print(f"Warning: PeeringDB collection failed after {max_retries} retries: {last_error}")
|
||||
logger.warning_event(
|
||||
"PeeringDB collection failed after retries",
|
||||
event="collector.peeringdb.retries_exhausted",
|
||||
context={"max_retries": max_retries, "last_error": last_error},
|
||||
)
|
||||
return {}
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
"""Space-Track TLE Collector
|
||||
"""Space-Track TLE Collector.
|
||||
|
||||
Collects satellite TLE (Two-Line Element) data from Space-Track.org.
|
||||
API documentation: https://www.space-track.org/documentation
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from typing import Any, Dict, List
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.core.logging import get_logger
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
logger = get_logger(__name__, service="collector")
|
||||
|
||||
|
||||
class SpaceTrackTLECollector(BaseCollector):
|
||||
@@ -53,10 +57,16 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
password = settings.SPACETRACK_PASSWORD
|
||||
|
||||
if not username or not password:
|
||||
print("SPACETRACK: No credentials configured, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track credentials are not configured; using sample data",
|
||||
event="collector.spacetrack.credentials_missing",
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch started",
|
||||
event="collector.spacetrack.fetch.start",
|
||||
)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
@@ -78,11 +88,17 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
logger.info_event(
|
||||
"Space-Track login response received",
|
||||
event="collector.spacetrack.login.response",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
|
||||
if login_response.status_code == 403:
|
||||
print("SPACETRACK: Trying alternate login method...")
|
||||
logger.warning_event(
|
||||
"Space-Track login returned forbidden; trying alternate method",
|
||||
event="collector.spacetrack.login.forbidden",
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
@@ -90,11 +106,6 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
) as alt_client:
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
"password": password,
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
@@ -102,77 +113,59 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"password": password,
|
||||
},
|
||||
)
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
logger.info_event(
|
||||
"Space-Track alternate login response received",
|
||||
event="collector.spacetrack.alt_login.response",
|
||||
context={"status_code": alt_login.status_code},
|
||||
)
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
logger.info_event(
|
||||
"Space-Track alternate query completed",
|
||||
event="collector.spacetrack.alt_query.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track login failed; using sample data",
|
||||
event="collector.spacetrack.login.failed",
|
||||
context={"status_code": login_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
print(f"SPACETRACK: Attempting to fetch TLE data with username: {username}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
},
|
||||
logger.info_event(
|
||||
"Space-Track TLE query response received",
|
||||
event="collector.spacetrack.query.response",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
print(f"SPACETRACK: Login response status: {login_response.status_code}")
|
||||
print(f"SPACETRACK: Login response URL: {login_response.url}")
|
||||
print(f"SPACETRACK: Login response body: {login_response.text[:500]}")
|
||||
|
||||
if login_response.status_code != 200:
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
print(f"SPACETRACK: Query failed, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE query failed; using sample data",
|
||||
event="collector.spacetrack.query.failed",
|
||||
context={"status_code": tle_response.status_code},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records")
|
||||
logger.info_event(
|
||||
"Space-Track TLE fetch completed",
|
||||
event="collector.spacetrack.fetch.completed",
|
||||
context={"record_count": len(data)},
|
||||
)
|
||||
return data
|
||||
except Exception as e:
|
||||
print(f"SPACETRACK: Error - {e}, using sample data")
|
||||
logger.warning_event(
|
||||
"Space-Track TLE fetch failed; using sample data",
|
||||
event="collector.spacetrack.fetch.failed",
|
||||
context={"error": str(e)},
|
||||
)
|
||||
return self._get_sample_data()
|
||||
|
||||
def transform(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -45,7 +45,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("earth-bgp-context.md", "earth-bgp-context", "docs_developer", "Earth", 14, "BGP 态势上下文", "BGP Context"),
|
||||
DocsMetadata("earth-interactable-usage.md", "earth-interactable-usage", "docs_developer", "Earth", 16, "智能星球可交互图标接入", "Intelligent Planet Interactable Usage"),
|
||||
DocsMetadata("earth-interactable-clustering.md", "earth-interactable-clustering", "docs_developer", "Earth", 17, "智能星球可交互图标聚类策略", "Intelligent Planet Interactable Clustering"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 17, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-toolbar-overlay-coordination.md", "earth-toolbar-overlay-coordination", "docs_developer", "Earth", 18, "智能星球工具栏与浮层协同", "Intelligent Planet Toolbar and Overlay Coordination"),
|
||||
DocsMetadata("earth-news-sources.md", "earth-news-sources", "docs_developer", "Earth", 19, "智能星球新闻源配置", "Intelligent Planet News Source Configuration"),
|
||||
DocsMetadata("frontend-admin-frontend-context.md", "frontend-admin-frontend-context", "docs_developer", "Frontend", 20, "控制台前端结构", "Admin Frontend Context"),
|
||||
DocsMetadata("frontend-layout-guidelines.md", "frontend-layout-guidelines", "docs_developer", "Frontend", 21, "前端布局指南", "Frontend Layout Guidelines"),
|
||||
@@ -56,7 +56,7 @@ DOCS_METADATA: tuple[DocsMetadata, ...] = (
|
||||
DocsMetadata("backend-datasources-api-performance.md", "backend-datasources-api-performance", "docs_developer", "Backend", 33, "数据源 API 性能", "Datasource API Performance"),
|
||||
DocsMetadata("data-job-earth-sync-architecture.md", "data-job-earth-sync-architecture", "docs_developer", "Backend", 34, "数据作业与 Outbox 技术架构", "Data Jobs and Outbox Architecture"),
|
||||
DocsMetadata("backend-enum-contracts.md", "backend-enum-contracts", "docs_developer", "Backend", 35, "后端枚举与字符串兼容契约", "Backend Enum and String Compatibility Contract"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 36, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("location-pipeline-development.md", "location-pipeline-development", "docs_developer", "Backend", 35, "通用位置估算管线开发说明", "Shared Location Resolution Pipeline Development Guide"),
|
||||
DocsMetadata("earth-news-live-streams-collector-format.md", "earth-news-live-streams-collector-format", "docs_developer", "Backend", 36, "新闻直播采集格式", "News Live Streams Collector Format"),
|
||||
DocsMetadata("docs-gatekeeper-development.md", "docs-gatekeeper-development", "docs_developer", "Backend", 37, "Docs Gatekeeper 开发说明", "Docs Gatekeeper Development Guide"),
|
||||
DocsMetadata("agents-aiprovider.md", "agents-aiprovider", "docs_developer", "Agents", 40, "AI Provider 指南", "AI Provider Guide"),
|
||||
|
||||
@@ -56,6 +56,8 @@ def build_earth_update_from_db_payload(payload: dict[str, Any]) -> dict[str, Any
|
||||
source_has_adapter = bool(get_earth_update_layers_for_source(source))
|
||||
effective_source = source if source_has_adapter else (table_name if table_name else source)
|
||||
operation = payload.get("operation")
|
||||
if "vessels" in layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
refresh_strategy = "reload"
|
||||
update: dict[str, Any] = {
|
||||
"event": "earth.layer.changed",
|
||||
"action": "database_changed",
|
||||
@@ -113,6 +115,8 @@ class PendingEarthDbChange:
|
||||
operation = payload.get("operation")
|
||||
if operation:
|
||||
self.operations.add(str(operation))
|
||||
if "vessels" in self.layers and operation in {"DELETE", "TRUNCATE"}:
|
||||
self.refresh_strategy = "reload"
|
||||
entity_keys = payload.get("entity_keys")
|
||||
if not isinstance(entity_keys, list):
|
||||
entity_key = payload.get("entity_key")
|
||||
@@ -176,6 +180,17 @@ class EarthDbChangeDispatcher:
|
||||
if not update:
|
||||
return False
|
||||
|
||||
if payload.get("table") == "vessel_current_state" and payload.get("operation") == "DELETE":
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
|
||||
keys = payload.get("entity_keys")
|
||||
if not isinstance(keys, list):
|
||||
keys = [payload.get("entity_key")]
|
||||
broadcaster.enqueue_vessel_update({
|
||||
"action": "remove",
|
||||
"vessels": [{"mmsi": key} for key in keys if key is not None],
|
||||
})
|
||||
|
||||
source = update["source"]
|
||||
pending = self._pending.get(source)
|
||||
if pending is None:
|
||||
|
||||
@@ -25,6 +25,7 @@ EARTH_LAYER_ADAPTERS: tuple[EarthLayerAdapter, ...] = (
|
||||
layers=("vessels",),
|
||||
cache_patterns=("vessels*", "summary*"),
|
||||
derived_models=("ais_raw_observations", "ais_conflict_records", "ais_source_health"),
|
||||
refresh_strategy="delta",
|
||||
),
|
||||
EarthLayerAdapter(
|
||||
sources=frozenset(
|
||||
|
||||
@@ -865,9 +865,9 @@ def _get_locale_text(
|
||||
if isinstance(fallback, dict):
|
||||
value = _coerce_str(fallback.get(key))
|
||||
if value:
|
||||
if locale == "en-US" and _contains_cjk_text(value):
|
||||
return ""
|
||||
return value
|
||||
if locale == "en-US" and _is_chinese_language(item.content_language):
|
||||
return item.title if key == "title" else item.summary
|
||||
return ""
|
||||
|
||||
|
||||
@@ -2282,6 +2282,53 @@ def _filter_news_items_by_source_ids(
|
||||
]
|
||||
|
||||
|
||||
def _news_item_source_id(item: ParsedNewsItem) -> str:
|
||||
return item.id.split(":", 1)[0] if ":" in item.id else ""
|
||||
|
||||
|
||||
def _is_news_item_display_ready(item: ParsedNewsItem, *, locale: str) -> bool:
|
||||
return bool(
|
||||
_get_locale_text(item, "title", locale=locale)
|
||||
and _get_locale_text(item, "summary", locale=locale)
|
||||
)
|
||||
|
||||
|
||||
def _diversify_news_items_for_locale(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
active_region: str,
|
||||
limit: int,
|
||||
locale: str,
|
||||
) -> list[ParsedNewsItem]:
|
||||
ranked = sorted(
|
||||
_rank_and_trim_items(items, active_region=active_region, limit=max(len(items), limit)),
|
||||
key=lambda item: (not _is_news_item_display_ready(item, locale=locale),),
|
||||
)
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for item in ranked:
|
||||
if active_region == "global":
|
||||
key = item.feed_region or "global"
|
||||
else:
|
||||
key = _news_item_source_id(item) or item.source or item.feed_name or item.id
|
||||
if key not in buckets:
|
||||
buckets[key] = []
|
||||
order.append(key)
|
||||
buckets[key].append(item)
|
||||
|
||||
diversified: list[ParsedNewsItem] = []
|
||||
while len(diversified) < limit and order:
|
||||
next_order: list[str] = []
|
||||
for source_id in order:
|
||||
bucket = buckets.get(source_id) or []
|
||||
if bucket and len(diversified) < limit:
|
||||
diversified.append(bucket.pop(0))
|
||||
if bucket:
|
||||
next_order.append(source_id)
|
||||
order = next_order
|
||||
return diversified
|
||||
|
||||
|
||||
async def _call_store_list_items(list_fn, db: AsyncSession, **kwargs):
|
||||
try:
|
||||
return await list_fn(db, **kwargs)
|
||||
@@ -2707,6 +2754,7 @@ async def get_earth_news_payload(
|
||||
)
|
||||
await record_earth_news_sources_health(db, health_by_source)
|
||||
ranked_fetched_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
await _enqueue_unverified_locations(ranked_fetched_items)
|
||||
await upsert_earth_news_items(db, ranked_fetched_items)
|
||||
|
||||
items = await _call_store_list_items(
|
||||
@@ -2717,6 +2765,37 @@ async def get_earth_news_payload(
|
||||
categories=categories,
|
||||
source_ids=source_ids,
|
||||
)
|
||||
if not source_ids:
|
||||
ready_sources = {
|
||||
_news_item_source_id(item)
|
||||
for item in items
|
||||
if _is_news_item_display_ready(item, locale=locale)
|
||||
}
|
||||
missing_ready_sources = [
|
||||
source.id
|
||||
for source in sources
|
||||
if source.id and source.id not in ready_sources
|
||||
]
|
||||
if missing_ready_sources:
|
||||
extra_items: list[ParsedNewsItem] = []
|
||||
for missing_source_id in missing_ready_sources:
|
||||
extra_items.extend(
|
||||
await _call_store_list_items(
|
||||
list_earth_news_items,
|
||||
db,
|
||||
active_region=active_region,
|
||||
limit=3,
|
||||
categories=categories,
|
||||
source_ids={missing_source_id},
|
||||
)
|
||||
)
|
||||
if extra_items:
|
||||
items = _diversify_news_items_for_locale(
|
||||
[*items, *extra_items],
|
||||
active_region=active_region,
|
||||
limit=limit,
|
||||
locale=locale,
|
||||
)
|
||||
if hasattr(db, "execute"):
|
||||
cruise_items = await _call_store_list_items(
|
||||
list_earth_news_cruise_items,
|
||||
@@ -2727,7 +2806,8 @@ async def get_earth_news_payload(
|
||||
)
|
||||
else:
|
||||
cruise_items = items
|
||||
await _enqueue_unverified_locations(items)
|
||||
enqueue_candidates = {item.id: item for item in [*items, *cruise_items] if item.id}
|
||||
await _enqueue_unverified_locations(list(enqueue_candidates.values()))
|
||||
stale = bool(errors and items)
|
||||
|
||||
return _build_payload(
|
||||
|
||||
693
backend/app/services/earth_news_manual.py
Normal file
@@ -0,0 +1,693 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.enums import NewsEnrichmentStatus, NewsSourceType, NewsTaggingSource
|
||||
from app.core.websocket.broadcaster import broadcaster
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.earth_news import (
|
||||
ALLOWED_NEWS_CATEGORY_KEYS,
|
||||
DEFAULT_NEWS_LOCALE,
|
||||
REGION_ANCHORS,
|
||||
NewsFeedEndpoint,
|
||||
NewsFeedSource,
|
||||
NewsTargetLocation,
|
||||
ParsedNewsItem,
|
||||
apply_news_classification,
|
||||
build_anchor_location_patch,
|
||||
build_target_location_job_payload,
|
||||
build_target_location_patch,
|
||||
_serialize_item,
|
||||
)
|
||||
from app.services.earth_news_queue import enqueue_target_location_job
|
||||
from app.services.earth_news_store import record_to_parsed_news_item
|
||||
|
||||
|
||||
MANUAL_NEWS_SOURCE_ID = "manual"
|
||||
MANUAL_NEWS_SOURCE_LABEL = "手动添加"
|
||||
MANUAL_NEWS_MAX_IMPORT_ITEMS = 500
|
||||
MANUAL_NEWS_MAX_TITLE_LENGTH = 500
|
||||
MANUAL_NEWS_MAX_SUMMARY_LENGTH = 1200
|
||||
MANUAL_NEWS_MAX_CONTENT_LENGTH = 12000
|
||||
EARTH_NEWS_MANUAL_GROUPS_CATEGORY = "earth_news_manual_groups"
|
||||
DEFAULT_MANUAL_NEWS_GROUP_ID = "manual-default"
|
||||
DEFAULT_MANUAL_NEWS_GROUP_NAME = "新建新闻组"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManualNewsWriteResult:
|
||||
item: EarthNewsItem
|
||||
created: bool
|
||||
queued: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManualNewsGroup:
|
||||
id: str
|
||||
name: str
|
||||
sort_order: int = 0
|
||||
|
||||
|
||||
def _clean_text(value: object, *, max_length: int) -> str:
|
||||
raw = "" if value is None else str(value)
|
||||
text = BeautifulSoup(html.unescape(raw), "html.parser").get_text(" ", strip=True)
|
||||
text = re.sub(r"\s+", " ", text).strip()
|
||||
if len(text) > max_length:
|
||||
return text[: max_length - 1].rstrip() + "…"
|
||||
return text
|
||||
|
||||
|
||||
def _parse_datetime(value: object) -> datetime | None:
|
||||
if value is None or str(value).strip() == "":
|
||||
return None
|
||||
if isinstance(value, datetime):
|
||||
parsed = value
|
||||
else:
|
||||
try:
|
||||
parsed = datetime.fromisoformat(str(value).strip().replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError("published_at 必须是 ISO8601 时间。") from exc
|
||||
if parsed.tzinfo is None:
|
||||
return parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
|
||||
|
||||
def _detect_language(*parts: str) -> str:
|
||||
text = " ".join(part for part in parts if part)
|
||||
cjk_count = len(re.findall(r"[\u4e00-\u9fff]", text))
|
||||
latin_count = len(re.findall(r"[A-Za-z]", text))
|
||||
return "zh-CN" if cjk_count >= max(4, latin_count // 3) else "en-US"
|
||||
|
||||
|
||||
def _manual_item_id(*, title: str, published_at: datetime | None, url: str, source: str) -> str:
|
||||
published = published_at.isoformat() if published_at else ""
|
||||
basis = "\n".join([title.strip().lower(), published, url.strip().lower(), source.strip().lower()])
|
||||
return f"manual:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:16]}"
|
||||
|
||||
|
||||
def _manual_group_id(name: str) -> str:
|
||||
basis = f"{name.strip().lower()}\n{datetime.now(UTC).isoformat()}"
|
||||
return f"manual-group:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:10]}"
|
||||
|
||||
|
||||
def _news_meta(record: EarthNewsItem) -> dict[str, Any]:
|
||||
location_meta = record.location_meta if isinstance(record.location_meta, dict) else {}
|
||||
news_meta = location_meta.get("news_meta")
|
||||
return dict(news_meta) if isinstance(news_meta, dict) else {}
|
||||
|
||||
|
||||
def _record_source_type(record: EarthNewsItem) -> str:
|
||||
return str(_news_meta(record).get("feed_type") or _news_meta(record).get("source_type") or "rss")
|
||||
|
||||
|
||||
def _record_manual_group_id(record: EarthNewsItem) -> str:
|
||||
return str(_news_meta(record).get("manual_group_id") or DEFAULT_MANUAL_NEWS_GROUP_ID)
|
||||
|
||||
|
||||
def _rss_group_id(record: EarthNewsItem) -> str:
|
||||
basis = "\n".join(
|
||||
[
|
||||
_record_source_type(record),
|
||||
str(record.feed_name or ""),
|
||||
str(record.source or ""),
|
||||
]
|
||||
)
|
||||
return f"rss:{hashlib.sha1(basis.encode('utf-8')).hexdigest()[:12]}"
|
||||
|
||||
|
||||
def _default_manual_group() -> dict[str, Any]:
|
||||
return {
|
||||
"id": DEFAULT_MANUAL_NEWS_GROUP_ID,
|
||||
"name": DEFAULT_MANUAL_NEWS_GROUP_NAME,
|
||||
"sort_order": 0,
|
||||
}
|
||||
|
||||
|
||||
def _normalize_manual_groups_payload(payload: Any) -> list[dict[str, Any]]:
|
||||
raw_groups = payload.get("groups") if isinstance(payload, dict) else None
|
||||
normalized: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
for index, item in enumerate(raw_groups if isinstance(raw_groups, list) else []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
group_id = str(item.get("id") or "").strip()
|
||||
name = _clean_text(item.get("name"), max_length=120)
|
||||
if not group_id or not name or group_id in seen:
|
||||
continue
|
||||
normalized.append(
|
||||
{
|
||||
"id": group_id,
|
||||
"name": name,
|
||||
"sort_order": int(item.get("sort_order") or index),
|
||||
}
|
||||
)
|
||||
seen.add(group_id)
|
||||
if DEFAULT_MANUAL_NEWS_GROUP_ID not in seen:
|
||||
normalized.insert(0, _default_manual_group())
|
||||
return sorted(normalized, key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
|
||||
|
||||
|
||||
async def _get_manual_groups_record(db: AsyncSession) -> SystemSetting | None:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == EARTH_NEWS_MANUAL_GROUPS_CATEGORY)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_manual_news_groups(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
record = await _get_manual_groups_record(db)
|
||||
return _normalize_manual_groups_payload(record.payload if record else None)
|
||||
|
||||
|
||||
async def _save_manual_news_groups(db: AsyncSession, groups: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
normalized = _normalize_manual_groups_payload({"groups": groups})
|
||||
record = await _get_manual_groups_record(db)
|
||||
payload = {"groups": normalized}
|
||||
if record is None:
|
||||
db.add(SystemSetting(category=EARTH_NEWS_MANUAL_GROUPS_CATEGORY, payload=payload))
|
||||
else:
|
||||
record.payload = payload
|
||||
await db.flush()
|
||||
return normalized
|
||||
|
||||
|
||||
async def resolve_manual_news_group(db: AsyncSession, group_id: str | None) -> ManualNewsGroup:
|
||||
normalized_id = str(group_id or DEFAULT_MANUAL_NEWS_GROUP_ID).strip() or DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
groups = await get_manual_news_groups(db)
|
||||
match = next((item for item in groups if item.get("id") == normalized_id), None)
|
||||
if match is None and normalized_id != DEFAULT_MANUAL_NEWS_GROUP_ID:
|
||||
raise ValueError(f"手动新闻组不存在:{normalized_id}")
|
||||
match = match or _default_manual_group()
|
||||
return ManualNewsGroup(
|
||||
id=str(match["id"]),
|
||||
name=str(match["name"]),
|
||||
sort_order=int(match.get("sort_order") or 0),
|
||||
)
|
||||
|
||||
|
||||
async def create_manual_news_group(db: AsyncSession, name: str) -> dict[str, Any]:
|
||||
group_name = _clean_text(name, max_length=120)
|
||||
if not group_name:
|
||||
raise ValueError("新闻组名称不能为空。")
|
||||
groups = await get_manual_news_groups(db)
|
||||
group = {"id": _manual_group_id(group_name), "name": group_name, "sort_order": len(groups)}
|
||||
groups.append(group)
|
||||
await _save_manual_news_groups(db, groups)
|
||||
return group
|
||||
|
||||
|
||||
async def rename_manual_news_group(db: AsyncSession, group_id: str, name: str) -> dict[str, Any]:
|
||||
group_name = _clean_text(name, max_length=120)
|
||||
if not group_name:
|
||||
raise ValueError("新闻组名称不能为空。")
|
||||
groups = await get_manual_news_groups(db)
|
||||
match = next((item for item in groups if item.get("id") == group_id), None)
|
||||
if match is None:
|
||||
raise ValueError(f"手动新闻组不存在:{group_id}")
|
||||
match["name"] = group_name
|
||||
await _save_manual_news_groups(db, groups)
|
||||
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem).where(
|
||||
EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == NewsSourceType.MANUAL.value
|
||||
)
|
||||
)
|
||||
for record in result.scalars().all():
|
||||
if _record_manual_group_id(record) != group_id:
|
||||
continue
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = dict(location_meta.get("news_meta") or {})
|
||||
news_meta["manual_group_name"] = group_name
|
||||
location_meta["news_meta"] = news_meta
|
||||
record.location_meta = location_meta
|
||||
await db.flush()
|
||||
return match
|
||||
|
||||
|
||||
def _normalize_region(value: object) -> str:
|
||||
region = str(value or "global").strip().lower() or "global"
|
||||
if region not in REGION_ANCHORS:
|
||||
raise ValueError(f"region 不支持:{region}")
|
||||
return region
|
||||
|
||||
|
||||
def _normalize_tags(value: object) -> list[str]:
|
||||
if value is None:
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
parts = re.split(r"[,,\n]", value)
|
||||
elif isinstance(value, list):
|
||||
parts = [str(item) for item in value]
|
||||
else:
|
||||
raise ValueError("tags 必须是字符串数组或逗号分隔字符串。")
|
||||
return [item.strip() for item in parts if item.strip()][:20]
|
||||
|
||||
|
||||
def _normalize_location(value: object) -> NewsTargetLocation | None:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("location 必须是对象。")
|
||||
lat = value.get("latitude")
|
||||
lon = value.get("longitude")
|
||||
if lat in (None, "") and lon in (None, ""):
|
||||
return None
|
||||
try:
|
||||
latitude = float(lat)
|
||||
longitude = float(lon)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("location.latitude / longitude 必须是数字。") from exc
|
||||
if not -90 <= latitude <= 90 or not -180 <= longitude <= 180:
|
||||
raise ValueError("location 经纬度超出范围。")
|
||||
label = _clean_text(value.get("label"), max_length=255)
|
||||
if not label:
|
||||
label = f"{latitude:.4f}, {longitude:.4f}"
|
||||
return NewsTargetLocation(
|
||||
latitude=latitude,
|
||||
longitude=longitude,
|
||||
label=label,
|
||||
source="manual_location",
|
||||
confidence=1.0,
|
||||
country=_clean_text(value.get("country"), max_length=100) or None,
|
||||
city=_clean_text(value.get("city"), max_length=100) or None,
|
||||
)
|
||||
|
||||
|
||||
def _manual_source(source_name: str, *, region: str) -> NewsFeedSource:
|
||||
return NewsFeedSource(
|
||||
id=MANUAL_NEWS_SOURCE_ID,
|
||||
name=source_name or MANUAL_NEWS_SOURCE_LABEL,
|
||||
region=region,
|
||||
feed_url="",
|
||||
homepage_url="",
|
||||
source_type=NewsSourceType.MANUAL.value,
|
||||
default_category="other",
|
||||
source_tags=("manual",),
|
||||
)
|
||||
|
||||
|
||||
def _manual_feed(category: str) -> NewsFeedEndpoint:
|
||||
return NewsFeedEndpoint(
|
||||
id=MANUAL_NEWS_SOURCE_ID,
|
||||
name=MANUAL_NEWS_SOURCE_LABEL,
|
||||
url="",
|
||||
type=NewsSourceType.MANUAL.value,
|
||||
default_category=category or "other",
|
||||
tags=("manual",),
|
||||
priority=1,
|
||||
)
|
||||
|
||||
|
||||
def parsed_manual_news_item(
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
item_id_override: str | None = None,
|
||||
) -> tuple[ParsedNewsItem, NewsTargetLocation | None, str]:
|
||||
title = _clean_text(payload.get("title"), max_length=MANUAL_NEWS_MAX_TITLE_LENGTH)
|
||||
if not title:
|
||||
raise ValueError("title 不能为空。")
|
||||
content = _clean_text(payload.get("content"), max_length=MANUAL_NEWS_MAX_CONTENT_LENGTH)
|
||||
summary = _clean_text(payload.get("summary"), max_length=MANUAL_NEWS_MAX_SUMMARY_LENGTH)
|
||||
if not summary:
|
||||
summary = _clean_text(content, max_length=240) if content else title
|
||||
source = _clean_text(payload.get("source"), max_length=255) or MANUAL_NEWS_SOURCE_LABEL
|
||||
region = _normalize_region(payload.get("region"))
|
||||
published_at = _parse_datetime(payload.get("published_at")) or datetime.now(UTC)
|
||||
url = str(payload.get("url") or "").strip()
|
||||
category = str(payload.get("category") or "other").strip().lower() or "other"
|
||||
if category not in ALLOWED_NEWS_CATEGORY_KEYS:
|
||||
raise ValueError(f"category 不支持:{category}")
|
||||
tags = _normalize_tags(payload.get("tags"))
|
||||
target = _normalize_location(payload.get("location"))
|
||||
language = str(payload.get("content_language") or "").strip() or _detect_language(title, summary, content)
|
||||
localizations = {
|
||||
language: {
|
||||
"title": title,
|
||||
"summary": summary,
|
||||
}
|
||||
}
|
||||
item = ParsedNewsItem(
|
||||
id=item_id_override
|
||||
or _manual_item_id(title=title, published_at=published_at, url=url, source=source),
|
||||
title=title,
|
||||
summary=summary,
|
||||
url=url,
|
||||
source=source,
|
||||
feed_name=MANUAL_NEWS_SOURCE_LABEL,
|
||||
feed_region=region,
|
||||
homepage_url=str(payload.get("homepage_url") or ""),
|
||||
published_at=published_at,
|
||||
content_language=language,
|
||||
localizations=localizations,
|
||||
enrichment_status=NewsEnrichmentStatus.PENDING.value,
|
||||
source_tags=["manual"],
|
||||
feed_id=MANUAL_NEWS_SOURCE_ID,
|
||||
feed_type=NewsSourceType.MANUAL.value,
|
||||
feed_default_category=category,
|
||||
category=category,
|
||||
item_tags=tags,
|
||||
tagging_source=NewsTaggingSource.MANUAL.value if payload.get("category") else NewsTaggingSource.RULES.value,
|
||||
tagging_confidence=0.9 if payload.get("category") else 0.0,
|
||||
)
|
||||
source_config = _manual_source(source, region=region)
|
||||
feed = _manual_feed(category)
|
||||
apply_news_classification(item, source_config, feed=feed)
|
||||
if payload.get("category"):
|
||||
item.category = category
|
||||
item.tagging_source = NewsTaggingSource.MANUAL.value
|
||||
item.tagging_confidence = 0.9
|
||||
if tags:
|
||||
item.item_tags = sorted(set([*item.item_tags, *tags]))
|
||||
return item, target, content
|
||||
|
||||
|
||||
def _manual_editable(record: EarthNewsItem) -> bool:
|
||||
if record.id.startswith("manual:"):
|
||||
return True
|
||||
news_meta = (record.location_meta or {}).get("news_meta") if isinstance(record.location_meta, dict) else None
|
||||
return isinstance(news_meta, dict) and news_meta.get("feed_type") == NewsSourceType.MANUAL.value
|
||||
|
||||
|
||||
async def _broadcast_news_reload() -> None:
|
||||
await broadcaster.broadcast_earth_update(
|
||||
{
|
||||
"action": "database_changed",
|
||||
"source": "earth_news_items",
|
||||
"layers": ["news"],
|
||||
"refresh_strategy": "reload",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
async def upsert_manual_news_item(
|
||||
db: AsyncSession,
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
item_id_override: str | None = None,
|
||||
group_id: str | None = None,
|
||||
) -> ManualNewsWriteResult:
|
||||
item, target, content = parsed_manual_news_item(payload, item_id_override=item_id_override)
|
||||
group = await resolve_manual_news_group(db, group_id or payload.get("group_id"))
|
||||
existing = await db.get(EarthNewsItem, item.id)
|
||||
created = existing is None
|
||||
patch = build_target_location_patch(item, target) if target else build_anchor_location_patch(item)
|
||||
patch_meta = dict(patch.get("location_meta") or {})
|
||||
patch_news_meta = dict(patch_meta.get("news_meta") or {})
|
||||
patch_news_meta["feed_type"] = NewsSourceType.MANUAL.value
|
||||
patch_news_meta["source_type"] = NewsSourceType.MANUAL.value
|
||||
patch_news_meta["manual_group_id"] = group.id
|
||||
patch_news_meta["manual_group_name"] = group.name
|
||||
patch_meta["news_meta"] = patch_news_meta
|
||||
patch["location_meta"] = patch_meta
|
||||
now = datetime.now(UTC)
|
||||
record = existing or EarthNewsItem(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
summary=item.summary,
|
||||
content_language=item.content_language,
|
||||
localizations=dict(item.localizations or {}),
|
||||
url=item.url,
|
||||
source=item.source,
|
||||
feed_name=item.feed_name,
|
||||
region=item.feed_region,
|
||||
homepage_url=item.homepage_url,
|
||||
published_at=item.published_at,
|
||||
latitude=patch["latitude"],
|
||||
longitude=patch["longitude"],
|
||||
location_label=patch["location_label"],
|
||||
location_source=patch["location_source"],
|
||||
verified=patch["verified"],
|
||||
location_meta=patch["location_meta"],
|
||||
first_seen_at=now,
|
||||
last_seen_at=now,
|
||||
resolved_at=now if patch["verified"] else None,
|
||||
enrichment_status=item.enrichment_status,
|
||||
)
|
||||
if existing is None:
|
||||
db.add(record)
|
||||
else:
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口编辑。")
|
||||
record.title = item.title
|
||||
record.summary = item.summary
|
||||
record.content_language = item.content_language
|
||||
record.localizations = dict(item.localizations or {})
|
||||
record.url = item.url
|
||||
record.source = item.source
|
||||
record.feed_name = item.feed_name
|
||||
record.region = item.feed_region
|
||||
record.homepage_url = item.homepage_url
|
||||
record.published_at = item.published_at
|
||||
record.last_seen_at = now
|
||||
if target is None and record.location_source == "manual_location":
|
||||
merged_meta = dict(record.location_meta or {})
|
||||
patch_meta = patch.get("location_meta") if isinstance(patch, dict) else None
|
||||
patch_news_meta = patch_meta.get("news_meta") if isinstance(patch_meta, dict) else None
|
||||
if isinstance(patch_news_meta, dict):
|
||||
merged_meta["news_meta"] = patch_news_meta
|
||||
record.location_meta = merged_meta
|
||||
else:
|
||||
record.location_meta = patch["location_meta"]
|
||||
if target:
|
||||
record.latitude = patch["latitude"]
|
||||
record.longitude = patch["longitude"]
|
||||
record.location_label = patch["location_label"]
|
||||
record.location_source = patch["location_source"]
|
||||
record.verified = patch["verified"]
|
||||
record.resolved_at = now
|
||||
elif record.location_source != "manual_location":
|
||||
record.latitude = patch["latitude"]
|
||||
record.longitude = patch["longitude"]
|
||||
record.location_label = patch["location_label"]
|
||||
record.location_source = patch["location_source"]
|
||||
record.verified = patch["verified"]
|
||||
record.resolved_at = None
|
||||
record.enrichment_status = NewsEnrichmentStatus.PENDING.value
|
||||
record.enrichment_error = None
|
||||
record.enriched_at = None
|
||||
if content:
|
||||
meta = dict(record.location_meta or {})
|
||||
meta["manual_content"] = content
|
||||
record.location_meta = meta
|
||||
await db.flush()
|
||||
|
||||
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
|
||||
if queued:
|
||||
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
|
||||
await db.flush()
|
||||
return ManualNewsWriteResult(item=record, created=created, queued=queued)
|
||||
|
||||
|
||||
async def import_manual_news_items(
|
||||
db: AsyncSession,
|
||||
payload: list[Any],
|
||||
*,
|
||||
group_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if len(payload) > MANUAL_NEWS_MAX_IMPORT_ITEMS:
|
||||
raise ValueError(f"单次最多导入 {MANUAL_NEWS_MAX_IMPORT_ITEMS} 条。")
|
||||
created = 0
|
||||
updated = 0
|
||||
queued = 0
|
||||
errors: list[dict[str, Any]] = []
|
||||
for index, raw_item in enumerate(payload):
|
||||
if not isinstance(raw_item, dict):
|
||||
errors.append({"index": index, "error": "条目必须是 JSON 对象。"})
|
||||
continue
|
||||
try:
|
||||
result = await upsert_manual_news_item(db, raw_item, group_id=group_id)
|
||||
created += 1 if result.created else 0
|
||||
updated += 0 if result.created else 1
|
||||
queued += 1 if result.queued else 0
|
||||
except Exception as exc:
|
||||
errors.append({"index": index, "error": str(exc)})
|
||||
if errors and created == 0 and updated == 0:
|
||||
raise ValueError("导入失败,未写入任何新闻。")
|
||||
return {"created": created, "updated": updated, "queued": queued, "failed": len(errors), "errors": errors}
|
||||
|
||||
|
||||
async def parse_manual_news_import_upload(raw_bytes: bytes) -> list[Any]:
|
||||
try:
|
||||
payload = json.loads(raw_bytes.decode("utf-8-sig"))
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ValueError("JSON 文件必须使用 UTF-8 编码。") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"JSON 解析失败:第 {exc.lineno} 行第 {exc.colno} 列。") from exc
|
||||
if not isinstance(payload, list):
|
||||
raise ValueError("JSON 顶层必须是数组。")
|
||||
return payload
|
||||
|
||||
|
||||
def serialize_news_record(record: EarthNewsItem, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
|
||||
item = record_to_parsed_news_item(record)
|
||||
payload = _serialize_item(item, active_region=item.feed_region, locale=locale)
|
||||
news_meta = _news_meta(record)
|
||||
payload["editable"] = _manual_editable(record)
|
||||
payload["source_type"] = payload.get("feed_type")
|
||||
payload["status"] = record.enrichment_status
|
||||
payload["translated"] = bool((record.localizations or {}).get("zh-CN") and (record.localizations or {}).get("en-US"))
|
||||
payload["manual_content"] = (record.location_meta or {}).get("manual_content") if isinstance(record.location_meta, dict) else None
|
||||
payload["manual_group_id"] = news_meta.get("manual_group_id")
|
||||
payload["manual_group_name"] = news_meta.get("manual_group_name")
|
||||
return payload
|
||||
|
||||
|
||||
def _record_matches_group(record: EarthNewsItem, group_id: str) -> bool:
|
||||
source_type = _record_source_type(record)
|
||||
if source_type == NewsSourceType.MANUAL.value:
|
||||
return _record_manual_group_id(record) == group_id
|
||||
return _rss_group_id(record) == group_id
|
||||
|
||||
|
||||
async def list_news_records(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
page: int,
|
||||
page_size: int,
|
||||
source_type: str | None = None,
|
||||
region: str | None = None,
|
||||
category: str | None = None,
|
||||
status_filter: str | None = None,
|
||||
group_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
page = max(page, 1)
|
||||
page_size = min(max(page_size, 1), 100)
|
||||
query = select(EarthNewsItem)
|
||||
count_query = select(func.count(EarthNewsItem.id))
|
||||
filters = []
|
||||
if region and region != "all":
|
||||
filters.append(EarthNewsItem.region == region)
|
||||
if status_filter and status_filter != "all":
|
||||
filters.append(EarthNewsItem.enrichment_status == status_filter)
|
||||
if source_type and source_type != "all":
|
||||
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("feed_type") == source_type)
|
||||
if category and category != "all":
|
||||
filters.append(EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("category") == category)
|
||||
for clause in filters:
|
||||
query = query.where(clause)
|
||||
count_query = count_query.where(clause)
|
||||
ordered_query = query.order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
|
||||
if group_id:
|
||||
result = await db.execute(ordered_query)
|
||||
all_records = [record for record in result.scalars().all() if _record_matches_group(record, group_id)]
|
||||
total = len(all_records)
|
||||
records = all_records[(page - 1) * page_size : page * page_size]
|
||||
else:
|
||||
total_result = await db.execute(count_query)
|
||||
result = await db.execute(
|
||||
ordered_query.offset((page - 1) * page_size).limit(page_size)
|
||||
)
|
||||
records = list(result.scalars().all())
|
||||
total = int(total_result.scalar() or 0)
|
||||
return {
|
||||
"items": [serialize_news_record(record) for record in records],
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
async def list_news_groups(db: AsyncSession, *, locale: str = DEFAULT_NEWS_LOCALE) -> dict[str, Any]:
|
||||
manual_groups = await get_manual_news_groups(db)
|
||||
manual_by_id: dict[str, dict[str, Any]] = {
|
||||
str(group["id"]): {
|
||||
"id": str(group["id"]),
|
||||
"name": str(group["name"]),
|
||||
"group_type": "manual",
|
||||
"source_type": NewsSourceType.MANUAL.value,
|
||||
"editable": True,
|
||||
"sort_order": int(group.get("sort_order") or 0),
|
||||
"count": 0,
|
||||
"items": [],
|
||||
}
|
||||
for group in manual_groups
|
||||
}
|
||||
rss_by_id: dict[str, dict[str, Any]] = {}
|
||||
result = await db.execute(
|
||||
select(EarthNewsItem).order_by(EarthNewsItem.published_at.desc().nullslast(), EarthNewsItem.last_seen_at.desc())
|
||||
)
|
||||
for record in result.scalars().all():
|
||||
serialized = serialize_news_record(record, locale=locale)
|
||||
source_type = _record_source_type(record)
|
||||
if source_type == NewsSourceType.MANUAL.value:
|
||||
group_id = _record_manual_group_id(record)
|
||||
group = manual_by_id.setdefault(
|
||||
group_id,
|
||||
{
|
||||
"id": group_id,
|
||||
"name": str(_news_meta(record).get("manual_group_name") or DEFAULT_MANUAL_NEWS_GROUP_NAME),
|
||||
"group_type": "manual",
|
||||
"source_type": NewsSourceType.MANUAL.value,
|
||||
"editable": True,
|
||||
"sort_order": len(manual_by_id),
|
||||
"count": 0,
|
||||
"items": [],
|
||||
},
|
||||
)
|
||||
else:
|
||||
group_id = _rss_group_id(record)
|
||||
group = rss_by_id.setdefault(
|
||||
group_id,
|
||||
{
|
||||
"id": group_id,
|
||||
"name": record.feed_name or record.source or "RSS 新闻",
|
||||
"group_type": "rss",
|
||||
"source_type": source_type,
|
||||
"editable": False,
|
||||
"region": record.region,
|
||||
"source": record.source,
|
||||
"feed_name": record.feed_name,
|
||||
"count": 0,
|
||||
"items": [],
|
||||
},
|
||||
)
|
||||
group["count"] = int(group.get("count") or 0) + 1
|
||||
group.setdefault("items", []).append(serialized)
|
||||
manual_items = sorted(manual_by_id.values(), key=lambda item: (int(item.get("sort_order") or 0), str(item.get("name") or "")))
|
||||
rss_items = sorted(rss_by_id.values(), key=lambda item: str(item.get("name") or ""))
|
||||
return {"groups": [*manual_items, *rss_items], "manual_groups": manual_items, "rss_groups": rss_items}
|
||||
|
||||
|
||||
async def get_news_record_or_404(db: AsyncSession, item_id: str) -> EarthNewsItem | None:
|
||||
return await db.get(EarthNewsItem, item_id)
|
||||
|
||||
|
||||
async def delete_manual_news_item(db: AsyncSession, item_id: str) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口删除。")
|
||||
await db.execute(delete(EarthNewsItem).where(EarthNewsItem.id == item_id))
|
||||
await db.flush()
|
||||
return True
|
||||
|
||||
|
||||
async def reprocess_manual_news_item(db: AsyncSession, item_id: str) -> bool:
|
||||
record = await db.get(EarthNewsItem, item_id)
|
||||
if record is None:
|
||||
return False
|
||||
if not _manual_editable(record):
|
||||
raise PermissionError("RSS 新闻不允许通过手动新闻接口重新处理。")
|
||||
item = record_to_parsed_news_item(record)
|
||||
queued = await enqueue_target_location_job(build_target_location_job_payload(item), force=True)
|
||||
if queued:
|
||||
record.enrichment_status = NewsEnrichmentStatus.QUEUED.value
|
||||
record.enrichment_error = None
|
||||
await db.flush()
|
||||
return queued
|
||||
|
||||
|
||||
async def broadcast_manual_news_changed() -> None:
|
||||
await _broadcast_news_reload()
|
||||
@@ -14,10 +14,14 @@ from app.core.logging import get_logger
|
||||
logger = get_logger(__name__, service="earth_news")
|
||||
|
||||
TARGET_LOCATION_STREAM = "earth_news:target_location:jobs"
|
||||
TARGET_LOCATION_PRIORITY_STREAM = "earth_news:target_location:priority"
|
||||
TARGET_LOCATION_GROUP = "earth_news_target_location"
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM = "earth_news:target_location:dead"
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS = 60 * 60 * 12
|
||||
TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS = 60 * 60 * 6
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS = 60 * 5
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS = 2 * 60 * 1000
|
||||
TARGET_LOCATION_PRIORITY_READ_BLOCK_MS = 1
|
||||
TARGET_LOCATION_MAX_ATTEMPTS = 3
|
||||
|
||||
_redis_client: redis.Redis | None = None
|
||||
@@ -28,6 +32,7 @@ class NewsTargetLocationMessage:
|
||||
message_id: str
|
||||
item_id: str
|
||||
payload: dict[str, Any]
|
||||
stream_name: str = TARGET_LOCATION_STREAM
|
||||
attempts: int = 0
|
||||
|
||||
|
||||
@@ -44,7 +49,7 @@ class NewsTargetLocationQueue(Protocol):
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
...
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
...
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
@@ -71,6 +76,10 @@ def _queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:queued:{item_id}"
|
||||
|
||||
|
||||
def _priority_queued_key(item_id: str) -> str:
|
||||
return f"earth_news:target_location:priority_queued:{item_id}"
|
||||
|
||||
|
||||
class RedisStreamsNewsTargetLocationQueue:
|
||||
def __init__(self, client: redis.Redis | None = None) -> None:
|
||||
self.client = client or _get_redis_client()
|
||||
@@ -79,34 +88,44 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
async def _ensure_group(self) -> None:
|
||||
if self._group_ready:
|
||||
return
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
TARGET_LOCATION_STREAM,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
for stream_name in (TARGET_LOCATION_PRIORITY_STREAM, TARGET_LOCATION_STREAM):
|
||||
try:
|
||||
await self.client.xgroup_create(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
id="0",
|
||||
mkstream=True,
|
||||
)
|
||||
except ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
self._group_ready = True
|
||||
|
||||
async def enqueue(self, *, item_id: str, payload: dict[str, Any], force: bool = False) -> bool:
|
||||
await self._ensure_group()
|
||||
if force:
|
||||
await self.client.delete(_result_key(item_id), _queued_key(item_id))
|
||||
await self.client.delete(_result_key(item_id))
|
||||
queued_key = _priority_queued_key(item_id)
|
||||
elif await self.client.exists(_result_key(item_id)):
|
||||
return False
|
||||
else:
|
||||
queued_key = _queued_key(item_id)
|
||||
dedup_ttl = (
|
||||
TARGET_LOCATION_PRIORITY_JOB_DEDUP_TTL_SECONDS
|
||||
if force
|
||||
else TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS
|
||||
)
|
||||
queued = await self.client.set(
|
||||
_queued_key(item_id),
|
||||
queued_key,
|
||||
"1",
|
||||
nx=True,
|
||||
ex=TARGET_LOCATION_JOB_DEDUP_TTL_SECONDS,
|
||||
ex=dedup_ttl,
|
||||
)
|
||||
if not queued:
|
||||
return bool(await self.client.exists(_queued_key(item_id)))
|
||||
return bool(await self.client.exists(queued_key))
|
||||
stream_name = TARGET_LOCATION_PRIORITY_STREAM if force else TARGET_LOCATION_STREAM
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
stream_name,
|
||||
{
|
||||
"item_id": item_id,
|
||||
"attempts": "0",
|
||||
@@ -123,39 +142,104 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
block_ms: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
await self._ensure_group()
|
||||
streams = await self.client.xreadgroup(
|
||||
streams = []
|
||||
priority_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_PRIORITY_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if priority_claimed:
|
||||
return priority_claimed
|
||||
|
||||
priority_messages = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
{TARGET_LOCATION_PRIORITY_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
block=TARGET_LOCATION_PRIORITY_READ_BLOCK_MS,
|
||||
)
|
||||
if priority_messages:
|
||||
streams = priority_messages
|
||||
else:
|
||||
regular_claimed = await self._claim_stale_messages(
|
||||
stream_name=TARGET_LOCATION_STREAM,
|
||||
consumer_name=consumer_name,
|
||||
count=count,
|
||||
)
|
||||
if regular_claimed:
|
||||
return regular_claimed
|
||||
streams = await self.client.xreadgroup(
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
{TARGET_LOCATION_STREAM: ">"},
|
||||
count=count,
|
||||
block=block_ms,
|
||||
)
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for _stream_name, stream_messages in streams:
|
||||
for stream_name, stream_messages in streams:
|
||||
for message_id, fields in stream_messages:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self.ack(message_id)
|
||||
continue
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
messages.append(
|
||||
NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
attempts=attempts,
|
||||
)
|
||||
)
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def ack(self, message_id: str) -> None:
|
||||
await self.client.xack(TARGET_LOCATION_STREAM, TARGET_LOCATION_GROUP, message_id)
|
||||
async def _claim_stale_messages(
|
||||
self,
|
||||
*,
|
||||
stream_name: str,
|
||||
consumer_name: str,
|
||||
count: int,
|
||||
) -> list[NewsTargetLocationMessage]:
|
||||
try:
|
||||
_next_id, claimed, _deleted = await self.client.xautoclaim(
|
||||
stream_name,
|
||||
TARGET_LOCATION_GROUP,
|
||||
consumer_name,
|
||||
TARGET_LOCATION_PENDING_RECLAIM_IDLE_MS,
|
||||
start_id="0-0",
|
||||
count=count,
|
||||
)
|
||||
except ResponseError:
|
||||
return []
|
||||
messages: list[NewsTargetLocationMessage] = []
|
||||
for message_id, fields in claimed:
|
||||
message = await self._message_from_fields(stream_name, message_id, fields)
|
||||
if message is not None:
|
||||
messages.append(message)
|
||||
return messages
|
||||
|
||||
async def _message_from_fields(
|
||||
self,
|
||||
stream_name: str,
|
||||
message_id: str,
|
||||
fields: dict[str, str],
|
||||
) -> NewsTargetLocationMessage | None:
|
||||
raw_payload = fields.get("payload")
|
||||
item_id = fields.get("item_id")
|
||||
if not raw_payload or not item_id:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
try:
|
||||
payload = json.loads(raw_payload)
|
||||
except json.JSONDecodeError:
|
||||
await self._discard_message(stream_name, message_id)
|
||||
return None
|
||||
attempts = int(fields.get("attempts") or 0)
|
||||
return NewsTargetLocationMessage(
|
||||
message_id=message_id,
|
||||
item_id=item_id,
|
||||
payload=payload,
|
||||
stream_name=stream_name,
|
||||
attempts=attempts,
|
||||
)
|
||||
|
||||
async def ack(self, message: NewsTargetLocationMessage) -> None:
|
||||
await self.client.xack(message.stream_name, TARGET_LOCATION_GROUP, message.message_id)
|
||||
await self.client.xdel(message.stream_name, message.message_id)
|
||||
|
||||
async def _discard_message(self, stream_name: str, message_id: str) -> None:
|
||||
await self.client.xack(stream_name, TARGET_LOCATION_GROUP, message_id)
|
||||
await self.client.xdel(stream_name, message_id)
|
||||
|
||||
async def retry_or_dead_letter(
|
||||
self,
|
||||
@@ -163,7 +247,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
*,
|
||||
error: str,
|
||||
) -> None:
|
||||
await self.ack(message.message_id)
|
||||
await self.ack(message)
|
||||
if message.attempts + 1 >= TARGET_LOCATION_MAX_ATTEMPTS:
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_DEAD_LETTER_STREAM,
|
||||
@@ -176,7 +260,7 @@ class RedisStreamsNewsTargetLocationQueue:
|
||||
)
|
||||
return
|
||||
await self.client.xadd(
|
||||
TARGET_LOCATION_STREAM,
|
||||
message.stream_name,
|
||||
{
|
||||
"item_id": message.item_id,
|
||||
"attempts": str(message.attempts + 1),
|
||||
@@ -231,4 +315,4 @@ async def save_target_location_patch(item_id: str, patch: dict[str, Any]) -> Non
|
||||
TARGET_LOCATION_RESULT_TTL_SECONDS,
|
||||
json.dumps(patch, ensure_ascii=False),
|
||||
)
|
||||
await client.delete(_queued_key(item_id))
|
||||
await client.delete(_queued_key(item_id), _priority_queued_key(item_id))
|
||||
|
||||
@@ -19,6 +19,17 @@ from app.services.earth_news_classification import (
|
||||
normalize_breaking_scope,
|
||||
)
|
||||
|
||||
CRUISE_REGION_ORDER = (
|
||||
"americas",
|
||||
"europe",
|
||||
"middle-east-africa",
|
||||
"asia-pacific",
|
||||
"global",
|
||||
)
|
||||
CRUISE_REGION_QUERY_MULTIPLIER = 12
|
||||
CRUISE_REGION_QUERY_MIN_LIMIT = 240
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT = 1000
|
||||
|
||||
|
||||
def _coerce_datetime(value: datetime | None) -> datetime | None:
|
||||
if value is None:
|
||||
@@ -106,6 +117,41 @@ def _sort_parsed_news_items(items: list[ParsedNewsItem], *, active_region: str)
|
||||
)
|
||||
|
||||
|
||||
def _diversify_parsed_news_items_by_region(
|
||||
items: list[ParsedNewsItem],
|
||||
*,
|
||||
limit: int,
|
||||
) -> list[ParsedNewsItem]:
|
||||
if limit <= 0:
|
||||
return []
|
||||
sorted_items = _sort_parsed_news_items(items, active_region="global")
|
||||
buckets: dict[str, list[ParsedNewsItem]] = {}
|
||||
for item in sorted_items:
|
||||
region = item.feed_region or "global"
|
||||
buckets.setdefault(region, []).append(item)
|
||||
|
||||
ordered_regions = [
|
||||
*[region for region in CRUISE_REGION_ORDER if buckets.get(region)],
|
||||
*sorted(region for region in buckets if region not in CRUISE_REGION_ORDER),
|
||||
]
|
||||
diversified: list[ParsedNewsItem] = []
|
||||
cursor = 0
|
||||
while len(diversified) < limit:
|
||||
added = False
|
||||
for region in ordered_regions:
|
||||
bucket = buckets.get(region) or []
|
||||
if cursor >= len(bucket):
|
||||
continue
|
||||
diversified.append(bucket[cursor])
|
||||
added = True
|
||||
if len(diversified) >= limit:
|
||||
break
|
||||
if not added:
|
||||
break
|
||||
cursor += 1
|
||||
return diversified
|
||||
|
||||
|
||||
def _query_sort_key(active_region: str):
|
||||
if active_region == "global":
|
||||
return (
|
||||
@@ -133,42 +179,6 @@ def _source_filter_clause(source_ids: set[str] | None):
|
||||
return EarthNewsItem.location_meta.op("->")("news_meta").op("->>")("source_id").in_(sorted(source_ids))
|
||||
|
||||
|
||||
def _record_source_id(record: EarthNewsItem) -> str:
|
||||
location_meta = dict(record.location_meta or {})
|
||||
news_meta = location_meta.get("news_meta") if isinstance(location_meta.get("news_meta"), dict) else {}
|
||||
source_id = str(news_meta.get("source_id") or "").strip()
|
||||
if source_id:
|
||||
return source_id
|
||||
if isinstance(record.id, str) and ":" in record.id:
|
||||
return record.id.split(":", 1)[0]
|
||||
return record.feed_name or record.source or record.id
|
||||
|
||||
|
||||
def _diversify_records_by_source(records: list[EarthNewsItem], *, limit: int) -> list[EarthNewsItem]:
|
||||
if limit <= 0 or len(records) <= limit:
|
||||
return records[:limit]
|
||||
buckets: dict[str, list[EarthNewsItem]] = {}
|
||||
order: list[str] = []
|
||||
for record in records:
|
||||
source_id = _record_source_id(record)
|
||||
if source_id not in buckets:
|
||||
buckets[source_id] = []
|
||||
order.append(source_id)
|
||||
buckets[source_id].append(record)
|
||||
|
||||
diversified: list[EarthNewsItem] = []
|
||||
while len(diversified) < limit and order:
|
||||
next_order: list[str] = []
|
||||
for source_id in order:
|
||||
bucket = buckets.get(source_id) or []
|
||||
if bucket and len(diversified) < limit:
|
||||
diversified.append(bucket.pop(0))
|
||||
if bucket:
|
||||
next_order.append(source_id)
|
||||
order = next_order
|
||||
return diversified
|
||||
|
||||
|
||||
async def list_earth_news_items(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -177,7 +187,7 @@ async def list_earth_news_items(
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
query_limit = limit if source_ids else min(max(limit * 8, limit), 200)
|
||||
query_limit = limit if source_ids else min(max(limit * 20, limit), 500)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(*_query_sort_key(active_region))
|
||||
@@ -203,6 +213,8 @@ async def list_earth_news_items(
|
||||
[record_to_parsed_news_item(record) for record in records],
|
||||
active_region=active_region,
|
||||
)
|
||||
if active_region == "global" and not source_ids:
|
||||
return _diversify_parsed_news_items_by_region(items, limit=limit)
|
||||
return items[:limit]
|
||||
|
||||
|
||||
@@ -213,15 +225,20 @@ async def list_earth_news_cruise_items(
|
||||
categories: set[str] | None = None,
|
||||
source_ids: set[str] | None = None,
|
||||
) -> list[ParsedNewsItem]:
|
||||
query_limit = min(
|
||||
max(limit * CRUISE_REGION_QUERY_MULTIPLIER, CRUISE_REGION_QUERY_MIN_LIMIT),
|
||||
CRUISE_REGION_QUERY_MAX_LIMIT,
|
||||
)
|
||||
query = (
|
||||
select(EarthNewsItem)
|
||||
.order_by(
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.published_at.is_(None),
|
||||
EarthNewsItem.published_at.desc().nullslast(),
|
||||
EarthNewsItem.last_seen_at.desc(),
|
||||
EarthNewsItem.region.asc(),
|
||||
EarthNewsItem.feed_name.asc(),
|
||||
)
|
||||
.limit(min(max(limit * 4, limit), 200))
|
||||
.limit(query_limit)
|
||||
)
|
||||
category_clause = _category_filter_clause(categories)
|
||||
if category_clause is not None:
|
||||
@@ -230,11 +247,10 @@ async def list_earth_news_cruise_items(
|
||||
if source_clause is not None:
|
||||
query = query.where(source_clause)
|
||||
result = await db.execute(query)
|
||||
items = _sort_parsed_news_items(
|
||||
return _diversify_parsed_news_items_by_region(
|
||||
[record_to_parsed_news_item(record) for record in result.scalars().all()],
|
||||
active_region="global",
|
||||
limit=limit,
|
||||
)
|
||||
return items[:limit]
|
||||
|
||||
|
||||
async def get_earth_news_freshness(
|
||||
@@ -382,13 +398,28 @@ async def update_earth_news_item_enrichment(
|
||||
if record is None:
|
||||
return False
|
||||
if "latitude" in patch:
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = dict(patch.get("location_meta") or {})
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
patch_meta = dict(patch.get("location_meta") or {})
|
||||
if record.location_source == "manual_location":
|
||||
current_meta = dict(record.location_meta or {})
|
||||
patch_news_meta = patch_meta.get("news_meta")
|
||||
if isinstance(patch_news_meta, dict):
|
||||
current_meta["news_meta"] = patch_news_meta
|
||||
current_meta["manual_enrichment"] = {
|
||||
"resolution_stage": patch_meta.get("resolution_stage"),
|
||||
"ai_attempted": patch_meta.get("ai_attempted"),
|
||||
"ai_status": patch_meta.get("ai_status"),
|
||||
"ai_error": patch_meta.get("ai_error"),
|
||||
"debug_note": patch_meta.get("debug_note"),
|
||||
}
|
||||
record.location_meta = current_meta
|
||||
else:
|
||||
record.latitude = float(patch["latitude"])
|
||||
record.longitude = float(patch["longitude"])
|
||||
record.location_label = str(patch["location_label"])
|
||||
record.location_source = str(patch["location_source"])
|
||||
record.verified = bool(patch["verified"])
|
||||
record.location_meta = patch_meta
|
||||
record.resolved_at = datetime.now(UTC) if record.verified else None
|
||||
if "content_language" in patch:
|
||||
record.content_language = str(patch.get("content_language") or "en")
|
||||
if "localizations" in patch:
|
||||
|
||||
@@ -29,6 +29,9 @@ logger = get_logger(__name__, service="earth_news")
|
||||
WORKER_BATCH_SIZE = 4
|
||||
WORKER_BLOCK_MS = 5000
|
||||
WORKER_BACKOFF_SECONDS = 5.0
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS = 20.0
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS = 90.0
|
||||
WORKER_JOB_TIMEOUT_GRACE_SECONDS = 10.0
|
||||
|
||||
_worker_task: asyncio.Task | None = None
|
||||
|
||||
@@ -109,12 +112,25 @@ async def _run_target_location_worker() -> None:
|
||||
if not messages:
|
||||
continue
|
||||
provider_client = await _build_provider_client()
|
||||
for message in messages:
|
||||
job_timeout = _get_worker_job_timeout(provider_client)
|
||||
|
||||
async def handle_message(message: NewsTargetLocationMessage) -> None:
|
||||
try:
|
||||
await process_target_location_message(message, provider_client=provider_client)
|
||||
await queue.ack(message.message_id)
|
||||
await asyncio.wait_for(
|
||||
process_target_location_message(message, provider_client=provider_client),
|
||||
timeout=job_timeout,
|
||||
)
|
||||
await queue.ack(message)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except TimeoutError as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job timed out",
|
||||
event="earth_news.target_location.worker_job_timeout",
|
||||
context={"item_id": message.item_id, "timeout_seconds": job_timeout},
|
||||
)
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc) or "job timed out")
|
||||
except Exception as exc:
|
||||
logger.warning_event(
|
||||
"Earth news target location worker job failed",
|
||||
@@ -124,6 +140,8 @@ async def _run_target_location_worker() -> None:
|
||||
with suppress(Exception):
|
||||
await queue.retry_or_dead_letter(message, error=str(exc))
|
||||
|
||||
await asyncio.gather(*(handle_message(message) for message in messages))
|
||||
|
||||
|
||||
def start_earth_news_target_worker() -> None:
|
||||
global _worker_task
|
||||
@@ -140,3 +158,15 @@ async def stop_earth_news_target_worker() -> None:
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
_worker_task = None
|
||||
|
||||
|
||||
def _get_worker_job_timeout(provider_client: AIProviderClient | None) -> float:
|
||||
timeout = float(getattr(provider_client, "timeout", 0) or WORKER_JOB_TIMEOUT_MIN_SECONDS)
|
||||
retry_attempts = float(getattr(provider_client, "retry_attempts", 1) or 1)
|
||||
return min(
|
||||
max(
|
||||
timeout * retry_attempts + WORKER_JOB_TIMEOUT_GRACE_SECONDS,
|
||||
WORKER_JOB_TIMEOUT_MIN_SECONDS,
|
||||
),
|
||||
WORKER_JOB_TIMEOUT_MAX_SECONDS,
|
||||
)
|
||||
|
||||
186
backend/app/services/llm_model_catalog.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Authenticated model discovery shared by refresh and connection checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
CATALOG_TIMEOUT_SECONDS = 30
|
||||
CATALOG_MAX_PAGES = 100
|
||||
CATALOG_PAGE_SIZE = 100
|
||||
CATALOG_REQUEST_ATTEMPTS = 2
|
||||
CATALOG_RETRY_DELAY_SECONDS = 0.2
|
||||
SUPPORTED_PROVIDER_APIS = {
|
||||
"openai-completions",
|
||||
"openai-responses",
|
||||
"anthropic-messages",
|
||||
"ollama-generate",
|
||||
}
|
||||
|
||||
|
||||
class LLMProviderCatalogError(RuntimeError):
|
||||
"""A safe, user-facing catalog failure with no upstream response or credentials."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ModelCatalog:
|
||||
url: str
|
||||
models: list[str]
|
||||
|
||||
|
||||
def model_catalog_url(provider: str, base_url: str, provider_api: str) -> str:
|
||||
parts = urlsplit(base_url.strip())
|
||||
if parts.scheme not in {"http", "https"} or not parts.hostname:
|
||||
raise LLMProviderCatalogError("请填写有效的 HTTP(S) 模型基础地址。")
|
||||
if parts.username or parts.password or parts.query or parts.fragment:
|
||||
raise LLMProviderCatalogError("模型基础地址不能包含账号、密码、查询参数或片段。")
|
||||
if provider_api not in SUPPORTED_PROVIDER_APIS:
|
||||
raise LLMProviderCatalogError("当前接口协议不支持模型目录查询。")
|
||||
path = parts.path.rstrip("/")
|
||||
if provider_api == "ollama-generate":
|
||||
path = path.removesuffix("/api").removesuffix("/v1") + "/api/tags"
|
||||
elif provider == "alibaba" and parts.hostname.endswith(".aliyuncs.com"):
|
||||
path = "/api/v1/models"
|
||||
else:
|
||||
if not path or (provider_api == "anthropic-messages" and path.endswith("/anthropic")):
|
||||
path += "/v1"
|
||||
path += "/models"
|
||||
return urlunsplit((parts.scheme, parts.netloc, path, "", ""))
|
||||
|
||||
|
||||
def catalog_error_message(exc: Exception) -> str:
|
||||
if isinstance(exc, LLMProviderCatalogError):
|
||||
return str(exc)
|
||||
if isinstance(exc, (httpx.TimeoutException, TimeoutError)):
|
||||
return "模型目录请求超时,请检查网络后重试。"
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
code = exc.response.status_code
|
||||
messages = {
|
||||
401: "API Key 验证失败,请检查当前供应商的凭证。",
|
||||
403: "当前 API Key 无权访问该模型目录,请检查账号权限和服务地域。",
|
||||
404: "模型目录接口不存在,请检查基础地址、地域和接口协议。",
|
||||
429: "供应商请求限流,请稍后重试。",
|
||||
}
|
||||
return messages.get(code, f"供应商模型目录返回 HTTP {code},请稍后重试。")
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return "无法连接模型目录,请检查基础地址和网络。"
|
||||
return "模型目录响应无效,请稍后重试。"
|
||||
|
||||
|
||||
def _model_rows(payload: object) -> tuple[list[dict[str, object]], dict[str, object]]:
|
||||
if not isinstance(payload, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
envelope = payload.get("output", payload)
|
||||
if not isinstance(envelope, dict):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录。")
|
||||
rows = envelope.get("data", envelope.get("models"))
|
||||
if not isinstance(rows, list):
|
||||
raise LLMProviderCatalogError("供应商响应中没有模型列表。")
|
||||
if any(not isinstance(row, dict) for row in rows):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型条目。")
|
||||
return rows, envelope
|
||||
|
||||
|
||||
def _model_id(row: dict[str, object]) -> str:
|
||||
value = row.get("id") or row.get("model") or row.get("name")
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise LLMProviderCatalogError("供应商返回了缺少 ID 的模型条目。")
|
||||
return value.strip()
|
||||
|
||||
|
||||
def _model_date(row: dict[str, object]) -> float:
|
||||
value = row.get("created_at") or row.get("published_time") or row.get("created")
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).timestamp()
|
||||
except ValueError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
async def _get_catalog_page(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
headers: dict[str, str],
|
||||
params: dict[str, str | int],
|
||||
) -> object:
|
||||
for attempt in range(CATALOG_REQUEST_ATTEMPTS):
|
||||
try:
|
||||
response = await client.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
if attempt or exc.response.status_code not in {502, 503, 504}:
|
||||
raise
|
||||
except httpx.TransportError:
|
||||
if attempt:
|
||||
raise
|
||||
await asyncio.sleep(CATALOG_RETRY_DELAY_SECONDS)
|
||||
raise LLMProviderCatalogError("模型目录请求失败。")
|
||||
|
||||
|
||||
async def fetch_model_catalog(
|
||||
provider: str,
|
||||
base_url: str,
|
||||
provider_api: str,
|
||||
api_key: str = "",
|
||||
anthropic_version: str = "2023-06-01",
|
||||
timeout_seconds: int = CATALOG_TIMEOUT_SECONDS,
|
||||
) -> ModelCatalog:
|
||||
url = model_catalog_url(provider, base_url, provider_api)
|
||||
public_catalog = provider in {"opencode-go", "openrouter"}
|
||||
if not api_key and provider_api != "ollama-generate" and not public_catalog:
|
||||
raise LLMProviderCatalogError("请先配置当前供应商的 API Key,再刷新模型列表。")
|
||||
headers = {"User-Agent": "Planet/1.0", "Accept": "application/json"}
|
||||
if provider_api == "anthropic-messages" and provider not in {
|
||||
"opencode-go",
|
||||
"openrouter",
|
||||
"alibaba",
|
||||
"moonshotai",
|
||||
}:
|
||||
headers.update({"x-api-key": api_key, "anthropic-version": anthropic_version})
|
||||
elif api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
native_dashscope = provider == "alibaba" and urlsplit(url).hostname.endswith(".aliyuncs.com")
|
||||
params: dict[str, str | int] = {}
|
||||
if native_dashscope:
|
||||
params = {"page_no": 1, "page_size": CATALOG_PAGE_SIZE, "capabilities": "TG"}
|
||||
rows_by_id: dict[str, dict[str, object]] = {}
|
||||
timeout = max(1, min(timeout_seconds, CATALOG_TIMEOUT_SECONDS))
|
||||
# Bound the complete pagination/retry cycle, not just each individual request.
|
||||
async with asyncio.timeout(timeout), httpx.AsyncClient(timeout=timeout) as client:
|
||||
for page in range(CATALOG_MAX_PAGES):
|
||||
payload = await _get_catalog_page(client, url, headers, params)
|
||||
rows, envelope = _model_rows(payload)
|
||||
previous_count = len(rows_by_id)
|
||||
for row in rows:
|
||||
rows_by_id[_model_id(row)] = row
|
||||
has_more = envelope.get("has_more") is True
|
||||
if native_dashscope:
|
||||
total = envelope.get("total")
|
||||
if not isinstance(total, int) or total < 0:
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
has_more = len(rows_by_id) < total
|
||||
params["page_no"] = page + 2
|
||||
elif has_more:
|
||||
cursor = envelope.get("last_id")
|
||||
if not isinstance(cursor, str) or not cursor or cursor == params.get("after_id"):
|
||||
raise LLMProviderCatalogError("供应商返回了无效的模型目录分页信息。")
|
||||
params["after_id"] = cursor
|
||||
if not has_more:
|
||||
models = sorted(
|
||||
rows_by_id, key=lambda key: _model_date(rows_by_id[key]), reverse=True
|
||||
)
|
||||
# An empty Ollama catalog is valid: no models have been installed yet.
|
||||
if not models and provider_api != "ollama-generate":
|
||||
raise LLMProviderCatalogError("供应商返回了空模型目录,已保留上次模型列表。")
|
||||
return ModelCatalog(url=url, models=models)
|
||||
if len(rows_by_id) == previous_count:
|
||||
raise LLMProviderCatalogError("供应商模型目录分页没有进展,已保留上次模型列表。")
|
||||
raise LLMProviderCatalogError("供应商模型目录分页超过限制,已保留上次模型列表。")
|
||||
@@ -3,17 +3,30 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
|
||||
MODELS_DEV_URL = "https://models.dev/api.json"
|
||||
OPENCODE_GO_MODELS_URL = "https://opencode.ai/zen/go/v1/models"
|
||||
from app.services.llm_model_catalog import fetch_model_catalog
|
||||
|
||||
OPENCODE_GO_MODEL_PROVIDER_APIS = {
|
||||
"minimax-m3": "anthropic-messages",
|
||||
"qwen3.8-max": "anthropic-messages",
|
||||
"qwen3.8-flash": "anthropic-messages",
|
||||
"qwen3.7-max": "anthropic-messages",
|
||||
"qwen3.7-plus": "anthropic-messages",
|
||||
"qwen3.6-plus": "anthropic-messages",
|
||||
"grok-4.6": "openai-responses",
|
||||
"gpt-5.6-luna": "openai-responses",
|
||||
"muse-spark-1.3-contributor": "openai-responses",
|
||||
"muse-spark-1.2-contributor": "openai-responses",
|
||||
"minimax-m2.7": "anthropic-messages",
|
||||
"minimax-m2.5": "anthropic-messages",
|
||||
}
|
||||
OPENCODE_GO_FALLBACK_MODELS = [
|
||||
"minimax-m3",
|
||||
"kimi-k3",
|
||||
"glm-5.3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"minimax-m2.7",
|
||||
"minimax-m2.5",
|
||||
"kimi-k2.6",
|
||||
@@ -29,24 +42,47 @@ OPENCODE_GO_FALLBACK_MODELS = [
|
||||
]
|
||||
|
||||
|
||||
OPENAI_MODEL_PROVIDER_APIS = {
|
||||
model: "openai-responses"
|
||||
for model in [
|
||||
"gpt-6-astra",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.1",
|
||||
"gpt-5.1-codex",
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"minimax": {
|
||||
"provider": "minimax",
|
||||
"label": "MiniMax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.minimaxi.com/anthropic",
|
||||
"model": "MiniMax-M2.7",
|
||||
"models": ["MiniMax-M2.7", "MiniMax-M2.7-highspeed", "MiniMax-M2.5", "MiniMax-M2"],
|
||||
"model": "MiniMax-M3",
|
||||
"models": [
|
||||
"MiniMax-M3",
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
"MiniMax-M2.1",
|
||||
"MiniMax-M2.1-highspeed",
|
||||
"MiniMax-M2",
|
||||
],
|
||||
"api_key_env": "MINIMAX_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
"openai": {
|
||||
"provider": "openai",
|
||||
"label": "OpenAI",
|
||||
"provider_api": "openai-completions",
|
||||
"provider_api": "openai-responses",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"model": "gpt-5.1",
|
||||
"models": ["gpt-5.1", "gpt-5.1-codex", "gpt-4.1", "gpt-4o"],
|
||||
"model": "gpt-6-astra",
|
||||
"models": ["gpt-6-astra", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna", "gpt-4.1"],
|
||||
"model_provider_apis": OPENAI_MODEL_PROVIDER_APIS,
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -55,8 +91,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Anthropic",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://api.anthropic.com/v1",
|
||||
"model": "claude-sonnet-4-6",
|
||||
"models": ["claude-sonnet-4-6", "claude-opus-4-5", "claude-3-5-haiku-20241022"],
|
||||
"model": "claude-opus-5",
|
||||
"models": ["claude-opus-5", "claude-sonnet-4-6"],
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -65,8 +101,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "DeepSeek",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"model": "deepseek-chat",
|
||||
"models": ["deepseek-chat", "deepseek-reasoner"],
|
||||
"model": "deepseek-flash",
|
||||
"models": ["deepseek-flash", "deepseek-v4-pro"],
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -75,8 +111,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Alibaba Qwen / DashScope",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
"model": "qwen3-max",
|
||||
"models": ["qwen3-max", "qwen3.5-plus", "qwen-max", "qwen-plus"],
|
||||
"model": "qwen3.8-max",
|
||||
"models": ["qwen3.8-max", "qwen3-max", "qwen-plus"],
|
||||
"api_key_env": "DASHSCOPE_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -85,8 +121,8 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
"label": "Moonshot AI / Kimi",
|
||||
"provider_api": "openai-completions",
|
||||
"base_url": "https://api.moonshot.ai/v1",
|
||||
"model": "kimi-k2.5",
|
||||
"models": ["kimi-k2.5", "kimi-k2-thinking", "kimi-k2-turbo-preview"],
|
||||
"model": "kimi-k3",
|
||||
"models": ["kimi-k3", "kimi-k2.5"],
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"source": "fallback",
|
||||
},
|
||||
@@ -123,16 +159,6 @@ FALLBACK_LLM_PROVIDER_PRESETS: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
}
|
||||
|
||||
MODELS_DEV_PROVIDER_KEYS = {
|
||||
"minimax": "minimax",
|
||||
"openai": "openai",
|
||||
"anthropic": "anthropic",
|
||||
"deepseek": "deepseek",
|
||||
"alibaba": "alibaba",
|
||||
"moonshotai": "moonshotai",
|
||||
"openrouter": "openrouter",
|
||||
}
|
||||
|
||||
|
||||
def list_fallback_llm_provider_presets() -> list[dict[str, Any]]:
|
||||
return [dict(value) for value in FALLBACK_LLM_PROVIDER_PRESETS.values()]
|
||||
@@ -152,65 +178,40 @@ def _opencode_go_model_provider_apis(model_ids: list[str]) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
async def refresh_llm_provider_preset(provider: str, api_key: str | None = None) -> dict[str, Any]:
|
||||
async def refresh_llm_provider_preset(
|
||||
provider: str,
|
||||
api_key: str | None = None,
|
||||
*,
|
||||
base_url: str | None = None,
|
||||
provider_api: str | None = None,
|
||||
anthropic_version: str = "2023-06-01",
|
||||
) -> dict[str, Any]:
|
||||
fallback = get_fallback_llm_provider_preset(provider)
|
||||
if fallback["provider"] == "opencode-go":
|
||||
headers = {"User-Agent": "Planet/1.0"}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
OPENCODE_GO_MODELS_URL,
|
||||
headers=headers,
|
||||
)
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
|
||||
data = payload.get("data") if isinstance(payload, dict) else []
|
||||
model_ids = [
|
||||
str(item.get("id"))
|
||||
for item in data
|
||||
if isinstance(item, dict) and item.get("id")
|
||||
][:120]
|
||||
if not model_ids:
|
||||
model_ids = fallback["models"]
|
||||
return {
|
||||
**fallback,
|
||||
"model": fallback["model"] if fallback["model"] in model_ids else model_ids[0],
|
||||
"models": model_ids,
|
||||
"model_provider_apis": _opencode_go_model_provider_apis(model_ids),
|
||||
"source": OPENCODE_GO_MODELS_URL,
|
||||
}
|
||||
|
||||
models_dev_key = MODELS_DEV_PROVIDER_KEYS.get(fallback["provider"])
|
||||
if not models_dev_key:
|
||||
return fallback
|
||||
|
||||
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
|
||||
response = await client.get(
|
||||
MODELS_DEV_URL,
|
||||
headers={"User-Agent": "Planet/1.0"},
|
||||
)
|
||||
response.raise_for_status()
|
||||
catalog = response.json()
|
||||
|
||||
upstream = catalog.get(models_dev_key)
|
||||
if not isinstance(upstream, dict):
|
||||
return fallback
|
||||
|
||||
upstream_models = upstream.get("models") if isinstance(upstream.get("models"), dict) else {}
|
||||
model_ids = list(upstream_models.keys())[:80]
|
||||
base_url = upstream.get("api") or fallback["base_url"]
|
||||
if fallback["provider"] == "deepseek" and base_url == "https://api.deepseek.com":
|
||||
base_url = "https://api.deepseek.com/v1"
|
||||
|
||||
refreshed = {
|
||||
resolved_base_url = base_url or fallback["base_url"]
|
||||
resolved_api = provider_api or fallback["provider_api"]
|
||||
catalog = await fetch_model_catalog(
|
||||
fallback["provider"],
|
||||
resolved_base_url,
|
||||
resolved_api,
|
||||
api_key or "",
|
||||
anthropic_version,
|
||||
)
|
||||
model_provider_apis = (
|
||||
_opencode_go_model_provider_apis(catalog.models)
|
||||
if fallback["provider"] == "opencode-go"
|
||||
else fallback.get("model_provider_apis", {})
|
||||
)
|
||||
if (
|
||||
fallback["provider"] == "openai"
|
||||
and urlsplit(resolved_base_url).hostname != "api.openai.com"
|
||||
):
|
||||
model_provider_apis = {}
|
||||
return {
|
||||
**fallback,
|
||||
"label": upstream.get("name") or fallback["label"],
|
||||
"base_url": base_url,
|
||||
"model": model_ids[0] if model_ids else fallback["model"],
|
||||
"models": model_ids or fallback["models"],
|
||||
"api_key_env": (upstream.get("env") or [fallback["api_key_env"]])[0],
|
||||
"source": MODELS_DEV_URL,
|
||||
"base_url": resolved_base_url,
|
||||
"provider_api": resolved_api,
|
||||
"model": catalog.models[0] if catalog.models else "",
|
||||
"models": catalog.models,
|
||||
"model_provider_apis": model_provider_apis,
|
||||
"source": catalog.url,
|
||||
}
|
||||
return refreshed
|
||||
|
||||
@@ -635,6 +635,7 @@ async def _run_assistant_message(
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"session_id": session_id,
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
|
||||
148
backend/app/services/tv_catalog.py
Normal file
@@ -0,0 +1,148 @@
|
||||
"""Search and paginate the public live TV catalog at the database boundary."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import Select, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.services.tv_streams import (
|
||||
TV_LIVE_SOURCE_COLLECTOR,
|
||||
TV_LIVE_SOURCE_DATA_TYPE,
|
||||
_build_collected_tv_source,
|
||||
build_public_tv_payload,
|
||||
get_tv_settings_payload,
|
||||
)
|
||||
|
||||
|
||||
def _source_key():
|
||||
return func.coalesce(
|
||||
func.nullif(CollectedData.extra_data["id"].as_string(), ""),
|
||||
func.nullif(CollectedData.source_id, ""),
|
||||
CollectedData.entity_key,
|
||||
)
|
||||
|
||||
|
||||
def _collected_catalog_query(configured_ids: list[str]) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
source_id = _source_key()
|
||||
enabled = func.lower(func.trim(func.coalesce(metadata["is_enabled"].as_string(), "true")))
|
||||
ranked = (
|
||||
select(
|
||||
CollectedData.id,
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=source_id,
|
||||
order_by=CollectedData.id.desc(),
|
||||
)
|
||||
.label("source_rank"),
|
||||
)
|
||||
.where(
|
||||
CollectedData.source == TV_LIVE_SOURCE_COLLECTOR,
|
||||
CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE,
|
||||
CollectedData.is_current.is_(True),
|
||||
CollectedData.is_valid == 1,
|
||||
CollectedData.deleted_at.is_(None),
|
||||
enabled.notin_(("false", "0", "no", "off")),
|
||||
source_id.notin_(configured_ids),
|
||||
)
|
||||
.subquery()
|
||||
)
|
||||
return (
|
||||
select(CollectedData)
|
||||
.join(ranked, ranked.c.id == CollectedData.id)
|
||||
.where(ranked.c.source_rank == 1)
|
||||
)
|
||||
|
||||
|
||||
def _filter_catalog_query(
|
||||
query: Select[tuple[CollectedData]], terms: list[str]
|
||||
) -> Select[tuple[CollectedData]]:
|
||||
metadata = CollectedData.extra_data
|
||||
searchable = func.lower(
|
||||
func.concat_ws(
|
||||
" ",
|
||||
CollectedData.name,
|
||||
CollectedData.title,
|
||||
CollectedData.source_id,
|
||||
metadata["name"].as_string(),
|
||||
metadata["provider"].as_string(),
|
||||
metadata["region"].as_string(),
|
||||
metadata["country"].as_string(),
|
||||
metadata["language"].as_string(),
|
||||
)
|
||||
)
|
||||
for term in terms:
|
||||
query = query.where(searchable.contains(term, autoescape=True))
|
||||
return query
|
||||
|
||||
|
||||
def _matches_source(source: dict[str, Any], terms: list[str]) -> bool:
|
||||
searchable = " ".join(
|
||||
str(source.get(key) or "") for key in ("id", "name", "provider", "region", "language")
|
||||
).lower()
|
||||
return all(term in searchable for term in terms)
|
||||
|
||||
|
||||
async def get_tv_catalog_page(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
offset: int = 0,
|
||||
limit: int = 50,
|
||||
q: str = "",
|
||||
selected_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
settings = await get_tv_settings_payload(db)
|
||||
payload = build_public_tv_payload(settings, [])
|
||||
configured = payload["sources"]
|
||||
query = _collected_catalog_query([source["id"] for source in settings["sources"]])
|
||||
if selected_id:
|
||||
selected = next((source for source in configured if source["id"] == selected_id), None)
|
||||
if selected is None:
|
||||
record = await db.scalar(query.where(_source_key() == selected_id).limit(1))
|
||||
selected = _build_collected_tv_source(record, 0) if record else None
|
||||
if selected:
|
||||
payload["selected_source"] = selected
|
||||
summary = (
|
||||
await db.execute(
|
||||
select(func.count(), func.max(CollectedData.collected_at))
|
||||
.select_from(CollectedData)
|
||||
.where(CollectedData.id.in_(query.with_only_columns(CollectedData.id)))
|
||||
)
|
||||
).one()
|
||||
total_collected, latest_update = summary
|
||||
terms = q.lower().split()
|
||||
matched_configured = [source for source in configured if _matches_source(source, terms)]
|
||||
filtered_query = _filter_catalog_query(query, terms)
|
||||
matched_collected = (
|
||||
await db.scalar(select(func.count()).select_from(filtered_query.subquery()))
|
||||
if terms
|
||||
else total_collected
|
||||
)
|
||||
sources = matched_configured[offset : offset + limit]
|
||||
remaining = limit - len(sources)
|
||||
if remaining:
|
||||
rows = await db.scalars(
|
||||
filtered_query.order_by(func.lower(CollectedData.name), CollectedData.id)
|
||||
.offset(max(0, offset - len(matched_configured)))
|
||||
.limit(remaining)
|
||||
)
|
||||
sources.extend(
|
||||
_build_collected_tv_source(record, index) for index, record in enumerate(rows)
|
||||
)
|
||||
total = len(matched_configured) + matched_collected
|
||||
next_offset = offset + len(sources)
|
||||
return {
|
||||
**payload,
|
||||
"sources": sources,
|
||||
"source_count": len(configured) + total_collected,
|
||||
"latest_updated_at": (
|
||||
to_iso8601_utc(latest_update) if latest_update else payload["latest_updated_at"]
|
||||
),
|
||||
"total": total,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
@@ -11,7 +11,7 @@ from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
DEFAULT_TV_SOURCE_ID = "aljazeera-mubasher"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
@@ -300,8 +300,12 @@ def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
default_source = next(
|
||||
source for source in DEFAULT_TV_SETTINGS["sources"]
|
||||
if source["id"] == DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
normalize_tv_source(default_source, index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
|
||||
@@ -614,6 +614,18 @@ async def get_current_vessels_snapshot(
|
||||
return [item.to_dict() for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def get_current_vessels_by_mmsi(
|
||||
db: AsyncSession, mmsis: list[int]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Read canonical render state for the vessels changed by a stream flush."""
|
||||
if not mmsis:
|
||||
return []
|
||||
result = await db.execute(
|
||||
select(VesselCurrentState).where(VesselCurrentState.mmsi.in_(mmsis))
|
||||
)
|
||||
return [_jsonable(item.to_dict()) for item in result.scalars().all()]
|
||||
|
||||
|
||||
async def aggregate_vessel_observations(
|
||||
db: AsyncSession,
|
||||
observations: Iterable[AISRawObservation],
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
[pytest]
|
||||
pythonpath = ..
|
||||
asyncio_mode = auto
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
|
||||
@@ -63,6 +63,21 @@ def test_build_earth_update_maps_derived_tables_to_layers():
|
||||
assert vessel_update is not None
|
||||
assert vessel_update["source"] == "vessel_position"
|
||||
assert vessel_update["layers"] == ["vessels"]
|
||||
assert vessel_update["refresh_strategy"] == "reload"
|
||||
|
||||
|
||||
def test_vessel_stream_changes_do_not_request_full_layer_rebuilds():
|
||||
from app.services.earth_db_change_listener import PendingEarthDbChange
|
||||
|
||||
for source in ("aisstream_vessels", "barentswatch_vessels"):
|
||||
update = build_earth_update_from_db_payload({
|
||||
"source": source, "table": "vessel_current_state", "operation": "UPDATE",
|
||||
})
|
||||
assert update["refresh_strategy"] == "delta"
|
||||
pending = PendingEarthDbChange(source=source, layers=["vessels"], refresh_strategy="delta")
|
||||
pending.add({"operation": "UPDATE"})
|
||||
pending.add({"operation": "DELETE"})
|
||||
assert pending.refresh_strategy == "reload"
|
||||
|
||||
|
||||
def test_build_earth_update_maps_interactable_delete_to_delta():
|
||||
|
||||
@@ -12,6 +12,7 @@ from app.services.earth_news import (
|
||||
default_earth_news_sources_payload,
|
||||
normalize_earth_news_sources_payload,
|
||||
_fetch_source,
|
||||
_diversify_news_items_for_locale,
|
||||
_enrich_items_with_target_locations,
|
||||
_extract_target_location_from_text,
|
||||
_parse_feed_entries,
|
||||
@@ -23,6 +24,7 @@ from app.services.earth_news import (
|
||||
from app.services.earth_news_queue import NewsTargetLocationMessage
|
||||
from app.services.earth_news_worker import process_target_location_message
|
||||
from app.services.collectors.media_news_archive import MediaNewsArchiveCollector
|
||||
from app.services.earth_news_store import _diversify_parsed_news_items_by_region
|
||||
|
||||
|
||||
def test_serialize_item_includes_region_anchor_for_cruise():
|
||||
@@ -130,6 +132,109 @@ def test_rank_and_trim_items_prioritizes_active_breaking():
|
||||
assert [item.id for item in ranked] == ["global:critical", "europe:expired", "europe:regular"]
|
||||
|
||||
|
||||
def test_diversify_news_items_prefers_display_ready_content_across_sources():
|
||||
published_at = datetime(2026, 6, 11, 3, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(source_id: str, suffix: str, *, zh_ready: bool) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{source_id}:{suffix}",
|
||||
title=f"{source_id} title {suffix}",
|
||||
summary=f"{source_id} summary {suffix}",
|
||||
url=f"https://example.com/{source_id}/{suffix}",
|
||||
source=source_id,
|
||||
feed_name=source_id,
|
||||
feed_region="global",
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at,
|
||||
content_language="en",
|
||||
localizations={
|
||||
"zh-CN": {
|
||||
"title": f"{source_id} 中文标题 {suffix}",
|
||||
"summary": f"{source_id} 中文摘要 {suffix}",
|
||||
}
|
||||
} if zh_ready else {},
|
||||
)
|
||||
|
||||
items = [
|
||||
make_item("source-a", "1", zh_ready=False),
|
||||
make_item("source-a", "2", zh_ready=False),
|
||||
make_item("source-a", "3", zh_ready=False),
|
||||
make_item("source-b", "1", zh_ready=True),
|
||||
make_item("source-c", "1", zh_ready=True),
|
||||
]
|
||||
|
||||
result = _diversify_news_items_for_locale(
|
||||
items,
|
||||
active_region="global",
|
||||
limit=3,
|
||||
locale="zh-CN",
|
||||
)
|
||||
|
||||
assert [item.id.split(":", 1)[0] for item in result] == ["source-b", "source-c", "source-a"]
|
||||
|
||||
|
||||
def test_cruise_news_diversity_keeps_regions_from_being_starved():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(40)],
|
||||
make_item("europe", 1),
|
||||
make_item("middle-east-africa", 1),
|
||||
make_item("americas", 1),
|
||||
make_item("global", 1),
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=8)
|
||||
regions = [item.feed_region for item in result]
|
||||
|
||||
assert "europe" in regions
|
||||
assert "middle-east-africa" in regions
|
||||
assert "americas" in regions
|
||||
assert regions.count("asia-pacific") < len(regions)
|
||||
|
||||
|
||||
def test_global_news_diversity_uses_same_region_balance():
|
||||
published_at = datetime(2026, 6, 26, 8, 0, tzinfo=UTC)
|
||||
|
||||
def make_item(region: str, index: int) -> ParsedNewsItem:
|
||||
return ParsedNewsItem(
|
||||
id=f"{region}:global:{index}",
|
||||
title=f"{region} story {index}",
|
||||
summary=f"{region} summary {index}",
|
||||
url=f"https://example.com/{region}/global/{index}",
|
||||
source=region,
|
||||
feed_name=region,
|
||||
feed_region=region,
|
||||
homepage_url="https://example.com",
|
||||
published_at=published_at - timedelta(minutes=index),
|
||||
)
|
||||
|
||||
items = [
|
||||
*[make_item("asia-pacific", index) for index in range(24)],
|
||||
*[make_item("europe", index) for index in range(2)],
|
||||
*[make_item("middle-east-africa", index) for index in range(2)],
|
||||
*[make_item("americas", index) for index in range(2)],
|
||||
]
|
||||
|
||||
result = _diversify_parsed_news_items_by_region(items, limit=6)
|
||||
regions = {item.feed_region for item in result}
|
||||
|
||||
assert {"europe", "middle-east-africa", "americas"}.issubset(regions)
|
||||
|
||||
|
||||
def test_serialize_item_falls_back_to_global_anchor():
|
||||
item = ParsedNewsItem(
|
||||
id="custom:test",
|
||||
@@ -279,7 +384,14 @@ def test_parse_chinese_rss_marks_source_language_and_keeps_zh_localization():
|
||||
assert items[0].content_language == "zh-CN"
|
||||
assert items[0].localizations["zh-CN"]["title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_zh["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == "中国电商平台发布季度增长数据"
|
||||
assert payload_en["display_title"] == ""
|
||||
|
||||
items[0].localizations["en-US"] = {
|
||||
"title": "Chinese e-commerce platform reports quarterly growth",
|
||||
"summary": "The platform said cross-border orders rose year over year.",
|
||||
}
|
||||
payload_en_ready = _serialize_item(items[0], active_region="global", locale="en-US")
|
||||
assert payload_en_ready["display_title"] == "Chinese e-commerce platform reports quarterly growth"
|
||||
|
||||
|
||||
def test_default_news_sources_include_business_and_ecommerce_sources():
|
||||
@@ -961,9 +1073,11 @@ async def test_earth_news_payload_uses_fresh_database_items_without_rss(monkeypa
|
||||
async def fake_get_earth_news_freshness(_db, *, active_region):
|
||||
return 12, datetime.now(UTC)
|
||||
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None):
|
||||
assert limit == 12
|
||||
return [item]
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
if source_ids is None:
|
||||
assert limit == 12
|
||||
return [item]
|
||||
return []
|
||||
|
||||
async def fail_fetch(_sources):
|
||||
raise AssertionError("fresh database items should not fetch RSS")
|
||||
@@ -1062,8 +1176,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
async def fake_list_earth_news_items(_db, *, active_region, limit, categories=None, source_ids=None):
|
||||
captured["items_region"] = active_region
|
||||
captured["items_categories"] = categories
|
||||
captured["items_source_ids"] = source_ids
|
||||
return [item]
|
||||
captured.setdefault("items_source_ids", []).append(source_ids)
|
||||
return [item] if source_ids is None else []
|
||||
|
||||
async def fake_list_earth_news_cruise_items(_db, *, limit, categories=None, source_ids=None):
|
||||
captured["cruise_categories"] = categories
|
||||
@@ -1090,7 +1204,8 @@ async def test_earth_news_payload_passes_region_and_category_filters_to_store(mo
|
||||
assert captured["freshness_region"] == "europe"
|
||||
assert captured["items_region"] == "europe"
|
||||
assert captured["items_categories"] == {"business", "ecommerce"}
|
||||
assert captured["items_source_ids"] is None
|
||||
assert captured["items_source_ids"][0] is None
|
||||
assert any(source_ids for source_ids in captured["items_source_ids"][1:])
|
||||
assert captured["cruise_categories"] == {"business", "ecommerce"}
|
||||
assert captured["cruise_source_ids"] is None
|
||||
assert payload["filters"] == {
|
||||
|
||||
252
backend/tests/test_earth_news_manual.py
Normal file
@@ -0,0 +1,252 @@
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from app.core.enums import NewsSourceType
|
||||
from app.models.earth_news import EarthNewsItem
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services.earth_news import REGION_ANCHORS
|
||||
from app.services.earth_news_manual import (
|
||||
DEFAULT_MANUAL_NEWS_GROUP_ID,
|
||||
create_manual_news_group,
|
||||
import_manual_news_items,
|
||||
list_news_groups,
|
||||
list_news_records,
|
||||
parse_manual_news_import_upload,
|
||||
rename_manual_news_group,
|
||||
upsert_manual_news_item,
|
||||
)
|
||||
|
||||
|
||||
class _FakeResult:
|
||||
def __init__(self, rows=None, scalar=None):
|
||||
self.rows = rows or []
|
||||
self._scalar = scalar
|
||||
|
||||
def scalar_one_or_none(self):
|
||||
return self._scalar
|
||||
|
||||
def scalar(self):
|
||||
return self._scalar
|
||||
|
||||
def scalars(self):
|
||||
return self
|
||||
|
||||
def all(self):
|
||||
return self.rows
|
||||
|
||||
|
||||
class _FakeNewsSession:
|
||||
def __init__(self, records=None, setting=None):
|
||||
self.records = dict(records or {})
|
||||
self.setting = setting
|
||||
|
||||
async def get(self, _model, item_id):
|
||||
return self.records.get(item_id)
|
||||
|
||||
def add(self, item):
|
||||
if isinstance(item, SystemSetting):
|
||||
self.setting = item
|
||||
else:
|
||||
self.records[item.id] = item
|
||||
|
||||
async def execute(self, stmt):
|
||||
statement = str(stmt)
|
||||
if "system_settings" in statement:
|
||||
return _FakeResult(scalar=self.setting)
|
||||
if "count" in statement.lower():
|
||||
return _FakeResult(scalar=len(self.records))
|
||||
return _FakeResult(rows=list(self.records.values()))
|
||||
|
||||
async def flush(self):
|
||||
return None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_news_queue(monkeypatch):
|
||||
queued = []
|
||||
|
||||
async def _enqueue(payload, force=False):
|
||||
queued.append({"payload": payload, "force": force})
|
||||
return True
|
||||
|
||||
monkeypatch.setattr("app.services.earth_news_manual.enqueue_target_location_job", _enqueue)
|
||||
return queued
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_upsert_uses_region_anchor_and_manual_metadata(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
result = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "手动添加的新闻",
|
||||
"summary": "一条用于测试的手动新闻。",
|
||||
"source": "人工录入",
|
||||
"region": "europe",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"tags": ["manual", "test"],
|
||||
},
|
||||
)
|
||||
|
||||
anchor = REGION_ANCHORS["europe"]
|
||||
assert result.created is True
|
||||
assert result.queued is True
|
||||
assert result.item.id.startswith("manual:")
|
||||
assert result.item.feed_name == "手动添加"
|
||||
assert result.item.source == "人工录入"
|
||||
assert result.item.latitude == anchor.latitude
|
||||
assert result.item.longitude == anchor.longitude
|
||||
assert result.item.location_source == "region_anchor"
|
||||
assert result.item.verified is False
|
||||
assert result.item.location_meta["news_meta"]["feed_type"] == NewsSourceType.MANUAL.value
|
||||
assert result.item.location_meta["news_meta"]["source_type"] == NewsSourceType.MANUAL.value
|
||||
assert result.item.location_meta["news_meta"]["manual_group_id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
assert len(fake_news_queue) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_duplicate_import_upserts_without_duplicate_rows(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
payload = {
|
||||
"title": "Same manual story",
|
||||
"source": "Manual Desk",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"region": "global",
|
||||
}
|
||||
|
||||
first = await upsert_manual_news_item(db, payload)
|
||||
second = await upsert_manual_news_item(db, {**payload, "summary": "Updated summary"})
|
||||
|
||||
assert first.created is True
|
||||
assert second.created is False
|
||||
assert len(db.records) == 1
|
||||
assert db.records[first.item.id].summary == "Updated summary"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_edit_without_location_preserves_manual_coordinates(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
created = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "Taipei-1 data center update",
|
||||
"summary": "Initial summary.",
|
||||
"region": "asia-pacific",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"location": {"label": "Kaohsiung, Taiwan", "latitude": 22.6273, "longitude": 120.3014},
|
||||
},
|
||||
)
|
||||
|
||||
updated = await upsert_manual_news_item(
|
||||
db,
|
||||
{
|
||||
"title": "Taipei-1 data center update",
|
||||
"summary": "Edited summary only.",
|
||||
"region": "asia-pacific",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
},
|
||||
item_id_override=created.item.id,
|
||||
)
|
||||
|
||||
assert updated.created is False
|
||||
assert updated.item.latitude == pytest.approx(22.6273)
|
||||
assert updated.item.longitude == pytest.approx(120.3014)
|
||||
assert updated.item.location_source == "manual_location"
|
||||
assert updated.item.verified is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_api_service_rejects_rss_records(fake_news_queue):
|
||||
rss_record = EarthNewsItem(
|
||||
id="bbc-world:example",
|
||||
title="RSS story",
|
||||
summary="RSS summary",
|
||||
source="BBC World",
|
||||
feed_name="BBC World",
|
||||
region="global",
|
||||
latitude=20,
|
||||
longitude=0,
|
||||
location_label="全球",
|
||||
location_source="region_anchor",
|
||||
verified=False,
|
||||
location_meta={"news_meta": {"feed_type": "rss"}},
|
||||
first_seen_at=datetime.now(UTC),
|
||||
last_seen_at=datetime.now(UTC),
|
||||
)
|
||||
db = _FakeNewsSession({rss_record.id: rss_record})
|
||||
|
||||
with pytest.raises(PermissionError):
|
||||
await upsert_manual_news_item(
|
||||
db,
|
||||
{"title": "Edited title", "region": "global"},
|
||||
item_id_override=rss_record.id,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_import_reports_per_item_errors(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
result = await import_manual_news_items(
|
||||
db,
|
||||
[
|
||||
{"title": "Valid manual news", "region": "global"},
|
||||
{"summary": "missing title"},
|
||||
],
|
||||
)
|
||||
|
||||
assert result["created"] == 1
|
||||
assert result["failed"] == 1
|
||||
assert result["errors"][0]["index"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_groups_default_create_and_rename(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
|
||||
initial = await list_news_groups(db)
|
||||
assert initial["manual_groups"][0]["id"] == DEFAULT_MANUAL_NEWS_GROUP_ID
|
||||
assert initial["manual_groups"][0]["name"] == "新建新闻组"
|
||||
|
||||
group = await create_manual_news_group(db, "专题组")
|
||||
assert group["name"] == "专题组"
|
||||
assert db.setting is not None
|
||||
|
||||
await upsert_manual_news_item(db, {"title": "Grouped story", "region": "global"}, group_id=group["id"])
|
||||
renamed = await rename_manual_news_group(db, group["id"], "重命名专题")
|
||||
|
||||
record = next(iter(db.records.values()))
|
||||
assert renamed["name"] == "重命名专题"
|
||||
assert record.location_meta["news_meta"]["manual_group_id"] == group["id"]
|
||||
assert record.location_meta["news_meta"]["manual_group_name"] == "重命名专题"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_list_filters_by_group_id(fake_news_queue):
|
||||
db = _FakeNewsSession()
|
||||
group = await create_manual_news_group(db, "导入组")
|
||||
|
||||
await import_manual_news_items(
|
||||
db,
|
||||
[
|
||||
{"title": "In group", "region": "global"},
|
||||
{"title": "Also in group", "region": "global"},
|
||||
],
|
||||
group_id=group["id"],
|
||||
)
|
||||
await upsert_manual_news_item(db, {"title": "Default group", "region": "global"})
|
||||
|
||||
grouped = await list_news_records(db, page=1, page_size=20, group_id=group["id"])
|
||||
default_group = await list_news_records(db, page=1, page_size=20, group_id=DEFAULT_MANUAL_NEWS_GROUP_ID)
|
||||
|
||||
assert grouped["total"] == 2
|
||||
assert {item["manual_group_id"] for item in grouped["items"]} == {group["id"]}
|
||||
assert default_group["total"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_news_import_parser_requires_json_array():
|
||||
with pytest.raises(ValueError, match="顶层必须是数组"):
|
||||
await parse_manual_news_import_upload(b'{"title":"not an array"}')
|
||||
@@ -28,7 +28,7 @@ def test_protocol_enum_values_remain_api_compatible() -> None:
|
||||
assert [item.value for item in NewsImportanceLevel] == ["low", "medium", "high", "critical"]
|
||||
assert [item.value for item in BreakingLevel] == ["none", "watch", "breaking", "critical"]
|
||||
assert [item.value for item in BreakingScope] == ["regional", "global"]
|
||||
assert [item.value for item in NewsSourceType] == ["rss", "atom", "aggregated", "reference"]
|
||||
assert [item.value for item in NewsSourceType] == ["rss", "atom", "aggregated", "reference", "manual"]
|
||||
assert [item.value for item in UserRole] == ["viewer", "admin", "super_admin"]
|
||||
assert JobStatus.RUNNING.value == "running"
|
||||
assert PlaygroundMessageKind.THINKING.value == "thinking"
|
||||
|
||||
219
backend/tests/test_llm_provider_catalog.py
Normal file
@@ -0,0 +1,219 @@
|
||||
from copy import deepcopy
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services import llm_provider_catalog as catalog
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
def mock_http(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"provider,path,auth",
|
||||
[
|
||||
("minimax", "/anthropic/v1/models", "x-api-key"),
|
||||
("anthropic", "/v1/models", "x-api-key"),
|
||||
("openai", "/v1/models", "authorization"),
|
||||
("deepseek", "/v1/models", "authorization"),
|
||||
("alibaba", "/api/v1/models", "authorization"),
|
||||
("moonshotai", "/v1/models", "authorization"),
|
||||
("openrouter", "/api/v1/models", "authorization"),
|
||||
("opencode-go", "/zen/go/v1/models", "authorization"),
|
||||
("ollama", "/api/tags", "authorization"),
|
||||
],
|
||||
)
|
||||
async def test_official_catalog_requests(monkeypatch, provider, path, auth):
|
||||
preset = catalog.get_fallback_llm_provider_preset(provider)
|
||||
|
||||
def handle(request):
|
||||
assert request.url.path == path
|
||||
assert request.url.host == httpx.URL(preset["base_url"]).host
|
||||
assert request.headers[auth] == ("test-key" if auth == "x-api-key" else "Bearer test-key")
|
||||
if provider == "alibaba":
|
||||
assert request.url.params["capabilities"] == "TG"
|
||||
return httpx.Response(
|
||||
200, json={"output": {"total": 1, "models": [{"model": "latest"}]}}
|
||||
)
|
||||
if provider == "ollama":
|
||||
return httpx.Response(200, json={"models": [{"name": "latest:7b"}]})
|
||||
return httpx.Response(200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset(provider, api_key="test-key")
|
||||
assert result["models"] == (["latest:7b"] if provider == "ollama" else ["latest"])
|
||||
assert result["base_url"] == preset["base_url"]
|
||||
assert "test-key" not in str(result)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"provider,base,api,path",
|
||||
[
|
||||
(
|
||||
"minimax",
|
||||
"https://api.minimax.io/anthropic/v1/",
|
||||
"anthropic-messages",
|
||||
"/anthropic/v1/models",
|
||||
),
|
||||
("anthropic", "https://api.anthropic.com", "anthropic-messages", "/v1/models"),
|
||||
(
|
||||
"alibaba",
|
||||
"https://workspace.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
(
|
||||
"alibaba",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
"openai-completions",
|
||||
"/api/v1/models",
|
||||
),
|
||||
("alibaba", "https://custom.test/gateway/v1", "openai-completions", "/gateway/v1/models"),
|
||||
("moonshotai", "https://api.moonshot.cn/v1", "openai-completions", "/v1/models"),
|
||||
("ollama", "http://localhost:11434/api", "ollama-generate", "/api/tags"),
|
||||
],
|
||||
)
|
||||
def test_urls_preserve_region_and_gateway(provider, base, api, path):
|
||||
url = httpx.URL(discovery.model_catalog_url(provider, base, api))
|
||||
assert url.host == httpx.URL(base).host
|
||||
assert url.path == path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_anthropic_pagination_and_release_order(monkeypatch):
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request.url.params.get("after_id"))
|
||||
if len(seen) == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"data": [{"id": "old", "created_at": "2025-01-01T00:00:00Z"}],
|
||||
"has_more": True,
|
||||
"last_id": "old",
|
||||
},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"data": [{"id": "new", "created_at": "2026-06-01T00:00:00Z"}], "has_more": False},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("anthropic", "test-key")
|
||||
assert seen == [None, "old"]
|
||||
assert result["models"] == ["new", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dashscope_pagination(monkeypatch):
|
||||
def handle(request):
|
||||
page = int(request.url.params["page_no"])
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"output": {
|
||||
"total": 2,
|
||||
"models": [
|
||||
{"model": f"page-{page}", "published_time": f"2026-06-0{page} 00:00:00"}
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
result = await catalog.refresh_llm_provider_preset("alibaba", "test-key")
|
||||
assert result["models"] == ["page-2", "page-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"payload",
|
||||
[
|
||||
{},
|
||||
{"data": []},
|
||||
{"data": [None]},
|
||||
{"data": [{}]},
|
||||
{"data": [{"id": "same"}], "has_more": True, "last_id": "same"},
|
||||
],
|
||||
)
|
||||
async def test_invalid_or_incomplete_catalog_fails(monkeypatch, payload):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
with pytest.raises(discovery.LLMProviderCatalogError):
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_models_retained_and_defaults_unchanged(monkeypatch):
|
||||
before = deepcopy(catalog.FALLBACK_LLM_PROVIDER_PRESETS)
|
||||
rows = [{"id": f"model-{i}", "created": i} for i in range(140)]
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"data": rows}))
|
||||
result = await catalog.refresh_llm_provider_preset("openai", "test-key")
|
||||
assert len(result["models"]) == 140
|
||||
assert result["models"][0] == "model-139"
|
||||
assert catalog.FALLBACK_LLM_PROVIDER_PRESETS == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_ollama_catalog_is_valid(monkeypatch):
|
||||
mock_http(monkeypatch, lambda request: httpx.Response(200, json={"models": []}))
|
||||
result = await catalog.refresh_llm_provider_preset("ollama")
|
||||
assert result["models"] == []
|
||||
assert result["model"] == ""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_opencode_documented_protocols(monkeypatch):
|
||||
models = [
|
||||
"minimax-m3",
|
||||
"qwen3.8-max",
|
||||
"gpt-5.6-luna",
|
||||
"grok-4.6",
|
||||
"muse-spark-1.3-contributor",
|
||||
"kimi-k3",
|
||||
]
|
||||
mock_http(
|
||||
monkeypatch,
|
||||
lambda request: httpx.Response(200, json={"data": [{"id": model} for model in models]}),
|
||||
)
|
||||
result = await catalog.refresh_llm_provider_preset("opencode-go")
|
||||
assert list(result["model_provider_apis"].values()) == [
|
||||
"anthropic-messages",
|
||||
"anthropic-messages",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-responses",
|
||||
"openai-completions",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_retry_transient_error_only(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def handle(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(503 if len(calls) == 1 else 200, json={"data": [{"id": "latest"}]})
|
||||
|
||||
mock_http(monkeypatch, handle)
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 2
|
||||
calls.clear()
|
||||
|
||||
def unauthorized(request):
|
||||
calls.append(request)
|
||||
return httpx.Response(401, text="secret-upstream-text")
|
||||
|
||||
mock_http(monkeypatch, unauthorized)
|
||||
with pytest.raises(httpx.HTTPStatusError) as error:
|
||||
await catalog.refresh_llm_provider_preset("minimax", "test-key")
|
||||
assert len(calls) == 1
|
||||
assert "secret-upstream-text" not in discovery.catalog_error_message(error.value)
|
||||
165
backend/tests/test_llm_provider_settings.py
Normal file
@@ -0,0 +1,165 @@
|
||||
from copy import deepcopy
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.v1 import settings as api
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.services import llm_model_catalog as discovery
|
||||
from app.services.llm_provider_catalog import list_fallback_llm_provider_presets
|
||||
|
||||
REAL_CLIENT = httpx.AsyncClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def settings_db(monkeypatch, tmp_path):
|
||||
# Real SQL reads, commits and reloads; do not mock the settings helpers.
|
||||
engine = create_engine("sqlite://")
|
||||
SystemSetting.__table__.create(engine)
|
||||
session = Session(engine)
|
||||
db = SimpleNamespace(
|
||||
execute=AsyncMock(side_effect=session.execute),
|
||||
add=session.add,
|
||||
commit=AsyncMock(side_effect=session.commit),
|
||||
refresh=AsyncMock(side_effect=session.refresh),
|
||||
)
|
||||
monkeypatch.setattr(api, "AI_PROVIDER_ENV_FILE", tmp_path / "missing.env")
|
||||
monkeypatch.setattr(api, "_resolve_env_secret", lambda *names: ("", ""))
|
||||
yield db
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def use_upstream(monkeypatch, handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
monkeypatch.setattr(
|
||||
discovery.httpx, "AsyncClient", lambda **kw: REAL_CLIENT(transport=transport, **kw)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fresh_database_lists_all_builtin_presets(settings_db):
|
||||
result = await api.get_ai_provider_presets(current_user=None, db=settings_db)
|
||||
assert len(result["data"]) == 9
|
||||
assert all(row["models"] for row in result["data"])
|
||||
assert "MiniMax-M3" in result["data"][0]["models"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_refresh_commits_reloads_and_preserves_runtime_and_draft(settings_db, monkeypatch):
|
||||
runtime = {
|
||||
"ai_provider": {
|
||||
"default_provider": "minimax",
|
||||
"service_token": "private-service-token",
|
||||
"providers": {
|
||||
"minimax": {
|
||||
"base_url": "https://saved.test/anthropic",
|
||||
"model": "old",
|
||||
"api_key": "saved-secret",
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
await api.save_setting_payload(settings_db, "external_integrations", deepcopy(runtime))
|
||||
before = await api.get_setting_payload(settings_db, "external_integrations")
|
||||
seen = []
|
||||
|
||||
def handle(request):
|
||||
seen.append(request)
|
||||
assert str(request.url) == "https://draft.test/anthropic/v1/models"
|
||||
assert request.headers["x-api-key"] == "draft-secret"
|
||||
return httpx.Response(200, json={"data": [{"id": "MiniMax-M3"}, {"id": "old"}]})
|
||||
|
||||
use_upstream(monkeypatch, handle)
|
||||
draft = api.AIProviderIntegrationUpdate(
|
||||
provider="minimax",
|
||||
base_url="https://draft.test/anthropic",
|
||||
model="unsaved-model",
|
||||
api_key="draft-secret",
|
||||
service_token="unsaved-service-token",
|
||||
)
|
||||
refreshed = await api.refresh_ai_provider_preset("minimax", None, settings_db, draft)
|
||||
result = await api.get_ai_provider_presets(None, settings_db)
|
||||
preset = next(row for row in result["data"] if row["provider"] == "minimax")
|
||||
assert len(seen) == 1
|
||||
assert preset["models"] == ["MiniMax-M3", "old"]
|
||||
assert preset["refreshed_at"] == refreshed["data"]["refreshed_at"]
|
||||
assert await api.get_setting_payload(settings_db, "external_integrations") == before
|
||||
assert "secret" not in str(result)
|
||||
assert "private-service-token" not in str(result)
|
||||
assert draft.model == "unsaved-model"
|
||||
stored = await settings_db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == "llm_provider_preset:minimax")
|
||||
)
|
||||
assert stored.scalar_one().payload["models"] == ["MiniMax-M3", "old"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_refresh_preserves_stored_catalog_and_sanitizes_errors(
|
||||
settings_db, monkeypatch
|
||||
):
|
||||
await api.save_setting_payload(
|
||||
settings_db, "llm_provider_preset:openrouter", {"models": ["saved-model"]}
|
||||
)
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(401, text="private-upstream-secret"))
|
||||
commits = settings_db.commit.await_count
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("openrouter", None, settings_db)
|
||||
assert error.value.status_code == 502
|
||||
assert "private-upstream-secret" not in error.value.detail
|
||||
assert settings_db.commit.await_count == commits
|
||||
assert (await api.get_setting_payload(settings_db, "llm_provider_preset:openrouter"))[
|
||||
"models"
|
||||
] == ["saved-model"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_key_is_actionable_and_does_not_request_upstream(settings_db, monkeypatch):
|
||||
def fail_request(request):
|
||||
pytest.fail("must not query a private catalog without credentials")
|
||||
|
||||
use_upstream(monkeypatch, fail_request)
|
||||
with pytest.raises(api.HTTPException) as error:
|
||||
await api.refresh_ai_provider_preset("minimax", None, settings_db)
|
||||
assert error.value.status_code == 400
|
||||
assert "API Key" in error.value.detail
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"preset", list_fallback_llm_provider_presets(), ids=lambda p: p["provider"]
|
||||
)
|
||||
async def test_404_never_passes_by_builtin_model_name(monkeypatch, preset):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(404, text="private-upstream-secret"))
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "api_key": "invalid-key", "preset_models": preset["models"]}, 5
|
||||
)
|
||||
assert result["success"] is False
|
||||
assert result["connected"] is False
|
||||
assert "private-upstream-secret" not in result["message"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("payload", [{}, {"data": []}, {"data": [{"id": "different-model"}]}])
|
||||
async def test_invalid_or_missing_models_never_pass(monkeypatch, payload):
|
||||
use_upstream(monkeypatch, lambda request: httpx.Response(200, json=payload))
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight({**preset, "api_key": "test-key"}, 5)
|
||||
assert result["success"] is False
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_model_does_not_need_a_builtin_whitelist(monkeypatch):
|
||||
use_upstream(
|
||||
monkeypatch, lambda request: httpx.Response(200, json={"data": [{"id": "future-model"}]})
|
||||
)
|
||||
preset = list_fallback_llm_provider_presets()[0]
|
||||
result = await api._check_ai_provider_lightweight(
|
||||
{**preset, "model": "future-model", "api_key": "test-key"}, 5
|
||||
)
|
||||
assert result["success"] is True
|
||||
assert result["url"].endswith("/anthropic/v1/models")
|
||||
@@ -40,6 +40,9 @@ def test_gesture_event_serializes_stable_protocol_fields():
|
||||
assert payload["seq"] == 7
|
||||
assert payload["source"] == "motion-agent"
|
||||
assert payload["mode"] == "single"
|
||||
assert payload["protocol_version"] == "motion.v2"
|
||||
assert payload["input_mode"] == "single"
|
||||
assert payload["camera_id"] == "unknown"
|
||||
assert payload["payload"] == {}
|
||||
|
||||
|
||||
@@ -88,8 +91,13 @@ def test_motion_server_status_includes_dry_run_camera_and_heartbeat():
|
||||
assert status["camera_count"] == 1
|
||||
assert status["active_camera_ids"] == ["dry-run:null-camera"]
|
||||
assert status["recognizer"] == "dry-run"
|
||||
assert status["protocol_version"] == "motion.v2"
|
||||
assert status["armed"] is False
|
||||
assert status["paused"] is False
|
||||
assert status["devices_open"] is False
|
||||
assert heartbeat == {
|
||||
"timestamp_ms": 123,
|
||||
"protocol_version": "motion.v2",
|
||||
"source": "motion-agent",
|
||||
"type": "heartbeat",
|
||||
}
|
||||
@@ -109,6 +117,7 @@ def test_skeleton_event_serializes_without_raw_image_fields():
|
||||
payload = json.loads(event.to_json())
|
||||
|
||||
assert payload["type"] == "skeleton"
|
||||
assert payload["protocol_version"] == "motion.v2"
|
||||
assert payload["matched_gesture"] == "rotate_left"
|
||||
assert payload["confidence"] == 0.91
|
||||
assert payload["camera_id"] == "usb:0"
|
||||
@@ -120,6 +129,25 @@ def test_skeleton_event_serializes_without_raw_image_fields():
|
||||
assert "frame" not in payload
|
||||
|
||||
|
||||
def test_v2_gesture_set_accepts_frontend_motion_gestures():
|
||||
state = GestureStateMachine(confidence_threshold=0.7, cooldown_ms=0)
|
||||
|
||||
for gesture in [
|
||||
"rotate_up",
|
||||
"rotate_down",
|
||||
"focus_prev",
|
||||
"focus_next",
|
||||
"layer_prev",
|
||||
"layer_next",
|
||||
]:
|
||||
event = state.accept(
|
||||
GestureObservation(gesture, confidence=0.9, intensity=0.8, timestamp_ms=1000)
|
||||
)
|
||||
|
||||
assert event is not None
|
||||
assert event.gesture == gesture
|
||||
|
||||
|
||||
def test_dry_run_recognizer_produces_debug_skeleton():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
@@ -240,3 +268,92 @@ async def test_motion_agent_cli_reports_dependency_error_without_traceback(monke
|
||||
assert exit_code == 2
|
||||
assert "Motion agent failed: missing cv stack" in captured.err
|
||||
assert "Traceback" not in captured.err
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_motion_agent_command_updates_control_state():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
armed = await server.handle_command(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_armed",
|
||||
"request_id": "req-armed",
|
||||
"payload": {"armed": True},
|
||||
}
|
||||
)
|
||||
)
|
||||
paused = await server.handle_command(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_paused",
|
||||
"request_id": "req-paused",
|
||||
"payload": {"paused": True},
|
||||
}
|
||||
)
|
||||
|
||||
assert armed.ok is True
|
||||
assert armed.request_id == "req-armed"
|
||||
assert armed.status["armed"] is True
|
||||
assert paused.ok is True
|
||||
assert paused.status["paused"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_motion_agent_open_devices_command_accepts_dual_mode():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
|
||||
try:
|
||||
result = await server.handle_command(
|
||||
{
|
||||
"type": "command",
|
||||
"command": "open_devices",
|
||||
"request_id": "req-open",
|
||||
"payload": {"input_mode": "dual_redundant"},
|
||||
}
|
||||
)
|
||||
|
||||
assert result.ok is True
|
||||
assert result.status["input_mode"] == "dual_redundant"
|
||||
assert result.status["active_camera_ids"] == ("dry-run:null-camera",)
|
||||
assert server._recognition_subprocess is not None
|
||||
assert server._recognition_subprocess.returncode is None
|
||||
finally:
|
||||
await server.stop_recognition_subprocess()
|
||||
|
||||
|
||||
def test_motion_agent_dual_fusion_merges_matching_observations():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True))
|
||||
server.state.mode = "dual_redundant"
|
||||
|
||||
selected, fusion = server._fuse_observations(
|
||||
[
|
||||
GestureObservation("zoom_in", confidence=0.82, intensity=0.4, camera_id="usb:0"),
|
||||
GestureObservation("zoom_in", confidence=0.86, intensity=0.8, camera_id="usb:1"),
|
||||
]
|
||||
)
|
||||
|
||||
assert selected.gesture == "zoom_in"
|
||||
assert selected.camera_id == "fusion"
|
||||
assert selected.confidence > 0.86
|
||||
assert fusion == {
|
||||
"source_cameras": ["usb:1", "usb:0"],
|
||||
"window_ms": 120,
|
||||
"reason": "matched_observations",
|
||||
}
|
||||
|
||||
|
||||
def test_motion_agent_dual_fusion_suppresses_close_conflict():
|
||||
server = MotionAgentServer(MotionAgentConfig(dry_run=True, confidence_threshold=0.7))
|
||||
|
||||
selected, fusion = server._fuse_observations(
|
||||
[
|
||||
GestureObservation("zoom_in", confidence=0.82, intensity=0.5, camera_id="usb:0"),
|
||||
GestureObservation("zoom_out", confidence=0.78, intensity=0.5, camera_id="usb:1"),
|
||||
]
|
||||
)
|
||||
|
||||
assert selected.gesture == "zoom_in"
|
||||
assert selected.confidence == 0
|
||||
assert fusion["reason"] == "conflict_ignored"
|
||||
|
||||
141
backend/tests/test_provider_protocols.py
Normal file
@@ -0,0 +1,141 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
with patch(
|
||||
"pydantic_settings.sources.providers.dotenv.DotEnvSettingsSource._read_env_files",
|
||||
return_value={},
|
||||
):
|
||||
from aiprovider.provider_service import ProviderService
|
||||
from aiprovider.schemas import SituationalAnalysisRequest
|
||||
from app.api.v1.settings import _runtime_config_from_ai_payload
|
||||
|
||||
|
||||
def test_custom_openai_gateway_keeps_its_configured_protocol():
|
||||
config = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "openai",
|
||||
"providers": {
|
||||
"openai": {
|
||||
"base_url": "https://gateway.test/v1",
|
||||
"provider_api": "openai-completions",
|
||||
"model": "gpt-6-astra",
|
||||
},
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService({**config, "api_key": "test-key"})
|
||||
assert service._resolve_model_provider_api("gpt-6-astra") == "openai-completions"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"model,path",
|
||||
[("minimax-m3", "/messages"), ("qwen3.8-max", "/messages"), ("gpt-5.6-luna", "/responses")],
|
||||
)
|
||||
async def test_runtime_model_routes_use_current_protocols(monkeypatch, model, path):
|
||||
llm = _runtime_config_from_ai_payload(
|
||||
{
|
||||
"default_provider": "opencode-go",
|
||||
"providers": {
|
||||
"opencode-go": {
|
||||
"model": model,
|
||||
"api_key": "test-key",
|
||||
"model_provider_apis": {model: "openai-completions"},
|
||||
}
|
||||
},
|
||||
}
|
||||
)["llm_config"]
|
||||
service = ProviderService(llm)
|
||||
seen = []
|
||||
|
||||
async def post(**request):
|
||||
seen.append(request)
|
||||
assert request["path"] == path
|
||||
assert request["request_body"]["model"] == model
|
||||
if path == "/responses":
|
||||
assert request["request_body"]["store"] is False
|
||||
assert "max_tokens" not in request["request_body"]
|
||||
return {
|
||||
"output": [
|
||||
{"type": "reasoning", "summary": [{"text": "reason"}]},
|
||||
{"type": "message", "content": [{"type": "output_text", "text": "OK"}]},
|
||||
]
|
||||
}
|
||||
return {"content": [{"type": "text", "text": "OK"}]}
|
||||
|
||||
monkeypatch.setattr(service, "_post", post)
|
||||
result = await service.analyze(SituationalAnalysisRequest(title="test", objective="reply OK"))
|
||||
assert len(seen) == 1
|
||||
assert result.content == "OK"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_minimax_messages_url_accepts_both_documented_base_forms(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
seen.append(str(request.url))
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for base in ["https://api.minimaxi.com/anthropic", "https://api.minimaxi.com/anthropic/v1"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "minimax",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": base,
|
||||
"api_key": "test-key",
|
||||
"model": "MiniMax-M3",
|
||||
}
|
||||
)
|
||||
result = await service.analyze(
|
||||
SituationalAnalysisRequest(title="test", objective="reply OK")
|
||||
)
|
||||
assert result.content == "OK"
|
||||
assert seen == ["https://api.minimaxi.com/anthropic/v1/messages"] * 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_m3_playground_thinking_and_opencode_session_headers(monkeypatch):
|
||||
seen = []
|
||||
client_type = httpx.AsyncClient
|
||||
|
||||
def handle(request):
|
||||
import json
|
||||
|
||||
seen.append(request)
|
||||
assert json.loads(request.content)["thinking"] == {"type": "adaptive"}
|
||||
assert request.headers["user-agent"] == "Planet/1.0"
|
||||
return httpx.Response(200, json={"content": [{"type": "text", "text": "OK"}]})
|
||||
|
||||
transport = httpx.MockTransport(handle)
|
||||
monkeypatch.setattr(httpx, "AsyncClient", lambda **kw: client_type(transport=transport, **kw))
|
||||
for session_id in ["conversation-one", "conversation-one", "conversation-two"]:
|
||||
service = ProviderService(
|
||||
{
|
||||
"provider": "opencode-go",
|
||||
"provider_api": "anthropic-messages",
|
||||
"base_url": "https://opencode.ai/zen/go/v1",
|
||||
"api_key": "test-key",
|
||||
"model": "minimax-m3",
|
||||
}
|
||||
)
|
||||
await service.analyze(
|
||||
SituationalAnalysisRequest(
|
||||
title="test",
|
||||
objective="reply OK",
|
||||
thinking={"type": "enabled"},
|
||||
context={"session_id": session_id},
|
||||
)
|
||||
)
|
||||
session_headers = [request.headers["x-opencode-session"] for request in seen]
|
||||
assert session_headers[0] == session_headers[1]
|
||||
assert session_headers[0] != session_headers[2]
|
||||
160
backend/tests/test_tv_catalog.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db.session import Base
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.task import CollectionTask
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
from app.services.tv_catalog import get_tv_catalog_page
|
||||
from app.services.tv_streams import normalize_tv_settings
|
||||
|
||||
|
||||
class CatalogSession:
|
||||
"""Execute the real catalog queries against an isolated SQLite database."""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
async def execute(self, query):
|
||||
return self.session.execute(query)
|
||||
|
||||
async def scalar(self, query):
|
||||
return self.session.scalar(query)
|
||||
|
||||
async def scalars(self, query):
|
||||
return self.session.scalars(query)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def catalog_db():
|
||||
engine = create_engine("sqlite:///:memory:")
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def register_functions(connection, _record):
|
||||
connection.create_function(
|
||||
"concat_ws",
|
||||
-1,
|
||||
lambda sep, *args: sep.join(str(arg) for arg in args if arg is not None),
|
||||
)
|
||||
|
||||
Base.metadata.create_all(
|
||||
engine,
|
||||
tables=[
|
||||
CollectionTask.__table__,
|
||||
DataSnapshot.__table__,
|
||||
CollectedData.__table__,
|
||||
SystemSetting.__table__,
|
||||
],
|
||||
)
|
||||
with Session(engine) as session:
|
||||
for index in range(135):
|
||||
session.add(
|
||||
CollectedData(
|
||||
source="news_live_streams",
|
||||
source_id=f"channel-{index:03}",
|
||||
data_type="news_live_stream",
|
||||
name=f"Channel {index:03}",
|
||||
collected_at=datetime(2026, 9, 13, tzinfo=UTC),
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
extra_data={
|
||||
"stream_url": "https://example.invalid/live.m3u8",
|
||||
"region": "Canada",
|
||||
},
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
yield CatalogSession(session)
|
||||
engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pages_include_entire_catalog_without_overlap(catalog_db):
|
||||
first = await get_tv_catalog_page(catalog_db, limit=50)
|
||||
second = await get_tv_catalog_page(catalog_db, offset=first["next_offset"], limit=50)
|
||||
third = await get_tv_catalog_page(catalog_db, offset=second["next_offset"], limit=50)
|
||||
ids = [source["id"] for page in (first, second, third) for source in page["sources"]]
|
||||
assert [len(page["sources"]) for page in (first, second, third)] == [50, 50, 45]
|
||||
assert len(set(ids)) == first["source_count"] == 145
|
||||
assert ids[-1] == "channel-134"
|
||||
assert third["next_offset"] is None and not third["has_more"]
|
||||
beyond = await get_tv_catalog_page(catalog_db, offset=200)
|
||||
assert beyond["sources"] == [] and not beyond["has_more"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_finds_later_pages_and_treats_wildcards_literally(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, q="CANADA 134")
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
assert payload["total"] == 1 and payload["source_count"] == 145
|
||||
assert (await get_tv_catalog_page(catalog_db, q="%"))["total"] == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_selection_survives_refresh_when_outside_first_page(catalog_db):
|
||||
payload = await get_tv_catalog_page(catalog_db, selected_id="channel-134")
|
||||
assert payload["selected_source"]["id"] == "channel-134"
|
||||
assert "channel-134" not in [source["id"] for source in payload["sources"]]
|
||||
assert payload["default_source_id"] == "aljazeera-mubasher"
|
||||
default = (await get_tv_catalog_page(catalog_db, selected_id="removed"))["selected_source"]
|
||||
assert default["id"] == "aljazeera-mubasher" and default["source_type"] == "hls"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_catalog_hides_inactive_records_and_deduplicates_ids(catalog_db):
|
||||
for name, values in [
|
||||
("Disabled", {"extra_data": {"is_enabled": False}}),
|
||||
("Historical", {"is_current": False}),
|
||||
("Invalid", {"is_valid": 0}),
|
||||
("Deleted", {"deleted_at": datetime.now(UTC)}),
|
||||
("Replacement", {"source_id": "channel-134"}),
|
||||
]:
|
||||
record = dict(
|
||||
source="news_live_streams",
|
||||
source_id=name,
|
||||
name=name,
|
||||
data_type="news_live_stream",
|
||||
is_current=True,
|
||||
is_valid=1,
|
||||
)
|
||||
catalog_db.session.add(CollectedData(**{**record, **values}))
|
||||
catalog_db.session.flush()
|
||||
payload = await get_tv_catalog_page(catalog_db, q="replacement")
|
||||
assert payload["source_count"] == 145
|
||||
assert [source["id"] for source in payload["sources"]] == ["channel-134"]
|
||||
|
||||
|
||||
def test_missing_builtin_default_adds_aljazeera_without_losing_custom_source():
|
||||
settings = normalize_tv_settings({"sources": [{"id": "custom", "name": "Custom"}]})
|
||||
assert settings["default_source_id"] == "aljazeera-mubasher"
|
||||
assert {source["id"] for source in settings["sources"]} == {"custom", "aljazeera-mubasher"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
"config, expected", [({}, 135), ({"max_sources": 0}, 135), ({"max_sources": 7}, 7)]
|
||||
)
|
||||
async def test_collector_keeps_all_matching_channels_unless_explicitly_limited(
|
||||
monkeypatch, config, expected
|
||||
):
|
||||
collector = NewsLiveStreamsCollector()
|
||||
channels = [
|
||||
{"id": f"channel-{i}", "name": f"Channel {i}", "categories": ["news"]} for i in range(135)
|
||||
]
|
||||
channels.append({"id": "sport", "name": "Sports", "categories": ["sports"]})
|
||||
streams = [
|
||||
{"channel": channel["id"], "url": "https://example.invalid/live.m3u8"}
|
||||
for channel in channels
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
collector, "_gather_iptv_org_payloads", AsyncMock(return_value=(channels, streams, []))
|
||||
)
|
||||
records = await collector._fetch_iptv_org("https://example.invalid/channels.json", config)
|
||||
assert len(records) == expected
|
||||
assert all(record["source_id"] != "sport" for record in records)
|
||||
@@ -1,5 +1,9 @@
|
||||
import pytest
|
||||
import importlib
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.core.websocket.manager import ConnectionManager
|
||||
from app.core.websocket.broadcaster import DataBroadcaster
|
||||
@@ -113,6 +117,11 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
broadcaster_module = importlib.import_module("app.core.websocket.broadcaster")
|
||||
monkeypatch.setattr(broadcaster_module.manager, "broadcast_vessels", fake_broadcast_vessels)
|
||||
broadcaster = DataBroadcaster()
|
||||
async def load_current(keys):
|
||||
assert keys == ["1"]
|
||||
return [{"mmsi": 1, "lat": 60.0, "lon": 11.0, "source": "barentswatch_vessels"}]
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", load_current)
|
||||
broadcaster.enqueue_vessel_update(
|
||||
{
|
||||
"source": "aisstream_vessels",
|
||||
@@ -129,10 +138,67 @@ async def test_vessel_broadcaster_keeps_latest_update_per_mmsi(monkeypatch):
|
||||
assert sent[0]["vessels"] == [
|
||||
{
|
||||
"mmsi": 1,
|
||||
"lat": 59.1,
|
||||
"lon": 10.1,
|
||||
"source": "aisstream_vessels",
|
||||
"lat": 60.0,
|
||||
"lon": 11.0,
|
||||
"source": "barentswatch_vessels",
|
||||
"action": "upsert",
|
||||
"created": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_subscription_delivers_every_item_in_bounded_frames():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
config = manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
json.dumps(config)
|
||||
vessels = [{"mmsi": index, "lat": 60, "lon": 10} for index in range(2501)]
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
|
||||
assert [len(frame["payload"]["vessels"]) for frame in socket.sent] == [1000, 1000, 501]
|
||||
assert [item for frame in socket.sent for item in frame["payload"]["vessels"]] == vessels
|
||||
manager.unsubscribe(socket, ["vessels"])
|
||||
await manager.broadcast_vessels({"vessels": vessels})
|
||||
assert len(socket.sent) == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_global_vessel_removal_does_not_require_coordinates():
|
||||
manager = ConnectionManager()
|
||||
socket = FakeWebSocket()
|
||||
manager.subscribe_vessels(socket, {"scope": "global", "zoom": 4})
|
||||
await manager.broadcast_vessels({"vessels": [{"mmsi": 123, "action": "remove"}]})
|
||||
assert socket.sent[0]["payload"]["vessels"] == [{"mmsi": 123, "action": "remove"}]
|
||||
|
||||
|
||||
def test_anonymous_earth_can_confirm_global_vessel_subscription(monkeypatch):
|
||||
websocket_module = importlib.import_module("app.api.v1.websocket")
|
||||
monkeypatch.setattr(websocket_module, "manager", ConnectionManager())
|
||||
app = FastAPI()
|
||||
app.include_router(websocket_module.router)
|
||||
with TestClient(app) as client, client.websocket_connect("/ws") as socket:
|
||||
assert socket.receive_json()["type"] == "connection_established"
|
||||
socket.send_json({
|
||||
"type": "subscribe",
|
||||
"data": {"channels": ["earth_updates", "vessels"], "scope": "global", "zoom": 4},
|
||||
})
|
||||
response = socket.receive_json()
|
||||
assert response["type"] == "subscription_confirmed"
|
||||
assert response["data"]["vessels"]["scope"] == "global"
|
||||
assert response["data"]["vessels"]["type"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_vessel_flush_retries_without_overwriting_newer_queued_updates(monkeypatch):
|
||||
broadcaster = DataBroadcaster()
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 59, "lon": 10}]})
|
||||
|
||||
async def fail_read(_keys):
|
||||
broadcaster.enqueue_vessel_update({"vessels": [{"mmsi": 1, "lat": 61, "lon": 12}]})
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
monkeypatch.setattr(broadcaster, "_load_current_vessel_updates", fail_read)
|
||||
with pytest.raises(RuntimeError, match="database unavailable"):
|
||||
await broadcaster.flush_vessel_updates()
|
||||
assert broadcaster._pending_vessel_updates["1"]["lat"] == 61
|
||||
|
||||
@@ -8,6 +8,178 @@ This project follows the repository versioning rule:
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.74.5] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 恢复模型供应商预设的加载与持久化,模型刷新和轻量连接检查统一使用供应商官方目录接口。
|
||||
- 修复 MiniMax M3 思考模式和 OpenCode Go 模型协议路由,补齐 Responses 适配。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 修复 `llm_provider_preset:*` 动态设置分类触发 KeyError 和列表 500,使用真实数据库读写回归验证保存与重载。
|
||||
- 统一九家供应商的目录 URL、鉴权、响应解析与分页,保留服务地域和自定义地址;明确处理限流、空目录和失败,移除 404 白名单误判成功。
|
||||
- 模型刷新读取表单草稿地址和凭证,保留默认模型、未保存修改及失败前目录;连接提示区分目录可用与实际生成成功。
|
||||
- 增加 Responses 请求与响应解析、M3 adaptive 思考适配和 OpenCode 会话标识;已知模型选择同步协议,自定义 OpenAI 网关保留手动协议。
|
||||
- 同步中英文接口说明、操作手册与快速开始,并覆盖模型目录失败、草稿保留和协议适配回归。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.4] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- 保留 Earth 全量对象与交互,优化海缆、登陆点和卫星绘制,并让 AIS / BarentsWatch 船只通过确认状态增量更新。
|
||||
- 新闻直播支持完整频道目录搜索和无限滚动;模型目录刷新可保存结果,启动流程减少不必要的依赖安装与等待。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 海缆和登陆点合批绘制,卫星 SGP4 计算移入 Worker、呼吸动画移入 GPU,并缩小动态文本的翻译扫描范围。
|
||||
- 船舶全局订阅按 MMSI 合并、拆包和原位更新,保留选择状态,并处理删除、重连及旧快照覆盖。
|
||||
- 新闻直播取消默认 120 项采集截断,增加数据库分页、完整目录搜索和固定计数栏,默认源改为半岛电视台 HLS。
|
||||
- 算力中心定位队列在当前 Earth 页面内独立于详情面板继续运行;模型刷新保存新目录并保留当前模型、凭证草稿与失败前目录。
|
||||
- AI Provider 镜像使用独立依赖组,容器直接运行已安装环境;启动提前验证数据库、及时识别后端失败,并保留可复用容器。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.3] — 2026-09-13
|
||||
|
||||
Released: 2026-09-13
|
||||
|
||||
### Highlights
|
||||
- Ubuntu / WSL 新机器初始化会自动检测并准备 Docker Engine、Compose v2、Buildx 和当前用户权限,减少手工安装步骤。
|
||||
- 数据库初始化先核对容器端口与后端真实连接,连接和认证通过后才创建表和默认数据。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 区分 Docker CLI 缺失、服务未安装、daemon 不可用及 socket 权限不足,修正未安装 Docker 时误提示启动 socket 的诊断。
|
||||
- 自动补齐缺失的 Docker 依赖并启动本地服务,以原用户身份刷新 Docker 组权限;保留参数和 PATH,不依赖 sg。
|
||||
- 通过 Compose 同步已有 PostgreSQL / Redis 容器配置,保留端口冲突等具体错误;端口映射异常时最多保留数据卷重建一次 PostgreSQL。
|
||||
- 新增后端数据库只读连接检查,对认证、库名和网络失败给出不含密码或完整连接串的诊断。
|
||||
- 将 Docker 与数据库启动隔离回归测试接入快速检查,并同步 README、harness 和中英文运维说明。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.2] — 2026-07-01
|
||||
|
||||
Released: 2026-07-01
|
||||
|
||||
### Highlights
|
||||
- 收敛 agent harness 到 `rules.md`、`AGENTS.md`、`docs/HARNESS.md` 和 `.codex/skills/`,删除重复维护的旧 Claude command 入口。
|
||||
- 强化视觉证据规则:截图或视觉引用路径打不开时必须先处理 WSL/Windows 路径、相对路径和附件位置,而不是跳过后猜测。
|
||||
- 明确 OCR 可作为文本类视觉证据或非多模态环境 fallback,同时要求布局、颜色、像素和渲染类问题保留真实视觉验证或明确限制说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `AGENTS.md` 替换旧 opencode/默认 Plan Mode 内容,保留最新单一入口和 harness 验证说明。
|
||||
- `rules.md` 与 `docs/HARNESS.md` 同步 Visual Evidence Gate,补齐路径解析、访问失败报告和 OCR fallback 边界。
|
||||
- 删除 `.claude/commands/*` 中与 `.codex/skills/*` 重复的旧 cleanup/docs/goal-driven/release 入口,并更新文档受众计划中的旧路径引用。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.1] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 将 `/earth-content` 的品牌标识上传收敛到 `Logo 地址` 与 `标题图地址` 字段内,移除旧的全局“选择资产/上传”工具栏。
|
||||
- 新增字段级图片拖拽反馈,拖到对应字段时直接提示将图片复制为 Logo 或标题图。
|
||||
- 对齐品牌上传按钮到现有 Tactile UI primary 按钮样式,并同步中英文使用手册、快速开始和控制台上下文文档。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- `BrandAssetInput` 支持字段内选择文件、拖拽上传、单字段 loading 和上传后回写草稿 URL。
|
||||
- `FieldGrid` 支持按字段注入自定义输入控件,同时复用统一草稿提交路径。
|
||||
- 品牌上传拖拽态改为低饱和 tactile 配色,上传按钮保持蓝色轻立体样式,容器内上下/右侧留白对齐为 3px。
|
||||
- 补齐品牌上传相关 legacy UI 英文翻译、术语对照和用户文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.74.0] — 2026-06-30
|
||||
|
||||
Released: 2026-06-30
|
||||
|
||||
### Highlights
|
||||
- 扩展统一 i18n 到 Web Earth、控制台、认证页和公开 Docs 的更多动态入口,减少英文界面中文残留。
|
||||
- 强化 Earth HUD 的通知胶囊、品牌栏、语言 switch、图例、tooltip、详情卡、新闻和 TV 文案展示,避免英文态裁切或错位。
|
||||
- 将容易遗漏的 i18n 入口和视觉回归加入 harness,让 smoke 覆盖通知位置、内容宽度、语言切换状态和动态文案。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 Earth runtime i18n 入口,统一高清材质、启动状态、错误提示和图层状态文案来源。
|
||||
- 补齐国家名、属性名、卫星 legend、详情页 tooltip、新闻/TV 默认文案和 API 错误提示的英文翻译与回退。
|
||||
- 更新控制台与 Earth 布局规则,保留 brand 尺寸语义,同时让标题、副标题和通知胶囊按内容完整显示。
|
||||
- 扩展 frontend smoke 与 harness 文档,固化一屏高度链、i18n 动态入口、胶囊/tag overflow 和截图证据要求。
|
||||
- 更新 Earth 新闻本地化服务与测试,确保英文界面新闻内容不再回退中文 UI 文案。
|
||||
|
||||
---
|
||||
|
||||
## [0.73.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 新增前端统一 i18n 基础设施,让认证页、Docs UI、控制台外壳、导航、搜索和核心共享组件共用 `zh-CN` / `en-US` 语言状态。
|
||||
- 控制台侧边栏偏好面板接入语言与主题切换,并修复一屏高度链、账号区、状态指示器和英文态文案裁切问题。
|
||||
- 扩展 harness 与 smoke 覆盖,确保 admin shell 高度、移动/缩放布局、语言切换、搜索、Docs 和核心控制台交互在发布前被验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `frontend/src/i18n/`,用 `i18next` / `react-i18next` 维护资源、locale 映射、Docs 兼容和过渡期 legacy UI 翻译桥。
|
||||
- 将 AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast、MarkdownRenderer 和 Users 页迁移到统一翻译资源。
|
||||
- 补齐 Planet Content、Collected Data、System Logs、Datasources、Settings 和 Collection Management 等英文态残留翻译,并覆盖动态计数字符串。
|
||||
- 改进控制台侧边栏账号区、语言 switch、状态 pill 自适应宽度和 admin shell overflow ownership,避免首屏溢出和状态词裁切。
|
||||
- 更新 i18n 计划、控制台前端上下文、harness 文档和规则,记录语言迁移边界、状态指示器布局约束和一屏验证要求。
|
||||
|
||||
---
|
||||
|
||||
## [0.72.0] — 2026-06-29
|
||||
|
||||
Released: 2026-06-29
|
||||
|
||||
### Highlights
|
||||
- 将 agent 入口收敛到单一 `AGENTS.md`,并让 harness 明确阻止小写入口再次分叉。
|
||||
- 新增完整本地 harness 验证层,覆盖 backend/frontend/docs/security 静态规则、前端 build 和 Playwright 路由/交互 smoke。
|
||||
- 扩展 Earth News 与控制台 smoke,确保新闻源测试、新增取消、手动新闻组创建、桌面/移动菜单和 zoom 布局都在发布前验证。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- 新增 `scripts/harness/*` 规则检查、doctor、validate 和前端 smoke 脚本,并将未跟踪 harness 设施纳入发布。
|
||||
- 清理 SpaceTrack 与 PeeringDB collector 的 stdout/debug 输出,改用结构化日志并移除 SpaceTrack 不可达重复 fetch 路径。
|
||||
- 强化控制台布局、auth 表单、Docs 页面、Earth shell 和 Earth toolbar 的响应式与无障碍细节。
|
||||
- 同步 README、CODEMAP、HARNESS、harness audit、用户手册、快速开始和开发者文档,明确当前 Web Earth / React admin / FastAPI / aiprovider 边界。
|
||||
- 将 backend、frontend、docs 和 Earth News 检查纳入 `scripts/harness/quick-check.sh` 与 `scripts/harness/validate.sh` 的稳定验证面。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.1] — 2026-06-26
|
||||
|
||||
Released: 2026-06-26
|
||||
|
||||
### Highlights
|
||||
- 修复 Earth 欧洲、美洲、中东与非洲等区域新闻被亚太来源和旧来源过滤饿死的问题,滚动条、面板和巡航重新回到同一批区域 payload。
|
||||
- 将当前可见新闻和巡航新闻提升到目标位置/翻译优先队列,避免历史普通 Redis backlog 阻塞用户正在看的新闻精修。
|
||||
- 新增 agent harness 入口、代码地图、验证脚本与双语技术说明,让后续维护能按现有 uv/Bun/Gitea 工作流检查而不替代项目规则。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- EarthFeed 在全局和巡航队列中按区域轮转候选新闻,保留区域视图的“当前区域 + global”规则,并补充回归测试。
|
||||
- `news.js` 按区域、类型、来源和数量隔离并发刷新请求,丢弃旧区域响应;跨区域时不再复用旧来源筛选。
|
||||
- 新闻展示在中文本地化未完成时回退原始标题和摘要,避免出现有内容却显示“新闻汉化中”的卡片。
|
||||
- 新闻目标位置 worker 新增优先 stream、pending reclaim、任务超时和并发处理;无效消息会确认并删除,减少队列堆积。
|
||||
- 补充 Earth 新闻源、Earth 前端结构、harness 和版本历史文档,并移除控制台 auth store 的调试日志。
|
||||
|
||||
---
|
||||
|
||||
## [0.71.0] — 2026-06-11
|
||||
|
||||
Released: 2026-06-11
|
||||
|
||||
### Highlights
|
||||
- 将 Motion Agent 升级为可供 Web/UE 共用的双向控制服务,补齐真实 MediaPipe 识别 worker、设备控制、动作白名单和 WSL 摄像头开箱启动链路。
|
||||
- 新增 Earth 手动新闻内容组、条目、导入与重处理能力,并改进按 locale 和启用来源进行的新闻补充与多样化。
|
||||
- 对齐动捕模式下的 Earth 点击、详情锁定和卫星轨迹交互,同时完善启动脚本、测试 harness 与运维说明。
|
||||
|
||||
### Added / Fixed / Improved
|
||||
- Motion Agent 支持命令结果、状态、骨架与手势事件,统一单路/双路输入配置,并随 `planet.sh` 默认启动;仓库内提供 usbipd-win fallback 安装包。
|
||||
- Earth 新闻服务集中处理显示就绪判断和来源多样化,避免存储层与编排层重复筛选;新增手动新闻 API 与回归测试。
|
||||
- 清理 Motion Agent 重复识别执行路径、前端不稳定随机 key 和过时计划描述,补齐 pytest 路径 harness、双语使用手册、快速开始与数据流文档。
|
||||
|
||||
---
|
||||
|
||||
## [0.70.0] — 2026-06-04
|
||||
|
||||
Released: 2026-06-04
|
||||
|
||||
325
docs/HARNESS.md
Normal file
@@ -0,0 +1,325 @@
|
||||
# Agent Harness
|
||||
|
||||
This harness improves discoverability, repeatability, and agent safety for the
|
||||
existing Planet project. It does not replace current project rules, scripts, CI,
|
||||
or release workflows.
|
||||
|
||||
## Authority And Conflicts
|
||||
|
||||
Existing project rules are authoritative:
|
||||
|
||||
1. `rules.md`
|
||||
2. `AGENTS.md`
|
||||
3. Current implementation docs under `docs/technical/`
|
||||
4. Existing scripts, especially `planet.sh`
|
||||
5. Existing Gitea workflow files under `.gitea/workflows/`
|
||||
|
||||
When harness guidance conflicts with any of the above, keep the existing rule,
|
||||
do not overwrite the existing workflow, and add a compatibility note here or in
|
||||
`docs/harness-audit.md`.
|
||||
|
||||
For frontend or documentation audits, also read the Rules Coverage Evidence
|
||||
section in `docs/harness-audit.md`. It maps `rules.md` clauses to the current
|
||||
static checks, Playwright smoke coverage, and remaining manual review areas, so
|
||||
an agent can distinguish a proved harness pass from a rule that still needs
|
||||
human-quality inspection.
|
||||
|
||||
When the user describes work with product words rather than module names, use
|
||||
the `rules.md` **Agent Discovery Index** before deciding which modules to load.
|
||||
It maps Chinese phrases such as `一屏`, `高度没控住`, `文档`, `数据源`,
|
||||
`地球`, `模型供应商`, and `发版` to the required rule modules.
|
||||
|
||||
## Starting Work
|
||||
|
||||
Recommended startup flow:
|
||||
|
||||
```bash
|
||||
git status --short
|
||||
scripts/harness/doctor.sh
|
||||
```
|
||||
|
||||
Then read only the relevant implementation docs:
|
||||
|
||||
- Backend/API/data work: `docs/technical/zh/backend-*.md` and matching English
|
||||
docs when public docs are affected.
|
||||
- Frontend/admin work: `docs/technical/zh/frontend-admin-frontend-context.md`.
|
||||
- Earth work: `docs/technical/zh/earth-frontend-context.md`,
|
||||
`docs/technical/zh/earth-render-layer-order.md`, and style docs when visual
|
||||
semantics change.
|
||||
- Operations work: `docs/technical/zh/ops-runbook.md` and
|
||||
`docs/technical/zh/ops-planet-sh-startup.md`.
|
||||
- AI Provider work: `docs/technical/zh/agents-aiprovider.md`.
|
||||
- Documentation work: `docs/documentation-coverage-rules.md`.
|
||||
|
||||
Use focused inspection commands before broad reads:
|
||||
|
||||
```bash
|
||||
rg -n "<symbol-or-term>" <path>
|
||||
git diff --stat HEAD
|
||||
git diff --name-only HEAD
|
||||
git diff --unified=0 HEAD -- <path>
|
||||
```
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Purpose | Command |
|
||||
| --- | --- |
|
||||
| First setup | `./planet.sh init` |
|
||||
| Start local stack | `./planet.sh start` |
|
||||
| Start with LAN access | `./planet.sh start --allow-lan` |
|
||||
| Restart all services | `./planet.sh restart` |
|
||||
| Restart one area | `./planet.sh restart -b`, `-f`, `-a`, or `-d` |
|
||||
| Health check | `./planet.sh health` |
|
||||
| Logs | `./planet.sh log`, `./planet.sh log -b`, `-f`, `-a`, or `-m` |
|
||||
| Create local user | `./planet.sh createuser` |
|
||||
| Destructive local reset | `./planet.sh destroy` |
|
||||
| Backend smoke tests | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` |
|
||||
| Mock AIS WebSocket | `bun run mock:ais-ws` |
|
||||
|
||||
## Harness Commands
|
||||
|
||||
| Tier | Command | What It Does |
|
||||
| --- | --- | --- |
|
||||
| Doctor | `scripts/harness/doctor.sh` | Checks required files, required tools, optional delivery tools, and forbidden frontend lockfiles. |
|
||||
| Security | `scripts/harness/security-check.sh` | Checks that environment/private-key files are not tracked and scans for high-confidence committed secret tokens. |
|
||||
| Backend Rules | `scripts/harness/backend-rules-check.sh` | Checks backend app Python for direct `print()`, `breakpoint()`, and `pdb.set_trace()` debug calls so service code uses structured logging. |
|
||||
| Frontend Rules | `scripts/harness/frontend-rules-check.sh` | Checks Bun-only scripts, admin route manifest coherence, literal internal route links, admin search route targets, frontend debug output, native button safety, icon-button accessibility, no nested Cards, no AntD/Space layout primitives, ConnectionTestInput usage, admin/docs shell height-chain sizing, same-category style owner warnings, viewport-scaled font sizes, zero letter spacing, and high-signal UI rule warnings. |
|
||||
| Docs Consistency | `scripts/harness/docs-consistency-check.sh` | Checks frontend Docs metadata against backend Gatekeeper metadata, public Docs registration, full technical-doc bilingual file pairs, public doc links, readable link titles, language-scoped technical links, README/project-context admin stack drift, supported credential collector contracts, manual console route coverage against the actual admin manifest, documented UI route drift, documented `?section=` deep-link validity against the actual admin section config in technical docs and active plan docs, and the harness rules-coverage notes. |
|
||||
| Quick | `scripts/harness/quick-check.sh` | Runs doctor, whitespace diff check, shell syntax checks, security scan, backend/frontend/doc consistency checks, and CI backend smoke tests. |
|
||||
| Full | `scripts/harness/validate.sh` | Runs quick check, frontend Bun install/build, Playwright route smoke, optional Helm checks, and opt-in Docker image smoke builds. |
|
||||
|
||||
Docker image smoke builds are expensive and are off by default:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_DOCKER_SMOKE=1 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
Frontend Playwright smoke runs by default in full validation after the frontend
|
||||
build. It starts a local Vite preview and checks the `/` to Earth redirect,
|
||||
public pages, unknown-route login fallback, protected admin route login
|
||||
fallback, authenticated unknown-route fallback to `/admin`, Docs loading with
|
||||
mocked API content, Docs detail page
|
||||
language/theme/search interactions, every Docs catalog slug exposed by the
|
||||
frontend/backend metadata, the Earth iframe entry point, login error handling,
|
||||
register + email verification, password reset, standalone email verification,
|
||||
and authenticated `super_admin` rendering for every admin route plus core
|
||||
`section` deep links derived from the actual admin route and section config.
|
||||
Authenticated admin
|
||||
checks run at desktop size, mobile size, and 125% / 150% zoom; desktop and
|
||||
mobile passes also fail on global horizontal overflow so table/detail panels
|
||||
must keep overflow ownership inside their own scroll regions. To enforce the
|
||||
existing `rules.md` `uiux` one-screen workspace rule, admin shell pages have a
|
||||
hard rendered check: the shell must resolve to the viewport height through the
|
||||
root 100% height chain, `#root`/document/body must not gain vertical overflow,
|
||||
and the desktop sidebar account/preferences area must remain inside the first
|
||||
viewport while the nav owns any excess scrolling. The smoke also
|
||||
derives the sidebar menu from the actual admin route manifest and clicks every
|
||||
visible `super_admin` menu entry on both desktop and mobile viewports, then
|
||||
exercises safe interaction paths for admin search, section tabs, the AI settings
|
||||
shortcut, logs view switching, user dialog opening, and data distribution toggles.
|
||||
It also exercises Earth News source testing, add/cancel source draft behavior,
|
||||
and manual news group creation against mocked `/earth/news-*` APIs.
|
||||
Documented AI and collector
|
||||
deep links such as `/ai?section=integrations`, `/ai?section=playground`, and
|
||||
`/collection-management?section=collector_credentials` are part of the rendered
|
||||
smoke surface:
|
||||
|
||||
```bash
|
||||
PLANET_HARNESS_FRONTEND_SMOKE=0 scripts/harness/validate.sh
|
||||
PLANET_HARNESS_FRONTEND_SMOKE_PORT=4174 scripts/harness/validate.sh
|
||||
```
|
||||
|
||||
### Visual Evidence And Style Consistency
|
||||
|
||||
- Treat user-provided screenshots and images as primary visual evidence. If a
|
||||
screenshot contradicts written text, inspect the image first and explicitly
|
||||
call out the mismatch before deciding what to change.
|
||||
- Path resolution is part of visual evidence handling. If a referenced
|
||||
screenshot path cannot be opened, try reasonable local equivalents first:
|
||||
WSL/Windows path conversion, workspace-relative paths, absolute paths, current
|
||||
thread attachments, repository files, and obvious local attachment/download
|
||||
locations.
|
||||
- If the image still cannot be found or opened, report the exact path/access
|
||||
blocker instead of guessing. Do not infer image content from the filename, alt
|
||||
text, surrounding prose, logs, or memory.
|
||||
- OCR is acceptable evidence for text-only questions or non-multimodal
|
||||
environments; state when OCR was the fallback. Layout, color, spacing, pixel,
|
||||
and rendering issues still require a real visual inspection or an explicit
|
||||
"could not verify visually" note.
|
||||
- Same-category UI surfaces must use one visual system per product area. Badges,
|
||||
chips, pills, tags, status labels, small buttons, cards, panels, and toolbar
|
||||
controls should reuse the shared component, shared token, or established CSS
|
||||
owner for that area instead of introducing a page-local lookalike.
|
||||
- `scripts/harness/frontend-rules-check.sh` warns when semantic
|
||||
`badge` / `chip` / `pill` / `tag` / `status` selectors appear outside the
|
||||
approved React and Earth CSS owner files. A warning means the reviewer should
|
||||
either move the style into the shared owner or document why this is a genuinely
|
||||
new visual family.
|
||||
|
||||
### Earth I18n Harness Rules
|
||||
|
||||
Earth i18n work must validate rendered behavior, not only static text lookup.
|
||||
Agents often miss dynamic strings that are created after initial page load, so
|
||||
the smoke treats these as first-class i18n surfaces:
|
||||
|
||||
- **Visible text and attributes**: translated checks must include `innerText`
|
||||
plus `title`, `aria-label`, `placeholder`, and `alt`. Tooltips and icon-only
|
||||
buttons are user-facing copy, not implementation details.
|
||||
- **Dynamic detail cards**: info cards opened from Earth markers, cruise cards,
|
||||
BGP markers, compute centers, vessels, and news must render field labels,
|
||||
status values, source tags, action buttons, and disabled/tooltips in the active
|
||||
language.
|
||||
- **English content safety**: English mode must not fall back to Chinese news
|
||||
titles, summaries, feed names, measure words, or generic status labels. If no
|
||||
English localization exists, hide the item or use a neutral English fallback.
|
||||
- **Brand assets**: locale switching must update both text and image assets.
|
||||
The default Earth HUD brand uses `title-zh.png` for Chinese and `title-en.png`
|
||||
for English while keeping the same top-left layout and logo position.
|
||||
- **Controls and state**: switch/segmented-control visuals must follow the real
|
||||
checked/pressed state after both direct clicks and programmatic panel changes.
|
||||
A control is not valid if the state changes but the thumb, active pill, or
|
||||
`aria-*` state stays stale.
|
||||
- **Runtime copy entrypoints**: dynamic status, loading, startup, and error copy
|
||||
must enter through `earthMessage(...)` plus the centralized
|
||||
`EARTH_MESSAGE_TEMPLATES` map. Do not hide direct strings behind
|
||||
`showStatusMessage`, `queueStatusMessage`, `showGestureStatusMessage`,
|
||||
`showError`, `setLoadingMessage`, `resolveStartupMessage`, `startupMessage`,
|
||||
or `earth:status` events.
|
||||
- **Capsules and tags**: pills, tags, chips, badges, and small buttons must not
|
||||
overflow their panel. Prefer a slightly wider owning panel for important
|
||||
status information; otherwise use `min-width: 0`, wrapping, or ellipsis with a
|
||||
translated tooltip.
|
||||
|
||||
Current frontend smoke explicitly covers the Earth English locale flow: brand
|
||||
image swap, settings language controls, panel switch visual sync, English news
|
||||
filtering, English detail-card text and tooltips, and TV default/source labels.
|
||||
|
||||
## Environment Requirements
|
||||
|
||||
Required for normal development:
|
||||
|
||||
- `zsh` for `planet.sh`
|
||||
- `uv` for Python dependency and test execution
|
||||
- `bun` for frontend dependency and build execution
|
||||
- Python resolved by `uv` from the root `pyproject.toml`
|
||||
|
||||
Harness command lookup first checks the current non-interactive `PATH`. If a
|
||||
required tool is not visible there, `scripts/harness/lib.sh` asks the user's
|
||||
login interactive shell (`$SHELL`, then `zsh`, then `bash`) for the command
|
||||
path. This avoids hardcoding a dotfile while still covering agent environments
|
||||
that do not inherit the user's normal shell setup.
|
||||
|
||||
Required for full local stack operation:
|
||||
|
||||
- Docker and Docker Compose
|
||||
- PostgreSQL and Redis containers started by `planet.sh`
|
||||
|
||||
Optional for delivery smoke:
|
||||
|
||||
- Docker daemon for image builds
|
||||
- Helm for chart lint/template checks
|
||||
|
||||
For routine harness validation, do not install missing system software
|
||||
automatically. Report the gap and point to the explicit bootstrap entry points.
|
||||
`./planet.sh init` can install missing Docker Engine, Compose v2, and Buildx on
|
||||
Ubuntu / Ubuntu WSL, start the local service, and configure Docker group access.
|
||||
This bootstrap behavior is intentional; do not invoke it merely to make harness
|
||||
checks pass. `scripts/bootstrap-dev.sh` only prepares application dependencies.
|
||||
|
||||
Docker bootstrap regression checks use isolated command stubs and never install
|
||||
packages or modify the host daemon:
|
||||
|
||||
```bash
|
||||
uv run --frozen --project . python scripts/harness/test_docker_bootstrap.py
|
||||
uv run --frozen --project . python scripts/harness/test_database_startup.py
|
||||
```
|
||||
|
||||
Database startup regressions also run in quick-check. They cover Compose
|
||||
reconciliation of existing containers, visible startup errors, published-port
|
||||
checks, bounded recreation that preserves volumes, and the backend connection
|
||||
gate before schema initialization. Their command stubs and driver mocks do not
|
||||
modify the host Docker environment.
|
||||
|
||||
## What Agents Must Not Change Automatically
|
||||
|
||||
- Do not replace Bun with npm, pnpm, or yarn.
|
||||
- Do not migrate CI from `.gitea/workflows/` to `.github/workflows/`.
|
||||
- Do not rewrite `planet.sh` lifecycle behavior as a parallel script.
|
||||
- Do not run `./planet.sh destroy` unless explicitly requested.
|
||||
- Do not commit `.env`, secrets, private keys, logs, or generated build output.
|
||||
- Do not add external integrations, hooks, or new dependency managers just to
|
||||
satisfy harness structure.
|
||||
- Do not publish internal harness docs into the product Docs UI unless a
|
||||
maintainer explicitly asks for it.
|
||||
|
||||
## Hooks And Reminders
|
||||
|
||||
No automatic hooks are installed in this phase. Manual reminders:
|
||||
|
||||
- Run `scripts/harness/quick-check.sh` before handing off small changes.
|
||||
- Run `scripts/harness/validate.sh` before larger cross-subsystem changes.
|
||||
- Run `scripts/harness/security-check.sh` after touching config, auth,
|
||||
credentials, docs examples, or generated fixtures.
|
||||
- Run `scripts/harness/backend-rules-check.sh` after backend service edits to
|
||||
catch direct stdout/debugger calls before they reach runtime logs.
|
||||
- Run `scripts/harness/frontend-rules-check.sh` after frontend edits to expose
|
||||
route, package-manager, debug-output, and UI rule warnings.
|
||||
- Run `scripts/harness/docs-consistency-check.sh` after docs edits or feature
|
||||
route changes.
|
||||
- Add focused tests before modifying backend service behavior or frontend
|
||||
workflows.
|
||||
- For docs changes, run the checks listed in
|
||||
`docs/documentation-coverage-rules.md`.
|
||||
|
||||
## Reusable Workflows
|
||||
|
||||
### Feature Work
|
||||
|
||||
1. Read `rules.md` modules for the touched area.
|
||||
2. Check `CODEMAP.md` for entry points and ownership boundaries.
|
||||
3. Inspect existing tests and docs before editing.
|
||||
4. Make the smallest behavior-preserving or feature-scoped change.
|
||||
5. Run `scripts/harness/quick-check.sh` or a narrower documented command.
|
||||
6. Update relevant docs when behavior, workflow, or operations change.
|
||||
7. For rendered frontend changes, verify the affected route with Playwright or
|
||||
the full harness smoke, because `bun run build` alone does not prove page
|
||||
usability.
|
||||
|
||||
### Bug Fix
|
||||
|
||||
1. Reproduce with a focused test or command.
|
||||
2. Patch the owning module, not a caller-side workaround.
|
||||
3. Run the focused regression test.
|
||||
4. Run `scripts/harness/quick-check.sh` when the change is safe to validate
|
||||
locally.
|
||||
|
||||
### Documentation Change
|
||||
|
||||
1. Read `docs/documentation-coverage-rules.md`.
|
||||
2. Route docs by audience: UI users, operations, or second-party developers.
|
||||
3. Keep Chinese and English technical docs paired by filename; public Docs also
|
||||
need matching frontend/backend metadata when exposed in the product Docs UI.
|
||||
4. Run the repository-specific docs checks that match the changed files.
|
||||
|
||||
### Release Or Delivery Change
|
||||
|
||||
Use the existing release skill/workflow and `.gitea/workflows/` files. Harness
|
||||
validation can smoke-check Helm and Docker locally, but it must not replace the
|
||||
release process.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `docs/harness-audit.md` records the discovery pass that led to this harness.
|
||||
- `AGENTS.md` is the single authoritative agent guide. The older lowercase
|
||||
`agents.md` entry has been merged into it and should remain absent.
|
||||
- `CODEMAP.md` is intentionally high level; deeper subsystem docs stay in
|
||||
`docs/technical/{zh,en}/`.
|
||||
- `scripts/harness/frontend-smoke.mjs` is a lightweight route/section smoke
|
||||
with mocked API data. It proves route shells, auth guards, and primary admin
|
||||
sections render, but it is not a replacement for feature-specific browser QA
|
||||
against a real backend.
|
||||
- Frontend smoke prints phase-level progress by default. Use
|
||||
`PLANET_FRONTEND_SMOKE_PROGRESS=verbose` to print each route/menu/doc item
|
||||
when diagnosing a slow or failing smoke run, or set it to `0` to suppress
|
||||
progress lines.
|
||||
161
docs/harness-audit.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Harness Audit
|
||||
|
||||
Last audited: 2026-06-26
|
||||
|
||||
This audit records the repository state used to add the agent harness. It is a
|
||||
compatibility note, not a replacement for existing rules or architecture docs.
|
||||
|
||||
## Existing Commands
|
||||
|
||||
| Area | Existing Command | Notes |
|
||||
| --- | --- | --- |
|
||||
| Bootstrap | `./planet.sh init` | Syncs uv/Bun dependencies, creates missing env files, starts data services, seeds defaults. |
|
||||
| Start | `./planet.sh start` | Starts backend, frontend, AI Provider, PostgreSQL/Redis, and Motion Agent when available. |
|
||||
| LAN start | `./planet.sh start --allow-lan` | Opens frontend/backend/AI Provider ports and requests Windows firewall/port cleanup when needed. |
|
||||
| Restart | `./planet.sh restart` | Supports scoped restart flags for backend, frontend, AI Provider, database, and Motion Agent. |
|
||||
| Health | `./planet.sh health` | Checks containers, backend `/health`, AI Provider `/health`, frontend, and Motion Agent state. |
|
||||
| Logs | `./planet.sh log` | Supports backend, frontend, AI Provider, and Motion Agent log views. |
|
||||
| User fallback | `./planet.sh createuser` | Interactive emergency/local account creation. |
|
||||
| Destructive reset | `./planet.sh destroy` | Requires confirmation and removes Planet-owned Docker/build/runtime state. Not a validation command. |
|
||||
| Backend CI smoke | `cd backend && uv run --frozen --group dev --project .. python -m pytest -s tests/test_api.py tests/test_realtime_sources.py -q` | Mirrors `.gitea/workflows/ci.yaml`. |
|
||||
| Frontend build | `cd frontend && bun install --frozen-lockfile && bun run build` | Bun-only workflow. |
|
||||
| Root helper | `bun run mock:ais-ws` | Runs `scripts/mock-ais-ws-server.ts` from the root package. |
|
||||
|
||||
## Existing Agent Instructions
|
||||
|
||||
| File | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `AGENTS.md` | Present | Single authoritative agent behavior guide. It references `rules.md`, `project_context.md`, harness validation, and high-risk areas. |
|
||||
| `rules.md` | Present | Mandatory modular rules. Always load `core`, `security`, and `workflow`; load topic modules as needed. |
|
||||
| `project_context.md` | Present | Static context. Some roadmap-era stack details are older than the current README/docs. |
|
||||
| `.claude/commands/*.md` | Present | Existing command docs for cleanup, docs, goal-driven, and release workflows. |
|
||||
| `.codex/skills/*.md` | Present | Existing local skills for cleanup, docs, goal-driven, and release. |
|
||||
|
||||
## Existing CI Gates
|
||||
|
||||
The repository uses `.gitea/workflows/`, not `.github/workflows/`.
|
||||
|
||||
| Workflow | Gate |
|
||||
| --- | --- |
|
||||
| `.gitea/workflows/ci.yaml` | Backend smoke tests, frontend Bun build, Docker build smoke, Helm lint/template. |
|
||||
| `.gitea/workflows/release.yaml` | Builds and pushes frontend, backend, and AI Provider images on main/tag/manual release events. |
|
||||
| `.gitea/workflows/deploy-staging.yaml` | Deploys Helm release to staging and runs curl smoke tests inside the cluster. |
|
||||
|
||||
## Existing Docs And Architecture Maps
|
||||
|
||||
| Area | Docs |
|
||||
| --- | --- |
|
||||
| Current architecture and startup | `README.md` |
|
||||
| Technical docs index | `docs/technical/zh/README.md`, `docs/technical/en/README.md` |
|
||||
| Documentation rules | `docs/documentation-coverage-rules.md` |
|
||||
| Operations | `docs/technical/zh/ops-runbook.md`, `docs/technical/en/ops-runbook.md` |
|
||||
| Startup internals | `docs/technical/zh/ops-planet-sh-startup.md`, `docs/technical/en/ops-planet-sh-startup.md` |
|
||||
| AI Provider | `docs/technical/zh/agents-aiprovider.md`, `docs/technical/en/agents-aiprovider.md` |
|
||||
| Frontend admin | `docs/technical/zh/frontend-admin-frontend-context.md`, `docs/technical/en/frontend-admin-frontend-context.md` |
|
||||
| Earth rendering | `docs/technical/zh/earth-frontend-context.md`, `docs/technical/zh/earth-render-layer-order.md`, `docs/technical/zh/earth-layer-style-reference.md` |
|
||||
| Plans and history | `docs/plans/README.md`, `docs/deprecated/README.md` |
|
||||
|
||||
## Release And Deploy Process
|
||||
|
||||
- Release workflow is documented in `.codex/skills/release/SKILL.md` and
|
||||
`.claude/commands/release.md`.
|
||||
- Version-bearing files include `VERSION`, `frontend/package.json`,
|
||||
`pyproject.toml`, `uv.lock`, `docs/CHANGELOG.md`, and
|
||||
`docs/version-history.md`.
|
||||
- Delivery automation lives in `.gitea/workflows/release.yaml` and
|
||||
`.gitea/workflows/deploy-staging.yaml`.
|
||||
- Helm chart entry point is `deploy/helm/planet/Chart.yaml`.
|
||||
|
||||
## Missing Or Unclear Areas
|
||||
|
||||
- The older lowercase `agents.md` entry has been merged into uppercase
|
||||
`AGENTS.md` so coding agents and harness tools use one source of truth.
|
||||
- `project_context.md` originally included older roadmap assumptions such as
|
||||
Celery, Kafka, TimescaleDB, MinIO, and UE5 as active stack elements. The
|
||||
harness pass updated it to separate active stack facts from future directions;
|
||||
current code and technical docs still remain authoritative when details drift.
|
||||
- No safe automatic hook system was already configured. This phase documents
|
||||
manual reminders instead of adding hooks.
|
||||
- `.github/workflows/` is absent by design; CI is under `.gitea/workflows/`.
|
||||
|
||||
## Conflicts And Preserved Rules
|
||||
|
||||
| Conflict Or Tension | Resolution |
|
||||
| --- | --- |
|
||||
| Prompt suggested `AGENTS.md`; repository already had `agents.md`. | Merged the lowercase guide into uppercase `AGENTS.md`; harness doctor now requires `AGENTS.md` and keeps `agents.md` absent to prevent split authority. |
|
||||
| Harness validation could duplicate CI. | Added wrapper scripts that call existing commands and mirror current CI gates where practical. |
|
||||
| Full Docker smoke builds are expensive locally. | Kept them opt-in with `PLANET_HARNESS_DOCKER_SMOKE=1`. |
|
||||
| Internal harness docs could clutter public Docs UI. | Kept `docs/HARNESS.md` and `docs/harness-audit.md` as repository docs, not product Docs entries. |
|
||||
| Existing frontend toolchain is Bun-only. | Harness scripts and docs use Bun only and flag npm/pnpm/yarn lockfiles as failures. |
|
||||
| Agents often miss user-installed Bun or uv in non-interactive shells. | Added `scripts/harness/lib.sh` to resolve tools from current `PATH` first and then the user's login interactive shell without hardcoding a dotfile. |
|
||||
| Always-loaded security rules had no standalone harness gate. | Added `scripts/harness/security-check.sh` to block tracked `.env` / key files and scan for high-confidence committed private keys or provider tokens; quick-check now runs it. |
|
||||
| Build success does not prove frontend page usability. | Added static frontend rules/doc checks and a Playwright route smoke for public pages, protected admin fallback, Docs loading and detail interactions, Earth iframe entry, login/register/verification/password-reset interactions, authenticated admin route/section rendering with mocked API data across desktop, mobile, and 125% / 150% zoom, plus manifest-derived desktop/mobile menu navigation and safe search/tab/dialog/Earth News interactions. |
|
||||
| Route fallback behavior can regress even when every named page renders. | Extended the frontend smoke to verify `/` redirects to Earth, unauthenticated unknown routes show the login page, and authenticated unknown routes navigate back to `/admin`. |
|
||||
| Frontend smoke route lists can drift from `AdminRoutes` and resource-page sections. | Updated the smoke to derive protected route checks and authenticated section deep-link checks from `AdminRoutes.tsx` and `PlainResourcePages.tsx`, including redirect-only `/alerts`. |
|
||||
| Docs smoke mocks can drift from the product Docs catalog. | Updated the frontend smoke to derive mocked Docs catalog/content from `frontend/src/pages/Docs/docs-content.ts` plus backend Gatekeeper access metadata, then open every Chinese Docs catalog slug. |
|
||||
| User manuals can miss a real console menu entry after route changes. | Added a docs consistency check that compares the manual console overview tables with `frontend/src/admin/routes/manifest.tsx`; fixed the missing `/docs` row in both user manuals. |
|
||||
| Rendered pages can still contain broken internal shortcuts. | Added literal internal route-link checks and an interaction smoke for the AI settings shortcut; this caught and fixed a stale `/admin/settings` link that should point to `/settings`. |
|
||||
| Global search entries can drift because their route targets live in data objects rather than JSX links. | Added a frontend rules check that validates every admin search `routePath` against the actual frontend route set. |
|
||||
| Responsive styling fixes can satisfy one viewport by breaking the no-viewport-font rule. | Added a frontend rules failure for `font-size` values that use viewport or container query width units, and replaced public auth shell `vw` font sizing with fixed desktop/mobile sizes. |
|
||||
| Typography polish can accidentally reintroduce squeezed non-zero letter spacing. | Normalized active frontend `letter-spacing` values to `0` and made the frontend rules check fail non-zero `letter-spacing` / `letterSpacing` declarations, with only inherit/default-zero forms allowed. |
|
||||
| Native buttons can accidentally submit forms or keep controls clickable while loading after a props-spread reorder. | Added a frontend rules failure for TSX `<button>` elements without explicit `type` and for buttons whose `disabled` state can be overridden by a later props spread; fixed the data distribution buttons and auth button disabled ordering. |
|
||||
| Admin/docs shell layouts can reintroduce brittle viewport sizing after a responsive fix. | Changed the admin and Docs route shells to use the existing `html/body/#root` 100% height chain, and added a frontend rules failure for exact `100vh` / `100vw` shell sizing in those CSS files. |
|
||||
| Compact workspaces can drift back into card-in-card layouts or implicit AntD `Space` wrappers. | Added frontend rules failures for nested `Card` components, AntD imports, and `<Space>` layout primitives in active frontend source. |
|
||||
| Connection-test controls can drift back into detached toolbar buttons. | Added a shared `ConnectionTestInput` suffix pattern for AI Provider and WebSearch Base URL fields, disabled WebSearch configuration/test controls when the tool is off, and made the frontend rules check fail detached AI/WebSearch connection-test buttons. |
|
||||
| Same-category UI styles can fragment into page-local lookalikes. | Added same-category style owner warnings for semantic `badge`, `chip`, `pill`, `tag`, and `status` CSS selectors outside the approved shared React and Earth CSS owner files. |
|
||||
| Visual fixes can go wrong when agents guess from missing screenshot paths. | Documented screenshots and images as primary visual evidence: if the path is not available, agents must search alternate attachment/local locations or report the blocker instead of inferring image content. |
|
||||
| Public docs can reference stale admin section URLs. | Added docs consistency validation for documented `?section=` links and rendered smoke coverage for documented AI / collector deep links. |
|
||||
| Active plan docs can preserve old admin deep-link assumptions after the technical docs are corrected. | Extended docs consistency checks to active `docs/plans/*.md` files for stale admin tab-query terms and actual `?section=` validity; corrected the docs audience split plan to current section routes. |
|
||||
| Top-level README can drift from the actual frontend stack while technical docs stay current. | Updated README from Ant Design Pro to Tactile UI / Radix primitives / lucide-react and added README stale admin-stack terms to docs consistency checks. |
|
||||
| Agent background context can reintroduce inactive stack assumptions. | Updated `project_context.md` and the root agent guide to label current stack facts versus future directions, then added exact stale-stack patterns for them to docs consistency checks. |
|
||||
| Docs `?section=` validation can drift if the harness owns its own route/section table. | Changed the docs consistency check to derive section keys from `AdminRoutes.tsx` and `PlainResourcePages.tsx` resource configs before validating documented deep links. |
|
||||
| Public Docs can drift between frontend catalog metadata and backend Gatekeeper authorization metadata. | Added a docs consistency check that compares filename, slug, group, order, and bilingual titles across both metadata sources; aligned existing order drift for toolbar overlay and location pipeline docs. |
|
||||
| Non-public technical docs can silently become Chinese-only or English-only. | Added a full `docs/technical/{zh,en}` filename-pair check so every technical Markdown file has a same-named counterpart before docs consistency passes. |
|
||||
| Credentialed collector docs can drift from backend support wiring. | Added docs consistency validation for every built-in collector marked `requires_credentials=true` and `credential_status=supported`: it must have a provider, default credential guide, supported connectivity provider, frontend credential UI/guidance, a regression test, and zh/en connectivity documentation. |
|
||||
| Backend collectors can leak debug output or credential-adjacent context through stdout. | Replaced SpaceTrack and PeeringDB collector `print()` calls with structured logger events, removed unreachable duplicate SpaceTrack fetch code, and added `scripts/harness/backend-rules-check.sh` to block future backend app `print()`, `breakpoint()`, or `pdb.set_trace()` calls. |
|
||||
|
||||
## Rules Coverage Evidence
|
||||
|
||||
This matrix records how the current harness checks the `rules.md` modules that
|
||||
matter for this frontend and documentation pass. "Automated" means the listed
|
||||
command fails when the rule regresses. "Smoke" means the rendered product route
|
||||
or interaction is opened with Playwright. "Manual" means the rule is still a
|
||||
judgment call and must be inspected during review.
|
||||
|
||||
Before using this matrix, start from `rules.md`'s **Agent Discovery Index** when
|
||||
the user describes work with Chinese/product terms instead of module names. The
|
||||
index is the routing layer; this table is the coverage/evidence layer.
|
||||
|
||||
| `rules.md` Area | Rule Surface | Harness Evidence | Remaining Review |
|
||||
| --- | --- | --- | --- |
|
||||
| `core` | Remove stale transitional paths, duplicated helpers, and naming drift after large changes. | `scripts/harness/docs-consistency-check.sh` blocks known stale stack terms, old `?tab=` links, public Docs metadata drift, and README/project context drift. `scripts/harness/frontend-rules-check.sh` blocks repeated detached AI/WebSearch connection-test buttons by requiring `ConnectionTestInput`. | Naming quality, function size, and whether a new abstraction is worth keeping remain manual review items. |
|
||||
| `core` | Keep one source of truth for route, Docs, and section state. | Frontend route, admin manifest, admin search targets, Docs catalog metadata, backend Gatekeeper metadata, manual route tables, and documented `?section=` links are all parsed from source and compared by `frontend-rules-check.sh`, `docs-consistency-check.sh`, and `frontend-smoke.mjs`. | Business-state ownership inside feature components still needs focused review when behavior changes. |
|
||||
| `security` | Do not commit secrets, tracked env files, private keys, or exposed tokens. | `scripts/harness/security-check.sh` fails on tracked `.env` / private-key files and high-confidence provider tokens. `backend-rules-check.sh` blocks backend stdout/debugger calls, and `frontend-rules-check.sh` fails frontend console output that includes token material. | Whether a newly added setting should be masked or stored server-side still requires feature-specific review. |
|
||||
| `workflow` | Frontend package management must stay Bun-only. | `scripts/harness/doctor.sh` and `frontend-rules-check.sh` fail forbidden frontend lockfiles and `npm` / `pnpm` / `yarn` script usage. `validate.sh` uses Bun for install, build, preview, and smoke. | New dependency legitimacy and maintenance quality are manual unless a dependency is actually added. |
|
||||
| `workflow` | Agents should find `bun` and `uv` even when non-interactive `PATH` is incomplete. | `scripts/harness/lib.sh` checks the current `PATH`, then asks `$SHELL`, `zsh`, and `bash` login interactive shells for the command path without hardcoding a dotfile. `doctor.sh`, `quick-check.sh`, and `validate.sh` all source it. | System package installation remains outside harness scope and should be reported instead of auto-fixed. |
|
||||
| `docs` | Keep public Docs whitelist-driven and synchronized with backend authorization metadata. | `docs-consistency-check.sh` compares frontend Docs metadata against backend Gatekeeper metadata, verifies files exist for both languages, checks public link titles, and blocks missing zh/en technical doc pairs. `frontend-smoke.mjs` opens every Chinese Docs catalog slug plus detail/search/language/theme interactions. | Quality of prose, examples, and whether a doc should be public are still editorial review items. |
|
||||
| `docs` | User manuals must match real console routes and deep links. | `docs-consistency-check.sh` compares manual console tables with `frontend/src/admin/routes/manifest.tsx` and validates documented `?section=` links from actual `AdminRoutes.tsx` plus `PlainResourcePages.tsx` section config. | Screenshots and UI-copy nuance are not exhaustively validated. |
|
||||
| `uiux` | Admin pages are compact single-screen workspaces with explicit overflow ownership. | `frontend-rules-check.sh` fails missing admin shell height-chain declarations (`.admin-theme-root`, `.admin`, `.admin__sider`, `.admin__nav-scroll`, `.admin__account`, `.admin__content`, `.admin__content-inner`), warns on suspicious `overflow: hidden`, and blocks exact `100vh` / `100vw` shell sizing in admin/Docs CSS. `frontend-smoke.mjs` checks every admin route at desktop/mobile and verifies `.admin` equals viewport height, `#root`/document/body have no vertical overflow, and desktop sidebar account/preferences stay in the first viewport. Zoom passes still cover 125% / 150% rendering. | Visual density, hierarchy, and whether a scroll owner feels ergonomic remain manual QA. |
|
||||
| `uiux` | Controls use expected patterns and accessible icon buttons. | `frontend-rules-check.sh` blocks icon `Button` without `aria-label` and `title`, native `<button>` without explicit `type`, nested Cards, AntD imports, `<Space>`, and detached connection-test buttons. Smoke exercises search, tabs, dialogs, data toggles, and connection-test actions. | Native buttons with visible text are not treated as icon-only by static checks; semantics still need review when adding custom controls. |
|
||||
| `uiux` | Same-category visual surfaces should use one style system per product area. | `frontend-rules-check.sh` warns when semantic `badge`, `chip`, `pill`, `tag`, or `status` selectors appear outside approved shared React and Earth CSS owner files. `docs/HARNESS.md` also makes screenshots/images primary evidence for visual fixes and forbids guessing when an image path is unavailable. | Final visual cohesion across screenshots still needs human/Playwright review, especially for page-specific cards, panels, and toolbar controls that static selector checks cannot classify perfectly. |
|
||||
| `uiux` | Text should fit, avoid viewport-scaled font sizes, and keep letter spacing at zero. | `frontend-rules-check.sh` fails viewport/container-width font-size units and non-zero `letter-spacing` / `letterSpacing`. `frontend-smoke.mjs` checks rendered routes for global overflow across desktop/mobile. | Per-element text clipping without page-level overflow is not exhaustively detected and needs visual review for changed screens. |
|
||||
| `frontend` | Keep shared behavior in reusable components and existing project patterns. | `frontend-rules-check.sh` enforces shared `ConnectionTestInput`, route/link/search consistency, no debug output, native button safety, Tactile/Radix/lucide direction instead of AntD/Space, and whitelist-driven public Docs. `bun x tsc --noEmit` and `bun run build` verify TypeScript/build health. | Broad casts, inline styles, and overflow issues are warnings when context may be legitimate; review changed lines before accepting them. |
|
||||
| `frontend` | Responsive adaptations must preserve the primary action path. | `frontend-smoke.mjs` clicks every visible admin menu entry on desktop and mobile, opens protected routes unauthenticated and authenticated, verifies root/unknown route fallback, and exercises core auth flows. | Deep feature workflows beyond smoke data, such as destructive or long-running actions, require targeted tests before behavior changes. |
|
||||
| `earth` | Earth render work needs real rendering checks. | Full smoke opens `/earth` and verifies the `3D Earth` iframe entry point. Earth News settings routes are included through the admin manifest/menu smoke, mocked `/earth/news-*` API responses, source test, add/cancel source draft, and manual news group creation checks. The broader Earth-specific layer/depth rules remain in `rules.md` and Earth docs. | The harness still does not claim full 3D layer visual verification; layer-depth and picking changes need targeted browser/canvas QA. |
|
||||
|
||||
## Harness Files Added
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `AGENTS.md` | Single authoritative agent guide and coding-agent entry point. |
|
||||
| `docs/HARNESS.md` | Harness workflow, validation tiers, conflict policy, and manual reminders. |
|
||||
| `CODEMAP.md` | High-level codebase map and validation references. |
|
||||
| `scripts/harness/lib.sh` | Shared command lookup and run helpers. |
|
||||
| `scripts/harness/doctor.sh` | Environment and repository-shape check. |
|
||||
| `scripts/harness/security-check.sh` | High-confidence secret and tracked environment/key file check. |
|
||||
| `scripts/harness/backend-rules-check.sh` | Backend app debug-call guard for direct stdout/debugger usage. |
|
||||
| `scripts/harness/frontend-rules-check.sh` | Bun-only, route manifest, literal internal link, admin-search route target, debug-output, native-button safety, icon-button accessibility, Card nesting, AntD/Space avoidance, ConnectionTestInput, admin shell one-screen height-chain declarations, admin/docs shell viewport sizing, same-category style owner warnings, viewport-font, zero-letter-spacing, and UI rules static check. |
|
||||
| `scripts/harness/docs-consistency-check.sh` | Frontend/backend Docs metadata alignment, public Docs metadata, full technical-doc bilingual pair, link-title, language-scoped technical link, supported credential collector contracts, manual console route coverage, documented route, admin-config-derived section deep-link consistency, and harness rules-coverage note check. |
|
||||
| `scripts/harness/frontend-smoke.mjs` | Playwright route, Docs detail/language/theme/search interaction, public auth form interaction, desktop/mobile/zoom rendering, admin shell one-screen/overflow checks, safe admin navigation/search/tab/dialog/Earth News interactions, and authenticated admin route/section smoke for the built frontend preview. |
|
||||
| `scripts/harness/quick-check.sh` | Fast deterministic local validation. |
|
||||
| `scripts/harness/validate.sh` | Full local validation wrapper with optional delivery smoke. |
|
||||
@@ -22,6 +22,7 @@
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [控制台 i18n 接入计划](/home/ray/dev/linkong/planet/docs/plans/admin-console-i18n-plan.md)
|
||||
- [Earth Mobile Drawer UI Plan](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [Earth Compute Center BGP Style Plan](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [Earth Renderer Architecture Separation Plan](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
@@ -33,6 +34,7 @@
|
||||
- [Earth News Cruise Summary Plan](/home/ray/dev/linkong/planet/docs/plans/earth-news-cruise-summary-plan.md)
|
||||
- [Earth 动作捕捉手势控制计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-capture-gesture-control-plan.md)
|
||||
- [Earth 动捕交互语义 V2 计划](/home/ray/dev/linkong/planet/docs/plans/earth-motion-gesture-interaction-v2-plan.md)
|
||||
- [Motion Agent v2 控制协议与 3D 标定路线](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md)
|
||||
- [Earth Presentation 解耦架构计划](/home/ray/dev/linkong/planet/docs/plans/earth-presentation-decoupled-architecture-plan.md)
|
||||
- [Earth Vessel Rendering Performance Plan](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-rendering-performance-plan.md)
|
||||
- [AIS 多源采集、冲突记录与聚合接口计划](/home/ray/dev/linkong/planet/docs/plans/earth-vessel-ais-aggregation-plan.md)
|
||||
|
||||
62
docs/plans/admin-console-i18n-plan.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# 控制台 i18n 接入计划
|
||||
|
||||
**状态**:基础设施已落地,大型业务页迁移继续进行
|
||||
**创建日期**:2026-06-29
|
||||
**核心目标**:把 Docs 已有的中英文文档能力提升为前端统一 i18n 体系,让未登录认证页、Docs UI、控制台外壳、导航、搜索和核心工作台文案共用同一个语言状态。
|
||||
|
||||
## 背景
|
||||
|
||||
Docs 站点已经有 `zh` / `en` 文档目录、Gatekeeper 权限和 `/api/v1/docs/{lang}/{slug}` 内容接口,但语言状态只保存在 `docs-lang`,不影响控制台。控制台页面、搜索索引、toast、dialog、表格和认证页仍以中文硬编码为主,导致用户切到英文文档后,控制台仍是中文。
|
||||
|
||||
本计划把前端语言偏好收敛到 `planet-locale`,默认 `zh-CN`,支持 `en-US`。Docs 继续使用后端现有 `zh` / `en` 文档接口,通过前端映射与全局 locale 对齐。
|
||||
|
||||
## 设计决策
|
||||
|
||||
- 使用 `i18next` 和 `react-i18next` 作为统一 i18n 层,避免长期维护自研插值、hook 和资源加载逻辑。
|
||||
- 前端统一语言枚举为 `zh-CN` / `en-US`;Docs 请求继续转换为 `zh` / `en`,新闻接口继续使用已有 `zh-CN` / `en-US` 口径。
|
||||
- 语言偏好首版只保存在浏览器 `localStorage`,不新增后端用户设置字段。
|
||||
- `docs-lang` 保留为兼容读取和写入项,让已访问过 Docs 的浏览器能平滑迁移。
|
||||
- 静态路由、导航、搜索目标和通用组件使用显式翻译 key;大型业务页在迁移期间通过 legacy UI 翻译桥补足常见硬编码文案。
|
||||
|
||||
## 分期
|
||||
|
||||
### P1:统一语言基础设施
|
||||
|
||||
- 在 `frontend/src/i18n/` 下维护 locale 类型、资源、初始化和 `useLocale()`。
|
||||
- 在 `frontend/src/main.tsx` 里初始化 i18n,并同步 `document.documentElement.lang`。
|
||||
- 在认证页和控制台侧边栏偏好面板提供语言切换入口。
|
||||
|
||||
### P2:高复用界面迁移
|
||||
|
||||
- 迁移 Docs UI、AdminLayout、route manifest、admin search、Auth、DataTable、Dialog、Toast 和 MarkdownRenderer。
|
||||
- 搜索索引按当前语言展示,同时保留中英文关键词以免降低可发现性。
|
||||
- 用户管理页作为独立业务页示范,迁移表头、按钮、toast、校验提示、角色和 Gatekeeper 标签。
|
||||
|
||||
当前已完成统一 `planet-locale`、Docs 兼容映射、认证页和控制台外壳语言入口、共享组件 key 化,以及 legacy UI 翻译桥。后续工作集中在把大型业务页从过渡桥迁移到显式 key。
|
||||
|
||||
### P3:大型业务页收敛
|
||||
|
||||
- 分批把 Dashboard、DataList、Logs 和 PlainResourcePages 的配置块改为显式翻译 key。
|
||||
- 过渡期保留 legacy UI 翻译桥,只处理 admin/auth 容器里的精确静态文本和属性。
|
||||
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥。
|
||||
|
||||
### P4:移除过渡桥
|
||||
|
||||
- 当 `rg -n "[\\p{Han}]" frontend/src/admin frontend/src/pages frontend/src/components` 只剩业务数据示例、中文文档标题或必须保留的中文品牌词时,删除 legacy UI 翻译桥。
|
||||
- 增加 key 完整性检查,确保 `zh-CN` 和 `en-US` 资源结构一致。
|
||||
|
||||
## 验证
|
||||
|
||||
- `cd frontend && bun run build`
|
||||
- `scripts/harness/frontend-rules-check.sh`
|
||||
- `scripts/harness/docs-consistency-check.sh`
|
||||
- `scripts/harness/quick-check.sh`
|
||||
- 前端 smoke 需要覆盖登录页、Docs、Admin 侧边栏语言切换、侧边栏和搜索结果在中英文下渲染。
|
||||
|
||||
## 相关文件
|
||||
|
||||
- `frontend/src/i18n/`:统一 locale、资源和过渡桥。
|
||||
- `frontend/src/pages/Docs/Docs.tsx`:Docs 语言状态改为读取全局 locale。
|
||||
- `frontend/src/admin/components/layout/AdminLayout.tsx`:控制台侧边栏语言切换、导航和搜索文案。
|
||||
- `frontend/src/admin/search/indexers.ts`:Admin 搜索目标本地化。
|
||||
- `docs/technical/{zh,en}/frontend-admin-frontend-context.md`:当前实现上下文。
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
**状态**:待实施
|
||||
**创建日期**:2026-05-12
|
||||
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.claude/commands/docs.md`,让以后写文档时自动按受众归档。
|
||||
**校正日期**:2026-06-26,控制台深链已从旧 tab 查询口径更新为当前 `?section=` 口径。
|
||||
**核心目标**:把 `docs/technical/{zh,en}/manual.md` 拆成"纯客户视角"的使用手册,把 `planet.sh`、日志、LAN、故障排查这类运维内容迁到独立 `ops-runbook.md`,并把分层规则写进 `documentation-coverage-rules.md` 和 `.codex/skills/docs/SKILL.md`,让以后写文档时自动按受众归档。
|
||||
|
||||
## 背景
|
||||
|
||||
@@ -31,12 +32,12 @@
|
||||
3. **登录与找回密码** — 登录页、忘记密码流程
|
||||
4. **账户设置** — 修改密码、修改邮箱(需重新验证)、查看权限组、登出
|
||||
5. **Console 总览** — 左侧菜单结构、各路由用途
|
||||
6. **配置数据采集器** — `/collection-management?tab=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?tab=providers`:默认 provider、模型、Base URL、API Key、本地代理;工具 tab(WebSearch、OCR)
|
||||
6. **配置数据采集器** — `/collection-management?section=collector_credentials`:选择 collector、连接测试、保存凭证;BarentsWatch / AISStream 两个典型例子
|
||||
7. **配置 AI 凭证** — `/ai?section=integrations`:默认 provider、模型、Base URL、API Key、本地代理;工具 section(WebSearch、OCR)位于 `/ai?section=tools`
|
||||
8. **系统设置** — `/settings` 其他子 tab(系统设置、电视直播源、SMTP 邮件)
|
||||
9. **用户管理(管理员)** — `/users`:创建、删除、改角色、Gatekeeper 权限组
|
||||
10. **数据探索** — `/datasources`、`/data`、`/bgp`、`/alerts/*`
|
||||
11. **AI 测试台** — `/ai?tab=playground`
|
||||
11. **AI 测试台** — `/ai?section=playground`
|
||||
12. **Earth 公开页面** — 现 manual.md 的 Earth 章节原样保留(图层、图例、搜索、位置候选、设置、视角、动捕、巡航、移动端)
|
||||
13. **Docs 文档站** — 当前 Docs 章节保留(权限组说明)
|
||||
|
||||
@@ -48,7 +49,7 @@
|
||||
|
||||
- 打开管理员给你的 URL
|
||||
- 注册账号 + 邮箱验证
|
||||
- 登录后第一次做什么(建议先到 `/collection-management?tab=collector_credentials` 配一个 collector,再到 `/ai` 配模型)
|
||||
- 登录后第一次做什么(建议先到 `/collection-management?section=collector_credentials` 配一个 collector,再到 `/ai?section=integrations` 配模型)
|
||||
- 看 Earth
|
||||
|
||||
部署/开发的 quickstart 内容并入 `ops-runbook.md` 的"首次部署"小节,**不**再单独出 `ops-quickstart.md`,避免新增维护点。
|
||||
@@ -79,7 +80,7 @@
|
||||
> - 新增客户可见 UI 流 → 同时更新 `manual.md` zh+en 与 `docs-content.ts`
|
||||
> - 新增 ops 命令或脚本 → 只更新 `ops-runbook.md` zh+en
|
||||
|
||||
## .claude/commands/docs.md 增量
|
||||
## `.codex/skills/docs/SKILL.md` 增量
|
||||
|
||||
在 "Step 2 — Decide Scope" 后插一段:
|
||||
|
||||
@@ -99,7 +100,7 @@
|
||||
- `docs/technical/zh/quickstart.md` & `en/quickstart.md` — 重写
|
||||
- `docs/technical/zh/ops-runbook.md` & `en/ops-runbook.md` *(新)*
|
||||
- `docs/documentation-coverage-rules.md` — 加受众分层段
|
||||
- `.claude/commands/docs.md` — 加 Document Audience Routing 段
|
||||
- `.codex/skills/docs/SKILL.md` — 加 Document Audience Routing 段
|
||||
- `frontend/src/pages/Docs/docs-content.ts` — 注册 `ops-runbook` 到 `DOCS_METADATA`(`docs_admin` 组)
|
||||
|
||||
## 依赖
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Earth Motion Capture Gesture Control Plan
|
||||
|
||||
> Update: Motion Agent process/device control, UE/Web shared command protocol, dual-camera redundant fusion, and the next 3D calibration route are now tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md). This document remains useful for the original provider split and gesture-control intent.
|
||||
|
||||
## Goal
|
||||
|
||||
为 Planet Earth 大屏和未来 3D 展示增加一套解耦的动作捕捉手势控制能力。实时输入分成两条路线:网页端可直接通过浏览器 `getUserMedia` 在本机识别;高级设备可继续使用本机 Motion Capture Edge Agent。两条路线都只输出轻量语义事件,客户端负责把“手势事件”映射到“具体交互函数”。
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
**状态**:已实现主体交互,并按实测调整。当前浏览器识别保留右手导航、头部切目标、左手上下切动捕图层、双手张开/收拢缩放;双手上举确认暂时关闭。Motion 目标展示已改为 `CruiseSequencer` + `PresentationController` 的 persistent 展示。
|
||||
|
||||
> Update: Agent-side bidirectional commands, UE/Web shared device control, dual-camera redundant fusion, and future calibrated 3D mode are tracked in [Motion Agent v2 Control Protocol And 3D Calibration Roadmap](/home/ray/dev/linkong/planet/docs/plans/motion-agent-v2-control-protocol-plan.md).
|
||||
|
||||
## Summary
|
||||
|
||||
把动捕从“几个单点手势触发函数”升级为一套更像大屏遥控器的交互层:右手负责地球导航,头部负责候选切换,左手上下切换动捕候选图层,双手负责缩放,调试面板支持“只显示骨骼”和暂停匹配。进入动捕模式后,Earth 自动软选中屏幕中心附近的正面可交互目标;确认动作预留为把目标升级为锁定,并用巡航/引导线式详情打开,不再模拟鼠标点击。
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
## 当前状态
|
||||
|
||||
最新实现继续保留分桶 `Points` 与全局数据范围,并已接通 `vessels` 全局实时订阅:后端读取确认状态后按 MMSI 拆包推送,前端原位修改缓冲;快照用于首次加载、重连和删除校准。旧快照不能覆盖较新的更新或删除,旋转与缩放仍不触发视口请求。下方 Sprite 问题分析和阶段方案保留为历史背景,当前约束以[地球前端上下文](../technical/zh/earth-frontend-context.md)为准。
|
||||
|
||||
该计划的前端核心部分已经在 `0.44.1` 落地,但最终实现不是原文设想的 `InstancedBufferGeometry` quad,而是更稳的分桶 `THREE.Points` 方案:
|
||||
|
||||
- 普通船只按 moving / anchored 和 `VESSEL_COURSE_BINS` 航向分桶,使用 `PointsMaterial` 批量绘制。
|
||||
|
||||
175
docs/plans/motion-agent-v2-control-protocol-plan.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Motion Agent v2 Control Protocol And 3D Calibration Roadmap
|
||||
|
||||
## Summary
|
||||
|
||||
Motion Agent v2 turns the local motion service into a shared WebSocket control plane for the Web Earth page and UE clients. The agent process is still started externally through `planet.sh`, a desktop service, or UE process management. Once the process is running, clients can open cameras, close cameras, switch input mode, arm or pause recognition, and inspect status through the same bidirectional WebSocket protocol.
|
||||
|
||||
This phase implements robust single-camera and dual-camera redundant fusion. True calibrated 3D skeleton reconstruction is deliberately reserved for the v3 calibration phase.
|
||||
|
||||
## v2 Protocol
|
||||
|
||||
The default endpoint remains:
|
||||
|
||||
```text
|
||||
ws://127.0.0.1:8765/ws/gestures
|
||||
```
|
||||
|
||||
The agent emits:
|
||||
|
||||
- `gesture`
|
||||
- `skeleton`
|
||||
- `status`
|
||||
- `heartbeat`
|
||||
- `command_result`
|
||||
|
||||
Clients send:
|
||||
|
||||
- `open_devices`
|
||||
- `close_devices`
|
||||
- `rescan_devices`
|
||||
- `set_armed`
|
||||
- `set_paused`
|
||||
- `set_input_mode`
|
||||
- `set_camera_config`
|
||||
- `set_fusion_config`
|
||||
- `set_debug_options`
|
||||
- `set_enabled_gestures`
|
||||
- `get_status`
|
||||
- `ping`
|
||||
|
||||
All v2 messages include additive compatibility fields such as `protocol_version`, `request_id`, `camera_id`, `input_mode`, and optional `fusion`. Older push-only clients can continue to consume `gesture`, `skeleton`, `status`, and `heartbeat`.
|
||||
|
||||
Example command:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command",
|
||||
"command": "open_devices",
|
||||
"request_id": "req-001",
|
||||
"payload": {
|
||||
"input_mode": "dual_redundant",
|
||||
"camera_indexes": [0, 1]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example gesture whitelist command:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command",
|
||||
"command": "set_enabled_gestures",
|
||||
"request_id": "earth-motion-enabled-gestures",
|
||||
"payload": {
|
||||
"gestures": ["rotate_left", "rotate_right", "zoom_in", "zoom_out", "confirm"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example result:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "command_result",
|
||||
"protocol_version": "motion.v2",
|
||||
"request_id": "req-001",
|
||||
"command": "open_devices",
|
||||
"ok": true,
|
||||
"status": {
|
||||
"armed": false,
|
||||
"paused": false,
|
||||
"input_mode": "dual_redundant",
|
||||
"devices_open": true,
|
||||
"active_camera_ids": ["usb:0", "usb:1"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Device And Control State
|
||||
|
||||
Process startup is not a WebSocket feature: the server must exist before a WebSocket client can connect. UE should start the agent as an external process or depend on a system service, then connect to the WebSocket endpoint.
|
||||
|
||||
Device wake and control wake are WebSocket features:
|
||||
|
||||
- `open_devices` opens USB cameras or URL cameras.
|
||||
- `close_devices` releases them.
|
||||
- `set_armed` enables gesture execution.
|
||||
- `set_paused` pauses recognition without closing the connection.
|
||||
|
||||
When `armed=false`, the agent may still emit skeleton and status events, but it does not emit actionable gesture events.
|
||||
|
||||
The Earth settings dialog owns a user-facing gesture whitelist. Unchecked gestures are ignored in the Earth client and are also sent to the Motion Agent through `set_enabled_gestures`, so the server does not broadcast disabled actions to UE/Web consumers. The default whitelist enables the full v2 gesture set; disabling gestures is a local display/control preference and does not change the installed recognition model.
|
||||
|
||||
## Input Modes
|
||||
|
||||
- `single`: one camera.
|
||||
- `dual_redundant`: two or more cameras observe the same gesture. Matching observations in a short window are fused into a higher-confidence event.
|
||||
- `single_fallback`: the primary camera is preferred and a secondary input is used as fallback.
|
||||
- `calibrated_3d`: reserved for v3 and should not be enabled unless a calibration profile exists.
|
||||
|
||||
The v2 dual-camera mode is redundant fusion, not 3D reconstruction. It is meant to improve reliability under occlusion and camera noise without requiring calibration.
|
||||
|
||||
## v3 3D Calibration Roadmap
|
||||
|
||||
The next phase is `Motion Agent v3 3D Calibration`. It upgrades from redundant fusion to calibrated multi-camera skeleton fusion.
|
||||
|
||||
Planned capabilities:
|
||||
|
||||
- Camera intrinsics: focal length, distortion, resolution.
|
||||
- Camera extrinsics: relative position, rotation, and baseline distance.
|
||||
- Calibration workflow: checkerboard, AprilTag, or ArUco board.
|
||||
- Local calibration profile JSON with query, load, reset, and validation commands.
|
||||
- `skeleton_3d` event with world-space joints, confidence, and source cameras.
|
||||
- UE coordinate mapping from Motion Agent coordinates to UE world or widget coordinates.
|
||||
|
||||
Reserved v3 input configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"input_mode": "calibrated_3d",
|
||||
"calibration_profile": "desk-dual-camera-v1"
|
||||
}
|
||||
```
|
||||
|
||||
Reserved v3 event:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "skeleton_3d",
|
||||
"protocol_version": "motion.v3",
|
||||
"profile": "desk-dual-camera-v1",
|
||||
"joints": [
|
||||
{
|
||||
"name": "right_wrist",
|
||||
"x": 0.42,
|
||||
"y": 1.13,
|
||||
"z": 0.76,
|
||||
"confidence": 0.91
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Command/result roundtrip for device open, close, rescan, armed, paused, and status.
|
||||
- Gesture protocol accepts the full Earth v2 gesture set.
|
||||
- Earth settings can disable individual gestures; disabled gestures are ignored locally and filtered server-side through `set_enabled_gestures`.
|
||||
- Motion Agent `status` reports the current enabled gesture list for Web/UE diagnostics.
|
||||
- Single and dual redundant fusion emit compatible gesture payloads.
|
||||
- Conflicting dual-camera observations below the confidence delta are ignored.
|
||||
- Web Earth `motion-agent-provider` can send commands over the same socket it uses for events.
|
||||
- UE mock clients can operate the service without browser-only assumptions.
|
||||
- Dry-run mode can test commands, status, skeleton, and fusion behavior without camera dependencies.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Production recognition uses a real MediaPipe pose pipeline and heuristic gesture recognizer. It still needs environment-specific threshold tuning, camera framing validation, and long-running reliability checks before it can be treated as calibration-free.
|
||||
- Dual-camera v2 does not triangulate 3D joint positions.
|
||||
- `calibrated_3d` is documented as a reserved mode and must not be treated as implemented until v3 lands.
|
||||
|
||||
## Implementation Status
|
||||
|
||||
- Implemented: bidirectional command/result protocol, device lifecycle controls, dry-run mode, subprocess recognition worker, MediaPipe pose recognition, gesture whitelist, single-camera mode, dual-redundant/fallback scaffolding, Web client integration, and default `planet.sh` lifecycle integration.
|
||||
- Remaining v2 hardening: tune recognition thresholds across camera placements, exercise UE command/control integration, and run longer soak tests for device reconnect and dual-camera conflicts.
|
||||
- Planned v3: calibrated multi-camera 3D skeleton fusion and Motion Agent-to-UE coordinate calibration.
|
||||
@@ -51,6 +51,7 @@ That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-com
|
||||
Supported request adapters:
|
||||
|
||||
- `openai-completions`
|
||||
- `openai-responses`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
@@ -95,12 +96,37 @@ The AI settings page uses:
|
||||
- `POST /api/v1/settings/integrations/ai-provider/connect`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/secrets`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/presets`
|
||||
- `POST /api/v1/settings/integrations/ai-provider/presets/{provider}/refresh`
|
||||
- `GET /api/v1/settings/ai-prompts`
|
||||
- `PUT /api/v1/settings/ai-prompts/{task_key}`
|
||||
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
|
||||
|
||||
These endpoints require an authenticated user. The `secrets` endpoint is only used when the settings page reveals a key or token; hiding the field restores the masked preview.
|
||||
|
||||
Model refresh and lightweight checks share `backend/app/services/llm_model_catalog.py` and query provider endpoints directly, without models.dev. Refresh uses the draft base URL, protocol and credentials, or saved configuration when no draft is supplied. Regional hosts and custom gateway paths are preserved. Missing required credentials return 400; upstream failures return safe errors and retain the previous catalog.
|
||||
|
||||
Successful catalogs and `refreshed_at` are stored under `llm_provider_preset:<provider>` in `system_settings`. Dynamic categories use the provider preset as their defaults and are validated before committing. Listing prefers saved catalogs and otherwise shows bundled suggestions. Refresh never modifies the active model, protocol, URL or credentials in `external_integrations`. The frontend retains the draft and previously loaded models when listing fails, with an explicit error message.
|
||||
|
||||
### Model Catalog Endpoints
|
||||
|
||||
These endpoints were checked against official documentation on 2026-09-13. The table lists discovery URLs; the form still takes the generation base URL. Credentials must belong to the configured service region.
|
||||
|
||||
| Provider | Model list GET endpoint | Authentication and parsing |
|
||||
| --- | --- | --- |
|
||||
| [MiniMax](https://platform.minimax.io/docs/api-reference/models/anthropic/list-models) | `https://api.minimaxi.com/anthropic/v1/models`; international: `api.minimax.io` | `x-api-key`; `data[].id`; `has_more/last_id` pagination |
|
||||
| [OpenAI](https://developers.openai.com/api/reference/resources/models/methods/list) | `https://api.openai.com/v1/models` | Bearer; `data[].id` |
|
||||
| [Anthropic](https://platform.claude.com/docs/en/api/models/list) | `https://api.anthropic.com/v1/models` | `x-api-key`, `anthropic-version`; paginate with `after_id` |
|
||||
| [DeepSeek](https://api-docs.deepseek.com/api/list-models) | `https://api.deepseek.com/v1/models` | Bearer; `data[].id` |
|
||||
| [Alibaba Model Studio](https://help.aliyun.com/zh/model-studio/list-models) | `/api/v1/models` on the configured regional host | Bearer; `output.models[].model`; `page_no/page_size/output.total`; filter `capabilities=TG` |
|
||||
| [Moonshot / Kimi](https://platform.kimi.ai/docs/api/list-models) | `https://api.moonshot.ai/v1/models`; China: `api.moonshot.cn` | Bearer; `data[].id` |
|
||||
| [OpenRouter](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties) | `https://openrouter.ai/api/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
|
||||
| [OpenCode Go](https://opencode.ai/docs/go/#models) | `https://opencode.ai/zen/go/v1/models` | Public catalog; Bearer when supplied; `data[].id` |
|
||||
| [Ollama](https://docs.ollama.com/api/tags) | `/api/tags` on the configured server | Local servers need no key; `models[].model/name`; empty means no installed models |
|
||||
|
||||
New Model Studio endpoints for Beijing, Tokyo, Frankfurt and Virginia require the actual workspace host, such as `<WorkspaceId>.cn-beijing.maas.aliyuncs.com`. Singapore uses `dashscope-intl.aliyuncs.com`; Hong Kong uses `cn-hongkong.dashscope.aliyuncs.com`. Discovery changes only the path, never guesses a workspace or switches credential regions. If a legacy Beijing host no longer accepts the account, update the base URL from the provider console.
|
||||
|
||||
Pagination and retries have a 30-second overall deadline. Network failures and 502/503/504 allow up to two attempts; authentication failures are not retried. Models are sorted by supplied creation/release dates, preserving upstream order on ties, without arbitrary list truncation. Public catalog membership does not establish account-specific generation permission.
|
||||
|
||||
Admin keeps the AI page aligned with the legacy information architecture:
|
||||
|
||||
- `Model Providers`
|
||||
@@ -283,22 +309,11 @@ Tool keys follow the same rule. WebSearch and OCR must match the current tool an
|
||||
|
||||
### Lightweight Connectivity Testing
|
||||
|
||||
The Admin plug button performs a lightweight connectivity check and does not save configuration. Common API-platform practice is two-tiered:
|
||||
|
||||
- Check a provider catalog or low-cost endpoint to validate base URL, authentication, and model reachability.
|
||||
- Send full model requests only when the user explicitly runs Playground or a business task.
|
||||
|
||||
Connectivity results should be explicit:
|
||||
|
||||
- `ok`: authentication, route, and model catalog are usable.
|
||||
- `warning`: service is reachable, but the current model is missing from the catalog or capability metadata is incomplete.
|
||||
- `error`: authentication, network, protocol, or model lookup failed.
|
||||
|
||||
Toast titles must match the result; failures must not be titled as a successful connection.
|
||||
The Admin plug button checks proxy configuration and queries the current provider catalog. It returns `success=true` only when discovery succeeds and contains the selected model. A 404, authentication error, invalid response or missing model never becomes success based on bundled suggestions. The message distinguishes discovery from generation; Playground or a business task verifies actual generation. Testing never saves the draft or switches the default provider.
|
||||
|
||||
### OpenCode Go Routing Model
|
||||
|
||||
Subscription channels such as OpenCode Go should not be handled by hard-coded frontend model sets. Prefer provider catalog or backend capability discovery that records per-model capabilities such as `chat_completions`, `anthropic_messages`, `models_endpoint`, and whether a subscription key is required. The frontend should display capabilities; the backend should map provider, base URL, model, and adapter into the real request.
|
||||
OpenCode Go protocol mappings are owned by the backend catalog: MiniMax M3/M2.7/M2.5 and the verified Qwen3.6/3.7/3.8 models use Anthropic Messages; GPT-5.6 Luna, Grok 4.6 and Muse Spark Contributor use Responses; other supported models use Chat Completions. The official `/models` response currently supplies IDs only, so new models still require checking the documented endpoint table. Selecting a model updates its protocol in the draft, and backend mappings supersede known stale mappings. Responses uses `input`, `max_output_tokens` and `store=false`, and parses text and reasoning summaries. OpenCode calls carry an application User-Agent and a stable Playground conversation identifier. Generic `thinking=enabled` is translated to MiniMax M3’s `adaptive` format.
|
||||
|
||||
#### Settings Page Behavior
|
||||
|
||||
|
||||
@@ -111,6 +111,12 @@ Snapshot lists should not show every snapshot of the same collector as separate
|
||||
|
||||
Credential guides are maintained by `backend/app/services/credential_guides.py`. The console uses read / generate / reset actions to load or create Markdown instructions. The frontend should render the guide Markdown for operators, not expose generation prompts or raw metadata.
|
||||
|
||||
### News Live Stream Catalog
|
||||
|
||||
The IPTV-org adapter in `news_live_streams` retains its news-category filters. Its default `max_sources` is `0`, meaning all matching channels; an explicit positive value still limits collection. Existing configurations retaining the old `120` limit must change it to `0` and collect again to populate the complete catalog.
|
||||
|
||||
`GET /api/v1/tv/streams` uses database pagination in `tv_catalog.py`: `offset` defaults to `0`, `limit` defaults to `50` with a maximum of `100`, and whitespace-separated `q` terms match channel names, providers, regions, and languages. Built-in and configured sources come first. Collected sources are deduplicated by channel ID, exclude configured overrides, and are filtered, counted, sorted, and paginated in SQL. The response's `total` counts matches, while `source_count` counts the complete available catalog; use `has_more` and `next_offset` for subsequent pages. `selected_id` can also return the selected channel outside the current page without consuming its quota. `tv_streams.py` remains the owner of default and fallback sources.
|
||||
|
||||
## IV. Data Format (stored in CollectedData table)
|
||||
|
||||
```python
|
||||
@@ -352,7 +358,11 @@ GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
||||
|
||||
`/api/v1/vessels/snapshot` requires `bbox` and `zoom`, and caps `limit` at `5000`. The Earth frontend uses a global bbox for current state and does not refetch on camera viewport changes. The endpoint reads `vessel_current_state` and reports `diagnostics.source = "vessel_current_state"`. The old `/api/v1/visualization/geo/vessels` route has been removed.
|
||||
|
||||
High-frequency AIS updates must not become per-delta full-layer rebuilds. If Earth uses the `/ws` `vessels` channel, it should send low-frequency reload/dirty hints and let the frontend merge snapshot refreshes. Tracks and conflicts still read historical facts through the single-vessel APIs.
|
||||
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
|
||||
|
||||
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
|
||||
|
||||
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
|
||||
|
||||
### Layer APIs And Global Stats
|
||||
|
||||
|
||||
@@ -70,7 +70,10 @@ The unified event model is `earth.layer.changed`:
|
||||
| --- | --- |
|
||||
| `clear_then_reload` | Clear local frontend layer objects first, then force a refetch. Prefer this for deletes. |
|
||||
| `reload` | Keep old objects until fresh data returns. Use it for location, metadata, or non-destructive updates. |
|
||||
| `delta` | Used only for `earth_interactables`; upsert or remove objects by id. |
|
||||
| `delta` | `earth_interactables` upserts/removes by id; ordinary vessel writes update confirmed state by MMSI through the dedicated `vessels` channel without clearing the layer. |
|
||||
|
||||
Vessel deletion still emits a `reload` reconciliation hint; individual `vessel_current_state` deletions also enter the vessel remove channel. Source notifications identify changed MMSIs, while transmitted values come from current state. Global subscriptions use `scope: "global"`; message-size limits split frames instead of discarding remaining vessels.
|
||||
|
||||
|
||||
APIs must return HTTP 200 with an empty collection for real zero-data states; 5xx is reserved for real endpoint failures. After a delete event, if refetch fails, the frontend should keep the cleared state and show a lightweight error instead of restoring stale objects.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ The console now separates the "data source catalog" from "collector configuratio
|
||||
- Lists all data sources, including built-in and custom sources.
|
||||
- Clicking a name only opens an information drawer.
|
||||
- Focuses on status, manual collection, and running collection tasks.
|
||||
- `/collection-management?tab=collector_credentials`
|
||||
- `/collection-management?section=collector_credentials`
|
||||
- Displays as "Collectors".
|
||||
- Owns endpoint, headers, base parameters, and credentials.
|
||||
- Every collector exposes a connection button for health checks.
|
||||
@@ -321,7 +321,7 @@ The new vessel list entry point is no longer the legacy `/api/v1/visualization/g
|
||||
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
||||
```
|
||||
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. If the `/ws` `vessels` channel is connected, it should act as a reload/dirty hint for merged refreshes, not as a per-AIS-delta full-layer rebuild path.
|
||||
That endpoint reads `vessel_current_state`, returning the latest point per MMSI inside the freshness window. Raw `ais_raw_observations` remain available for tracks, audit, and situational analysis, but the display endpoint no longer scans and aggregates history on the fly. Earth sends a global bbox rather than the current camera viewport. Realtime updates use the global `/ws` `vessels` subscription to deliver canonical state by MMSI and update existing markers in place. Larger updates are split across frames without dropping vessels. Initial load, reconnection, and deletion reconciliation still use snapshots; see [Collector Architecture](backend-collectors.md).
|
||||
|
||||
## Custom REST / WebSocket Mapping Runtime
|
||||
|
||||
|
||||
@@ -75,7 +75,13 @@ This is currently the most critical UI control entry point for the Earth fronten
|
||||
|
||||
Earth settings are now grouped by `data-settings-tab` and `data-settings-tab-panel`. Desktop and mobile share the same category semantics: Runtime, Display, Panels, Motion, Shortcuts, and System. When adding a setting, first choose its category, then add the DOM, persistence field, and restore logic; do not keep growing one long undifferentiated panel.
|
||||
|
||||
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories to `/api/v1/news/earth-feed?categories=...&locale=zh-CN`, so Web and UE clients share the same backend category filtering path.
|
||||
Earth runs inside an independent iframe / static application, so it cannot directly reuse React Admin's `react-i18next` context. `public/earth` uses its own [i18n.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/i18n.js) runtime to read and write the global `planet-locale`, while keeping the legacy `docs-lang` in sync. This keeps Docs, the console, and Earth on the same language preference. The language switch belongs in the Settings `System` tab, and desktop/mobile both use the same `data-earth-locale` buttons; do not put language selection into layer, display, or runtime-mode settings.
|
||||
|
||||
The news category selector in Display reuses the same chip-selector pattern as Cruise Modules. It only filters news categories for the current browser on the Earth frontend. It does not toggle layers, basemap, boundaries, TV, data points, BGP, vessels, satellites, or compute centers; those remain owned by the layer panel, media panel, and admin configuration. `controls.js` persists only `shared.newsCategoryFilters` and broadcasts `earth:news-category-filters-change`; `news.js` sends the selected categories and current locale to `/api/v1/news/earth-feed?categories=...&locale=...`, so Web and UE clients share the same backend category filtering path.
|
||||
|
||||
In English mode, Earth news must render only English title/summary text. Chinese source items without `en-US` localization are filtered from the visible cards/ticker/cruise until the backend enrichment finishes, and source/feed labels fall back to English-safe names instead of rendering Chinese labels.
|
||||
|
||||
The news panel, ticker, and news cruise must consume `items` / `cruise_items` from the same `/api/v1/news/earth-feed` response instead of keeping separate regional caches. `news.js` builds a refresh request key from region, category, source, and limit; only concurrent requests with the same key reuse the promise, and stale responses from an older region are dropped by token. Source filtering is also region-scoped: when the user moves from Asia Pacific to Europe or another region, source IDs saved for the old region must not be appended to the next fetch. After the new payload arrives, the saved source list is intersected with the available `sources`; if the intersection is empty, the current region falls back to all available sources. This keeps the ticker, panel, and cruise cards aligned after region switches.
|
||||
|
||||
Shortcut configuration is a device-local preference owned by `controls.js`: read, capture, enable/disable, and reset all stay in the Earth frontend. It should not be written to backend user settings and should not affect other browsers. New shortcuts must provide a default key, display label, disabled/enabled state, and reset path instead of being hard-coded only in a keydown handler.
|
||||
|
||||
@@ -102,9 +108,9 @@ Responsibilities:
|
||||
- Map `rotate_left`, `rotate_right`, `rotate_up`, `rotate_down`, `zoom_in`, `zoom_out`, `focus_prev`, `focus_next`, `layer_prev`, `layer_next`, and `confirm` to the action entry points exposed by `main.js`.
|
||||
- Parse `skeleton` debug events and dispatch `earth:motion-debug-frame`.
|
||||
|
||||
Gesture recognition may run locally in the browser or inside the local Agent, but neither path sends realtime camera frames to the SaaS cloud. `main.js` exposes rotation, zoom, target focus, layer switching, and confirm entry points, plus a `window.__planetEarth.motion` debug entry. The adapter starts only when `?motion=1` is present, browser local storage contains `planet-earth-motion-control-enabled=true`, or Earth settings enable Motion Debug Mode.
|
||||
Gesture recognition may run locally in the browser or inside the local Agent, but neither path sends realtime camera frames to the SaaS cloud. `main.js` exposes rotation, zoom, target focus, layer switching, and confirm entry points, plus a `window.__planetEarth.motion` debug entry. The adapter starts only when `?motion=1` is present, browser local storage contains `planet-earth-motion-control-enabled=true`, or Earth settings enable Motion Debug Mode. `shared.motionEnabledGestures` stores the user-approved gesture whitelist; the browser filters locally, and Motion Agent mode also synchronizes it through `set_enabled_gestures`.
|
||||
|
||||
[motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) owns the debug panel. It listens for `earth:motion-debug-frame` and draws normalized skeleton joints and bones on a canvas. The Browser Camera provider also emits `earth:motion-debug-video-source` with the local `<video>` element so the panel can show a local preview behind the skeleton; `shared.motionDebugSkeletonOnly` switches the panel back to skeleton-only rendering. `Stop Matching Gestures` dispatches `earth:motion-recognition-pause`, which suppresses gesture execution while video and skeleton drawing continue. Unmatched skeletons are red; matched gestures turn green and display the gesture name. Settings are persisted under `shared.motionDebugEnabled`, `shared.motionProvider`, and `shared.motionDebugSkeletonOnly` in `planet.earth.settings.v2`, and both the switch and provider selector reserve `data-gatekeeper-permission="earth.motion_debug"`.
|
||||
[motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) owns the debug panel. It listens for `earth:motion-debug-frame` and draws normalized skeleton joints and bones on a canvas. The Browser Camera provider also emits `earth:motion-debug-video-source` with the local `<video>` element so the panel can show a local preview behind the skeleton; `shared.motionDebugSkeletonOnly` switches the panel back to skeleton-only rendering. `Stop Matching Gestures` dispatches `earth:motion-recognition-pause`, which suppresses gesture execution while video and skeleton drawing continue. Unmatched skeletons are red; matched gestures turn green and display the gesture name. Settings are persisted under `shared.motionDebugEnabled`, `shared.motionProvider`, `shared.motionDebugSkeletonOnly`, and `shared.motionEnabledGestures` in `planet.earth.settings.v2`; the switch, provider selector, and gesture whitelist reserve `data-gatekeeper-permission="earth.motion_debug"`.
|
||||
|
||||
The Browser Camera provider's gesture pipeline lives in `recognizeGesture()` inside [motion-browser-provider.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-browser-provider.js). Detectors are evaluated in this order, first match wins:
|
||||
|
||||
@@ -153,6 +159,8 @@ Each module is responsible for its own:
|
||||
|
||||
`tv.js` owns the live / aggregation-news tabs inside `media-panel`. Toolbar open and tab-switch actions write back through `earth:tv-visibility-change` and `earth:tv-tab-change`: panel visibility remains viewport-scoped at `views.<scope>.panelVisibility.media-panel`, while the active tab is stored at `shared.mediaPanelActiveTab`. Refreshing the page therefore restores the user's last live/news state. Temporary hides from `closeTransientMobileOverlays()` carry `persist:false` and do not overwrite the preference.
|
||||
|
||||
`tv-source-menu.js` reuses HUD and legend-list styling in a popover with search, a scrolling list, and a fixed count footer. `tv.js` requests 50 entries from `/api/v1/tv/streams` and caches channel details; search runs against the complete backend catalog, and pagination uses `next_offset`. Cancellation and a request generation counter prevent stale results from replacing a newer search. Catalog refresh uses `selected_id` to restore a channel outside the first page.
|
||||
|
||||
`brand.js` manages Earth HUD brand resources. Static assets provide the default brand; runtime overrides come from `/api/v1/earth/brand`, and uploaded images are served from `/earth-brand-assets/...`. The frontend must treat logo/title images and text fallback separately: if an image fails, show the text title; if text fields are empty, rely on backend defaults so the HUD brand area never renders blank. The console Earth Content page owns saving and resetting brand configuration; the Earth frontend only consumes it.
|
||||
|
||||
`about.js` manages the About card inside Earth settings. Frontend defaults remain as a fallback, while runtime content is loaded from `/api/v1/earth/about`. If the request fails or fields are missing, the renderer must fall back per field so the settings page never renders an empty card. Admin exposes an Earth Content `About` tab; saving uses `PUT /api/v1/earth/about`, and restoring defaults uses `DELETE /api/v1/earth/about`.
|
||||
@@ -173,6 +181,8 @@ The compute-center layer row has a notification badge for GeoJSON `unresolved` r
|
||||
|
||||
Location candidate state in the details card is cached in [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) by `entityType:entityId`. If the user closes the details card or unresolved queue and reopens the same compute center / BGP collector, previously collected candidates and status text are restored. Header-level `一键采用` prefers cached candidates, avoiding repeated online geocoding or LLM factcheck calls. After a location is saved, that entity's candidate list is cleared to a "refreshing layer" status so stale candidates do not keep misleading the user.
|
||||
|
||||
`runUnresolvedComputeCenterBatch()` owns a queue of entity contexts rather than panel DOM. `locationCollectStateCache` retains candidates, progress, and save results so rebuilt cards can hydrate the current state. `earth:compute-center-location-batch-change` updates the badge, `savedCandidate` excludes completed entries, and a ✓ entry remains after completion. Queue state lives only in the current Earth page: refresh, close, or route navigation interrupts unfinished work, while saved coordinates remain authoritative in the backend.
|
||||
|
||||
The `预览 / 保存` buttons on each candidate row use a single delegated `click` handler per candidate root (the `[data-collect-cache-key]` block in the details card, or `[data-unresolved-item]` in the unresolved queue), guarded by a `data-candidate-actions-bound` flag so it cannot be double-bound. Direct `pointerup` / `click` listeners on individual buttons and overlapping delegated handlers were removed. Candidate objects are no longer JSON-stringified into an HTML attribute and parsed back; buttons only carry `data-candidate-index`, and the handler resolves the candidate object from a module-level `Map` keyed by cache-key. This removes the entire class of failures caused by HTML entity escaping of `&` / `<` / `"` in candidate fields. Clicking `预览` dispatches `earth:preview-location-candidate`; `main.js`'s `previewLocationCandidate()` calls `showComputeCenterLocationPreview()`, which attaches a hollow breathing-ring sprite pair at the candidate coordinates (visually mirroring the BGP event ring) and focuses the camera on the candidate. Previewing another candidate replaces the ring; saving clears it and `spawnSavedComputeCenterLocation()` immediately spawns the formal compute-center interactable. Note that `main.js` has no module-level `earth` variable — every location-save / preview handler must call `const earth = getEarth();` first, otherwise the event handler throws a `ReferenceError` that the surrounding `.catch` swallows, producing the failure mode where the button "does nothing".
|
||||
|
||||
The `earth:compute-center-location-saved` reconciliation pipeline is deliberately silent on background-refresh failures. `spawnComputeCenterAfterLocationSave()` already presents the success toast and locked state; `refreshComputeCentersAfterLocationSave()` only reloads backend data when the scene is ready and no longer emits its own `已保存` toast. `handleComputeCenterLocationSaved()` runs refresh in the background after a successful spawn; only when spawn returns `null` (scene not ready) or throws does refresh take over the success toast. A refresh error is only `console.warn`'d — it must never surface as a `保存失败` message, because the save itself succeeded and the refresh is a follow-up sync.
|
||||
@@ -185,7 +195,11 @@ The backend snapshot endpoint still requires `bbox`, but the Earth runtime treat
|
||||
|
||||
Vessel markers are rendered through `createInteractableLayer()` as batched `THREE.Points`, with `cluster` and `avoidance` explicitly disabled. Dense waterways may overlap. Dragging or inertial rotation skips hover picking; normal hover uses screen-space nearest-point picking. Do not reconnect vessels to dynamic screen clustering or per-frame Points rebuilds, because those interaction costs are what make a 3000-marker layer feel heavy.
|
||||
|
||||
If the `/ws` `vessels` channel is used by Earth, it should be a low-frequency reload/dirty hint only. Do not create multiple viewport subscriptions, and do not turn every AIS delta into a full layer rebuild.
|
||||
AISStream and BarentsWatch share the `/ws` `vessels` delta channel. Earth subscribes with `scope: "global"` on its existing WebSocket, without changing subscriptions as the camera moves. The backend coalesces notifications by MMSI for one second, then reads confirmed `vessel_current_state` rows. Frames contain at most 1000 items; further frames carry the remaining vessels rather than truncating them. Raw source messages must not overwrite confirmed client positions. Deleting current-state rows also produces MMSI-based remove notifications.
|
||||
|
||||
The frontend coalesces short bursts per MMSI and uses `Interactable.updateItems()` to update position and color buffers in place. Heading-bucket changes touch only affected buckets, growing capacity when needed. Existing marker identities, selections, and materials survive. Initial entry, reconnects, deletion hints, and the minute reconciliation still use snapshots without first clearing the layer. A bounded snapshot must not treat truncated vessels as deleted, and older responses must not overwrite newer stream updates or removals received while the request was in flight. Snapshots include query-start `generated_at`; it is compared with stream frame time to prevent stale cached snapshots from rolling back new vessels or removals.
|
||||
|
||||
Ordinary vessel writes use the `delta` strategy on `earth_updates`; while the dedicated channel is connected, they no longer request whole-layer reloads. Deletion or disconnected reconciliation uses `reload` while reusing existing objects. Hiding the layer unsubscribes and clears queued changes. Tracks, conflicts, and audit data continue through the single-vessel historical APIs.
|
||||
|
||||
The legacy `/api/v1/visualization/geo/vessels` route has been removed. Frontend code should keep using `PATHS.vesselsApi` and can verify the current-state path through `diagnostics.source == "vessel_current_state"`.
|
||||
|
||||
@@ -215,6 +229,14 @@ The cruise sequencer handles generic logic: current target, queue order, camera
|
||||
|
||||
All material, layer, satellite, BGP, cable, terrain, celestial, and other style parameters are maintained here. Do not scatter magic numbers in module files.
|
||||
|
||||
## Render hot paths
|
||||
|
||||
`cable-batches.js` batches the complete cable and landing-point sets. `cables.js` still owns original business objects, picking, occlusion, and selection state. Do not make these interaction proxies draw individually again or return a render batch as the detail-card business object.
|
||||
|
||||
Satellite breathing runs in vertex shaders. `satellite-position-worker.js` moves full SGP4 position and initial trail calculations off the main thread; the main thread applies snapshots to interaction coordinates and render buffers. `satellite-propagation.js` supplies the same mathematics to the Worker, predicted orbits, and synchronous fallback. Reload, clear, and altitude changes must terminate prior work so stale snapshots cannot resurrect cleared layers.
|
||||
|
||||
The `i18n.js` MutationObserver translates only added subtrees, changed text, or changed attributes. Local clocks, status labels, and download progress must not synchronize all locale controls. Global locale synchronization belongs to initialization, locale changes, or subtrees containing new locale controls.
|
||||
|
||||
## Current Style Layers
|
||||
|
||||
CSS files in `frontend/public/earth/css/` each correspond to a specific component scope. Do not write global Earth styles into `base.css` unless they genuinely apply to everything.
|
||||
@@ -240,6 +262,8 @@ Earth settings are stored in `localStorage`. The key is typically a namespaced s
|
||||
|
||||
Settings that affect visual layers and surface interaction (terrain opacity, day/night mode, satellite display style, satellite idle breathing, real satellite altitude, track display, hover tooltip mode, etc.) are read during initialization and applied immediately.
|
||||
|
||||
Language preference is the exception: Earth language is not a private `planet.earth.settings.v2` field. It shares `planet-locale` with Docs/Admin. When language changes, `i18n.js` updates `document.documentElement.lang`, translates static and dynamically inserted DOM, synchronizes language button state, and passes the current locale to news requests. The news module refreshes after a language change so an English UI does not reuse a Chinese news payload.
|
||||
|
||||
The surface hover tooltip preference is persisted by `controls.js` as `shared.surfaceHoverInfoMode`, while `main.js` composes the actual tooltip in the globe-surface hover branch. `Country` shows country details only when a country polygon is hit and stays silent over ocean; `Position` shows latitude, longitude, and sampled terrain elevation and clears country-boundary hover; `Full` shows country + position on land and position over ocean.
|
||||
|
||||
The real satellite altitude preference is persisted by `controls.js`, while the rendering state lives in `satellites.js`. When enabled, the real radius from SGP4 is compressed logarithmically into the current Earth visual radius range. When disabled, satellite dots, trails, and predicted orbits all return to the legacy same-sphere display. Toggling this setting must refresh satellite positions and clear trail buffers so a trail never mixes both height models. `maxRealAltitudeOffset = 25` is a visual cap tuned for the current camera and `earthRadius = 100`: GEO / MEO remain clearly higher than LEO, but the highest orbits stay within about 25% beyond the globe radius so selection targets, red trails, and the globe do not feel disconnected.
|
||||
@@ -325,7 +349,7 @@ If future cable, satellite, or news cruise is added, do not copy a new set of `m
|
||||
|
||||
[controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) owns the Earth zoom state, and every zoom entry point must ultimately call `setZoomLevel()` to write the camera distance. Do not write `camera.position.z` from other modules, or the zoom percentage, drag sensitivity, and Interactable clustering thresholds will diverge again.
|
||||
|
||||
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps the projection-based behavior for high-frequency realtime layers such as vessels, and `none` disables clustering. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
|
||||
Interactable clustering is selected per layer through `cluster.strategy`. `stable-spherical` uses discrete zoom bands and local 3D bucket clustering, so BGP, compute centers, and Earth interactables do not regroup while the globe rotates inside the same band. `dynamic-screen` keeps projection-based clustering, and `none` disables it. Vessels explicitly disable clustering and avoidance so incremental updates do not rebuild cluster topology. Stable cluster dots stay rigidly aligned to their 3D centroid projection and do not participate in 2D avoidance. See [Earth Interactable Clustering](/home/ray/dev/linkong/planet/docs/technical/en/earth-interactable-clustering.md) for strategy configuration and tuning.
|
||||
|
||||
Wheel input has two paths. Traditional mouse wheels keep the 10% step and short animation, using `wheelZoomTarget` as the logical base for continuous wheel input. Trackpads and high-precision wheels use the pixel delta for continuous zoom and call `setZoomLevel()` directly instead of passing through the 10% stepped animation. The trackpad path also filters a short-window, old-direction residual delta after a real direction change so inertia tails do not pull a just-reversed zoom back in the previous direction.
|
||||
|
||||
@@ -350,7 +374,7 @@ For future Earth changes:
|
||||
|
||||
The Earth frontend and the console frontend are not the same UI system:
|
||||
|
||||
- Console frontend: React + Ant Design workbench
|
||||
- Console frontend: React + Tactile UI / Radix primitives / lucide workbench
|
||||
- Earth frontend: native HUD + Three.js display under `public/earth`
|
||||
|
||||
Therefore:
|
||||
|
||||
@@ -143,6 +143,8 @@ The land/ocean base is an Earth base-map asset and preloads at startup; the "Bor
|
||||
|
||||
## Submarine Cables and Landing Points
|
||||
|
||||
`cable-batches.js` batches cable lines and landing points separately. The Sprite material and size parameters below remain owned by picking and selection proxies in `cables.js`; their color, opacity, scale, and visibility update the style texture or instance attributes. Drawing uses `LineSegments` and instanced billboards with the existing textures, colors, render order, and globe occlusion rules.
|
||||
|
||||
| Name | Variable | Current Value | Location / Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Default cable color | `CABLE_COLORS.default` | `0xffff44` | Used when no data color available |
|
||||
|
||||
@@ -86,6 +86,55 @@ Importance levels are fixed: `low` 0–34, `medium` 35–59, `high` 60–79, and
|
||||
|
||||
`GET /api/v1/earth/news-sources` returns the default or saved configuration. `PUT /api/v1/earth/news-sources` saves it, increments `cache_version`, and clears the process region cache. `POST /api/v1/earth/news-sources/reset` restores defaults. `POST /api/v1/earth/news-sources/test` tests one RSS/Atom/Aggregated source without writing news items.
|
||||
|
||||
## Manual News Content
|
||||
|
||||
Manual news is a content-management path, not an RSS source configuration. The Admin entry is `Earth Content -> News Content`; the left rail groups items by RSS source and manual news group. After opening a manual group, administrators can add one item or upload a JSON array. Manual items are written to `earth_news_items` with `feed_type/source_type = manual`; the display label is “手动添加” / “Manual”. They do not participate in RSS connectivity tests and are not routed through RSS fetching.
|
||||
|
||||
Manual news uses a publish-first, enrich-later flow:
|
||||
|
||||
1. Saving immediately writes the item to `earth_news_items`.
|
||||
2. If no manual coordinates are provided, the selected region anchor is used and `verified=false`.
|
||||
3. If manual coordinates are provided, the item uses `location_source=manual_location` and `verified=true`; later AI enrichment does not overwrite that location.
|
||||
4. Create and reprocess actions enqueue the item for cleanup, translation, classification, importance, Breaking, and target-location inference.
|
||||
5. When enrichment finishes, the same row is updated and Earth receives a news reload / patch so the frontend replaces the item without a manual refresh.
|
||||
|
||||
The Admin API is under `/api/v1/earth/news-items`:
|
||||
|
||||
- `GET /earth/news-groups`: return RSS virtual source groups and manual news groups.
|
||||
- `POST /earth/news-groups`: create a manual news group.
|
||||
- `PUT /earth/news-groups/{group_id}`: rename a manual news group and synchronize metadata for items in that group.
|
||||
- `GET /earth/news-items`: paginated RSS and manual news list, with filters for source type, region, category, and status.
|
||||
- `POST /earth/news-items`: create one manual news item.
|
||||
- `POST /earth/news-items/import`: upload a JSON array; `group_id` selects the current manual news group.
|
||||
- `PUT /earth/news-items/{id}`: edit a manual news item; RSS items are read-only.
|
||||
- `DELETE /earth/news-items/{id}`: delete a manual news item and trigger an Earth news reload.
|
||||
- `POST /earth/news-items/{id}/reprocess`: requeue cleanup, translation, and geolocation.
|
||||
|
||||
The first JSON import format supports arrays only:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"title": "Required title",
|
||||
"summary": "Optional summary",
|
||||
"content": "Optional body",
|
||||
"url": "https://example.com/story",
|
||||
"source": "Manual",
|
||||
"region": "global",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"category": "business",
|
||||
"tags": ["manual", "analysis"],
|
||||
"location": {
|
||||
"label": "Beijing, China",
|
||||
"latitude": 39.9057,
|
||||
"longitude": 116.3913
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Imports deduplicate through a stable `manual:{hash}` ID derived from title, published time, URL, and source. Importing the same manual item again updates the existing row instead of creating a duplicate EarthFeed entry.
|
||||
|
||||
## Feed Query and Category Filtering
|
||||
|
||||
The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. The endpoint supports server-side filtering, so clients do not need to fetch the full list and apply the primary category filter locally.
|
||||
@@ -95,6 +144,8 @@ The Web Earth client and UE client both consume `GET /api/v1/news/earth-feed`. T
|
||||
- `categories`: comma-separated news category keys, for example `business,ecommerce`. Omit it when all categories are selected.
|
||||
- `locale`: display locale, currently `zh-CN` or `en-US`, defaulting to `zh-CN`. Chinese RSS items are stored as Chinese source content and enriched with `en-US`; English RSS items are enriched with `zh-CN`.
|
||||
|
||||
For `locale=en-US`, the API must not fall back to Chinese source title/summary. If a Chinese source item has not yet received an `en-US` localization, `display_title` and `display_summary` stay empty so the Earth client can show an English pending/empty state instead of mixing languages.
|
||||
|
||||
Examples:
|
||||
|
||||
```http
|
||||
@@ -107,8 +158,23 @@ Unknown category or locale values return `422` with the allowed values. The resp
|
||||
|
||||
The Web Earth category chips only store the current browser preference; changing them triggers a new API request. UE should pass its selected categories through the `categories` query parameter and does not need to perform the primary filtering itself.
|
||||
|
||||
When `sources` is omitted, the service layer prefers stories that already have displayable title and summary content for the requested `locale`, supplements candidates from enabled sources, and rotates sources so one source's newest pending items cannot occupy all 12 default slots. An explicit `sources` filter remains precise and does not supplement other sources. The database query layer owns region, category, source, and ordering constraints only; it does not own locale presentation policy.
|
||||
|
||||
Source testing only proves that a specific RSS/Atom/XML feed can be parsed. It does not mean those items have already been written to the news table or are visible in the current region/category view. Saving or resetting news sources increments the configuration version and clears cache; if an enabled feed has no recent stored items, the next `earth-feed` request supplements from RSS so newly enabled sources such as 36Kr and Ebrun are not masked by fresh Google News rows.
|
||||
|
||||
## Regional Balancing and Async Enrichment
|
||||
|
||||
`earth_news_items` is the current-state table for EarthFeed. When no explicit `sources` filter is present, the endpoint sorts by Breaking state, region, and publish time, then applies a regional round-robin so Asia Pacific or any other high-volume source cannot fill the entire global view and cruise queue. The default region order is Americas, Europe, Middle East / Africa, Asia Pacific, then Global; unknown regions participate after the known regions. Regional views still follow the active-region-plus-global rule and do not mix unrelated regions into the regional panel.
|
||||
|
||||
Both `items` and `cruise_items` are enqueued for target-location and localization enrichment, but the frontend must not wait for AI before rendering. If the requested display locale is not ready yet, Web Earth falls back to the original `title / summary` so a card does not show a "translation pending" placeholder when readable source content already exists. After translation, classification, Breaking, or target-location enrichment completes, the same `earth_news_items` row is updated and Earth receives a news reload / patch.
|
||||
|
||||
Target-location enrichment uses two Redis Streams queues:
|
||||
|
||||
- `earth_news:target_location:priority`: priority jobs for currently visible `items` and `cruise_items`, with a short dedupe TTL.
|
||||
- `earth_news:target_location:jobs`: normal background enrichment jobs, with a longer dedupe TTL.
|
||||
|
||||
The worker always drains the priority queue before the regular queue; stale pending messages are reclaimed after the idle threshold, each AI job has a hard timeout, and failures go through retry or dead-letter handling. This prevents a large historical regular backlog from starving the Europe, Americas, or other regional news currently visible to the user.
|
||||
|
||||
## Breaking News Insertion
|
||||
|
||||
The news system keeps three separate decisions:
|
||||
@@ -163,9 +229,11 @@ Reference links show that they only record a homepage, report page, or future co
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Admin["Admin: Earth Content / News Sources"] --> Source["Source config"]
|
||||
ManualAdmin["Admin: Earth Content / News Content"] --> ManualAPI["/api/v1/earth/news-items"]
|
||||
Source --> Feed["Feed children"]
|
||||
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
|
||||
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
|
||||
ManualAPI --> Store
|
||||
|
||||
Earth["Earth News Panel"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
|
||||
@@ -20,7 +20,7 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| 0.86 | Land/ocean base fill | `country-boundaries.js` | `landAltitudeOffset = 0.32`; ocean `#010609`, land `#080f1b` | Raycast disabled | Base map remains usable even when country borders are off; radius is separated from the base sphere to avoid far-zoom z-fighting. |
|
||||
| 0.96 | HD Earth texture | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | Surface picking target when visible | HD texture always overlays the land/ocean base fill; radius must stay above the land/ocean base and far enough from the base sphere. |
|
||||
| 1 | Atmospheric glow and clouds | `earth.js` | Atmosphere / cloud spheres | Not in normal object selection path | Cloud layer controlled by the "Cloud Layer" toggle. |
|
||||
| 1 | Submarine cables | `cables.js` | `CABLE_CONFIG.line.renderOrder` | Cable picking path | Preserves existing cable layer level. |
|
||||
| 1 | Submarine cables / landing points | `cables.js`, `cable-batches.js` | All cable segments share one `LineSegments`; all landing points use instanced billboards; render order `1` and altitude offset `0.2` are unchanged | Original `Line` / `Sprite` objects retain per-item picking and selection state but no longer draw individually; landing-point sphere occlusion remains, with `depthTest: false` on the batch | Style textures and instance attributes preserve color, pulse, size, visibility, and click behavior. |
|
||||
| 1.2 | Real terrain | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` plus terrain displacement | Raycast disabled | Terrain overlays HD texture; temporarily hidden when HD texture is off, restores to prior state when re-enabled. |
|
||||
| 2.05 | Grid lines | `earth.js` | `CONFIG.earthRadius + 0.14` | Raycast disabled | Low-opacity lines over HD texture. |
|
||||
| 2.2 | Country borders | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`; claim lines have no extra lift | Raycast disabled | Line geometry still has its own `renderOrder`, but it shares the exact same radius as the HD texture shell to avoid parallax while the globe rotates. |
|
||||
@@ -36,6 +36,14 @@ Note: the layer control panel order and the registration / startup load order ar
|
||||
| 12+ | Satellite locked ring, halo, predicted orbit | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` and offsets; predicted orbit follows the same real-altitude toggle and fixes the lock-time globe pose to draw a closed inertial orbit; returns to same-sphere mode when real altitude is disabled | Satellite overlay path | Used for selected/locked satellite emphasis. |
|
||||
| 98-100 | Sun / moon halo and sprite | `celestial.js` | Fixed renderOrder | Celestial picking disabled | Foreground celestial sprites. |
|
||||
|
||||
## Full-set rendering and updates
|
||||
|
||||
- Batching changes GPU submission, not the number of satellites, cable segments, or landing points. It does not filter data by the visible hemisphere. Every adjacent cable vertex pair remains present, and picking still returns the original business object.
|
||||
- Satellite foreground and backdrop remain two `Points` draws. Vertex shaders compute breathing from static per-point parameters and a shared frame-time uniform; existing hover/locked state still controls point masking.
|
||||
- `satellite-position-worker.js` uses the same Three.js / SGP4 versions as the main thread. `satellite-propagation.js` owns shared orbit, display-altitude, and fallback calculations. Full positions and initial trail samples use transferable arrays; only one calculation is in flight. Reload, clear, and altitude-mode changes terminate the previous Worker.
|
||||
- Worker startup failure or `SATELLITE_CONFIG.workerStartupTimeoutMs` expiry falls back to the shared synchronous calculation. Counts, APIs, the existing update interval, and trail length are unchanged.
|
||||
- Verify full draw ranges, per-item picking, locked overlays, trails, rear-side occlusion, visibility toggles, and resource release after clearing. Frame-time comparisons require identical data, view, and resolution.
|
||||
|
||||
## Toggle Behavior
|
||||
|
||||
| Toggle | Behavior |
|
||||
|
||||
@@ -98,6 +98,8 @@ Admin status labels should use [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
|
||||
`StatusText` is an indicator-light pill: the pill background and border stay on the component base color, while only the dot and text use the status color. `Badge` does not carry the indicator-light meaning, so it may use a light same-tone background and border for stronger hierarchy.
|
||||
|
||||
Status indicators must show the full state word. In lists, hierarchy groups, and detail headers, the title/description area should shrink or wrap while the status pill keeps content-sized width and does not get compressed by flex/grid layout; do not truncate state words such as `Configured` or `Available` just to save horizontal space.
|
||||
|
||||
| Tone | Color variable | Meaning | Examples |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | `--an-success` | available, successful, connected, enabled | log source `Available`, collection `Success` |
|
||||
@@ -155,7 +157,6 @@ Purpose:
|
||||
Current usage:
|
||||
|
||||
- Admin data sources, collected data, collection management, logs, alerts, and BGP pages
|
||||
- Old AntD legacy pages continue using shared scrolling behavior through compatibility wrappers
|
||||
|
||||
### 3. `TableScrollRegion`
|
||||
|
||||
@@ -217,7 +218,31 @@ Current constraints:
|
||||
- Prefer CSS variable overrides for colors instead of hard-coding theme colors in feature components
|
||||
- Best for a small set of mutually exclusive choices; do not use it as a long list, navigation menu, or select replacement
|
||||
|
||||
### 6. `MarkdownRenderer`
|
||||
### 6. Console i18n
|
||||
|
||||
Files:
|
||||
|
||||
- [i18n/index.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/index.ts)
|
||||
- [i18n/locale.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/locale.ts)
|
||||
- [i18n/resources.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/resources.ts)
|
||||
- [LegacyI18nBridge.tsx](/home/ray/dev/linkong/planet/frontend/src/i18n/LegacyI18nBridge.tsx)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Share one `zh-CN` / `en-US` language state across the console, auth pages, and Docs UI
|
||||
- Store the language preference in `planet-locale` while keeping compatibility with the old `docs-lang`
|
||||
- Keep Docs API requests mapped to the backend's existing `zh` / `en` document interface
|
||||
- Provide language switchers in the console sidebar preferences panel and auth panel
|
||||
|
||||
Current constraints:
|
||||
|
||||
- New console copy should be added to `resources.ts`, then consumed with `useTranslation()` or `useLocale()`
|
||||
- Routes, menus, search indexes, and shared components must use explicit translation keys
|
||||
- `LegacyI18nBridge` is transitional and only handles exact static text and attributes inside admin/auth containers
|
||||
- Business data, raw logs, API field names, provider ids, commands, and Markdown body content are not translated by the legacy bridge
|
||||
- Future large-page migrations should shrink the legacy dictionary rather than grow it
|
||||
|
||||
### 7. `MarkdownRenderer`
|
||||
|
||||
File:
|
||||
|
||||
@@ -275,17 +300,16 @@ File:
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- `/ai` now owns LLM Provider, AI Tool configuration, and the testbench instead of nesting them under `/settings`
|
||||
- The `模型供应商` tab manages default provider, model, base URL, provider key, local `aiprovider` proxy, and connection test; provider and model fields use editable comboboxes so users can manually enter new providers/models if the models.dev catalog stops updating
|
||||
- The `工具` tab first selects a tool from a dropdown menu, then renders that tool's configuration; it currently includes WebSearch and OCR
|
||||
- `/ai` now owns LLM Provider, AI Tool configuration, and Playground instead of nesting them under `/settings`
|
||||
- The `模型供应商` section manages default provider, model, base URL, provider key, local `aiprovider` proxy, and connection test; provider and model fields use editable comboboxes so users can manually enter new providers/models if the models.dev catalog stops updating
|
||||
- The `工具调用` section first selects a tool from a dropdown menu, then renders that tool's configuration; it currently includes WebSearch and OCR
|
||||
- WebSearch configuration includes provider, search key, base URL, timeout, result count, and advanced provider options
|
||||
- OCR configuration includes provider, Base URL, API key, model/engine, languages, timeout, file-size limit, and output format
|
||||
- The `测试台` tab embeds the former Playground real session, preset prompts, and AI Provider status debugging
|
||||
- The page reuses the Settings single-screen tabs, panel card, and internal scrolling style
|
||||
- The `Playground` section embeds the former Playground real session, preset prompts, and AI Provider status debugging
|
||||
- The page reuses the Settings single-screen section, panel card, and internal scrolling style
|
||||
- AI Provider and WebSearch connection tests use `ConnectionTestInput`, with the connector icon fixed at the end of the Base URL input; when WebSearch is disabled, every configuration field and the test entry point are greyed out except the switch
|
||||
|
||||
Legacy `/settings?tab=ai` should redirect to `/ai?tab=providers`.
|
||||
Legacy `/playground` should redirect to `/ai?tab=playground`.
|
||||
AI configuration no longer lives under `/settings`; `/playground` should redirect to `/ai?section=playground`.
|
||||
|
||||
### 3. Business Data Gateway
|
||||
|
||||
@@ -342,11 +366,11 @@ Current page boundary:
|
||||
- Endpoint, headers, and config are displayed here, not edited.
|
||||
- Credential-bearing collectors point users to `Collection Management -> Collectors`.
|
||||
|
||||
Keep this boundary: do not put custom datasource editing, built-in endpoint overrides, or credential forms back into `/datasources`. Those configuration entry points live at `/collection-management?tab=collector_credentials`.
|
||||
Keep this boundary: do not put custom datasource editing, built-in endpoint overrides, or credential forms back into `/datasources`. Those configuration entry points live at `/collection-management?section=collector_credentials`.
|
||||
|
||||
### Collectors Page
|
||||
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` tab is shown as `Collectors` under `/collection-management`.
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) has three route modes: `/settings` for System Settings, `/earth-content` for Earth Content, and `/collection-management` for Collection Management. The `collector_credentials` section is shown as `Collectors` under `/collection-management`.
|
||||
|
||||
The `System Display` section under `/settings` includes the `Demo Mode` switch. When enabled, Earth OOBE ignores existing current collected data and the local `browse first` temporary skip state, then opens the initialization guide directly. This switch is only for demos and acceptance checks; it does not change datasources, collection queues, or Earth content resources.
|
||||
|
||||
@@ -363,6 +387,7 @@ Current boundary:
|
||||
`/earth-content` reuses the same single-screen tab container from [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx), but its ownership is separate from System Settings:
|
||||
|
||||
- `TV Livestream` owns the Earth media-panel source configuration.
|
||||
- `Branding` owns Earth HUD brand assets. `Logo URL` and `Title Image URL` use inline upload controls inside the fields; the upload buttons keep the primary `TactileButton` style, and dropping an image onto the matching field shows a low-saturation drag reaction instead of the old global asset-picker toolbar.
|
||||
- `Boundary Precision` owns the Earth static boundary asset state: provider, low-precision fallback, high-precision manifest/PMTiles, source JSON, and build action.
|
||||
- `Base Map`, `Layer Resources`, `3D Assets`, and `News Anchor Strategy` are placeholders only. They show module status and do not invent fake APIs or fake data.
|
||||
|
||||
@@ -391,6 +416,7 @@ These principles have been repeatedly validated in the project:
|
||||
4. Do not use `overflow: hidden` to mask structural issues
|
||||
5. Do not compress the main work area to make summary cards show completely
|
||||
6. Custom scrollbars must be floating overlays; they must not squeeze content width
|
||||
7. The Admin shell relies on the root `height: 100%` chain and should not use exact `100vh` sizing at the workspace root
|
||||
|
||||
For detailed experience, see:
|
||||
|
||||
@@ -411,7 +437,7 @@ Do not write local CSS patches first, then retrofit the structure.
|
||||
|
||||
The console frontend and the Earth frontend are not the same system:
|
||||
|
||||
- Console frontend: React + Ant Design workbench
|
||||
- Console frontend: React + Tactile UI / Radix primitives / lucide workbench
|
||||
- Earth frontend: independent native HUD system under `public/earth`
|
||||
|
||||
Therefore:
|
||||
|
||||
@@ -210,6 +210,11 @@ Inspection order:
|
||||
3. Does the real scroll node explicitly use `overflow: auto`?
|
||||
4. Have intermediate wrapper layers silently changed layout semantics?
|
||||
|
||||
The current Admin and Docs root shells rely on the `html` / `body` / `#root`
|
||||
`height: 100%` chain. Do not reintroduce exact `100vh` / `100vw` sizing on
|
||||
these embedded workspace shells; modals, overlays, and narrow-screen safety
|
||||
boundaries may still use `calc(100vh - ...)` as a maximum-size constraint.
|
||||
|
||||
### 5. UI State and Display State Out of Sync
|
||||
|
||||
Repeated in Earth-related changes:
|
||||
|
||||
@@ -22,7 +22,7 @@ URLs below use the local default `http://localhost:3000`. Replace the prefix wit
|
||||
1. Open `http://localhost:3000/login` and click "Register" under the form.
|
||||
2. On `/register`, fill in:
|
||||
- **Username**: 3–50 characters, used to log in
|
||||
- **Email**: receives the verification code; editable later in account settings
|
||||
- **Email**: receives the verification code; in the current version, ask an administrator to maintain email changes in user management
|
||||
- **Password**: at least 8 characters
|
||||
3. After submission you are taken to the verify page. A 6-digit code is sent to your email. It expires in 10 minutes.
|
||||
4. Enter the code and click "Verify and Sign In". On success the system stores a session and sends you to the console.
|
||||
@@ -52,23 +52,23 @@ If you see "Email not verified", the page automatically redirects to `/verify-em
|
||||
3. After receiving the code, enter it together with a new password (≥ 8 characters) and click "Reset Password".
|
||||
4. The system sends you back to `/login` — sign in with the new password.
|
||||
|
||||
## Account Settings
|
||||
## Account Area And Sign Out
|
||||
|
||||
Click your username at the top-right of the console to open account settings:
|
||||
The account area at the bottom of the console sidebar shows the current username, version, and theme control. The current version does not include a signed-in self-service account settings page:
|
||||
|
||||
- Change password: enter current password + new password
|
||||
- Change email: the system sends a verification code to the new address; the change applies only after verification
|
||||
- View Gatekeeper groups: lists current groups (`docs_user` / `docs_developer` / `docs_admin`)
|
||||
- Log out: clears the current session
|
||||
- Use `/forgot-password` for password reset through email verification
|
||||
- Administrators maintain email, role, and Gatekeeper groups at `/users`
|
||||
- The sign-out icon in the account area clears the current session and returns to the login page
|
||||
|
||||
## Console Overview
|
||||
|
||||
The console at `http://localhost:3000/admin` is built with React + Ant Design. The left menu is organized by work domain.
|
||||
The console at `http://localhost:3000/admin` is built with React plus Tactile UI / Radix primitives and lucide icons. The left menu is organized by work domain.
|
||||
|
||||
| Page | Route | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Dashboard | `/admin` | System overview |
|
||||
| Earth | `/earth` | Open the public Earth page |
|
||||
| Docs | `/docs` | Open the docs site and show documents allowed by Gatekeeper permissions |
|
||||
| Datasources | `/datasources` | Source directory and collection triggers |
|
||||
| Collected Data | `/data` | Data already ingested |
|
||||
| BGP | `/bgp` | BGP situational view |
|
||||
@@ -86,7 +86,7 @@ Menu items hide automatically when you lack permission. If a menu is missing, ch
|
||||
|
||||
## Configure Data Collectors
|
||||
|
||||
`/collection-management?tab=collector_credentials` is the "Collectors" page. It manages connection configuration for every collector, not just credentials. Legacy `/settings?tab=collector_credentials` redirects here; the datasource directory remains at `/datasources`.
|
||||
`/collection-management?section=collector_credentials` is the "Collectors" page. It manages connection configuration for every collector, not just credentials; the datasource directory remains at `/datasources`.
|
||||
|
||||
Steps:
|
||||
|
||||
@@ -123,11 +123,14 @@ The default guide follows the BarentsWatch official tutorial and reminds you to
|
||||
|
||||
### AISStream Realtime Vessels
|
||||
|
||||
Once the vessel layer is open, subsequent AISStream and BarentsWatch positions update automatically. Selected vessels stay selected during updates, and reconnecting automatically reconciles the display without repeatedly toggling the layer.
|
||||
|
||||
|
||||
`AISStream Realtime Vessels` is the global AIS WebSocket collector. A passing connection test only confirms API key + endpoint format. Actual global vessel data requires the backend `aisstream_vessels` collector to stay connected and write to `ais_raw_observations`.
|
||||
|
||||
Steps:
|
||||
|
||||
1. Open `/collection-management?tab=collector_credentials` and select `AISStream Realtime Vessels : aisstream_vessels`
|
||||
1. Open `/collection-management?section=collector_credentials` and select `AISStream Realtime Vessels : aisstream_vessels`
|
||||
2. Fill the AISStream API Key
|
||||
3. Keep the default endpoint `wss://stream.aisstream.io/v0/stream`
|
||||
4. Click the plug icon to test; confirm it reports `Reachable`
|
||||
@@ -141,11 +144,12 @@ Steps:
|
||||
|
||||
## Configure AI Credentials
|
||||
|
||||
`/ai?tab=providers` is the AI management entry. Three key sub-tabs:
|
||||
`/ai?section=integrations` is the AI management entry. Key sections:
|
||||
|
||||
- `Model Providers`: default LLM provider, model, base URL, API key, local `aiprovider` proxy, connection test
|
||||
- `Tools`: a dropdown for specific tools — currently WebSearch and OCR
|
||||
- `Tool Calls`: a dropdown for specific tools — currently WebSearch and OCR
|
||||
- `Prompts`: a task dropdown for news localization, alert analysis, BGP briefs, and other LLM tasks. Operators can edit the prompt or reset it to the default
|
||||
- `Playground`: real session, preset request, and AI Provider status debugging
|
||||
|
||||
### Model Providers
|
||||
|
||||
@@ -159,7 +163,11 @@ Providers and models accept presets or arbitrary custom IDs. Common fields:
|
||||
- Max Tokens, Anthropic Version: keep defaults if unsure
|
||||
- Timeout / Retry: timeout and retry attempts
|
||||
|
||||
The plug icon at the end of the Base URL input runs a connection test. A passing test echoes the model's short reply.
|
||||
The plug icon at the end of the Base URL input runs a lightweight connection check: it checks the proxy and model catalog, then confirms the selected model is listed. Use Playground afterward to verify generation access and the model's reply; the catalog check itself does not generate a response.
|
||||
|
||||
Use the refresh icon at the top right to update the available models. A successful refresh saves the catalog for later visits; catalogs with release dates show newer models first. Refresh preserves the current model, credentials, URLs, and unsaved edits. Select a model and click Save to change the model used by the application. A failed refresh displays an error and keeps the previous catalog.
|
||||
|
||||
Refresh uses the base URL, protocol, and API key currently entered in the form, so you can fill new configuration and refresh before saving. Credentialed providers require a key for the matching service region; an empty Ollama catalog means no models are installed. Selecting a model with a known protocol also updates the protocol field. Custom OpenAI gateways retain the manually selected protocol.
|
||||
|
||||
### Tools
|
||||
|
||||
@@ -170,7 +178,7 @@ The plug icon at the end of the Base URL input runs a connection test. A passing
|
||||
|
||||
After selecting a task, the page shows the effective prompt, whether it is customized, the shipped default version, and a reset button. Saving affects only that task. Reset restores the default prompt from the current release package. Business facts, context, and output schemas are still assembled by the backend for each task.
|
||||
|
||||
The legacy link `/settings?tab=ai` redirects to `/ai?tab=providers`.
|
||||
AI configuration no longer lives in System Settings; `/playground` redirects to `/ai?section=playground`.
|
||||
|
||||
## Datasources and Task Logs
|
||||
|
||||
@@ -191,9 +199,10 @@ TV livestreams and boundary precision moved to `/earth-content`; collectors and
|
||||
|
||||
`/earth-content` is under the console's Operations and Configuration group and owns resources used by the Earth frontend:
|
||||
|
||||
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. Uploaded images are saved as Earth brand assets and read by the Earth page immediately.
|
||||
- **Brand Assets**: manages the logo, title image, title text, subtitle, and description used by the Earth HUD. The `Logo URL` and `Title Image URL` fields each include their own Upload button, and image files can be dropped directly onto the matching field. After upload, the field receives the new asset URL; save the brand configuration to make the Earth page use it.
|
||||
- **About**: manages the About card shown in Earth settings, including logo, kicker, title, version, description, and metadata.
|
||||
- **TV Livestream**: manages sources shown in the Earth media panel.
|
||||
- **News Content**: browses news grouped by RSS source and manual group. RSS items remain read-only; manual groups support create, JSON import, edit, delete, and reprocess.
|
||||
- **Boundary Precision**: shows the current provider, low-precision fallback, high-precision PMTiles/manifest status, local source JSON, and manual build action.
|
||||
- **Base Map**, **Layer Resources**, **3D Assets**, and **News Anchor Strategy**: placeholder tabs for future configuration. They do not display fake data.
|
||||
|
||||
@@ -244,7 +253,7 @@ To let a regular user read developer or operations docs, add `docs_developer` or
|
||||
|
||||
## AI Testbench
|
||||
|
||||
`/ai?tab=playground` is for real-pipeline debugging:
|
||||
`/ai?section=playground` is for real-pipeline debugging:
|
||||
|
||||
- Pick the active provider
|
||||
- Run preset requests or custom prompts
|
||||
@@ -280,6 +289,10 @@ AIS vessel legend colors by type: cargo, tanker, passenger, fishing, military, m
|
||||
|
||||
Search finds cables, landing points, satellites, compute centers, BGP events, BGP observers. Results jump to and focus the object.
|
||||
|
||||
### Choosing a News Live Stream
|
||||
|
||||
Open the live tab in the media panel and click the current channel to open the search menu. Search covers the complete catalog. The list loads 50 channels at a time and appends another page when you scroll to the bottom. A fixed footer shows the loaded and matching channel counts; a failed request can be retried below the list. The default source is Al Jazeera Mubasher, using its HLS playback URL. Other channels remain subject to availability of their playback service.
|
||||
|
||||
### Coordinate Candidate Collection
|
||||
|
||||
Compute center and BGP observer detail cards support automatic coordinate-candidate collection. Click the object then use "Collect Coordinate Candidates" or "Re-collect Coordinates". The backend assembles candidates from source coordinates, public-org registry APIs, and online geocoders. When regular sources have no candidate, the current default AI Provider runs one LLM factcheck fallback. BGP observers' stored coordinates only fill query context; they are not returned as candidates.
|
||||
@@ -296,9 +309,11 @@ Recommended single-object flow:
|
||||
|
||||
Adopt All is for batch processing the compute-center unresolved queue. It starts from the top and adopts the highest-confidence candidate. Records without factual support remain in the queue. When WebSearch is disabled, single locate and Adopt All are disabled because location validation depends on factual lookup.
|
||||
|
||||
The batch continues within the current Earth page when you close the candidate panel, inspect another object, or switch browser tabs. Reopening the queue restores progress and results. Saved entries are not collected again, and a ✓ entry beside the layer remains available after completion. Refreshing, closing, or navigating away from Earth interrupts unfinished work; coordinates already saved remain stored.
|
||||
|
||||
### Settings
|
||||
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
The settings panel is grouped into Runtime, Display, Panels, Motion, Shortcuts, and System. It covers rotate / cruise / motion mode, cruise modules (BGP/news/compute centers/vessels/cables/satellites), view (satellite display style, hover tooltip, satellite idle breathing, real satellite altitude, track display, compact dots, day-night mode, panel toggles), motion debug mode / input source / skeleton-only / recognized-gesture whitelist, shortcut enablement and remapping, default globe size, terrain opacity, reset.
|
||||
|
||||
News categories use the same chip selector as Cruise Modules. They only filter the news panel and news cruise items in the current browser; they do not affect layers, TV, data points, basemap, boundaries, collector jobs, or admin news-source configuration.
|
||||
|
||||
@@ -336,6 +351,8 @@ Enable via the settings toggle "Motion Debug Mode", or with URL parameter `?moti
|
||||
|
||||
Neither mode uploads camera frames or live gestures; neither reuses the news/RSS aggregation API.
|
||||
|
||||
Recognized Gestures can disable rotation, zoom, focus switching, layer switching, or confirmation independently. The browser ignores unchecked actions; when Motion Agent is active, the same whitelist is synchronized through the control protocol.
|
||||
|
||||
Gesture semantics:
|
||||
|
||||
| Event | Effect |
|
||||
|
||||
@@ -28,6 +28,9 @@ This document standardizes terms used across Intelligent Planet, the console, ba
|
||||
| AI Provider | AI Provider | Service name |
|
||||
| tool | 工具 | Web Search, OCR, and similar integrations |
|
||||
| Playground | Playground | Interactive debugging entry |
|
||||
| branding | 品牌标识 | Earth HUD brand configuration section under `/earth-content` |
|
||||
| brand assets | 品牌资源 | Logo, title image, and related HUD copy assets |
|
||||
| title image | 标题图 | Earth HUD title image |
|
||||
|
||||
## Data Types
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
## Background
|
||||
|
||||
Use `zsh ./planet.sh start --non-motion-agent` for daily startup; use `init` when preparing a new environment. When investigating latency, distinguish initial dependency downloads, container readiness, and application initialization using the stage timestamps.
|
||||
|
||||
Before preparing the AI Provider image, startup verifies the backend's actual database connection and recreates a missing port mapping once while preserving the volume. A terminated backend process or Uvicorn initialization, ASGI loading, import, or syntax failure stops waiting and identical retries immediately. Normally slow initialization keeps its existing timeout budget. AI Provider readiness probes the host `/health` endpoint directly, without waiting for Docker's first scheduled health check.
|
||||
|
||||
`planet.sh` manages start, stop, restart, health checks, and logs for all local services. The previous implementation had several startup issues:
|
||||
|
||||
1. AI Provider rebuilt every time, even when code had not changed.
|
||||
@@ -37,27 +41,9 @@ write_ai_provider_build_stamp() {
|
||||
}
|
||||
```
|
||||
|
||||
### Faster Fingerprint
|
||||
### Fingerprint Scope
|
||||
|
||||
The previous implementation tarred the whole `aiprovider/` directory before hashing, which could take seconds in large trees. The new version uses `find + stat` and reads only file metadata:
|
||||
|
||||
```bash
|
||||
compute_ai_provider_build_fingerprint() {
|
||||
find aiprovider \
|
||||
-type f \
|
||||
! -path '*/__pycache__/*' \
|
||||
! -name '.env' \
|
||||
! -name '.env.*' \
|
||||
! -name '*.pyc' \
|
||||
! -name '*.pyo' \
|
||||
| LC_ALL=C sort \
|
||||
| xargs -r stat --format="%Y %s %n" 2>/dev/null
|
||||
sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null
|
||||
python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null
|
||||
}
|
||||
```
|
||||
|
||||
This is roughly 10 times faster for many-small-file workloads while preserving the same practical rebuild signal. `.env` and `.env.*` are excluded because runtime model, key, and Base URL changes should not force an image rebuild.
|
||||
The fingerprint hashes file contents from `aiprovider/`, the Dockerfile, the root manifest and lockfile, and the provider dependency information. It does not traverse frontend assets or downloaded data. `.env` and `.env.*` are excluded because they are runtime configuration. The Dockerfile applies its fingerprint label after dependency installation so a changed build marker alone does not invalidate dependency layers.
|
||||
|
||||
### Docker Build Context
|
||||
|
||||
@@ -83,13 +69,15 @@ The Dockerfile copies only AI Provider inputs:
|
||||
```dockerfile
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
```
|
||||
|
||||
`uv sync` uses a BuildKit cache mount. The first build may still depend on network speed, but later builds reuse `/root/.cache/uv`.
|
||||
|
||||
The `aiprovider` dependency group in the root `pyproject.toml` uses the same `uv.lock` and installs only the API, HTTP client, settings, and ASGI runtime dependencies. The image excludes backend collectors and OpenCV / MediaPipe motion dependencies. The build fingerprint label comes after dependency installation and code copying, so a fingerprint change alone does not invalidate dependency layers. The container starts the installed `.venv/bin/python` directly, without runtime dependency synchronization. Dependency changes must update this group and the lockfile and validate image imports and `/health`.
|
||||
|
||||
### Runtime Configuration
|
||||
|
||||
Before starting AI Provider, `planet.sh` generates a current-user runtime env-file and passes it to Compose or the manual `docker run` fallback. The default path is `${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_runtime.env`. Configuration priority:
|
||||
@@ -117,7 +105,7 @@ When the fingerprint matches, the script skips `docker compose build` and starts
|
||||
docker start planet_aiprovider
|
||||
```
|
||||
|
||||
`docker stop` stops the container without deleting the image. `cleanup_exit_containers` removes exited containers but not images, so the next `docker start` can reuse the existing image.
|
||||
`docker stop` stops the container without deleting the image. `start` and `restart` retain stopped containers instead of scanning and deleting all exited containers on the host. An unchanged AI Provider can be reused, while Compose still reconciles database configuration. Existing recreation paths remain responsible for image or configuration changes.
|
||||
|
||||
## Issue 2: Slow Port Cleanup
|
||||
|
||||
@@ -244,22 +232,27 @@ The recommended direction is a small platform compatibility layer for port liste
|
||||
|
||||
The production frontend shape is `vite build` static output served by nginx/Caddy or an equivalent HTTP server. Do not use `bun run dev` or `vite preview` in production. The project does not maintain a parallel Webpack build chain; if a future enterprise requirement needs closer Webpack-ecosystem compatibility, run an Rsbuild/Rspack spike first. Electron should only be evaluated when the official target becomes an offline desktop application.
|
||||
|
||||
## Optional Motion Agent Startup
|
||||
## Default Motion Agent Startup
|
||||
|
||||
`planet.sh` can now manage the local Motion Capture Agent. It is disabled by default so ordinary development machines do not fail startup when cameras, OpenCV, or MediaPipe are unavailable.
|
||||
`planet.sh` now starts the local Motion Agent by default during `start` and full `restart`. This makes the Earth page, UE clients, and debug clients able to connect to `ws://127.0.0.1:8765/ws/gestures` immediately. If the machine has no usable camera, implicit default startup falls back to dry-run protocol mode and does not block backend/frontend startup. Explicit Motion Agent startup through `--motion-agent`, camera indexes, camera URLs, or WSL USB options still treats live camera failures as real errors.
|
||||
|
||||
Start it with:
|
||||
To skip Motion Agent for this run:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent
|
||||
./planet.sh start --non-motion-agent
|
||||
./planet.sh restart --non-motion-agent
|
||||
```
|
||||
|
||||
Common options:
|
||||
|
||||
- `--motion-agent` / `-m`: start or restart the Motion Agent for this command.
|
||||
- `--non-motion-agent`: do not start Motion Agent for this `start` or full `restart`.
|
||||
- `--motion-agent` / `-m`: explicitly start or restart the Motion Agent for this command; live camera failures are reported as failures.
|
||||
- `--motion-agent-port <port>`: override the default WebSocket port `8765`.
|
||||
- `--motion-agent-mode <mode>`: choose `auto`, `single`, `dual_redundant`, `single_fallback`, or `calibrated_3d`; `dual` is kept as a compatibility alias for redundant dual-camera mode.
|
||||
- `--motion-agent-camera-indexes <indexes>`: override auto-detected camera indexes, for example `0` or `0,1`. The same can be provided through `MOTION_AGENT_CAMERA_INDEXES=0,1`.
|
||||
- `--motion-agent-camera-urls <urls>`: use RTSP/HTTP camera streams, useful for WSL, phone cameras, or network cameras. The same can be provided through `MOTION_AGENT_CAMERA_URLS=...`.
|
||||
- `--motion-agent-wsl-usbipd`: in WSL, try to attach the single detected Windows USB camera through `usbipd-win`.
|
||||
- `--motion-agent-wsl-usbipd-busid <BUSID>`: in WSL, attach the camera matching a `usbipd list` BUSID; use this when multiple cameras are present.
|
||||
- `--motion-agent-dry-run`: start only the protocol service without opening cameras or loading CV dependencies; useful for Web client debugging.
|
||||
|
||||
Non-dry-run live mode checks `mediapipe` and `opencv-python` before startup. If the current `.venv` is missing them, the script automatically runs:
|
||||
@@ -268,13 +261,17 @@ Non-dry-run live mode checks `mediapipe` and `opencv-python` before startup. If
|
||||
uv add mediapipe opencv-python
|
||||
```
|
||||
|
||||
To disable startup-time auto-install:
|
||||
To disable startup-time Python CV dependency auto-install:
|
||||
|
||||
```bash
|
||||
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start --motion-agent
|
||||
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start
|
||||
```
|
||||
|
||||
Live mode auto-detects `/dev/video*` and passes the first two indexes to the Motion Agent. In WSL, Windows cameras usually do not appear as `/dev/video*` automatically. Check available devices first:
|
||||
`./planet.sh init` also performs a WSL host dependency preflight for `usbipd-win`. If `usbipd.exe` is missing, the script first tries `winget install -e --id dorssel.usbipd-win`, then reuses the repository-bundled dorssel.usbipd-win MSI fallback. If that cached MSI is missing or an architecture-specific MSI is needed, it downloads one and requests an Administrator PowerShell installation. This is best-effort: failure prints next steps but does not block normal initialization. Use `./planet.sh init --non-motion-agent` to skip this preflight.
|
||||
|
||||
Live mode auto-detects `/dev/video*`, then prefers an OpenCV probe to keep only indexes that can open and return frames before passing them to the Motion Agent. In WSL/USB camera setups, one camera can expose multiple `/dev/video*` nodes, and some of them are metadata or non-capture nodes; the script skips those unreadable indexes. In WSL, Windows cameras usually do not appear as `/dev/video*` automatically. Check available devices first:
|
||||
|
||||
Live capture defaults to low-latency settings: `640x360` input and roughly `15Hz` recognition events. The worker uses latest-frame reader threads and keeps only the newest frame from each camera, so a slow MediaPipe frame does not make the recognizer drain stale camera backlog. The skeleton debug stream is disabled by default and is only sent at roughly `8Hz` while the Earth motion debug panel is open, so normal gesture control is not slowed down by debug data. Status events report both capture FPS and recognition FPS to separate camera throughput issues from recognition cost.
|
||||
|
||||
```bash
|
||||
ls /dev/video*
|
||||
@@ -292,20 +289,42 @@ In WSL, the more general path is to connect a phone or network camera through an
|
||||
./planet.sh start --motion-agent --motion-agent-camera-urls http://192.168.1.20:8080/video
|
||||
```
|
||||
|
||||
If WSL has no `/dev/video*` and no `--motion-agent-camera-urls` is provided, live startup stops and prints guidance instead of silently falling back to dry-run. Choose one of:
|
||||
To use a Windows USB camera directly from WSL, let the script call `usbipd-win`. This is opt-in because an attached camera is usually temporarily unavailable to Windows apps while WSL owns it.
|
||||
|
||||
When there is only one camera:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
|
||||
```
|
||||
|
||||
When there are multiple cameras, inspect the BUSID first and pass it explicitly:
|
||||
|
||||
```bash
|
||||
usbipd.exe list
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd-busid 3-2
|
||||
```
|
||||
|
||||
If `usbipd attach` says the device is not shared or bound, the script tries to open an Administrator PowerShell to run `usbipd bind`, then retries attach. If UAC is canceled or automatic bind fails, run this manually from an Administrator PowerShell:
|
||||
|
||||
```powershell
|
||||
usbipd bind --busid 3-2
|
||||
usbipd attach --wsl --busid 3-2
|
||||
```
|
||||
|
||||
If WSL has no `/dev/video*` and no `--motion-agent-camera-urls` is provided, implicit default startup falls back to dry-run. Explicit live startup stops and prints guidance. Choose one of:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent --motion-agent-camera-urls http://<phone-ip>:8080/video
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
|
||||
./planet.sh start --motion-agent --motion-agent-dry-run
|
||||
```
|
||||
|
||||
Automatic dry-run fallback only happens when `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` is explicitly set.
|
||||
For explicit live startup, automatic dry-run fallback only happens when `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` is set.
|
||||
|
||||
Environment-variable startup is also supported:
|
||||
`--non-motion-agent` is the command-level opt-out. Environment variables can still tune how the service starts:
|
||||
|
||||
```bash
|
||||
PLANET_START_MOTION_AGENT=1 ./planet.sh start
|
||||
MOTION_AGENT_DRY_RUN=1 PLANET_START_MOTION_AGENT=1 ./planet.sh start
|
||||
MOTION_AGENT_DRY_RUN=1 ./planet.sh start
|
||||
```
|
||||
|
||||
Logs:
|
||||
|
||||
@@ -2,6 +2,48 @@
|
||||
|
||||
This runbook is for deployment, on-call, and maintenance engineers. End-user UI flows live in the [Intelligent Planet Manual](/home/ray/dev/linkong/planet/docs/technical/en/manual.md); this document only covers shell, Docker, logs, environment variables, and troubleshooting.
|
||||
|
||||
## Docker Initialization and Access
|
||||
|
||||
Initialize a new machine before starting the application services:
|
||||
|
||||
```bash
|
||||
zsh ./planet.sh init --non-motion-agent && zsh ./planet.sh start --non-motion-agent
|
||||
```
|
||||
|
||||
The entry point still requires `zsh`, `curl`, and reachable package repositories. Before synchronizing Python and frontend dependencies, `init` prepares Docker:
|
||||
|
||||
- Reuse working Docker, Compose v2, and Buildx (at least 0.17.0).
|
||||
- On Ubuntu / Ubuntu WSL, use apt to install the missing parts of `docker.io`, `docker-compose-v2`, and `docker-buildx`. When Docker CE CLI is already installed, use the configured Docker CE repository and corresponding plugin packages to keep the package family consistent.
|
||||
- If the local daemon is unavailable, check that `docker.service` exists, then enable and start it. WSL must have systemd enabled; unavailable service management produces an explicit Docker preparation error.
|
||||
- If the current user cannot read and write the Docker socket, check for `usermod`, install its `passwd` package when needed, and add the user to the `docker` group. This group grants privileged control of the local Docker engine. The script uses `sudo` to refresh group access as the original user and continue the original command with its arguments preserved. It does not depend on `sg` or run application processes as root.
|
||||
|
||||
When elevation is needed, sudo authentication runs in the foreground. Missing sudo for an unprivileged user, authentication failure, repository errors, or insufficient versions after installation stop initialization with a specific error.
|
||||
|
||||
Subsequent `planet.sh start` and other service commands in the same old terminal also refresh Docker group membership when it has been granted but is not yet active. Open a new Ubuntu session to use `docker` directly in the terminal.
|
||||
|
||||
When Docker Desktop is present but its WSL integration is unavailable, the script asks the operator to start Desktop and enable WSL Integration for the distribution. Unreachable remote or rootless endpoints produce a diagnostic for that environment; neither case installs a second local engine automatically. Automatic installation on other operating systems is not currently supported.
|
||||
|
||||
`planet.sh` calls `scripts/lib/docker-bootstrap.zsh` for preparation. Missing CLI, missing service units, socket permissions, and stopped daemons receive separate diagnostics. Advice to start `docker.socket` is shown only after confirming that the unit exists. Verify the result with:
|
||||
|
||||
```bash
|
||||
docker info
|
||||
docker compose version
|
||||
docker buildx version
|
||||
```
|
||||
|
||||
## Database Initialization and Connection Checks
|
||||
|
||||
`init` and `start` reconcile PostgreSQL / Redis containers through Compose, including port configuration on existing containers. A plain `docker start` cannot apply configuration changes. Compose failures retain their specific errors, such as an occupied port, instead of falling back to an old container and reporting success.
|
||||
|
||||
The container's `pg_isready` check only establishes that the server accepts connections; it does not validate the host backend's address and credentials. Once containers are healthy, both initialization and backend startup run `scripts/check_database_connection.py` using the backend's effective `DATABASE_URL`. It checks the local PostgreSQL published port and executes a read-only `SELECT 1`. Startup performs this check before preparing the AI Provider image and stops immediately on failure. Initialization only creates tables and seed data after the check passes.
|
||||
|
||||
- If the actual local port mapping is still missing or mismatched, the script recreates PostgreSQL once from Compose while preserving its data volume, then checks again. A second failure stops initialization.
|
||||
- Authentication, database-name, and network failures stop before schema changes. Diagnostics show the host, port, and database name without passwords, full connection strings, or raw driver exceptions.
|
||||
- A process-level `DATABASE_URL` overrides `backend/.env`. Changing `POSTGRES_PASSWORD` alone updates neither the connection string nor the password stored in an existing data volume. Existing environment files are retained and their effective configuration must be checked.
|
||||
- Explicit external databases do not require a local container mapping. Host networking also does not require published ports. Both still require the real connection check.
|
||||
|
||||
For `port is already allocated` or `address already in use`, inspect `docker ps` port information and `ss -ltnp '( sport = :5432 )'`. With WSL mirrored networking, also inspect Windows listeners. Initialization does not kill other database services to acquire a port, delete data volumes, or reset passwords.
|
||||
|
||||
## First Startup
|
||||
|
||||
```bash
|
||||
|
||||
@@ -39,7 +39,7 @@ flowchart TB
|
||||
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
|
||||
VesselDerived --> VesselLayer["vessels layer"]
|
||||
Interactables["earth_interactables"] --> InteractableLayer["interactables layer"]
|
||||
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
|
||||
NewsRaw["RSS / Manual News / Live"] --> NewsItems["earth_news_items"]
|
||||
NewsItems --> NewsLayer["news / media layer"]
|
||||
```
|
||||
|
||||
@@ -51,7 +51,7 @@ flowchart TB
|
||||
| BGP context | Collectors, anomalies, incidents, route events, and regional context | `ris_live_bgp`, `bgpstream_bgp`, prefix geography sources | `bgp_observations`, `bgp_anomalies`, `bgp_incidents`, `bgp_collector_locations` | `bgp` | `clear_then_reload` |
|
||||
| Vessels | AIS vessels, positions, tracks, legend, and source health | AIS sources, `barentswatch_vessels` | `vessel_static`, `vessel_position`, `ais_raw_observations`, `ais_source_health` | `vessels` | `clear_then_reload` |
|
||||
| Interactables | Generic surface icons, manual objects, and future small layers | `earth_interactables` | None | `interactables` | `delta` |
|
||||
| News and media | Earth news, live streams, cruise summaries, and situation content | news sources | `earth_news_items` | `news` / `media` | `reload` |
|
||||
| News and media | Earth news, live streams, cruise summaries, and situation content | RSS news sources, manual news, live streams | `earth_news_items` | `news` / `media` | `reload` |
|
||||
|
||||
## Satellites
|
||||
|
||||
@@ -141,12 +141,12 @@ Vessel data shows AIS vessels, navigation state, vessel-type legend, and source
|
||||
|
||||
News and media support the Earth news ticker, live stream panel, news cruise, and situation summaries. They are content refresh paths rather than stable geographic object layers, so they default to `reload`.
|
||||
|
||||
- **Collection entry**: RSS, live streams, news sources.
|
||||
- **Fact table**: news source rows in `collected_data`.
|
||||
- **Collection entry**: RSS news sources, manual news from `Earth Content -> News Content`, and live streams.
|
||||
- **Fact table**: news source rows in `collected_data`; manual news writes directly to `earth_news_items` and marks the content source with `feed_type/source_type=manual`.
|
||||
- **Derived table**: `earth_news_items`.
|
||||
- **API**: news, live stream, and media content APIs.
|
||||
- **API**: `/api/v1/news/earth-feed` reads `earth_news_items`; the Admin API `/api/v1/earth/news-items` supports manual create, JSON import, edit, delete, and reprocess.
|
||||
- **Delete semantics**: deleting news sources or `earth_news_items` broadcasts `news` / `media` reload; empty responses hide the corresponding content.
|
||||
- **Common failure**: the live panel shows stale content. Usually the media component ignored the layer update or the content API cache was not invalidated.
|
||||
- **Common failure**: a newly saved manual item may initially show source text or a region anchor; this is the normal publish-first enrichment window. If it never updates, check the `earth_news_enrichment` queue, AI / Web Search configuration, and `enrichment_status`. If the live panel shows stale content, the media component likely ignored the layer update or the content API cache was not invalidated.
|
||||
|
||||
## Adding a New Layer
|
||||
|
||||
|
||||
@@ -32,11 +32,12 @@ The default role is `viewer`: you can sign in but only see public pages. For col
|
||||
|
||||
After landing on the `/admin` dashboard, here's a recommended walk-through:
|
||||
|
||||
1. `/collection-management?tab=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
|
||||
2. `/ai?tab=providers`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to test. WebSearch / OCR tools are optional
|
||||
3. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
|
||||
4. `/alerts/system`: verify system alerts look right
|
||||
5. `/users` (super_admin only): open accounts for teammates or adjust their groups
|
||||
1. `/collection-management?section=collector_credentials`: pick a collector and click the plug icon to test connectivity. Free collectors (e.g. open BGP) usually work right away; credential-bearing ones like `AISStream` or `BarentsWatch` need an API key / client secret first
|
||||
2. `/ai?section=integrations`: fill an LLM provider (e.g. `minimax` / `openai`), model, base URL, API key, and click the plug at the end of the base URL to check the model catalog; verify generation in Playground. The refresh icon updates and saves the available model catalog while preserving your form; select a model and click Save to apply it. WebSearch / OCR tools are optional
|
||||
3. `/earth-content?section=brand`: maintain the Earth HUD logo and title image in Branding. The Upload button inside each URL field opens a file picker, and image files can also be dropped directly onto the matching field. Save the brand configuration after upload
|
||||
4. `/datasources` or `/data`: check whether collectors have produced data. Use `/datasources -> Built-in Sources` for finite collectors: with no rows selected, click `Trigger All`; after selecting rows, the primary button becomes `Trigger Selected N`. The top-right queue button shows progress. Use `/datasources -> Realtime Sources` for AISStream / WebSocket health and counters
|
||||
5. `/alerts/system`: verify system alerts look right
|
||||
6. `/users` (super_admin only): open accounts for teammates or adjust their groups
|
||||
|
||||
## 4. Open Earth
|
||||
|
||||
@@ -49,8 +50,11 @@ Once in, verify:
|
||||
- The globe renders, and the right-side layer panel can toggle layers
|
||||
- Search finds cables, satellites, compute centers, BGP events
|
||||
- Compute-center and BGP collector detail cards can collect coordinate candidates and preview them on Earth
|
||||
- Close the candidate panel or switch browser tabs during batch location, then return to see progress; refreshing or leaving Earth interrupts unfinished work
|
||||
- The vessel layer receives ongoing AISStream / BarentsWatch positions and reconciles after reconnecting
|
||||
- The live-stream menu searches the complete channel catalog and loads more on scroll; Al Jazeera Mubasher is the default
|
||||
- Mouse drag, wheel zoom, and the zoom percentage indicator work
|
||||
- The settings panel can switch rotate / cruise / motion modes; view settings can switch hover tooltip content, and satellite settings can toggle real-altitude layering and track display
|
||||
- The settings panel can switch rotate / cruise / motion modes; motion settings can select the input source and allowed gestures; view settings can switch hover tooltip content, and satellite settings can toggle real-altitude layering and track display
|
||||
|
||||
## 5. Recover a Lost Password
|
||||
|
||||
|
||||
@@ -51,6 +51,7 @@
|
||||
支持的请求适配器:
|
||||
|
||||
- `openai-completions`
|
||||
- `openai-responses`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
@@ -95,12 +96,37 @@ AI 配置页使用的接口:
|
||||
- `POST /api/v1/settings/integrations/ai-provider/connect`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/secrets`
|
||||
- `GET /api/v1/settings/integrations/ai-provider/presets`
|
||||
- `POST /api/v1/settings/integrations/ai-provider/presets/{provider}/refresh`
|
||||
- `GET /api/v1/settings/ai-prompts`
|
||||
- `PUT /api/v1/settings/ai-prompts/{task_key}`
|
||||
- `POST /api/v1/settings/ai-prompts/{task_key}/reset`
|
||||
|
||||
这些接口都需要用户登录。`secrets` 接口只用于配置页点击显示 key/token 时取回明文,隐藏时前端恢复为脱敏预览。
|
||||
|
||||
模型目录刷新和轻量连通性测试共用 `backend/app/services/llm_model_catalog.py`,直接请求当前供应商的模型接口,不再依赖 models.dev。使用当前表单草稿中的基础地址、协议和凭证;不传草稿时读取已保存配置。国内、国际和自定义网关地址保持各自的地域与路径。缺少必需凭证时返回 400 并提示配置 API Key;上游超时、鉴权或响应错误返回安全提示,并保留上次目录。
|
||||
|
||||
成功目录和 `refreshed_at` 按供应商保存到 `system_settings` 的 `llm_provider_preset:<provider>` 分类。该动态分类使用对应供应商预设作为默认值,提交前校验分类,避免已提交却返回失败。列表优先返回已保存目录,未刷新过的供应商显示内置建议。刷新不修改 `external_integrations` 的当前模型、协议、地址或凭证;页面保留草稿,目录加载失败时保留已显示模型并明确提示失败。
|
||||
|
||||
### 模型目录接口
|
||||
|
||||
以下接口于 2026-09-13 对照官方文档核对。表中的 URL 是模型列表地址,配置表单仍填写生成接口的基础地址。需要凭证的目录必须使用对应服务地域的 API Key。
|
||||
|
||||
| 供应商 | 模型列表 GET 接口 | 鉴权与解析 |
|
||||
| --- | --- | --- |
|
||||
| [MiniMax](https://platform.minimax.io/docs/api-reference/models/anthropic/list-models) | `https://api.minimaxi.com/anthropic/v1/models`;国际站使用 `api.minimax.io` | `x-api-key`;`data[].id`;处理 `has_more/last_id` |
|
||||
| [OpenAI](https://developers.openai.com/api/reference/resources/models/methods/list) | `https://api.openai.com/v1/models` | Bearer;`data[].id` |
|
||||
| [Anthropic](https://platform.claude.com/docs/en/api/models/list) | `https://api.anthropic.com/v1/models` | `x-api-key`、`anthropic-version`;按 `after_id` 翻页 |
|
||||
| [DeepSeek](https://api-docs.deepseek.com/api/list-models) | `https://api.deepseek.com/v1/models` | Bearer;`data[].id` |
|
||||
| [阿里百炼](https://help.aliyun.com/zh/model-studio/list-models) | 同地域主机的 `/api/v1/models` | Bearer;`output.models[].model`;按 `page_no/page_size/output.total` 翻页,筛选 `capabilities=TG` |
|
||||
| [Moonshot / Kimi](https://platform.kimi.ai/docs/api/list-models) | `https://api.moonshot.ai/v1/models`;国内站使用 `api.moonshot.cn` | Bearer;`data[].id` |
|
||||
| [OpenRouter](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties) | `https://openrouter.ai/api/v1/models` | 支持公共目录;有凭证时附带 Bearer;`data[].id` |
|
||||
| [OpenCode Go](https://opencode.ai/docs/go/#models) | `https://opencode.ai/zen/go/v1/models` | 支持公共目录;有凭证时附带 Bearer;`data[].id` |
|
||||
| [Ollama](https://docs.ollama.com/api/tags) | 本机或配置服务器的 `/api/tags` | 本地无需 Key;`models[].model/name`;空目录表示尚未安装模型 |
|
||||
|
||||
百炼的北京、东京、法兰克福、弗吉尼亚新接口要求实际业务空间主机,例如 `<WorkspaceId>.cn-beijing.maas.aliyuncs.com`;新加坡使用 `dashscope-intl.aliyuncs.com`,香港使用 `cn-hongkong.dashscope.aliyuncs.com`。系统只替换目录路径,不猜测业务空间或跨地域切换凭证。旧北京域名若不再接受账号凭证,应按控制台给出的业务空间地址更新基础地址。
|
||||
|
||||
完整分页和重试受 30 秒总超时保护;网络故障及 502/503/504 最多尝试两次,鉴权失败不重试。按接口提供的创建/发布日期倒序排列;日期相同时保留上游顺序,不截断为固定数量。公共目录只表示供应商公开的模型集合,不代表账号已获得每个模型的生成权限。
|
||||
|
||||
Admin 的 AI 页面按业务信息架构组织为:
|
||||
|
||||
- `模型供应商`
|
||||
@@ -283,22 +309,11 @@ Admin 的密钥状态必须按 provider / tool 精确判断:
|
||||
|
||||
### 轻量连通性测试
|
||||
|
||||
Admin 的插头按钮走轻量连通性测试,不承担保存职责。业界常见做法是分两层:
|
||||
|
||||
- 快速检查 provider 目录或低成本 endpoint,确认 base URL、鉴权和当前模型是否可达。
|
||||
- 只有在用户明确运行 Playground 或业务任务时才发完整模型请求。
|
||||
|
||||
因此,连通性测试应尽量使用低成本请求,并返回明确状态:
|
||||
|
||||
- `ok`: 鉴权、路由和模型目录可用。
|
||||
- `warning`: 服务可达,但当前模型不在目录或能力声明不完整。
|
||||
- `error`: 鉴权失败、网络失败、协议错误或模型不可用。
|
||||
|
||||
错误 toast 标题必须和结果一致,不能在失败时显示“连通性正常”。
|
||||
Admin 插头按钮检查代理服务配置,再查询当前供应商目录;只有目录请求成功且包含所选模型时才返回 `success=true`。404、鉴权失败、无效响应和缺失模型都不会因命中内置建议而被改判成功。提示明确区分“找到模型”和“实际生成成功”;完整调用由 Playground 或业务任务验证。测试不保存表单或切换默认供应商。
|
||||
|
||||
### OpenCode Go 路由模型
|
||||
|
||||
OpenCode Go 这类订阅通道不要靠前端硬编码模型集合判断协议。推荐在 provider catalog 或后端能力发现中记录每个模型的协议能力,例如 `chat_completions`、`anthropic_messages`、`models_endpoint` 和是否需要订阅 key。前端只展示能力结果;后端负责把 provider、base URL、model 和协议适配映射为实际请求。
|
||||
OpenCode Go 的模型协议映射由后端目录集中维护:MiniMax M3/M2.7/M2.5 和已核对的 Qwen3.6/3.7/3.8 模型使用 Anthropic Messages;GPT-5.6 Luna、Grok 4.6 和 Muse Spark Contributor 使用 Responses;其余已支持模型使用 Chat Completions。官方 `/models` 当前只提供模型 ID,新增模型的协议仍需对照官方端点表核对。前端选择模型时同步对应协议,后端覆盖旧的已知错误映射。Responses 请求使用 `input`、`max_output_tokens` 和 `store=false`;回复解析文本及推理摘要。OpenCode 请求携带应用 User-Agent 和会话标识;Playground 同一会话保持标识稳定。MiniMax M3 的通用 `thinking=enabled` 转为官方 `adaptive` 格式。
|
||||
|
||||
#### 配置页行为
|
||||
|
||||
|
||||
@@ -102,6 +102,12 @@ AIS 船只类采集器和其它 `CollectedData` 采集器的落库路径不同
|
||||
|
||||
TOP500 和 Epoch AI 算力数据的公开源不总是提供可用经纬度。Earth 统一算力中心接口在主地图启动链路中只使用源数据自带坐标或 `compute_center_locations` 维表坐标;缺少坐标的记录会进入 `unresolved`,不会通过本地注册表、国家质心或猜测城市自动渲染。用户手动采集候选时,后端会用源字段调用 ROR 组织注册 API 和 Nominatim/OpenStreetMap 在线搜索;候选经前端保存后写入 `compute_center_locations`,后续地图刷新再从维表渲染。
|
||||
|
||||
### 新闻直播频道目录
|
||||
|
||||
`news_live_streams` 的 IPTV-org 适配器保留现有新闻分类筛选,`max_sources` 默认值为 `0`,表示不截断匹配频道;显式正数仍限制采集数量。已有配置若保留旧的 `120`,需要改为 `0` 并重新采集才能补齐目录。
|
||||
|
||||
`GET /api/v1/tv/streams` 由 `tv_catalog.py` 提供数据库分页:`offset` 默认 `0`,`limit` 默认 `50`、最大 `100`,`q` 按空白拆词并匹配频道名、来源、地区和语言。内置及配置源优先,采集源按频道标识去重并排除配置覆盖项,再在数据库中搜索、计数、排序和分页,避免全表读取后切片。响应的 `total` 是匹配总数,`source_count` 是完整可用目录数量;后续页使用 `next_offset` 和 `has_more`。`selected_id` 可额外取得不在当前页的已选频道,不占本页配额。默认和兜底源继续由 `tv_streams.py` 统一管理。
|
||||
|
||||
## 四、数据格式 (统一存储到 CollectedData 表)
|
||||
|
||||
```python
|
||||
@@ -382,7 +388,11 @@ GET /api/v1/visualization/vessels/{mmsi}/conflicts
|
||||
|
||||
`/api/v1/vessels/snapshot` 必须携带 `bbox` 和 `zoom`,后端最大 `limit=5000`。Earth 前端使用全球 bbox 读取当前状态,不随相机视口变化反复请求。接口消费 `vessel_current_state`,并在 `diagnostics.source` 返回 `vessel_current_state`;旧 `/api/v1/visualization/geo/vessels` 路由已移除。
|
||||
|
||||
高频 AIS 更新不要直接推送成每条 delta 的整层重建。`/ws` 的 `vessels` channel 如用于 Earth,应广播低频 reload/dirty 提示,由前端合并刷新 snapshot;轨迹和冲突详情仍按单船接口读取历史事实。
|
||||
AISStream 与 BarentsWatch 共用 `/ws` 的 `vessels` 增量通道。Earth 使用现有 WebSocket 连接订阅 `scope: "global"`,不跟随镜头改变订阅范围。后端按 MMSI 合并一秒内的通知,再读取 `vessel_current_state` 的已确认状态;单帧最多 1000 项,超过时继续发送后续帧,不截断不同船只。原始源消息不能直接覆盖客户端的确认位置。当前状态删除也会产生按 MMSI 的 remove 通知。
|
||||
|
||||
前端将短时间内同一 MMSI 的变更合并为最新值,通过 `Interactable.updateItems()` 原位修改位置和颜色缓冲。航向分桶变更只更新受影响的桶,容量不足才扩容;保留已有 marker 身份、锁定状态和材质。首次进入、重连、删除提示及每分钟巡检仍用 snapshot 校准,但不先清空整个图层。收到有数量上限的快照时,不能把被截断的船只当成删除;快照返回期间到达的较新更新和删除也不能被旧响应覆盖。快照携带查询开始时的 `generated_at`,与实时帧时间一起用于识别缓存旧快照,避免新船或删除状态被回滚。
|
||||
|
||||
`earth_updates` 中船舶普通写入的策略是 `delta`;专用通道连通时不再触发整层重拉。删除或断连后的校准使用 `reload`,并复用现有对象。关闭图层会取消船舶订阅并清空待应用变更。轨迹、冲突详情和审计继续按单船接口读取历史事实。
|
||||
|
||||
### 图层接口与全量统计分离
|
||||
|
||||
|
||||
@@ -70,7 +70,10 @@ DB 变化不再默认创建 `earth_refresh` 任务,因此不会被同 source
|
||||
| --- | --- |
|
||||
| `clear_then_reload` | 先清前端本地图层对象,再强制重拉接口。删除数据时优先使用。 |
|
||||
| `reload` | 保留旧对象直到新数据返回,适合定位、元数据或非破坏性更新。 |
|
||||
| `delta` | 只用于 `earth_interactables`,按 id upsert 或 remove。 |
|
||||
| `delta` | `earth_interactables` 按 id upsert/remove;船舶普通写入由专用 `vessels` 通道按 MMSI 更新确认状态,不触发整层清空。 |
|
||||
|
||||
船舶删除仍发出 `reload` 校准提示,`vessel_current_state` 的逐项删除同时进入船舶 remove 通道。源通知只决定哪些 MMSI 需要更新,推送值以当前状态表为准。全局订阅使用 `scope: "global"`;消息大小上限用于拆包,不用于丢弃其余船只。
|
||||
|
||||
|
||||
接口在真实 0 数据时必须返回 200 和空集合;只有真实接口异常才返回 5xx。前端收到删除事件后,如果重拉失败,应保持已清空状态并显示轻量错误,不恢复旧对象。
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
- 展示所有数据源,包括内置和自定义。
|
||||
- 点击名称只打开信息抽屉。
|
||||
- 负责查看状态、触发采集和查看采集中任务。
|
||||
- `/collection-management?tab=collector_credentials`
|
||||
- `/collection-management?section=collector_credentials`
|
||||
- 显示为“采集器”。
|
||||
- 负责 endpoint、请求头、基础参数和凭证配置。
|
||||
- 所有采集器都提供连接按钮,用于健康检查。
|
||||
@@ -323,7 +323,7 @@ AISStream 使用 WebSocket 实时流,采集器只写入 `ais_raw_observations`
|
||||
GET /api/v1/vessels/snapshot?bbox=-180,-85.05112878,180,85.05112878&zoom=12&limit=3000
|
||||
```
|
||||
|
||||
该接口查询 `vessel_current_state` 当前状态表,只返回有效窗口内每个 MMSI 的最新点;原始 `ais_raw_observations` 继续保留给轨迹、审计和态势分析,但不再由展示接口临时扫描聚合。Earth 前端统一传全球 bbox,不随当前镜头视口反复请求。实时更新如接入 `/ws` 的 `vessels` channel,应作为 reload/dirty 提示触发合并刷新,不能把每条 AIS delta 直接变成整层重建。
|
||||
该接口查询 `vessel_current_state` 当前状态表,只返回有效窗口内每个 MMSI 的最新点;原始 `ais_raw_observations` 继续保留给轨迹、审计和态势分析,但不再由展示接口临时扫描聚合。Earth 前端统一传全球 bbox,不随当前镜头视口反复请求。实时更新使用 `/ws` 的 `vessels` 全局订阅,按 MMSI 推送确认状态并原位更新;超出单帧上限时拆包,保留所有更新。首次加载、重连和删除校准仍使用快照,详见[采集器架构](backend-collectors.md)。
|
||||
|
||||
## 自定义 REST / WebSocket 映射运行时
|
||||
|
||||
|
||||
@@ -56,7 +56,7 @@ React 路由入口:
|
||||
- 各图层集成
|
||||
- Earth 级别状态同步
|
||||
|
||||
Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,真实数据仍通过 `/api/v1/visualization/...` 接口重新 GET。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload`、`reload` 或 `delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉,summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,普通图层仍通过 `/api/v1/visualization/...` 接口重新 GET;船舶使用下文的 `vessels` 专用确认状态增量通道。数据库驱动的刷新由后端 listener 直接清理缓存再广播,不再默认经过 `earth_refresh` 作业队列;前端收到 `database_changed` 后会按 layer 读取 `clear_then_reload`、`reload` 或 `delta` 策略。`clear_then_reload` 必须先清 Three.js 对象再 no-store 重拉,summary 只做一致性校验,不能用 `0` 作为跳过图层重拉的理由。技术链路见 [数据作业与 Outbox 技术架构](/home/ray/dev/linkong/planet/docs/technical/zh/data-job-earth-sync-architecture.md),业务数据流见 [业务架构与数据流转](/home/ray/dev/linkong/planet/docs/technical/zh/platform-data-flows.md)。
|
||||
|
||||
### 3. 地球控制层
|
||||
|
||||
@@ -75,7 +75,13 @@ Earth 收到 `/ws` 的 `earth_updates` 时只把它当作刷新提示,真实
|
||||
|
||||
Earth 设置面板现在按 `data-settings-tab` 和 `data-settings-tab-panel` 分类组织。桌面端和移动端使用同一组分类语义:运行、显示、面板、动捕、快捷键、系统。新增设置项时应先判断它属于哪个分类,再补 DOM、持久化字段和恢复逻辑;不要把所有控件继续堆到一个长面板里。
|
||||
|
||||
`显示` 分类里的新闻类型选择复用巡航模块的 chip 选择器形态,只控制星球端当前浏览器的新闻分类显示。它不会打开或关闭图层、底图、边界、TV、数据点、BGP、船舶、卫星或算力中心;这些仍由图层面板、媒体面板和控制台配置各自负责。`controls.js` 只持久化 `shared.newsCategoryFilters` 并广播 `earth:news-category-filters-change`,`news.js` 会把选中的类型拼到 `/api/v1/news/earth-feed?categories=...&locale=zh-CN`,让 Web 和 UE 走同一套后端类型过滤。
|
||||
Earth 运行在独立 iframe / 静态应用里,不能直接复用 React Admin 的 `react-i18next` 上下文。`public/earth` 自己通过 [i18n.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/i18n.js) 读取并写回全局 `planet-locale`,同时同步旧的 `docs-lang`,这样 Docs、控制台和 Earth 的语言偏好保持一致。语言切换控件属于设置里的 `系统` tab,桌面和移动端都使用同一组 `data-earth-locale` 按钮;不要把语言选择塞进图层、显示或运行模式设置里。
|
||||
|
||||
`显示` 分类里的新闻类型选择复用巡航模块的 chip 选择器形态,只控制星球端当前浏览器的新闻分类显示。它不会打开或关闭图层、底图、边界、TV、数据点、BGP、船舶、卫星或算力中心;这些仍由图层面板、媒体面板和控制台配置各自负责。`controls.js` 只持久化 `shared.newsCategoryFilters` 并广播 `earth:news-category-filters-change`,`news.js` 会把选中的类型和当前 locale 拼到 `/api/v1/news/earth-feed?categories=...&locale=...`,让 Web 和 UE 走同一套后端类型过滤。
|
||||
|
||||
英文模式下,Earth 新闻只能渲染英文标题和摘要。中文来源新闻如果还没有 `en-US` 本地化,会先从可见卡片、滚动条和巡航中隐藏,直到后端增强完成;来源和 feed 标签也必须回退到英文安全名称,避免英文界面混入中文标签。
|
||||
|
||||
新闻面板、滚动条和新闻巡航必须消费同一次 `/api/v1/news/earth-feed` 响应里的 `items` / `cruise_items`,不能各自缓存区域状态。`news.js` 的刷新请求以区域、类型、来源和数量生成 request key;只有 key 相同的并发请求才复用 promise,旧区域请求返回时会被 token 丢弃。来源过滤也必须按区域作用域处理:当用户从亚太切到欧洲等其它区域时,旧区域保存的来源 ID 不允许继续拼到下一次 fetch 里;拿到新 payload 后再与 `sources` 列表做交集,若没有交集则回退到当前区域所有可用来源。这样滚动条、面板和巡航才会在区域切换后展示同一批新闻。
|
||||
|
||||
快捷键配置属于设备本地偏好,由 `controls.js` 负责读取、捕获、启用/禁用和重置。它不应写入后端用户设置,也不应影响其它浏览器。后续新增快捷键时,必须同时提供默认键、显示标签、可禁用状态和重置路径,避免只在 keydown handler 中硬编码。
|
||||
|
||||
@@ -109,9 +115,9 @@ Earth 设置面板现在按 `data-settings-tab` 和 `data-settings-tab-panel`
|
||||
- 将 `rotate_left`、`rotate_right`、`rotate_up`、`rotate_down`、`zoom_in`、`zoom_out`、`focus_prev`、`focus_next`、`layer_prev`、`layer_next`、`confirm` 映射到 `main.js` 暴露的动作入口。
|
||||
- 解析 `skeleton` 调试事件并派发 `earth:motion-debug-frame`。
|
||||
|
||||
动作捕捉识别可以在浏览器本地执行,也可以在本地 Agent 中执行,但两者都不会把实时视频帧发给 SaaS 云端。`main.js` 暴露旋转、缩放、目标切换、图层切换和确认入口,并通过 `window.__planetEarth.motion` 提供调试入口。默认只有 URL 参数 `?motion=1`、本地存储 `planet-earth-motion-control-enabled=true`,或 Earth 设置中的“动捕调试模式”打开时才启动当前 provider。
|
||||
动作捕捉识别可以在浏览器本地执行,也可以在本地 Agent 中执行,但两者都不会把实时视频帧发给 SaaS 云端。`main.js` 暴露旋转、缩放、目标切换、图层切换和确认入口,并通过 `window.__planetEarth.motion` 提供调试入口。默认只有 URL 参数 `?motion=1`、本地存储 `planet-earth-motion-control-enabled=true`,或 Earth 设置中的“动捕调试模式”打开时才启动当前 provider。`shared.motionEnabledGestures` 保存用户允许识别的动作;浏览器端先过滤,Motion Agent 模式还会通过 `set_enabled_gestures` 同步给服务端。
|
||||
|
||||
动捕调试面板由 [motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) 负责。它监听 `earth:motion-debug-frame`,用 canvas 绘制归一化骨架点和连线;Browser Camera provider 会额外通过 `earth:motion-debug-video-source` 提供本机 `<video>` 作为调试预览底图,`shared.motionDebugSkeletonOnly` 可切换为只显示骨骼。`停止匹配动作` 通过 `earth:motion-recognition-pause` 暂停 gesture 执行,但继续显示视频和骨架。未匹配动作为红色,匹配后变绿并显示动作名。设置项持久化在 `planet.earth.settings.v2` 的 `shared.motionDebugEnabled`、`shared.motionProvider` 与 `shared.motionDebugSkeletonOnly`,switch 和输入源控件都预留 `data-gatekeeper-permission="earth.motion_debug"`。
|
||||
动捕调试面板由 [motion-debug-panel.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-debug-panel.js) 负责。它监听 `earth:motion-debug-frame`,用 canvas 绘制归一化骨架点和连线;Browser Camera provider 会额外通过 `earth:motion-debug-video-source` 提供本机 `<video>` 作为调试预览底图,`shared.motionDebugSkeletonOnly` 可切换为只显示骨骼。`停止匹配动作` 通过 `earth:motion-recognition-pause` 暂停 gesture 执行,但继续显示视频和骨架。未匹配动作为红色,匹配后变绿并显示动作名。设置项持久化在 `planet.earth.settings.v2` 的 `shared.motionDebugEnabled`、`shared.motionProvider`、`shared.motionDebugSkeletonOnly` 与 `shared.motionEnabledGestures`,switch、输入源和动作白名单控件都预留 `data-gatekeeper-permission="earth.motion_debug"`。
|
||||
|
||||
Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/motion-browser-provider.js) 的 `recognizeGesture()`,按以下顺序匹配,前者命中即返回:
|
||||
|
||||
@@ -164,6 +170,8 @@ Browser Camera provider 的手势识别管线在 [motion-browser-provider.js](/h
|
||||
|
||||
`tv.js` 管理 `media-panel` 里的直播 / 态势新闻 tab。toolbar 打开或切换 TV/新闻时,会通过 `earth:tv-visibility-change` 和 `earth:tv-tab-change` 回写 Earth 设置:面板可见性仍按 desktop/mobile viewport 存在 `views.<scope>.panelVisibility.media-panel`,当前 tab 存在 `shared.mediaPanelActiveTab`,因此刷新页面后能恢复用户上次打开的直播或新闻状态。`closeTransientMobileOverlays()` 这类临时收起会带 `persist:false`,不会覆盖用户偏好。
|
||||
|
||||
`tv-source-menu.js` 复用 HUD 面板与图例列表样式,使用 popover 显示搜索、滚动列表和固定计数栏。`tv.js` 按 50 条请求 `/api/v1/tv/streams`,维护当前频道缓存;搜索交给后端完整目录,翻页用 `next_offset`。菜单通过取消请求和请求代数屏蔽过期响应,不能让旧搜索覆盖新输入。刷新目录时通过 `selected_id` 恢复不在第一页的频道。
|
||||
|
||||
`brand.js` 管理智能星球 HUD 品牌资源。默认品牌来自静态资源,运行时覆盖值来自 `/api/v1/earth/brand`,上传的图片通过 `/earth-brand-assets/...` 读取。前端必须把 logo/title 图片和文本 fallback 分开处理:图片加载失败时显示文本标题,文本字段为空时使用后端默认值,避免 HUD 品牌区空白。控制台的智能星球内容页负责保存和重置品牌配置,智能星球前端只消费结果。
|
||||
|
||||
`about.js` 管理智能星球设置里的“关于”卡片。默认内容仍保留在前端作为兜底,运行时优先读取 `/api/v1/earth/about`。接口失败或字段缺失时必须回退默认值,避免设置页出现空白。控制台的智能星球内容页提供“关于”tab,保存走 `PUT /api/v1/earth/about`,恢复默认走 `DELETE /api/v1/earth/about`。
|
||||
@@ -203,6 +211,14 @@ TV 预览需要尽量复用 Earth 运行时的直播卡片结构和状态标签
|
||||
|
||||
新闻巡航摘要的未来计划保存在仓库路径 `docs/plans/earth-news-cruise-summary-plan.md`,不作为公开 Docs 页面入口。
|
||||
|
||||
## 渲染热路径
|
||||
|
||||
`cable-batches.js` 将全部海缆线段与全部登陆点分别合批绘制;`cables.js` 继续拥有原始业务对象、拾取、遮挡和选择状态。避免在后续功能中把这些代理对象重新加入逐对象绘制,或把绘制批次当作业务对象返回给详情卡。
|
||||
|
||||
卫星的逐点呼吸在 GPU 中计算。全量 SGP4 位置和初始轨迹计算通过 `satellite-position-worker.js` 离开主线程;主线程只接收位置快照、更新共享交互坐标及绘制缓冲。轨道数学由 `satellite-propagation.js` 同时供 Worker、预测轨道和同步降级路径使用,避免多套公式漂移。数据重载、清空和高度切换必须同步终止旧计算,不能让旧快照恢复已清空的图层。
|
||||
|
||||
`i18n.js` 的 MutationObserver 只翻译新增子树、变动文本或变动属性。局部时钟、状态和下载进度更新不能触发整页语言控件同步;全局语言同步只属于初始化、语言切换或包含新语言控件的子树。
|
||||
|
||||
## 当前样式分层
|
||||
|
||||
Earth 的 CSS 不是一份大样式表,而是分层管理:
|
||||
@@ -338,7 +354,13 @@ AIS 船只图层入口:
|
||||
|
||||
AISStream 的 `PositionReport` 常带实时位置和 `MetaData.ShipName`,但船型通常来自低频 `ShipStaticData.Type`。后端会把 `MetaData.ShipName` 补进船名,并将类型码映射为 Cargo / Tanker / Passenger / Fishing / Military;仍缺失的船型需要等待静态 AIS 消息或后续船舶资料 enrichment,不能在前端凭颜色之外的信息臆造细分类。
|
||||
|
||||
旧 `/api/v1/visualization/geo/vessels` 路由已移除。前端打开船只图层时只应拉取一次全局 `/api/v1/vessels/snapshot`;API 参数里的 bbox 是后端接口约束,Earth 运行时传全球范围,不表示当前镜头视口。`/ws` 的 `vessels` channel 如启用,只作为低频 reload/dirty 提示,不能把每条 AIS delta 直接变成整层重建。后端通过 `diagnostics.source == "vessel_current_state"` 暴露当前状态链路。
|
||||
旧 `/api/v1/visualization/geo/vessels` 路由已移除;初始数据仍通过全球 bbox 的 `/api/v1/vessels/snapshot` 读取。
|
||||
|
||||
AISStream 与 BarentsWatch 共用 `/ws` 的 `vessels` 增量通道。Earth 使用现有 WebSocket 连接订阅 `scope: "global"`,不跟随镜头改变订阅范围。后端按 MMSI 合并一秒内的通知,再读取 `vessel_current_state` 的已确认状态;单帧最多 1000 项,超过时继续发送后续帧,不截断不同船只。原始源消息不能直接覆盖客户端的确认位置。当前状态删除也会产生按 MMSI 的 remove 通知。
|
||||
|
||||
前端将短时间内同一 MMSI 的变更合并为最新值,通过 `Interactable.updateItems()` 原位修改位置和颜色缓冲。航向分桶变更只更新受影响的桶,容量不足才扩容;保留已有 marker 身份、锁定状态和材质。首次进入、重连、删除提示及每分钟巡检仍用 snapshot 校准,但不先清空整个图层。收到有数量上限的快照时,不能把被截断的船只当成删除;快照返回期间到达的较新更新和删除也不能被旧响应覆盖。快照携带查询开始时的 `generated_at`,与实时帧时间一起用于识别缓存旧快照,避免新船或删除状态被回滚。
|
||||
|
||||
`earth_updates` 中船舶普通写入的策略是 `delta`;专用通道连通时不再触发整层重拉。删除或断连后的校准使用 `reload`,并复用现有对象。关闭图层会取消船舶订阅并清空待应用变更。轨迹、冲突详情和审计继续按单船接口读取历史事实。
|
||||
|
||||
新的图层接口族是 `/api/v1/layers/*`,用于把地图渲染数据和聚合面板统计分开。地图层请求必须带 `bbox`、`zoom` 和受控 `limit`,响应会返回 `visible_count`、`returned_count` 和 `diagnostics`,其中 `degraded/truncated/limit_clamped` 用于前端提示降级。右侧聚合统计不要从图层响应累加,应读取 `/api/v1/data-products` 或 `/api/v1/data-products/{product_id}/status`,因为这些统计保持全量/全局口径,不随当前视口变化。
|
||||
|
||||
@@ -374,6 +396,8 @@ AISStream 的 `PositionReport` 常带实时位置和 `MetaData.ShipName`,但
|
||||
|
||||
详情卡里的坐标候选状态由 [info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js) 按 `entityType:entityId` 缓存在模块内存中。用户关闭详情卡或待定位列表后再次打开同一个算力中心 / BGP 观测站,已经采集到的候选和状态文案会恢复;`一键采用` 会优先使用缓存候选,避免重复调用在线地理编码或 LLM factcheck。保存成功后该实体的候选列表会清空为“正在刷新图层”状态,避免旧候选在刷新后继续误导用户。
|
||||
|
||||
批量定位由 `runUnresolvedComputeCenterBatch()` 持有实体上下文队列,不持有面板 DOM;`locationCollectStateCache` 保存候选、进度和保存结果,重建卡片时重新 hydrate。`earth:compute-center-location-batch-change` 同步任务状态与气泡入口,成功条目根据 `savedCandidate` 排除,全部完成后保留 ✓ 入口。状态只存在当前 Earth 页面内存中,刷新、关闭或路由离开会终止未完成队列;已保存坐标仍以后端为准。
|
||||
|
||||
候选行的 `预览 / 保存` 按钮采用单一的事件委托模型:每个候选根(详情卡里的 `[data-collect-cache-key]` 块,或待定位列表里的 `[data-unresolved-item]`)只挂一个 `click` 监听,由 `data-candidate-actions-bound` 幂等标记,不再混用 `pointerup` / `click` 直绑或重复委托。候选对象不再以 JSON 字符串塞进 HTML 属性后再 `JSON.parse`,按钮只携带 `data-candidate-index`,handler 通过 cache-key 在模块内存的 `Map` 里取出原对象,避开 HTML 实体转义对 `&` / `<` / `"` 的破坏。点击 `预览` 会派发 `earth:preview-location-candidate`,由 `main.js` 的 `previewLocationCandidate()` 调用 `showComputeCenterLocationPreview()`:在候选经纬度上挂双层空心呼吸 sprite(视觉参考 BGP 事件 ring),并把视角聚焦到候选坐标;切换到另一个候选会替换为新呼吸圈,保存时立即清除并由 `spawnSavedComputeCenterLocation()` 即时生成正式算力中心交互图标。注意 `main.js` 没有模块级 `earth` 变量,所有 location-save / preview 处理函数必须先 `const earth = getEarth();`,否则会在事件 handler 里抛 `ReferenceError` 被 `.catch` 静默掉,外观上等同于按钮“没有反应”。
|
||||
|
||||
`earth:compute-center-location-saved` 之后的图层校准链路对后台刷新失败保持沉默:`spawnComputeCenterAfterLocationSave()` 已经把 toast 和 locked 状态都给了用户,`refreshComputeCentersAfterLocationSave()` 只在场景就绪时重新拉取后端数据,本身不再吐 `已保存` toast;`handleComputeCenterLocationSaved()` 在 spawn 成功路径让 refresh 静默后台运行,只在 spawn 返回 `null`(场景未就绪)或抛错时才让 refresh 接管成功 toast,refresh 自身报错只走 `console.warn`,绝不冒泡成 `保存失败` 文案——保存请求本身已经成功,刷新失败属于后续同步问题。
|
||||
@@ -386,7 +410,7 @@ asset 图标大小由 `Interactable` 的 `icon.fitSize` 控制。SVG / 图片文
|
||||
|
||||
跨 Interactable 的同坐标关系也在公共层记录,但真实位置必须始终以 `icon_base_position` 为准。缩放、避让、聚合和后续 spiderfy 展开都只能改变屏幕表现,不能写回 `marker.position` 或 `THREE.Points` 里的业务锚点;巡航定位、详情卡、搜索定位和 picking 返回对象都必须落回真实经纬度。多个图标归入同一个经纬度 key 时,公共层只写 `icon_avoidance_*` 元数据,供业务层弱化 halo 或显示聚合提示;真正的低缩放聚合应通过独立 cluster glyph / screen layout 层实现,而不是把对象沿地表切平面挪开。
|
||||
|
||||
`Interactable` 的单点显示只由全局地图缩放决定:170% 及以下强制显示小圆点,超过 170% 显示原图标。cluster 现在由 `cluster.strategy` 决定:`stable-spherical` 使用离散 zoom band 和 3D 球面分桶,BGP、算力中心和 Earth interactable 在同一 band 内旋转或细微缩放时不会重新计算聚合拓扑;`dynamic-screen` 保留屏幕空间聚类,适合船只这类实时高频图层;`none` 关闭聚类。稳定球面聚类的 cluster 圆点刚性落在成员 3D 质心投影上,不参与 2D 避让,避免缩放时被推离真实地理位置。cluster 圆点大小随包含对象数量增长,数量过多时按稳定地理顺序拆成多个较小圆点;数量默认只在 hover tooltip 中显示。这个过程只设置 `icon_cluster_*` 展示元数据和重建渲染 Points,不改变每个 marker 的真实经纬度。
|
||||
`Interactable` 的单点显示只由全局地图缩放决定:170% 及以下强制显示小圆点,超过 170% 显示原图标。cluster 现在由 `cluster.strategy` 决定:`stable-spherical` 使用离散 zoom band 和 3D 球面分桶,BGP、算力中心和 Earth interactable 在同一 band 内旋转或细微缩放时不会重新计算聚合拓扑;`dynamic-screen` 保留屏幕空间聚类;`none` 关闭聚类。船只显式关闭聚类与避让,保证增量更新时不重建聚类拓扑。稳定球面聚类的 cluster 圆点刚性落在成员 3D 质心投影上,不参与 2D 避让,避免缩放时被推离真实地理位置。cluster 圆点大小随包含对象数量增长,数量过多时按稳定地理顺序拆成多个较小圆点;数量默认只在 hover tooltip 中显示。这个过程只设置 `icon_cluster_*` 展示元数据和重建渲染 Points,不改变每个 marker 的真实经纬度。
|
||||
|
||||
接口细节、生命周期和接入示例见:
|
||||
|
||||
@@ -455,6 +479,8 @@ Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/
|
||||
|
||||
也就是说,Earth 设置不是一次性 UI 状态了,而是本地设备级偏好。后续如果再加入新的设置项,应优先接入同一条持久化链,而不是各自散着写 `localStorage`。
|
||||
|
||||
语言偏好例外:Earth 语言不是 `planet.earth.settings.v2` 的私有字段,而是与 Docs/Admin 共用 `planet-locale`。切换语言时,`i18n.js` 会更新 `document.documentElement.lang`、翻译静态和动态插入的 DOM、同步语言按钮状态,并向新闻请求传递当前 locale;新闻模块在语言变化后会重新刷新,避免英文界面复用中文新闻 payload。
|
||||
|
||||
地表 hover 提示由 `controls.js` 持久化为 `shared.surfaceHoverInfoMode`,实际 tooltip 在 `main.js` 的地表 hover 分支组合。`国家` 模式只在命中国家时显示国家信息,海洋区域不显示地表 tooltip;`位置` 模式只显示纬度、经度和地形采样海拔,并清除国家边界 hover;`完整` 模式在陆地显示国家 + 位置,在海洋显示位置。
|
||||
|
||||
卫星真实高度开关由 `controls.js` 持久化,实际渲染状态在 `satellites.js`。开启时,SGP4 得到的真实半径会按对数压缩到当前 Earth 视觉半径范围;关闭时,卫星点、轨迹和预测轨道都回到旧版同层球面。切换时必须刷新卫星位置并清理轨迹缓存,避免同一条轨迹混入两个高度模型。`maxRealAltitudeOffset = 25` 是当前相机和 `earthRadius = 100` 下的视觉上限:它让 GEO / MEO 比 LEO 明显更高,但把最高轨道控制在地球半径外约 25%,避免选择点、红色轨迹和主体地球之间出现过大的空场。
|
||||
|
||||
@@ -149,6 +149,8 @@
|
||||
|
||||
## 海缆与登陆点
|
||||
|
||||
`cable-batches.js` 将海缆线与登陆点分别合批绘制。下表的 Sprite 材质和尺寸仍属于 `cables.js` 中的拾取、选择代理;每帧把颜色、透明度、缩放和可见性同步到样式纹理或实例属性。实际绘制使用 `LineSegments` 和实例化 billboard,沿用原纹理、色彩、层级与球体遮挡规则。
|
||||
|
||||
| 正式名称 | 变量名 | 当前值 | 使用位置 / 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 默认海缆颜色 | `CABLE_COLORS.default` | `0xffff44` | 无数据颜色时使用 |
|
||||
|
||||
@@ -86,6 +86,55 @@ Earth 态势新闻使用 `/api/v1/news/earth-feed` 输出给前端。新闻源
|
||||
|
||||
`GET /api/v1/earth/news-sources` 返回默认或已保存配置。`PUT /api/v1/earth/news-sources` 保存配置并递增 `cache_version`,同时清理进程内 region cache。`POST /api/v1/earth/news-sources/reset` 恢复默认源。`POST /api/v1/earth/news-sources/test` 只测试单个 RSS/Atom/Aggregated 源,不写入新闻表。
|
||||
|
||||
## 手动新闻内容
|
||||
|
||||
手动新闻是内容管理能力,不是 RSS 源配置。Admin 入口是 `智能星球内容 -> 新闻内容`,左栏按 RSS 来源和手动新闻组聚合;进入手动组详情后可以按条添加新闻或上传 JSON 数组批量导入。手动新闻写入 `earth_news_items`,并标记 `feed_type/source_type = manual`;前台展示文案是“手动添加”。它不参与 RSS 连通性测试,也不会进入 RSS 抓取流程。
|
||||
|
||||
手动新闻采用“先展示,再精修”的策略:
|
||||
|
||||
1. 保存后立即写入 `earth_news_items`。
|
||||
2. 没有人工坐标时使用所选区域锚点,`verified=false`。
|
||||
3. 有人工坐标时使用 `location_source=manual_location`,`verified=true`,后续 AI 精修不会覆盖该坐标。
|
||||
4. 创建或重新处理后进入新闻增强队列,后台补清洗、翻译、分类、重要度、Breaking 和目标位置推断。
|
||||
5. 精修完成后更新同一条新闻,并通过 Earth news reload / patch 让前端无感替换。
|
||||
|
||||
后台接口挂在 `/api/v1/earth/news-items`:
|
||||
|
||||
- `GET /earth/news-groups`:返回 RSS 虚拟来源组和手动新闻组。
|
||||
- `POST /earth/news-groups`:新建手动新闻组。
|
||||
- `PUT /earth/news-groups/{group_id}`:重命名手动新闻组,并同步组内新闻 meta。
|
||||
- `GET /earth/news-items`:分页查询 RSS 与手动新闻,支持来源类型、区域、类型和状态过滤。
|
||||
- `POST /earth/news-items`:新增一条手动新闻。
|
||||
- `POST /earth/news-items/import`:上传 JSON 数组批量导入,`group_id` 指定当前手动新闻组。
|
||||
- `PUT /earth/news-items/{id}`:编辑手动新闻;RSS 新闻只读。
|
||||
- `DELETE /earth/news-items/{id}`:删除手动新闻,并触发 Earth 新闻重载。
|
||||
- `POST /earth/news-items/{id}/reprocess`:重新进入清洗、翻译和定位队列。
|
||||
|
||||
JSON 导入首版只支持数组:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"title": "必填标题",
|
||||
"summary": "可选摘要",
|
||||
"content": "可选正文",
|
||||
"url": "https://example.com/story",
|
||||
"source": "手动添加",
|
||||
"region": "global",
|
||||
"published_at": "2026-05-15T03:00:00Z",
|
||||
"category": "business",
|
||||
"tags": ["manual", "analysis"],
|
||||
"location": {
|
||||
"label": "北京市, 中国",
|
||||
"latitude": 39.9057,
|
||||
"longitude": 116.3913
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
重复导入使用稳定 ID 去重,ID 由标题、发布时间、URL 和来源生成,格式为 `manual:{hash}`。同一条手动新闻再次导入会更新原记录,不会重复出现在 EarthFeed。
|
||||
|
||||
## Feed 查询与类型过滤
|
||||
|
||||
星球端和 UE 端统一使用 `GET /api/v1/news/earth-feed` 获取新闻。接口支持服务端过滤,不要求客户端拿全量列表后自行筛选。
|
||||
@@ -95,6 +144,8 @@ Earth 态势新闻使用 `/api/v1/news/earth-feed` 输出给前端。新闻源
|
||||
- `categories`:逗号分隔的新闻类型 key,例如 `business,ecommerce`。全选时可以不传。
|
||||
- `locale`:展示语言,支持 `zh-CN` 和 `en-US`,默认 `zh-CN`。中文 RSS 会以中文原文入库,并由后台补 `en-US`;英文 RSS 则由后台补 `zh-CN`。
|
||||
|
||||
当 `locale=en-US` 时,接口不能回退展示中文原文标题或摘要。中文来源新闻尚未生成 `en-US` 本地化时,`display_title` 和 `display_summary` 保持为空,由 Earth 前端展示英文待处理或空态,避免英文界面混入中文新闻内容。
|
||||
|
||||
示例:
|
||||
|
||||
```http
|
||||
@@ -107,8 +158,23 @@ GET /api/v1/news/earth-feed?region=global&categories=business,ecommerce&locale=z
|
||||
|
||||
Web 星球端的新闻类型按钮只保存当前浏览器的显示偏好;偏好变化后会重新请求接口。UE 端应直接把类型选择拼到 `categories` 参数里,不需要再做主过滤。
|
||||
|
||||
未指定 `sources` 时,服务层会优先选择当前 `locale` 已有可展示标题和摘要的新闻,并从当前启用来源补齐候选后做来源轮转,避免一个来源的最新待处理条目占满默认 12 条。指定 `sources` 时保持精确来源过滤,不做跨来源补齐。数据库查询层只负责区域、类型、来源和排序条件,不包含语言展示策略。
|
||||
|
||||
源测试只证明当前 RSS/Atom/XML 能解析到条目,不等于这些条目已经入库展示。展示链路还会检查区域、类型过滤和数据库新鲜度。保存或重置新闻源会递增配置版本并清理缓存;如果当前启用的 Feed 子项在库里没有近期条目,下一次 `earth-feed` 请求会补抓,避免新启用的 36氪、亿邦被旧 Google News 缓存挡住。
|
||||
|
||||
## 区域均衡与异步精修
|
||||
|
||||
`earth_news_items` 是 EarthFeed 的当前状态表。接口在没有显式 `sources` 过滤时会先按 Breaking、区域、发布时间排序,再做区域轮转,避免亚太或任一高频来源把全球视图和巡航队列全部占满。默认区域顺序是美洲、欧洲、中东与非洲、亚太、全球;未知区域只在已知区域之后参与轮转。区域视图仍遵守“当前区域 + global”的规则,不会把其它区域混进区域面板。
|
||||
|
||||
`items` 和 `cruise_items` 都会进入目标位置与本地化精修队列,但前端不能等 AI 完成后再展示。标题或摘要没有目标语言翻译时,Web Earth 先显示原始 `title / summary`,避免卡片出现“新闻汉化中”而实际内容已经可读。后台完成翻译、分类、Breaking 或目标坐标后,会更新同一条 `earth_news_items` 并通过 Earth news reload / patch 刷新前端。
|
||||
|
||||
目标位置队列使用 Redis Streams 两级队列:
|
||||
|
||||
- `earth_news:target_location:priority`:当前可见 `items` 与 `cruise_items` 的优先精修任务,短 TTL 去重。
|
||||
- `earth_news:target_location:jobs`:普通后台精修任务,长 TTL 去重。
|
||||
|
||||
Worker 总是先处理优先队列,再处理普通队列;pending 消息超过空闲阈值会被 reclaim,单条 AI 任务有超时保护,失败后进入重试或 dead letter。这样即使历史普通队列有大量 backlog,当前打开的欧洲、美洲等区域新闻也不会被长队列饿死。
|
||||
|
||||
## Breaking News 插队
|
||||
|
||||
新闻体系里有三套互不替代的判断:
|
||||
@@ -163,9 +229,11 @@ Admin 入口是 `Earth 内容 -> 新闻源`。界面不是整包 JSON 编辑,
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Admin["Admin: Earth 内容 / 新闻源"] --> Source["Source 配置"]
|
||||
ManualAdmin["Admin: Earth 内容 / 新闻内容"] --> ManualAPI["/api/v1/earth/news-items"]
|
||||
Source --> Feed["Feed 子项"]
|
||||
Feed --> ConfigAPI["/api/v1/earth/news-sources"]
|
||||
ConfigAPI --> Config["SystemSetting: earth_news_sources"]
|
||||
ManualAPI --> Store
|
||||
|
||||
Earth["Earth 新闻面板"] --> NewsAPI["/api/v1/news/earth-feed"]
|
||||
NewsAPI --> Resolver["Source Resolver"]
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
| 0.86 | 海陆基座填充 | `country-boundaries.js` | `landAltitudeOffset = 0.32`; 海洋 `#010609`,陆地 `#080f1b` | 禁用 raycast | 即使国界线关闭,基座地图仍保持可用;半径与基座球拉开以避免远距 z-fighting。 |
|
||||
| 0.96 | 高清 Earth 材质 | `earth.js` | `textureOverlayAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48` | 可见时作为地表拾取目标 | 高清材质始终压过海陆基座填充;半径必须高于海陆基座并与基座球保持足够间距。 |
|
||||
| 1 | 大气辉光和云图 | `earth.js` | 大气 / 云层球 | 不走普通对象选择路径 | 云图由“大气云图”图层开关控制。 |
|
||||
| 1 | 海缆 / 登陆点 | `cables.js` | 海缆线和登陆点都使用 `renderOrder = 1`;半径偏移都为 `0.2`;登陆点是专用 `THREE.Sprite` 黄色扁平球 | 海缆走海缆拾取路径;登陆点 `depthTest: false` 保持球体完整,并用相机到球心的球体遮挡判断避免背面穿透 | 登陆点和海缆同层贴地,避免地表设施层的凌空感。 |
|
||||
| 1 | 海缆 / 登陆点 | `cables.js`, `cable-batches.js` | 海缆全量线段合并为一个 `LineSegments`;登陆点全量使用实例化 billboard;`renderOrder = 1`、半径偏移 `0.2` 保持不变 | 原始 `Line` / `Sprite` 仅作为逐项拾取和选择状态对象,材质不再单独绘制;登陆点仍按球体遮挡判断背面可见性,批量材质 `depthTest: false` | 样式通过线缆样式纹理和登陆点实例属性同步;保留颜色、脉冲、尺寸、显隐与点击语义。 |
|
||||
| 1.2 | 真实地形 | `earth.js`, `terrain.js` | `TERRAIN_CONFIG.baseRadiusOffset` 加地形位移 | 禁用 raycast | 地形压过高清材质;高清材质关闭时临时隐藏,重新开启后恢复原状态。 |
|
||||
| 2.05 | 经纬线 | `earth.js` | `CONFIG.earthRadius + 0.14` | 禁用 raycast | 低透明度显示在高清材质上。 |
|
||||
| 2.2 | 国界线 | `country-boundaries.js` | `lineAltitudeOffset = EARTH_SURFACE_TEXTURE_ALTITUDE_OFFSET = 0.48`;claim 线不再额外抬高 | `depthTest: true`,禁用 raycast | 线层使用独立 line geometry 与 `renderOrder` 控制,但半径与高清材质壳完全一致,避免转动地球时与高清贴图出现视差。 |
|
||||
@@ -37,6 +37,14 @@
|
||||
| 12+ | 卫星锁定 ring、halo、预测轨道 | `satellites.js` | `SATELLITE_CONFIG.overlayRenderOrder` 及偏移;预测轨道使用同一真实高度开关,并固定锁定时刻的地球姿态来绘制闭合惯性轨道;关闭真实高度时回到同层球面 | 卫星覆盖层路径 | 用于选中 / 锁定卫星强调。 |
|
||||
| 98-100 | 太阳 / 月亮 halo 和 sprite | `celestial.js` | 固定 renderOrder | 天体拾取禁用 | 前景天体 sprite。 |
|
||||
|
||||
## 全量绘制与更新约束
|
||||
|
||||
- 合批只改变 GPU 提交方式,不减少卫星、线段或登陆点数量,也不按相机半球裁剪数据。海缆每对相邻顶点都保留,拾取仍返回原始业务对象。
|
||||
- 卫星普通点与背景点保持两个 `Points` 绘制;呼吸动画由顶点着色器使用静态逐点参数和每帧统一时间计算。hover / locked 的隐藏标记仍由原有选择状态控制。
|
||||
- `satellite-position-worker.js` 使用与主线程相同的 Three.js / SGP4 版本,通过 `satellite-propagation.js` 共享轨道、显示高度和 fallback 计算;全量位置及初始轨迹以可转移数组交给主线程。主线程维持一个在途计算,不堆积过期帧;重载、清空和高度模式变更会终止旧 Worker。
|
||||
- Worker 启动失败或超过 `SATELLITE_CONFIG.workerStartupTimeoutMs` 时退回共享的同步计算,避免图层无限等待。计数、数据接口、原有更新周期和轨迹长度不变。
|
||||
- 验证时同时检查全量 draw range、逐项拾取、锁定覆盖层、轨迹、背面遮挡、开关及清空后的资源释放;比较帧耗时必须使用相同数据、相同视角和分辨率。
|
||||
|
||||
## 开关联动
|
||||
|
||||
| 开关 | 行为 |
|
||||
|
||||
@@ -98,6 +98,8 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
|
||||
`StatusText` 是带圆点的指示灯:胶囊背景和边框保持组件原色,只让圆点和文字变成状态色。`Badge` 不带指示灯语义,可以使用同 tone 的浅色背景和边框强化信息层级。
|
||||
|
||||
状态指示器必须完整显示状态词。列表、树形组和详情栏里的状态列应让标题/描述区域收缩或换行,状态 pill 本身使用内容自适应宽度并禁止被 flex/grid 挤压;不要为了紧凑把 `Configured` / `Available` 这类状态裁成省略号。
|
||||
|
||||
| Tone | 颜色变量 | 语义 | 示例 |
|
||||
| --- | --- | --- | --- |
|
||||
| `success` | `--an-success` | 可用、成功、已连接、已启用 | 日志源 `可用`、采集 `成功` |
|
||||
@@ -216,7 +218,31 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
- 颜色优先通过 CSS 变量覆盖,避免在业务组件里硬编码主题色
|
||||
- 适合少量互斥选项,不适合用作长列表、导航菜单或表单下拉
|
||||
|
||||
### 6. `MarkdownRenderer`
|
||||
### 6. 控制台 i18n
|
||||
|
||||
文件:
|
||||
|
||||
- [i18n/index.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/index.ts)
|
||||
- [i18n/locale.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/locale.ts)
|
||||
- [i18n/resources.ts](/home/ray/dev/linkong/planet/frontend/src/i18n/resources.ts)
|
||||
- [LegacyI18nBridge.tsx](/home/ray/dev/linkong/planet/frontend/src/i18n/LegacyI18nBridge.tsx)
|
||||
|
||||
用途:
|
||||
|
||||
- 控制台、认证页和 Docs UI 共用 `zh-CN` / `en-US` 语言状态
|
||||
- 语言偏好保存在 `planet-locale`,同时兼容旧的 `docs-lang`
|
||||
- Docs 请求仍映射到后端现有 `zh` / `en` 文档接口
|
||||
- 控制台侧边栏偏好面板和认证页面板提供语言切换入口
|
||||
|
||||
当前约束:
|
||||
|
||||
- 新增控制台文案优先写入 `resources.ts`,组件使用 `useTranslation()` 或 `useLocale()`
|
||||
- 路由、菜单、搜索索引和通用组件必须使用显式翻译 key
|
||||
- `LegacyI18nBridge` 只作为过渡层,负责 admin/auth 容器内未迁移的精确静态文本和属性
|
||||
- 业务数据、日志原文、API 字段名、provider id、命令和 Markdown 正文不走 legacy 翻译桥
|
||||
- 后续迁移大型业务页时应减少 legacy 字典,而不是继续扩大它
|
||||
|
||||
### 7. `MarkdownRenderer`
|
||||
|
||||
文件:
|
||||
|
||||
@@ -274,17 +300,16 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
|
||||
职责:
|
||||
|
||||
- `/ai` 独立承载 LLM Provider、AI Tool 配置和测试台,不再放在 `/settings` 的系统配置 tabs 中
|
||||
- `模型供应商` tab 管理默认 provider、模型、base URL、provider key、本地 `aiprovider` 代理和连接测试;provider 和模型输入使用可输入组合框,models.dev 目录停更时用户仍可手动填新 provider/model
|
||||
- `工具` tab 先通过下拉菜单选择工具,再管理对应配置;当前包含 WebSearch 和 OCR
|
||||
- `/ai` 独立承载 LLM Provider、AI Tool 配置和 Playground,不再放在 `/settings` 的系统配置分区中
|
||||
- `模型供应商` 分区管理默认 provider、模型、base URL、provider key、本地 `aiprovider` 代理和连接测试;provider 和模型输入使用可输入组合框,models.dev 目录停更时用户仍可手动填新 provider/model
|
||||
- `工具调用` 分区先通过下拉菜单选择工具,再管理对应配置;当前包含 WebSearch 和 OCR
|
||||
- WebSearch 配置包含 provider、搜索 key、base URL、超时、结果数和高级 provider 参数
|
||||
- OCR 配置包含 provider、Base URL、API Key、模型/engine、语言、超时、文件大小上限和输出格式
|
||||
- `测试台` tab 嵌入原 Playground 的真实会话、预设请求和 AI Provider 状态调试
|
||||
- 页面复用 Settings 的单屏 tabs、panel card 和内部滚动样式
|
||||
- `Playground` 分区嵌入原 Playground 的真实会话、预设请求和 AI Provider 状态调试
|
||||
- 页面复用 Settings 的单屏分区、panel card 和内部滚动样式
|
||||
- AI Provider / WebSearch 的连接测试使用 `ConnectionTestInput`,连接器图标固定在 Base URL 输入框末端;WebSearch 未启用时,除开关外的配置项和测试入口都置灰
|
||||
|
||||
旧的 `/settings?tab=ai` 应跳转到 `/ai?tab=providers`。
|
||||
旧的 `/playground` 应跳转到 `/ai?tab=playground`。
|
||||
AI 配置不再挂在 `/settings` 下;`/playground` 应跳转到 `/ai?section=playground`。
|
||||
|
||||
### 3. 业务数据网关
|
||||
|
||||
@@ -342,7 +367,7 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
- endpoint、headers、config 只展示,不在这里编辑。
|
||||
- 需要凭证的采集器提示用户到“采集管理 -> 采集器”维护。
|
||||
|
||||
这个边界很重要:后续不要把自定义数据源编辑、内置 endpoint 覆盖或凭证表单再塞回 `/datasources`。这些配置入口统一放在 `/collection-management?tab=collector_credentials`。
|
||||
这个边界很重要:后续不要把自定义数据源编辑、内置 endpoint 覆盖或凭证表单再塞回 `/datasources`。这些配置入口统一放在 `/collection-management?section=collector_credentials`。
|
||||
|
||||
页面顶部的总进度区域新增 `采集中 N` 标签:
|
||||
|
||||
@@ -355,7 +380,7 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
|
||||
### 采集器设置页
|
||||
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content` 是智能星球内容,`/collection-management` 是采集管理。`collector_credentials` tab 当前在 `/collection-management` 下显示为“采集器”。
|
||||
[Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 会按路由进入三种模式:`/settings` 是系统设置,`/earth-content` 是智能星球内容,`/collection-management` 是采集管理。`collector_credentials` section 当前在 `/collection-management` 下显示为“采集器”。
|
||||
|
||||
`/settings` 的“系统显示”分区包含 `演示模式` 开关。开启后,智能星球的 OOBE 会忽略“已有当前采集数据”和本地“先浏览”临时跳过状态,直接展示初始化引导;该开关仅用于演示/验收流程,不改变数据源、采集队列或智能星球内容资源配置。
|
||||
|
||||
@@ -368,7 +393,7 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
- 不需要凭证的采集器只显示基础配置:endpoint、默认 endpoint、请求头、timeout、retry。
|
||||
- `BarentsWatch AIS` 使用专用凭证表单。
|
||||
|
||||
连接图标使用内联 `PlugConnectIcon`,视觉语义来自 Tabler `plug-connected`。后续如果控制台重写图标体系,应迁移到 Tabler Icons,而不是继续使用 Ant Design 刷新图标表达连接。
|
||||
连接测试入口使用现有 `Button icon="test"` 图标语义。后续如果控制台重写图标体系,应迁移到现有 lucide / Tactile UI 图标体系,而不是用普通刷新图标表达连接。
|
||||
|
||||
`Client Secret` 的表单语义:
|
||||
|
||||
@@ -393,6 +418,7 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
`/earth-content` 复用 [Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/admin/pages/PlainResourcePages.tsx) 的单屏 tab 容器,但页面责任与系统设置分离:
|
||||
|
||||
- `电视直播` 迁移原直播源配置,继续管理 Earth 媒体面板内容源。
|
||||
- `品牌标识` 管理 Earth HUD 的品牌资源。`Logo 地址` 与 `标题图地址` 使用字段内上传控件,上传按钮保持 `TactileButton` 的 primary 样式;图片拖到对应字段时显示低饱和拖拽反应,避免回到旧的全局“选择资产/上传”工具栏。
|
||||
- `国界精度` 管理 Earth 静态国界资产:provider 状态、低精 fallback、高精 manifest/PMTiles、源配置 JSON 和构建动作。
|
||||
- `地球底图`、`图层资源`、`三维素材`、`新闻锚点策略` 是占位页,只显示模块待接入,不造假接口或假数据。
|
||||
|
||||
@@ -421,6 +447,7 @@ Admin 的状态标签统一走 [StatusText](/home/ray/dev/linkong/planet/fronten
|
||||
4. 不要用 `overflow: hidden` 掩盖结构问题
|
||||
5. 不要为了摘要卡完整显示去压缩主工作区
|
||||
6. 自定义滚动条必须是浮层,不得挤压内容宽度
|
||||
7. Admin shell 依赖 root `height: 100%` 高度链,不在根工作区重新写精确 `100vh`
|
||||
|
||||
详细经验见:
|
||||
|
||||
|
||||
@@ -210,6 +210,10 @@
|
||||
3. 真正滚动节点是否明确 `overflow: auto`
|
||||
4. 中间包装层是否偷偷改了布局语义
|
||||
|
||||
当前 Admin 和 Docs 根 shell 依赖 `html` / `body` / `#root` 的 `height: 100%`
|
||||
链路。不要在这些嵌入式工作区根容器上重新写精确 `100vh` / `100vw`;
|
||||
弹窗、浮层和窄屏安全边界可以继续使用 `calc(100vh - ...)` 作为最大尺寸约束。
|
||||
|
||||
### 5. UI 状态和显示状态不同步
|
||||
|
||||
Earth 相关改动里反复出现:
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
1. 打开 `http://localhost:3000/login`,点击表单下方"注册账户"。
|
||||
2. 在 `/register` 填写:
|
||||
- **用户名**:3–50 位字符,登录时使用
|
||||
- **邮箱**:用于接收验证码,可在账户设置中修改
|
||||
- **邮箱**:用于接收验证码;当前版本如需修改邮箱,请联系管理员在用户管理中维护
|
||||
- **密码**:至少 8 位
|
||||
3. 提交后会跳到验证页,已将 6 位验证码发到你的邮箱。10 分钟内有效。
|
||||
4. 输入验证码,点击"验证并登录"。验证通过后系统会自动写入登录态并跳到控制台。
|
||||
@@ -52,23 +52,23 @@
|
||||
3. 收到验证码后,在下一步填入验证码 + 新密码(至少 8 位),点击"重置密码"。
|
||||
4. 系统会跳回 `/login`,用新密码登录即可。
|
||||
|
||||
## 账户设置
|
||||
## 账户区与退出
|
||||
|
||||
控制台右上角点击你的用户名进入账户设置,可以:
|
||||
控制台左侧底部的账户区会显示当前用户名、版本号和主题切换。当前版本还没有登录后的自助账户设置页:
|
||||
|
||||
- 修改密码:输入当前密码 + 新密码
|
||||
- 修改邮箱:输入新邮箱后系统会发验证码到新地址,验证通过后才生效
|
||||
- 查看权限组:列出你目前拥有的 Gatekeeper 权限组(`docs_user` / `docs_developer` / `docs_admin`)
|
||||
- 登出:清除当前会话
|
||||
- 忘记密码或需要重置密码时,使用 `/forgot-password` 邮件验证码流程
|
||||
- 邮箱、角色和 Gatekeeper 权限组由管理员在 `/users` 维护
|
||||
- 点击账户区的退出图标会清除当前会话并返回登录页
|
||||
|
||||
## 控制台总览
|
||||
|
||||
控制台 `http://localhost:3000/admin` 使用 React + Ant Design,左侧菜单按工作域组织。
|
||||
控制台 `http://localhost:3000/admin` 使用 React + Tactile UI / Radix 基础组件和 lucide 图标,左侧菜单按工作域组织。
|
||||
|
||||
| 页面 | 路由 | 用途 |
|
||||
| --- | --- | --- |
|
||||
| 仪表盘 | `/admin` | 系统概览 |
|
||||
| 智能星球 | `/earth` | 跳到公开智能星球页面 |
|
||||
| 文档 | `/docs` | 打开文档站并按 Gatekeeper 权限查看可见文档 |
|
||||
| 数据源 | `/datasources` | 数据源目录、触发采集 |
|
||||
| 采集数据 | `/data` | 已落库的数据 |
|
||||
| BGP 观测 | `/bgp` | BGP 专题观测 |
|
||||
@@ -86,7 +86,7 @@
|
||||
|
||||
## 配置数据采集器
|
||||
|
||||
`/collection-management?tab=collector_credentials` 是"采集器"页。这里统一维护所有采集器的连接配置,不仅是凭证。旧链接 `/settings?tab=collector_credentials` 会自动跳转到这个入口;数据源目录仍保留在 `/datasources`。
|
||||
`/collection-management?section=collector_credentials` 是"采集器"页。这里统一维护所有采集器的连接配置,不仅是凭证;数据源目录仍保留在 `/datasources`。
|
||||
|
||||
操作步骤:
|
||||
|
||||
@@ -126,11 +126,14 @@
|
||||
|
||||
### AISStream 实时船舶
|
||||
|
||||
在智能星球打开船只图层后,AISStream 与 BarentsWatch 的后续位置会自动更新。更新时会保留已选中的船只;短暂断线重连后会自动校准,无需反复关闭、开启图层。
|
||||
|
||||
|
||||
`AISStream 实时船舶` 是全球 AIS WebSocket 采集器。连接测试通过只说明 API Key 和 endpoint 格式可用;真正的全球船只数据来自后台 `aisstream_vessels` collector 长连接运行并写入 `ais_raw_observations`。
|
||||
|
||||
操作步骤:
|
||||
|
||||
1. `/collection-management?tab=collector_credentials` 选择 `AISStream 实时船舶 : aisstream_vessels`
|
||||
1. `/collection-management?section=collector_credentials` 选择 `AISStream 实时船舶 : aisstream_vessels`
|
||||
2. 在 `AISStream 凭证` 填入 API Key
|
||||
3. Endpoint 保持默认 `wss://stream.aisstream.io/v0/stream`
|
||||
4. 点击插头图标进行连接测试,确认显示 `可用`
|
||||
@@ -144,11 +147,12 @@
|
||||
|
||||
## 配置 AI 凭证
|
||||
|
||||
`/ai?tab=providers` 是 AI 模型管理入口。包含三个核心子 tab:
|
||||
`/ai?section=integrations` 是 AI 模型管理入口。主要分区包括:
|
||||
|
||||
- `模型供应商`:默认 LLM provider、模型、Base URL、API Key、本地 `aiprovider` 代理和连接测试
|
||||
- `工具`:通过下拉菜单选择具体工具,当前支持 WebSearch 和 OCR
|
||||
- `工具调用`:通过下拉菜单选择具体工具,当前支持 WebSearch 和 OCR
|
||||
- `提示词`:通过功能入口下拉菜单选择新闻汉化、告警研判、BGP 简报等 LLM 任务,手动调整提示词或重置为缺省
|
||||
- `Playground`:真实会话、预设请求和 AI Provider 状态调试
|
||||
|
||||
### 模型供应商
|
||||
|
||||
@@ -162,7 +166,11 @@ provider 和模型既可选预设也可直接输入自定义 id/name。常用字
|
||||
- Max Tokens、Anthropic Version:可保持默认
|
||||
- Timeout / Retry:超时和重试次数
|
||||
|
||||
Base URL 输入框尾端的插头图标会触发连接测试。测试通过会显示当前模型返回的简短回复。
|
||||
Base URL 输入框尾端的插头图标会触发轻量连接测试:检查代理和模型目录,并确认当前模型出现在目录中。通过后可到 Playground 发起实际生成,核对账号权限和模型回复;目录检查本身不执行生成调用。
|
||||
|
||||
点击右上角的刷新图标可更新“可选模型”。成功后目录会保存,重新打开页面仍可使用;有发布日期的目录按新到旧排列。刷新保留当前模型、密钥、地址和未保存的修改。点击一个可选模型后,再点“保存”才会改变实际使用的模型。刷新失败时会显示错误并保留上次目录。
|
||||
|
||||
刷新使用当前表单中的基础地址、协议和 API Key,因此可以先填写新配置再刷新,无需提前保存。需要凭证的供应商必须填写对应地域的 Key;Ollama 的空目录表示尚未安装模型。选择已知协议的模型时,协议适配会同步到表单;自定义 OpenAI 网关保留手动选择的协议。
|
||||
|
||||
### 工具
|
||||
|
||||
@@ -173,7 +181,7 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
|
||||
|
||||
选择功能入口后,页面会显示当前提示词、是否已自定义、缺省版本和重置按钮。保存只影响该功能入口;重置会恢复当前发布包中的缺省提示词。业务事实、上下文和输出 schema 仍由后端按功能入口自动传入。
|
||||
|
||||
旧链接 `/settings?tab=ai` 会跳到 `/ai?tab=providers`。
|
||||
旧的 AI 配置入口不再放在系统设置里;`/playground` 会跳到 `/ai?section=playground`。
|
||||
|
||||
## 系统设置
|
||||
|
||||
@@ -190,9 +198,10 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
|
||||
|
||||
`/earth-content` 位于控制台“运维与配置”下,面向智能星球前端体验资源:
|
||||
|
||||
- **品牌资源**:维护智能星球 HUD 使用的 logo、标题图、标题文本、副标题和描述;上传的图片会保存为智能星球品牌资产并立即供智能星球页面读取。
|
||||
- **品牌资源**:维护智能星球 HUD 使用的 logo、标题图、标题文本、副标题和描述。`Logo 地址` 和 `标题图地址` 字段内各有独立的“上传”按钮,也可以把图片直接拖到对应字段;上传成功后字段会写入新的资产地址,保存品牌配置后供智能星球页面读取。
|
||||
- **关于**:维护智能星球设置面板里的关于卡片,包括 logo、眉标、标题、版本、描述和元信息。
|
||||
- **电视直播**:维护智能星球媒体面板里的直播源。
|
||||
- **新闻内容**:按 RSS 来源和手动新闻组查看新闻。RSS 新闻保持只读;手动新闻组可以新增、批量导入 JSON、编辑、删除和重新处理。
|
||||
- **国界精度**:查看当前国界 provider、低精 fallback、高精 PMTiles/manifest 状态,编辑本机源配置并手动构建。
|
||||
- **地球底图**、**图层资源**、**三维素材**、**新闻锚点策略**:目前是待接入占位页,不展示假数据。
|
||||
|
||||
@@ -243,7 +252,7 @@ Base URL 输入框尾端的插头图标会触发连接测试。测试通过会
|
||||
|
||||
## AI 测试台
|
||||
|
||||
`/ai?tab=playground` 用于真实分析链路调试。可以:
|
||||
`/ai?section=playground` 用于真实分析链路调试。可以:
|
||||
|
||||
- 选择当前 provider
|
||||
- 用预设请求或自定义 prompt 触发分析
|
||||
@@ -279,6 +288,10 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
支持查找海缆、登陆点、卫星、算力中心、BGP 事件、BGP 观测站。结果可快速定位并打开详情。
|
||||
|
||||
### 新闻直播源选择
|
||||
|
||||
打开媒体面板的直播页,点击当前频道可展开搜索菜单。搜索会检索完整频道库;列表每次加载 50 个频道,滚动到底部自动追加。底部固定显示已加载数量和搜索结果总数,加载失败时可在列表下方重试。默认新闻直播源为 Al Jazeera Mubasher(半岛电视台),使用 HLS 播放地址;其他频道仍取决于各自的播放服务是否可用。
|
||||
|
||||
### 位置候选采集
|
||||
|
||||
算力中心和 BGP 观测站详情卡支持自动采集坐标候选。点击对象后用"自动采集坐标候选"或"重新自动采集坐标"按钮,后端会从源坐标、开放组织注册 API 和在线地理编码中整理候选;常规来源没有候选时使用当前默认 AI Provider 做 LLM factcheck 兜底。BGP 观测站的已存储位置只用于补齐查询上下文,不会作为候选直接返回。
|
||||
@@ -295,9 +308,11 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
一键定位用于批量处理算力中心待定位队列。它会从列表顶部开始采用最高置信候选;仍没有事实依据的记录会保留在队列中。未开启 WebSearch 时,单个定位和一键定位会置灰,因为位置核验依赖事实查询。
|
||||
|
||||
在当前智能星球页面内,关闭候选面板、查看其他对象或切换浏览器标签页不会取消队列;返回候选列表后可继续查看进度和结果。已保存条目不会再次采集,全部完成后仍可通过图层旁的 ✓ 入口查看结果。刷新、关闭页面或跳转离开智能星球会中断尚未完成的队列,已经保存的坐标会保留。
|
||||
|
||||
### 设置
|
||||
|
||||
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
|
||||
设置面板按分类组织:运行、显示、面板、动捕、快捷键、系统。里面包含旋转模式 / 巡航模式 / 动捕模式、巡航模块(BGP/新闻/算力中心/船只/海缆/卫星)、视图设置(卫星显示风格、悬停提示、卫星呼吸闪烁、真实卫星高度、轨迹显示、低缩放圆点、日夜模式、面板显示开关)、动捕调试模式 / 输入源 / 只显示骨骼 / 识别动作白名单、快捷键启用与改键、地球默认大小、地形透明度、重置设置。
|
||||
|
||||
新闻类型使用与巡航模块一致的标签选择器,只筛选当前浏览器里的新闻面板和新闻巡航条目,不影响图层、TV、数据点、底图、边界、采集任务或后台新闻源配置。
|
||||
|
||||
@@ -335,6 +350,8 @@ AIS 船只图例按船型显示颜色:货轮、油轮、客船、渔船、军
|
||||
|
||||
两种模式都不会把摄像头帧或实时手势发到云端,也不会复用新闻/RSS 聚合接口。
|
||||
|
||||
“识别动作”可以单独关闭旋转、缩放、焦点切换、图层切换或确认手势。浏览器端会忽略未勾选动作;使用 Motion Agent 时,同一白名单也会通过控制协议同步给 Agent。
|
||||
|
||||
手势语义:
|
||||
|
||||
| 手势事件 | 作用 |
|
||||
|
||||
@@ -28,6 +28,9 @@
|
||||
| AI Provider | AI Provider | 服务名,保留英文 |
|
||||
| tool | 工具 | Web Search、OCR 等工具配置 |
|
||||
| Playground | Playground | 交互调试入口,保留英文 |
|
||||
| branding | 品牌标识 | `/earth-content` 中的 Earth HUD 品牌配置分区 |
|
||||
| brand assets | 品牌资源 | Logo、标题图和相关 HUD 文案资源 |
|
||||
| title image | 标题图 | Earth HUD 标题图片,不写作“标题图片地址”以外的混合名 |
|
||||
|
||||
## 数据类型
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
## 背景
|
||||
|
||||
日常启动使用 `zsh ./planet.sh start --non-motion-agent`;新环境首次准备才需要 `init`。排查耗时时,应区分首次依赖下载、容器就绪和应用初始化,结合阶段日志时间判断。
|
||||
|
||||
当前启动流程在准备 AI Provider 镜像前验证后端实际数据库连接,端口映射缺失时保留数据卷重建一次。后端进程退出,或 Uvicorn 日志出现应用初始化失败、ASGI 加载失败、导入或语法错误时,会立即停止等待和重复启动;正常的慢启动仍保留原有等待预算。AI Provider 直接探测宿主机 `/health`,无需再等待 Docker 周期性健康检查首次运行。
|
||||
|
||||
`planet.sh` 管理所有服务的启动/停止/重启。原有实现存在以下问题:
|
||||
|
||||
1. AI Provider 每次都重新构建(即使代码未变)
|
||||
@@ -37,27 +41,9 @@ write_ai_provider_build_stamp() {
|
||||
}
|
||||
```
|
||||
|
||||
### fingerprint 计算提速
|
||||
### fingerprint 检查范围
|
||||
|
||||
原实现对整个 `aiprovider/` 打 tar 包再算 SHA,大目录下耗时可达数秒。改为 `find + stat`(只读文件元信息,不读内容):
|
||||
|
||||
```bash
|
||||
compute_ai_provider_build_fingerprint() {
|
||||
find aiprovider \
|
||||
-type f \
|
||||
! -path '*/__pycache__/*' \
|
||||
! -name '.env' \
|
||||
! -name '.env.*' \
|
||||
! -name '*.pyc' \
|
||||
! -name '*.pyo' \
|
||||
| LC_ALL=C sort \
|
||||
| xargs -r stat --format="%Y %s %n" 2>/dev/null
|
||||
sha256sum docker-compose.yml docker-compose.simple.yml 2>/dev/null
|
||||
python3 "$SCRIPT_DIR/scripts/compute_aiprovider_dependency_fingerprint.py" 2>/dev/null
|
||||
}
|
||||
```
|
||||
|
||||
速度提升约 10 倍(大量小文件场景),误报率相同(mtime+size 变化 ≡ 文件被修改)。
|
||||
当前指纹使用内容 SHA,覆盖 `aiprovider/`、Dockerfile、根依赖清单与锁文件,以及代理服务相关依赖信息;不会遍历前端资源或下载数据。`.env` 配置不参与镜像内容指纹。构建标记只用于判断是否需要构建,Dockerfile 中的指纹标签位于依赖安装之后,以保留前置依赖层缓存。
|
||||
|
||||
`.env` 和 `.env.*` 被排除在 fingerprint 外。它们属于运行期配置,不应该因为修改模型、密钥或 Base URL 触发镜像重建。
|
||||
|
||||
@@ -85,13 +71,15 @@ Dockerfile 也从全仓复制改为只复制 AI Provider 代码:
|
||||
```dockerfile
|
||||
COPY pyproject.toml uv.lock /app/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv sync --frozen --no-dev
|
||||
uv sync --frozen --only-group aiprovider
|
||||
|
||||
COPY aiprovider /app/aiprovider
|
||||
```
|
||||
|
||||
`uv sync` 使用 BuildKit cache mount 后,首次构建仍可能受网络影响;后续构建会复用 `/root/.cache/uv`,依赖下载不再重复从零开始。
|
||||
|
||||
`aiprovider` 依赖组在根 `pyproject.toml` 定义,并由同一份 `uv.lock` 锁定,只安装 FastAPI、HTTP 客户端、配置读取和 ASGI 服务所需依赖。镜像不安装后端采集或 OpenCV / MediaPipe 动捕依赖。构建指纹标签放在依赖安装和代码复制之后,指纹改变不会单独使依赖层缓存失效。容器直接启动已安装的 `.venv/bin/python`,运行时不再执行 `uv sync`。修改代理服务的依赖时,应同步更新该依赖组和锁文件,并验证镜像导入及 `/health`。
|
||||
|
||||
### 运行期配置来源
|
||||
|
||||
`planet.sh` 启动 AI Provider 前会生成受当前用户保护的运行期 env-file,并把它传给 Compose 或手动 `docker run` fallback。默认路径位于 `${XDG_STATE_HOME:-$HOME/.local/state}/planet/aiprovider_runtime.env`。配置优先来自:
|
||||
@@ -119,7 +107,7 @@ fingerprint 一致时不执行 `docker compose build`,而是:
|
||||
docker start planet_aiprovider # 启动已存在的容器,几秒内完成
|
||||
```
|
||||
|
||||
`docker stop` 停容器,不删镜像;`cleanup_exit_containers` 删已退出容器,不删镜像。下次 `docker start` 会从现有镜像直接创建并启动容器。
|
||||
`docker stop` 停容器,不删镜像。`start` 和 `restart` 保留已停止的容器,不再扫描删除全机已退出容器;未变化的 AI Provider 可直接复用,数据库仍由 Compose 同步配置。只有需要更新镜像或容器配置时才按原有流程重建。
|
||||
|
||||
## 问题二:杀端口速度慢
|
||||
|
||||
@@ -258,22 +246,27 @@ HTTP 健康检查统一使用 `curl -fsS --max-time`。因此 `/health` 返回 4
|
||||
|
||||
前端生产形态是 `vite build` 生成静态资源,再由 nginx/Caddy 等 HTTP 服务器托管。不要在生产中使用 `bun run dev` 或 `vite preview`。当前不维护 Webpack 双构建链;如果未来需要评估更企业化的构建生态,优先做 Rsbuild/Rspack spike。Electron 仅在正式目标变成离线桌面软件时再单独评估。
|
||||
|
||||
## Motion Agent 可选启动
|
||||
## Motion Agent 默认启动
|
||||
|
||||
`planet.sh` 现在可以管理本地动作捕捉 Agent,但默认不会启动它,避免普通开发机因为没有摄像头、OpenCV 或 MediaPipe 而影响后端/前端启动。
|
||||
`planet.sh` 现在默认随 `start` 和全量 `restart` 启动本地 Motion Agent。这样星球端、UE 或调试客户端可以直接连接 `ws://127.0.0.1:8765/ws/gestures`。如果当前机器没有可用摄像头,默认隐式启动会降级为 dry-run 协议服务,不会阻断后端/前端启动;只有显式传入 `--motion-agent`、摄像头 index、摄像头 URL 或 WSL USB 参数时,live 模式缺摄像头才会硬失败。
|
||||
|
||||
启动方式:
|
||||
如果本次不需要 Motion Agent:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent
|
||||
./planet.sh start --non-motion-agent
|
||||
./planet.sh restart --non-motion-agent
|
||||
```
|
||||
|
||||
常用参数:
|
||||
|
||||
- `--motion-agent` / `-m`:随本次启动或重启拉起 Motion Agent。
|
||||
- `--non-motion-agent`:本次启动或全量重启不拉起 Motion Agent。
|
||||
- `--motion-agent` / `-m`:显式要求本次启动或重启拉起 Motion Agent;此时 live 摄像头失败会作为错误反馈。
|
||||
- `--motion-agent-port <端口>`:覆盖默认 WebSocket 端口 `8765`。
|
||||
- `--motion-agent-mode <模式>`:指定输入模式,可选 `auto`、`single`、`dual_redundant`、`single_fallback`、`calibrated_3d`;`dual` 作为兼容别名会进入双路冗余。
|
||||
- `--motion-agent-camera-indexes <indexes>`:覆盖自动发现的摄像头 index,例如 `0` 或 `0,1`。也可以用环境变量 `MOTION_AGENT_CAMERA_INDEXES=0,1`。
|
||||
- `--motion-agent-camera-urls <urls>`:使用 RTSP/HTTP 摄像头流,适合 WSL、手机摄像头或网络摄像头。也可以用环境变量 `MOTION_AGENT_CAMERA_URLS=...`。
|
||||
- `--motion-agent-wsl-usbipd`:在 WSL 中尝试通过 `usbipd-win` 自动把唯一的 Windows USB 摄像头透传到 Linux。
|
||||
- `--motion-agent-wsl-usbipd-busid <BUSID>`:在 WSL 中指定 `usbipd list` 里的摄像头 BUSID 后透传,适合多摄像头设备。
|
||||
- `--motion-agent-dry-run`:不打开摄像头、不加载 CV 依赖,只启动协议服务,适合调试 Web 端连接。
|
||||
|
||||
非 dry-run 的 live 模式会在启动前检查 `mediapipe` 和 `opencv-python`。如果当前 `.venv` 缺包,脚本会自动执行:
|
||||
@@ -282,13 +275,17 @@ HTTP 健康检查统一使用 `curl -fsS --max-time`。因此 `/health` 返回 4
|
||||
uv add mediapipe opencv-python
|
||||
```
|
||||
|
||||
如需禁止启动时自动安装,可设置:
|
||||
如需禁止启动时自动安装 Python CV 依赖,可设置:
|
||||
|
||||
```bash
|
||||
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start --motion-agent
|
||||
PLANET_MOTION_AGENT_AUTO_INSTALL=0 ./planet.sh start
|
||||
```
|
||||
|
||||
live 模式会自动寻找 `/dev/video*`,优先取前两个 index 传给 Motion Agent。在 WSL 中,Windows 摄像头通常不会自动出现在 `/dev/video*`。可先用下面命令看设备:
|
||||
`./planet.sh init` 会在 WSL 中预检查 `usbipd-win`。如果没有 `usbipd.exe`,脚本会先尝试 `winget install -e --id dorssel.usbipd-win`,失败后复用仓库内置的 dorssel.usbipd-win MSI fallback;如果缓存缺失或需要其他架构版本,再下载 MSI 并请求管理员 PowerShell 安装。该步骤是 best-effort:失败会提示后续处理方式,但不会阻断普通初始化。可通过 `./planet.sh init --non-motion-agent` 跳过该预检。
|
||||
|
||||
live 模式会自动寻找 `/dev/video*`,并优先用 OpenCV 实测过滤出真正能打开并读帧的 index,再传给 Motion Agent。在 WSL/USB 摄像头场景中,一个摄像头可能暴露多个 `/dev/video*` 节点,其中部分是 metadata 或非采集节点,脚本会跳过这类不可读 index。在 WSL 中,Windows 摄像头通常不会自动出现在 `/dev/video*`。可先用下面命令看设备:
|
||||
|
||||
默认 live 采集使用低延迟参数:`640x360` 输入、约 `15Hz` 识别事件;worker 内部用 latest-frame 读帧线程,只保留每路摄像头的最新帧,避免 MediaPipe 慢帧时继续排队识别旧画面。骨架调试流默认关闭,只在星球端打开动捕调试面板时按约 `8Hz` 推送,避免日常手势控制被调试数据拖慢。状态事件会同时上报采集 FPS 与识别 FPS,方便区分摄像头掉帧和识别耗时。
|
||||
|
||||
```bash
|
||||
ls /dev/video*
|
||||
@@ -306,20 +303,42 @@ WSL 下更通用的方式是把手机摄像头或网络摄像头以 RTSP/HTTP
|
||||
./planet.sh start --motion-agent --motion-agent-camera-urls http://192.168.1.20:8080/video
|
||||
```
|
||||
|
||||
如果 WSL 中没有发现 `/dev/video*`,且没有提供 `--motion-agent-camera-urls`,脚本会停止 live 启动并提示处理方式,不会自动降级为 dry-run。可选处理:
|
||||
如果希望直接使用 Windows USB 摄像头,可以让脚本调用 `usbipd-win` 透传。该能力是显式开启的,因为摄像头附加到 WSL 期间通常会从 Windows 应用中暂时断开。
|
||||
|
||||
只有一个摄像头时:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
|
||||
```
|
||||
|
||||
多个摄像头时,先查看 BUSID,再指定设备:
|
||||
|
||||
```bash
|
||||
usbipd.exe list
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd-busid 3-2
|
||||
```
|
||||
|
||||
如果 `usbipd attach` 提示设备未共享或未绑定,脚本会尝试弹出 Windows 管理员 PowerShell 自动执行 `usbipd bind`,然后重试 attach。若 UAC 被取消或自动 bind 失败,可在 Windows 管理员 PowerShell 中手动执行:
|
||||
|
||||
```powershell
|
||||
usbipd bind --busid 3-2
|
||||
usbipd attach --wsl --busid 3-2
|
||||
```
|
||||
|
||||
如果 WSL 中没有发现 `/dev/video*`,且没有提供 `--motion-agent-camera-urls`,默认隐式启动会降级为 dry-run。显式 live 启动会停止并提示处理方式。可选处理:
|
||||
|
||||
```bash
|
||||
./planet.sh start --motion-agent --motion-agent-camera-urls http://<手机IP>:8080/video
|
||||
./planet.sh start --motion-agent --motion-agent-wsl-usbipd
|
||||
./planet.sh start --motion-agent --motion-agent-dry-run
|
||||
```
|
||||
|
||||
只有显式设置 `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1` 时,WSL 无摄像头才会自动降级。
|
||||
显式 live 启动时,只有设置 `PLANET_MOTION_AGENT_WSL_ALLOW_DRY_RUN_FALLBACK=1`,WSL 无摄像头才会自动降级。
|
||||
|
||||
也可以用环境变量启用:
|
||||
`--non-motion-agent` 是命令级跳过入口。环境变量仍可调整服务启动方式:
|
||||
|
||||
```bash
|
||||
PLANET_START_MOTION_AGENT=1 ./planet.sh start
|
||||
MOTION_AGENT_DRY_RUN=1 PLANET_START_MOTION_AGENT=1 ./planet.sh start
|
||||
MOTION_AGENT_DRY_RUN=1 ./planet.sh start
|
||||
```
|
||||
|
||||
日志入口:
|
||||
|
||||
@@ -2,6 +2,48 @@
|
||||
|
||||
这份手册面向部署、值班和二次开发的运维人员。客户面向的 UI 使用流程见 [智能星球使用手册](/home/ray/dev/linkong/planet/docs/technical/zh/manual.md),本手册只覆盖 shell、Docker、日志、环境变量和故障排查。
|
||||
|
||||
## Docker 初始化与访问权限
|
||||
|
||||
新机器应先执行初始化,再启动应用服务:
|
||||
|
||||
```bash
|
||||
zsh ./planet.sh init --non-motion-agent && zsh ./planet.sh start --non-motion-agent
|
||||
```
|
||||
|
||||
脚本入口仍需先安装 `zsh`、`curl`,并保证软件源可访问。`init` 在同步 Python 和前端依赖之前准备 Docker:
|
||||
|
||||
- 已有可用的 Docker、Compose v2 和 Buildx(至少 0.17.0)时直接复用。
|
||||
- Ubuntu / Ubuntu WSL 缺少依赖时,通过 apt 安装 `docker.io`、`docker-compose-v2`、`docker-buildx` 中缺失的部分。若已安装 Docker CE CLI,则使用已配置的 Docker CE 软件源和对应插件包,避免混用软件包系列。
|
||||
- 本地 Docker daemon 未运行时,确认 `docker.service` 存在后启用并启动它。WSL 必须启用 systemd;如果服务管理不可用,脚本会在 Docker 准备阶段明确报错。
|
||||
- 当前用户不能读写 Docker socket 时,检查并补装提供 `usermod` 的 `passwd` 包,将用户加入 `docker` 组。该组拥有管理本机 Docker 的高权限。脚本使用 `sudo` 以原用户身份刷新组权限并继续原命令,保留参数,不依赖 `sg`,也不会把应用进程改为 root 用户运行。
|
||||
|
||||
需要提权时,脚本会在前台请求 sudo 认证。普通用户缺少 sudo、认证失败、软件源不可用或安装后版本仍不满足要求时,初始化会停止并报告具体原因。
|
||||
|
||||
同一旧终端随后执行 `planet.sh start` 等命令时,也会检测已加入但尚未生效的 Docker 组权限并刷新。若要在终端直接使用 `docker`,重新打开 Ubuntu 会话即可。
|
||||
|
||||
Docker Desktop 已存在但 WSL 集成不可用时,脚本提示启动 Desktop 并启用当前发行版的 WSL Integration。已有远程或 rootless endpoint 无法连接时,提示检查当前环境;这些情况不会自动安装另一套本地引擎。其他操作系统的自动安装暂未支持。
|
||||
|
||||
安装逻辑由 `planet.sh` 调用 `scripts/lib/docker-bootstrap.zsh`;缺少 CLI、没有服务单元、socket 权限不足和 daemon 未启动会分别诊断。仅在确认 `docker.socket` 单元存在时才给出启动该单元的建议。验证准备结果可执行:
|
||||
|
||||
```bash
|
||||
docker info
|
||||
docker compose version
|
||||
docker buildx version
|
||||
```
|
||||
|
||||
## 数据库初始化与连接检查
|
||||
|
||||
`init` 和 `start` 会先通过 Compose 同步 PostgreSQL / Redis 容器配置,包括已有容器的端口映射;仅执行 `docker start` 无法应用配置变化。Compose 同步失败时会保留具体错误,例如端口被占用,不会继续复用旧容器并报告成功。
|
||||
|
||||
容器内部的 `pg_isready` 只检查服务是否接受连接,不能证明宿主机上的后端使用正确地址和密码。容器健康后,`init` 和后端启动流程通过 `scripts/check_database_connection.py` 读取与后端相同的有效 `DATABASE_URL`,检查本地 PostgreSQL 的实际发布端口并执行只读 `SELECT 1`。启动流程在准备 AI Provider 镜像之前完成此检查;失败会立即停止。`init` 通过检查后才创建表和默认数据。
|
||||
|
||||
- 如果本地实际端口映射仍缺失或不匹配,脚本会保留数据卷,按 Compose 配置重建一次 PostgreSQL 并重新检查;再次失败就停止。
|
||||
- 认证、库名或网络错误会在建表前停止,诊断只显示目标主机、端口和库名,不输出密码、完整连接串或驱动异常原文。
|
||||
- 进程环境变量中的 `DATABASE_URL` 优先于 `backend/.env`。单独修改 `POSTGRES_PASSWORD` 不会自动更新连接串,也不会改变已有数据卷内的密码。已有环境文件会保留,需要核对其有效配置。
|
||||
- 显式配置的外部数据库不要求本地容器端口匹配;host 网络模式也不要求发布端口,两者仍须通过实际连接检查。
|
||||
|
||||
出现 `port is already allocated` 或 `address already in use` 时,检查 `docker ps` 的端口信息和 `ss -ltnp '( sport = :5432 )'`;WSL 镜像网络下还需检查 Windows 侧监听。初始化不会为了占用数据库端口而自动结束其他数据库服务,也不会删除数据卷或重设密码。
|
||||
|
||||
## 首次启动
|
||||
|
||||
```bash
|
||||
|
||||
@@ -39,7 +39,7 @@ flowchart TB
|
||||
VesselRaw["AIS / BarentsWatch"] --> VesselDerived["vessel_static / vessel_position"]
|
||||
VesselDerived --> VesselLayer["vessels 图层"]
|
||||
Interactables["earth_interactables"] --> InteractableLayer["interactables 图层"]
|
||||
NewsRaw["RSS / Live / News"] --> NewsItems["earth_news_items"]
|
||||
NewsRaw["RSS / 手动新闻 / Live"] --> NewsItems["earth_news_items"]
|
||||
NewsItems --> NewsLayer["news / media 图层"]
|
||||
```
|
||||
|
||||
@@ -51,7 +51,7 @@ flowchart TB
|
||||
| BGP 态势 | 展示观测站、异常事件、路由事件和区域态势 | `ris_live_bgp`、`bgpstream_bgp`、prefix geography sources | `bgp_observations`、`bgp_anomalies`、`bgp_incidents`、`bgp_collector_locations` | `bgp` | `clear_then_reload` |
|
||||
| 船舶 | 展示 AIS 船只、位置、轨迹和源健康 | AIS sources、`barentswatch_vessels` | `vessel_static`、`vessel_position`、`ais_raw_observations`、`ais_source_health` | `vessels` | `clear_then_reload` |
|
||||
| 可交互对象 | 支撑通用地表图标、人工点位和未来扩展对象 | `earth_interactables` | 无 | `interactables` | `delta` |
|
||||
| 新闻与媒体 | 支撑 Earth 新闻、直播和巡航摘要 | news sources | `earth_news_items` | `news` / `media` | `reload` |
|
||||
| 新闻与媒体 | 支撑 Earth 新闻、直播和巡航摘要 | RSS news sources、手动新闻、直播源 | `earth_news_items` | `news` / `media` | `reload` |
|
||||
|
||||
## 卫星链路
|
||||
|
||||
@@ -141,12 +141,12 @@ sequenceDiagram
|
||||
|
||||
新闻与媒体数据用于 Earth 顶部新闻条、直播面板、新闻巡航和态势摘要。它们的视觉状态比地理对象更偏内容刷新,因此默认使用 `reload`。
|
||||
|
||||
- **采集入口**:RSS、直播源、新闻 source。
|
||||
- **事实表**:新闻 source 的 `collected_data`。
|
||||
- **采集入口**:RSS 新闻源、`智能星球内容 -> 新闻内容` 的手动新闻、直播源。
|
||||
- **事实表**:新闻 source 的 `collected_data`;手动新闻直接写入 `earth_news_items`,并以 `feed_type/source_type=manual` 标记内容来源。
|
||||
- **派生表**:`earth_news_items`。
|
||||
- **接口**:新闻、直播和媒体 visualization / content API。
|
||||
- **接口**:`/api/v1/news/earth-feed` 读取 `earth_news_items`;后台管理接口 `/api/v1/earth/news-items` 支持新增、JSON 导入、编辑、删除和重新处理手动新闻。
|
||||
- **删除语义**:删除新闻 source 或 `earth_news_items` 后广播 `news` / `media` reload;前端重拉后列表为空即隐藏对应内容。
|
||||
- **常见异常**:直播面板仍显示旧内容,通常是媒体组件本地状态没有响应 layer update,或内容接口缓存未失效。
|
||||
- **常见异常**:手动新闻保存后只显示原文或大区锚点是正常的“先展示再精修”窗口;若长期不更新,应检查 `earth_news_enrichment` 队列、AI / Web Search 配置和 `enrichment_status`。直播面板仍显示旧内容,通常是媒体组件本地状态没有响应 layer update,或内容接口缓存未失效。
|
||||
|
||||
## 扩展新图层
|
||||
|
||||
|
||||