Compare commits
65 Commits
codex/aipr
...
v0.37.0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
67f82dc41c | ||
|
|
abe04030fb | ||
|
|
6a5f9f7ad4 | ||
|
|
439a512148 | ||
|
|
f73fa1ea6d | ||
|
|
5b623a6385 | ||
|
|
0082cf3fbd | ||
|
|
3ae4acdff8 | ||
|
|
437efc848c | ||
|
|
003a46ac30 | ||
|
|
4b0be4cb76 | ||
|
|
b7647379de | ||
|
|
0f89372d71 | ||
|
|
2b0d4cfc49 | ||
|
|
e6d0332fba | ||
|
|
fe45a99cbd | ||
|
|
ae77b06c3c | ||
|
|
b5dd4f12f8 | ||
|
|
75cb214f23 | ||
|
|
4c21973197 | ||
|
|
51ae5e6ec9 | ||
|
|
1cf1f32ddd | ||
|
|
8f3ab88743 | ||
|
|
f8b43a995b | ||
|
|
d9adaf4134 | ||
|
|
40e51d5b20 | ||
|
|
93c1c1e550 | ||
|
|
48eb13b993 | ||
|
|
11179e7e67 | ||
|
|
07e26d6d5a | ||
|
|
7cd29cf9c0 | ||
|
|
2ee4773f4f | ||
|
|
b1d0624061 | ||
|
|
812c825dc6 | ||
|
|
a359d94127 | ||
|
|
c92be9c054 | ||
|
|
10e2bae8c2 | ||
|
|
60ed88b609 | ||
|
|
e85a9fc614 | ||
|
|
a2210f0f78 | ||
|
|
62ad09e816 | ||
|
|
89a71e6f29 | ||
|
|
60f5ff9bab | ||
|
|
fbb6adfbf5 | ||
|
|
749e6e76b6 | ||
|
|
83839b8b11 | ||
|
|
ed898aef9c | ||
|
|
abe0b5c11b | ||
|
|
306ba7f850 | ||
|
|
39f90bd575 | ||
|
|
c4ea918fac | ||
|
|
34d94a6b6b | ||
|
|
ef65acd49c | ||
|
|
5639546990 | ||
|
|
c8fe8cad59 | ||
|
|
d395769df6 | ||
|
|
8bd9d34376 | ||
|
|
2d43263b9e | ||
|
|
2da6ed166b | ||
|
|
da587398d9 | ||
|
|
f5308340af | ||
|
|
981617ee80 | ||
|
|
7abf391c74 | ||
|
|
f12719914d | ||
|
|
bc90e00e25 |
120
.claude/commands/cleanup.md
Normal file
120
.claude/commands/cleanup.md
Normal file
@@ -0,0 +1,120 @@
|
||||
---
|
||||
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
|
||||
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /cleanup — 垃圾代码审查与清理
|
||||
|
||||
分析当前工作区(git diff)中的未提交代码,找出并修复常见垃圾代码,**不得改变任何运行逻辑**。
|
||||
|
||||
## 检查范围
|
||||
|
||||
若 `$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
|
||||
|
||||
## 审查清单
|
||||
|
||||
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
|
||||
|
||||
### 1. 重复逻辑 (Duplicate Logic)
|
||||
- 完全相同或高度相似的代码块在多处出现
|
||||
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
|
||||
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
|
||||
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
|
||||
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
|
||||
|
||||
### 3. 命名问题
|
||||
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`)
|
||||
- 命名与实际用途不符
|
||||
- 同一概念在不同地方用不同名字表达
|
||||
|
||||
### 4. 死代码 / 无效代码
|
||||
- 注释掉的旧代码块(3行以上)
|
||||
- 声明后从未使用的变量/参数/导入
|
||||
- 永远不会执行的条件分支
|
||||
|
||||
### 5. 代码风格问题
|
||||
- 尾部空白字符(trailing whitespace)
|
||||
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
|
||||
- 空行使用不一致(连续多个空行等)
|
||||
|
||||
### 6. 其他常见问题
|
||||
- 私有辅助函数应被 export 但没有,导致调用方重复实现
|
||||
- 类型/接口重复定义
|
||||
- 过于冗长的条件表达式可以简化(不改逻辑)
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 获取待检查文件列表
|
||||
|
||||
```bash
|
||||
# 无参数时:获取所有未提交修改
|
||||
git diff HEAD --name-only
|
||||
|
||||
# 有参数时:用 $ARGUMENTS 过滤
|
||||
```
|
||||
|
||||
### Step 2 — 逐文件阅读并分析
|
||||
|
||||
- 用 Read 工具读取完整文件(不只读 diff)
|
||||
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
|
||||
|
||||
### Step 3 — 报告问题清单
|
||||
|
||||
在修改前,先以列表形式输出所有发现的问题:
|
||||
|
||||
```
|
||||
发现 N 个问题:
|
||||
|
||||
[文件] js/foo.js
|
||||
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
|
||||
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
|
||||
|
||||
[文件] js/bar.js
|
||||
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
|
||||
...
|
||||
```
|
||||
|
||||
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
|
||||
|
||||
### Step 4 — 执行修复
|
||||
|
||||
对每个问题,使用 Edit 工具进行**最小化修改**:
|
||||
|
||||
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
|
||||
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
|
||||
- **命名问题**:重命名变量,更新所有使用处
|
||||
- **死代码**:直接删除
|
||||
- **尾部空白/风格**:修正
|
||||
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
|
||||
|
||||
**修复原则:**
|
||||
- 只改在审查清单中发现的问题,不做额外优化
|
||||
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
|
||||
- 改完后用 `grep` 验证旧的坏代码已消失
|
||||
|
||||
### Step 5 — 输出总结
|
||||
|
||||
```
|
||||
清理完成:
|
||||
|
||||
修复了 N 个问题:
|
||||
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
|
||||
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 14(4处引用)
|
||||
✓ controls.js — export updateLayerButtonState,移除 main.js 中的重复实现
|
||||
...
|
||||
|
||||
未修改的问题(需人工确认):
|
||||
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
|
||||
```
|
||||
|
||||
## 约束
|
||||
|
||||
- **禁止**改变函数签名、接口定义、导出 API(除非问题正是私有函数应被 export)
|
||||
- **禁止**添加新功能、新抽象、新参数
|
||||
- **禁止**修改注释内容(只删除注释掉的死代码)
|
||||
- **禁止**修改测试文件逻辑
|
||||
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"
|
||||
91
.claude/commands/goal-driven.md
Normal file
91
.claude/commands/goal-driven.md
Normal file
@@ -0,0 +1,91 @@
|
||||
---
|
||||
description: 用 goal-driven 方法推动一个复杂任务持续执行,直到明确成功标准被满足
|
||||
argument-hint: 建议填写任务目标;若同时给出成功标准更好
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
|
||||
---
|
||||
|
||||
# /goal-driven — 目标驱动执行模式
|
||||
|
||||
使用 `lidangzzz/goal-driven` 的核心思想来推进复杂任务:先固定目标与成功标准,再持续执行和反复验收,直到标准真正满足。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 长周期实现任务
|
||||
- 高复杂度工程任务
|
||||
- 可被明确验收的研究、实现、迁移、验证类工作
|
||||
|
||||
不适用场景:
|
||||
|
||||
- 纯脑暴
|
||||
- 无法定义成功标准的模糊任务
|
||||
- 很小的一次性修改
|
||||
|
||||
## 输入要求
|
||||
|
||||
若 `$ARGUMENTS` 只包含目标,没有成功标准,先补全一版可执行的成功标准再开始。
|
||||
|
||||
启动时先输出:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- ...
|
||||
```
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先把任务固化为两个核心块:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. 成功标准必须尽量客观,可验证,可落地。
|
||||
优先写成:
|
||||
- 需要交付什么
|
||||
- 需要通过哪些测试或验证
|
||||
- 如何判断结果真的完成
|
||||
|
||||
3. 进入持续执行循环:
|
||||
- 完成一个阶段
|
||||
- 检查当前结果是否满足成功标准
|
||||
- 若未满足,明确剩余差距并继续推进
|
||||
|
||||
4. 任何“完成了”“差不多了”“已实现”之类的结论,都必须经过验证,不能直接接受。
|
||||
|
||||
5. 如果验证失败:
|
||||
- 明确指出哪条成功标准没满足
|
||||
- 继续工作,不要把阶段性进展误判为完成
|
||||
|
||||
6. 只有在以下情况之一才能停止:
|
||||
- 成功标准已满足
|
||||
- 用户明确要求停止
|
||||
|
||||
## 执行风格
|
||||
|
||||
- 重证据,轻口头判断
|
||||
- 重验收,轻自我感觉
|
||||
- 优先用测试、日志、产物、对比结果来证明完成
|
||||
- 对长期任务保持“未达标就继续”的节奏
|
||||
|
||||
## 简版模板
|
||||
|
||||
```md
|
||||
Goal: [[[[[在此填写最终目标]]]]]
|
||||
|
||||
Criteria for success: [[[[[在此填写成功标准]]]]]
|
||||
|
||||
循环执行:
|
||||
1. 推进任务
|
||||
2. 检查是否满足成功标准
|
||||
3. 若未满足,继续工作
|
||||
4. 直到满足标准或用户明确停止
|
||||
```
|
||||
146
.claude/commands/release.md
Normal file
146
.claude/commands/release.md
Normal file
@@ -0,0 +1,146 @@
|
||||
---
|
||||
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog,运行验证,commit 并 push
|
||||
argument-hint: 可选:feature | bugfix | 或直接描述本次发布内容
|
||||
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
# /release — Planet 发版工作流
|
||||
|
||||
## 版本号规则
|
||||
|
||||
| 变更类型 | 版本跳动 | 适用场景 |
|
||||
|---------|---------|---------|
|
||||
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
|
||||
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
|
||||
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
|
||||
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
|
||||
|
||||
意图混合时以用户明确描述为准;bugfix + 小 feature 混合默认判定为 `improvement`(`+0.0.1`)。
|
||||
|
||||
## 必须同步更新的文件
|
||||
|
||||
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json`(`"version"` 字段)
|
||||
- `pyproject.toml`(`version =` 字段)
|
||||
- `uv.lock`(**不要手动编辑**,通过 `uv lock` 重新生成)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## 执行步骤
|
||||
|
||||
### Step 1 — 环境检查
|
||||
|
||||
```bash
|
||||
git branch --show-current # 确认在 dev 分支
|
||||
git status --short # 检查是否有无关的未暂存修改
|
||||
cat VERSION # 读取当前版本
|
||||
```
|
||||
|
||||
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
|
||||
|
||||
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
|
||||
|
||||
### Step 2 — 确定发版类型与新版本号
|
||||
|
||||
- 若 `$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
|
||||
- 否则根据当前 `git diff HEAD` 和 `git log` 推断
|
||||
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`)
|
||||
- **先输出发版计划供用户确认**:
|
||||
|
||||
```
|
||||
发版计划:
|
||||
类型:bugfix
|
||||
版本:0.26.2 → 0.26.3
|
||||
分支:dev
|
||||
将更新:VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — 更新版本号文件
|
||||
|
||||
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
|
||||
|
||||
1. `VERSION` — 直接替换全部内容为新版本号
|
||||
2. `frontend/package.json` — 替换 `"version": "x.x.x"` 行
|
||||
3. `pyproject.toml` — 替换 `version = "x.x.x"` 行
|
||||
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
|
||||
|
||||
### Step 4 — 更新 CHANGELOG.md
|
||||
|
||||
在文件顶部插入新条目,格式:
|
||||
|
||||
```markdown
|
||||
## [x.x.x] — YYYY-MM-DD
|
||||
|
||||
### ✨ Features / 🐛 Fixes / 🔧 Improvements
|
||||
- ...(只列高信号条目,最多 5 条)
|
||||
- ...
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
日期使用 `date +%Y-%m-%d` 获取今天的日期。
|
||||
|
||||
### Step 5 — 更新 docs/version-history.md
|
||||
|
||||
- 更新文件头部的"当前开发版本"字段
|
||||
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
|
||||
|
||||
### Step 6 — 验证
|
||||
|
||||
针对本次变更范围做最小验证:
|
||||
|
||||
- Python 文件有修改:`python3 -m py_compile <changed_files>`
|
||||
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
|
||||
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — 提交前预览
|
||||
|
||||
展示将要提交的文件列表:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
|
||||
|
||||
### Step 8 — Commit & Push(用户确认后)
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# 若有代码变更也一并 stage
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
commit message 固定格式:`release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — 完成确认
|
||||
|
||||
输出摘要:
|
||||
|
||||
```
|
||||
✓ 版本号已更新:0.26.2 → 0.26.3
|
||||
✓ CHANGELOG 已更新
|
||||
✓ version-history 已更新
|
||||
✓ uv.lock 已重新生成
|
||||
✓ 验证通过
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ 已 push 到 origin/dev
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
|
||||
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
|
||||
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行
|
||||
3
.codex/config.toml
Normal file
3
.codex/config.toml
Normal file
@@ -0,0 +1,3 @@
|
||||
approval_policy = "never"
|
||||
|
||||
sandbox_mode = "danger-full-access"
|
||||
124
.codex/skills/cleanup/SKILL.md
Normal file
124
.codex/skills/cleanup/SKILL.md
Normal file
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: cleanup
|
||||
description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
|
||||
---
|
||||
|
||||
# Cleanup
|
||||
|
||||
Review and fix code quality issues in the current working tree without altering any logic or behavior.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to clean up, tidy, or lint uncommitted changes
|
||||
- The user wants a code smell review before releasing or committing
|
||||
- The user mentions magic numbers, duplicate logic, dead code, or naming issues
|
||||
|
||||
Do not refactor architecture, add features, or change behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
|
||||
|
||||
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
|
||||
|
||||
## Checklist
|
||||
|
||||
### 1. Duplicate Logic
|
||||
- Identical or near-identical code blocks appearing in multiple places
|
||||
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
|
||||
- Repeated DOM queries, regex literals, or template strings within the same file
|
||||
|
||||
### 2. Magic Numbers / Magic Strings
|
||||
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
|
||||
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
|
||||
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
|
||||
|
||||
### 3. Naming Issues
|
||||
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
|
||||
- Names that do not match actual behavior
|
||||
- The same concept referred to by different names in different places
|
||||
|
||||
### 4. Dead Code
|
||||
- Commented-out code blocks (3+ lines)
|
||||
- Variables, parameters, or imports declared but never used
|
||||
- Branches that can never execute
|
||||
|
||||
### 5. Style Inconsistencies
|
||||
- Trailing whitespace
|
||||
- Mixed quote styles or indentation within the same file
|
||||
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
|
||||
|
||||
### 6. Other
|
||||
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
|
||||
- Overly verbose conditions that can be simplified without changing logic
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1 — Get the file list
|
||||
|
||||
```bash
|
||||
git diff HEAD --name-only
|
||||
```
|
||||
|
||||
Filter to the user-specified path if one was provided.
|
||||
|
||||
### Step 2 — Read and analyze each file
|
||||
|
||||
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
|
||||
|
||||
### Step 3 — Report findings before touching anything
|
||||
|
||||
Print a structured list:
|
||||
|
||||
```
|
||||
Found N issues:
|
||||
|
||||
[file] js/foo.js
|
||||
· L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
|
||||
· L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
|
||||
|
||||
[file] js/bar.js
|
||||
· L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
|
||||
...
|
||||
```
|
||||
|
||||
If no issues are found, output "No code smells detected. Code quality looks good." and stop.
|
||||
|
||||
### Step 4 — Fix each issue
|
||||
|
||||
Use the Edit tool for **minimal, targeted changes**:
|
||||
|
||||
- **Duplicate logic**: extract to a shared constant or function; update all call sites
|
||||
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
|
||||
- **Naming**: rename the variable/function; update all references
|
||||
- **Dead code**: delete it
|
||||
- **Trailing whitespace / style**: fix in place
|
||||
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
|
||||
|
||||
Principles:
|
||||
- Only fix issues identified in the checklist — no extra improvements
|
||||
- Keep each Edit as small as possible
|
||||
- After fixing, verify the old bad pattern is gone with grep
|
||||
|
||||
### Step 5 — Summary
|
||||
|
||||
```
|
||||
Cleanup complete:
|
||||
|
||||
Fixed N issues:
|
||||
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
|
||||
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
|
||||
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
|
||||
...
|
||||
|
||||
Skipped (needs manual review):
|
||||
! foo.js L45 — large commented-out block; confirm it is safe to delete
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
|
||||
- **Do not** add new features, abstractions, or parameters
|
||||
- **Do not** rewrite comments (only delete commented-out dead code)
|
||||
- **Do not** touch test file logic
|
||||
- If a magic number's intent is uncertain, skip it and flag it in the summary
|
||||
101
.codex/skills/goal-driven/SKILL.md
Executable file
101
.codex/skills/goal-driven/SKILL.md
Executable file
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: goal-driven
|
||||
description: Run a goal-driven execution loop for very large, long-horizon, rigorously verifiable tasks. Use when the user explicitly wants the lidangzzz/goal-driven method, a master-agent plus worker-agent style workflow, or a persistent loop that keeps working until concrete success criteria are satisfied.
|
||||
---
|
||||
|
||||
# Goal-Driven
|
||||
|
||||
Use this skill when the user wants a strict goal-driven workflow for a hard task with:
|
||||
|
||||
- one clear end goal
|
||||
- explicit success criteria
|
||||
- repeated verification against those criteria
|
||||
- continued execution until the criteria are actually met
|
||||
|
||||
This skill is adapted from `lidangzzz/goal-driven`, but trimmed for local skill use to avoid bloating context.
|
||||
|
||||
## When To Use
|
||||
|
||||
Use it for tasks like:
|
||||
|
||||
- compilers, interpreters, theorem-like proof work, deep refactors
|
||||
- long-running system design or implementation work
|
||||
- problems that are expensive and complex, but still objectively testable
|
||||
|
||||
Do not use it for:
|
||||
|
||||
- vague brainstorming without a success condition
|
||||
- short one-shot edits
|
||||
- tasks where "done" cannot be evaluated in a meaningful way
|
||||
|
||||
## Core Model
|
||||
|
||||
The workflow has two roles:
|
||||
|
||||
1. Master role
|
||||
Defines the goal, defines the success criteria, audits progress, and decides whether the work is actually complete.
|
||||
|
||||
2. Worker role
|
||||
Keeps advancing the task toward the goal. If a result is partial, stalled, or unverifiable, the worker continues.
|
||||
|
||||
In Codex, only use actual subagents when the user explicitly asks for delegation or subagent work and the platform supports it. Otherwise emulate the same loop locally: keep working, checkpointing, and re-verifying until the criteria are satisfied.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Normalize the task into two blocks:
|
||||
- `Goal`
|
||||
- `Criteria for success`
|
||||
|
||||
2. Make the criteria concrete and testable.
|
||||
Good criteria usually include:
|
||||
- required outputs
|
||||
- required validations or tests
|
||||
- edge cases or coverage thresholds
|
||||
- what evidence proves completion
|
||||
|
||||
3. Break the work into milestones that can each produce evidence.
|
||||
|
||||
4. Execute the next milestone.
|
||||
If subagents are explicitly allowed, the master may delegate bounded worker tasks.
|
||||
If not, do the work locally but keep the master/worker mindset.
|
||||
|
||||
5. Whenever work pauses, stalls, or appears complete, audit against the criteria directly.
|
||||
Check artifacts, tests, logs, diffs, metrics, or other real evidence.
|
||||
|
||||
6. If the criteria are not met, continue with a specific delta:
|
||||
- what is still missing
|
||||
- what evidence failed
|
||||
- what the next worker pass must improve
|
||||
|
||||
7. Stop only when the criteria are met, or when the user explicitly stops the process.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Prefer objective checks over self-reported completion.
|
||||
- Do not confuse progress with completion.
|
||||
- If the worker says "done", verify it.
|
||||
- If verification fails, continue from the gap instead of restarting blindly.
|
||||
- Keep the goal stable unless the user changes it.
|
||||
- Tighten fuzzy criteria before sinking large amounts of effort.
|
||||
|
||||
## Recommended Response Shape
|
||||
|
||||
When starting a goal-driven task, structure the kickoff like this:
|
||||
|
||||
```md
|
||||
Goal
|
||||
- ...
|
||||
|
||||
Criteria for success
|
||||
- ...
|
||||
|
||||
Current plan
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
Verification
|
||||
- What evidence will prove completion
|
||||
```
|
||||
|
||||
For a reusable prompt template, read [references/prompt-template.md](references/prompt-template.md).
|
||||
7
.codex/skills/goal-driven/agents/openai.yaml
Normal file
7
.codex/skills/goal-driven/agents/openai.yaml
Normal file
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Goal-Driven"
|
||||
short_description: "Drive complex work until explicit success criteria are met."
|
||||
default_prompt: "Use $goal-driven to turn this task into a concrete goal, explicit success criteria, and a verification-driven execution loop."
|
||||
|
||||
policy:
|
||||
allow_implicit_invocation: true
|
||||
38
.codex/skills/goal-driven/references/prompt-template.md
Executable file
38
.codex/skills/goal-driven/references/prompt-template.md
Executable file
@@ -0,0 +1,38 @@
|
||||
# Goal-Driven Prompt Template
|
||||
|
||||
Use this when you want a reusable kickoff prompt for a master/worker execution loop.
|
||||
|
||||
```md
|
||||
# Goal-Driven System
|
||||
|
||||
Goal: [[[[[DEFINE THE FINAL GOAL HERE]]]]]
|
||||
|
||||
Criteria for success: [[[[[DEFINE THE SUCCESS CRITERIA HERE]]]]]
|
||||
|
||||
You are the master agent.
|
||||
|
||||
Your job is to:
|
||||
1. Keep the goal and criteria fixed.
|
||||
2. Start worker execution toward the goal.
|
||||
3. Audit any claimed progress against the criteria.
|
||||
4. If the criteria are not met, continue the work with a precise next delta.
|
||||
5. Stop only when the criteria are satisfied or the user explicitly stops the process.
|
||||
|
||||
Worker requirements:
|
||||
1. Break the task into subproblems.
|
||||
2. Keep producing concrete progress toward the goal.
|
||||
3. Report evidence, not just claims.
|
||||
4. Continue until the criteria are satisfied.
|
||||
|
||||
Master audit loop:
|
||||
1. Check whether the worker is still making progress.
|
||||
2. If the worker stalls or claims completion, verify against the criteria.
|
||||
3. If verification fails, resume work from the remaining gap.
|
||||
4. Repeat until the criteria are met.
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Stronger criteria produce better results than stronger rhetoric.
|
||||
- Prefer measurable checks such as tests, parity checks, generated artifacts, benchmarks, or reviewable outputs.
|
||||
- If the environment does not support subagents, emulate the same loop locally.
|
||||
157
.codex/skills/release/SKILL.md
Normal file
157
.codex/skills/release/SKILL.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: release
|
||||
description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
|
||||
---
|
||||
|
||||
# Release Workflow
|
||||
|
||||
Use this skill for release-oriented work in this repository.
|
||||
|
||||
## When To Use
|
||||
|
||||
- The user asks to `发版`
|
||||
- The user asks to bump a version
|
||||
- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
|
||||
- The user asks to commit/push a release or a publishable bugfix/feature bundle
|
||||
|
||||
Do not use this skill for ordinary commits that are not being released.
|
||||
|
||||
## Versioning Rules
|
||||
|
||||
- `feature` -> bump `+0.1.0`
|
||||
- `bugfix` -> bump `+0.0.1`
|
||||
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
|
||||
|
||||
When intent is mixed, prefer the user's stated release intent.
|
||||
|
||||
## Required Files
|
||||
|
||||
Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
|
||||
|
||||
- `VERSION`
|
||||
- `frontend/package.json` (`"version"` field)
|
||||
- `pyproject.toml` (`version =` field)
|
||||
- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
|
||||
- `docs/CHANGELOG.md`
|
||||
- `docs/version-history.md`
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1 — Environment check
|
||||
|
||||
```bash
|
||||
git branch --show-current # must be on dev
|
||||
git status --short # check for unrelated uncommitted changes
|
||||
cat VERSION # read current version
|
||||
```
|
||||
|
||||
If not on `dev`, stop and tell the user. Do not proceed.
|
||||
|
||||
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
|
||||
|
||||
### Step 2 — Determine release type and next version
|
||||
|
||||
- If the user provided an explicit type (`feature` / `bugfix`), use it
|
||||
- Otherwise infer from `git diff HEAD` and recent `git log`
|
||||
- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
|
||||
- **Show the release plan before making any changes:**
|
||||
|
||||
```
|
||||
Release plan:
|
||||
Type: bugfix
|
||||
Version: 0.26.2 → 0.26.3
|
||||
Branch: dev
|
||||
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
|
||||
```
|
||||
|
||||
### Step 3 — Update version files
|
||||
|
||||
Update in order (use Edit for precise replacement, never rewrite whole files):
|
||||
|
||||
1. `VERSION` — replace entire content with new version string
|
||||
2. `frontend/package.json` — replace `"version": "x.x.x"` line
|
||||
3. `pyproject.toml` — replace `version = "x.x.x"` line
|
||||
4. Run `uv lock` at repo root to regenerate `uv.lock`
|
||||
|
||||
### Step 4 — Update CHANGELOG.md
|
||||
|
||||
Insert a new entry at the top of the file:
|
||||
|
||||
```markdown
|
||||
## x.x.x
|
||||
|
||||
Released: YYYY-MM-DD
|
||||
|
||||
### Highlights
|
||||
|
||||
- ...
|
||||
|
||||
### Added / Fixed / Improved
|
||||
|
||||
- ... (high-signal items only, max 5)
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
Get today's date with `date +%Y-%m-%d`.
|
||||
|
||||
### Step 5 — Update docs/version-history.md
|
||||
|
||||
- Update the "current dev version" field in the file header
|
||||
- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
|
||||
|
||||
### Step 6 — Validate
|
||||
|
||||
Run the smallest relevant validation for the changes in scope:
|
||||
|
||||
- Python files changed: `python3 -m py_compile <changed_files>`
|
||||
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
|
||||
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
|
||||
|
||||
```bash
|
||||
grep -h "version" VERSION frontend/package.json pyproject.toml
|
||||
```
|
||||
|
||||
### Step 7 — Pre-commit preview
|
||||
|
||||
Show what will be committed:
|
||||
|
||||
```bash
|
||||
git diff --stat HEAD
|
||||
```
|
||||
|
||||
Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
|
||||
|
||||
### Step 8 — Commit, tag, and push
|
||||
|
||||
```bash
|
||||
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
|
||||
# also stage any code changes included in this release
|
||||
git add <code_files>
|
||||
|
||||
git commit -m "release: bump version to x.x.x"
|
||||
git tag vx.x.x
|
||||
git push origin dev
|
||||
git push origin vx.x.x
|
||||
```
|
||||
|
||||
Commit message format is fixed: `release: bump version to x.x.x`
|
||||
|
||||
### Step 9 — Completion summary
|
||||
|
||||
```
|
||||
✓ Version bumped: 0.26.2 → 0.26.3
|
||||
✓ CHANGELOG updated
|
||||
✓ version-history updated
|
||||
✓ uv.lock regenerated
|
||||
✓ Validation passed
|
||||
✓ commit: release: bump version to 0.26.3
|
||||
✓ tag: v0.26.3
|
||||
✓ Pushed to origin/dev
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `uv.lock` must only be updated by running `uv lock`, never manually
|
||||
- The release commit should include only version files + the code for this release — no unrelated changes
|
||||
- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually
|
||||
5
.gitignore
vendored
5
.gitignore
vendored
@@ -145,3 +145,8 @@ docs/.venv/
|
||||
*.temp
|
||||
tmp/
|
||||
temp/
|
||||
|
||||
# ----------------------
|
||||
# Runtime Data
|
||||
# ----------------------
|
||||
data/ai/bgp-briefs/
|
||||
|
||||
177
README.md
177
README.md
@@ -102,6 +102,13 @@
|
||||
| Axios | HTTP 客户端 |
|
||||
| Socket.io-client | WebSocket 客户端 |
|
||||
| ECharts | 统计图表 |
|
||||
| Bun | 前端包管理与脚本运行 |
|
||||
|
||||
前端工程统一使用 Bun:
|
||||
|
||||
- 安装依赖使用 `bun install`
|
||||
- 运行脚本使用 `bun run <script>`
|
||||
- 不使用 `npm`、`pnpm`、`yarn`
|
||||
|
||||
### 虚幻引擎客户端
|
||||
|
||||
@@ -205,10 +212,161 @@
|
||||
./planet.sh health
|
||||
```
|
||||
|
||||
前端命令约定:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
bun install
|
||||
bun run dev
|
||||
bun run build
|
||||
```
|
||||
|
||||
不要使用 `npm run ...`,避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
|
||||
|
||||
## API 文档
|
||||
|
||||
启动服务后访问: `http://localhost:8000/docs`
|
||||
|
||||
## WSL / Windows 局域网访问
|
||||
|
||||
如果服务运行在 WSL 中,而你希望:
|
||||
|
||||
- Windows 本机浏览器访问开发服务
|
||||
- 同一局域网内的手机或其他电脑访问开发服务
|
||||
|
||||
推荐按下面顺序排查和配置。
|
||||
|
||||
### 1. 在 WSL 中启动服务
|
||||
|
||||
```bash
|
||||
./planet.sh start --allow-lan
|
||||
```
|
||||
|
||||
这会让前端监听 `0.0.0.0:3000`,后端监听 `0.0.0.0:8000`。
|
||||
|
||||
### 2. 先确认 WSL 内部服务正常
|
||||
|
||||
在 WSL 中执行:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
ss -ltnp | grep -E ':3000|:8000'
|
||||
```
|
||||
|
||||
预期:
|
||||
|
||||
- `3000` 返回前端 HTML
|
||||
- `8000/health` 返回健康检查 JSON
|
||||
- `ss` 中能看到 `0.0.0.0:3000` 和 `0.0.0.0:8000`
|
||||
|
||||
如果这一步不通,先不要继续做 Windows 转发。
|
||||
|
||||
### 3. 在 Windows 本机验证 localhost 直通
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
curl http://localhost:3000
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
在常见的 WSL2 开发环境下,Windows 通常可以直接通过 `localhost` 访问 WSL 中的服务。
|
||||
|
||||
### 4. 如果需要让局域网设备访问,再做 Windows 端口转发
|
||||
|
||||
注意:下面的命令必须在“以管理员身份运行”的 PowerShell 中执行。
|
||||
|
||||
先把 Windows 对外网卡上的 `3000` / `8000` 转发到 Windows 本机 `127.0.0.1`:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=3000
|
||||
netsh interface portproxy delete v4tov4 listenaddress=0.0.0.0 listenport=8000
|
||||
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=3000 connectaddress=127.0.0.1 connectport=3000
|
||||
netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=8000 connectaddress=127.0.0.1 connectport=8000
|
||||
```
|
||||
|
||||
再放行 Windows 防火墙:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 3000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3000
|
||||
New-NetFirewallRule -DisplayName "WSL Planet 8000" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 8000
|
||||
```
|
||||
|
||||
检查转发规则是否生效:
|
||||
|
||||
```powershell
|
||||
netsh interface portproxy show all
|
||||
```
|
||||
|
||||
预期能看到:
|
||||
|
||||
- `0.0.0.0:3000 -> 127.0.0.1:3000`
|
||||
- `0.0.0.0:8000 -> 127.0.0.1:8000`
|
||||
|
||||
### 5. 查 Windows 局域网 IP,并让其他设备访问
|
||||
|
||||
在 Windows PowerShell 中执行:
|
||||
|
||||
```powershell
|
||||
ipconfig
|
||||
```
|
||||
|
||||
找到当前联网网卡的 IPv4 地址,例如 `192.168.8.228`。
|
||||
|
||||
局域网其他设备可访问:
|
||||
|
||||
- `http://<Windows局域网IP>:3000/earth`
|
||||
- `http://<Windows局域网IP>:3000/admin`
|
||||
|
||||
例如:
|
||||
|
||||
- `http://192.168.8.228:3000/earth`
|
||||
|
||||
### 6. 常见现象与判断
|
||||
|
||||
- WSL 中 `curl localhost:3000` 能通,但 Windows 访问 `WSL 的局域网 IP:3000` 不通:这是正常现象之一,优先验证 Windows 的 `localhost:3000`
|
||||
- Windows `localhost:3000` 能通,但局域网设备访问 `Windows 局域网 IP:3000` 不通:通常缺少 `portproxy` 或防火墙放行
|
||||
- `whoami /groups` 中 `S-1-5-32-544` 显示 `deny only`:说明当前 PowerShell 不是提权管理员窗口
|
||||
|
||||
### 7. 本项目一次性验证顺序
|
||||
|
||||
建议固定按这个顺序验证:
|
||||
|
||||
1. WSL 中执行 `curl http://localhost:3000`
|
||||
2. WSL 中执行 `curl http://localhost:8000/health`
|
||||
3. Windows 中执行 `curl http://localhost:3000`
|
||||
4. Windows 中执行 `curl http://localhost:8000/health`
|
||||
5. 管理员 PowerShell 配置 `portproxy` 和防火墙
|
||||
6. 用手机或其他电脑访问 `http://<Windows局域网IP>:3000/earth`
|
||||
|
||||
## 启动容错参数
|
||||
|
||||
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
|
||||
|
||||
可通过环境变量临时调整:
|
||||
|
||||
```bash
|
||||
# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
|
||||
AI_PROVIDER_START_MAX_RETRIES=5 \
|
||||
AI_PROVIDER_RETRY_INTERVAL=10 \
|
||||
DATABASE_START_MAX_RETRIES=5 \
|
||||
DATABASE_RETRY_INTERVAL=10 \
|
||||
./planet.sh restart
|
||||
```
|
||||
|
||||
常用参数:
|
||||
|
||||
- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync`、`bun install` 的重试次数与间隔,默认 `3` 次、`5` 秒
|
||||
- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres`、`redis` 的启动/重启与健康检查自愈,默认 `3` 次、`5` 秒
|
||||
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5` 秒
|
||||
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
|
||||
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
|
||||
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2` 秒
|
||||
|
||||
## AI 接口预留
|
||||
|
||||
项目现在采用“两层”设计:
|
||||
@@ -284,8 +442,25 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
详细文档:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
|
||||
|
||||
## 前端页面布局规范
|
||||
|
||||
管理后台页面默认遵循“单屏工作区”原则:
|
||||
|
||||
- 页头、摘要区、主工作区应在一屏内形成稳定结构
|
||||
- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
|
||||
- 模块内容超出时优先在卡片、表格、标签页内部滚动
|
||||
- 不依赖整页纵向撑开来容纳主要工作区
|
||||
|
||||
当前推荐参考实现:
|
||||
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
15
TODO.md
15
TODO.md
@@ -13,7 +13,22 @@
|
||||
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
|
||||
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
|
||||
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback,而不是主来源
|
||||
- [ ] 为 `aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json` 或 `yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
|
||||
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
|
||||
- [ ] 在兼容配置中补齐可声明项:`api adapter`、`base_url pattern`、`auth header`、`thinking default`、`reasoning block mapping`、`stream path`、`tool-call capability`、`multimodal capability`、`provider-specific request patch`
|
||||
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
|
||||
- [x] 在 activity layer 之后继续补 `route leak` 和 `path instability / flap` detector
|
||||
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation,降低后续维护复杂度
|
||||
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker(参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
|
||||
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
|
||||
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
|
||||
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
|
||||
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector
|
||||
- [ ] 为未知位置的算力中心建立分层坐标补全链路:优先 `精确坐标 > 站点/园区命中 > 城市 > 州/省 > 国家内主要算力城市 > 国家质心`,并把每次回退的 `confidence / reason / precision` 明确写进统一 GeoJSON
|
||||
- [ ] 为算力中心补一份可维护的本地位置注册表,例如 `canonical_name / aliases / operator / country / region / city / lat / lon / confidence / source_note`,避免把地点知识长期硬编码在 `visualization.py`
|
||||
- [ ] 增强 `epoch_ai_gpu` 和相关算力采集器的源页面解析:即使公开 API 不给坐标,也继续尝试从详情页、HTML、内嵌 JSON、schema.org、OpenGraph、脚本变量和 PDF/新闻稿链接里抽地点线索
|
||||
- [ ] 为未知位置算力中心增加外部富化策略评估:可选接入公开知识源或搜索兜底,只抓“站点名/园区名/城市名”级别线索,不直接抓经纬度结论,并把结果作为候选证据而不是真值
|
||||
- [ ] 为算力中心建立 `operator / cluster name / facility alias` 归一化层,先解决 `xAI / Colossus / Memphis`、`OpenAI / Stargate`、`CoreWeave`、`Lambda`、`Crusoe` 这类同一对象多种写法导致的地点匹配失败
|
||||
- [ ] 为估算位置增加更细的视觉和产品表达:除了问号角标,还要支持 tooltip/详情中的“估算依据”“精度级别”“最后核验时间”,并允许在设置中单独开关“仅看精确位置”
|
||||
- [ ] 为国家级估算点设计更合理的落点策略:优先落在“该国主要算力/数据中心城市候选集”而不是几何质心,必要时同国多节点做稳定散列分配,避免大量节点堆在荒漠或海上
|
||||
- [ ] 为未知位置算力中心建立人工校验工作流:支持导出待核验清单、记录人工确认结果,并把人工确认反哺到位置注册表,逐步减少问号点比例
|
||||
|
||||
@@ -6,29 +6,51 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_HTTP_RETRY_ATTEMPTS=2
|
||||
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
|
||||
|
||||
# Select one provider mode:
|
||||
# - openai_compatible
|
||||
# - claude_compatible
|
||||
# Provider identity. Recommended values:
|
||||
# - minimax
|
||||
# - openai
|
||||
# - ollama
|
||||
AI_PROVIDER=ollama
|
||||
# Compatibility aliases still accepted:
|
||||
# - openai_compatible
|
||||
# - anthropic_compatible
|
||||
# - claude_compatible
|
||||
AI_PROVIDER=minimax
|
||||
|
||||
# Request adapter style, following OpenClaw's API-seam pattern:
|
||||
# - auto
|
||||
# - openai-completions
|
||||
# - anthropic-messages
|
||||
# - ollama-generate
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
|
||||
# Common model selection
|
||||
AI_MODEL=qwen2.5:7b
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
|
||||
# MiniMax CN Anthropic-compatible example
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-change-me
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
|
||||
# AI_PROVIDER=openai_compatible
|
||||
# AI_PROVIDER=openai
|
||||
# AI_PROVIDER_API=openai-completions
|
||||
# AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-local-model
|
||||
|
||||
# Claude-compatible example (Anthropic / MiniMax / Claude-compatible gateway)
|
||||
# AI_PROVIDER=claude_compatible
|
||||
# AI_BASE_URL=http://127.0.0.1:8002
|
||||
# Anthropic-compatible example (Claude-compatible gateway)
|
||||
# AI_PROVIDER=anthropic
|
||||
# AI_PROVIDER_API=anthropic-messages
|
||||
# AI_BASE_URL=http://127.0.0.1:8002/anthropic
|
||||
# AI_API_KEY=local-key
|
||||
# AI_MODEL=your-model
|
||||
# AI_MAX_TOKENS=1200
|
||||
# AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
# Ollama native example
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
# AI_PROVIDER=ollama
|
||||
# AI_PROVIDER_API=ollama-generate
|
||||
# AI_BASE_URL=http://127.0.0.1:11434
|
||||
# AI_API_KEY=
|
||||
# AI_MODEL=qwen2.5:7b
|
||||
|
||||
@@ -4,21 +4,31 @@
|
||||
|
||||
完整使用说明见:
|
||||
|
||||
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
|
||||
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
|
||||
|
||||
当前支持:
|
||||
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=openai_compatible`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=anthropic_compatible`
|
||||
- `AI_PROVIDER=claude_compatible`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- provider identity:
|
||||
- `AI_PROVIDER=openai`
|
||||
- `AI_PROVIDER=anthropic`
|
||||
- `AI_PROVIDER=minimax`
|
||||
- `AI_PROVIDER=ollama`
|
||||
- request adapter:
|
||||
- `AI_PROVIDER_API=openai-completions`
|
||||
- `AI_PROVIDER_API=anthropic-messages`
|
||||
- `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
兼容别名仍然保留:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
|
||||
典型配置:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=https://api.openai.com/v1
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=gpt-4o-mini
|
||||
@@ -26,13 +36,14 @@ AI_TIMEOUT_SECONDS=60
|
||||
AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
Claude 兼容供应商示例:
|
||||
MiniMax 中国大陆节点示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-claude-compatible-model
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_TIMEOUT_SECONDS=60
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
@@ -43,12 +54,15 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
- Anthropic 官方 Claude API
|
||||
- Claude 兼容网关
|
||||
- MiniMax 等提供 Claude/Anthropic 风格消息接口的服务
|
||||
- MiniMax 等提供 Anthropic Messages 风格接口的服务
|
||||
|
||||
这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
|
||||
|
||||
Ollama 原生示例:
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
@@ -58,8 +72,8 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
|
||||
|
||||
本地模型接入建议:
|
||||
|
||||
- `vLLM`、`LM Studio`、`One API`:优先使用 `openai_compatible`
|
||||
- `MiniMax`、Claude 兼容网关:使用 `claude_compatible`
|
||||
- `vLLM`、`LM Studio`、`One API`:`AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`、Claude 兼容网关:`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`:可直接使用 `ollama`
|
||||
|
||||
启动模板:
|
||||
|
||||
@@ -9,6 +9,7 @@ class Settings(BaseSettings):
|
||||
SERVICE_VERSION: str = "0.1.0"
|
||||
|
||||
AI_PROVIDER: str = "disabled"
|
||||
AI_PROVIDER_API: str = "auto"
|
||||
AI_BASE_URL: str = "https://api.openai.com/v1"
|
||||
AI_API_KEY: str = ""
|
||||
AI_MODEL: str = ""
|
||||
|
||||
@@ -8,6 +8,7 @@ from fastapi import HTTPException, status
|
||||
|
||||
from aiprovider.config import settings
|
||||
from aiprovider.schemas import (
|
||||
AIContentBlock,
|
||||
AIProviderStatusResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
@@ -18,9 +19,39 @@ def _normalize_provider(value: str) -> str:
|
||||
return (value or "disabled").strip().lower()
|
||||
|
||||
|
||||
def _normalize_provider_api(value: str) -> str:
|
||||
return (value or "auto").strip().lower().replace("_", "-")
|
||||
|
||||
|
||||
def _resolve_provider_api(provider: str, configured_api: str) -> str:
|
||||
if configured_api and configured_api != "auto":
|
||||
return configured_api
|
||||
|
||||
if provider in {"openai", "openai-compatible", "openai_compatible"}:
|
||||
return "openai-completions"
|
||||
if provider in {
|
||||
"anthropic",
|
||||
"anthropic-compatible",
|
||||
"anthropic_compatible",
|
||||
"claude-compatible",
|
||||
"claude_compatible",
|
||||
"minimax",
|
||||
"kimi-coding",
|
||||
"moonshot-anthropic",
|
||||
}:
|
||||
return "anthropic-messages"
|
||||
if provider == "ollama":
|
||||
return "ollama-generate"
|
||||
return "disabled"
|
||||
|
||||
|
||||
class ProviderService:
|
||||
def __init__(self) -> None:
|
||||
self.provider = _normalize_provider(settings.AI_PROVIDER)
|
||||
self.provider_api = _resolve_provider_api(
|
||||
self.provider,
|
||||
_normalize_provider_api(settings.AI_PROVIDER_API),
|
||||
)
|
||||
self.base_url = settings.AI_BASE_URL.rstrip("/")
|
||||
self.api_key = settings.AI_API_KEY
|
||||
self.default_model = settings.AI_MODEL
|
||||
@@ -32,9 +63,11 @@ class ProviderService:
|
||||
|
||||
def get_status(self) -> AIProviderStatusResponse:
|
||||
enabled = self.provider != "disabled"
|
||||
configured = enabled and bool(self.base_url and self.api_key and self.default_model)
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
configured = enabled and bool(self.base_url and has_credentials and self.default_model)
|
||||
return AIProviderStatusResponse(
|
||||
provider=self.provider,
|
||||
api=self.provider_api if enabled else None,
|
||||
enabled=enabled,
|
||||
configured=configured,
|
||||
model=self.default_model or None,
|
||||
@@ -49,7 +82,8 @@ class ProviderService:
|
||||
)
|
||||
|
||||
model = payload.preferred_model or self.default_model
|
||||
if not self.base_url or not self.api_key or not model:
|
||||
has_credentials = bool(self.api_key) if self._requires_api_key() else True
|
||||
if not self.base_url or not has_credentials or not model:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
|
||||
@@ -57,28 +91,40 @@ class ProviderService:
|
||||
|
||||
prompt = self._build_prompt(payload)
|
||||
|
||||
if self.provider in {"openai", "openai_compatible"}:
|
||||
if self.provider_api == "openai-completions":
|
||||
data = await self._request_openai_compatible(model, prompt)
|
||||
content = self._extract_openai_content(data)
|
||||
elif self.provider in {"anthropic", "anthropic_compatible", "claude_compatible"}:
|
||||
data = await self._request_anthropic_compatible(model, prompt)
|
||||
content_blocks = self._extract_openai_blocks(data)
|
||||
elif self.provider_api == "anthropic-messages":
|
||||
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
|
||||
content = self._extract_anthropic_content(data)
|
||||
elif self.provider == "ollama":
|
||||
content_blocks = self._extract_anthropic_blocks(data)
|
||||
elif self.provider_api == "ollama-generate":
|
||||
data = await self._request_ollama(model, prompt)
|
||||
content = self._extract_ollama_content(data)
|
||||
content_blocks = self._extract_ollama_blocks(data)
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported AI provider: {self.provider}",
|
||||
detail=f"Unsupported AI provider API: {self.provider_api}",
|
||||
)
|
||||
|
||||
text_blocks = [block.text for block in content_blocks if block.text]
|
||||
thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
|
||||
|
||||
return SituationalAnalysisResponse(
|
||||
provider=self.provider,
|
||||
model=model,
|
||||
content=content,
|
||||
content_blocks=content_blocks,
|
||||
text_blocks=text_blocks,
|
||||
thinking_blocks=thinking_blocks,
|
||||
raw_response=data,
|
||||
)
|
||||
|
||||
def _requires_api_key(self) -> bool:
|
||||
return self.provider_api != "ollama-generate"
|
||||
|
||||
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
|
||||
sections = [
|
||||
f"任务标题:\n{payload.title}",
|
||||
@@ -113,7 +159,12 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
async def _request_anthropic_compatible(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
async def _request_anthropic_messages(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
"system": self.system_prompt,
|
||||
@@ -131,8 +182,15 @@ class ProviderService:
|
||||
"max_tokens": self.max_tokens,
|
||||
"temperature": 0.2,
|
||||
}
|
||||
resolved_thinking = self._resolve_anthropic_thinking(thinking)
|
||||
if resolved_thinking:
|
||||
request_body["thinking"] = resolved_thinking
|
||||
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
|
||||
path = "/v1/messages"
|
||||
else:
|
||||
path = "/messages"
|
||||
return await self._post(
|
||||
path="/messages",
|
||||
path=path,
|
||||
headers={
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": self.anthropic_version,
|
||||
@@ -141,6 +199,25 @@ class ProviderService:
|
||||
request_body=request_body,
|
||||
)
|
||||
|
||||
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if thinking:
|
||||
return thinking
|
||||
|
||||
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
|
||||
# disable thinking by default unless the caller explicitly opts in.
|
||||
if self.provider == "minimax":
|
||||
return {"type": "disabled"}
|
||||
|
||||
return None
|
||||
|
||||
async def _request_anthropic_compatible(
|
||||
self,
|
||||
model: str,
|
||||
prompt: str,
|
||||
thinking: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
return await self._request_anthropic_messages(model, prompt, thinking)
|
||||
|
||||
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
|
||||
request_body = {
|
||||
"model": model,
|
||||
@@ -218,6 +295,30 @@ class ProviderService:
|
||||
)
|
||||
return ""
|
||||
|
||||
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
choices = payload.get("choices") or []
|
||||
if not choices:
|
||||
return []
|
||||
|
||||
message = choices[0].get("message") or {}
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return [AIContentBlock(type="text", text=content)]
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "text")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
|
||||
content = payload.get("content")
|
||||
if isinstance(content, str):
|
||||
@@ -233,8 +334,39 @@ class ProviderService:
|
||||
fragments.append(item["text"])
|
||||
return "".join(fragments)
|
||||
|
||||
def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
content = payload.get("content")
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
|
||||
blocks: list[AIContentBlock] = []
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
blocks.append(
|
||||
AIContentBlock(
|
||||
type=str(item.get("type", "unknown")),
|
||||
text=item.get("text") if isinstance(item.get("text"), str) else None,
|
||||
thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
|
||||
signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
|
||||
metadata={
|
||||
k: v
|
||||
for k, v in item.items()
|
||||
if k not in {"type", "text", "thinking", "signature"}
|
||||
},
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str):
|
||||
return response
|
||||
return ""
|
||||
|
||||
def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
|
||||
response = payload.get("response")
|
||||
if isinstance(response, str) and response:
|
||||
return [AIContentBlock(type="text", text=response)]
|
||||
return []
|
||||
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,17 +18,22 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
|
||||
@@ -13,7 +13,9 @@ from app.api.v1 import (
|
||||
collected_data,
|
||||
visualization,
|
||||
bgp,
|
||||
news,
|
||||
system_control,
|
||||
tv,
|
||||
)
|
||||
|
||||
api_router = APIRouter()
|
||||
@@ -33,3 +35,5 @@ api_router.include_router(settings.router, prefix="/settings", tags=["settings"]
|
||||
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
|
||||
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
|
||||
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
|
||||
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
|
||||
api_router.include_router(news.router, prefix="/news", tags=["news"])
|
||||
|
||||
@@ -1,15 +1,52 @@
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.security import get_current_user
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
AlertBriefRequest,
|
||||
AlertBriefResponse,
|
||||
BGPBriefRequest,
|
||||
BGPBriefRecordResponse,
|
||||
BGPBriefRecordSummary,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAlertBriefRequest,
|
||||
SituationalAlertBriefResponse,
|
||||
SituationalAnalysisRequest,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
from app.services.alert_ai_brief import build_alert_brief_request
|
||||
from app.services.ai_client import AIProviderClient, get_ai_provider_client
|
||||
from app.services.bgp_ai_brief import build_bgp_brief_request
|
||||
from app.services.bgp_ai_brief_store import (
|
||||
get_bgp_brief_record,
|
||||
get_latest_bgp_brief_record,
|
||||
list_bgp_brief_records,
|
||||
save_bgp_brief_record,
|
||||
)
|
||||
from app.services.playground_session_store import (
|
||||
get_playground_session,
|
||||
upsert_playground_session,
|
||||
)
|
||||
from app.services.playground_chat_service import (
|
||||
create_turn,
|
||||
edit_user_message,
|
||||
get_thread,
|
||||
resend_turn,
|
||||
stop_message,
|
||||
)
|
||||
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -37,3 +74,208 @@ async def analyze_situational_awareness(
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
return await provider_client.analyze(payload, request_id=request_id)
|
||||
|
||||
|
||||
@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
|
||||
async def get_playground_thread(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_thread(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
|
||||
async def get_saved_playground_session(
|
||||
session_key: str = "default",
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
session_key=session_key,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/playground/session", response_model=PlaygroundSessionResponse)
|
||||
async def save_playground_session(
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await upsert_playground_session(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
|
||||
async def create_playground_message(
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await create_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
|
||||
async def stop_playground_message(
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await stop_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
|
||||
async def resend_playground_message(
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await resend_turn(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
|
||||
async def edit_playground_message(
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await edit_user_message(
|
||||
db,
|
||||
user_id=current_user.id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
|
||||
async def list_saved_bgp_briefs(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return list_bgp_brief_records()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
|
||||
async def get_latest_saved_bgp_brief(
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
return get_latest_bgp_brief_record()
|
||||
|
||||
|
||||
@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
|
||||
async def get_saved_bgp_brief(
|
||||
brief_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
record = get_bgp_brief_record(brief_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail="BGP brief not found")
|
||||
return record
|
||||
|
||||
|
||||
@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
|
||||
async def analyze_bgp_brief(
|
||||
payload: BGPBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_bgp_brief_request(
|
||||
db,
|
||||
incident_limit=payload.incident_limit,
|
||||
anomaly_limit=payload.anomaly_limit,
|
||||
collector_limit=payload.collector_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return save_bgp_brief_record(
|
||||
analysis,
|
||||
request_id=request_id,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/alerts/brief", response_model=AlertBriefResponse)
|
||||
async def analyze_alert_brief(
|
||||
payload: AlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_alert_brief_request(
|
||||
db,
|
||||
alert_limit=payload.alert_limit,
|
||||
)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return AlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
|
||||
async def analyze_situational_alert_brief(
|
||||
payload: SituationalAlertBriefRequest,
|
||||
request: Request,
|
||||
response: Response,
|
||||
current_user: User = Depends(get_current_user),
|
||||
provider_client: AIProviderClient = Depends(get_ai_provider_client),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
request_id = request.headers.get("X-Request-ID") or str(uuid4())
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
brief_request, facts, context = await build_situational_alert_brief_request(db)
|
||||
brief_request.preferred_model = payload.preferred_model
|
||||
brief_request.thinking = payload.thinking
|
||||
|
||||
analysis = await provider_client.analyze(brief_request, request_id=request_id)
|
||||
return SituationalAlertBriefResponse(
|
||||
**analysis.model_dump(),
|
||||
title=brief_request.title,
|
||||
objective=brief_request.objective,
|
||||
facts=facts,
|
||||
context=context,
|
||||
)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select, func, case
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
@@ -9,6 +9,7 @@ from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.core.security import get_current_user
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.alert import AlertResolutionRequest
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -77,7 +78,7 @@ async def acknowledge_alert(
|
||||
@router.post("/{alert_id}/resolve")
|
||||
async def resolve_alert(
|
||||
alert_id: int,
|
||||
resolution: str,
|
||||
payload: AlertResolutionRequest,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -85,12 +86,12 @@ async def resolve_alert(
|
||||
alert = result.scalar_one_or_none()
|
||||
|
||||
if not alert:
|
||||
return {"error": "Alert not found"}
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
alert.status = AlertStatus.RESOLVED
|
||||
alert.resolved_by = current_user.id
|
||||
alert.resolved_at = datetime.now(UTC)
|
||||
alert.resolution_notes = resolution
|
||||
alert.resolution_notes = payload.resolution
|
||||
await db.commit()
|
||||
|
||||
return {"message": "Alert resolved", "alert": alert.to_dict()}
|
||||
@@ -101,25 +102,44 @@ async def get_alert_stats(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
critical_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
result = await db.execute(
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == AlertStatus.ACTIVE),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info"),
|
||||
)
|
||||
)
|
||||
warning_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
info_query = select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == AlertStatus.ACTIVE,
|
||||
)
|
||||
|
||||
critical_result = await db.execute(critical_query)
|
||||
warning_result = await db.execute(warning_query)
|
||||
info_result = await db.execute(info_query)
|
||||
row = result.one()
|
||||
|
||||
return {
|
||||
"critical": critical_result.scalar() or 0,
|
||||
"warning": warning_result.scalar() or 0,
|
||||
"info": info_result.scalar() or 0,
|
||||
"critical": row.critical or 0,
|
||||
"warning": row.warning or 0,
|
||||
"info": row.info or 0,
|
||||
}
|
||||
|
||||
@@ -22,16 +22,161 @@ def _parse_dt(value: Optional[str]) -> Optional[datetime]:
|
||||
if not value:
|
||||
return None
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
def _event_filters(
|
||||
*,
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
peer_asn: Optional[int],
|
||||
collector: Optional[str],
|
||||
event_type: Optional[str],
|
||||
source: Optional[str],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
if source:
|
||||
filters.append(BGPObservation.source == source)
|
||||
if prefix:
|
||||
filters.append(BGPObservation.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPObservation.origin_asn == origin_asn)
|
||||
if peer_asn is not None:
|
||||
filters.append(BGPObservation.peer_asn == peer_asn)
|
||||
if collector:
|
||||
filters.append(BGPObservation.collector == collector)
|
||||
if event_type:
|
||||
filters.append(BGPObservation.event_type == event_type)
|
||||
if time_from:
|
||||
filters.append(BGPObservation.observed_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPObservation.observed_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _matches_time(value: Optional[datetime], time_from: Optional[datetime], time_to: Optional[datetime]) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if time_from and value < time_from:
|
||||
return False
|
||||
if time_to and value > time_to:
|
||||
return False
|
||||
return True
|
||||
def _anomaly_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
anomaly_type: Optional[str],
|
||||
status: Optional[str],
|
||||
prefix: Optional[str],
|
||||
origin_asn: Optional[int],
|
||||
time_from: Optional[datetime],
|
||||
time_to: Optional[datetime],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
filters.append(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
filters.append(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
filters.append(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
filters.append(BGPAnomaly.origin_asn == origin_asn)
|
||||
if time_from:
|
||||
filters.append(BGPAnomaly.created_at >= time_from)
|
||||
if time_to:
|
||||
filters.append(BGPAnomaly.created_at <= time_to)
|
||||
return filters
|
||||
|
||||
|
||||
def _incident_filters(
|
||||
*,
|
||||
severity: Optional[str],
|
||||
incident_type: Optional[str],
|
||||
status: Optional[str],
|
||||
):
|
||||
filters = []
|
||||
if severity:
|
||||
filters.append(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
filters.append(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
filters.append(BGPIncident.status == status)
|
||||
return filters
|
||||
|
||||
|
||||
async def _build_event_summary_payload(db: AsyncSession) -> dict:
|
||||
base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
|
||||
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*base_filters)
|
||||
)
|
||||
collectors_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.collector))).where(
|
||||
*base_filters, BGPObservation.collector.isnot(None)
|
||||
)
|
||||
)
|
||||
prefixes_result = await db.execute(
|
||||
select(func.count(func.distinct(BGPObservation.prefix))).where(
|
||||
*base_filters, BGPObservation.prefix.isnot(None)
|
||||
)
|
||||
)
|
||||
type_result = await db.execute(
|
||||
select(BGPObservation.event_type, func.count(BGPObservation.id))
|
||||
.where(*base_filters)
|
||||
.group_by(BGPObservation.event_type)
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"collector_count": collectors_result.scalar() or 0,
|
||||
"prefix_count": prefixes_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
async def _build_incident_summary_payload(db: AsyncSession) -> dict:
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events")
|
||||
@@ -49,41 +194,36 @@ async def list_bgp_events(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = (
|
||||
select(BGPObservation)
|
||||
.where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
)
|
||||
if source:
|
||||
stmt = stmt.where(BGPObservation.source == source)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
|
||||
filtered = []
|
||||
for record in records:
|
||||
if prefix and record.prefix != prefix:
|
||||
continue
|
||||
if origin_asn is not None and record.origin_asn != origin_asn:
|
||||
continue
|
||||
if peer_asn is not None and record.peer_asn != peer_asn:
|
||||
continue
|
||||
if collector and record.collector != collector:
|
||||
continue
|
||||
if event_type and record.event_type != event_type:
|
||||
continue
|
||||
if (dt_from or dt_to) and not _matches_time(record.observed_at, dt_from, dt_to):
|
||||
continue
|
||||
filtered.append(record)
|
||||
|
||||
filters = _event_filters(
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
peer_asn=peer_asn,
|
||||
collector=collector,
|
||||
event_type=event_type,
|
||||
source=source,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
count_result = await db.execute(
|
||||
select(func.count(BGPObservation.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPObservation)
|
||||
.where(*filters)
|
||||
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(filtered),
|
||||
"total": count_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in filtered[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -92,21 +232,7 @@ async def get_bgp_event_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
result = await db.execute(select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES)))
|
||||
records = result.scalars().all()
|
||||
|
||||
collectors = sorted({record.collector for record in records if record.collector})
|
||||
prefixes = sorted({record.prefix for record in records if record.prefix})
|
||||
by_type: dict[str, int] = {}
|
||||
for record in records:
|
||||
by_type[record.event_type] = by_type.get(record.event_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"collector_count": len(collectors),
|
||||
"prefix_count": len(prefixes),
|
||||
"by_type": by_type,
|
||||
}
|
||||
return await _build_event_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
@@ -138,6 +264,32 @@ async def get_bgp_collector_summary(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/overview/summary")
|
||||
async def get_bgp_overview_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
event_summary = await _build_event_summary_payload(db)
|
||||
anomaly_summary = await _build_anomaly_summary_payload(db)
|
||||
incident_summary = await _build_incident_summary_payload(db)
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
return {
|
||||
"incidentSummary": incident_summary,
|
||||
"anomalySummary": anomaly_summary,
|
||||
"eventSummary": event_summary,
|
||||
"collectorSummary": {
|
||||
"total": len(collectors),
|
||||
"active_collectors": len(active_collectors),
|
||||
"observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
|
||||
"observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
|
||||
"recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
|
||||
"recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/events/{event_id}")
|
||||
async def get_bgp_event(
|
||||
event_id: int,
|
||||
@@ -164,31 +316,35 @@ async def list_bgp_anomalies(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPAnomaly.severity == severity)
|
||||
if anomaly_type:
|
||||
stmt = stmt.where(BGPAnomaly.anomaly_type == anomaly_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPAnomaly.status == status)
|
||||
if prefix:
|
||||
stmt = stmt.where(BGPAnomaly.prefix == prefix)
|
||||
if origin_asn is not None:
|
||||
stmt = stmt.where(BGPAnomaly.origin_asn == origin_asn)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
dt_from = _parse_dt(time_from)
|
||||
dt_to = _parse_dt(time_to)
|
||||
if dt_from or dt_to:
|
||||
records = [record for record in records if _matches_time(record.created_at, dt_from, dt_to)]
|
||||
|
||||
filters = _anomaly_filters(
|
||||
severity=severity,
|
||||
anomaly_type=anomaly_type,
|
||||
status=status,
|
||||
prefix=prefix,
|
||||
origin_asn=origin_asn,
|
||||
time_from=dt_from,
|
||||
time_to=dt_to,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.where(*filters)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -197,29 +353,7 @@ async def get_bgp_anomaly_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.severity)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
|
||||
.group_by(BGPAnomaly.status)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_anomaly_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/anomalies/{anomaly_id}")
|
||||
@@ -244,22 +378,29 @@ async def list_bgp_incidents(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
if severity:
|
||||
stmt = stmt.where(BGPIncident.severity == severity)
|
||||
if incident_type:
|
||||
stmt = stmt.where(BGPIncident.incident_type == incident_type)
|
||||
if status:
|
||||
stmt = stmt.where(BGPIncident.status == status)
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = result.scalars().all()
|
||||
filters = _incident_filters(
|
||||
severity=severity,
|
||||
incident_type=incident_type,
|
||||
status=status,
|
||||
)
|
||||
offset = (page - 1) * page_size
|
||||
total_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(*filters)
|
||||
)
|
||||
data_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.where(*filters)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.offset(offset)
|
||||
.limit(page_size)
|
||||
)
|
||||
records = data_result.scalars().all()
|
||||
|
||||
return {
|
||||
"total": len(records),
|
||||
"total": total_result.scalar() or 0,
|
||||
"page": page,
|
||||
"page_size": page_size,
|
||||
"data": [record.to_dict() for record in records[offset : offset + page_size]],
|
||||
"data": [record.to_dict() for record in records],
|
||||
}
|
||||
|
||||
|
||||
@@ -268,29 +409,7 @@ async def get_bgp_incident_summary(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
total_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
type_result = await db.execute(
|
||||
select(BGPIncident.incident_type, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.incident_type)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.severity)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
status_result = await db.execute(
|
||||
select(BGPIncident.status, func.count(BGPIncident.id))
|
||||
.group_by(BGPIncident.status)
|
||||
.order_by(func.count(BGPIncident.id).desc())
|
||||
)
|
||||
|
||||
return {
|
||||
"total": total_result.scalar() or 0,
|
||||
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
|
||||
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
|
||||
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
|
||||
}
|
||||
return await _build_incident_summary_payload(db)
|
||||
|
||||
|
||||
@router.get("/incidents/{incident_id}")
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy import select, func, text
|
||||
from sqlalchemy import case, select, func, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
@@ -118,58 +118,77 @@ async def get_stats(
|
||||
built_in_count = len(COLLECTOR_INFO)
|
||||
built_in_active = built_in_count # Built-in are always "active" for counting purposes
|
||||
|
||||
# Count custom configs from database
|
||||
result = await db.execute(select(func.count(DataSourceConfig.id)))
|
||||
custom_count = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
|
||||
select(
|
||||
func.count(DataSourceConfig.id).label("custom_count"),
|
||||
func.sum(
|
||||
case((DataSourceConfig.is_active == True, 1), else_=0)
|
||||
).label("custom_active"),
|
||||
)
|
||||
)
|
||||
custom_active = result.scalar() or 0
|
||||
datasource_stats = result.one()
|
||||
custom_count = datasource_stats.custom_count or 0
|
||||
custom_active = datasource_stats.custom_active or 0
|
||||
|
||||
# Total datasources
|
||||
total_datasources = built_in_count + custom_count
|
||||
active_datasources = built_in_active + custom_active
|
||||
|
||||
# Tasks today (from database)
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
tasks_today = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(CollectionTask.id)).where(
|
||||
CollectionTask.status == "success",
|
||||
CollectionTask.started_at >= today_start,
|
||||
select(
|
||||
func.count(CollectionTask.id).label("tasks_today"),
|
||||
func.sum(
|
||||
case(
|
||||
(CollectionTask.status == "success", 1),
|
||||
else_=0,
|
||||
)
|
||||
).label("success_tasks"),
|
||||
)
|
||||
.where(CollectionTask.started_at >= today_start)
|
||||
)
|
||||
success_tasks = result.scalar() or 0
|
||||
task_stats = result.one()
|
||||
tasks_today = task_stats.tasks_today or 0
|
||||
success_tasks = task_stats.success_tasks or 0
|
||||
success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
|
||||
|
||||
# Alerts
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.CRITICAL,
|
||||
Alert.status == "active",
|
||||
select(
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.CRITICAL)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("critical_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.WARNING)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("warning_alerts"),
|
||||
func.sum(
|
||||
case(
|
||||
(
|
||||
(Alert.severity == AlertSeverity.INFO)
|
||||
& (Alert.status == "active"),
|
||||
1,
|
||||
),
|
||||
else_=0,
|
||||
)
|
||||
).label("info_alerts"),
|
||||
)
|
||||
)
|
||||
critical_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.WARNING,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
warning_alerts = result.scalar() or 0
|
||||
|
||||
result = await db.execute(
|
||||
select(func.count(Alert.id)).where(
|
||||
Alert.severity == AlertSeverity.INFO,
|
||||
Alert.status == "active",
|
||||
)
|
||||
)
|
||||
info_alerts = result.scalar() or 0
|
||||
alert_stats = result.one()
|
||||
critical_alerts = alert_stats.critical_alerts or 0
|
||||
warning_alerts = alert_stats.warning_alerts or 0
|
||||
info_alerts = alert_stats.info_alerts or 0
|
||||
|
||||
response = {
|
||||
"total_datasources": total_datasources,
|
||||
|
||||
@@ -3,7 +3,7 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -11,10 +11,17 @@ from app.core.security import get_current_user
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.db.session import get_db
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
from app.models.datasource import DataSource
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.models.task import CollectionTask
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import get_latest_task_id_for_datasource, run_collector_now, sync_datasource_job
|
||||
from app.services.scheduler import (
|
||||
cancel_running_collector_now,
|
||||
get_latest_task_id_for_datasource,
|
||||
run_collector_now,
|
||||
sync_datasource_job,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
|
||||
@@ -34,6 +41,156 @@ def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
|
||||
return datasource.last_run_at + timedelta(minutes=datasource.frequency_minutes) <= now
|
||||
|
||||
|
||||
def _task_rank_column(order_column):
|
||||
return func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=(order_column.desc().nullslast(), CollectionTask.id.desc()),
|
||||
).label("row_num")
|
||||
|
||||
|
||||
async def _load_latest_running_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.started_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.status == "running")
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_completed_tasks(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, CollectionTask]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
_task_rank_column(CollectionTask.completed_at),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {task.datasource_id: task for task in result.scalars().all()}
|
||||
|
||||
|
||||
async def _load_latest_task_ids(
|
||||
db: AsyncSession,
|
||||
datasource_ids: list[int],
|
||||
) -> dict[int, int]:
|
||||
if not datasource_ids:
|
||||
return {}
|
||||
|
||||
ranked_tasks = (
|
||||
select(
|
||||
CollectionTask.id.label("task_id"),
|
||||
CollectionTask.datasource_id.label("datasource_id"),
|
||||
func.row_number().over(
|
||||
partition_by=CollectionTask.datasource_id,
|
||||
order_by=CollectionTask.id.desc(),
|
||||
).label("row_num"),
|
||||
)
|
||||
.where(CollectionTask.datasource_id.in_(datasource_ids))
|
||||
.subquery()
|
||||
)
|
||||
result = await db.execute(
|
||||
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
|
||||
.where(ranked_tasks.c.row_num == 1)
|
||||
)
|
||||
return {datasource_id: task_id for datasource_id, task_id in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_data_counts(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, int]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(CollectedData.source, func.count(CollectedData.id))
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.group_by(CollectedData.source)
|
||||
)
|
||||
return {source: count for source, count in result.all()}
|
||||
|
||||
|
||||
async def _load_datasource_endpoint_overrides(
|
||||
db: AsyncSession,
|
||||
sources: list[str],
|
||||
) -> dict[str, str]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(DataSourceConfig.name, DataSourceConfig.endpoint)
|
||||
.where(DataSourceConfig.name.in_(sources))
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.where(DataSourceConfig.endpoint.isnot(None))
|
||||
)
|
||||
return {
|
||||
name: endpoint
|
||||
for name, endpoint in result.all()
|
||||
if endpoint
|
||||
}
|
||||
|
||||
|
||||
async def _load_datasource_list_context(
|
||||
db: AsyncSession,
|
||||
datasources: list[DataSource],
|
||||
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
|
||||
datasource_ids = [datasource.id for datasource in datasources]
|
||||
sources = [datasource.source for datasource in datasources]
|
||||
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
datasource_by_id = {datasource.id: datasource for datasource in datasources}
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
stale_datasource_ids: list[int] = []
|
||||
for datasource_id, task in running_tasks.items():
|
||||
started_at = task.started_at
|
||||
if started_at is None:
|
||||
continue
|
||||
if started_at.tzinfo is None:
|
||||
started_at = started_at.replace(tzinfo=timezone.utc)
|
||||
if now - started_at > timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
datasource = datasource_by_id.get(datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
stale_datasource_ids.append(datasource_id)
|
||||
|
||||
if stale_datasource_ids:
|
||||
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
|
||||
|
||||
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
|
||||
data_counts = await _load_datasource_data_counts(db, sources)
|
||||
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
|
||||
return running_tasks, completed_tasks, data_counts, endpoint_overrides
|
||||
|
||||
|
||||
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
|
||||
datasource = None
|
||||
try:
|
||||
@@ -52,18 +209,6 @@ async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[Da
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_last_completed_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
.where(CollectionTask.datasource_id == datasource_id)
|
||||
.where(CollectionTask.completed_at.isnot(None))
|
||||
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
|
||||
.order_by(CollectionTask.completed_at.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
|
||||
result = await db.execute(
|
||||
select(CollectionTask)
|
||||
@@ -87,17 +232,152 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
|
||||
if now - started_at <= timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
|
||||
return task
|
||||
|
||||
existing_error = (task.error_message or "").strip()
|
||||
datasource = await db.get(DataSource, datasource_id)
|
||||
if datasource is not None:
|
||||
await fail_and_rollback_stale_running_task(db, datasource, task)
|
||||
else:
|
||||
existing_error = (task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
async def rollback_orphaned_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "orphaned_running_task_after_backend_restart"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
running_task.status = "cancelled"
|
||||
running_task.phase = "cancelled"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
|
||||
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
|
||||
datasource.last_status = "cancelled"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
|
||||
|
||||
async def fail_and_rollback_stale_running_task(
|
||||
db: AsyncSession,
|
||||
datasource: DataSource,
|
||||
running_task: CollectionTask,
|
||||
) -> None:
|
||||
snapshot_result = await db.execute(
|
||||
select(DataSnapshot)
|
||||
.where(
|
||||
DataSnapshot.datasource_id == datasource.id,
|
||||
DataSnapshot.task_id == running_task.id,
|
||||
)
|
||||
.order_by(DataSnapshot.id.desc())
|
||||
.limit(1)
|
||||
)
|
||||
snapshot = snapshot_result.scalar_one_or_none()
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": datasource.source},
|
||||
)
|
||||
|
||||
if snapshot is not None:
|
||||
snapshot.status = "failed"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(timezone.utc)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = "stale_running_task_timeout"
|
||||
snapshot.summary = summary
|
||||
|
||||
if snapshot.parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": snapshot.parent_snapshot_id},
|
||||
)
|
||||
|
||||
existing_error = (running_task.error_message or "").strip()
|
||||
stale_reason = (
|
||||
f"Marked failed automatically after stale running timeout "
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
|
||||
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
|
||||
)
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.completed_at = now
|
||||
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
running_task.status = "failed"
|
||||
running_task.phase = "failed"
|
||||
running_task.completed_at = datetime.now(timezone.utc)
|
||||
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
|
||||
datasource.last_status = "failed"
|
||||
datasource.last_run_at = datetime.now(timezone.utc)
|
||||
await db.commit()
|
||||
return None
|
||||
|
||||
|
||||
@router.get("")
|
||||
@@ -121,14 +401,17 @@ async def list_datasources(
|
||||
|
||||
collector_list = []
|
||||
config = get_data_sources_config()
|
||||
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
|
||||
db,
|
||||
datasources,
|
||||
)
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
last_task = await get_last_completed_task(db, datasource.id)
|
||||
endpoint = await config.get_url(datasource.source, db)
|
||||
data_count_result = await db.execute(
|
||||
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
last_task = completed_tasks.get(datasource.id)
|
||||
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
|
||||
datasource.source,
|
||||
)
|
||||
data_count = data_count_result.scalar() or 0
|
||||
data_count = data_counts.get(datasource.source, 0)
|
||||
|
||||
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
|
||||
last_run = to_iso8601_utc(last_run_at)
|
||||
@@ -137,6 +420,7 @@ async def list_datasources(
|
||||
collector_list.append(
|
||||
{
|
||||
"id": datasource.id,
|
||||
"source": datasource.source,
|
||||
"name": datasource.name,
|
||||
"module": datasource.module,
|
||||
"priority": datasource.priority,
|
||||
@@ -189,9 +473,13 @@ async def trigger_all_datasources(
|
||||
skipped_sources: list[dict] = []
|
||||
failed_sources: list[dict] = []
|
||||
now = datetime.now(timezone.utc)
|
||||
running_tasks = await _load_latest_running_tasks(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
|
||||
for datasource in datasources:
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
running_task = running_tasks.get(datasource.id)
|
||||
if running_task is not None:
|
||||
skipped_sources.append(
|
||||
{
|
||||
@@ -219,7 +507,7 @@ async def trigger_all_datasources(
|
||||
)
|
||||
continue
|
||||
|
||||
previous_task_ids[datasource.id] = await get_latest_task_id_for_datasource(datasource.id)
|
||||
previous_task_ids[datasource.id] = None
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
failed_sources.append(
|
||||
@@ -241,13 +529,24 @@ async def trigger_all_datasources(
|
||||
}
|
||||
)
|
||||
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[datasource.id for datasource in datasources],
|
||||
)
|
||||
for datasource_id in previous_task_ids:
|
||||
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
|
||||
|
||||
for _ in range(20):
|
||||
await asyncio.sleep(0.1)
|
||||
pending = [item for item in triggered_sources if item["task_id"] is None]
|
||||
if not pending:
|
||||
break
|
||||
latest_task_ids = await _load_latest_task_ids(
|
||||
db,
|
||||
[item["id"] for item in pending],
|
||||
)
|
||||
for item in pending:
|
||||
task_id = await get_latest_task_id_for_datasource(item["id"])
|
||||
task_id = latest_task_ids.get(item["id"])
|
||||
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
|
||||
item["task_id"] = task_id
|
||||
|
||||
@@ -346,6 +645,7 @@ async def get_datasource_stats(
|
||||
@router.post("/{source_id}/trigger")
|
||||
async def trigger_datasource(
|
||||
source_id: str,
|
||||
force: bool = Query(False),
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
@@ -356,6 +656,26 @@ async def trigger_datasource(
|
||||
if not datasource.is_active:
|
||||
raise HTTPException(status_code=400, detail="Data source is disabled")
|
||||
|
||||
running_task = await get_running_task(db, datasource.id)
|
||||
if running_task is not None and not force:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"reason": "running_task_in_progress",
|
||||
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
|
||||
"task_id": running_task.id,
|
||||
"phase": running_task.phase,
|
||||
"progress": running_task.progress,
|
||||
"records_processed": running_task.records_processed,
|
||||
"total_records": running_task.total_records,
|
||||
},
|
||||
)
|
||||
|
||||
if running_task is not None and force:
|
||||
cancelled = await cancel_running_collector_now(datasource.source)
|
||||
if not cancelled:
|
||||
await rollback_orphaned_running_task(db, datasource, running_task)
|
||||
|
||||
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
|
||||
success = run_collector_now(datasource.source)
|
||||
if not success:
|
||||
@@ -375,6 +695,7 @@ async def trigger_datasource(
|
||||
"source_id": datasource.id,
|
||||
"task_id": task_id,
|
||||
"collector_name": datasource.source,
|
||||
"force": force,
|
||||
"message": f"Collector '{datasource.source}' has been triggered",
|
||||
}
|
||||
|
||||
@@ -412,6 +733,7 @@ async def clear_datasource_data(
|
||||
async def get_task_status(
|
||||
source_id: str,
|
||||
task_id: Optional[int] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
datasource = await get_datasource_record(db, source_id)
|
||||
|
||||
13
backend/app/api/v1/news.py
Normal file
13
backend/app/api/v1/news.py
Normal file
@@ -0,0 +1,13 @@
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from app.services.earth_news import get_earth_news_payload
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/earth-feed")
|
||||
async def get_earth_feed(
|
||||
lat: float | None = Query(None, description="Current Earth view center latitude"),
|
||||
lon: float | None = Query(None, description="Current Earth view center longitude"),
|
||||
):
|
||||
return await get_earth_news_payload(lat=lat, lon=lon)
|
||||
@@ -1,3 +1,4 @@
|
||||
from copy import deepcopy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Optional
|
||||
|
||||
@@ -13,6 +14,7 @@ from app.models.datasource import DataSource
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.user import User
|
||||
from app.services.scheduler import sync_datasource_job
|
||||
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -36,6 +38,7 @@ DEFAULT_SETTINGS = {
|
||||
"max_login_attempts": 5,
|
||||
"password_policy": "medium",
|
||||
},
|
||||
"tv": DEFAULT_TV_SETTINGS,
|
||||
}
|
||||
|
||||
|
||||
@@ -67,8 +70,34 @@ class CollectorSettingsUpdate(BaseModel):
|
||||
frequency_minutes: int = Field(default=60, ge=1, le=10080)
|
||||
|
||||
|
||||
class TVStreamSourceUpdate(BaseModel):
|
||||
id: str = Field(min_length=1, max_length=100)
|
||||
name: str = Field(min_length=1, max_length=200)
|
||||
provider: str = Field(default="Unknown", max_length=100)
|
||||
region: str = Field(default="Global", max_length=100)
|
||||
language: str = Field(default="und", max_length=32)
|
||||
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
|
||||
embed_url: str = ""
|
||||
stream_url: str = ""
|
||||
homepage_url: str = ""
|
||||
poster_url: str = ""
|
||||
youtube_video_id: str = ""
|
||||
youtube_channel: str = ""
|
||||
is_enabled: bool = True
|
||||
is_fallback: bool = False
|
||||
sort_order: int = Field(default=10, ge=0, le=9999)
|
||||
collector_source: Optional[str] = None
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class TVSettingsUpdate(BaseModel):
|
||||
default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
|
||||
auto_fallback: bool = True
|
||||
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
|
||||
|
||||
|
||||
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
|
||||
merged = DEFAULT_SETTINGS[category].copy()
|
||||
merged = deepcopy(DEFAULT_SETTINGS[category])
|
||||
if payload:
|
||||
merged.update(payload)
|
||||
return merged
|
||||
@@ -79,6 +108,26 @@ async def get_setting_record(db: AsyncSession, category: str) -> Optional[System
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
|
||||
if not categories:
|
||||
return {}
|
||||
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category.in_(categories))
|
||||
)
|
||||
records_by_category = {
|
||||
record.category: record
|
||||
for record in result.scalars().all()
|
||||
}
|
||||
return {
|
||||
category: merge_with_defaults(
|
||||
category,
|
||||
records_by_category.get(category).payload if records_by_category.get(category) else None,
|
||||
)
|
||||
for category in categories
|
||||
}
|
||||
|
||||
|
||||
async def get_setting_payload(db: AsyncSession, category: str) -> dict:
|
||||
record = await get_setting_record(db, category)
|
||||
return merge_with_defaults(category, record.payload if record else None)
|
||||
@@ -175,6 +224,25 @@ async def update_security_settings(
|
||||
return {"status": "updated", "security": payload}
|
||||
|
||||
|
||||
@router.get("/tv")
|
||||
async def get_tv_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return {"tv": await get_tv_settings_payload(db)}
|
||||
|
||||
|
||||
@router.put("/tv")
|
||||
async def update_tv_settings(
|
||||
settings: TVSettingsUpdate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = normalize_tv_settings(settings.model_dump())
|
||||
saved = await save_setting_payload(db, "tv", payload)
|
||||
return {"status": "updated", "tv": normalize_tv_settings(saved)}
|
||||
|
||||
|
||||
@router.get("/collectors")
|
||||
async def get_collector_settings(
|
||||
current_user: User = Depends(get_current_user),
|
||||
@@ -212,10 +280,15 @@ async def get_all_settings(
|
||||
):
|
||||
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
|
||||
datasources = result.scalars().all()
|
||||
setting_payloads = await get_setting_payloads(
|
||||
db,
|
||||
["system", "notifications", "security"],
|
||||
)
|
||||
return {
|
||||
"system": await get_setting_payload(db, "system"),
|
||||
"notifications": await get_setting_payload(db, "notifications"),
|
||||
"security": await get_setting_payload(db, "security"),
|
||||
"system": setting_payloads["system"],
|
||||
"notifications": setting_payloads["notifications"],
|
||||
"security": setting_payloads["security"],
|
||||
"tv": await get_tv_settings_payload(db),
|
||||
"collectors": [serialize_collector(datasource) for datasource in datasources],
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
70
backend/app/api/v1/tv.py
Normal file
70
backend/app/api/v1/tv.py
Normal file
@@ -0,0 +1,70 @@
|
||||
from urllib.parse import quote, urljoin
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import get_db
|
||||
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_public_tv_streams(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
return await get_public_tv_payload(db)
|
||||
|
||||
|
||||
@router.get("/proxy")
|
||||
async def proxy_tv_stream(
|
||||
url: str = Query(..., description="Upstream TV stream or manifest URL"),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
payload = await get_public_tv_payload(db)
|
||||
if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
|
||||
raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
|
||||
upstream = await client.get(
|
||||
url,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Referer": "https://tv.cctv.com/live/cctv4/",
|
||||
},
|
||||
)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
|
||||
|
||||
content_type = upstream.headers.get("content-type", "application/octet-stream")
|
||||
raw_content = upstream.content
|
||||
response_url = str(upstream.url)
|
||||
is_manifest = (
|
||||
response_url.endswith(".m3u8")
|
||||
or "mpegurl" in content_type.lower()
|
||||
or raw_content.lstrip().startswith(b"#EXTM3U")
|
||||
)
|
||||
|
||||
headers = {"Cache-Control": "no-store"}
|
||||
|
||||
if is_manifest:
|
||||
manifest_text = upstream.text
|
||||
rewritten_lines: list[str] = []
|
||||
for line in manifest_text.splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
rewritten_lines.append(line)
|
||||
continue
|
||||
absolute_url = urljoin(response_url, stripped)
|
||||
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
|
||||
return Response(
|
||||
content="\n".join(rewritten_lines),
|
||||
media_type="application/vnd.apple.mpegurl",
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
return Response(content=raw_content, media_type=content_type, headers=headers)
|
||||
@@ -6,12 +6,14 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
|
||||
|
||||
from datetime import UTC, datetime
|
||||
import math
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Depends, Query, Response
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy import select, func
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from app.core.collected_data_fields import get_record_field
|
||||
from app.core.countries import get_country_centroid
|
||||
from app.core.satellite_tle import build_tle_lines_from_elements
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.db.session import get_db
|
||||
@@ -23,6 +25,9 @@ from app.services.cable_graph import build_graph_from_data, CableGraph, haversin
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
router = APIRouter()
|
||||
TERRAIN_TILE_URL_TEMPLATE = (
|
||||
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
|
||||
)
|
||||
|
||||
|
||||
# ============== Converter Functions ==============
|
||||
@@ -205,42 +210,84 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def dedupe_satellite_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each satellite identity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
norad_id = metadata.get("norad_cat_id")
|
||||
dedupe_key = (
|
||||
str(norad_id)
|
||||
if norad_id not in (None, "")
|
||||
else str(record.source_id or record.entity_key or record.name or record.id)
|
||||
)
|
||||
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
def _current_collected_data_stmt(source: str):
|
||||
return (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == source)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
|
||||
|
||||
def dedupe_collected_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
"""Keep only the newest record for each collected entity."""
|
||||
latest_by_key: Dict[str, CollectedData] = {}
|
||||
async def _load_current_collected_data(
|
||||
db: AsyncSession,
|
||||
source: str,
|
||||
*,
|
||||
exclude_unknown_name: bool = False,
|
||||
limit: Optional[int] = None,
|
||||
) -> List[CollectedData]:
|
||||
stmt = _current_collected_data_stmt(source)
|
||||
if exclude_unknown_name:
|
||||
stmt = stmt.where(CollectedData.name != "Unknown")
|
||||
if limit is not None:
|
||||
stmt = stmt.limit(limit)
|
||||
|
||||
for record in records:
|
||||
dedupe_key = str(
|
||||
record.source_id
|
||||
or record.entity_key
|
||||
or record.name
|
||||
or record.id
|
||||
)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
result = await db.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
|
||||
|
||||
async def _load_current_collected_data_by_sources(
|
||||
db: AsyncSession,
|
||||
sources: List[str],
|
||||
) -> Dict[str, List[CollectedData]]:
|
||||
if not sources:
|
||||
return {}
|
||||
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source.in_(sources))
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
|
||||
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return grouped_records
|
||||
|
||||
|
||||
def _build_landing_point_cable_maps(
|
||||
relation_records: List[CollectedData],
|
||||
cable_records: List[CollectedData],
|
||||
) -> tuple[Dict[int, List[int]], Dict[int, str]]:
|
||||
city_to_cable_ids_map: Dict[int, List[int]] = {}
|
||||
for relation_record in relation_records:
|
||||
if not relation_record.extra_data:
|
||||
continue
|
||||
city_id = relation_record.extra_data.get("city_id")
|
||||
cable_id = relation_record.extra_data.get("cable_id")
|
||||
if city_id is None or cable_id is None:
|
||||
continue
|
||||
city_to_cable_ids_map.setdefault(city_id, [])
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map: Dict[int, str] = {}
|
||||
for cable_record in cable_records:
|
||||
if not cable_record.extra_data:
|
||||
continue
|
||||
cable_id = cable_record.extra_data.get("cable_id")
|
||||
cable_name = cable_record.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
|
||||
return city_to_cable_ids_map, cable_id_to_name_map
|
||||
|
||||
|
||||
def _filter_known_records(records: List[CollectedData]) -> List[CollectedData]:
|
||||
return [record for record in records if record.name != "Unknown"]
|
||||
|
||||
|
||||
def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
@@ -317,6 +364,215 @@ def convert_gpu_cluster_to_geojson(records: List[CollectedData]) -> Dict[str, An
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def _parse_float(value: Any) -> Optional[float]:
|
||||
try:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
COMPUTE_CENTER_COORDINATE_HINTS = (
|
||||
("el capitan", 37.6819, -121.7681),
|
||||
("livermore", 37.6819, -121.7681),
|
||||
("llnl", 37.6819, -121.7681),
|
||||
("lawrence livermore", 37.6819, -121.7681),
|
||||
("frontier", 35.9319, -84.3107),
|
||||
("oak ridge", 35.9319, -84.3107),
|
||||
("ornl", 35.9319, -84.3107),
|
||||
("aurora", 41.7130, -87.9820),
|
||||
("argonne", 41.7130, -87.9820),
|
||||
("anl", 41.7130, -87.9820),
|
||||
("fugaku", 34.6953, 135.1974),
|
||||
("kobe", 34.6953, 135.1974),
|
||||
("riken", 34.6953, 135.1974),
|
||||
("summit", 35.9319, -84.3107),
|
||||
("leonardo", 44.4949, 11.3426),
|
||||
("bologna", 44.4949, 11.3426),
|
||||
("alps", 46.0037, 8.9511),
|
||||
("lugano", 46.0037, 8.9511),
|
||||
("sunway taihulight", 31.4912, 120.3119),
|
||||
("wuxi", 31.4912, 120.3119),
|
||||
("tianhe-2", 23.1291, 113.2644),
|
||||
("tianhe-2a", 23.1291, 113.2644),
|
||||
("guangzhou", 23.1291, 113.2644),
|
||||
("colossus", 35.1495, -90.0490),
|
||||
("memphis", 35.1495, -90.0490),
|
||||
("xai", 35.1495, -90.0490),
|
||||
)
|
||||
|
||||
|
||||
def _normalize_hint_text(*parts: Any) -> str:
|
||||
return " ".join(
|
||||
str(part).strip().lower()
|
||||
for part in parts
|
||||
if part not in (None, "")
|
||||
)
|
||||
|
||||
|
||||
def _resolve_compute_center_coordinates(
|
||||
record: CollectedData,
|
||||
metadata: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
latitude = _parse_float(get_record_field(record, "latitude"))
|
||||
longitude = _parse_float(get_record_field(record, "longitude"))
|
||||
if latitude not in (None, 0.0) and longitude not in (None, 0.0):
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "precise",
|
||||
"geography_mode": "source_coordinates",
|
||||
"is_estimated": False,
|
||||
"estimated_reason": None,
|
||||
}
|
||||
|
||||
hint_text = _normalize_hint_text(
|
||||
record.name,
|
||||
get_record_field(record, "city"),
|
||||
get_record_field(record, "country"),
|
||||
metadata.get("site"),
|
||||
metadata.get("organization"),
|
||||
metadata.get("operator"),
|
||||
)
|
||||
for needle, resolved_latitude, resolved_longitude in COMPUTE_CENTER_COORDINATE_HINTS:
|
||||
if needle in hint_text:
|
||||
return {
|
||||
"latitude": resolved_latitude,
|
||||
"longitude": resolved_longitude,
|
||||
"location_precision": "estimated_site",
|
||||
"geography_mode": "site_hint",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": f"Matched known site hint: {needle}",
|
||||
}
|
||||
|
||||
centroid = get_country_centroid(get_record_field(record, "country"))
|
||||
if centroid:
|
||||
return {
|
||||
"latitude": centroid.get("latitude"),
|
||||
"longitude": centroid.get("longitude"),
|
||||
"location_precision": "estimated_country",
|
||||
"geography_mode": "country_centroid",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "Estimated from country centroid",
|
||||
}
|
||||
|
||||
return {
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"location_precision": "unknown",
|
||||
"geography_mode": "unknown",
|
||||
"is_estimated": True,
|
||||
"estimated_reason": "No resolvable location hints",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_capacity_band(capacity_value: Optional[float], capacity_unit: str) -> str:
|
||||
if capacity_value is None:
|
||||
return "unknown"
|
||||
|
||||
unit = str(capacity_unit or "").strip().lower()
|
||||
if unit in {"pflop/s", "pflops", "pflop"}:
|
||||
normalized_tflops = capacity_value * 1000
|
||||
elif unit in {"gflop/s", "gflops", "gflop"}:
|
||||
normalized_tflops = capacity_value / 1000
|
||||
else:
|
||||
normalized_tflops = capacity_value
|
||||
|
||||
if normalized_tflops >= 1_000_000:
|
||||
return "exascale"
|
||||
if normalized_tflops >= 100_000:
|
||||
return "ultra"
|
||||
if normalized_tflops >= 10_000:
|
||||
return "large"
|
||||
if normalized_tflops > 0:
|
||||
return "regional"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def convert_compute_centers_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
|
||||
"""Convert compute infrastructure records into a unified GeoJSON layer."""
|
||||
features = []
|
||||
|
||||
for record in records:
|
||||
metadata = record.extra_data or {}
|
||||
coordinate_info = _resolve_compute_center_coordinates(record, metadata)
|
||||
latitude = coordinate_info.get("latitude")
|
||||
longitude = coordinate_info.get("longitude")
|
||||
site_type = (
|
||||
"supercomputer"
|
||||
if record.source == "top500" or record.data_type == "supercomputer"
|
||||
else "gpu_cluster"
|
||||
)
|
||||
if latitude in (None, 0.0) or longitude in (None, 0.0):
|
||||
continue
|
||||
|
||||
if site_type == "supercomputer":
|
||||
capacity_value = _parse_float(get_record_field(record, "rmax"))
|
||||
capacity_unit = "GFlops"
|
||||
else:
|
||||
capacity_value = _parse_float(get_record_field(record, "value"))
|
||||
capacity_unit = str(get_record_field(record, "unit") or "TFlop/s")
|
||||
|
||||
vendor = (
|
||||
metadata.get("manufacturer")
|
||||
or metadata.get("vendor")
|
||||
or metadata.get("gpu_type")
|
||||
)
|
||||
operator = (
|
||||
metadata.get("organization")
|
||||
or metadata.get("operator")
|
||||
or metadata.get("owner")
|
||||
)
|
||||
rank = metadata.get("rank")
|
||||
if rank in (None, "") and site_type == "supercomputer":
|
||||
rank = get_record_field(record, "rank")
|
||||
|
||||
updated_at = to_iso8601_utc(record.reference_date or record.collected_at)
|
||||
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"id": record.id,
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [longitude or 0, latitude or 0],
|
||||
},
|
||||
"properties": {
|
||||
"id": record.id,
|
||||
"source_id": record.source_id,
|
||||
"name": record.name,
|
||||
"site_type": site_type,
|
||||
"country": get_record_field(record, "country"),
|
||||
"city": get_record_field(record, "city"),
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
"operator": operator,
|
||||
"vendor": vendor,
|
||||
"capacity_value": capacity_value,
|
||||
"capacity_unit": capacity_unit,
|
||||
"capacity_band": _normalize_capacity_band(capacity_value, capacity_unit),
|
||||
"rank": rank,
|
||||
"gpu_count": metadata.get("gpu_count"),
|
||||
"gpu_type": metadata.get("gpu_type"),
|
||||
"cores": get_record_field(record, "cores"),
|
||||
"power": get_record_field(record, "power"),
|
||||
"source": record.source,
|
||||
"updated_at": updated_at,
|
||||
"status": "observed",
|
||||
"location_precision": coordinate_info.get("location_precision"),
|
||||
"geography_mode": coordinate_info.get("geography_mode"),
|
||||
"is_estimated": coordinate_info.get("is_estimated", False),
|
||||
"estimated_reason": coordinate_info.get("estimated_reason"),
|
||||
"data_type": "compute_center",
|
||||
"metadata": metadata,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {"type": "FeatureCollection", "features": features}
|
||||
|
||||
|
||||
def convert_bgp_anomalies_to_geojson(
|
||||
records: List[BGPAnomaly],
|
||||
geography_hints: Optional[Dict[str, Dict[str, Any]]] = None,
|
||||
@@ -722,9 +978,7 @@ def convert_bgp_incidents_to_geojson(
|
||||
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
"""获取海底电缆 GeoJSON 数据 (LineString)"""
|
||||
try:
|
||||
stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -742,36 +996,25 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
|
||||
@router.get("/geo/landing-points")
|
||||
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
try:
|
||||
landing_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_result = await db.execute(landing_stmt)
|
||||
records = dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
cable_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cable_result = await db.execute(cable_stmt)
|
||||
cable_records = dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cable_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
],
|
||||
)
|
||||
records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get(
|
||||
"arcgis_cable_landing_relation",
|
||||
[],
|
||||
)
|
||||
cable_records = records_by_source.get("arcgis_cables", [])
|
||||
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cable_records,
|
||||
)
|
||||
|
||||
if not records:
|
||||
raise HTTPException(
|
||||
@@ -786,38 +1029,67 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
|
||||
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
|
||||
async def get_terrarium_tile(z: int, x: int, y: int):
|
||||
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
|
||||
if z < 0 or x < 0 or y < 0:
|
||||
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
|
||||
|
||||
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=20.0,
|
||||
follow_redirects=True,
|
||||
) as client:
|
||||
upstream = await client.get(url)
|
||||
upstream.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.response.status_code,
|
||||
detail=f"Terrain tile upstream error: {exc.response.status_code}",
|
||||
) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"Terrain tile fetch failed: {exc}",
|
||||
) from exc
|
||||
|
||||
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
|
||||
etag = upstream.headers.get("etag")
|
||||
last_modified = upstream.headers.get("last-modified")
|
||||
headers = {
|
||||
"Cache-Control": cache_control,
|
||||
}
|
||||
if etag:
|
||||
headers["ETag"] = etag
|
||||
if last_modified:
|
||||
headers["Last-Modified"] = last_modified
|
||||
|
||||
return Response(
|
||||
content=upstream.content,
|
||||
media_type=upstream.headers.get("content-type", "image/png"),
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/geo/all")
|
||||
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
relation_result = await db.execute(relation_stmt)
|
||||
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids_map = {}
|
||||
for rel in relation_records:
|
||||
if rel.extra_data:
|
||||
city_id = rel.extra_data.get("city_id")
|
||||
cable_id = rel.extra_data.get("cable_id")
|
||||
if city_id is not None and cable_id is not None:
|
||||
if city_id not in city_to_cable_ids_map:
|
||||
city_to_cable_ids_map[city_id] = []
|
||||
if cable_id not in city_to_cable_ids_map[city_id]:
|
||||
city_to_cable_ids_map[city_id].append(cable_id)
|
||||
|
||||
cable_id_to_name_map = {}
|
||||
for cable in cables_records:
|
||||
if cable.extra_data:
|
||||
cable_id = cable.extra_data.get("cable_id")
|
||||
cable_name = cable.name
|
||||
if cable_id and cable_name:
|
||||
cable_id_to_name_map[cable_id] = cable_name
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
],
|
||||
)
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
|
||||
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
|
||||
relation_records,
|
||||
cables_records,
|
||||
)
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -850,17 +1122,12 @@ async def get_satellites_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取卫星 TLE GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"celestrak_tle",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_satellite_records(list(result.scalars().all()))
|
||||
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -878,15 +1145,12 @@ async def get_supercomputers_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 TOP500 超算中心 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"top500",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -904,15 +1168,12 @@ async def get_gpu_clusters_geojson(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取 GPU 集群 GeoJSON 数据"""
|
||||
stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
.order_by(CollectedData.id.desc())
|
||||
records = await _load_current_collected_data(
|
||||
db,
|
||||
"epoch_ai_gpu",
|
||||
exclude_unknown_name=True,
|
||||
limit=limit,
|
||||
)
|
||||
result = await db.execute(stmt)
|
||||
records = dedupe_collected_records(list(result.scalars().all()))
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {"type": "FeatureCollection", "features": [], "count": 0}
|
||||
@@ -924,6 +1185,53 @@ async def get_gpu_clusters_geojson(
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/compute-centers")
|
||||
async def get_compute_centers_geojson(
|
||||
limit: int = Query(200, ge=1, le=1000),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取统一算力中心 GeoJSON 数据"""
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
["top500", "epoch_ai_gpu"],
|
||||
)
|
||||
records = _filter_known_records(
|
||||
records_by_source.get("top500", []) + records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
if limit is not None:
|
||||
records = records[:limit]
|
||||
|
||||
if not records:
|
||||
return {
|
||||
"type": "FeatureCollection",
|
||||
"features": [],
|
||||
"count": 0,
|
||||
"stats": {
|
||||
"total": 0,
|
||||
"supercomputers": 0,
|
||||
"gpu_clusters": 0,
|
||||
},
|
||||
}
|
||||
|
||||
geojson = convert_compute_centers_to_geojson(records)
|
||||
features = geojson.get("features", [])
|
||||
return {
|
||||
**geojson,
|
||||
"count": len(features),
|
||||
"stats": {
|
||||
"total": len(features),
|
||||
"supercomputers": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "supercomputer"
|
||||
),
|
||||
"gpu_clusters": sum(
|
||||
1 for feature in features
|
||||
if feature.get("properties", {}).get("site_type") == "gpu_cluster"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/geo/bgp-anomalies")
|
||||
async def get_bgp_anomalies_geojson(
|
||||
severity: Optional[str] = Query(None),
|
||||
@@ -990,37 +1298,27 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
|
||||
- supercomputers: TOP500 超算
|
||||
- gpu_clusters: GPU 集群
|
||||
"""
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = dedupe_collected_records(list(points_result.scalars().all()))
|
||||
|
||||
satellites_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "celestrak_tle")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
records_by_source = await _load_current_collected_data_by_sources(
|
||||
db,
|
||||
[
|
||||
"arcgis_cables",
|
||||
"arcgis_landing_points",
|
||||
"celestrak_tle",
|
||||
"top500",
|
||||
"epoch_ai_gpu",
|
||||
],
|
||||
)
|
||||
satellites_result = await db.execute(satellites_stmt)
|
||||
satellites_records = dedupe_satellite_records(list(satellites_result.scalars().all()))
|
||||
|
||||
supercomputers_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "top500")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
cables_records = records_by_source.get("arcgis_cables", [])
|
||||
points_records = records_by_source.get("arcgis_landing_points", [])
|
||||
satellites_records = _filter_known_records(
|
||||
records_by_source.get("celestrak_tle", []),
|
||||
)
|
||||
supercomputers_result = await db.execute(supercomputers_stmt)
|
||||
supercomputers_records = dedupe_collected_records(list(supercomputers_result.scalars().all()))
|
||||
|
||||
gpu_stmt = (
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "epoch_ai_gpu")
|
||||
.where(CollectedData.name != "Unknown")
|
||||
supercomputers_records = _filter_known_records(
|
||||
records_by_source.get("top500", []),
|
||||
)
|
||||
gpu_records = _filter_known_records(
|
||||
records_by_source.get("epoch_ai_gpu", []),
|
||||
)
|
||||
gpu_result = await db.execute(gpu_stmt)
|
||||
gpu_records = dedupe_collected_records(list(gpu_result.scalars().all()))
|
||||
|
||||
cables = (
|
||||
convert_cable_to_geojson(cables_records)
|
||||
@@ -1084,13 +1382,8 @@ async def get_cable_graph(db: AsyncSession) -> CableGraph:
|
||||
global _cable_graph
|
||||
|
||||
if _cable_graph is None:
|
||||
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
cables_result = await db.execute(cables_stmt)
|
||||
cables_records = list(cables_result.scalars().all())
|
||||
|
||||
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
points_result = await db.execute(points_stmt)
|
||||
points_records = list(points_result.scalars().all())
|
||||
cables_records = await _load_current_collected_data(db, "arcgis_cables")
|
||||
points_records = await _load_current_collected_data(db, "arcgis_landing_points")
|
||||
|
||||
cables_data = convert_cable_to_geojson(cables_records)
|
||||
points_data = convert_landing_point_to_geojson(points_records)
|
||||
|
||||
@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
|
||||
"fao_landing_points": "fao.landing_point_url",
|
||||
"telegeography_cables": "telegeography.cable_url",
|
||||
"telegeography_landing": "telegeography.landing_point_url",
|
||||
"telegeography_systems": "telegeography.cable_url",
|
||||
"huggingface_models": "huggingface.models_url",
|
||||
"huggingface_datasets": "huggingface.datasets_url",
|
||||
"huggingface_spaces": "huggingface.spaces_url",
|
||||
@@ -23,11 +24,13 @@ COLLECTOR_URL_KEYS = {
|
||||
"top500": "top500.url",
|
||||
"epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
|
||||
"spacetrack_tle": "spacetrack.tle_query_url",
|
||||
"celestrak_tle": "celestrak.base_url",
|
||||
"ris_live_bgp": "ris_live.url",
|
||||
"bgpstream_bgp": "bgpstream.url",
|
||||
"iptoasn_prefix_geo": "iptoasn.combined_url",
|
||||
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
|
||||
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
|
||||
"news_live_streams": "news_live_streams.channels_url",
|
||||
}
|
||||
|
||||
|
||||
@@ -41,18 +44,22 @@ class DataSourcesConfig:
|
||||
with open(config_path, "r") as f:
|
||||
self._yaml_config = yaml.safe_load(f) or {}
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
def get_yaml_value(self, key: str):
|
||||
if not key:
|
||||
return ""
|
||||
return None
|
||||
|
||||
parts = key.split(".")
|
||||
value = self._yaml_config
|
||||
for part in parts:
|
||||
if isinstance(value, dict):
|
||||
value = value.get(part, "")
|
||||
value = value.get(part)
|
||||
else:
|
||||
return ""
|
||||
return None
|
||||
return value
|
||||
|
||||
def get_yaml_url(self, collector_name: str) -> str:
|
||||
key = COLLECTOR_URL_KEYS.get(collector_name, "")
|
||||
value = self.get_yaml_value(key)
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
async def get_url(self, collector_name: str, db) -> str:
|
||||
|
||||
@@ -2,53 +2,95 @@
|
||||
# All external data source URLs should be configured here
|
||||
|
||||
arcgis:
|
||||
# ArcGIS 海缆 GeoJSON 查询接口
|
||||
cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
|
||||
# ArcGIS 登陆点 GeoJSON 查询接口
|
||||
landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
|
||||
# ArcGIS 海缆与登陆点关联关系查询接口
|
||||
cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
|
||||
|
||||
fao:
|
||||
# FAO 登陆点 CSV 下载地址
|
||||
landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
telegeography:
|
||||
# TeleGeography 海缆/系统主数据源,当前使用 GitHub 镜像 JSON
|
||||
cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
# TeleGeography 登陆点主数据源,当前使用 GitHub 镜像 JSON
|
||||
landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
# TeleGeography 历史 API 存档,用于 cable collector 的 fallback
|
||||
archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
|
||||
# TeleGeography 官网页面,用于 cable collector 的最终 HTML 抓取 fallback
|
||||
live_map_url: "https://www.submarinecablemap.com"
|
||||
|
||||
huggingface:
|
||||
# Hugging Face 模型目录 API
|
||||
models_url: "https://huggingface.co/api/models"
|
||||
# Hugging Face 数据集目录 API
|
||||
datasets_url: "https://huggingface.co/api/datasets"
|
||||
# Hugging Face Spaces 目录 API
|
||||
spaces_url: "https://huggingface.co/api/spaces"
|
||||
|
||||
cloudflare:
|
||||
# Cloudflare Radar 设备类型摘要接口
|
||||
radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
|
||||
# Cloudflare Radar 请求量时间序列接口
|
||||
radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
|
||||
# Cloudflare Radar 热点地理位置接口
|
||||
radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
|
||||
|
||||
peeringdb:
|
||||
# PeeringDB IXP API
|
||||
ixp_url: "https://www.peeringdb.com/api/ix"
|
||||
# PeeringDB Network API
|
||||
network_url: "https://www.peeringdb.com/api/net"
|
||||
# PeeringDB Facility API
|
||||
facility_url: "https://www.peeringdb.com/api/fac"
|
||||
|
||||
top500:
|
||||
# TOP500 榜单页面,用于主表抓取
|
||||
url: "https://top500.org/lists/top500/list/2025/11/"
|
||||
# TOP500 站点根地址,用于拼详情页链接
|
||||
base_url: "https://top500.org"
|
||||
|
||||
epoch_ai:
|
||||
# Epoch AI GPU Cluster 页面
|
||||
gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
|
||||
|
||||
spacetrack:
|
||||
# Space-Track 站点根地址,用于首页访问和登录地址推导
|
||||
base_url: "https://www.space-track.org"
|
||||
# Space-Track TLE 主查询接口
|
||||
tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
|
||||
|
||||
celestrak:
|
||||
# CelesTrak TLE 基础接口,collector 会在其后拼接 GROUP / FORMAT 参数
|
||||
base_url: "https://celestrak.org/NORAD/elements/gp.php"
|
||||
|
||||
ris_live:
|
||||
# RIPE RIS Live 流式订阅地址
|
||||
url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
|
||||
bgpstream:
|
||||
# CAIDA BGPStream Broker API
|
||||
url: "https://broker.bgpstream.caida.org/v2"
|
||||
|
||||
iptoasn:
|
||||
# IPtoASN prefix geography 合并数据下载地址
|
||||
combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
|
||||
|
||||
opengeofeed:
|
||||
# OpenGeoFeed 公共 geofeed CSV
|
||||
public_csv_url: "https://opengeofeed.org/feed/public.csv"
|
||||
|
||||
nro:
|
||||
# NRO delegated stats 下载地址
|
||||
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
|
||||
|
||||
news_live_streams:
|
||||
# IPTV-org 频道元数据 JSON
|
||||
channels_url: "https://iptv-org.github.io/api/channels.json"
|
||||
# IPTV-org 频道播放流 JSON
|
||||
streams_url: "https://iptv-org.github.io/api/streams.json"
|
||||
# IPTV-org 台标 JSON
|
||||
logos_url: "https://iptv-org.github.io/api/logos.json"
|
||||
|
||||
@@ -155,6 +155,13 @@ DEFAULT_DATASOURCES = {
|
||||
"priority": "P1",
|
||||
"frequency_minutes": 1440,
|
||||
},
|
||||
"news_live_streams": {
|
||||
"id": 26,
|
||||
"name": "News Live Streams",
|
||||
"module": "L4",
|
||||
"priority": "P2",
|
||||
"frequency_minutes": 720,
|
||||
},
|
||||
}
|
||||
|
||||
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}
|
||||
|
||||
@@ -95,6 +95,8 @@ async def init_db():
|
||||
import app.models.bgp_observation # noqa: F401
|
||||
import app.models.collected_data # noqa: F401
|
||||
import app.models.system_setting # noqa: F401
|
||||
import app.models.playground_session # noqa: F401
|
||||
import app.models.playground_message # noqa: F401
|
||||
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
@@ -9,6 +9,8 @@ from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.models.system_setting import SystemSetting
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
|
||||
40
backend/app/models/playground_message.py
Normal file
40
backend/app/models/playground_message.py
Normal file
@@ -0,0 +1,40 @@
|
||||
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundMessage(Base):
|
||||
__tablename__ = "playground_messages"
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
public_id = Column(String(64), unique=True, index=True, nullable=False)
|
||||
session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
|
||||
role = Column(String(20), nullable=False)
|
||||
kind = Column(String(20), nullable=False, default="message")
|
||||
status = Column(String(20), nullable=False, default="done")
|
||||
title = Column(String(255), nullable=True)
|
||||
content = Column(Text, nullable=False, default="")
|
||||
thinking_content = Column(Text, nullable=False, default="")
|
||||
meta = Column(JSON, nullable=False, default=list)
|
||||
provider = Column(String(100), nullable=True)
|
||||
model = Column(String(200), nullable=True)
|
||||
request_id = Column(String(100), nullable=True)
|
||||
raw_response = Column(JSON, nullable=False, default=dict)
|
||||
content_blocks = Column(JSON, nullable=False, default=list)
|
||||
text_blocks = Column(JSON, nullable=False, default=list)
|
||||
thinking_blocks = Column(JSON, nullable=False, default=list)
|
||||
sort_order = Column(Integer, nullable=False, default=0, index=True)
|
||||
is_visible = Column(Boolean, nullable=False, default=True)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"
|
||||
27
backend/app/models/playground_session.py
Normal file
27
backend/app/models/playground_session.py
Normal file
@@ -0,0 +1,27 @@
|
||||
from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
|
||||
from sqlalchemy.sql import func
|
||||
|
||||
from app.db.session import Base
|
||||
|
||||
|
||||
class PlaygroundSession(Base):
|
||||
__tablename__ = "playground_sessions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
|
||||
)
|
||||
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
session_key = Column(String(100), nullable=False, default="default")
|
||||
title = Column(String(200), nullable=False, default="Playground 会话")
|
||||
state = Column(JSON, nullable=False, default={})
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
server_default=func.now(),
|
||||
onupdate=func.now(),
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"
|
||||
@@ -3,6 +3,14 @@ from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AIContentBlock(BaseModel):
|
||||
type: str
|
||||
text: str | None = None
|
||||
thinking: str | None = None
|
||||
signature: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAnalysisRequest(BaseModel):
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
@@ -10,18 +18,158 @@ class SituationalAnalysisRequest(BaseModel):
|
||||
observations: list[str] = Field(default_factory=list)
|
||||
constraints: list[str] = Field(default_factory=list)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class BGPBriefRequest(BaseModel):
|
||||
incident_limit: int = Field(default=5, ge=1, le=10)
|
||||
anomaly_limit: int = Field(default=6, ge=1, le=12)
|
||||
collector_limit: int = Field(default=5, ge=1, le=10)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AlertBriefRequest(BaseModel):
|
||||
alert_limit: int = Field(default=8, ge=1, le=20)
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAlertBriefRequest(BaseModel):
|
||||
preferred_model: str | None = Field(default=None, max_length=200)
|
||||
thinking: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SituationalAnalysisResponse(BaseModel):
|
||||
provider: str
|
||||
model: str
|
||||
content: str
|
||||
content_blocks: list[AIContentBlock] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BGPBriefRecordSummary(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None = None
|
||||
generated_at: str
|
||||
|
||||
|
||||
class BGPBriefRecordResponse(BGPBriefRecordSummary):
|
||||
content_markdown: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class SituationalAlertBriefResponse(SituationalAnalysisResponse):
|
||||
title: str
|
||||
objective: str
|
||||
facts: list[str] = Field(default_factory=list)
|
||||
context: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AIProviderStatusResponse(BaseModel):
|
||||
provider: str
|
||||
api: str | None = None
|
||||
enabled: bool
|
||||
configured: bool
|
||||
model: str | None = None
|
||||
base_url: str | None = None
|
||||
|
||||
|
||||
class PlaygroundSessionState(BaseModel):
|
||||
messages: list[dict[str, Any]] = Field(default_factory=list)
|
||||
selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
|
||||
title: str = Field(default="", max_length=200)
|
||||
objective: str = Field(default="", max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
inputValue: str = Field(default="")
|
||||
analysis: dict[str, Any] | None = None
|
||||
latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
|
||||
analysisMeta: dict[str, Any] = Field(default_factory=dict)
|
||||
helpExpanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundSessionUpsertRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str | None = Field(default=None, max_length=200)
|
||||
state: PlaygroundSessionState
|
||||
|
||||
|
||||
class PlaygroundMessageRecord(BaseModel):
|
||||
id: str
|
||||
role: str
|
||||
kind: str = "message"
|
||||
status: str = "done"
|
||||
title: str | None = None
|
||||
content: str = ""
|
||||
thinking_content: str = ""
|
||||
meta: list[str] = Field(default_factory=list)
|
||||
markdown: bool = True
|
||||
provider: str | None = None
|
||||
model: str | None = None
|
||||
request_id: str | None = None
|
||||
raw_response: dict[str, Any] = Field(default_factory=dict)
|
||||
content_blocks: list[dict[str, Any]] = Field(default_factory=list)
|
||||
text_blocks: list[str] = Field(default_factory=list)
|
||||
thinking_blocks: list[str] = Field(default_factory=list)
|
||||
parent_message_id: str | None = None
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundSessionResponse(BaseModel):
|
||||
id: str
|
||||
session_key: str
|
||||
title: str
|
||||
state: PlaygroundSessionState
|
||||
created_at: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
class PlaygroundThreadResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PlaygroundMessageCreateRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
title: str = Field(..., min_length=1, max_length=200)
|
||||
objective: str = Field(..., min_length=1, max_length=1000)
|
||||
constraints: str = Field(default="")
|
||||
input: str = Field(..., min_length=1)
|
||||
selected_preset_key: str = Field(default="bgp-brief", max_length=100)
|
||||
help_expanded: bool = True
|
||||
|
||||
|
||||
class PlaygroundMessageActionResponse(BaseModel):
|
||||
session: PlaygroundSessionResponse
|
||||
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
|
||||
active_message_id: str | None = None
|
||||
|
||||
|
||||
class PlaygroundMessageStopRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageResendRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
|
||||
|
||||
class PlaygroundMessageEditRequest(BaseModel):
|
||||
session_key: str = Field(default="default", min_length=1, max_length=100)
|
||||
user_message_id: str = Field(..., min_length=1, max_length=64)
|
||||
content: str = Field(..., min_length=1)
|
||||
|
||||
5
backend/app/schemas/alert.py
Normal file
5
backend/app/schemas/alert.py
Normal file
@@ -0,0 +1,5 @@
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AlertResolutionRequest(BaseModel):
|
||||
resolution: str = Field(..., min_length=1, max_length=1000)
|
||||
103
backend/app/services/alert_ai_brief.py
Normal file
103
backend/app/services/alert_ai_brief.py
Normal file
@@ -0,0 +1,103 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
|
||||
|
||||
|
||||
def _format_counter(counter: Counter[str], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
async def build_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
alert_limit: int = 8,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(max(alert_limit, 1))
|
||||
)
|
||||
total_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
|
||||
acknowledged_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
|
||||
)
|
||||
resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
total_alerts = total_result.scalar() or 0
|
||||
active_alerts = active_result.scalar() or 0
|
||||
acknowledged_alerts = acknowledged_result.scalar() or 0
|
||||
resolved_alerts = resolved_result.scalar() or 0
|
||||
|
||||
severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
|
||||
status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
|
||||
datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
|
||||
active_datasource_counts = Counter(
|
||||
(item.datasource_name or "未命名数据源")
|
||||
for item in recent_alerts
|
||||
if item.status == AlertStatus.ACTIVE
|
||||
)
|
||||
|
||||
facts = [
|
||||
f"告警总量 {total_alerts} 条,其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
|
||||
f"最近告警严重度分布:{_format_counter(severity_counts)}。",
|
||||
f"最近告警状态分布:{_format_counter(status_counts)}。",
|
||||
f"最近告警数据源分布:{_format_counter(Counter(dict(datasource_counts.most_common(6))))}。",
|
||||
]
|
||||
|
||||
if active_datasource_counts:
|
||||
facts.append(
|
||||
"当前待处理告警主要集中在:"
|
||||
+ _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
|
||||
for item in recent_alerts[:6]
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "alerts",
|
||||
"total_alerts": total_alerts,
|
||||
"active_alerts": active_alerts,
|
||||
"acknowledged_alerts": acknowledged_alerts,
|
||||
"resolved_alerts": resolved_alerts,
|
||||
"severity_distribution": dict(severity_counts),
|
||||
"status_distribution": dict(status_counts),
|
||||
"top_datasources": dict(datasource_counts.most_common(6)),
|
||||
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
|
||||
}
|
||||
|
||||
return (
|
||||
SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍处于 active 状态且高严重度的告警簇。",
|
||||
"不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
|
||||
"如果证据不足,请明确指出缺失的上下文。",
|
||||
],
|
||||
context=context,
|
||||
),
|
||||
facts,
|
||||
context,
|
||||
)
|
||||
259
backend/app/services/bgp_ai_brief.py
Normal file
259
backend/app/services/bgp_ai_brief.py
Normal file
@@ -0,0 +1,259 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.v1.bgp import BGP_SOURCES
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.models.bgp_observation import BGPObservation
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_collectors import build_bgp_collector_coverage
|
||||
from app.services.bgp_enrichment import lookup_prefix_geography
|
||||
|
||||
|
||||
def _format_counter(counter: dict[str, int], empty_text: str = "无") -> str:
|
||||
if not counter:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in counter.items())
|
||||
|
||||
|
||||
def _severity_rank(value: str | None) -> int:
|
||||
order = {
|
||||
"critical": 0,
|
||||
"high": 1,
|
||||
"medium": 2,
|
||||
"low": 3,
|
||||
"info": 4,
|
||||
}
|
||||
return order.get((value or "").lower(), 99)
|
||||
|
||||
|
||||
def _normalize_geo_key(country: str | None, city: str | None) -> str:
|
||||
if city and country:
|
||||
return f"{city}, {country}"
|
||||
return city or country or "未知区域"
|
||||
|
||||
|
||||
def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
|
||||
return {name: count for name, count in counter.most_common(limit) if name}
|
||||
|
||||
|
||||
def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in incidents:
|
||||
for region in item.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
|
||||
return counter
|
||||
|
||||
|
||||
def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
|
||||
counter: Counter[str] = Counter()
|
||||
for item in collectors:
|
||||
counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
|
||||
return counter
|
||||
|
||||
|
||||
def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
|
||||
if not prefix_geographies:
|
||||
return "没有命中 prefix geography 证据。"
|
||||
|
||||
rows = []
|
||||
for prefix, item in list(prefix_geographies.items())[:limit]:
|
||||
region = _normalize_geo_key(item.get("country"), item.get("city"))
|
||||
source = item.get("source") or item.get("geography_mode") or "unknown"
|
||||
as_hint = item.get("asn")
|
||||
as_name = item.get("as_name")
|
||||
as_text = ""
|
||||
if as_hint:
|
||||
as_text = f" / ASN AS{as_hint}"
|
||||
if as_name:
|
||||
as_text += f" ({as_name})"
|
||||
rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
|
||||
return ";".join(rows)
|
||||
|
||||
|
||||
async def build_bgp_brief_request(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
incident_limit: int = 5,
|
||||
anomaly_limit: int = 6,
|
||||
collector_limit: int = 5,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
|
||||
incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(max(incident_limit, 1))
|
||||
)
|
||||
anomalies_result = await db.execute(
|
||||
select(BGPAnomaly)
|
||||
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
|
||||
.limit(max(anomaly_limit, 1))
|
||||
)
|
||||
observations_result = await db.execute(
|
||||
select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
|
||||
)
|
||||
incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
|
||||
incidents = incidents_result.scalars().all()
|
||||
anomalies = anomalies_result.scalars().all()
|
||||
observations = observations_result.scalars().all()
|
||||
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
|
||||
|
||||
total_incidents = incident_count_result.scalar() or 0
|
||||
total_anomalies = anomaly_count_result.scalar() or 0
|
||||
total_observations = len(observations)
|
||||
active_collectors = [item for item in collectors if item["observation_count"] > 0]
|
||||
|
||||
incident_status_counts = Counter((item.status or "unknown") for item in incidents)
|
||||
incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
|
||||
incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
|
||||
anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
|
||||
event_type_counts = Counter((item.event_type or "unknown") for item in observations)
|
||||
incident_region_counts = _collect_incident_regions(incidents)
|
||||
|
||||
top_collectors = sorted(
|
||||
active_collectors,
|
||||
key=lambda item: (
|
||||
-int(item["recent_24h_observation_count"]),
|
||||
-int(item["observation_count"]),
|
||||
str(item["collector"]),
|
||||
),
|
||||
)[: max(collector_limit, 1)]
|
||||
collector_region_counts = _collect_collector_regions(top_collectors)
|
||||
|
||||
prefix_candidates = sorted(
|
||||
{
|
||||
prefix
|
||||
for item in incidents
|
||||
for prefix in (item.affected_prefixes or [])
|
||||
if prefix
|
||||
}
|
||||
| {item.prefix for item in anomalies if item.prefix}
|
||||
)
|
||||
prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
|
||||
geography_region_counts = Counter(
|
||||
_normalize_geo_key(item.get("country"), item.get("city"))
|
||||
for item in prefix_geographies.values()
|
||||
if item.get("country") or item.get("city")
|
||||
)
|
||||
hotspot_region_counts = geography_region_counts + incident_region_counts
|
||||
collector_bias_regions = [
|
||||
region
|
||||
for region, count in collector_region_counts.most_common(3)
|
||||
if count > hotspot_region_counts.get(region, 0)
|
||||
]
|
||||
|
||||
observations_lines: list[str] = [
|
||||
f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
|
||||
f"活跃观测站 {len(active_collectors)} 个;近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}。",
|
||||
f"最近 incidents 严重度分布:{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}。",
|
||||
f"最近 incidents 状态分布:{_format_counter(dict(incident_status_counts))}。",
|
||||
f"最近 incidents 类型分布:{_format_counter(dict(incident_type_counts.most_common(5)))}。",
|
||||
f"最近 anomalies 类型分布:{_format_counter(dict(anomaly_type_counts.most_common(6)))}。",
|
||||
f"观测事件类型分布:{_format_counter(dict(event_type_counts.most_common(6)))}。",
|
||||
]
|
||||
|
||||
if hotspot_region_counts:
|
||||
observations_lines.append(
|
||||
"区域热点事实层:"
|
||||
+ _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
|
||||
+ "。"
|
||||
)
|
||||
|
||||
if prefix_geographies:
|
||||
observations_lines.append("Prefix geography 证据:" + _format_geo_evidence(prefix_geographies))
|
||||
|
||||
if collector_bias_regions:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:重点观测站最近 24h 活跃度更集中在 "
|
||||
+ "、".join(collector_bias_regions)
|
||||
+ ",这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
|
||||
)
|
||||
elif top_collectors:
|
||||
observations_lines.append(
|
||||
"观测偏差提示:当前未发现明显高于区域热点事实层的单一观测站集中区域,但仍需区分 collector coverage 与真实区域风险。"
|
||||
)
|
||||
|
||||
if incidents:
|
||||
observations_lines.append(
|
||||
"最近 incident 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.incident_type} / {item.severity} / {item.status}"
|
||||
f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
|
||||
f" / 观测站 {len(item.affected_collectors or [])} 个"
|
||||
for item in incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if anomalies:
|
||||
observations_lines.append(
|
||||
"最近 anomaly 摘要:" + ";".join(
|
||||
[
|
||||
f"{item.anomaly_type} / {item.severity}"
|
||||
f" / 前缀 {item.prefix or '-'}"
|
||||
f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
|
||||
for item in anomalies
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if top_collectors:
|
||||
observations_lines.append(
|
||||
"重点观测站:" + ";".join(
|
||||
[
|
||||
f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
|
||||
f" / 近24h {item['recent_24h_observation_count']} 条"
|
||||
f" / 前缀 {item['prefix_count']} 个"
|
||||
for item in top_collectors
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "bgp-overview",
|
||||
"incident_total": total_incidents,
|
||||
"anomaly_total": total_anomalies,
|
||||
"observation_total": total_observations,
|
||||
"active_collectors": len(active_collectors),
|
||||
"top_incident_types": dict(incident_type_counts.most_common(5)),
|
||||
"top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
|
||||
"top_event_types": dict(event_type_counts.most_common(6)),
|
||||
"region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
|
||||
"incident_regions": _top_counter_items(incident_region_counts, limit=6),
|
||||
"collector_bias_regions": collector_bias_regions,
|
||||
"prefix_geography_sources": dict(
|
||||
Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
|
||||
),
|
||||
"prefix_geography_sample": {
|
||||
prefix: {
|
||||
"country": item.get("country"),
|
||||
"city": item.get("city"),
|
||||
"source": item.get("source"),
|
||||
"asn": item.get("asn"),
|
||||
"as_name": item.get("as_name"),
|
||||
}
|
||||
for prefix, item in list(prefix_geographies.items())[:8]
|
||||
},
|
||||
}
|
||||
|
||||
return SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
|
||||
observations=observations_lines,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出需要立即关注的高严重度 incident 或异常模式。",
|
||||
"需要单独指出哪些区域结论来自 prefix geography / affected regions,哪些可能受 collector coverage 偏差影响。",
|
||||
"结论应服务值班排障,不要写成泛泛的模型演示文案。",
|
||||
"如果证据不足,要明确指出缺失数据。",
|
||||
],
|
||||
context=context,
|
||||
), observations_lines, context
|
||||
160
backend/app/services/bgp_ai_brief_store.py
Normal file
160
backend/app/services/bgp_ai_brief_store.py
Normal file
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.config import ROOT_DIR
|
||||
from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
|
||||
|
||||
|
||||
_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
|
||||
_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
|
||||
_METADATA_SUFFIX = " -->"
|
||||
_BRIEF_TITLE = "BGP AI 简报"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _StoredBrief:
|
||||
id: str
|
||||
title: str
|
||||
provider: str
|
||||
model: str
|
||||
request_id: str | None
|
||||
generated_at: str
|
||||
content_markdown: str
|
||||
facts: list[str]
|
||||
context: dict[str, Any]
|
||||
path: Path
|
||||
|
||||
|
||||
def _ensure_storage_dir() -> Path:
|
||||
_BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
return _BRIEF_STORAGE_DIR
|
||||
|
||||
|
||||
def _build_metadata_line(metadata: dict[str, Any]) -> str:
|
||||
return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
|
||||
|
||||
|
||||
def _parse_brief_file(path: Path) -> _StoredBrief | None:
|
||||
try:
|
||||
raw_text = path.read_text(encoding="utf-8")
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
first_line, separator, remainder = raw_text.partition("\n")
|
||||
if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
|
||||
return None
|
||||
|
||||
metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
|
||||
|
||||
try:
|
||||
metadata = json.loads(metadata_payload)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
return _StoredBrief(
|
||||
id=str(metadata.get("id") or path.stem),
|
||||
title=str(metadata.get("title") or _BRIEF_TITLE),
|
||||
provider=str(metadata.get("provider") or "-"),
|
||||
model=str(metadata.get("model") or "-"),
|
||||
request_id=metadata.get("request_id"),
|
||||
generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
|
||||
content_markdown=remainder.lstrip("\n"),
|
||||
facts=list(metadata.get("facts") or []),
|
||||
context=dict(metadata.get("context") or {}),
|
||||
path=path,
|
||||
)
|
||||
|
||||
|
||||
def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
|
||||
storage_dir = _ensure_storage_dir()
|
||||
records: list[_StoredBrief] = []
|
||||
|
||||
for path in storage_dir.glob("*.md"):
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is not None:
|
||||
records.append(parsed)
|
||||
|
||||
records.sort(key=lambda item: item.generated_at, reverse=True)
|
||||
|
||||
return [
|
||||
BGPBriefRecordSummary(
|
||||
id=item.id,
|
||||
title=item.title,
|
||||
provider=item.provider,
|
||||
model=item.model,
|
||||
request_id=item.request_id,
|
||||
generated_at=item.generated_at,
|
||||
)
|
||||
for item in records[: max(limit, 1)]
|
||||
]
|
||||
|
||||
|
||||
def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
parsed = _parse_brief_file(path)
|
||||
if parsed is None:
|
||||
return None
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=parsed.id,
|
||||
title=parsed.title,
|
||||
provider=parsed.provider,
|
||||
model=parsed.model,
|
||||
request_id=parsed.request_id,
|
||||
generated_at=parsed.generated_at,
|
||||
content_markdown=parsed.content_markdown,
|
||||
facts=parsed.facts,
|
||||
context=parsed.context,
|
||||
)
|
||||
|
||||
|
||||
def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
|
||||
summaries = list_bgp_brief_records(limit=1)
|
||||
if not summaries:
|
||||
return None
|
||||
return get_bgp_brief_record(summaries[0].id)
|
||||
|
||||
|
||||
def save_bgp_brief_record(
|
||||
analysis: SituationalAnalysisResponse,
|
||||
*,
|
||||
request_id: str | None,
|
||||
facts: list[str] | None = None,
|
||||
context: dict[str, Any] | None = None,
|
||||
generated_at: datetime | None = None,
|
||||
) -> BGPBriefRecordResponse:
|
||||
created_at = generated_at or datetime.now(UTC)
|
||||
brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
|
||||
path = _ensure_storage_dir() / f"{brief_id}.md"
|
||||
|
||||
metadata = {
|
||||
"id": brief_id,
|
||||
"title": _BRIEF_TITLE,
|
||||
"provider": analysis.provider,
|
||||
"model": analysis.model,
|
||||
"request_id": request_id,
|
||||
"generated_at": created_at.isoformat(),
|
||||
"facts": facts or [],
|
||||
"context": context or {},
|
||||
}
|
||||
|
||||
markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
|
||||
path.write_text(markdown_text, encoding="utf-8")
|
||||
|
||||
return BGPBriefRecordResponse(
|
||||
id=brief_id,
|
||||
title=_BRIEF_TITLE,
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
generated_at=created_at.isoformat(),
|
||||
content_markdown=analysis.content,
|
||||
facts=facts or [],
|
||||
context=context or {},
|
||||
)
|
||||
@@ -6,7 +6,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import case, distinct, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
@@ -14,6 +14,16 @@ from app.models.bgp_observation import BGPObservation
|
||||
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
|
||||
|
||||
|
||||
def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
|
||||
filters: list[Any] = [
|
||||
BGPObservation.collector.isnot(None),
|
||||
func.length(func.btrim(BGPObservation.collector)) > 0,
|
||||
]
|
||||
if source_filter:
|
||||
filters.append(BGPObservation.source.in_(source_filter))
|
||||
return filters
|
||||
|
||||
|
||||
async def build_bgp_collector_coverage(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -24,88 +34,148 @@ async def build_bgp_collector_coverage(
|
||||
recent_24h_threshold = now - timedelta(hours=24)
|
||||
recent_7d_threshold = now - timedelta(days=7)
|
||||
|
||||
stmt = select(BGPObservation).order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
|
||||
if source_filter:
|
||||
stmt = stmt.where(BGPObservation.source.in_(source_filter))
|
||||
filters = _collector_base_filters(source_filter)
|
||||
country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
|
||||
city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
|
||||
|
||||
result = await db.execute(stmt)
|
||||
records = list(result.scalars().all())
|
||||
aggregate_stmt = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
func.count(BGPObservation.id).label("observation_count"),
|
||||
func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
|
||||
func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
|
||||
func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
|
||||
func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
|
||||
func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
|
||||
func.max(BGPObservation.observed_at).label("latest_observed_at"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector)
|
||||
)
|
||||
aggregate_rows = (await db.execute(aggregate_stmt)).all()
|
||||
|
||||
latest_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("latest_event_type"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.subquery()
|
||||
)
|
||||
latest_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
latest_subquery.c.collector,
|
||||
latest_subquery.c.latest_event_type,
|
||||
latest_subquery.c.country,
|
||||
latest_subquery.c.city,
|
||||
).where(latest_subquery.c.rn == 1)
|
||||
)
|
||||
).all()
|
||||
|
||||
event_counts_subquery = (
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
BGPObservation.event_type.label("event_type"),
|
||||
func.count(BGPObservation.id).label("count"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=BGPObservation.collector,
|
||||
order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
|
||||
)
|
||||
.label("rn"),
|
||||
)
|
||||
.where(*filters)
|
||||
.group_by(BGPObservation.collector, BGPObservation.event_type)
|
||||
.subquery()
|
||||
)
|
||||
top_event_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
event_counts_subquery.c.collector,
|
||||
event_counts_subquery.c.event_type,
|
||||
event_counts_subquery.c.count,
|
||||
).where(event_counts_subquery.c.rn <= 3)
|
||||
)
|
||||
).all()
|
||||
|
||||
scope_rows = (
|
||||
await db.execute(
|
||||
select(
|
||||
BGPObservation.collector.label("collector"),
|
||||
country_expr.label("country"),
|
||||
city_expr.label("city"),
|
||||
)
|
||||
.where(*filters)
|
||||
.distinct()
|
||||
)
|
||||
).all()
|
||||
|
||||
latest_by_collector = {
|
||||
row.collector: {
|
||||
"latest_event_type": row.latest_event_type,
|
||||
"country": row.country,
|
||||
"city": row.city,
|
||||
}
|
||||
for row in latest_rows
|
||||
}
|
||||
|
||||
scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
|
||||
for row in scope_rows:
|
||||
if row.country:
|
||||
scope_by_collector[row.collector]["countries"].add(row.country)
|
||||
if row.city:
|
||||
scope_by_collector[row.collector]["cities"].add(row.city)
|
||||
|
||||
top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in top_event_rows:
|
||||
top_events_by_collector[row.collector].append(
|
||||
{"event_type": row.event_type, "count": row.count}
|
||||
)
|
||||
|
||||
by_collector: dict[str, dict[str, Any]] = {}
|
||||
for record in records:
|
||||
collector = str(record.collector or "").strip()
|
||||
if not collector:
|
||||
continue
|
||||
for row in aggregate_rows:
|
||||
collector = row.collector
|
||||
latest = latest_by_collector.get(collector, {})
|
||||
fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
|
||||
|
||||
coverage = by_collector.get(collector)
|
||||
if coverage is None:
|
||||
location = record.collector_geo or RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
|
||||
coverage = {
|
||||
"collector": collector,
|
||||
"city": location.get("city"),
|
||||
"country": location.get("country"),
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": set(),
|
||||
"cities": set(),
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
}
|
||||
by_collector[collector] = coverage
|
||||
|
||||
coverage["observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["prefixes"].add(record.prefix)
|
||||
if record.origin_asn is not None:
|
||||
coverage["origin_asns"].add(record.origin_asn)
|
||||
if record.peer_asn is not None:
|
||||
coverage["peer_asns"].add(record.peer_asn)
|
||||
if record.event_type:
|
||||
coverage["event_types"][record.event_type] += 1
|
||||
|
||||
observed_at = record.observed_at
|
||||
if observed_at is not None:
|
||||
aware_observed_at = (
|
||||
observed_at.astimezone(UTC)
|
||||
if observed_at.tzinfo
|
||||
else observed_at.replace(tzinfo=UTC)
|
||||
)
|
||||
if aware_observed_at >= recent_15m_threshold:
|
||||
coverage["recent_15m_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_15m_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_24h_threshold:
|
||||
coverage["recent_24h_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_24h_prefixes"].add(record.prefix)
|
||||
if aware_observed_at >= recent_7d_threshold:
|
||||
coverage["recent_7d_observation_count"] += 1
|
||||
if record.prefix:
|
||||
coverage["recent_7d_prefixes"].add(record.prefix)
|
||||
|
||||
geo = record.collector_geo or {}
|
||||
if geo.get("country"):
|
||||
coverage["countries"].add(geo["country"])
|
||||
if geo.get("city"):
|
||||
coverage["cities"].add(geo["city"])
|
||||
|
||||
current_latest = coverage["latest_observed_at"]
|
||||
if current_latest is None or (
|
||||
record.observed_at is not None and record.observed_at > current_latest
|
||||
):
|
||||
coverage["latest_observed_at"] = record.observed_at
|
||||
coverage["latest_event_type"] = record.event_type
|
||||
by_collector[collector] = {
|
||||
"collector": collector,
|
||||
"city": latest.get("city") or fallback_location.get("city"),
|
||||
"country": latest.get("country") or fallback_location.get("country"),
|
||||
"latitude": fallback_location.get("latitude"),
|
||||
"longitude": fallback_location.get("longitude"),
|
||||
"observation_count": row.observation_count or 0,
|
||||
"prefix_count": row.prefix_count or 0,
|
||||
"origin_asn_count": row.origin_asn_count or 0,
|
||||
"peer_asn_count": row.peer_asn_count or 0,
|
||||
"recent_15m_observation_count": row.recent_15m_observation_count or 0,
|
||||
"recent_24h_observation_count": row.recent_24h_observation_count or 0,
|
||||
"recent_7d_observation_count": row.recent_7d_observation_count or 0,
|
||||
"recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
|
||||
"recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
|
||||
"recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
|
||||
"top_event_types": top_events_by_collector.get(collector, []),
|
||||
"latest_observed_at": to_iso8601_utc(row.latest_observed_at),
|
||||
"latest_event_type": latest.get("latest_event_type"),
|
||||
"baseline_scope": {
|
||||
"countries": sorted(scope["countries"]),
|
||||
"cities": sorted(scope["cities"]),
|
||||
},
|
||||
}
|
||||
|
||||
for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
|
||||
if collector in by_collector:
|
||||
@@ -117,57 +187,22 @@ async def build_bgp_collector_coverage(
|
||||
"latitude": location.get("latitude"),
|
||||
"longitude": location.get("longitude"),
|
||||
"observation_count": 0,
|
||||
"prefixes": set(),
|
||||
"origin_asns": set(),
|
||||
"peer_asns": set(),
|
||||
"event_types": defaultdict(int),
|
||||
"countries": {location.get("country")} if location.get("country") else set(),
|
||||
"cities": {location.get("city")} if location.get("city") else set(),
|
||||
"prefix_count": 0,
|
||||
"origin_asn_count": 0,
|
||||
"peer_asn_count": 0,
|
||||
"recent_15m_observation_count": 0,
|
||||
"recent_24h_observation_count": 0,
|
||||
"recent_7d_observation_count": 0,
|
||||
"recent_15m_prefixes": set(),
|
||||
"recent_24h_prefixes": set(),
|
||||
"recent_7d_prefixes": set(),
|
||||
"recent_15m_prefix_count": 0,
|
||||
"recent_24h_prefix_count": 0,
|
||||
"recent_7d_prefix_count": 0,
|
||||
"top_event_types": [],
|
||||
"latest_observed_at": None,
|
||||
"latest_event_type": None,
|
||||
"baseline_scope": {
|
||||
"countries": [location["country"]] if location.get("country") else [],
|
||||
"cities": [location["city"]] if location.get("city") else [],
|
||||
},
|
||||
}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for collector in sorted(by_collector.keys()):
|
||||
item = by_collector[collector]
|
||||
top_event_types = sorted(
|
||||
item["event_types"].items(),
|
||||
key=lambda pair: (-pair[1], pair[0]),
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"collector": item["collector"],
|
||||
"city": item["city"],
|
||||
"country": item["country"],
|
||||
"latitude": item["latitude"],
|
||||
"longitude": item["longitude"],
|
||||
"observation_count": item["observation_count"],
|
||||
"prefix_count": len(item["prefixes"]),
|
||||
"origin_asn_count": len(item["origin_asns"]),
|
||||
"peer_asn_count": len(item["peer_asns"]),
|
||||
"recent_15m_observation_count": item["recent_15m_observation_count"],
|
||||
"recent_24h_observation_count": item["recent_24h_observation_count"],
|
||||
"recent_7d_observation_count": item["recent_7d_observation_count"],
|
||||
"recent_15m_prefix_count": len(item["recent_15m_prefixes"]),
|
||||
"recent_24h_prefix_count": len(item["recent_24h_prefixes"]),
|
||||
"recent_7d_prefix_count": len(item["recent_7d_prefixes"]),
|
||||
"top_event_types": [
|
||||
{"event_type": event_type, "count": count}
|
||||
for event_type, count in top_event_types[:3]
|
||||
],
|
||||
"latest_observed_at": to_iso8601_utc(item["latest_observed_at"]),
|
||||
"latest_event_type": item["latest_event_type"],
|
||||
"baseline_scope": {
|
||||
"countries": sorted(country for country in item["countries"] if country),
|
||||
"cities": sorted(city for city in item["cities"] if city),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return results
|
||||
return [by_collector[collector] for collector in sorted(by_collector.keys())]
|
||||
|
||||
@@ -7,7 +7,7 @@ from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy import Integer, cast, select, text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.countries import get_country_centroid, normalize_country
|
||||
@@ -231,6 +231,13 @@ async def _lookup_prefix_geography(
|
||||
return results
|
||||
|
||||
|
||||
async def lookup_prefix_geography(
|
||||
db: AsyncSession,
|
||||
prefix_values: list[str],
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
return await _lookup_prefix_geography(db, prefix_values)
|
||||
|
||||
|
||||
async def enrich_bgp_events_for_batch(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
@@ -261,29 +268,40 @@ async def enrich_bgp_events_for_batch(
|
||||
historical_prefix_baseline: dict[str, dict[str, Any]] = {}
|
||||
if prefix_values:
|
||||
previous_result = await db.execute(
|
||||
select(BGPObservation).where(
|
||||
select(
|
||||
BGPObservation.prefix,
|
||||
BGPObservation.origin_asn,
|
||||
BGPObservation.collector,
|
||||
BGPObservation.collector_geo,
|
||||
).where(
|
||||
BGPObservation.source == source,
|
||||
BGPObservation.prefix.in_(prefix_values),
|
||||
)
|
||||
)
|
||||
by_prefix: defaultdict[str, list[BGPObservation]] = defaultdict(list)
|
||||
for observation in previous_result.scalars().all():
|
||||
if observation.prefix:
|
||||
by_prefix[observation.prefix].append(observation)
|
||||
by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||
for prefix, origin_asn, collector, collector_geo in previous_result.all():
|
||||
if prefix:
|
||||
by_prefix[str(prefix)].append(
|
||||
{
|
||||
"origin_asn": origin_asn,
|
||||
"collector": collector,
|
||||
"collector_geo": collector_geo or {},
|
||||
}
|
||||
)
|
||||
|
||||
for prefix, observations in by_prefix.items():
|
||||
unique_origins = sorted(
|
||||
{
|
||||
observation.origin_asn
|
||||
observation["origin_asn"]
|
||||
for observation in observations
|
||||
if observation.origin_asn is not None
|
||||
if observation["origin_asn"] is not None
|
||||
}
|
||||
)
|
||||
unique_collectors = sorted(
|
||||
{
|
||||
observation.collector
|
||||
observation["collector"]
|
||||
for observation in observations
|
||||
if observation.collector
|
||||
if observation["collector"]
|
||||
}
|
||||
)
|
||||
historical_prefix_baseline[prefix] = {
|
||||
@@ -292,9 +310,9 @@ async def enrich_bgp_events_for_batch(
|
||||
"historical_observation_count": len(observations),
|
||||
"historical_regions": _compact_locations(
|
||||
[
|
||||
observation.collector_geo or {}
|
||||
observation["collector_geo"] or {}
|
||||
for observation in observations
|
||||
if observation.collector_geo
|
||||
if observation["collector_geo"]
|
||||
]
|
||||
),
|
||||
}
|
||||
@@ -303,7 +321,13 @@ async def enrich_bgp_events_for_batch(
|
||||
prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
|
||||
if origin_asns:
|
||||
peeringdb_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "peeringdb_network")
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == "peeringdb_network")
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(
|
||||
cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
|
||||
)
|
||||
.order_by(CollectedData.id.desc())
|
||||
)
|
||||
for record in peeringdb_result.scalars().all():
|
||||
metadata = record.extra_data or {}
|
||||
|
||||
@@ -48,14 +48,36 @@ def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
|
||||
return collected
|
||||
|
||||
|
||||
def _dedupe_collected_records(records: list[CollectedData]) -> list[CollectedData]:
|
||||
latest_by_key: dict[str, CollectedData] = {}
|
||||
for record in records:
|
||||
dedupe_key = str(record.source_id or record.entity_key or record.name or record.id)
|
||||
existing = latest_by_key.get(dedupe_key)
|
||||
if existing is None or (record.id or 0) > (existing.id or 0):
|
||||
latest_by_key[dedupe_key] = record
|
||||
return list(latest_by_key.values())
|
||||
async def _load_current_infrastructure_records(
|
||||
db: AsyncSession,
|
||||
) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source.in_(
|
||||
(
|
||||
"arcgis_landing_points",
|
||||
"arcgis_cable_landing_relation",
|
||||
"arcgis_cables",
|
||||
)
|
||||
)
|
||||
)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
|
||||
)
|
||||
grouped_records = {
|
||||
"arcgis_landing_points": [],
|
||||
"arcgis_cable_landing_relation": [],
|
||||
"arcgis_cables": [],
|
||||
}
|
||||
for record in result.scalars().all():
|
||||
grouped_records.setdefault(record.source, []).append(record)
|
||||
|
||||
return (
|
||||
grouped_records["arcgis_landing_points"],
|
||||
grouped_records["arcgis_cable_landing_relation"],
|
||||
grouped_records["arcgis_cables"],
|
||||
)
|
||||
|
||||
|
||||
async def infer_related_infrastructure(
|
||||
@@ -75,19 +97,9 @@ async def infer_related_infrastructure(
|
||||
if not valid_regions:
|
||||
return {"related_cables": [], "related_ixps": []}
|
||||
|
||||
landing_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
|
||||
landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
|
||||
db,
|
||||
)
|
||||
relation_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
|
||||
)
|
||||
cable_result = await db.execute(
|
||||
select(CollectedData).where(CollectedData.source == "arcgis_cables")
|
||||
)
|
||||
|
||||
landing_records = _dedupe_collected_records(list(landing_result.scalars().all()))
|
||||
relation_records = _dedupe_collected_records(list(relation_result.scalars().all()))
|
||||
cable_records = _dedupe_collected_records(list(cable_result.scalars().all()))
|
||||
|
||||
city_to_cable_ids: dict[int, list[int]] = {}
|
||||
for relation in relation_records:
|
||||
|
||||
@@ -35,6 +35,7 @@ from app.services.collectors.bgpstream import BGPStreamBackfillCollector
|
||||
from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
|
||||
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
|
||||
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
|
||||
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
|
||||
|
||||
collector_registry.register(TOP500Collector())
|
||||
collector_registry.register(EpochAIGPUCollector())
|
||||
@@ -61,3 +62,4 @@ collector_registry.register(BGPStreamBackfillCollector())
|
||||
collector_registry.register(IPtoASNPrefixGeoCollector())
|
||||
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
|
||||
collector_registry.register(NRODelegatedPrefixGeoCollector())
|
||||
collector_registry.register(NewsLiveStreamsCollector())
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Base collector class for all data sources"""
|
||||
|
||||
import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Dict, List, Any, Optional
|
||||
from datetime import UTC, datetime
|
||||
@@ -166,6 +167,59 @@ class BaseCollector(ABC):
|
||||
await db.commit()
|
||||
return snapshot.id
|
||||
|
||||
async def _rollback_incomplete_run(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
*,
|
||||
task_id: int,
|
||||
snapshot_id: Optional[int],
|
||||
reason: str,
|
||||
) -> None:
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.data_snapshot import DataSnapshot
|
||||
|
||||
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
|
||||
|
||||
parent_snapshot_id: Optional[int] = None
|
||||
if snapshot_id is not None:
|
||||
snapshot = await db.get(DataSnapshot, snapshot_id)
|
||||
if snapshot:
|
||||
parent_snapshot_id = snapshot.parent_snapshot_id
|
||||
snapshot.status = "cancelled"
|
||||
snapshot.is_current = False
|
||||
snapshot.completed_at = datetime.now(UTC)
|
||||
summary = dict(snapshot.summary or {})
|
||||
summary["rollback"] = True
|
||||
summary["rollback_reason"] = reason
|
||||
snapshot.summary = summary
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = FALSE
|
||||
WHERE source = :source
|
||||
"""
|
||||
),
|
||||
{"source": self.name},
|
||||
)
|
||||
|
||||
if parent_snapshot_id is not None:
|
||||
parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
|
||||
if parent_snapshot:
|
||||
parent_snapshot.is_current = True
|
||||
|
||||
await db.execute(
|
||||
text(
|
||||
"""
|
||||
UPDATE collected_data
|
||||
SET is_current = TRUE
|
||||
WHERE snapshot_id = :snapshot_id
|
||||
"""
|
||||
),
|
||||
{"snapshot_id": parent_snapshot_id},
|
||||
)
|
||||
|
||||
async def run(self, db: AsyncSession) -> Dict[str, Any]:
|
||||
"""Full pipeline: fetch -> transform -> save"""
|
||||
from app.services.collectors.registry import collector_registry
|
||||
@@ -227,7 +281,24 @@ class BaseCollector(ABC):
|
||||
"records_processed": records_count,
|
||||
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
|
||||
}
|
||||
except asyncio.CancelledError:
|
||||
await db.rollback()
|
||||
task.status = "cancelled"
|
||||
task.phase = "cancelled"
|
||||
task.error_message = "Collection cancelled by operator and rolled back"
|
||||
task.completed_at = datetime.now(UTC)
|
||||
if snapshot_id is not None:
|
||||
await self._rollback_incomplete_run(
|
||||
db,
|
||||
task_id=task_id,
|
||||
snapshot_id=snapshot_id,
|
||||
reason="cancelled_by_operator",
|
||||
)
|
||||
await db.commit()
|
||||
await self._publish_task_update(force=True)
|
||||
raise
|
||||
except Exception as e:
|
||||
await db.rollback()
|
||||
task.status = "failed"
|
||||
task.phase = "failed"
|
||||
task.error_message = str(e)
|
||||
@@ -276,20 +347,34 @@ class BaseCollector(ABC):
|
||||
updated_count = 0
|
||||
unchanged_count = 0
|
||||
seen_entity_keys: set[str] = set()
|
||||
previous_current_keys: set[str] = set()
|
||||
progress_commit_interval = 1000
|
||||
|
||||
previous_current_result = await db.execute(
|
||||
select(CollectedData.entity_key).where(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_current_keys = {row[0] for row in previous_current_result.fetchall() if row[0]}
|
||||
previous_current_records = previous_current_result.scalars().all()
|
||||
previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
|
||||
previous_current_map: dict[str, CollectedData] = {}
|
||||
stale_previous_records: list[CollectedData] = []
|
||||
|
||||
for existing_record in previous_current_records:
|
||||
entity_key = existing_record.entity_key
|
||||
if not entity_key:
|
||||
continue
|
||||
if entity_key not in previous_current_map:
|
||||
previous_current_map[entity_key] = existing_record
|
||||
continue
|
||||
stale_previous_records.append(existing_record)
|
||||
|
||||
for stale_record in stale_previous_records:
|
||||
stale_record.is_current = False
|
||||
|
||||
for i, item in enumerate(data):
|
||||
print(
|
||||
f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
|
||||
)
|
||||
raw_metadata = item.get("metadata", {})
|
||||
extra_data = build_dynamic_metadata(
|
||||
raw_metadata,
|
||||
@@ -318,20 +403,9 @@ class BaseCollector(ABC):
|
||||
previous_record = None
|
||||
|
||||
if entity_key and entity_key not in seen_entity_keys:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(
|
||||
CollectedData.source == self.name,
|
||||
CollectedData.entity_key == entity_key,
|
||||
CollectedData.is_current == True,
|
||||
)
|
||||
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
previous_records = result.scalars().all()
|
||||
if previous_records:
|
||||
previous_record = previous_records[0]
|
||||
for old_record in previous_records:
|
||||
old_record.is_current = False
|
||||
previous_record = previous_current_map.get(entity_key)
|
||||
if previous_record is not None:
|
||||
previous_record.is_current = False
|
||||
|
||||
record = CollectedData(
|
||||
snapshot_id=snapshot_id,
|
||||
@@ -375,7 +449,7 @@ class BaseCollector(ABC):
|
||||
seen_entity_keys.add(entity_key)
|
||||
records_added += 1
|
||||
|
||||
if i % 100 == 0:
|
||||
if (i + 1) % progress_commit_interval == 0:
|
||||
await self.update_progress(i + 1, commit=True)
|
||||
|
||||
if snapshot_id is not None:
|
||||
|
||||
@@ -21,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
return "https://celestrak.org/NORAD/elements/gp.php"
|
||||
return self._resolved_url or ""
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
satellite_groups = [
|
||||
@@ -40,7 +40,7 @@ class CelesTrakTLECollector(BaseCollector):
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
for group in satellite_groups:
|
||||
try:
|
||||
url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
|
||||
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
|
||||
response = await client.get(url)
|
||||
|
||||
if response.status_code == 200:
|
||||
|
||||
@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar device type response"""
|
||||
data = []
|
||||
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar traffic timeseries response"""
|
||||
data = []
|
||||
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
|
||||
if CLOUDFLARE_API_TOKEN:
|
||||
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Cloudflare Radar top locations response"""
|
||||
data = []
|
||||
|
||||
@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch Epoch AI GPU clusters data from webpage"""
|
||||
url = "https://epoch.ai/data/gpu-clusters"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
|
||||
frequency_hours = 168
|
||||
data_type = "landing_point"
|
||||
|
||||
csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.csv_url)
|
||||
response = await client.get(self._resolved_url or "")
|
||||
response.raise_for_status()
|
||||
return self.parse_csv(response.text)
|
||||
|
||||
|
||||
@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
|
||||
data_type = "model"
|
||||
base_url = "https://huggingface.co/api/models"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face models API response"""
|
||||
data = []
|
||||
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
|
||||
data_type = "dataset"
|
||||
base_url = "https://huggingface.co/api/datasets"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face datasets API response"""
|
||||
data = []
|
||||
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
|
||||
data_type = "space"
|
||||
base_url = "https://huggingface.co/api/spaces"
|
||||
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
return self._resolved_url or self.base_url
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from httpx import AsyncClient
|
||||
|
||||
async with AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(response.json())
|
||||
|
||||
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Hugging Face Spaces API response"""
|
||||
data = []
|
||||
|
||||
570
backend/app/services/collectors/news_live_streams.py
Normal file
570
backend/app/services/collectors/news_live_streams.py
Normal file
@@ -0,0 +1,570 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.models.datasource_config import DataSourceConfig
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
class NewsLiveStreamsCollector(BaseCollector):
|
||||
"""Collect normalized news live-stream sources from a JSON endpoint."""
|
||||
|
||||
name = "news_live_streams"
|
||||
priority = "P2"
|
||||
module = "L4"
|
||||
frequency_hours = 12
|
||||
data_type = "news_live_stream"
|
||||
fail_on_empty = False
|
||||
|
||||
DEFAULT_TIMEOUT = 45.0
|
||||
DEFAULT_HEADERS = {
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
|
||||
DEFAULT_ADAPTER = "iptv_org"
|
||||
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
|
||||
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
|
||||
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
|
||||
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
|
||||
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
|
||||
|
||||
async def fetch(self) -> list[dict[str, Any]]:
|
||||
request_url = (self._resolved_url or "").strip()
|
||||
if not request_url:
|
||||
return []
|
||||
|
||||
datasource_config = await self._load_datasource_config()
|
||||
effective_config = self._get_effective_config(datasource_config)
|
||||
adapter = str(effective_config.get("adapter") or "").strip().lower()
|
||||
if adapter == "iptv_org":
|
||||
return await self._fetch_iptv_org(request_url, effective_config)
|
||||
|
||||
request_headers = self._build_request_headers(datasource_config)
|
||||
request_config = self._get_request_config(datasource_config)
|
||||
request_params = self._build_request_params(datasource_config)
|
||||
request_json = self._build_request_json_body(datasource_config)
|
||||
request_data = self._build_request_form_body(datasource_config)
|
||||
timeout = self._get_timeout(datasource_config)
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
response = await client.request(
|
||||
request_config["method"],
|
||||
request_url,
|
||||
headers=request_headers,
|
||||
params=request_params or None,
|
||||
json=request_json,
|
||||
data=request_data,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return self.parse_response(
|
||||
response.json(),
|
||||
response_path=request_config["response_path"],
|
||||
)
|
||||
|
||||
async def _load_datasource_config(self) -> DataSourceConfig | None:
|
||||
if not self._db_session:
|
||||
return None
|
||||
|
||||
result = await self._db_session.execute(
|
||||
select(DataSourceConfig)
|
||||
.where(DataSourceConfig.name == self.name)
|
||||
.where(DataSourceConfig.is_active.is_(True))
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = dict(datasource_config.config or {}) if datasource_config else {}
|
||||
if payload:
|
||||
return payload
|
||||
|
||||
yaml_config = get_data_sources_config()
|
||||
return {
|
||||
"adapter": self.DEFAULT_ADAPTER,
|
||||
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
|
||||
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
|
||||
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
|
||||
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
|
||||
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
|
||||
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
|
||||
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
|
||||
}
|
||||
|
||||
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
raw_method = payload.get("method") or payload.get("request_method") or "GET"
|
||||
method = str(raw_method).strip().upper() or "GET"
|
||||
if method not in {"GET", "POST"}:
|
||||
method = "GET"
|
||||
|
||||
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
|
||||
if isinstance(response_path, str):
|
||||
response_path = response_path.strip()
|
||||
else:
|
||||
response_path = None
|
||||
|
||||
return {
|
||||
"method": method,
|
||||
"response_path": response_path or None,
|
||||
}
|
||||
|
||||
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
|
||||
payload = self._get_effective_config(datasource_config)
|
||||
try:
|
||||
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
return self.DEFAULT_TIMEOUT
|
||||
|
||||
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
if datasource_config:
|
||||
headers.update(self._normalize_headers(datasource_config.headers))
|
||||
headers.update(self._build_auth_headers(datasource_config))
|
||||
return headers
|
||||
|
||||
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
|
||||
params: dict[str, Any] = {}
|
||||
if not datasource_config:
|
||||
return params
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
candidate = payload.get("params") or payload.get("query_params")
|
||||
if isinstance(candidate, dict):
|
||||
params.update(candidate)
|
||||
|
||||
if datasource_config.auth_type == "api_key":
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
|
||||
api_key = auth_config.get("api_key")
|
||||
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
|
||||
if api_key and key_name:
|
||||
params[str(key_name)] = api_key
|
||||
|
||||
return params
|
||||
|
||||
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
body = payload.get("json_body")
|
||||
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, (dict, list)):
|
||||
body = candidate
|
||||
return body
|
||||
|
||||
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
|
||||
if not datasource_config:
|
||||
return None
|
||||
|
||||
payload = datasource_config.config or {}
|
||||
form_body = payload.get("form_body")
|
||||
if form_body is not None:
|
||||
return form_body
|
||||
|
||||
if str(payload.get("body_type") or "").lower() == "form":
|
||||
candidate = payload.get("body")
|
||||
if isinstance(candidate, dict):
|
||||
return candidate
|
||||
return None
|
||||
|
||||
def _normalize_headers(self, headers: Any) -> dict[str, str]:
|
||||
if not isinstance(headers, dict):
|
||||
return {}
|
||||
normalized: dict[str, str] = {}
|
||||
for key, value in headers.items():
|
||||
header_name = str(key).strip()
|
||||
if not header_name or value is None:
|
||||
continue
|
||||
normalized[header_name] = str(value)
|
||||
return normalized
|
||||
|
||||
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
|
||||
if not datasource_config:
|
||||
return {}
|
||||
|
||||
auth_type = str(datasource_config.auth_type or "none").lower()
|
||||
auth_config = datasource_config.auth_config or {}
|
||||
if auth_type == "bearer" and auth_config.get("token"):
|
||||
return {"Authorization": f"Bearer {auth_config['token']}"}
|
||||
|
||||
if auth_type == "api_key" and auth_config.get("api_key"):
|
||||
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
|
||||
if location == "query":
|
||||
return {}
|
||||
key_name = auth_config.get("key_name") or "X-API-Key"
|
||||
return {str(key_name): str(auth_config["api_key"])}
|
||||
|
||||
if auth_type == "basic":
|
||||
username = str(auth_config.get("username") or "")
|
||||
password = str(auth_config.get("password") or "")
|
||||
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
|
||||
return {"Authorization": f"Basic {encoded}"}
|
||||
|
||||
return {}
|
||||
|
||||
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
|
||||
if response_path:
|
||||
extracted = self._extract_from_path(response, response_path)
|
||||
if isinstance(extracted, list):
|
||||
return extracted
|
||||
if isinstance(extracted, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = extracted.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return [extracted]
|
||||
|
||||
if isinstance(response, dict):
|
||||
for key in self.RESPONSE_CANDIDATE_KEYS:
|
||||
nested = response.get(key)
|
||||
if isinstance(nested, list):
|
||||
return nested
|
||||
return []
|
||||
|
||||
if isinstance(response, list):
|
||||
return response
|
||||
return []
|
||||
|
||||
def _extract_from_path(self, payload: Any, path: str) -> Any:
|
||||
current = payload
|
||||
for segment in (part.strip() for part in path.split(".") if part.strip()):
|
||||
if isinstance(current, dict):
|
||||
current = current.get(segment)
|
||||
continue
|
||||
if isinstance(current, list):
|
||||
try:
|
||||
current = current[int(segment)]
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
continue
|
||||
return None
|
||||
return current
|
||||
|
||||
def _infer_source_type(self, item: dict[str, Any]) -> str:
|
||||
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
|
||||
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
return explicit
|
||||
|
||||
youtube_video_id = self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
)
|
||||
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
|
||||
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
|
||||
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
|
||||
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
|
||||
|
||||
if youtube_video_id or youtube_channel:
|
||||
return "youtube"
|
||||
if stream_url.endswith(".m3u8"):
|
||||
return "hls"
|
||||
if stream_url:
|
||||
return "video"
|
||||
if embed_url:
|
||||
parsed = urlparse(embed_url)
|
||||
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
|
||||
return "youtube"
|
||||
return "iframe"
|
||||
if homepage_url:
|
||||
return "external"
|
||||
return "iframe"
|
||||
|
||||
def _parse_enabled(self, item: dict[str, Any]) -> bool:
|
||||
if "is_enabled" in item:
|
||||
return self._to_bool(item.get("is_enabled"), default=True)
|
||||
if "enabled" in item:
|
||||
return self._to_bool(item.get("enabled"), default=True)
|
||||
if "active" in item:
|
||||
return self._to_bool(item.get("active"), default=True)
|
||||
if "status" in item:
|
||||
status = str(item.get("status") or "").strip().lower()
|
||||
if status in {"disabled", "inactive", "offline"}:
|
||||
return False
|
||||
if status in {"enabled", "active", "online", "live"}:
|
||||
return True
|
||||
return True
|
||||
|
||||
def _to_bool(self, value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
def _clean_text(self, value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
def _clean_url(self, value: Any) -> str:
|
||||
text = self._clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
|
||||
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
|
||||
news_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
exclude_categories = {
|
||||
self._clean_text(value).lower()
|
||||
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
|
||||
if self._clean_text(value)
|
||||
}
|
||||
try:
|
||||
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
|
||||
except (TypeError, ValueError):
|
||||
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
|
||||
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
try:
|
||||
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
|
||||
except (TypeError, ValueError):
|
||||
timeout = self.DEFAULT_TIMEOUT
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
|
||||
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
|
||||
client,
|
||||
channels_url,
|
||||
streams_url,
|
||||
logos_url,
|
||||
)
|
||||
|
||||
channels = channels_payload if isinstance(channels_payload, list) else []
|
||||
streams = streams_payload if isinstance(streams_payload, list) else []
|
||||
logos = logos_payload if isinstance(logos_payload, list) else []
|
||||
|
||||
logo_by_channel = {
|
||||
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
|
||||
for item in logos
|
||||
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
|
||||
}
|
||||
|
||||
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
|
||||
for stream in streams:
|
||||
if not isinstance(stream, dict):
|
||||
continue
|
||||
channel_id = self._clean_text(stream.get("channel"))
|
||||
if not channel_id:
|
||||
continue
|
||||
streams_by_channel.setdefault(channel_id, []).append(stream)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for channel in channels:
|
||||
if not isinstance(channel, dict):
|
||||
continue
|
||||
|
||||
categories = [
|
||||
self._clean_text(value).lower()
|
||||
for value in (channel.get("categories") or [])
|
||||
if self._clean_text(value)
|
||||
]
|
||||
if news_categories and not any(category in news_categories for category in categories):
|
||||
continue
|
||||
if exclude_categories and any(category in exclude_categories for category in categories):
|
||||
continue
|
||||
if channel.get("is_nsfw") is True:
|
||||
continue
|
||||
if channel.get("closed"):
|
||||
continue
|
||||
|
||||
channel_id = self._clean_text(channel.get("id"))
|
||||
if not channel_id:
|
||||
continue
|
||||
|
||||
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
|
||||
if not stream:
|
||||
continue
|
||||
|
||||
stream_url = self._clean_url(stream.get("url"))
|
||||
if not stream_url:
|
||||
continue
|
||||
|
||||
name = self._clean_text(channel.get("name")) or channel_id
|
||||
notes_parts = [
|
||||
f"Imported from IPTV-org catalog ({channel_id})",
|
||||
f"Categories: {', '.join(categories)}" if categories else "",
|
||||
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
|
||||
]
|
||||
metadata = {
|
||||
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
|
||||
"region": self._clean_text(channel.get("country")) or "Global",
|
||||
"language": "und",
|
||||
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
|
||||
"embed_url": "",
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": self._clean_url(channel.get("website")),
|
||||
"poster_url": logo_by_channel.get(channel_id, ""),
|
||||
"youtube_video_id": "",
|
||||
"youtube_channel": "",
|
||||
"sort_order": 400 + len(normalized),
|
||||
"notes": "; ".join(part for part in notes_parts if part),
|
||||
"is_enabled": True,
|
||||
"collector_adapter": "iptv_org",
|
||||
"channel_id": channel_id,
|
||||
"categories": categories,
|
||||
"quality": self._clean_text(stream.get("quality")),
|
||||
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
|
||||
"stream_referrer": self._clean_text(stream.get("referrer")),
|
||||
"stream_user_agent": self._clean_text(stream.get("user_agent")),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": channel_id,
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
if len(normalized) >= max_sources:
|
||||
break
|
||||
|
||||
return normalized
|
||||
|
||||
async def _gather_iptv_org_payloads(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
channels_url: str,
|
||||
streams_url: str,
|
||||
logos_url: str,
|
||||
) -> tuple[Any, Any, Any]:
|
||||
headers = dict(self.DEFAULT_HEADERS)
|
||||
channels_payload, streams_payload, logos_payload = await asyncio.gather(
|
||||
client.get(channels_url, headers=headers),
|
||||
client.get(streams_url, headers=headers),
|
||||
client.get(logos_url, headers=headers),
|
||||
)
|
||||
channels_payload.raise_for_status()
|
||||
streams_payload.raise_for_status()
|
||||
logos_payload.raise_for_status()
|
||||
return channels_payload.json(), streams_payload.json(), logos_payload.json()
|
||||
|
||||
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
|
||||
if not streams:
|
||||
return None
|
||||
|
||||
def score(stream: dict[str, Any]) -> tuple[int, int]:
|
||||
url = self._clean_url(stream.get("url"))
|
||||
quality = self._clean_text(stream.get("quality")).lower()
|
||||
quality_score = 0
|
||||
if quality.endswith("p"):
|
||||
try:
|
||||
quality_score = int(quality[:-1])
|
||||
except ValueError:
|
||||
quality_score = 0
|
||||
stream_score = 1000 if url.endswith(".m3u8") else 0
|
||||
return stream_score, quality_score
|
||||
|
||||
sorted_streams = sorted(streams, key=score, reverse=True)
|
||||
return sorted_streams[0]
|
||||
|
||||
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
|
||||
candidates = self._extract_candidates(response, response_path)
|
||||
|
||||
normalized: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(candidates):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
stream_id = (
|
||||
item.get("id")
|
||||
or item.get("source_id")
|
||||
or item.get("slug")
|
||||
or item.get("channel_id")
|
||||
or item.get("code")
|
||||
or f"news-live-{index + 1}"
|
||||
)
|
||||
name = self._clean_text(
|
||||
item.get("name")
|
||||
or item.get("title")
|
||||
or item.get("channel")
|
||||
or item.get("display_name")
|
||||
or f"News Live {index + 1}"
|
||||
)
|
||||
if not name:
|
||||
continue
|
||||
|
||||
source_type = self._infer_source_type(item)
|
||||
stream_url = self._clean_url(
|
||||
item.get("stream_url")
|
||||
or item.get("stream")
|
||||
or item.get("playback_url")
|
||||
or item.get("hls_url")
|
||||
or item.get("m3u8_url")
|
||||
)
|
||||
embed_url = self._clean_url(
|
||||
item.get("embed_url")
|
||||
or item.get("embed")
|
||||
or item.get("page_url")
|
||||
or (item.get("url") if source_type == "iframe" else "")
|
||||
)
|
||||
homepage_url = self._clean_url(
|
||||
item.get("homepage_url")
|
||||
or item.get("source_url")
|
||||
or item.get("website")
|
||||
or item.get("url")
|
||||
)
|
||||
metadata = {
|
||||
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
|
||||
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
|
||||
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": embed_url,
|
||||
"stream_url": stream_url,
|
||||
"homepage_url": homepage_url,
|
||||
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
|
||||
"youtube_video_id": self._clean_text(
|
||||
item.get("youtube_video_id")
|
||||
or item.get("video_id")
|
||||
or item.get("youtubeVideoId")
|
||||
),
|
||||
"youtube_channel": self._clean_text(
|
||||
item.get("youtube_channel")
|
||||
or item.get("channel_handle")
|
||||
or item.get("youtubeChannel")
|
||||
),
|
||||
"sort_order": item.get("sort_order", 200 + index),
|
||||
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
|
||||
"is_enabled": self._parse_enabled(item),
|
||||
}
|
||||
|
||||
normalized.append(
|
||||
{
|
||||
"source_id": str(stream_id),
|
||||
"name": name,
|
||||
"description": metadata["notes"],
|
||||
"metadata": metadata,
|
||||
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
return normalized
|
||||
@@ -16,6 +16,7 @@ from typing import Dict, Any, List
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from urllib.parse import urlencode
|
||||
from app.services.collectors.base import HTTPCollector
|
||||
|
||||
|
||||
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
# API key is added to URL as query parameter
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
# Rate limited - wait and retry with exponential backoff
|
||||
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
if PEERINGDB_API_KEY:
|
||||
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
|
||||
@property
|
||||
def request_url(self) -> str:
|
||||
base = self._resolved_url or self.base_url
|
||||
if not PEERINGDB_API_KEY:
|
||||
return base
|
||||
separator = "&" if "?" in base else "?"
|
||||
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
|
||||
|
||||
async def fetch_with_retry(
|
||||
self, max_retries: int = 3, base_delay: float = 2.0
|
||||
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(self.base_url, headers=self.headers)
|
||||
response = await client.get(self.request_url, headers=self.headers)
|
||||
|
||||
if response.status_code == 429:
|
||||
delay = base_delay * (2**attempt)
|
||||
|
||||
@@ -33,7 +33,7 @@ class RISLiveCollector(BaseCollector):
|
||||
|
||||
def _fetch_via_stream(self) -> list[dict[str, Any]]:
|
||||
events: list[dict[str, Any]] = []
|
||||
stream_url = "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
|
||||
stream_url = self._resolved_url or ""
|
||||
subscribe = json.dumps(
|
||||
{
|
||||
"host": "rrc00",
|
||||
|
||||
@@ -7,6 +7,7 @@ API documentation: https://www.space-track.org/documentation
|
||||
import json
|
||||
from typing import Dict, Any, List
|
||||
import httpx
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.services.collectors.base import BaseCollector
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
@@ -21,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
data_type = "satellite_tle"
|
||||
|
||||
@property
|
||||
def base_url(self) -> str:
|
||||
def query_url(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
if self._resolved_url:
|
||||
return self._resolved_url
|
||||
return config.get_yaml_url("spacetrack_tle")
|
||||
|
||||
@property
|
||||
def site_root(self) -> str:
|
||||
config = get_data_sources_config()
|
||||
configured_root = config.get_yaml_value("spacetrack.base_url")
|
||||
if isinstance(configured_root, str) and configured_root:
|
||||
return configured_root.rstrip("/")
|
||||
|
||||
parsed = urlparse(self.query_url)
|
||||
return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
|
||||
|
||||
@property
|
||||
def login_url(self) -> str:
|
||||
return f"{self.site_root}/ajaxauth/login"
|
||||
|
||||
@property
|
||||
def probe_url(self) -> str:
|
||||
return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
from app.core.config import settings
|
||||
|
||||
@@ -47,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept": "application/json, text/html, */*",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Referer": "https://www.space-track.org/",
|
||||
"Referer": f"{self.site_root}/",
|
||||
},
|
||||
) as client:
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -69,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
timeout=120.0,
|
||||
follow_redirects=True,
|
||||
) as alt_client:
|
||||
await alt_client.get("https://www.space-track.org/")
|
||||
await alt_client.get(f"{self.site_root}/")
|
||||
|
||||
form_data = {
|
||||
"username": username,
|
||||
@@ -77,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
|
||||
}
|
||||
alt_login = await alt_client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -86,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
|
||||
|
||||
if alt_login.status_code == 200:
|
||||
tle_response = await alt_client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await alt_client.get(self.probe_url)
|
||||
if tle_response.status_code == 200:
|
||||
data = tle_response.json()
|
||||
print(f"SPACETRACK: Received {len(data)} records via alt method")
|
||||
@@ -98,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
print(f"SPACETRACK: Login failed, using sample data")
|
||||
return self._get_sample_data()
|
||||
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.probe_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
@@ -127,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
},
|
||||
) as client:
|
||||
# First, visit the main page to get any cookies
|
||||
await client.get("https://www.space-track.org/")
|
||||
await client.get(f"{self.site_root}/")
|
||||
|
||||
# Login to get session cookie
|
||||
login_response = await client.post(
|
||||
"https://www.space-track.org/ajaxauth/login",
|
||||
self.login_url,
|
||||
data={
|
||||
"identity": username,
|
||||
"password": password,
|
||||
@@ -146,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
|
||||
return self._get_sample_data()
|
||||
|
||||
# Query for TLE data (get first 1000 satellites)
|
||||
tle_response = await client.get(
|
||||
"https://www.space-track.org/basicspacedata/query"
|
||||
"/class/gp"
|
||||
"/orderby/EPOCH%20desc"
|
||||
"/limit/1000"
|
||||
"/format/json"
|
||||
)
|
||||
tle_response = await client.get(self.query_url)
|
||||
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
|
||||
|
||||
if tle_response.status_code != 200:
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import UTC, datetime
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch submarine cable data from Wayback Machine"""
|
||||
config = get_data_sources_config()
|
||||
# Try multiple data sources
|
||||
sources = [
|
||||
# Wayback Machine archive of TeleGeography
|
||||
"https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
|
||||
# Alternative: Try scraping the page
|
||||
"https://www.submarinecablemap.com",
|
||||
self._resolved_url or "",
|
||||
str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
|
||||
str(config.get_yaml_value("telegeography.live_map_url") or ""),
|
||||
]
|
||||
|
||||
for url in sources:
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch landing point data from GitHub mirror"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch cable system data"""
|
||||
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.get(url)
|
||||
|
||||
@@ -10,6 +10,7 @@ from typing import Dict, Any, List
|
||||
from bs4 import BeautifulSoup
|
||||
import httpx
|
||||
|
||||
from app.core.data_sources import get_data_sources_config
|
||||
from app.services.collectors.base import BaseCollector
|
||||
|
||||
|
||||
@@ -22,7 +23,7 @@ class TOP500Collector(BaseCollector):
|
||||
|
||||
async def fetch(self) -> List[Dict[str, Any]]:
|
||||
"""Fetch TOP500 list data and enrich each row with detail-page metadata."""
|
||||
url = "https://top500.org/lists/top500/list/2025/11/"
|
||||
url = self._resolved_url or ""
|
||||
|
||||
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
@@ -48,11 +49,13 @@ class TOP500Collector(BaseCollector):
|
||||
return await asyncio.gather(*(enrich(entry) for entry in entries))
|
||||
|
||||
def _extract_system_fields(self, system_cell) -> Dict[str, str]:
|
||||
config = get_data_sources_config()
|
||||
top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
|
||||
link = system_cell.find("a")
|
||||
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
|
||||
detail_url = ""
|
||||
if link and link.get("href"):
|
||||
detail_url = f"https://top500.org{link.get('href')}"
|
||||
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
|
||||
|
||||
manufacturer = ""
|
||||
if link and link.next_sibling:
|
||||
|
||||
490
backend/app/services/earth_news.py
Normal file
490
backend/app/services/earth_news.py
Normal file
@@ -0,0 +1,490 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
import hashlib
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
|
||||
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
|
||||
REQUEST_TIMEOUT = 12.0
|
||||
MAX_ITEMS_PER_SOURCE = 6
|
||||
MAX_ITEMS_TOTAL = 12
|
||||
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegionProfile:
|
||||
key: str
|
||||
label: str
|
||||
query: str
|
||||
accent: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NewsFeedSource:
|
||||
id: str
|
||||
name: str
|
||||
region: str
|
||||
feed_url: str
|
||||
homepage_url: str
|
||||
source_type: str = "rss"
|
||||
priority: int = 100
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedNewsItem:
|
||||
id: str
|
||||
title: str
|
||||
summary: str
|
||||
url: str
|
||||
source: str
|
||||
feed_name: str
|
||||
feed_region: str
|
||||
homepage_url: str
|
||||
published_at: datetime | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class CachedRegionFeed:
|
||||
region: str
|
||||
fetched_at: datetime
|
||||
items: list[ParsedNewsItem]
|
||||
sources: list[NewsFeedSource]
|
||||
|
||||
|
||||
REGION_PROFILES: dict[str, RegionProfile] = {
|
||||
"americas": RegionProfile(
|
||||
key="americas",
|
||||
label="美洲焦点",
|
||||
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
|
||||
accent="#79d3ff",
|
||||
),
|
||||
"europe": RegionProfile(
|
||||
key="europe",
|
||||
label="欧洲焦点",
|
||||
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
|
||||
accent="#8fd4ff",
|
||||
),
|
||||
"middle-east-africa": RegionProfile(
|
||||
key="middle-east-africa",
|
||||
label="中东与非洲焦点",
|
||||
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
|
||||
accent="#ffb56a",
|
||||
),
|
||||
"asia-pacific": RegionProfile(
|
||||
key="asia-pacific",
|
||||
label="亚太焦点",
|
||||
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
|
||||
accent="#78f2cf",
|
||||
),
|
||||
"global": RegionProfile(
|
||||
key="global",
|
||||
label="全球焦点",
|
||||
query='"world news" OR geopolitics OR "global affairs"',
|
||||
accent="#d6e6ff",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
|
||||
return (
|
||||
"https://news.google.com/rss/search?q="
|
||||
+ quote(query, safe="")
|
||||
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
|
||||
)
|
||||
|
||||
|
||||
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
|
||||
NewsFeedSource(
|
||||
id="bbc-world",
|
||||
name="BBC World",
|
||||
region="global",
|
||||
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||
homepage_url="https://www.bbc.com/news/world",
|
||||
priority=10,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="dw-top",
|
||||
name="DW Top Stories",
|
||||
region="europe",
|
||||
feed_url="https://rss.dw.com/rdf/rss-en-top",
|
||||
homepage_url="https://www.dw.com/en/top-stories/s-9097",
|
||||
priority=20,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="global-scan",
|
||||
name="Global Monitor / World",
|
||||
region="global",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["global"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=30,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-americas",
|
||||
name="Global Monitor / Americas",
|
||||
region="americas",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["americas"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-europe",
|
||||
name="Global Monitor / Europe",
|
||||
region="europe",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["europe"].query,
|
||||
hl="en-GB",
|
||||
gl="GB",
|
||||
ceid="GB:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-mea",
|
||||
name="Global Monitor / MEA",
|
||||
region="middle-east-africa",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["middle-east-africa"].query,
|
||||
hl="en-US",
|
||||
gl="US",
|
||||
ceid="US:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
NewsFeedSource(
|
||||
id="google-apac",
|
||||
name="Global Monitor / APAC",
|
||||
region="asia-pacific",
|
||||
feed_url=_google_news_feed(
|
||||
REGION_PROFILES["asia-pacific"].query,
|
||||
hl="en-SG",
|
||||
gl="SG",
|
||||
ceid="SG:en",
|
||||
),
|
||||
homepage_url="https://news.google.com/",
|
||||
source_type="aggregated",
|
||||
priority=40,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
|
||||
|
||||
|
||||
def determine_focus_region(lat: float | None, lon: float | None) -> str:
|
||||
if lat is None or lon is None:
|
||||
return "global"
|
||||
if -170 <= lon <= -30:
|
||||
return "americas"
|
||||
if -30 < lon <= 45:
|
||||
return "europe" if lat >= 30 else "middle-east-africa"
|
||||
if 45 < lon <= 150:
|
||||
return "middle-east-africa" if lat < 10 else "asia-pacific"
|
||||
return "asia-pacific"
|
||||
|
||||
|
||||
def get_region_profile(region: str) -> RegionProfile:
|
||||
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
|
||||
|
||||
|
||||
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
|
||||
return sorted(
|
||||
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
|
||||
key=lambda source: (source.priority, source.name),
|
||||
)
|
||||
|
||||
|
||||
def _strip_html(value: str) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
soup = BeautifulSoup(value, "html.parser")
|
||||
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
|
||||
|
||||
|
||||
def _truncate(value: str, limit: int = 180) -> str:
|
||||
text = value.strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _normalize_source_name(raw: str, fallback: str) -> str:
|
||||
text = html.unescape((raw or "").strip())
|
||||
if " - " in text:
|
||||
return text.split(" - ")[-1].strip() or fallback
|
||||
return text or fallback
|
||||
|
||||
|
||||
def _parse_datetime(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
|
||||
for parser in (
|
||||
lambda value: parsedate_to_datetime(value),
|
||||
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
|
||||
):
|
||||
try:
|
||||
parsed = parser(text)
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
return parsed.astimezone(UTC)
|
||||
except Exception:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def _extract_item_text(element: ET.Element, *names: str) -> str:
|
||||
for name in names:
|
||||
node = element.find(name)
|
||||
if node is not None and node.text:
|
||||
return node.text.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
|
||||
root = ET.fromstring(xml_text)
|
||||
items: list[ParsedNewsItem] = []
|
||||
|
||||
rss_items = root.findall("./channel/item")
|
||||
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
|
||||
nodes = rss_items or atom_entries
|
||||
|
||||
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
|
||||
if node.tag.endswith("entry"):
|
||||
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
|
||||
summary = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}summary",
|
||||
"{http://www.w3.org/2005/Atom}content",
|
||||
)
|
||||
link_node = node.find("{http://www.w3.org/2005/Atom}link")
|
||||
link = link_node.get("href", "").strip() if link_node is not None else ""
|
||||
published = _extract_item_text(
|
||||
node,
|
||||
"{http://www.w3.org/2005/Atom}updated",
|
||||
"{http://www.w3.org/2005/Atom}published",
|
||||
)
|
||||
else:
|
||||
title = _extract_item_text(node, "title")
|
||||
summary = _extract_item_text(node, "description", "content")
|
||||
link = _extract_item_text(node, "link")
|
||||
published = _extract_item_text(node, "pubDate", "published", "updated")
|
||||
|
||||
clean_title = html.unescape(title).strip()
|
||||
clean_summary = _truncate(_strip_html(summary), 180)
|
||||
if not clean_title or not link:
|
||||
continue
|
||||
|
||||
item_source = _normalize_source_name(clean_title, source.name)
|
||||
display_title = clean_title
|
||||
if source.source_type == "aggregated" and " - " in clean_title:
|
||||
parts = clean_title.rsplit(" - ", 1)
|
||||
display_title = parts[0].strip()
|
||||
item_source = _normalize_source_name(parts[1], source.name)
|
||||
|
||||
items.append(
|
||||
ParsedNewsItem(
|
||||
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
|
||||
title=display_title,
|
||||
summary=clean_summary,
|
||||
url=link,
|
||||
source=item_source,
|
||||
feed_name=source.name,
|
||||
feed_region=source.region,
|
||||
homepage_url=source.homepage_url,
|
||||
published_at=_parse_datetime(published),
|
||||
)
|
||||
)
|
||||
|
||||
return items
|
||||
|
||||
|
||||
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"id": source.id,
|
||||
"name": source.name,
|
||||
"region": source.region,
|
||||
"homepage_url": source.homepage_url,
|
||||
}
|
||||
for source in sources
|
||||
]
|
||||
|
||||
|
||||
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
|
||||
published_at = item.published_at
|
||||
return {
|
||||
"id": item.id,
|
||||
"title": item.title,
|
||||
"summary": item.summary,
|
||||
"url": item.url,
|
||||
"source": item.source,
|
||||
"feed_name": item.feed_name,
|
||||
"region": item.feed_region,
|
||||
"homepage_url": item.homepage_url,
|
||||
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
|
||||
"is_focus_match": item.feed_region == active_region,
|
||||
}
|
||||
|
||||
|
||||
def _build_payload(
|
||||
*,
|
||||
lat: float | None,
|
||||
lon: float | None,
|
||||
active_region: str,
|
||||
items: list[ParsedNewsItem],
|
||||
sources: list[NewsFeedSource],
|
||||
errors: list[str],
|
||||
stale: bool,
|
||||
generated_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
profile = get_region_profile(active_region)
|
||||
timestamp = generated_at or datetime.now(UTC)
|
||||
return {
|
||||
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"focus": {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"region": active_region,
|
||||
"label": profile.label,
|
||||
"accent": profile.accent,
|
||||
},
|
||||
"sources": _serialize_sources(sources),
|
||||
"items": [_serialize_item(item, active_region=active_region) for item in items],
|
||||
"errors": errors,
|
||||
"stale": stale,
|
||||
}
|
||||
|
||||
|
||||
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
|
||||
deduped: dict[str, ParsedNewsItem] = {}
|
||||
for item in items:
|
||||
key = item.url.strip() or item.title.strip().lower()
|
||||
if key not in deduped:
|
||||
deduped[key] = item
|
||||
|
||||
return sorted(
|
||||
deduped.values(),
|
||||
key=lambda item: (
|
||||
item.feed_region != active_region,
|
||||
item.published_at is None,
|
||||
-(item.published_at.timestamp() if item.published_at else 0),
|
||||
item.feed_name,
|
||||
),
|
||||
)[:MAX_ITEMS_TOTAL]
|
||||
|
||||
|
||||
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
|
||||
cached = _REGION_CACHE.get(region)
|
||||
if not cached:
|
||||
return None
|
||||
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
|
||||
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
|
||||
return None
|
||||
return cached
|
||||
|
||||
|
||||
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
|
||||
_REGION_CACHE[region] = CachedRegionFeed(
|
||||
region=region,
|
||||
fetched_at=datetime.now(UTC),
|
||||
items=list(items),
|
||||
sources=list(sources),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch_source(
|
||||
client: httpx.AsyncClient,
|
||||
source: NewsFeedSource,
|
||||
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
|
||||
try:
|
||||
response = await client.get(source.feed_url)
|
||||
response.raise_for_status()
|
||||
return source, _parse_feed_entries(response.text, source), None
|
||||
except Exception as exc:
|
||||
return source, [], str(exc)
|
||||
|
||||
|
||||
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
|
||||
active_region = determine_focus_region(lat, lon)
|
||||
sources = get_sources_for_region(active_region)
|
||||
errors: list[str] = []
|
||||
|
||||
async with httpx.AsyncClient(
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
follow_redirects=True,
|
||||
headers={"User-Agent": USER_AGENT},
|
||||
) as client:
|
||||
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
|
||||
|
||||
fetched_items: list[ParsedNewsItem] = []
|
||||
for source, items, error in results:
|
||||
if error:
|
||||
errors.append(f"{source.name}: {error}")
|
||||
continue
|
||||
fetched_items.extend(items)
|
||||
|
||||
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
|
||||
if ranked_items:
|
||||
_store_region_cache(active_region, items=ranked_items, sources=sources)
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=ranked_items,
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
|
||||
cached = _get_cached_region_feed(active_region)
|
||||
if cached:
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=cached.items,
|
||||
sources=cached.sources,
|
||||
errors=errors,
|
||||
stale=True,
|
||||
generated_at=cached.fetched_at,
|
||||
)
|
||||
|
||||
return _build_payload(
|
||||
lat=lat,
|
||||
lon=lon,
|
||||
active_region=active_region,
|
||||
items=[],
|
||||
sources=sources,
|
||||
errors=errors,
|
||||
stale=False,
|
||||
)
|
||||
694
backend/app/services/playground_chat_service.py
Normal file
694
backend/app/services/playground_chat_service.py
Normal file
@@ -0,0 +1,694 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
from time import perf_counter
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException, status
|
||||
from sqlalchemy import func, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.session import async_session_factory
|
||||
from app.models.playground_message import PlaygroundMessage
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundMessageEditRequest,
|
||||
PlaygroundMessageActionResponse,
|
||||
PlaygroundMessageCreateRequest,
|
||||
PlaygroundMessageRecord,
|
||||
PlaygroundMessageResendRequest,
|
||||
PlaygroundMessageStopRequest,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
PlaygroundThreadResponse,
|
||||
SituationalAnalysisRequest,
|
||||
)
|
||||
from app.services.ai_client import AIProviderClient
|
||||
from app.services.playground_session_store import _to_response as session_to_response
|
||||
from app.services.playground_session_store import upsert_playground_session
|
||||
|
||||
STREAM_CHUNK_SIZE = 24
|
||||
STREAM_INTERVAL_SECONDS = 0.08
|
||||
THINKING_PREVIEW_SECONDS = 2.6
|
||||
|
||||
|
||||
class _ActiveRun:
|
||||
def __init__(self, task: asyncio.Task[None]) -> None:
|
||||
self.task = task
|
||||
self.stop_requested = asyncio.Event()
|
||||
|
||||
|
||||
_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
|
||||
|
||||
|
||||
async def _get_session_by_key(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
async def _require_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundSession:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
|
||||
return session
|
||||
|
||||
|
||||
async def _require_visible_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
public_id: str,
|
||||
role: str | None = None,
|
||||
) -> PlaygroundMessage:
|
||||
conditions = [
|
||||
PlaygroundMessage.user_id == user_id,
|
||||
PlaygroundMessage.public_id == public_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
]
|
||||
if role is not None:
|
||||
conditions.append(PlaygroundMessage.role == role)
|
||||
|
||||
result = await db.execute(select(PlaygroundMessage).where(*conditions))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
detail = "User message not found" if role == "user" else "Playground message not found"
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
|
||||
return message
|
||||
|
||||
|
||||
def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
|
||||
return PlaygroundMessageRecord(
|
||||
id=message.public_id,
|
||||
role=message.role,
|
||||
kind=message.kind,
|
||||
status=message.status,
|
||||
title=message.title,
|
||||
content=message.content or "",
|
||||
thinking_content=message.thinking_content or "",
|
||||
meta=list(message.meta or []),
|
||||
markdown=message.role != "system",
|
||||
provider=message.provider,
|
||||
model=message.model,
|
||||
request_id=message.request_id,
|
||||
raw_response=dict(message.raw_response or {}),
|
||||
content_blocks=list(message.content_blocks or []),
|
||||
text_blocks=list(message.text_blocks or []),
|
||||
thinking_blocks=list(message.thinking_blocks or []),
|
||||
parent_message_id=parent_public_id,
|
||||
created_at=message.created_at.isoformat(),
|
||||
updated_at=message.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
title: str,
|
||||
state: PlaygroundSessionState | None = None,
|
||||
) -> PlaygroundSession:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
session = result.scalar_one_or_none()
|
||||
if session is not None:
|
||||
if title:
|
||||
session.title = title[:200]
|
||||
if state is not None:
|
||||
session.state = state.model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
payload = PlaygroundSessionUpsertRequest(
|
||||
session_key=session_key,
|
||||
title=title[:200],
|
||||
state=state or PlaygroundSessionState(title=title[:200]),
|
||||
)
|
||||
await upsert_playground_session(db, user_id=user_id, payload=payload)
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
return result.scalar_one()
|
||||
|
||||
|
||||
async def _list_visible_messages(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session_id: int,
|
||||
) -> list[PlaygroundMessage]:
|
||||
result = await db.execute(
|
||||
select(PlaygroundMessage)
|
||||
.where(
|
||||
PlaygroundMessage.session_id == session_id,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
.order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
async def _build_thread_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
) -> PlaygroundThreadResponse:
|
||||
messages = await _list_visible_messages(db, session_id=session.id)
|
||||
id_map = {item.id: item.public_id for item in messages}
|
||||
return PlaygroundThreadResponse(
|
||||
session=session_to_response(session),
|
||||
messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
|
||||
)
|
||||
|
||||
|
||||
async def get_thread(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str,
|
||||
) -> PlaygroundThreadResponse | None:
|
||||
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
|
||||
if session is None:
|
||||
return None
|
||||
return await _build_thread_response(db, session=session)
|
||||
|
||||
|
||||
async def _build_action_response(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
active_message_id: str | None = None,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
thread = await _build_thread_response(db, session=session)
|
||||
return PlaygroundMessageActionResponse(
|
||||
session=thread.session,
|
||||
messages=thread.messages,
|
||||
active_message_id=active_message_id,
|
||||
)
|
||||
|
||||
|
||||
def _collect_constraints(raw_constraints: str) -> list[str]:
|
||||
return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
|
||||
|
||||
|
||||
async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
|
||||
result = await db.execute(
|
||||
select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
|
||||
)
|
||||
current = result.scalar_one_or_none()
|
||||
return int(current or 0)
|
||||
|
||||
|
||||
async def _set_session_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
session: PlaygroundSession,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
) -> PlaygroundSession:
|
||||
session.state = PlaygroundSessionState(
|
||||
messages=[],
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
analysis=None,
|
||||
latestAnalysisMessageId=None,
|
||||
analysisMeta={},
|
||||
helpExpanded=payload.help_expanded,
|
||||
).model_dump(mode="json")
|
||||
session.title = payload.title[:200]
|
||||
await db.flush()
|
||||
await db.refresh(session)
|
||||
return session
|
||||
|
||||
|
||||
def _spawn_assistant_run(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
assistant_public_id: str,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
task = asyncio.create_task(
|
||||
_run_assistant_message(
|
||||
user_id=user_id,
|
||||
session_id=session_id,
|
||||
session_key=session_key,
|
||||
user_message_id=user_message_id,
|
||||
assistant_message_id=assistant_message_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
)
|
||||
_ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
|
||||
|
||||
|
||||
async def create_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _ensure_session(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=payload.title,
|
||||
state=PlaygroundSessionState(
|
||||
selectedPresetKey=payload.selected_preset_key,
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
constraints=payload.constraints,
|
||||
inputValue="",
|
||||
helpExpanded=payload.help_expanded,
|
||||
),
|
||||
)
|
||||
session = await _set_session_state(db, session=session, payload=payload)
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
|
||||
user_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
role="user",
|
||||
kind="message",
|
||||
status="done",
|
||||
title=payload.selected_preset_key,
|
||||
content=payload.input,
|
||||
meta=[payload.title],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=None,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 20,
|
||||
)
|
||||
db.add(user_message)
|
||||
await db.flush()
|
||||
assistant_message.parent_message_id = user_message.id
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(user_message)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def _create_assistant_retry_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session: PlaygroundSession,
|
||||
user_message: PlaygroundMessage,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
base_order = await _next_sort_order(db, session.id)
|
||||
assistant_message = PlaygroundMessage(
|
||||
public_id=uuid4().hex,
|
||||
session_id=session.id,
|
||||
user_id=user_id,
|
||||
parent_message_id=user_message.id,
|
||||
role="assistant",
|
||||
kind="thinking",
|
||||
status="pending",
|
||||
title="AI 回应",
|
||||
content="",
|
||||
thinking_content="",
|
||||
meta=[],
|
||||
sort_order=base_order + 10,
|
||||
)
|
||||
db.add(assistant_message)
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(session)
|
||||
await db.refresh(assistant_message)
|
||||
|
||||
_spawn_assistant_run(
|
||||
user_id=user_id,
|
||||
session_id=session.id,
|
||||
session_key=payload.session_key,
|
||||
user_message_id=user_message.id,
|
||||
assistant_message_id=assistant_message.id,
|
||||
assistant_public_id=assistant_message.public_id,
|
||||
payload=payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
return await _build_action_response(
|
||||
db,
|
||||
session=session,
|
||||
active_message_id=assistant_message.public_id,
|
||||
)
|
||||
|
||||
|
||||
async def stop_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageStopRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
|
||||
|
||||
if message.status not in {"pending", "thinking", "answering"}:
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
active_run = _ACTIVE_RUNS.get(message.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def resend_turn(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageResendRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
later_messages = await db.execute(
|
||||
select(PlaygroundMessage).where(
|
||||
PlaygroundMessage.session_id == session.id,
|
||||
PlaygroundMessage.sort_order > user_message.sort_order,
|
||||
PlaygroundMessage.is_visible.is_(True),
|
||||
)
|
||||
)
|
||||
for item in later_messages.scalars().all():
|
||||
item.is_visible = False
|
||||
if item.status in {"pending", "thinking", "answering"}:
|
||||
active_run = _ACTIVE_RUNS.get(item.public_id)
|
||||
if active_run is not None:
|
||||
active_run.stop_requested.set()
|
||||
active_run.task.cancel()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
create_payload = PlaygroundMessageCreateRequest(
|
||||
session_key=payload.session_key,
|
||||
title=session_state.title or session.title,
|
||||
objective=session_state.objective or "继续当前对话",
|
||||
constraints=session_state.constraints or "",
|
||||
input=user_message.content,
|
||||
selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
|
||||
help_expanded=session_state.helpExpanded,
|
||||
)
|
||||
return await _create_assistant_retry_turn(
|
||||
db,
|
||||
user_id=user_id,
|
||||
session=session,
|
||||
user_message=user_message,
|
||||
payload=create_payload,
|
||||
provider_client=provider_client,
|
||||
)
|
||||
|
||||
|
||||
async def edit_user_message(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundMessageEditRequest,
|
||||
) -> PlaygroundMessageActionResponse:
|
||||
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
|
||||
user_message = await _require_visible_message(
|
||||
db,
|
||||
user_id=user_id,
|
||||
public_id=payload.user_message_id,
|
||||
role="user",
|
||||
)
|
||||
|
||||
user_message.content = payload.content.strip()
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
await db.refresh(user_message)
|
||||
|
||||
return await _build_action_response(db, session=session)
|
||||
|
||||
|
||||
async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is None:
|
||||
return
|
||||
if meta_line not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), meta_line]
|
||||
await db.flush()
|
||||
|
||||
|
||||
async def _should_stop(message_public_id: str) -> bool:
|
||||
active_run = _ACTIVE_RUNS.get(message_public_id)
|
||||
return active_run.stop_requested.is_set() if active_run is not None else False
|
||||
|
||||
|
||||
async def _mark_message_state(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
message_id: int,
|
||||
**updates,
|
||||
) -> PlaygroundMessage:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
|
||||
message = result.scalar_one()
|
||||
for key, value in updates.items():
|
||||
setattr(message, key, value)
|
||||
await db.flush()
|
||||
await db.refresh(message)
|
||||
return message
|
||||
|
||||
|
||||
def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
|
||||
history: list[dict] = []
|
||||
for item in messages:
|
||||
if item.id >= current_user_message_id:
|
||||
break
|
||||
if item.role == "system":
|
||||
continue
|
||||
history.append(
|
||||
{
|
||||
"role": item.role,
|
||||
"kind": item.kind or "message",
|
||||
"title": item.title,
|
||||
"content": item.content or "",
|
||||
}
|
||||
)
|
||||
return history[-8:]
|
||||
|
||||
|
||||
async def _run_assistant_message(
|
||||
*,
|
||||
user_id: int,
|
||||
session_id: int,
|
||||
session_key: str,
|
||||
user_message_id: int,
|
||||
assistant_message_id: int,
|
||||
payload: PlaygroundMessageCreateRequest,
|
||||
provider_client: AIProviderClient,
|
||||
) -> None:
|
||||
request_id = str(uuid4())
|
||||
started_at = perf_counter()
|
||||
assistant_public_id: str | None = None
|
||||
try:
|
||||
async with async_session_factory() as db:
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
user_message = await db.get(PlaygroundMessage, user_message_id)
|
||||
assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
|
||||
if session is None or user_message is None or assistant_message is None:
|
||||
return
|
||||
assistant_public_id = assistant_message.public_id
|
||||
|
||||
visible_messages = await _list_visible_messages(db, session_id=session_id)
|
||||
conversation_history = _build_conversation_history(visible_messages, user_message_id)
|
||||
|
||||
request_payload = SituationalAnalysisRequest(
|
||||
title=payload.title,
|
||||
objective=payload.objective,
|
||||
observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
|
||||
constraints=_collect_constraints(payload.constraints),
|
||||
context={
|
||||
"source": "playground",
|
||||
"preset": payload.selected_preset_key,
|
||||
"conversation_history": conversation_history,
|
||||
"history_size": len(conversation_history),
|
||||
},
|
||||
thinking={"type": "enabled"},
|
||||
)
|
||||
|
||||
analysis = await provider_client.analyze(request_payload, request_id=request_id)
|
||||
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="thinking" if analysis.thinking_blocks else "answering",
|
||||
title=f"{analysis.provider} / {analysis.model}",
|
||||
provider=analysis.provider,
|
||||
model=analysis.model,
|
||||
request_id=request_id,
|
||||
raw_response=analysis.raw_response,
|
||||
content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
|
||||
text_blocks=analysis.text_blocks,
|
||||
thinking_blocks=analysis.thinking_blocks,
|
||||
thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
|
||||
if assistant_public_id and analysis.thinking_blocks:
|
||||
await asyncio.sleep(THINKING_PREVIEW_SECONDS)
|
||||
if await _should_stop(assistant_public_id):
|
||||
return
|
||||
|
||||
content = analysis.content or ""
|
||||
cursor = 0
|
||||
while cursor < len(content):
|
||||
if assistant_public_id and await _should_stop(assistant_public_id):
|
||||
return
|
||||
cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
|
||||
async with async_session_factory() as db:
|
||||
await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="answering",
|
||||
content=content[:cursor],
|
||||
)
|
||||
await db.commit()
|
||||
await asyncio.sleep(STREAM_INTERVAL_SECONDS)
|
||||
|
||||
duration_ms = round((perf_counter() - started_at) * 1000)
|
||||
async with async_session_factory() as db:
|
||||
assistant_message = await _mark_message_state(
|
||||
db,
|
||||
message_id=assistant_message_id,
|
||||
status="done",
|
||||
content=content,
|
||||
meta=[
|
||||
f"Request ID: {request_id}",
|
||||
f"耗时: {duration_ms} ms",
|
||||
f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
|
||||
],
|
||||
)
|
||||
session = await db.get(PlaygroundSession, session_id)
|
||||
if session is not None:
|
||||
session_state = PlaygroundSessionState.model_validate(session.state or {})
|
||||
session.state = session_state.model_copy(
|
||||
update={
|
||||
"latestAnalysisMessageId": assistant_message.public_id,
|
||||
"analysis": analysis.model_dump(mode="json"),
|
||||
"analysisMeta": {
|
||||
"requestId": request_id,
|
||||
"durationMs": duration_ms,
|
||||
"completedAt": datetime.now().isoformat(),
|
||||
},
|
||||
}
|
||||
).model_dump(mode="json")
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
except asyncio.CancelledError:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None and message.status in {"pending", "thinking", "answering"}:
|
||||
message.status = "stopped"
|
||||
if "已手动停止生成" not in (message.meta or []):
|
||||
message.meta = [*(message.meta or []), "已手动停止生成"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
raise
|
||||
except Exception as exc:
|
||||
async with async_session_factory() as db:
|
||||
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
|
||||
message = result.scalar_one_or_none()
|
||||
if message is not None:
|
||||
message.status = "error"
|
||||
message.content = message.content or "分析失败,请检查 AI Provider 配置或稍后再试。"
|
||||
message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
|
||||
await db.flush()
|
||||
await db.commit()
|
||||
finally:
|
||||
if assistant_public_id:
|
||||
_ACTIVE_RUNS.pop(assistant_public_id, None)
|
||||
72
backend/app/services/playground_session_store.py
Normal file
72
backend/app/services/playground_session_store.py
Normal file
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.playground_session import PlaygroundSession
|
||||
from app.schemas.ai import (
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
PlaygroundSessionUpsertRequest,
|
||||
)
|
||||
|
||||
|
||||
def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
|
||||
return PlaygroundSessionResponse(
|
||||
id=str(record.id),
|
||||
session_key=record.session_key,
|
||||
title=record.title,
|
||||
state=PlaygroundSessionState.model_validate(record.state or {}),
|
||||
created_at=record.created_at.isoformat(),
|
||||
updated_at=record.updated_at.isoformat(),
|
||||
)
|
||||
|
||||
|
||||
async def get_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
session_key: str = "default",
|
||||
) -> PlaygroundSessionResponse | None:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
if record is None:
|
||||
return None
|
||||
return _to_response(record)
|
||||
|
||||
|
||||
async def upsert_playground_session(
|
||||
db: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
payload: PlaygroundSessionUpsertRequest,
|
||||
) -> PlaygroundSessionResponse:
|
||||
result = await db.execute(
|
||||
select(PlaygroundSession).where(
|
||||
PlaygroundSession.user_id == user_id,
|
||||
PlaygroundSession.session_key == payload.session_key,
|
||||
)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
|
||||
|
||||
if record is None:
|
||||
record = PlaygroundSession(
|
||||
user_id=user_id,
|
||||
session_key=payload.session_key,
|
||||
title=title,
|
||||
state=payload.state.model_dump(mode="json"),
|
||||
)
|
||||
db.add(record)
|
||||
else:
|
||||
record.title = title
|
||||
record.state = payload.state.model_dump(mode="json")
|
||||
|
||||
await db.flush()
|
||||
await db.refresh(record)
|
||||
return _to_response(record)
|
||||
@@ -19,6 +19,30 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
scheduler = AsyncIOScheduler()
|
||||
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
|
||||
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
|
||||
|
||||
|
||||
def _collector_task_name(collector_name: str) -> str:
|
||||
return f"collector:{collector_name}"
|
||||
|
||||
|
||||
def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
|
||||
task = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if task is not None and not task.done():
|
||||
return task
|
||||
|
||||
if task is not None and task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
target_name = _collector_task_name(collector_name)
|
||||
for candidate in asyncio.all_tasks():
|
||||
if candidate.done():
|
||||
continue
|
||||
if candidate.get_name() == target_name:
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = candidate
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def _update_next_run_at(datasource: DataSource, session) -> None:
|
||||
@@ -133,6 +157,12 @@ async def run_collector_task(collector_name: str):
|
||||
datasource.last_status = task_result.get("status")
|
||||
await _update_next_run_at(datasource, db)
|
||||
logger.info("Collector %s completed: %s", collector_name, task_result)
|
||||
except asyncio.CancelledError:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "cancelled"
|
||||
await db.commit()
|
||||
logger.warning("Collector %s cancelled by operator", collector_name)
|
||||
raise
|
||||
except Exception as exc:
|
||||
datasource.last_run_at = datetime.now(UTC)
|
||||
datasource.last_status = "failed"
|
||||
@@ -244,10 +274,37 @@ def run_collector_now(collector_name: str) -> bool:
|
||||
logger.error("Collector not found: %s", collector_name)
|
||||
return False
|
||||
|
||||
existing_task = get_running_collector_task(collector_name)
|
||||
if existing_task is not None and not existing_task.done():
|
||||
logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
|
||||
return False
|
||||
|
||||
try:
|
||||
asyncio.create_task(run_collector_task(collector_name))
|
||||
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
|
||||
RUNNING_COLLECTOR_TASKS[collector_name] = task
|
||||
|
||||
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
|
||||
current = RUNNING_COLLECTOR_TASKS.get(collector_name)
|
||||
if current is done_task:
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
|
||||
task.add_done_callback(_cleanup_task)
|
||||
logger.info("Triggered collector: %s", collector_name)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
async def cancel_running_collector_now(collector_name: str) -> bool:
|
||||
task = get_running_collector_task(collector_name)
|
||||
if task is None or task.done():
|
||||
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
|
||||
return False
|
||||
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
return True
|
||||
return task.cancelled()
|
||||
|
||||
174
backend/app/services/situational_alert_ai_brief.py
Normal file
174
backend/app/services/situational_alert_ai_brief.py
Normal file
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.models.alert import Alert, AlertSeverity, AlertStatus
|
||||
from app.models.bgp_anomaly import BGPAnomaly
|
||||
from app.models.bgp_incident import BGPIncident
|
||||
from app.schemas.ai import SituationalAnalysisRequest
|
||||
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
|
||||
|
||||
|
||||
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "无") -> str:
|
||||
if not pairs:
|
||||
return empty_text
|
||||
return ",".join(f"{key} {value}" for key, value in pairs if key)
|
||||
|
||||
|
||||
async def build_situational_alert_brief_request(
|
||||
db: AsyncSession,
|
||||
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
|
||||
total_alerts_result = await db.execute(select(func.count(Alert.id)))
|
||||
active_alerts_result = await db.execute(
|
||||
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
|
||||
)
|
||||
alert_severity_result = await db.execute(
|
||||
select(Alert.severity, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.severity)
|
||||
)
|
||||
alert_source_result = await db.execute(
|
||||
select(Alert.datasource_name, func.count(Alert.id))
|
||||
.where(Alert.status == AlertStatus.ACTIVE)
|
||||
.group_by(Alert.datasource_name)
|
||||
.order_by(func.count(Alert.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
recent_alerts_result = await db.execute(
|
||||
select(Alert)
|
||||
.order_by(Alert.created_at.desc(), Alert.id.desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
|
||||
active_incidents_result = await db.execute(
|
||||
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
|
||||
)
|
||||
bgp_severity_result = await db.execute(
|
||||
select(BGPIncident.severity, func.count(BGPIncident.id))
|
||||
.where(BGPIncident.status == "active")
|
||||
.group_by(BGPIncident.severity)
|
||||
)
|
||||
bgp_region_counter: Counter[str] = Counter()
|
||||
recent_incidents_result = await db.execute(
|
||||
select(BGPIncident)
|
||||
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
|
||||
.limit(5)
|
||||
)
|
||||
|
||||
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
|
||||
active_anomalies_result = await db.execute(
|
||||
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
|
||||
)
|
||||
anomaly_type_result = await db.execute(
|
||||
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
|
||||
.where(BGPAnomaly.status == "active")
|
||||
.group_by(BGPAnomaly.anomaly_type)
|
||||
.order_by(func.count(BGPAnomaly.id).desc())
|
||||
.limit(6)
|
||||
)
|
||||
|
||||
recent_incidents = recent_incidents_result.scalars().all()
|
||||
for incident in recent_incidents:
|
||||
for region in incident.affected_regions or []:
|
||||
if not isinstance(region, dict):
|
||||
continue
|
||||
label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
|
||||
bgp_region_counter[label] += 1
|
||||
|
||||
latest_bgp_brief = get_latest_bgp_brief_record()
|
||||
active_alert_severities = [
|
||||
(item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
|
||||
for item in alert_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_bgp_severities = [
|
||||
(str(item[0]), item[1])
|
||||
for item in bgp_severity_result.fetchall()
|
||||
if item[0]
|
||||
]
|
||||
active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
|
||||
active_alert_sources = [
|
||||
(str(item[0] or "未命名数据源"), item[1])
|
||||
for item in alert_source_result.fetchall()
|
||||
]
|
||||
|
||||
facts = [
|
||||
(
|
||||
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0} 条,active {active_alerts_result.scalar() or 0} 条;"
|
||||
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP态势侧:累计 incidents {total_incidents_result.scalar() or 0} 条,active incidents {active_incidents_result.scalar() or 0} 条;"
|
||||
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}。"
|
||||
),
|
||||
(
|
||||
f"BGP异常侧:累计 anomalies {total_anomalies_result.scalar() or 0} 条,active anomalies {active_anomalies_result.scalar() or 0} 条;"
|
||||
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}。"
|
||||
),
|
||||
]
|
||||
|
||||
if active_alert_sources:
|
||||
facts.append(f"当前系统告警主要集中在:{_format_pairs(active_alert_sources)}。")
|
||||
if bgp_region_counter:
|
||||
facts.append(f"BGP近期高风险区域线索:{_format_pairs(bgp_region_counter.most_common(5))}。")
|
||||
|
||||
recent_alerts = recent_alerts_result.scalars().all()
|
||||
if recent_alerts:
|
||||
facts.append(
|
||||
"最近系统告警摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
|
||||
for alert in recent_alerts
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if recent_incidents:
|
||||
facts.append(
|
||||
"最近BGP事件摘录:"
|
||||
+ ";".join(
|
||||
[
|
||||
f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
|
||||
for incident in recent_incidents
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
if latest_bgp_brief:
|
||||
facts.append(
|
||||
f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at},模型 {latest_bgp_brief.model},可作为当前态势的补充说明。"
|
||||
)
|
||||
|
||||
context = {
|
||||
"source": "situational-alerts",
|
||||
"active_system_alerts": active_alerts_result.scalar() or 0,
|
||||
"active_system_alert_severities": dict(active_alert_severities),
|
||||
"top_system_alert_sources": dict(active_alert_sources),
|
||||
"active_bgp_incidents": active_incidents_result.scalar() or 0,
|
||||
"active_bgp_incident_severities": dict(active_bgp_severities),
|
||||
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
|
||||
"active_bgp_anomaly_types": dict(active_anomaly_types),
|
||||
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
|
||||
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
|
||||
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
|
||||
}
|
||||
|
||||
request = SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
|
||||
observations=facts,
|
||||
constraints=[
|
||||
"明确区分事实、推断与建议。",
|
||||
"优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
|
||||
"不要把单一数据源的局部异常夸大成全局态势。",
|
||||
"如果证据不足,请明确写出仍缺哪些模块或区域信息。",
|
||||
],
|
||||
context=context,
|
||||
)
|
||||
return request, facts, context
|
||||
466
backend/app/services/tv_streams.py
Normal file
466
backend/app/services/tv_streams.py
Normal file
@@ -0,0 +1,466 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.time import to_iso8601_utc
|
||||
from app.models.collected_data import CollectedData
|
||||
from app.models.system_setting import SystemSetting
|
||||
|
||||
DEFAULT_TV_SOURCE_ID = "cgtn-en"
|
||||
TV_SETTINGS_CATEGORY = "tv"
|
||||
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
|
||||
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
|
||||
|
||||
DEFAULT_TV_SETTINGS = {
|
||||
"default_source_id": DEFAULT_TV_SOURCE_ID,
|
||||
"auto_fallback": True,
|
||||
"sources": [
|
||||
{
|
||||
"id": "cctv4",
|
||||
"name": "CCTV-4 中文国际",
|
||||
"provider": "CCTV",
|
||||
"region": "China",
|
||||
"language": "zh-CN",
|
||||
"source_type": "hls",
|
||||
"embed_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
|
||||
"homepage_url": "https://tv.cctv.com/live/cctv4/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": True,
|
||||
"sort_order": 10,
|
||||
"collector_source": None,
|
||||
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
|
||||
},
|
||||
{
|
||||
"id": "reuters-tv",
|
||||
"name": "Reuters TV",
|
||||
"provider": "Reuters",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
|
||||
"homepage_url": "https://www.reuters.com/video/live/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 20,
|
||||
"collector_source": None,
|
||||
"notes": "参考 worldmonitor 的默认新闻频道清单,优先作为全球英文新闻直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-en",
|
||||
"name": "CGTN English",
|
||||
"provider": "CGTN",
|
||||
"region": "Global",
|
||||
"language": "en",
|
||||
"source_type": "youtube",
|
||||
"embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
|
||||
"youtube_video_id": "BOy2xDU1LC8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 30,
|
||||
"collector_source": None,
|
||||
"notes": "优先使用官方 YouTube 直播源,保留 HLS 直播放流作为候选信息。",
|
||||
},
|
||||
{
|
||||
"id": "cgtn-es",
|
||||
"name": "CGTN Espanol",
|
||||
"provider": "CGTN",
|
||||
"region": "Latin America",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
|
||||
"homepage_url": "https://news.cgtn.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 40,
|
||||
"collector_source": None,
|
||||
"notes": "西语国际新闻频道,覆盖拉美方向态势。",
|
||||
},
|
||||
{
|
||||
"id": "dw-espanol",
|
||||
"name": "DW Espanol",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Europe",
|
||||
"language": "es",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
|
||||
"homepage_url": "https://www.dw.com/es/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 50,
|
||||
"collector_source": None,
|
||||
"notes": "来自 worldmonitor 可选频道清单的直播放源。",
|
||||
},
|
||||
{
|
||||
"id": "dw-arabic",
|
||||
"name": "DW Arabic",
|
||||
"provider": "Deutsche Welle",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
|
||||
"homepage_url": "https://www.dw.com/ar/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 60,
|
||||
"collector_source": None,
|
||||
"notes": "阿拉伯语新闻流,适合作为中东方向新闻补充源。",
|
||||
},
|
||||
{
|
||||
"id": "aljazeera-mubasher",
|
||||
"name": "Al Jazeera Mubasher",
|
||||
"provider": "Al Jazeera",
|
||||
"region": "Middle East",
|
||||
"language": "ar",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
|
||||
"homepage_url": "https://www.aljazeera.net/live",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 70,
|
||||
"collector_source": None,
|
||||
"notes": "中东实时新闻流,来自 worldmonitor HLS 频道目录。",
|
||||
},
|
||||
{
|
||||
"id": "arirang-news",
|
||||
"name": "Arirang News",
|
||||
"provider": "Arirang",
|
||||
"region": "Korea",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.arirang.com/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 80,
|
||||
"collector_source": None,
|
||||
"notes": "东北亚英语新闻源,适合补充韩半岛与东亚视角。",
|
||||
},
|
||||
{
|
||||
"id": "abp-news",
|
||||
"name": "ABP News",
|
||||
"provider": "ABP",
|
||||
"region": "India",
|
||||
"language": "hi",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
|
||||
"homepage_url": "https://news.abplive.com/live-tv",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 90,
|
||||
"collector_source": None,
|
||||
"notes": "印度新闻直播放源,补充南亚区域视角。",
|
||||
},
|
||||
{
|
||||
"id": "sabc-news",
|
||||
"name": "SABC News",
|
||||
"provider": "SABC",
|
||||
"region": "Africa",
|
||||
"language": "en",
|
||||
"source_type": "hls",
|
||||
"embed_url": "",
|
||||
"stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
|
||||
"homepage_url": "https://www.sabcnews.com/sabcnews/",
|
||||
"poster_url": "",
|
||||
"is_enabled": True,
|
||||
"is_fallback": False,
|
||||
"sort_order": 100,
|
||||
"collector_source": None,
|
||||
"notes": "非洲英语新闻源,补充非洲区域新闻覆盖。",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _clean_text(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
return str(value).strip()
|
||||
|
||||
|
||||
def _clean_url(value: Any) -> str:
|
||||
text = _clean_text(value)
|
||||
if not text:
|
||||
return ""
|
||||
|
||||
parsed = urlparse(text)
|
||||
if parsed.scheme and parsed.scheme not in {"http", "https"}:
|
||||
return ""
|
||||
if parsed.scheme and not parsed.netloc:
|
||||
return ""
|
||||
return text
|
||||
|
||||
|
||||
def _clean_bool(value: Any, *, default: bool) -> bool:
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if value in (None, ""):
|
||||
return default
|
||||
if isinstance(value, str):
|
||||
lowered = value.strip().lower()
|
||||
if lowered in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if lowered in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return bool(value)
|
||||
|
||||
|
||||
def _clean_int(value: Any, *, default: int) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
|
||||
payload = dict(source or {})
|
||||
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
|
||||
source_type = _clean_text(payload.get("source_type")).lower()
|
||||
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
|
||||
youtube_channel = _clean_text(payload.get("youtube_channel"))
|
||||
if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
|
||||
if youtube_video_id or youtube_channel:
|
||||
source_type = "youtube"
|
||||
else:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
if source_type == "youtube" and not youtube_video_id and not youtube_channel:
|
||||
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
|
||||
|
||||
return {
|
||||
"id": source_id,
|
||||
"name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
|
||||
"provider": _clean_text(payload.get("provider")) or "Unknown",
|
||||
"region": _clean_text(payload.get("region")) or "Global",
|
||||
"language": _clean_text(payload.get("language")) or "und",
|
||||
"source_type": source_type,
|
||||
"embed_url": _clean_url(payload.get("embed_url")),
|
||||
"stream_url": _clean_url(payload.get("stream_url")),
|
||||
"homepage_url": _clean_url(payload.get("homepage_url")),
|
||||
"poster_url": _clean_url(payload.get("poster_url")),
|
||||
"youtube_video_id": youtube_video_id,
|
||||
"youtube_channel": youtube_channel,
|
||||
"is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
|
||||
"is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
|
||||
"sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
|
||||
"collector_source": payload.get("collector_source"),
|
||||
"notes": _clean_text(payload.get("notes")),
|
||||
"updated_at": _clean_text(payload.get("updated_at")),
|
||||
}
|
||||
|
||||
|
||||
def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
|
||||
merged = {
|
||||
"default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
|
||||
"auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
"sources": [],
|
||||
}
|
||||
|
||||
raw_sources = []
|
||||
if isinstance(payload, dict):
|
||||
merged["default_source_id"] = (
|
||||
_clean_text(payload.get("default_source_id")) or merged["default_source_id"]
|
||||
)
|
||||
merged["auto_fallback"] = _clean_bool(
|
||||
payload.get("auto_fallback"),
|
||||
default=DEFAULT_TV_SETTINGS["auto_fallback"],
|
||||
)
|
||||
if isinstance(payload.get("sources"), list):
|
||||
raw_sources = payload["sources"]
|
||||
|
||||
if not raw_sources:
|
||||
raw_sources = DEFAULT_TV_SETTINGS["sources"]
|
||||
|
||||
normalized_sources = [
|
||||
normalize_tv_source(source, index=index)
|
||||
for index, source in enumerate(raw_sources)
|
||||
]
|
||||
|
||||
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
|
||||
normalized_sources.append(
|
||||
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
|
||||
)
|
||||
|
||||
default_source_exists = any(
|
||||
source["id"] == merged["default_source_id"] and source["is_enabled"]
|
||||
for source in normalized_sources
|
||||
)
|
||||
if not default_source_exists:
|
||||
fallback_source = next(
|
||||
(source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
first_enabled_source = next(
|
||||
(source for source in normalized_sources if source["is_enabled"]),
|
||||
None,
|
||||
)
|
||||
merged["default_source_id"] = (
|
||||
fallback_source["id"]
|
||||
if fallback_source
|
||||
else first_enabled_source["id"]
|
||||
if first_enabled_source
|
||||
else DEFAULT_TV_SOURCE_ID
|
||||
)
|
||||
|
||||
merged["sources"] = sorted(
|
||||
normalized_sources,
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
return merged
|
||||
|
||||
|
||||
async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
result = await db.execute(
|
||||
select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
|
||||
)
|
||||
record = result.scalar_one_or_none()
|
||||
payload = record.payload if record else None
|
||||
return normalize_tv_settings(payload)
|
||||
|
||||
|
||||
def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
|
||||
metadata = dict(record.extra_data or {})
|
||||
return normalize_tv_source(
|
||||
{
|
||||
"id": metadata.get("id") or record.source_id or record.entity_key,
|
||||
"name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
|
||||
"provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
|
||||
"region": metadata.get("region") or metadata.get("country") or "Global",
|
||||
"language": metadata.get("language") or "und",
|
||||
"source_type": metadata.get("source_type") or "iframe",
|
||||
"embed_url": metadata.get("embed_url") or metadata.get("url") or "",
|
||||
"stream_url": metadata.get("stream_url") or "",
|
||||
"homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
|
||||
"poster_url": metadata.get("poster_url") or "",
|
||||
"youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
|
||||
"youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
|
||||
"is_enabled": metadata.get("is_enabled", True),
|
||||
"is_fallback": False,
|
||||
"sort_order": metadata.get("sort_order", 200 + index),
|
||||
"collector_source": record.source,
|
||||
"notes": record.description or metadata.get("notes") or "",
|
||||
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
|
||||
},
|
||||
index=index,
|
||||
)
|
||||
|
||||
|
||||
async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
|
||||
result = await db.execute(
|
||||
select(CollectedData)
|
||||
.where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
|
||||
.where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
|
||||
.where(CollectedData.is_current.is_(True))
|
||||
.where(CollectedData.is_valid == 1)
|
||||
.order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
|
||||
)
|
||||
rows = result.scalars().all()
|
||||
return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
|
||||
|
||||
|
||||
def build_public_tv_payload(
|
||||
settings_payload: dict[str, Any],
|
||||
collected_sources: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
configured_sources = [
|
||||
source for source in settings_payload["sources"] if source["is_enabled"]
|
||||
]
|
||||
|
||||
merged_by_id = {source["id"]: source for source in configured_sources}
|
||||
for source in collected_sources:
|
||||
if source["id"] in merged_by_id or not source["is_enabled"]:
|
||||
continue
|
||||
merged_by_id[source["id"]] = source
|
||||
|
||||
available_sources = sorted(
|
||||
merged_by_id.values(),
|
||||
key=lambda item: (item["sort_order"], item["name"], item["id"]),
|
||||
)
|
||||
|
||||
default_source = next(
|
||||
(
|
||||
source
|
||||
for source in available_sources
|
||||
if source["id"] == settings_payload["default_source_id"]
|
||||
),
|
||||
None,
|
||||
)
|
||||
fallback_source = next(
|
||||
(source for source in available_sources if source["is_fallback"]),
|
||||
None,
|
||||
)
|
||||
|
||||
resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
|
||||
latest_updated_at = max(
|
||||
(source.get("updated_at") or "" for source in available_sources),
|
||||
default="",
|
||||
)
|
||||
|
||||
return {
|
||||
"default_source_id": settings_payload["default_source_id"],
|
||||
"auto_fallback": settings_payload["auto_fallback"],
|
||||
"selected_source": resolved_source,
|
||||
"fallback_source": fallback_source,
|
||||
"sources": available_sources,
|
||||
"source_count": len(available_sources),
|
||||
"latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
|
||||
"generated_at": to_iso8601_utc(datetime.now(UTC)),
|
||||
}
|
||||
|
||||
|
||||
async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
|
||||
settings_payload = await get_tv_settings_payload(db)
|
||||
collected_sources = await get_collected_tv_sources(db)
|
||||
return build_public_tv_payload(settings_payload, collected_sources)
|
||||
|
||||
|
||||
def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
|
||||
hosts: set[str] = set()
|
||||
for source in sources:
|
||||
for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
|
||||
value = _clean_url(source.get(field))
|
||||
if not value:
|
||||
continue
|
||||
parsed = urlparse(value)
|
||||
if parsed.hostname:
|
||||
hosts.add(parsed.hostname.lower())
|
||||
return hosts
|
||||
|
||||
|
||||
def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
|
||||
cleaned = _clean_url(url)
|
||||
if not cleaned:
|
||||
return False
|
||||
|
||||
parsed = urlparse(cleaned)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if not hostname:
|
||||
return False
|
||||
|
||||
allowed_hosts = _extract_allowed_tv_hosts(sources)
|
||||
if hostname in allowed_hosts:
|
||||
return True
|
||||
return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)
|
||||
@@ -10,7 +10,12 @@ from app.core.config import settings
|
||||
from app.core.security import create_access_token
|
||||
from app.db.session import get_db
|
||||
from app.models.user import User
|
||||
from app.schemas.ai import AIProviderStatusResponse, SituationalAnalysisResponse
|
||||
from app.schemas.ai import (
|
||||
AIProviderStatusResponse,
|
||||
PlaygroundSessionResponse,
|
||||
PlaygroundSessionState,
|
||||
SituationalAnalysisResponse,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -90,6 +95,15 @@ async def test_alerts_without_auth():
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_datasource_task_status_without_auth():
|
||||
"""Test datasource task-status endpoint requires authentication"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/datasources/1/task-status")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_alerts_endpoint_with_auth(auth_headers):
|
||||
"""Test alerts endpoint with authentication"""
|
||||
@@ -165,7 +179,8 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def get_status(self, request_id=None):
|
||||
return AIProviderStatusResponse(
|
||||
provider="openai_compatible",
|
||||
provider="minimax",
|
||||
api="anthropic-messages",
|
||||
enabled=True,
|
||||
configured=True,
|
||||
model="test-model",
|
||||
@@ -193,6 +208,7 @@ async def test_ai_provider_status_with_auth(auth_headers):
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "provider" in data
|
||||
assert "api" in data
|
||||
assert "configured" in data
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -207,6 +223,9 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
provider="openai_compatible",
|
||||
model="test-model",
|
||||
content="1) 态势摘要: 测试返回",
|
||||
content_blocks=[],
|
||||
text_blocks=["1) 态势摘要: 测试返回"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-response"},
|
||||
)
|
||||
|
||||
@@ -241,5 +260,297 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
|
||||
data = response.json()
|
||||
assert data["provider"] == "openai_compatible"
|
||||
assert data["content"]
|
||||
assert "content_blocks" in data
|
||||
assert "text_blocks" in data
|
||||
assert "thinking_blocks" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session restore endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.get_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="Playground 会话",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_key"] == "default"
|
||||
assert data["state"]["messages"][0]["content"] == "hello"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_save_playground_session_with_auth(auth_headers):
|
||||
"""Test playground session save endpoint."""
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch(
|
||||
"app.api.v1.ai.upsert_playground_session",
|
||||
new=AsyncMock(
|
||||
return_value=PlaygroundSessionResponse(
|
||||
id="1",
|
||||
session_key="default",
|
||||
title="测试标题",
|
||||
state=PlaygroundSessionState(
|
||||
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
title="测试标题",
|
||||
objective="测试目标",
|
||||
),
|
||||
created_at="2026-04-10T00:00:00+00:00",
|
||||
updated_at="2026-04-10T00:00:00+00:00",
|
||||
)
|
||||
),
|
||||
):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.put(
|
||||
"/api/v1/ai/playground/session",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"session_key": "default",
|
||||
"title": "测试标题",
|
||||
"state": {
|
||||
"messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
|
||||
"selectedPresetKey": "bgp-brief",
|
||||
"title": "测试标题",
|
||||
"objective": "测试目标",
|
||||
"constraints": "",
|
||||
"inputValue": "",
|
||||
"analysis": None,
|
||||
"latestAnalysisMessageId": None,
|
||||
"analysisMeta": {},
|
||||
"helpExpanded": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "测试标题"
|
||||
assert data["state"]["objective"] == "测试目标"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.5",
|
||||
content="# BGP AI 简报\n\n事实摘要:测试",
|
||||
content_blocks=[],
|
||||
text_blocks=["# BGP AI 简报\n\n事实摘要:测试"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-bgp-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_bgp_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="BGP 态势 AI 简报",
|
||||
objective="生成值班简报",
|
||||
observations=["事实A", "事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"incident_total": 2, "active_collectors": 3},
|
||||
)
|
||||
return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["facts"] == ["事实A", "事实B"]
|
||||
assert data["context"]["incident_total"] == 2
|
||||
assert data["content_markdown"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-alert-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_alert_brief_request(_db, **_kwargs):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="告警态势 AI 简报",
|
||||
objective="输出告警简报",
|
||||
observations=["告警事实A", "告警事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
|
||||
)
|
||||
return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "告警态势 AI 简报"
|
||||
assert data["facts"] == ["告警事实A", "告警事实B"]
|
||||
assert data["context"]["active_alerts"] == 3
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
|
||||
class _FakeAIProviderClient:
|
||||
async def analyze(self, _payload, request_id=None):
|
||||
return SituationalAnalysisResponse(
|
||||
provider="minimax",
|
||||
model="MiniMax-M2.7",
|
||||
content="事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。",
|
||||
content_blocks=[],
|
||||
text_blocks=["事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。"],
|
||||
thinking_blocks=[],
|
||||
raw_response={"id": "mock-situational-brief"},
|
||||
)
|
||||
|
||||
def override_get_current_user():
|
||||
return User(
|
||||
id=1,
|
||||
username="testuser",
|
||||
email="test@example.com",
|
||||
password_hash="hashed",
|
||||
role="admin",
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
async def override_get_db():
|
||||
yield AsyncMock()
|
||||
|
||||
async def _fake_build_situational_alert_brief_request(_db):
|
||||
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
|
||||
title="态势告警 AI 简报",
|
||||
objective="输出态势告警简报",
|
||||
observations=["态势事实A", "态势事实B"],
|
||||
constraints=["不要编造"],
|
||||
context={"active_system_alerts": 2, "active_bgp_incidents": 1},
|
||||
)
|
||||
return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
|
||||
|
||||
app.dependency_overrides = {
|
||||
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
|
||||
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
|
||||
get_db: override_get_db,
|
||||
}
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["title"] == "态势告警 AI 简报"
|
||||
assert data["facts"] == ["态势事实A", "态势事实B"]
|
||||
assert data["context"]["active_system_alerts"] == 2
|
||||
assert data["content"]
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
217
backend/tests/test_visualization_compute_centers.py
Normal file
217
backend/tests/test_visualization_compute_centers.py
Normal file
@@ -0,0 +1,217 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from app.api.v1.visualization import convert_compute_centers_to_geojson
|
||||
from app.db.session import get_db
|
||||
from app.main import app
|
||||
from app.models.collected_data import CollectedData
|
||||
|
||||
|
||||
def _build_record(
|
||||
*,
|
||||
record_id: int,
|
||||
source: str,
|
||||
data_type: str,
|
||||
name: str,
|
||||
country: str,
|
||||
city: str,
|
||||
latitude: float,
|
||||
longitude: float,
|
||||
metadata: dict,
|
||||
):
|
||||
return CollectedData(
|
||||
id=record_id,
|
||||
source=source,
|
||||
data_type=data_type,
|
||||
source_id=f"{source}-{record_id}",
|
||||
name=name,
|
||||
extra_data={
|
||||
"country": country,
|
||||
"city": city,
|
||||
"latitude": latitude,
|
||||
"longitude": longitude,
|
||||
**metadata,
|
||||
},
|
||||
collected_at=datetime(2026, 4, 22, tzinfo=timezone.utc),
|
||||
reference_date=datetime(2026, 4, 21, tzinfo=timezone.utc),
|
||||
is_current=True,
|
||||
)
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_unifies_sources():
|
||||
top500_record = _build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={
|
||||
"rank": 1,
|
||||
"manufacturer": "HPE",
|
||||
"organization": "ORNL",
|
||||
"rmax": 1102000.0,
|
||||
"cores": 8730112,
|
||||
"power": 21510.0,
|
||||
},
|
||||
)
|
||||
gpu_record = _build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={
|
||||
"organization": "xAI",
|
||||
"gpu_type": "H100",
|
||||
"gpu_count": 100000,
|
||||
"value": "20000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([top500_record, gpu_record])
|
||||
|
||||
assert payload["type"] == "FeatureCollection"
|
||||
assert len(payload["features"]) == 2
|
||||
|
||||
supercomputer_feature = payload["features"][0]
|
||||
assert supercomputer_feature["properties"]["site_type"] == "supercomputer"
|
||||
assert supercomputer_feature["properties"]["capacity_unit"] == "GFlops"
|
||||
assert supercomputer_feature["properties"]["capacity_band"] == "exascale"
|
||||
assert supercomputer_feature["properties"]["operator"] == "ORNL"
|
||||
assert supercomputer_feature["properties"]["location_precision"] == "precise"
|
||||
assert supercomputer_feature["properties"]["is_estimated"] is False
|
||||
|
||||
gpu_feature = payload["features"][1]
|
||||
assert gpu_feature["properties"]["site_type"] == "gpu_cluster"
|
||||
assert gpu_feature["properties"]["vendor"] == "H100"
|
||||
assert gpu_feature["properties"]["gpu_count"] == 100000
|
||||
assert gpu_feature["properties"]["capacity_band"] == "large"
|
||||
assert gpu_feature["properties"]["location_precision"] == "precise"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_uses_coordinate_hints():
|
||||
hinted_record = _build_record(
|
||||
record_id=3,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Oak Ridge National Laboratory",
|
||||
"rmax": 1102000.0,
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([hinted_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-84.3107)
|
||||
assert coords[1] == pytest.approx(35.9319)
|
||||
assert payload["features"][0]["properties"]["is_estimated"] is True
|
||||
assert payload["features"][0]["properties"]["location_precision"] == "estimated_site"
|
||||
|
||||
|
||||
def test_convert_compute_centers_to_geojson_falls_back_to_country_centroid():
|
||||
centroid_record = _build_record(
|
||||
record_id=4,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Unknown Cluster",
|
||||
country="United States",
|
||||
city="",
|
||||
latitude=0.0,
|
||||
longitude=0.0,
|
||||
metadata={
|
||||
"organization": "Unknown Operator",
|
||||
"value": "10000",
|
||||
"unit": "TFlop/s",
|
||||
},
|
||||
)
|
||||
|
||||
payload = convert_compute_centers_to_geojson([centroid_record])
|
||||
|
||||
assert len(payload["features"]) == 1
|
||||
props = payload["features"][0]["properties"]
|
||||
coords = payload["features"][0]["geometry"]["coordinates"]
|
||||
assert coords[0] == pytest.approx(-98.5795)
|
||||
assert coords[1] == pytest.approx(39.8283)
|
||||
assert props["is_estimated"] is True
|
||||
assert props["location_precision"] == "estimated_country"
|
||||
assert props["geography_mode"] == "country_centroid"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_compute_centers_geojson_endpoint_returns_stats():
|
||||
records = [
|
||||
_build_record(
|
||||
record_id=1,
|
||||
source="top500",
|
||||
data_type="supercomputer",
|
||||
name="Frontier",
|
||||
country="United States",
|
||||
city="Oak Ridge",
|
||||
latitude=35.93,
|
||||
longitude=-84.31,
|
||||
metadata={"rank": 1, "rmax": 1102000.0},
|
||||
),
|
||||
_build_record(
|
||||
record_id=2,
|
||||
source="epoch_ai_gpu",
|
||||
data_type="gpu_cluster",
|
||||
name="Colossus",
|
||||
country="United States",
|
||||
city="Memphis",
|
||||
latitude=35.15,
|
||||
longitude=-90.05,
|
||||
metadata={"value": "20000", "unit": "TFlop/s"},
|
||||
),
|
||||
]
|
||||
|
||||
class _ScalarResult:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def scalars(self):
|
||||
class _Scalars:
|
||||
def __init__(self, rows):
|
||||
self._rows = rows
|
||||
|
||||
def all(self):
|
||||
return self._rows
|
||||
|
||||
return _Scalars(self._rows)
|
||||
|
||||
class _FakeSession:
|
||||
async def execute(self, _query):
|
||||
return _ScalarResult(records)
|
||||
|
||||
async def override_get_db():
|
||||
yield _FakeSession()
|
||||
|
||||
app.dependency_overrides[get_db] = override_get_db
|
||||
transport = ASGITransport(app=app)
|
||||
try:
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/v1/visualization/geo/compute-centers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["count"] == 2
|
||||
assert data["stats"]["supercomputers"] == 1
|
||||
assert data["stats"]["gpu_clusters"] == 1
|
||||
assert data["features"][0]["properties"]["data_type"] == "compute_center"
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
@@ -5,6 +5,8 @@ services:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: aiprovider/Dockerfile
|
||||
env_file:
|
||||
- ./aiprovider/.env
|
||||
container_name: planet_aiprovider
|
||||
ports:
|
||||
- "8010:8010"
|
||||
|
||||
@@ -5,8 +5,824 @@ All notable changes to `planet` are documented here.
|
||||
This project follows the repository versioning rule:
|
||||
|
||||
- `feature` -> `+0.1.0`
|
||||
- `improvement` -> `+0.0.1`(bugfix + 小功能混合)
|
||||
- `bugfix` -> `+0.0.1`
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.37.0] — 2026-04-23
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 连线系统正式从巡航里解耦成通用 callout connector:桌面端和移动端统一支持对象级锚点、四边切换与临界区边缘滑动
|
||||
- BGP 巡航展示继续收口为稳定的“先定位卡片、再连真实锚点、再展示卡片”链路,移动端 popup 与桌面 info panel 的路线规则统一
|
||||
|
||||
### 🔧 Improvements
|
||||
- connector 配置从 `CRUISE_CONFIG` 拆到独立 `CONNECTOR_CONFIG`,默认类名、动画名和实例命名也全部去 cruise 语义
|
||||
- 移动端 popup 增加更稳定的 dock/obstacle 处理,拖动卡片时连线起终点会持续按几何关系自适应刷新
|
||||
- Earth 多个图层与控制逻辑继续收口,补充算力中心/BGP 风格对齐、layer panel 与相关交互细节调整
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下终点只像“视觉锚点”而不是真实绑定对象的问题,卡片拖动后终点现在会跟随
|
||||
- 修复移动端与桌面端多类连线路线异常:压线、反向、临界区折返、起点遮挡事件点等问题
|
||||
- 修复对象矩形临界区内连线仍强制中点到中点导致路线像“先钻进 source 内部”再出去的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
## [0.36.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增统一“算力中心”图层:接入超算与 GPU 集群,支持搜索、统计、图例、详情卡与独立图层开关
|
||||
- 算力中心支持精确位置与估算位置两种状态,估算点会以问号角标区分,避免数据不全时整批节点在地图上消失
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 详情卡拖拽与地球拖拽交互继续收口,减少拖动卡片和旋转地球时的选中文本与 pointer 竞争
|
||||
- `planet.sh` 改为通过独立脚本计算 AI Provider 依赖指纹,降低与根仓库依赖版本文件的无关耦合
|
||||
- README 补充 WSL / Windows 局域网访问排查与转发配置说明,便于开发环境联调
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Earth 算力中心图层在无原始坐标时无法显示的问题,支持站点提示和国家级估算回退
|
||||
- 修复信息卡拖拽事件可能被卡片级 stopPropagation 吞掉,导致拖拽流中断的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.1] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 统计展示改为统一 `data-earth-stat` 绑定机制,桌面 HUD 和移动端抽屉复用同一套状态更新入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- 收口海缆、登陆点、卫星、BGP 事件与 BGP 状态的统计写入逻辑,减少后续继续补桌面/移动双写分支的成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端态势抽屉中的海缆、登陆点与 BGP 统计在图层切换后可能停留旧值的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.35.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 移动端底部抽屉系统全面上线:响应式布局自动切换、Tab 导航、手势上拉/下滑开合、惯性速度判定
|
||||
- 移动端点击可交互物件(海缆、登陆点、卫星、BGP)后弹出智能定位悬浮卡片,可拖动,点击跳转详情
|
||||
|
||||
### 🔧 Improvements
|
||||
- 抽屉把手区域缩小至 36px(collapsed 时仅露出把手,不遮挡地球操作区)
|
||||
- 抽屉定期弹跳动画提示用户可上拉,5 秒间隔,打开后自动停止
|
||||
- 通知胶囊位置调整,不再覆盖品牌 logo
|
||||
- 移动端单指旋转、双指捏合缩放地球,触控事件冲突修复(pointer-events 级联)
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复移动端抽屉 shell 因 layout 高度(240px+)遮挡地球触控区域,pointer-events 改为按层级精确控制
|
||||
- 修复悬浮卡片因 setPointerCapture 在 iOS Safari 抑制合成 click 事件导致无法点击的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.34.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 搜索面板正式接入,支持搜索海缆、登陆点、卫星、BGP 事件与观测站,并可直接聚焦到对应对象
|
||||
- `planet.sh --allow-lan` 打通 Bun + Vite 的局域网开放链路,启动成功后自动打印推荐访问地址与后端健康检查地址
|
||||
|
||||
### 🔧 Improvements
|
||||
- 前端开发启动链统一改成 Bun 直接执行 Vite 入口,不再依赖 shell 中额外暴露的 Node 路径
|
||||
- Earth 搜索结果接入登陆点详情卡片与对象聚焦,搜索后可直接进入对应详情流
|
||||
- `planet.sh` 补充局域网 IPv4 自动识别与推荐地址输出,减少 WSL 局域网调试成本
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `./planet.sh restart --allow-lan` 全量重启时未把 `--allow-lan` 继续传给 `start()`,导致前端退回本机监听的问题
|
||||
- 修复 WSL + Bun 环境下前端偶发因 Vite 启动链不稳定而无法正确监听 `0.0.0.0:3000` 的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.33.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- `news_live_streams` 采集器默认接入 `iptv-org` 频道目录,并将采集结果稳定并入 Earth TV 直播源列表
|
||||
- 数据源页支持直接编辑内置数据源 override,并为内置源提供一键恢复默认配置入口
|
||||
|
||||
### 🔧 Improvements
|
||||
- `News Live Streams` 现在作为可直接触发的内置默认数据源提供,无需先手工补 override 才能采集
|
||||
- TV 播放源菜单会直接区分 `[内置]` 和 `[采集]` 来源,频道来源信息也会同步展示
|
||||
- 新增 [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md),正式规划 Earth 态势新闻源配置化与后续采集器化路线
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 `news_live_streams` 采集完成后 `/api/v1/tv/streams` 因读取不存在的 `updated_at` 字段而导致默认频道全部消失的问题
|
||||
- 修复内置数据源操作列按钮显示不全,以及编辑抽屉中多个 `Collapse` 紧贴的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.32.0] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 设置新增“地球默认大小”持久化项,重置视角、缩放百分比重置和 BGP 巡航视图现在统一复用这一份默认 zoom
|
||||
- 卫星焦点层次继续收口:巡航进入 presentation 前不再过早 dim,非焦点卫星改成“降亮度/尾迹/背板”而不是去饱和度
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth 设置面板区块和左右留白进一步收紧,整体更贴近 HUD 面板的密度
|
||||
- toolbar 展开边界缓存改为按需刷新,减少 document 级 mousemove 期间的重复布局读取
|
||||
- Scrollbar 和 ScrollbarOverlay 收窄 observer 范围,减少大表格和动态菜单下的额外刷新成本
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md),补充默认视图大小已进入 Earth 设置持久化真源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复开启巡航后,尚未进入连线/presentation 时卫星已经整体变暗的问题
|
||||
- 修复默认大小重置链路分散在多个入口、实际 reset/cruise/缩放提示不一致的问题
|
||||
- 修复开启地形后卫星反馈层与地球背面可见性之间的一组表现问题,保留正面反馈同时恢复背面轨道遮挡
|
||||
|
||||
---
|
||||
|
||||
## [0.31.3] — 2026-04-22
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层注册表和启动任务框架继续收口,启动顺序、启动模式、启动提示和任务注册现在都能从统一入口扩展
|
||||
- 修复 Earth 普通旋转模式与巡航模式切换时的一组交互回归,同时让卫星/地形/昼夜模式的表现更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js) 启动任务注册表,支持 `registerLayerStartupTask(id, taskFactory)`,并拆成海缆 / 卫星 / BGP 独立注册函数
|
||||
- Earth 图层控制改成注册表驱动,统一承载 `startupPriority`、`startupMode`、`startupLabel`、`startupMessage` 与图层持久化元信息
|
||||
- Earth 设置支持持久化图层开关、旋转模式、HUD 面板显示状态、地形透明度与日夜模式,并提供一键重置
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 记录图层注册表、启动任务、设置持久化与巡航适配边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复普通旋转模式下点击海缆 / 卫星 / BGP 后卡片和选中表现会被异常清空的问题
|
||||
- 修复巡航模式切回旋转再切回巡航后无法继续自动巡航的问题
|
||||
- 修复开启地形后卫星选中反馈层被高海拔区域吞掉的问题,并恢复轨道只在地球前半侧可见
|
||||
- 修复关闭日夜模式后地球照明仍沿真实昼夜切换、亮部过曝和偏色的问题,改成更中性的 inspection lighting
|
||||
- 修复 toolbar 收起态仍挡住地球交互,以及首帧短暂展开闪现的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 巡航模式重构为“通用巡航队列 + 通用连线动画 + BGP 业务适配”三层结构,后续扩到海缆、卫星或新闻巡航时不必再复制一套 `main.js` 状态机
|
||||
- 修复巡航重构后的交互回归:空白点击重新稳定切到下一项,连线按“起点 → 引导线 → 终点”顺序入场
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js) 统一管理队列推进、停留时长、打断与恢复
|
||||
- 新增 [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js) 统一管理 SVG 连线、折线路径与描边动画
|
||||
- 新增 [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 收口 BGP 巡航目标排序、卡片落点、轮询去重与连线适配
|
||||
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 说明新的巡航分层与复用边界
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复巡航模式下点击空白处无法稳定跳转到下一项、切回旋转再切回巡航后直接卡住的问题
|
||||
- 修复巡航连线被实时重定位覆盖导致“直接出现”而非绘制动画的问题
|
||||
- 修复连线动画节点入场节奏不对的问题,改为先出现起点,再绘制连线,最后出现终点
|
||||
|
||||
---
|
||||
|
||||
## [0.31.1] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 图层开关状态统一成可复用的 `active / loading` 状态机,首次启用地形和卫星时不再像按钮失效
|
||||
- 文档目录重构为 `docs/technical`、`docs/plans`、`docs/deprecated`,并吸收 `.sisyphus/plans` 中有价值的 Earth / 卫星 / UE5 草案
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js),统一按钮 tooltip、`aria-busy`、禁用态和状态文本同步
|
||||
- 地形图层支持 hover/focus 预热与空闲预热,首次点击等待前移,加载中状态持续可见
|
||||
- 卫星图层启用前会立即切换为 `loading` 中间态,请求完成后再切回正常开关表现
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复地形首次加载时通知过早消失、开关仍像关闭状态导致用户误判按钮损坏的问题
|
||||
- 修复卫星接口较慢时按钮没有任何中间态反馈的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.31.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增"巡航展示"模式:自动轮播 BGP 异常事件,逐帧追踪连接线位置,支持外部交互立即中断序列(cancel notifier 模式)
|
||||
- 巡航目标事件点高亮显示:hover 外观 + 锁定脉冲动画,并与点击行为统一展示周边受影响卫星与海缆
|
||||
- BGP 事件图标新增填充 W 形波动符号(flap 类型),替换原有难以辨认的贝塞尔细线
|
||||
- 巡航/点击激活时其余卫星自动降饱和度 + 增加透明度以突出焦点;海缆未受影响时同步变暗
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复巡航轮播期间 BGP 事件 polling 刷新导致标记闪烁消失的问题(clearBGPData 延迟到请求完成后执行)
|
||||
- 点击与巡航锁定颜色统一为 hover 色(0.92, 0.98, 1.0 全透明),移除锁定态脉冲动画
|
||||
- 巡航连接折线转折点从尖角调整为钝角(linkElbowDropPx),提升连线可读性
|
||||
|
||||
---
|
||||
|
||||
## [0.30.0] — 2026-04-21
|
||||
|
||||
### ✨ Features
|
||||
- Earth 新增真实地形图层:后端代理 Terrarium DEM 瓦片(`/api/v1/visualization/terrain/terrarium/{z}/{x}/{y}.png`),前端新增 `terrain.js` 负责瓦片拉取、顶点位移与按海拔着色
|
||||
- 设置弹窗新增"地形"分组,支持通过滑块实时调整地形图层透明度
|
||||
|
||||
### 🔧 Improvements
|
||||
- 地形按钮改为异步加载,首次点击显示进度提示并在失败时自动回退
|
||||
- 启动阶段改用 `applyImmediateView` 直接应用初始视角,`showStatusMessage` / `queueStatusMessage` 区分即时与队列态状态消息,加载中不再被临时状态打断
|
||||
- 控制面板抽取 `applyTerrainUiState` / `getViewRotation` 收敛地形切换与视角旋转的重复 UI 同步逻辑
|
||||
|
||||
---
|
||||
|
||||
## [0.29.2] — 2026-04-21
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 继续收口 HUD 交互与设置面板表现,设置弹窗改成更接近从按钮展开的窗口感,同时加入系统级 admin 入口
|
||||
- 修正天球太阳方向与地球受光解耦后的日照逻辑,地表昼夜判断改为按太阳直射点经纬度落到地球贴图坐标
|
||||
|
||||
### 🔧 Improvements
|
||||
- toolbar 进一步收成更贴近 hub 的浅弓形排列,并统一成与 HUD panel 一致的液态玻璃配色与透明度
|
||||
- 设置弹窗与各 HUD panel 继续统一样式、等比缩放和头部基线,设置列表补充系统分组与 admin 跳转
|
||||
- 所有 HUD panel 增加更统一的液态玻璃高光与 hover / press 反馈
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复设置弹窗仍像旧圆角矩形、标题文案重复和从底边直直飞出的动画问题
|
||||
- 修复天球与太阳方向混用显示校准导致中国白天仍落在夜面的日照错误
|
||||
|
||||
---
|
||||
|
||||
## [0.29.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 加载状态条改成单一队列式通知面板,加载阶段不再因为步骤文案变化而回缩,也不会被其他通知打断
|
||||
- 调整 brand panel 的呈现方式与昼夜/选中态可读性,让品牌区更自然、交互高亮在白天和黑夜里都更稳定
|
||||
|
||||
### 🔧 Improvements
|
||||
- 移除旧的地球加载浮层结构,统一由 HUD 状态消息承载三点脉冲加载过程
|
||||
- brand panel 改为无边框品牌层,仅保留轻微氛围光,不再因为非常规尺寸显得像第五块功能面板
|
||||
- 温和收敛地球昼夜材质与主背光强度,保留昼夜辨识度的同时提升白天地表纹理和夜面交互可见性
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复加载地球时通知条在步骤切换中反复缩短、其他状态消息抢占加载流程的问题
|
||||
- 修复海缆、登陆点和 BGP 选中高亮在黑夜中过暗、在高光中过亮导致难以辨识的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.29.0] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- Earth 新增天球层第一版:引入真实全天星图、亮星层与太阳/月亮位置计算,地球场景首次具备可校准的天文背景
|
||||
- 地球昼夜分隔升级为更明显的日夜增强效果,夜面、晨昏带和太阳方向联动更容易直接读出来
|
||||
|
||||
### 🔧 Improvements
|
||||
- 新增 `celestial.js` 模块和 `assets/celestial/` 资源目录,统一管理星图、亮星数据以及太阳/月亮与光照同步
|
||||
- 卫星图例改为按倾角分组,严格固定为“赤道轨道 → 低倾角轨道 → 中倾角轨道 → 高倾角轨道 → 逆行轨道”顺序,并全部中文化
|
||||
- 图层面板补齐关闭按钮,拖拽脱离左列后不再被流布局 margin 影响,能够真正贴到品牌面板下沿
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复天球球壳放大后被相机 far plane 裁剪导致的外层黑环问题
|
||||
- 修复图层面板在左侧上移时始终与 brand panel 保持额外间距的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.28.2] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 修正媒体情报面板在 `电视直播 / 态势聚合` tab 间切换时的尺寸记忆逻辑,切回原 tab 后可恢复各自大小状态
|
||||
- 清理 `docs/` 根目录遗留的旧路径文档,只保留新的分组目录和归档目录,结束同一文档双路径并存状态
|
||||
|
||||
### 🔧 Improvements
|
||||
- `media-panel` 切换逻辑改成按 tab 分别记忆尺寸状态,避免 `A -> B -> A` 时继续共用同一套外层尺寸
|
||||
- 目录整理真正完成收尾:旧的 `docs/*.md` 平铺计划文档删除,继续以 `docs/agents / earth / backend / frontend / ops / ue5 / deprecated` 为唯一入口
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复拉伸 `media-panel` 后切换 tab 时,`news-panel` 高度回退到旧默认值的问题
|
||||
- 修复拉伸后切换 tab 导致面板视觉锚点异常的问题,切换时改为围绕当前卡片自身右下角进行尺寸恢复
|
||||
|
||||
---
|
||||
|
||||
## [0.28.1] — 2026-04-20
|
||||
|
||||
### ✨ Highlights
|
||||
- 收口 Earth 媒体情报面板命名,明确外层 `media-panel` 与内部 `tv-panel / news-panel` 的职责边界
|
||||
- 整理 `docs/` 目录分组,并将已完成或已废弃的计划文档归档到 `docs/deprecated`
|
||||
|
||||
### 🔧 Improvements
|
||||
- 底部 tab 语义统一为 `media-panel-tabs / media-panel-tab`,并将文案更新为“电视直播 / 态势聚合”
|
||||
- 补充媒体面板、聚合新闻模块的注释说明,减少 `tv-panel` 同时指代外层壳和内层直播 pane 的阅读歧义
|
||||
- 更新 README、AI Provider README 与历史文档互链,适配新的 `docs/agents / docs/earth / docs/frontend / docs/backend / docs/ops / docs/ue5` 分组结构
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板标题组在头部撑出多余空白的问题,去掉 `hud-panel__title-group` 的无效弹性占位
|
||||
- 修复聚合 tab 头部仍保留冗余固定标题的问题,现在仅显示区域标签
|
||||
|
||||
---
|
||||
|
||||
## [0.28.0] — 2026-04-20
|
||||
|
||||
### ✨ Features
|
||||
- 将 Earth 的“新闻直播”和“全球态势聚合”合并为统一的“媒体情报”面板,支持底部 tab 切换与共享标题栏操作区
|
||||
- 聚合新闻不再单独占据一个 HUD 面板,而是作为媒体情报面板内的第二视图与直播协同呈现
|
||||
|
||||
说明:
|
||||
- 当前结构中,外层 HUD 壳为 `media-panel`,内部 tab 内容区分别为 `tv-panel` 和 `news-panel`
|
||||
|
||||
### 🔧 Improvements
|
||||
- TV / News 面板切换加入底边锚定的 reform 动画,并继续保留拖拽、缩放和共享 HUD 行为收口
|
||||
- 聚合新闻视图新增默认高度约束与内部滚动填充逻辑,避免初始高度过度膨胀
|
||||
- `tv.js`、`news.js` 进一步清理共享 HUDPanel 迁移后的残留逻辑,收紧 tab / resize / reform 相关局部 helper
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复媒体情报面板右下角缩放时高度异常抬升、标题栏被顶出视口的问题
|
||||
- 修复直播/聚合 tab 切换时按钮高亮、内容切换和底边基准表现不一致的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
## [0.27.8] — 2026-04-20
|
||||
|
||||
### 🔧 Improvements
|
||||
- Earth HUD 共享 `HUDPanel` 默认展开/收缩逻辑继续收口,图例与图层面板统一使用同一套边缘阈值与箭头状态机
|
||||
- 保持新闻直播面板现有特例折叠行为不变,避免播放器区域被默认折叠逻辑影响
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复图例与图层面板展开/收缩箭头方向和实际动作不一致的问题
|
||||
- 修复拖动到屏幕底边附近时初始箭头、拖动中箭头和点击后动作不同步的问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.7] — 2026-04-16
|
||||
|
||||
### 🔧 Improvements
|
||||
- 用户管理、数据源配置、电视直播源表格统一接入可折叠操作列,窄宽度下自动收起到下拉菜单,减少操作区挤压
|
||||
- 电视直播设置改为表格总览 + 弹窗编辑模式,主表内容更紧凑,适合控制台一屏浏览
|
||||
- Earth TV 面板新增失败源探测与自动回退恢复标记,便于值班时快速识别异常直播源
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 Settings 电视直播源新增后取消编辑会残留未保存草稿的问题
|
||||
- 修复 Settings 删除直播源只改本地状态、刷新后恢复的问题,删除现在会立即持久化
|
||||
- 修复 Users / DataSources / Settings 表格“备注/状态”和“操作”之间的空白占位列问题
|
||||
- 修复 `useCollapsedActions` 未释放 `ResizeObserver` 导致的潜在内存泄漏与重复回调问题
|
||||
|
||||
---
|
||||
|
||||
## [0.27.6] — 2026-04-15
|
||||
|
||||
### 🔧 Improvements
|
||||
- BGP 告警页表格纵向 overflow 修复:补全 flex 布局链,tabs content-holder 正确撑满剩余高度
|
||||
- 用户管理表格横向滚动修复:采用 flex-fill 方案替换 `height: auto !important`,自定义滚动条 X 轨道位置对齐表格底部
|
||||
- Playground 宽布局隐藏"服务状态"按钮:侧边栏可见时不显示冗余入口
|
||||
- AI Chatbox 输入框失焦收起为单行,聚焦或有内容时展开完整 composer
|
||||
|
||||
---
|
||||
|
||||
## [0.27.4] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- info-card 改为懒加载动态挂载:页面初始 DOM 不再含隐藏的 `#info-panel` 节点,仅首次点击交互元素时创建
|
||||
|
||||
---
|
||||
|
||||
## [0.27.5] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 统一控制台多页面滚动体验:BGP、alerts、采集数据、用户管理、任务、设置、Playground 等区域接入自定义滚动条与表格滚动容器
|
||||
- 优化 BGP 与 alerts 页响应式布局:顶部概览卡在窄宽度下优先重排,必要时才启用横向滚动,避免卡片裁切和全局滚动条接管
|
||||
- 调整 `situational alerts` 布局策略:统计卡按宽度在单行、两列和横滚之间切换,下方详情卡保持单行高度优先
|
||||
- 实时采集进度优化:一键采集完成后在未刷新页面时保留 100% 完成态,不再错误归零
|
||||
- 补充 UE5 MVP 融合方案文档,完善后续集成规划沉淀
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复 BGP summary 与 alerts 顶部卡片在无真实溢出时误出现横向滚动的问题
|
||||
- 修复 alerts 页面缩窄后外层全局竖向滚动被接管的问题,恢复“一屏内、内部滚动”的布局逻辑
|
||||
- 修复 `situational alerts` 在两排布局下详情卡竖向溢出的问题,改为更稳定的分区响应式排版
|
||||
- 修复自定义滚动条交互反馈,悬停、聚焦、拖拽时颜色加深但不再显示多余外圈
|
||||
|
||||
---
|
||||
|
||||
## [0.27.3] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- TV panel meta 折叠展开方向稳定:底部锚定时向上生长,拖拽后(顶部锚定)通过 JS 补偿 top 保持播放器底部位置不变
|
||||
- 修复 TV panel 展开/折叠时视频区域跳动问题:移除面板 min-height,使播放器高度在两种状态下保持一致
|
||||
- 修正 TV panel meta toggle 箭头方向:展开朝下,折叠朝上
|
||||
- 修复图例面板折叠按钮失效(legend-bar-btn 补充进拖拽排除列表)
|
||||
- 调整图层搜索框图标尺寸为 20px,BR 缩放角标改为直角 L 形
|
||||
|
||||
---
|
||||
|
||||
## [0.27.2] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 修复 brand copy 宽度不随内容收缩的问题,现在与 title 图片宽度保持一致
|
||||
- 提取 `--brand-copy-width` CSS 自定义属性,消除 160px / 172px 魔法数字重复
|
||||
|
||||
---
|
||||
|
||||
## [0.27.1] — 2026-04-14
|
||||
|
||||
### 🔧 Improvements
|
||||
- 面板拖拽新增 L 形边界约束,其他面板无法覆盖 brand 面板区域,并从右侧/底部自然卡边
|
||||
- brand 组件引入 `--brand-scale` 整体缩放变量,padding 与内容尺寸独立控制
|
||||
- 图层控制面板宽度收窄(260px),与 brand 面板错落排列,间距调大
|
||||
|
||||
### 🐛 Fixes
|
||||
- 修复搜索框 `type="search"` 导致清除按钮重复显示的问题
|
||||
- 修复 `[hidden]` 属性被组件 `display` 规则覆盖的问题
|
||||
|
||||
---
|
||||
|
||||
## 0.27.0
|
||||
|
||||
Released: 2026-04-14
|
||||
|
||||
### Highlights
|
||||
|
||||
- 全面重构 Earth HUD 布局:品牌面板、图层控制面板、信息详情卡片各自独立,支持拖拽与折叠,信息卡片改为跟随点击位置悬浮显示。
|
||||
- 新增图层控制面板(Layer Panel),海缆、卫星、地形、BGP 等图层集中管理,支持关键字搜索过滤。
|
||||
- Earth 大气层渲染升级,引入 Fresnel 着色器双层辉光效果和深度遮挡球体。
|
||||
|
||||
### ✨ Features
|
||||
|
||||
- 新增独立 Layer Panel(`js/controls.js`, `css/layer-panel.css`),图层开关、搜索过滤、折叠收起,取代原工具栏弹出菜单
|
||||
- 信息详情面板(info-panel)改为点击时定位到鼠标附近(`js/info-card.js`),悬停改为轻量 tooltip,降低视觉干扰
|
||||
- Earth 材质重构(`js/earth.js`, `js/constants.js`):新增 `EARTH_MATERIAL_CONFIG`,Fresnel 内外大气层辉光、深度遮挡球体,纹理加载独立为 `loadEarthTexture()`
|
||||
|
||||
### 🔧 Improvements
|
||||
|
||||
- Earth Stats 面板改为 2 列 KPI 网格布局,支持拖拽和关闭(`css/earth-stats.css`)
|
||||
- 数据加载改为分步串行(登陆点 → 海缆 → 卫星 → BGP → 纹理),每步之间 yield 帧,改善视觉渐现体验(`js/main.js`)
|
||||
- 图层按钮状态更新逻辑统一至 `updateLayerButtonState()`,消除重复实现
|
||||
- 海缆状态识别新增 `active` 枚举值(兼容旧 `In Service`)
|
||||
|
||||
---
|
||||
|
||||
## 0.26.1
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Cleaned up the first TV follow-up and replaced the dashboard sidebar's brittle one-off scroll behavior with a reusable `Scrollbar` component, so the new live module code is easier to maintain and the console navigation can scroll without layout jitter.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by extracting the sidebar scrollbar into a dedicated component with explicit `x / y / both` axis support, hidden native scrollbars, compact account/version rows, and a sidebar-only vertical setup instead of the earlier patchwork CSS glued directly onto the layout.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) by refactoring repeated iframe/video reset paths into shared helpers, so TV source switching, empty-state fallback, and playback retry handling no longer duplicate cleanup logic across multiple branches.
|
||||
|
||||
## 0.26.2
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Stabilized the new reusable scrollbar work by restoring reliable sidebar visibility and extending the same floating scrollbar language to the Data Sources tables without letting scrollbars squeeze layout width or regress the console navigation.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/components/Scrollbar/ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/ScrollbarOverlay.tsx) as an overlay variant that binds to existing scroll containers such as Ant Table bodies, so heavy data grids can adopt the new scrollbar visuals without replacing their built-in scrolling mechanics.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by reverting the sidebar to a reliable vertical-first scrollbar path, then reintroducing automatic dual-axis support with independent floating tracks that no longer hide the thumb when only the sidebar needs vertical scrolling.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the built-in and custom datasource tables now use the new overlay scrollbar instead of native table scrollbars, keeping horizontal and vertical scrolling available without changing Ant Table’s internal layout behavior.
|
||||
|
||||
## 0.26.0
|
||||
|
||||
Released: 2026-04-12
|
||||
|
||||
### Highlights
|
||||
|
||||
- Added an operator-facing TV live module to Earth, including backend-configurable live sources, a draggable/resizable live-news HUD window, default global news channels, and a dedicated settings workflow so the Earth page can open real news playback instead of only static telemetry.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [backend/app/api/v1/tv.py](/home/ray/dev/linkong/planet/backend/app/api/v1/tv.py), [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py), and [backend/app/services/collectors/news_live_streams.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/news_live_streams.py) to provide TV source configuration, public stream payloads, a guarded HLS proxy path, and a collector entry point for future world-news live-source ingestion.
|
||||
- Added the Earth TV HUD workspace through [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js), and [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css), including toolbar access, draggable/closable behavior, resize support, direct video/HLS playback, iframe fallback, and per-channel external-open handling.
|
||||
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/technical/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/technical/earth-news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Settings/Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py), and [backend/app/core/datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py) by adding TV source administration to system settings and registering the `news_live_streams` datasource as a first-class configurable collector.
|
||||
- Improved [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py) by seeding a curated first-pass news channel catalog that now defaults to `CGTN English` YouTube playback while keeping `CCTV-4` as a built-in fallback and exposing additional Reuters, CGTN, DW, Al Jazeera, Arirang, ABP, and SABC entries for operator testing.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) and [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) so HLS/video playback now actively attempts autoplay in the TV panel instead of only loading metadata and leaving the player visually idle.
|
||||
- Fixed [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css) so the TV source selector and action controls better match the Earth HUD dark theme instead of falling back to a bright native dropdown presentation.
|
||||
|
||||
## 0.25.3
|
||||
|
||||
Released: 2026-04-11
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the Earth HUD visual system into a calmer operator-facing style, turned the top-left Earth brand into a real reusable component with language-driven rendering, and cleaned up duplicated brand assets so the page now has a single source of truth for HUD branding.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/public/earth/js/brand.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/brand.js) as a reusable Earth `brand` component that renders the top-left logo/title/subtitle block from a shared config instead of hardcoding the structure in HTML.
|
||||
- Added [frontend/public/earth/assets/brand/earth-logo.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/earth-logo.svg), [frontend/public/earth/assets/brand/title-zh.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-zh.svg), and [frontend/public/earth/assets/brand/title-en.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-en.svg) as the canonical Earth brand asset set.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css), [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css), and [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css) by rebalancing the Earth HUD into a more restrained deep-blue control-room look instead of the earlier over-layered glass-and-neon mix.
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), and [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) by moving the Earth brand mount to a dedicated root and letting `HUD_CONFIG.brandLanguage` choose between `zh` and `en` without embedding the language decision in the DOM.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css) so the Earth brand area now uses consistent `earth-brand` component selectors and English-specific typography hooks, avoiding the earlier one-off `brand-banner` naming drift and duplicated title styles.
|
||||
|
||||
## 0.25.2
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the Earth HUD operator polish so settings now behave like a true bounded menu, HUD panels render at the correct scale from the first frame, and dragged panels animate cleanly into and out of maximized layout targets without drifting to screen edges.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by precomputing the initial `--hud-scale` before Earth CSS loads, so HUD panels no longer flash at full size before shrinking to the target scale.
|
||||
- Improved [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) by cleaning up duplicated settings-modal close-button styles, aligning the settings title with the shared HUD title system, and constraining the settings sheet to a stable centered width instead of viewport-relative modal sizing.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) so dragged HUD panels now fly from their dragged positions into maximized layout targets, closed panels stay out of the transition, and restoring layout clears drag overrides back to the initial positions.
|
||||
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) and [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) so layout transitions no longer visibly stick to screen edges before landing; the FLIP motion now composes with the existing corner transforms instead of fighting them.
|
||||
|
||||
## 0.25.1
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Cleaned up the first persistent Playground rollout, fixed sidebar submenu persistence to match the intended operator behavior, and added reusable code-hygiene rules to prevent this class of drift from accumulating again.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) and [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by extracting repeated lookup, response, and request-action paths into shared helpers, reducing duplicated Playground flow code without changing behavior.
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding a new `Code Hygiene - MANDATORY` section covering single-source-of-truth state, transitional cleanup, repeated-logic extraction, layout debugging order, and post-feature cleanup expectations.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) so first-level menu expansion now behaves correctly across in-app navigation: `采集与数据` remains the default expanded group after refresh, while manually expanded groups stay open when navigating to their own child routes and reset only on page reload.
|
||||
|
||||
## 0.25.0
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Turned `AI Playground` into a persistent backend-backed chat workspace, added dedicated alert workspaces as a foundation for future situational analysis, and aligned the operator UI around a more structured AI + alerts workflow instead of one-off playground calls.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [backend/app/models/playground_session.py](/home/ray/dev/linkong/planet/backend/app/models/playground_session.py), [backend/app/models/playground_message.py](/home/ray/dev/linkong/planet/backend/app/models/playground_message.py), and [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so Playground conversations, execution state, edits, retries, and stop/resume semantics are persisted in the backend database rather than living only in browser state.
|
||||
- Added [backend/app/services/alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/alert_ai_brief.py), [backend/app/services/situational_alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/situational_alert_ai_brief.py), and the dedicated alert pages [frontend/src/pages/Alerts/SystemAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SystemAlerts.tsx), [frontend/src/pages/Alerts/BGPAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/BGPAlerts.tsx), and [frontend/src/pages/Alerts/SituationalAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SituationalAlerts.tsx) to establish the alert-analysis foundation for later situational awareness expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by rebuilding Playground into a true chatbox workflow with persistent history, edit-and-resend behavior, grounded message actions, responsive composer behavior, bottom-stick scrolling, and tighter mobile layout handling.
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx), and [frontend/src/pages/Alerts/Alerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/Alerts.tsx) by reorganizing navigation around `采集与数据`, `专题观测`, and split alert entries so the app can scale to more observability and situational modules without turning the top-level UI into a single overloaded page.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so background Playground runs explicitly commit state transitions, allowing frontend polling to observe real pending/thinking/answering/done states instead of seeing stale empty threads.
|
||||
- Fixed [frontend/src/services/situational-awareness/index.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/index.ts) by removing the temporary mock gateway path, so Playground and alert-related AI flows now reflect the real backend/provider chain instead of local fake responses.
|
||||
|
||||
## 0.24.8
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the BGP AI brief operator workflow so the tab now stays compact and metadata-focused, while the full Markdown brief opens in a bounded modal that respects the repo’s single-screen layout rules.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by turning the brief action row into a clearer `历史简报下拉框 + 查看 + 生成` flow, keeping inline metadata visible in the tab while moving full Markdown reading into a dedicated modal workspace.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so the BGP brief modal no longer spills below the viewport; long brief content now scrolls inside the modal body instead of extending past the visible screen.
|
||||
|
||||
## 0.24.7
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Formalized repository release hygiene and frontend layout guardrails so repeated versioning chores and recurring layout regressions now have explicit repo-level rules instead of living only in conversation context.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [release-workflow/SKILL.md](/home/ray/dev/linkong/planet/.codex/skills/release-workflow/SKILL.md), defining the repository release workflow for version bumps, changelog/version-history updates, minimal validation, and commit/push sequencing.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding mandatory release-workflow requirements and a new frontend layout constraint section covering single-screen workspaces, overflow ownership, tab-pane behavior, compact-mode expectations, and readable-card fallbacks.
|
||||
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
|
||||
|
||||
## 0.24.6
|
||||
|
||||
Released: 2026-04-10
|
||||
|
||||
### Highlights
|
||||
|
||||
- Tightened several backend hot paths outside the original BGP page fixes, stabilized BGP collector coverage after the recent query refactors, and rebuilt the BGP AI brief tab so saved Markdown briefs render and scroll like a proper operator workspace instead of collapsing inside the shared table layout.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [backend/app/api/v1/datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py) by replacing per-datasource task, count, and endpoint lookups with batched prefetch helpers, reducing the worst `1 + N` behavior on the datasource list and `trigger-all` flow.
|
||||
- Improved [backend/app/api/v1/visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py) by switching the main Earth-facing `CollectedData` endpoints to `is_current` records, batching multi-source loads for aggregate endpoints, and removing stale Python-side dedupe paths from the hot route.
|
||||
- Improved [backend/app/services/bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) by avoiding historical full-table infrastructure scans, narrowing observation baseline payloads to required columns, and pushing more ASN filtering into the database.
|
||||
- Improved [backend/app/api/v1/alerts.py](/home/ray/dev/linkong/planet/backend/app/api/v1/alerts.py), [backend/app/api/v1/dashboard.py](/home/ray/dev/linkong/planet/backend/app/api/v1/dashboard.py), and [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py) by collapsing several repeated count and settings queries into fewer aggregate or batched reads.
|
||||
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx), [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css), and [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx) by rebuilding the `AI 简报` tab layout, fixing saved brief scrolling behavior, and extending the renderer to handle tables, separators, and stored metadata comments more gracefully.
|
||||
- Improved [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [backend/app/services/bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) so JSON field extraction no longer depends on the less portable `.astext` path that could break BGP collector endpoints in local environments.
|
||||
- Fixed the BGP AI brief tab in [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so long saved briefs are no longer compressed into a tiny clipped viewport by the shared tab/table overflow rules.
|
||||
|
||||
## 0.24.4
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the `planet.sh` AI Provider rebuild UX so image rebuilds now feel like first-class scripted tasks, with clearer stage boundaries and cleaner fallback messaging instead of leaking raw Compose output into the terminal.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by keeping AI Provider image build logs in a temporary file, surfacing stage-specific detail copy for `docker compose v2` and `docker-compose v1`, and showing an explicit success line when the image rebuild finishes before container health checks begin.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the AI Provider image rebuild stage no longer dumps raw Compose build output into the main spinner flow during normal successful runs.
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the `构建 AI Provider 镜像` phase now ends with an explicit completion signal instead of visually blending into the subsequent container health-check phase.
|
||||
|
||||
## 0.24.3
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Extended `AI Playground` from a minimal prompt form into a more repeatable diagnostics workspace, and hardened `planet.sh` so AI Provider restarts can rebuild changed images and expose clearer Compose fallback behavior.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by adding preset scenarios, response metadata, `thinking` block visibility, raw JSON inspection, and copy actions so the page works more like a proper AI diagnostics console.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by styling Playground presets, result metadata, raw response sections, and responsive action groups without breaking the single-screen workspace layout.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so AI Provider restarts detect code/config changes, rebuild the image when needed, and surface which Compose path is being used during image and container operations.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the AI Provider restart path in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so code changes inside `aiprovider/` no longer stay hidden behind an old container image after `restart -a`.
|
||||
- Fixed Compose error reporting in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by making the script explicitly show `docker compose v2` first, then `docker-compose v1`, and only fail after both execution paths are exhausted.
|
||||
|
||||
## 0.24.2
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the public-entry and AI diagnostics regressions introduced during the recent frontend routing and playground work, while also reducing the main frontend bundle by switching to route-level lazy loading and more targeted vendor chunking.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by lazy-loading admin pages and large workspaces through `React.lazy()` plus `Suspense`, so the initial frontend entry no longer pulls every route into the first bundle.
|
||||
- Improved [frontend/vite.config.ts](/home/ray/dev/linkong/planet/frontend/vite.config.ts) by adding targeted manual chunking for React, icon, network, and Earth-related vendor dependencies instead of leaving everything in one monolithic application bundle.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding a shared route-loading state and making the Playground help panel size to its content instead of stretching to fill the sidebar.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) so anonymous visits to `/` once again reach the public Earth entry through the existing `/ -> /earth` redirect instead of being intercepted by the login screen.
|
||||
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) so cached provider status is shown immediately but still refreshed from the backend, avoiding stale diagnostics after `.env` or provider changes within the same browser tab.
|
||||
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the `测试说明` card no longer over-expands and now fits its content more naturally in the sidebar.
|
||||
|
||||
## 0.24.1
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined the `/earth` HUD into a cleaner class-first structure with responsive scaling, clearer CSS layer boundaries, and lower coupling between HTML, CSS, and runtime UI updates.
|
||||
- Hardened local startup conventions around Bun so frontend tooling, docs, and `planet.sh` behave more predictably in fresh Ubuntu and mixed WSL environments.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), extracting shared HUD panel surfaces, shared HUD typography rows, status messaging, tooltip overlays, and layout-expanded panel transitions out of the old monolithic base layer.
|
||||
- Added [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css), isolating Earth toolbar, popover, zoom dock, liquid-glass button, and toolbar-tooltip behavior into a dedicated toolbar layer.
|
||||
- Added explicit Bun package-manager metadata to [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json) so the frontend package manager choice is declared instead of inferred from the lockfile alone.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css) by reducing it to app-shell concerns only: global tokens, Earth app container, loading panel, and shared animation primitives.
|
||||
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by wiring the new CSS layer order, standardizing HUD utility classes on `hud-panel-*`, and replacing generic toolbar/tooltip hooks with more explicit Earth-specific classes.
|
||||
- Improved [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), and [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css) by leaving only panel-specific responsibilities in each file after the shared HUD and toolbar primitives moved out.
|
||||
- Improved [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js), [frontend/public/earth/js/ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js), and [frontend/public/earth/js/legend.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/legend.js) by aligning runtime DOM queries and generated markup with the new class-first HUD and toolbar structure.
|
||||
- Improved [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) and [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) by keeping HUD scaling configurable through extracted constants instead of scattering scaling assumptions through the main Earth entrypoint.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by prepending `~/.local/bin` and `~/.bun/bin` automatically, preferring direct `~/.bun/bin/bun` detection, and auto-installing missing `uv`/`bun` instead of requiring the user's interactive shell config to expose them first.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md), [project_context.md](/home/ray/dev/linkong/planet/project_context.md), [rules.md](/home/ray/dev/linkong/planet/rules.md), and [scripts/bootstrap-dev.sh](/home/ray/dev/linkong/planet/scripts/bootstrap-dev.sh) by making the frontend Bun-only workflow explicit in both onboarding docs and command-line guidance.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the Earth HUD cleanup path where splitting styles previously left the page with a missing `base.css` entry and mismatched utility class names during the refactor.
|
||||
- Fixed several Earth UI update paths so toolbar tooltip text, legend re-rendering, status message classes, and mouse coordinate styling continue to work after removing legacy fallback selectors.
|
||||
- Fixed the local developer bootstrap path where `planet.sh start` could fail in a clean Ubuntu or agent shell session simply because Bun or uv were installed outside the current shell's inherited `PATH`.
|
||||
|
||||
## 0.24.0
|
||||
|
||||
Released: 2026-04-09
|
||||
|
||||
### Highlights
|
||||
|
||||
- Added a dedicated `AI Playground` admin entry so operators can validate provider connectivity and run controlled situational-analysis prompts from the main frontend without introducing a separate UI service.
|
||||
- Established a first explicit frontend layout rulebook centered on single-screen workspaces, internal module scrolling, and BGP-style page composition for future admin pages.
|
||||
|
||||
### Added
|
||||
|
||||
- Added [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx), introducing the first dedicated AI testing workspace with provider status visibility, prompt/result tabs, and collapsible operator guidance.
|
||||
- Added [docs/frontend/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
|
||||
- Added [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) and [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by wiring `AI Playground` into the main admin navigation and route tree instead of pointing operators to a nonexistent `aiprovider` chat page.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by replacing the incorrect `localhost:8010/chat` closeout link with the frontend `AI Playground` entry.
|
||||
- Improved [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml) by attaching `./aiprovider/.env` to the `aiprovider` service so provider identity, model, and credentials actually reach the running container.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by linking the new frontend layout guidance and AI Playground development plan.
|
||||
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by refining the Playground workspace into a notebook-friendly left-sidebar plus right-tabbed layout, reusing thin scrollbars, and making provider/help/result regions degrade more gracefully under constrained height.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the local AI status flow so provider configuration no longer appeared permanently `disabled / not configured` merely because `aiprovider/.env` was not mounted into the container.
|
||||
- Fixed repeated `Provider 状态` refetching when switching away from and back to `/playground` by caching the last known provider status within the browser session until the operator explicitly refreshes it.
|
||||
- Fixed several Playground layout regressions where auxiliary panels could push the result area out of view or clip provider details without exposing internal scrolling.
|
||||
|
||||
## 0.23.4
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the BGP overview workspace so summary cards and tabular data now share viewport space more predictably, keeping the table header visible while preserving internal horizontal and vertical scrolling.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching BGP tables from frontend pagination to in-table scrolling, setting explicit horizontal scroll baselines per tab, and reshaping the summary cards into a desktop two-row layout with a compact single-row horizontal strip on tighter screens.
|
||||
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by hardening the BGP table layout chain from card body to Ant Design table body, so width overflow stays inside the table region and compact summary cards no longer consume unnecessary vertical space above the workspace.
|
||||
- Improved release consistency by aligning [VERSION](/home/ray/dev/linkong/planet/VERSION), [pyproject.toml](/home/ray/dev/linkong/planet/pyproject.toml), [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json), and [uv.lock](/home/ray/dev/linkong/planet/uv.lock) on the same `0.23.4` bugfix version.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the BGP collector/events tables so shrinking the browser window no longer clips the table card frame without exposing a usable horizontal scrollbar.
|
||||
- Fixed the BGP table region so the Ant Design header row is no longer obscured by an overgrown table body after the page switched to full-height internal scrolling.
|
||||
- Fixed the BGP summary area so medium and large screens no longer collapse the six KPI cards into overly narrow single-row tiles.
|
||||
|
||||
## 0.23.3
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Refined `planet.sh` startup and restart presentation so service bring-up, health checks, and frontend dev-server readiness are easier to follow in real time.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by standardizing the script on `zsh`, tightening frontend startup stage rendering, and making spinner-driven subtask feedback show up step by step instead of bunching at the end.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by clarifying health-check subtask copy, filtering noisy frontend startup lines, and aligning terminal output spacing across spinner and status rows.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed several `zsh` compatibility issues in [planet.sh](/home/ray/dev/linkong/planet/planet.sh), including reserved variable name collisions during `restart` and container health checks.
|
||||
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so frontend readiness animation no longer appears frozen while waiting for Vite startup and health probes.
|
||||
|
||||
## 0.23.2
|
||||
|
||||
Released: 2026-04-08
|
||||
|
||||
### Highlights
|
||||
|
||||
- Hardened datasource retrigger handling so operators can safely force reruns without losing control of task state visibility or rollback behavior.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) by mapping phase-local task percentages into a continuous overall progress bar, pre-checking running task status before retriggering, and consolidating the trigger/force-confirm flow.
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by polishing startup CLI output, reducing duplicated spinner cleanup, and standardizing log/help output around the newer terminal presentation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed forced datasource reruns in [backend/app/services/collectors/base.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/base.py) by rolling back invalid SQLAlchemy session state before marking cancelled or failed task cleanup, preventing `PendingRollbackError` during operator-triggered cancellation.
|
||||
- Fixed datasource trigger UX in [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) so batch and single-source progress no longer jump straight to `100%` on frontend-side status coercion while the backend is still reporting real progress.
|
||||
|
||||
## 0.23.1
|
||||
|
||||
Released: 2026-04-07
|
||||
|
||||
### Highlights
|
||||
|
||||
- Fixed the BGP overview page so high-DPI and lower-height screens can keep the observation workspace visible within a single viewport.
|
||||
|
||||
### Improved
|
||||
|
||||
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching the page to a height-aware shell, compact summary layout, and tabbed data workspace so observation tables retain the majority of the screen.
|
||||
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding responsive BGP-specific compact spacing, denser table paddings, and an internal scroll region for collector coverage and event tables.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the BGP observation page on small or high-scale displays where stacked stats and full-height blocks consumed too much vertical space, leaving only a couple of visible table rows.
|
||||
|
||||
## 0.23.0
|
||||
|
||||
Released: 2026-04-07
|
||||
@@ -23,7 +839,7 @@ Released: 2026-04-07
|
||||
- Added [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py), introducing an internal HTTP client for `backend -> aiprovider` calls with request-id propagation and lightweight retry.
|
||||
- Added [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py), [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py), and related config/schema files to stand up the dedicated adapter service.
|
||||
- Added [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) and [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) as ready-to-edit local-model templates.
|
||||
- Added [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
|
||||
- Added a dedicated `重启 AI Provider` control path in [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx), [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py), and [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
|
||||
### Improved
|
||||
@@ -31,6 +847,8 @@ Released: 2026-04-07
|
||||
- Improved backend-to-provider tracing by propagating `X-Request-ID` through the AI call chain and returning the same header from both backend and `aiprovider`.
|
||||
- Improved resilience by adding lightweight retry handling to both `backend -> aiprovider` and `aiprovider -> model provider` HTTP calls.
|
||||
- Improved operator workflow by folding `aiprovider` startup, health checks, restart support, and log viewing into [planet.sh](/home/ray/dev/linkong/planet/planet.sh).
|
||||
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by adding bounded retry and container-health self-recovery for dependency installs, database startup, `aiprovider` startup, and interactive PostgreSQL boot paths.
|
||||
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by documenting the new `planet.sh` retry and health-check tuning environment variables with concrete override examples.
|
||||
- Improved container consistency by switching [backend/Dockerfile](/home/ray/dev/linkong/planet/backend/Dockerfile) and [aiprovider/Dockerfile](/home/ray/dev/linkong/planet/aiprovider/Dockerfile) to `uv sync` / `uv run`.
|
||||
|
||||
### Changed
|
||||
@@ -147,7 +965,7 @@ Released: 2026-04-02
|
||||
|
||||
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
|
||||
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/prefix-geography-plan.md).
|
||||
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-prefix-geography-plan.md).
|
||||
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
|
||||
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
|
||||
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
|
||||
@@ -162,7 +980,7 @@ Released: 2026-04-02
|
||||
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
|
||||
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
|
||||
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -320,7 +1138,7 @@ Released: 2026-03-31
|
||||
- Added restart-task Redis helpers and whitelist command mapping in [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py).
|
||||
- Added detached restart runner orchestration in [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
|
||||
- Added `-d` / `--database` support to [planet.sh](/home/ray/dev/linkong/planet/planet.sh) for database-only restarts.
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/system-service-control.md).
|
||||
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md).
|
||||
|
||||
### Improved
|
||||
|
||||
@@ -523,7 +1341,7 @@ Released: 2026-03-26
|
||||
|
||||
### Added
|
||||
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/earth-module-plan.md).
|
||||
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-module-plan.md).
|
||||
- Added backend TLE helpers in [satellite_tle.py](/home/ray/dev/linkong/planet/backend/app/core/satellite_tle.py).
|
||||
- Added backend support for returning `tle_line1` and `tle_line2` from the satellite visualization API.
|
||||
|
||||
|
||||
22
docs/deprecated/README.md
Normal file
22
docs/deprecated/README.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Deprecated Docs
|
||||
|
||||
这个目录用于存放两类文档:
|
||||
|
||||
1. 已经完成、主要保留为历史记录的实施计划
|
||||
2. 已被现有实现或新方案替代的旧计划
|
||||
|
||||
放到这里并不代表这些文档“错误”,而是表示:
|
||||
|
||||
- 它们不再适合作为当前开发的主指导文档
|
||||
- 如果要了解历史决策、演进路径或旧设计背景,仍然可以参考
|
||||
|
||||
当前归档原则:
|
||||
|
||||
- 明确写明“已完成”的计划,优先归档
|
||||
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
|
||||
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
|
||||
|
||||
补充说明:
|
||||
|
||||
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
|
||||
- 这类文档如果有可用内容,应先吸收到 `docs/plans/` 或 `docs/technical/`,再归档保留来源记录
|
||||
117
docs/deprecated/earth-tv-live-module-plan.md
Normal file
117
docs/deprecated/earth-tv-live-module-plan.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# Earth 电视直播模块计划
|
||||
|
||||
## 目标
|
||||
|
||||
为 `Earth` 页面增加一个可配置、可扩展、可拖拽的电视直播模块:
|
||||
|
||||
- 后台可配置新闻直播源
|
||||
- 默认兜底源为央视 `CCTV-4`
|
||||
- 未来可通过采集器接入世界各地新闻直播源
|
||||
- Earth 工具栏 `显示控制` 子菜单新增电视按钮
|
||||
- 点击后打开一个与其他 HUD 一致的可拖拽/可关闭窗口
|
||||
- 窗口内部可播放或承载新闻直播页面
|
||||
|
||||
## 设计原则
|
||||
|
||||
- 第一阶段先交付“后台可配 + Earth 可用 + 默认可回退”的版本
|
||||
- 公开读取接口与后台管理接口分离
|
||||
- 手工配置源与采集器源共用统一的前端消费结构
|
||||
- Earth 里的电视窗口必须复用现有 HUD 拖拽、关闭、布局最大化逻辑
|
||||
- 小屏下优先保证窗口完整显示,超出部分在窗口内部滚动
|
||||
|
||||
## 分阶段实现
|
||||
|
||||
### Phase 1:后端配置与公开读取
|
||||
|
||||
- 在系统设置中新增 `tv` 分类
|
||||
- 定义直播源配置结构:
|
||||
- `default_source_id`
|
||||
- `auto_fallback`
|
||||
- `sources[]`
|
||||
- 每个直播源至少包含:
|
||||
- `id`
|
||||
- `name`
|
||||
- `provider`
|
||||
- `region`
|
||||
- `language`
|
||||
- `source_type`
|
||||
- `embed_url`
|
||||
- `stream_url`
|
||||
- `homepage_url`
|
||||
- `is_enabled`
|
||||
- `is_fallback`
|
||||
- `sort_order`
|
||||
- `collector_source`
|
||||
- `notes`
|
||||
- 默认兜底源使用央视官网 `CCTV-4` 直播页
|
||||
- 新增公开读取接口,供 Earth 页面无登录态读取直播源配置
|
||||
|
||||
### Phase 2:采集器扩展位
|
||||
|
||||
- 新增 `news_live_streams` collector 占位
|
||||
- 规范采集器入库数据结构,使其能与后台手工配置源合并
|
||||
- TV 公开接口支持合并:
|
||||
- 后台手工配置源
|
||||
- 采集器入库源
|
||||
- 保持手工配置源优先级更高,避免采集器覆盖人工兜底配置
|
||||
|
||||
### Phase 3:后台配置界面
|
||||
|
||||
- 在系统配置页新增 `电视直播` tab
|
||||
- 支持:
|
||||
- 查看当前默认源
|
||||
- 开关自动回退
|
||||
- 新增直播源
|
||||
- 编辑直播源
|
||||
- 删除直播源
|
||||
- 启用/禁用直播源
|
||||
- 将某个直播源设为默认源
|
||||
- 明确区分:
|
||||
- 手工配置源
|
||||
- 采集器来源
|
||||
|
||||
### Phase 4:Earth HUD 集成
|
||||
|
||||
- 在 `显示控制` 子菜单加入电视按钮
|
||||
- 新增 TV HUD 面板:
|
||||
- 可拖拽
|
||||
- 可关闭
|
||||
- 支持显示/隐藏状态同步
|
||||
- 参与布局最大化与恢复布局
|
||||
- 面板内容至少包含:
|
||||
- 当前频道标题
|
||||
- 源切换下拉菜单
|
||||
- 刷新按钮
|
||||
- 打开官网按钮
|
||||
- 播放区域
|
||||
|
||||
### Phase 5:播放策略
|
||||
|
||||
- 第一版优先支持 `iframe`/嵌入页类直播源
|
||||
- 为未来扩展保留:
|
||||
- `hls`
|
||||
- `video`
|
||||
- `external`
|
||||
- 如果默认源不可用:
|
||||
- 优先回退到标记为 `is_fallback=true` 的源
|
||||
- 若无明确回退源,则回退到第一个可用源
|
||||
- 面板内要有清晰的加载、错误、回退提示
|
||||
|
||||
### Phase 6:打磨与清理
|
||||
|
||||
- 统一 HUD 风格
|
||||
- 小屏下限制窗口尺寸并启用内部滚动
|
||||
- 避免窗口超出屏幕
|
||||
- 补最小验证
|
||||
- 清理临时代码、重复样式和无用资源
|
||||
|
||||
## 首版交付定义
|
||||
|
||||
当以下条件满足时,认为首版可用:
|
||||
|
||||
- 后台可以配置新闻直播源
|
||||
- Earth 可以读取并显示默认直播源
|
||||
- 工具栏可打开电视窗口
|
||||
- 电视窗口可拖拽、可关闭
|
||||
- 央视 `CCTV-4` 作为默认兜底源可被使用
|
||||
- 代码结构已为后续采集器接入预留统一接口
|
||||
165
docs/deprecated/hud-panel-component-plan.md
Normal file
165
docs/deprecated/hud-panel-component-plan.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# HUD Panel Component Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Unify Earth HUD panels into a reusable component layer so new panels can share:
|
||||
|
||||
- a consistent shell
|
||||
- a consistent header
|
||||
- a consistent action-button system
|
||||
- a consistent body and collapse pattern
|
||||
|
||||
## Scope
|
||||
|
||||
Target panels:
|
||||
|
||||
- `tv-panel`
|
||||
- `news-panel`
|
||||
- `legend`
|
||||
- `layer-panel`
|
||||
- `earth-stats`
|
||||
- `info-card`
|
||||
- settings modal header/actions
|
||||
|
||||
## Component Model
|
||||
|
||||
### Base shell
|
||||
|
||||
- `.hud-panel`
|
||||
- `.hud-panel--compact`
|
||||
- `.hud-panel--media`
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel-hidden`
|
||||
- `.hud-panel.is-dragging`
|
||||
- `.hud-panel.is-layout-animating`
|
||||
|
||||
### Header
|
||||
|
||||
- `.hud-panel__header`
|
||||
- `.hud-panel__title-group`
|
||||
- `.hud-panel__title`
|
||||
- `.hud-panel__subtitle`
|
||||
- `.hud-panel__chip`
|
||||
- `.hud-panel__actions`
|
||||
|
||||
Header baseline rule:
|
||||
|
||||
- Header title styling is fixed by the component layer and should not drift per panel
|
||||
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
|
||||
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
|
||||
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
|
||||
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
|
||||
|
||||
### Actions
|
||||
|
||||
- `.hud-panel__action`
|
||||
- `.hud-panel__action--icon`
|
||||
- `.hud-panel__action--collapse`
|
||||
- `.hud-panel__action--close`
|
||||
- `.hud-panel__action--refresh`
|
||||
- `.hud-panel__action--external`
|
||||
|
||||
Action-button baseline rule:
|
||||
|
||||
- Header action buttons must have one fixed default style baseline across all HUD panels
|
||||
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
|
||||
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
|
||||
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
|
||||
|
||||
### Body
|
||||
|
||||
- `.hud-panel__body`
|
||||
- `.hud-panel__body--scroll`
|
||||
- `.hud-panel__body--collapsible`
|
||||
|
||||
### Collapse behavior
|
||||
|
||||
- `.hud-panel--collapsed`
|
||||
- `.hud-panel--expand-up`
|
||||
- `.hud-panel--expand-down`
|
||||
|
||||
Adaptive collapse / expand rule:
|
||||
|
||||
- HUD panels support two expansion directions:
|
||||
- top-to-bottom expansion
|
||||
- bottom-to-top expansion
|
||||
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
|
||||
- Use:
|
||||
- `d` = available distance from the header anchor to the viewport bottom edge
|
||||
- `h` = expected expanded panel height
|
||||
- buffer = `20px`
|
||||
- Collapsed-state direction rule:
|
||||
- if `d > h + 20px`, the next action direction is `expand-up`
|
||||
- if `d <= h + 20px`, the next action direction is `expand-down`
|
||||
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
|
||||
- if the current direction is already `up`, keep it until `d <= h`
|
||||
- if the current direction is already `down`, keep it until `d > h + 20px`
|
||||
- The opposite edge is still a safety guard:
|
||||
- if the chosen side cannot fit at all, fall back to the other side if it can fit
|
||||
- if neither side fully fits, choose the side with more space and let the body scroll
|
||||
- If neither direction fully fits, choose the direction with more available space and let the body scroll
|
||||
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
|
||||
- The collapse icon describes the next action, not the current state
|
||||
- This mapping is fixed component behavior and must not drift per panel:
|
||||
- collapsed + expand-down => `expand_more`
|
||||
- expanded + expand-down => `expand_less`
|
||||
- collapsed + expand-up => `expand_less`
|
||||
- expanded + expand-up => `expand_more`
|
||||
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
|
||||
- Expansion direction and icon direction must come from one shared source of truth in the component controller
|
||||
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
|
||||
|
||||
## Tokens
|
||||
|
||||
Promote panel differences into CSS variables instead of duplicating selectors:
|
||||
|
||||
- `--hud-panel-padding`
|
||||
- `--hud-header-padding`
|
||||
- `--hud-header-gap`
|
||||
- `--hud-action-padding`
|
||||
- `--hud-action-gap`
|
||||
- `--hud-action-icon-size`
|
||||
- `--hud-body-gap`
|
||||
- `--hud-body-max-height`
|
||||
- `--hud-chip-radius`
|
||||
- `--hud-title-font-size`
|
||||
- `--hud-title-font-weight`
|
||||
- `--hud-title-letter-spacing`
|
||||
- `--hud-title-line-height`
|
||||
- `--hud-title-color`
|
||||
- `--hud-header-border-color`
|
||||
- `--hud-header-divider-opacity`
|
||||
- `--hud-expand-direction`
|
||||
|
||||
## Migration Order
|
||||
|
||||
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
|
||||
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
|
||||
3. Migrate `legend` and `layer-panel` into a compact variant
|
||||
4. Migrate `earth-stats` and `info-card`
|
||||
5. Align settings modal header/actions with the same action system
|
||||
6. Remove legacy one-off button selectors after verification
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not change panel behavior and data flow during the first pass
|
||||
- Keep old class names temporarily as compatibility hooks
|
||||
- Prefer variable overrides over per-panel reimplementation
|
||||
- Treat header action-button default styling as fixed component API, not per-panel design space
|
||||
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
|
||||
- Treat collapse direction as a component behavior contract, not a one-off panel trick
|
||||
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
|
||||
- Verify header alignment and drag/collapse behavior after each migration batch
|
||||
|
||||
## First Implementation Batch
|
||||
|
||||
Batch 1 should only do:
|
||||
|
||||
- shared header structure
|
||||
- shared action-button system
|
||||
- shared title typography and header border/divider baseline
|
||||
- shared collapsible body pattern
|
||||
- adaptive collapse direction logic and direction-aware collapse icons
|
||||
- migration of `tv-panel` and `news-panel`
|
||||
|
||||
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 地球3D可视化架构重构计划
|
||||
|
||||
## 背景
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# 卫星预测轨道显示功能
|
||||
|
||||
## TL;DR
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# UE5 3D 大屏客户端开发计划
|
||||
|
||||
## 项目概述
|
||||
@@ -1,3 +1,5 @@
|
||||
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
|
||||
|
||||
# WebGL Instancing 卫星渲染优化计划
|
||||
|
||||
## 背景
|
||||
37
docs/plans/README.md
Normal file
37
docs/plans/README.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Plans Docs
|
||||
|
||||
这里放“未来实施方案和未完成计划”的文档,重点回答:
|
||||
|
||||
- 我们准备做什么
|
||||
- 为什么要做
|
||||
- 分几期做
|
||||
- 当前差距和下一步是什么
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- Earth / BGP / 地形 / 天球实施方案
|
||||
- AI Playground 发展计划
|
||||
- backend / datasource / agent roadmap
|
||||
- UE5 MVP 方案
|
||||
|
||||
当前重点入口:
|
||||
|
||||
- [earth-mobile-drawer-ui-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-mobile-drawer-ui-plan.md)
|
||||
- [earth-compute-center-bgp-style-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-compute-center-bgp-style-plan.md)
|
||||
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
|
||||
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
|
||||
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
|
||||
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
|
||||
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
|
||||
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
|
||||
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 当前代码结构说明
|
||||
- 组件现状和实现入口
|
||||
- 已经落地的技术上下文说明
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)
|
||||
647
docs/plans/agents-agent-architecture-plan.md
Normal file
647
docs/plans/agents-agent-architecture-plan.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Agent Architecture Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the agent architecture for Planet.
|
||||
|
||||
The architecture is intentionally broader than datasource health checking.
|
||||
|
||||
It is designed to support both:
|
||||
|
||||
- datasource health governance
|
||||
- future situational-awareness workflows
|
||||
|
||||
The core idea is to avoid building a one-off "repair broken API links" agent.
|
||||
|
||||
Instead, Planet should grow a reusable agent runtime that can:
|
||||
|
||||
- collect evidence
|
||||
- evaluate signals
|
||||
- reason over incomplete information
|
||||
- generate proposals
|
||||
- produce assessments
|
||||
- execute limited actions under policy
|
||||
|
||||
|
||||
## Design Goal
|
||||
|
||||
Build an agent foundation that can evolve in this order:
|
||||
|
||||
1. datasource health checks
|
||||
2. datasource repair proposals
|
||||
3. signal correlation
|
||||
4. situational assessments
|
||||
5. controlled runtime actions
|
||||
|
||||
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
|
||||
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. Separate evidence from reasoning
|
||||
|
||||
- raw signals should be gathered first
|
||||
- deterministic checks should run before LLM reasoning
|
||||
|
||||
2. Agents do not own the defaults
|
||||
|
||||
- repository defaults remain human-owned
|
||||
- agents operate on runtime state, proposals, and overrides
|
||||
|
||||
3. Reasoning and action are different responsibilities
|
||||
|
||||
- many agents should be read-only or propose-only
|
||||
- only tightly controlled flows may apply changes
|
||||
|
||||
4. Shared runtime, specialized roles
|
||||
|
||||
- multiple agent roles should share the same object model and orchestration patterns
|
||||
- health and situational-awareness agents should not invent incompatible payloads
|
||||
|
||||
5. Auditability is mandatory
|
||||
|
||||
- every proposal, assessment, and applied action should be attributable
|
||||
|
||||
|
||||
## System Layers
|
||||
|
||||
Planet agent architecture should be split into four layers.
|
||||
|
||||
### 1. Signal Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- gather raw evidence from internal and external systems
|
||||
|
||||
Example sources:
|
||||
|
||||
- collector outputs
|
||||
- datasource health checks
|
||||
- logs
|
||||
- snapshots
|
||||
- alerts
|
||||
- web search results
|
||||
- scraped pages
|
||||
- external APIs
|
||||
- operator inputs
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- fetch
|
||||
- normalize
|
||||
- timestamp
|
||||
- tag with source and trust level
|
||||
|
||||
This layer should not make high-level judgments.
|
||||
|
||||
|
||||
### 2. Evaluation Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- perform deterministic analysis
|
||||
|
||||
Examples:
|
||||
|
||||
- reachability checks
|
||||
- schema validation
|
||||
- threshold checks
|
||||
- time-window comparisons
|
||||
- anomaly counters
|
||||
- completeness checks
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- classify signals into machine-readable findings
|
||||
- attach deterministic evidence
|
||||
|
||||
This layer should avoid LLM dependency whenever possible.
|
||||
|
||||
|
||||
### 3. Reasoning Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
|
||||
|
||||
Examples:
|
||||
|
||||
- endpoint migration inference
|
||||
- multi-source event correlation
|
||||
- causality hypotheses
|
||||
- ambiguity reduction
|
||||
- assessment narrative generation
|
||||
- action recommendation generation
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- synthesize evidence
|
||||
- produce hypotheses
|
||||
- rank confidence
|
||||
- explain reasoning boundaries
|
||||
|
||||
This is the main place where `aiprovider` and web search are used.
|
||||
|
||||
|
||||
### 4. Action Layer
|
||||
|
||||
Purpose:
|
||||
|
||||
- convert proposals or assessments into controlled system actions
|
||||
|
||||
Examples:
|
||||
|
||||
- create runtime override
|
||||
- create proposal
|
||||
- publish alert
|
||||
- update operator task queue
|
||||
- generate summary artifact
|
||||
- trigger follow-up verification
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- enforce policy
|
||||
- enforce approval requirements
|
||||
- verify post-action outcomes
|
||||
- record audit trails
|
||||
|
||||
|
||||
## Architecture Sketch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
|
||||
W["Web Search / Page Fetch / Docs"] --> B
|
||||
B --> C["Evaluation Layer"]
|
||||
C --> D["Findings"]
|
||||
D --> E["Reasoning Layer (LLM + Tools)"]
|
||||
E --> F["Proposals"]
|
||||
E --> G["Assessments"]
|
||||
F --> H["Action Layer"]
|
||||
H --> I["Runtime Overrides / Alerts / Tasks"]
|
||||
H --> J["Verification Loop"]
|
||||
J --> B
|
||||
|
||||
K["Policy Engine"] --> H
|
||||
L["Audit / History Store"] --> H
|
||||
L --> E
|
||||
L --> C
|
||||
```
|
||||
|
||||
|
||||
## Agent Roles
|
||||
|
||||
The first version should define these logical roles.
|
||||
|
||||
### 1. Health Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- datasource health governance
|
||||
|
||||
Inputs:
|
||||
|
||||
- datasource metadata
|
||||
- current endpoint
|
||||
- latest health records
|
||||
- latest failures
|
||||
- deterministic findings
|
||||
|
||||
Outputs:
|
||||
|
||||
- health interpretation
|
||||
- repair proposal
|
||||
- confidence
|
||||
- evidence references
|
||||
|
||||
Typical action level:
|
||||
|
||||
- propose-only
|
||||
|
||||
|
||||
### 2. Correlation Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- identify whether multiple signals describe the same event or related events
|
||||
|
||||
Inputs:
|
||||
|
||||
- findings from multiple collectors
|
||||
- time windows
|
||||
- region / ASN / prefix / cable relationships
|
||||
- prior incidents
|
||||
|
||||
Outputs:
|
||||
|
||||
- grouped event candidates
|
||||
- correlation rationale
|
||||
- confidence per relationship
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only
|
||||
|
||||
|
||||
### 3. Assessment Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- produce situational-awareness outputs
|
||||
|
||||
Inputs:
|
||||
|
||||
- grouped events
|
||||
- findings
|
||||
- current context
|
||||
- historical context
|
||||
- operator constraints
|
||||
|
||||
Outputs:
|
||||
|
||||
- structured assessment
|
||||
- risk summary
|
||||
- evidence-backed recommendations
|
||||
- missing-information list
|
||||
|
||||
Typical action level:
|
||||
|
||||
- read-only or propose-only
|
||||
|
||||
|
||||
### 4. Recovery Agent
|
||||
|
||||
Primary use case:
|
||||
|
||||
- carry low-risk proposals into controlled runtime actions
|
||||
|
||||
Inputs:
|
||||
|
||||
- approved proposal
|
||||
- policy constraints
|
||||
- trusted-domain rules
|
||||
- verification checks
|
||||
|
||||
Outputs:
|
||||
|
||||
- applied override
|
||||
- failed application
|
||||
- rollback request
|
||||
|
||||
Typical action level:
|
||||
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Shared Object Model
|
||||
|
||||
All agents should work on a shared object model.
|
||||
|
||||
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
|
||||
|
||||
### Signal
|
||||
|
||||
Represents a raw observed fact.
|
||||
|
||||
Examples:
|
||||
|
||||
- a datasource returned HTTP 404
|
||||
- a collector returned empty results
|
||||
- BGP updates spiked in one region
|
||||
- a known endpoint now redirects elsewhere
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sig_123",
|
||||
"type": "datasource.http_failure",
|
||||
"source": "ris_live_bgp",
|
||||
"occurred_at": "2026-04-08T10:00:00Z",
|
||||
"severity": "medium",
|
||||
"payload": {},
|
||||
"trust": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Finding
|
||||
|
||||
Represents a deterministic or semi-deterministic interpretation of one or more signals.
|
||||
|
||||
Examples:
|
||||
|
||||
- `schema_changed`
|
||||
- `endpoint_unreachable`
|
||||
- `data_volume_abnormally_low`
|
||||
- `event_cluster_detected`
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "find_123",
|
||||
"type": "datasource.schema_changed",
|
||||
"source_ids": ["sig_123"],
|
||||
"confidence": 0.92,
|
||||
"evidence": [],
|
||||
"details": {}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Proposal
|
||||
|
||||
Represents a recommended action, not an already-applied action.
|
||||
|
||||
Examples:
|
||||
|
||||
- switch endpoint to new URL
|
||||
- disable bad override
|
||||
- escalate issue for manual review
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "prop_123",
|
||||
"kind": "endpoint_override",
|
||||
"target": "telegeography_cables",
|
||||
"confidence": 0.84,
|
||||
"reason": "Official docs now point to a new API path",
|
||||
"payload": {},
|
||||
"evidence_urls": [],
|
||||
"status": "proposed"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Assessment
|
||||
|
||||
Represents a structured situational-awareness output for operators or downstream systems.
|
||||
|
||||
Examples:
|
||||
|
||||
- current network posture summary
|
||||
- incident impact assessment
|
||||
- risk and response recommendations
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "assess_123",
|
||||
"scope": "regional-network",
|
||||
"risk_level": "high",
|
||||
"summary": "Regional routing instability is increasing.",
|
||||
"key_risks": [],
|
||||
"evidence": [],
|
||||
"recommendations": [],
|
||||
"missing_data": []
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## State Machine
|
||||
|
||||
The shared orchestration flow should look like this:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Collect
|
||||
Collect --> Validate
|
||||
Validate --> Classify
|
||||
Classify --> Reason
|
||||
Reason --> Propose
|
||||
Reason --> Assess
|
||||
Propose --> Review
|
||||
Review --> Apply
|
||||
Apply --> Verify
|
||||
Verify --> Archive
|
||||
Assess --> Archive
|
||||
Archive --> [*]
|
||||
```
|
||||
|
||||
Definitions:
|
||||
|
||||
- `Collect`: gather signals
|
||||
- `Validate`: run deterministic checks
|
||||
- `Classify`: create findings
|
||||
- `Reason`: invoke LLM reasoning when needed
|
||||
- `Propose`: create change proposals
|
||||
- `Review`: policy or human approval
|
||||
- `Apply`: perform limited runtime action
|
||||
- `Verify`: confirm action effect
|
||||
- `Archive`: store artifacts and decisions
|
||||
|
||||
|
||||
## Permission Model
|
||||
|
||||
Each agent role should be assigned one of these action levels.
|
||||
|
||||
### `read-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- read signals
|
||||
- search web
|
||||
- fetch pages
|
||||
- read internal state
|
||||
- generate findings and assessments
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate config
|
||||
- write overrides
|
||||
- change live runtime behavior
|
||||
|
||||
|
||||
### `propose-only`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `read-only`
|
||||
- create proposals
|
||||
- create review tasks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- apply live changes
|
||||
|
||||
|
||||
### `apply-limited`
|
||||
|
||||
Allowed:
|
||||
|
||||
- everything in `propose-only`
|
||||
- write approved runtime overrides
|
||||
- trigger verification checks
|
||||
|
||||
Not allowed:
|
||||
|
||||
- mutate repository defaults
|
||||
- make destructive data changes
|
||||
- bypass policy engine
|
||||
|
||||
|
||||
## Runtime Components
|
||||
|
||||
The first durable architecture should introduce these components.
|
||||
|
||||
### 1. Signal Store
|
||||
|
||||
Stores normalized evidence and health outputs.
|
||||
|
||||
|
||||
### 2. Finding Store
|
||||
|
||||
Stores deterministic classifications that can be reused by multiple agents.
|
||||
|
||||
|
||||
### 3. Proposal Store
|
||||
|
||||
Stores recommended actions with evidence and confidence.
|
||||
|
||||
|
||||
### 4. Assessment Store
|
||||
|
||||
Stores structured situational-awareness outputs.
|
||||
|
||||
|
||||
### 5. Policy Engine
|
||||
|
||||
Decides:
|
||||
|
||||
- whether agent may run
|
||||
- whether proposal requires review
|
||||
- whether proposal may auto-apply
|
||||
- whether post-apply verification passed
|
||||
|
||||
|
||||
### 6. Override Store
|
||||
|
||||
Stores runtime-only configuration changes.
|
||||
|
||||
This is where endpoint repairs should live.
|
||||
|
||||
|
||||
## Relation To `aiprovider`
|
||||
|
||||
`aiprovider` should remain the model gateway.
|
||||
|
||||
It should not become the full agent runtime.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `aiprovider`
|
||||
- provider adaptation
|
||||
- prompt transport
|
||||
- model execution
|
||||
- protocol compatibility
|
||||
|
||||
- agent runtime
|
||||
- orchestration
|
||||
- signal handling
|
||||
- tool selection
|
||||
- proposal generation
|
||||
- policy and audit
|
||||
|
||||
This keeps provider concerns and agent behavior concerns separate.
|
||||
|
||||
|
||||
## Relation To Datasource Health
|
||||
|
||||
Datasource health becomes one vertical slice of this architecture.
|
||||
|
||||
Mapping:
|
||||
|
||||
- signal:
|
||||
- endpoint unreachable
|
||||
- schema mismatch
|
||||
- bad content type
|
||||
- finding:
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `moved_endpoint_suspected`
|
||||
- proposal:
|
||||
- runtime override suggestion
|
||||
- assessment:
|
||||
- datasource health summary for operators
|
||||
|
||||
|
||||
## Relation To Situational Awareness
|
||||
|
||||
Future situational-awareness capabilities should reuse the same flow:
|
||||
|
||||
- raw telemetry becomes signals
|
||||
- anomaly detection becomes findings
|
||||
- LLM correlation becomes reasoning
|
||||
- operator-facing output becomes assessments
|
||||
- policy-approved mitigations become actions
|
||||
|
||||
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
|
||||
|
||||
|
||||
## Suggested Delivery Sequence
|
||||
|
||||
### Phase A
|
||||
|
||||
- finalize shared object model
|
||||
- implement health-oriented signal and finding storage
|
||||
|
||||
### Phase B
|
||||
|
||||
- implement Health Agent
|
||||
- generate proposals only
|
||||
|
||||
### Phase C
|
||||
|
||||
- implement Assessment Agent
|
||||
- expose structured assessments via API
|
||||
|
||||
### Phase D
|
||||
|
||||
- implement Correlation Agent
|
||||
- support multi-source incident grouping
|
||||
|
||||
### Phase E
|
||||
|
||||
- implement Recovery Agent with policy-gated runtime actions
|
||||
|
||||
|
||||
## Recommended First Build
|
||||
|
||||
The first build should not try to implement every agent role.
|
||||
|
||||
Recommended initial slice:
|
||||
|
||||
- shared object model
|
||||
- health signals
|
||||
- health findings
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
This gives immediate value while preserving the longer-term architecture.
|
||||
|
||||
|
||||
## Non-Goals For The First Iteration
|
||||
|
||||
- repository YAML auto-rewrites
|
||||
- unrestricted autonomous action
|
||||
- full incident graph reasoning
|
||||
- automatic large-scale remediation
|
||||
- agent-owned configuration source of truth
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
|
||||
|
||||
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
|
||||
|
||||
The safest path is:
|
||||
|
||||
- deterministic checks first
|
||||
- agent reasoning second
|
||||
- proposals before actions
|
||||
- runtime overrides instead of default mutation
|
||||
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
346
docs/plans/agents-agent-runtime-roadmap.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# Agent Runtime Roadmap
|
||||
|
||||
## Overview
|
||||
|
||||
This document connects three existing planning threads into one implementation roadmap:
|
||||
|
||||
- `aiprovider` as the model gateway
|
||||
- datasource health governance as the first practical agent use case
|
||||
- situational awareness as the broader long-term target
|
||||
|
||||
Related documents:
|
||||
|
||||
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
|
||||
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
|
||||
|
||||
|
||||
## Big Picture
|
||||
|
||||
Planet should evolve in layers:
|
||||
|
||||
1. stable model gateway
|
||||
2. deterministic health and evidence collection
|
||||
3. agent runtime for reasoning and proposal generation
|
||||
4. situational-awareness assessments and controlled actions
|
||||
|
||||
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
|
||||
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
|
||||
B --> H["Datasource Health Services"]
|
||||
B --> R["Agent Runtime"]
|
||||
R --> P["aiprovider"]
|
||||
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
|
||||
|
||||
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
|
||||
H --> S
|
||||
S --> E["Evaluation Layer"]
|
||||
E --> F["Findings"]
|
||||
F --> R
|
||||
|
||||
W["Web Search / Page Fetch / Docs Fetch"] --> R
|
||||
R --> PR["Proposals"]
|
||||
R --> AS["Assessments"]
|
||||
|
||||
PR --> O["Runtime Overrides / Review Queue / Tasks"]
|
||||
AS --> SA["Situational Awareness APIs / UI"]
|
||||
|
||||
O --> V["Verification Loop"]
|
||||
V --> S
|
||||
```
|
||||
|
||||
|
||||
## Role Boundaries
|
||||
|
||||
### `aiprovider`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- provider compatibility
|
||||
- protocol adaptation
|
||||
- auth and model transport
|
||||
- request/response normalization
|
||||
|
||||
Not responsible for:
|
||||
|
||||
- agent orchestration
|
||||
- business workflows
|
||||
- datasource repair policy
|
||||
- situational-awareness domain logic
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- stable business APIs
|
||||
- auth and permissions
|
||||
- task orchestration
|
||||
- health records
|
||||
- proposal and override persistence
|
||||
- assessment exposure
|
||||
|
||||
|
||||
### Agent Runtime
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- consume findings and context
|
||||
- invoke LLMs via `aiprovider`
|
||||
- invoke tools such as web search
|
||||
- create proposals
|
||||
- create assessments
|
||||
- route to policy-controlled action paths
|
||||
|
||||
|
||||
## Delivery Sequence
|
||||
|
||||
## Stage 1: Gateway Foundation
|
||||
|
||||
Status:
|
||||
|
||||
- already in place
|
||||
|
||||
Delivered by current work:
|
||||
|
||||
- `aiprovider`
|
||||
- multi-provider compatibility
|
||||
- backend AI facade
|
||||
- MiniMax / Anthropic-compatible support
|
||||
- request-id propagation
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- the system already has a stable way to call models
|
||||
|
||||
|
||||
## Stage 2: Datasource Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- establish deterministic health observability
|
||||
|
||||
Key work:
|
||||
|
||||
- health check task runner
|
||||
- health result table
|
||||
- datasource health APIs
|
||||
- UI visibility
|
||||
- collector endpoint override precedence cleanup
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet knows which collectors are healthy before asking an LLM anything
|
||||
|
||||
|
||||
## Stage 3: Health Agent
|
||||
|
||||
Goal:
|
||||
|
||||
- let the first agent role operate on health failures
|
||||
|
||||
Key work:
|
||||
|
||||
- convert health failures into signals/findings
|
||||
- invoke agent only for failed or suspicious cases
|
||||
- produce repair proposals with evidence and confidence
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet can suggest endpoint repairs without mutating defaults
|
||||
|
||||
|
||||
## Stage 4: Runtime Repair Application
|
||||
|
||||
Goal:
|
||||
|
||||
- safely apply approved datasource repair proposals
|
||||
|
||||
Key work:
|
||||
|
||||
- override storage
|
||||
- policy-gated apply flow
|
||||
- verification after apply
|
||||
- rollback path
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- datasource repair becomes operationally useful without polluting repository defaults
|
||||
|
||||
|
||||
## Stage 5: Situational Awareness Assessments
|
||||
|
||||
Goal:
|
||||
|
||||
- reuse the same runtime for broader operator-facing assessment
|
||||
|
||||
Key work:
|
||||
|
||||
- normalize telemetry and incident evidence into signals/findings
|
||||
- build Assessment Agent
|
||||
- expose structured assessments through backend APIs and UI
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
|
||||
|
||||
|
||||
## Stage 6: Correlation and Controlled Actions
|
||||
|
||||
Goal:
|
||||
|
||||
- connect multiple sources into higher-level posture and event groupings
|
||||
|
||||
Key work:
|
||||
|
||||
- event correlation
|
||||
- incident grouping
|
||||
- recommendation scoring
|
||||
- controlled action routing
|
||||
|
||||
Primary outcome:
|
||||
|
||||
- Planet becomes a true agent-assisted situational-awareness system
|
||||
|
||||
|
||||
## Implementation Tracks
|
||||
|
||||
These tracks can progress in parallel, but they should stay loosely coupled.
|
||||
|
||||
### Track A: Config and Runtime Resolution
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource defaults
|
||||
- overrides
|
||||
- runtime precedence
|
||||
- audit trails
|
||||
|
||||
First milestone:
|
||||
|
||||
- health-safe override layer
|
||||
|
||||
|
||||
### Track B: Health and Evidence
|
||||
|
||||
Scope:
|
||||
|
||||
- deterministic checks
|
||||
- failure categorization
|
||||
- signal and finding persistence
|
||||
|
||||
First milestone:
|
||||
|
||||
- datasource health record system
|
||||
|
||||
|
||||
### Track C: Agent Runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- shared object model
|
||||
- orchestration flow
|
||||
- prompt/tool pipeline
|
||||
- policy integration
|
||||
|
||||
First milestone:
|
||||
|
||||
- Health Agent proposal pipeline
|
||||
|
||||
|
||||
### Track D: Situational Awareness
|
||||
|
||||
Scope:
|
||||
|
||||
- assessment schema
|
||||
- multi-source context assembly
|
||||
- operator-facing outputs
|
||||
|
||||
First milestone:
|
||||
|
||||
- structured assessment API
|
||||
|
||||
|
||||
## Shared Artifacts
|
||||
|
||||
To avoid fragmentation, these artifacts should be shared across all future agent work.
|
||||
|
||||
### Shared object model
|
||||
|
||||
- `Signal`
|
||||
- `Finding`
|
||||
- `Proposal`
|
||||
- `Assessment`
|
||||
|
||||
### Shared orchestration flow
|
||||
|
||||
- collect
|
||||
- validate
|
||||
- classify
|
||||
- reason
|
||||
- propose or assess
|
||||
- review or apply
|
||||
- verify
|
||||
- archive
|
||||
|
||||
### Shared policy model
|
||||
|
||||
- read-only
|
||||
- propose-only
|
||||
- apply-limited
|
||||
|
||||
|
||||
## Recommended Next Concrete Steps
|
||||
|
||||
1. Build Stage 2 first
|
||||
|
||||
- datasource health records
|
||||
- deterministic checks
|
||||
- no automatic repair
|
||||
|
||||
2. Then build Stage 3
|
||||
|
||||
- Health Agent
|
||||
- proposal generation only
|
||||
|
||||
3. Then Stage 4
|
||||
|
||||
- override apply flow
|
||||
- rollback and verification
|
||||
|
||||
4. Only after that start Stage 5
|
||||
|
||||
- broader situational-awareness assessment workflows
|
||||
|
||||
|
||||
## Why This Order
|
||||
|
||||
Because situational-awareness quality depends on reliable upstream data.
|
||||
|
||||
If datasource health is weak:
|
||||
|
||||
- agent reasoning quality will degrade
|
||||
- false explanations will increase
|
||||
- assessment trust will drop
|
||||
|
||||
So datasource health is not a side task.
|
||||
|
||||
It is the first operational foundation for the later situational-awareness system.
|
||||
|
||||
|
||||
## Summary
|
||||
|
||||
Planet should be built as:
|
||||
|
||||
- `aiprovider` for model access
|
||||
- backend services for orchestration and persistence
|
||||
- datasource health as the first evidence-governance layer
|
||||
- agent runtime as the reusable reasoning core
|
||||
- situational awareness as the long-term application layer
|
||||
|
||||
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.
|
||||
486
docs/plans/agents-datasource-health-plan.md
Normal file
486
docs/plans/agents-datasource-health-plan.md
Normal file
@@ -0,0 +1,486 @@
|
||||
# Datasource Health Plan
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines a phased plan for datasource health governance.
|
||||
|
||||
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
|
||||
|
||||
The key principle is:
|
||||
|
||||
- do not let runtime automation rewrite repository default config
|
||||
|
||||
Instead, split responsibilities across:
|
||||
|
||||
- default config
|
||||
- runtime overrides
|
||||
- health check records
|
||||
- agent-generated repair proposals
|
||||
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
|
||||
|
||||
These upstream dependencies can fail in several ways:
|
||||
|
||||
- endpoint becomes unreachable
|
||||
- endpoint still responds but schema changes
|
||||
- content-type changes
|
||||
- website shuts down or moves
|
||||
- mirror link disappears
|
||||
- HTML structure changes and scraping fails
|
||||
- endpoint requires a new path or new host
|
||||
|
||||
We want a system that can:
|
||||
|
||||
- detect datasource health degradation early
|
||||
- identify likely cause
|
||||
- search for updated endpoints when reasonable
|
||||
- apply safe runtime fixes without polluting default repo config
|
||||
- preserve auditability and rollback
|
||||
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. Default config is stable
|
||||
|
||||
- `backend/app/core/data_sources.yaml` remains the repository baseline.
|
||||
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
|
||||
|
||||
2. Runtime fixes are isolated
|
||||
|
||||
- Emergency or adaptive fixes should live in a runtime override layer.
|
||||
- Overrides should be reversible and auditable.
|
||||
|
||||
3. Deterministic checks come first
|
||||
|
||||
- Use normal programmatic health checks before using LLMs.
|
||||
- Only call an agent when deterministic checks indicate a meaningful failure.
|
||||
|
||||
4. Agents suggest before they mutate
|
||||
|
||||
- Agents should produce proposals with evidence and confidence.
|
||||
- Application of a proposal should be controlled by policy.
|
||||
|
||||
5. Every repair is attributable
|
||||
|
||||
- Store what changed, why, who or what suggested it, and when it was applied.
|
||||
|
||||
|
||||
## Configuration Layers
|
||||
|
||||
Recommended runtime precedence:
|
||||
|
||||
1. datasource endpoint override
|
||||
2. datasource DB endpoint override
|
||||
3. repository default YAML
|
||||
4. collector internal fallback logic
|
||||
|
||||
Definitions:
|
||||
|
||||
- repository default YAML:
|
||||
- `backend/app/core/data_sources.yaml`
|
||||
- versioned baseline
|
||||
- datasource DB endpoint override:
|
||||
- existing `DataSourceConfig.endpoint`
|
||||
- current runtime override entrypoint
|
||||
- datasource endpoint override:
|
||||
- a dedicated new override table
|
||||
- used for health-repair and proposal application
|
||||
- collector internal fallback logic:
|
||||
- final defensive fallback
|
||||
- should be minimized over time
|
||||
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
### 1. Deterministic Health Checks
|
||||
|
||||
Each collector gets a health profile with checks such as:
|
||||
|
||||
- endpoint resolves
|
||||
- HTTP request succeeds
|
||||
- status code is acceptable
|
||||
- content-type is expected
|
||||
- body parses successfully
|
||||
- minimum structural fields exist
|
||||
- sample item count is plausible
|
||||
- latency is within threshold
|
||||
|
||||
Output states:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
|
||||
|
||||
### 2. Agent-Assisted Repair Discovery
|
||||
|
||||
Only triggered when deterministic health checks fail or return suspicious structure.
|
||||
|
||||
Agent responsibilities:
|
||||
|
||||
- search for current official endpoint or replacement path
|
||||
- inspect likely upstream documentation or landing pages
|
||||
- compare candidate endpoint output to collector expectations
|
||||
- produce a repair proposal with confidence and evidence
|
||||
|
||||
Agent should not directly modify repository defaults.
|
||||
|
||||
|
||||
### 3. Safe Runtime Repair Application
|
||||
|
||||
Repair proposals can be:
|
||||
|
||||
- reviewed manually
|
||||
- auto-applied only under strict low-risk policy
|
||||
|
||||
Auto-apply should be limited to cases like:
|
||||
|
||||
- same trusted domain
|
||||
- highly similar response structure
|
||||
- repeated successful verification
|
||||
- confidence above threshold
|
||||
|
||||
|
||||
## Phased Delivery Plan
|
||||
|
||||
## Phase 1: Deterministic Health MVP
|
||||
|
||||
Goal:
|
||||
|
||||
- build health observability without automated repair
|
||||
|
||||
Scope:
|
||||
|
||||
- datasource health check task runner
|
||||
- datasource health result persistence
|
||||
- endpoint reachability + parse checks
|
||||
- dashboard or API visibility into health status
|
||||
|
||||
Deliverables:
|
||||
|
||||
- health check service
|
||||
- health check record table
|
||||
- status endpoint
|
||||
- scheduled or manual check trigger
|
||||
|
||||
No agent usage yet.
|
||||
|
||||
|
||||
## Phase 2: Agent Repair Proposals
|
||||
|
||||
Goal:
|
||||
|
||||
- let agent investigate failing sources and propose updated endpoints
|
||||
|
||||
Scope:
|
||||
|
||||
- invoke agent only when datasource health is `failed` or `schema_changed`
|
||||
- web search + page inspection
|
||||
- candidate endpoint extraction
|
||||
- proposal persistence
|
||||
|
||||
Deliverables:
|
||||
|
||||
- repair proposal schema
|
||||
- proposal generation pipeline
|
||||
- confidence and evidence model
|
||||
- operator review view or API
|
||||
|
||||
Still no automatic config mutation.
|
||||
|
||||
|
||||
## Phase 3: Runtime Overrides
|
||||
|
||||
Goal:
|
||||
|
||||
- allow approved proposals to take effect safely at runtime
|
||||
|
||||
Scope:
|
||||
|
||||
- add dedicated override storage
|
||||
- runtime resolution prefers override over default config
|
||||
- proposal application writes override only
|
||||
|
||||
Deliverables:
|
||||
|
||||
- endpoint override table
|
||||
- override-aware resolution logic
|
||||
- apply/reject endpoints
|
||||
- rollback endpoint
|
||||
|
||||
Repository default YAML remains untouched.
|
||||
|
||||
|
||||
## Phase 4: Limited Auto-Apply
|
||||
|
||||
Goal:
|
||||
|
||||
- safely automate a narrow slice of low-risk repairs
|
||||
|
||||
Scope:
|
||||
|
||||
- policy engine for auto-apply
|
||||
- same-domain or trusted-domain checks
|
||||
- structure validation
|
||||
- staged verification after apply
|
||||
|
||||
Deliverables:
|
||||
|
||||
- auto-apply rules
|
||||
- audit logs
|
||||
- automatic post-apply health verification
|
||||
- auto-disable or rollback on regression
|
||||
|
||||
|
||||
## Data Model Draft
|
||||
|
||||
### datasource_health_checks
|
||||
|
||||
Purpose:
|
||||
|
||||
- store each health evaluation result
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
`details` can store structured diagnostic data such as:
|
||||
|
||||
- parsed fields
|
||||
- schema mismatch summary
|
||||
- retry count
|
||||
- exception class
|
||||
|
||||
|
||||
### datasource_repair_proposals
|
||||
|
||||
Purpose:
|
||||
|
||||
- store agent-generated repair suggestions
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `old_endpoint`
|
||||
- `candidate_endpoint`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `evidence_urls`
|
||||
- `evidence_summary`
|
||||
- `status`
|
||||
- `created_by`
|
||||
- `created_at`
|
||||
- `reviewed_at`
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `proposed`
|
||||
- `approved`
|
||||
- `rejected`
|
||||
- `applied`
|
||||
- `expired`
|
||||
|
||||
|
||||
### datasource_endpoint_overrides
|
||||
|
||||
Purpose:
|
||||
|
||||
- runtime endpoint override layer
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint`
|
||||
- `reason`
|
||||
- `source`
|
||||
- `proposal_id`
|
||||
- `enabled`
|
||||
- `created_at`
|
||||
- `updated_at`
|
||||
|
||||
Suggested `source` values:
|
||||
|
||||
- `manual`
|
||||
- `health-agent`
|
||||
- `migration`
|
||||
|
||||
|
||||
## API Draft
|
||||
|
||||
### Health
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
### Repair proposals
|
||||
|
||||
- `GET /api/v1/datasources/{id}/repair-proposals`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
|
||||
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
|
||||
|
||||
### Overrides
|
||||
|
||||
- `GET /api/v1/datasources/{id}/overrides`
|
||||
- `POST /api/v1/datasources/{id}/overrides`
|
||||
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
|
||||
|
||||
|
||||
## Agent Contract Draft
|
||||
|
||||
When deterministic health fails, the agent should receive:
|
||||
|
||||
- datasource name
|
||||
- collector name
|
||||
- current endpoint
|
||||
- current failure mode
|
||||
- expected response shape summary
|
||||
- known trusted domains
|
||||
|
||||
Expected output:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "proposal",
|
||||
"candidate_endpoint": "https://example.com/api/v2/data",
|
||||
"confidence": 0.86,
|
||||
"reason": "Official docs now point to v2 endpoint",
|
||||
"evidence_urls": [
|
||||
"https://example.com/docs/api",
|
||||
"https://example.com/changelog"
|
||||
],
|
||||
"notes": "Response shape appears compatible after light field remapping"
|
||||
}
|
||||
```
|
||||
|
||||
The agent should never output "rewrite the default yaml" as its primary action.
|
||||
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
### Risk: wrong endpoint chosen by agent
|
||||
|
||||
Mitigation:
|
||||
|
||||
- use trusted-domain allowlists
|
||||
- require evidence URLs
|
||||
- require confidence threshold
|
||||
- add manual review for medium-risk sources
|
||||
|
||||
|
||||
### Risk: endpoint responds but schema silently changed
|
||||
|
||||
Mitigation:
|
||||
|
||||
- deterministic schema checks
|
||||
- parse and sample validation
|
||||
- content-type checks
|
||||
- collector-specific required fields
|
||||
|
||||
|
||||
### Risk: automatic runtime override causes hidden drift
|
||||
|
||||
Mitigation:
|
||||
|
||||
- store all overrides explicitly
|
||||
- mark source of override
|
||||
- keep default YAML unchanged
|
||||
- expose active overrides in API/UI
|
||||
|
||||
|
||||
### Risk: persistent bad override breaks data collection
|
||||
|
||||
Mitigation:
|
||||
|
||||
- allow rollback
|
||||
- keep parent/default endpoint visible
|
||||
- re-run verification after apply
|
||||
- auto-disable override on repeated failure
|
||||
|
||||
|
||||
## Operational Policy Recommendations
|
||||
|
||||
1. Do not auto-apply for high-value or high-fragility sources initially.
|
||||
|
||||
2. Use manual approval for:
|
||||
|
||||
- scraped HTML sources
|
||||
- unofficial mirrors
|
||||
- sources with auth or rate-limit complexity
|
||||
- sources with legal or trust ambiguity
|
||||
|
||||
3. Allow auto-apply only for:
|
||||
|
||||
- same-domain version bumps
|
||||
- obvious official migration paths
|
||||
- repeated passing verification
|
||||
|
||||
4. Expose health + proposal + override state together in one operator view.
|
||||
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Phase 1
|
||||
- health result table
|
||||
- deterministic checks
|
||||
- API and UI visibility
|
||||
|
||||
2. Phase 2
|
||||
- proposal table
|
||||
- agent prompt/output contract
|
||||
- proposal generation job
|
||||
|
||||
3. Phase 3
|
||||
- runtime override table
|
||||
- resolver precedence update
|
||||
- apply/reject endpoints
|
||||
|
||||
4. Phase 4
|
||||
- auto-apply rules
|
||||
- rollback policy
|
||||
- operator automation
|
||||
|
||||
|
||||
## Out Of Scope For The First Iteration
|
||||
|
||||
- direct automatic mutation of repository default YAML
|
||||
- automatic git commits by repair agents
|
||||
- unrestricted autonomous endpoint replacement
|
||||
- fully generalized schema remapping engine
|
||||
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted results
|
||||
- manual visibility
|
||||
- no automatic repair
|
||||
|
||||
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.
|
||||
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
478
docs/plans/agents-datasource-health-stage2-tasks.md
Normal file
@@ -0,0 +1,478 @@
|
||||
# Datasource Health Stage 2 Tasks
|
||||
|
||||
## Goal
|
||||
|
||||
Stage 2 focuses on the first practical operational layer:
|
||||
|
||||
- deterministic datasource health checks
|
||||
- persisted health results
|
||||
- health visibility through API and UI
|
||||
- no agent-assisted repair yet
|
||||
|
||||
This stage should make Planet capable of answering:
|
||||
|
||||
- which collectors are healthy
|
||||
- which collectors are degraded
|
||||
- which collectors are failing
|
||||
- why they are failing at a basic deterministic level
|
||||
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- datasource health data model
|
||||
- deterministic health check service
|
||||
- manual and scheduled health check triggers
|
||||
- health result APIs
|
||||
- frontend visibility
|
||||
|
||||
Excluded:
|
||||
|
||||
- LLM reasoning
|
||||
- web-search-based repair proposals
|
||||
- automatic endpoint rewriting
|
||||
- runtime override application
|
||||
|
||||
|
||||
## Delivery Target
|
||||
|
||||
At the end of Stage 2, an operator should be able to:
|
||||
|
||||
1. see health status for each collector
|
||||
2. trigger a health check manually
|
||||
3. inspect the latest failure reason
|
||||
4. inspect the last checked endpoint
|
||||
5. understand whether the problem is:
|
||||
- unreachable
|
||||
- auth-related
|
||||
- rate-limit-related
|
||||
- schema-related
|
||||
- empty-data-related
|
||||
|
||||
|
||||
## Work Breakdown
|
||||
|
||||
## A. Data Model
|
||||
|
||||
### A1. Add datasource health record table
|
||||
|
||||
Create a new model, for example:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `id`
|
||||
- `datasource_id`
|
||||
- `collector_name`
|
||||
- `endpoint_checked`
|
||||
- `status`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
- `checked_at`
|
||||
|
||||
Suggested status enum values:
|
||||
|
||||
- `healthy`
|
||||
- `degraded`
|
||||
- `failed`
|
||||
- `schema_changed`
|
||||
- `rate_limited`
|
||||
- `auth_required`
|
||||
- `empty_result`
|
||||
|
||||
|
||||
### A2. Add datasource health summary fields
|
||||
|
||||
Option A:
|
||||
|
||||
- keep summary only in the health check table
|
||||
|
||||
Option B:
|
||||
|
||||
- also add summary fields on `data_sources`
|
||||
|
||||
Recommended first step:
|
||||
|
||||
- do not mutate `data_sources` schema yet
|
||||
- derive summary from the latest health record
|
||||
|
||||
|
||||
### A3. Migration task
|
||||
|
||||
Add migration for the health table.
|
||||
|
||||
Deliverables:
|
||||
|
||||
- migration file
|
||||
- model registration
|
||||
|
||||
|
||||
## B. Health Check Engine
|
||||
|
||||
### B1. Define health check service
|
||||
|
||||
Add a new service module, for example:
|
||||
|
||||
- `backend/app/services/datasource_health.py`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- resolve effective endpoint
|
||||
- execute deterministic check
|
||||
- classify result
|
||||
- persist health record
|
||||
|
||||
|
||||
### B2. Define shared result schema
|
||||
|
||||
Create a typed result object, for example:
|
||||
|
||||
- `HealthCheckResult`
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `status`
|
||||
- `endpoint_checked`
|
||||
- `http_status`
|
||||
- `content_type`
|
||||
- `latency_ms`
|
||||
- `sample_count`
|
||||
- `error_message`
|
||||
- `details`
|
||||
|
||||
|
||||
### B3. Implement base deterministic checks
|
||||
|
||||
Every datasource should go through a minimal baseline check:
|
||||
|
||||
1. resolve endpoint
|
||||
2. perform request
|
||||
3. measure latency
|
||||
4. inspect status code
|
||||
5. inspect content type
|
||||
6. inspect body shape
|
||||
|
||||
Classification rules:
|
||||
|
||||
- network error -> `failed`
|
||||
- HTTP 401/403 -> `auth_required`
|
||||
- HTTP 429 -> `rate_limited`
|
||||
- HTTP 404/410 -> `failed`
|
||||
- parse failure -> `schema_changed`
|
||||
- zero or suspiciously empty results -> `empty_result` or `degraded`
|
||||
- valid parse -> `healthy`
|
||||
|
||||
|
||||
### B4. Add collector-aware adapters
|
||||
|
||||
Some collectors do not use the same fetch semantics.
|
||||
|
||||
Add adapter profiles such as:
|
||||
|
||||
- `http_json`
|
||||
- `http_csv`
|
||||
- `html_scrape`
|
||||
- `stream_probe`
|
||||
- `auth_session_http`
|
||||
|
||||
Initial mapping suggestion:
|
||||
|
||||
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
|
||||
- `fao` -> `http_csv`
|
||||
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
|
||||
- `ris_live` -> `stream_probe`
|
||||
- `spacetrack` -> `auth_session_http`
|
||||
|
||||
|
||||
### B5. Add sample validation hooks
|
||||
|
||||
For each adapter, add a lightweight validation rule.
|
||||
|
||||
Examples:
|
||||
|
||||
- JSON array length > 0
|
||||
- CSV rows > 1
|
||||
- HTML page contains expected table or script patterns
|
||||
- stream source yields at least one valid event within timeout
|
||||
|
||||
|
||||
## C. Persistence and Query Layer
|
||||
|
||||
### C1. Save every check run
|
||||
|
||||
Each health check should insert a record.
|
||||
|
||||
Do not overwrite history in Stage 2.
|
||||
|
||||
|
||||
### C2. Add latest-health query helpers
|
||||
|
||||
Add helper functions to fetch:
|
||||
|
||||
- latest health record by datasource
|
||||
- latest failed health record
|
||||
- recent health history
|
||||
|
||||
|
||||
### C3. Optional retention policy
|
||||
|
||||
For Stage 2, retention can be deferred.
|
||||
|
||||
If desired, keep only:
|
||||
|
||||
- last N records per datasource
|
||||
|
||||
|
||||
## D. API Layer
|
||||
|
||||
### D1. Add health list endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- current endpoint
|
||||
- latest health status
|
||||
- last checked time
|
||||
- short reason
|
||||
|
||||
|
||||
### D2. Add per-datasource health detail endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `GET /api/v1/datasources/{id}/health`
|
||||
|
||||
Returns:
|
||||
|
||||
- latest record
|
||||
- recent history
|
||||
- detailed classification fields
|
||||
|
||||
|
||||
### D3. Add manual health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/{id}/health-check`
|
||||
|
||||
Behavior:
|
||||
|
||||
- run a health check now
|
||||
- persist the result
|
||||
- return the new record
|
||||
|
||||
|
||||
### D4. Add bulk health trigger endpoint
|
||||
|
||||
Suggested endpoint:
|
||||
|
||||
- `POST /api/v1/datasources/health-check-all`
|
||||
|
||||
Behavior:
|
||||
|
||||
- enqueue or run health checks for all active datasources
|
||||
|
||||
|
||||
## E. Scheduling
|
||||
|
||||
### E1. Add health scheduler task
|
||||
|
||||
Decide scheduling strategy.
|
||||
|
||||
Recommended first version:
|
||||
|
||||
- run collector jobs and health checks separately
|
||||
- health checks run on a lower frequency
|
||||
|
||||
Suggested frequency:
|
||||
|
||||
- every 6h or 12h for most datasources
|
||||
- optionally on-demand only in the very first cut
|
||||
|
||||
|
||||
### E2. Prevent health check collision with collection
|
||||
|
||||
Rules:
|
||||
|
||||
- health checks should not disrupt active collection
|
||||
- they should use light requests
|
||||
- if a collector is currently running, health check may:
|
||||
- skip
|
||||
- or use a lightweight endpoint probe only
|
||||
|
||||
|
||||
## F. Frontend
|
||||
|
||||
### F1. Add health columns to datasource list
|
||||
|
||||
Update:
|
||||
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
|
||||
Suggested new columns:
|
||||
|
||||
- health status
|
||||
- last checked
|
||||
- reason summary
|
||||
|
||||
|
||||
### F2. Add manual health check action
|
||||
|
||||
Per datasource:
|
||||
|
||||
- button or dropdown action:
|
||||
- `健康检查`
|
||||
|
||||
|
||||
### F3. Add health detail drawer or modal
|
||||
|
||||
Show:
|
||||
|
||||
- endpoint checked
|
||||
- status
|
||||
- HTTP status
|
||||
- content type
|
||||
- sample count
|
||||
- error message
|
||||
- last few results
|
||||
|
||||
|
||||
### F4. Add basic visual language
|
||||
|
||||
Suggested colors:
|
||||
|
||||
- green -> healthy
|
||||
- yellow -> degraded
|
||||
- orange -> rate-limited / auth-required
|
||||
- red -> failed / schema-changed
|
||||
|
||||
|
||||
## G. Observability
|
||||
|
||||
### G1. Structured logging
|
||||
|
||||
Every health check should log:
|
||||
|
||||
- datasource id
|
||||
- collector name
|
||||
- endpoint
|
||||
- status
|
||||
- latency
|
||||
- failure class
|
||||
|
||||
|
||||
### G2. Optional metrics
|
||||
|
||||
If metrics are added later, useful counters include:
|
||||
|
||||
- health checks total
|
||||
- health checks failed
|
||||
- schema changes detected
|
||||
- rate limited checks
|
||||
|
||||
|
||||
## H. Tests
|
||||
|
||||
### H1. Unit tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- status classification
|
||||
- content type classification
|
||||
- adapter behavior
|
||||
- latest-health query helpers
|
||||
|
||||
|
||||
### H2. API tests
|
||||
|
||||
Add tests for:
|
||||
|
||||
- health endpoints require auth
|
||||
- manual trigger endpoint works
|
||||
- list endpoint returns latest status
|
||||
|
||||
|
||||
### H3. Failure-path tests
|
||||
|
||||
Add coverage for:
|
||||
|
||||
- HTTP 404
|
||||
- HTTP 429
|
||||
- invalid JSON
|
||||
- empty response
|
||||
- parse mismatch
|
||||
|
||||
|
||||
## Suggested File Plan
|
||||
|
||||
Possible implementation files:
|
||||
|
||||
- `backend/app/models/datasource_health_check.py`
|
||||
- `backend/app/services/datasource_health.py`
|
||||
- `backend/app/schemas/datasource_health.py`
|
||||
- `backend/app/api/v1/datasource_health.py`
|
||||
- migration file under the project migration system
|
||||
|
||||
Likely touched existing files:
|
||||
|
||||
- `backend/app/api/main.py`
|
||||
- `frontend/src/pages/DataSources/DataSources.tsx`
|
||||
- `backend/tests/test_api.py`
|
||||
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. Add model and migration
|
||||
2. Add service and result schema
|
||||
3. Add deterministic adapters
|
||||
4. Add manual trigger API
|
||||
5. Add list/detail API
|
||||
6. Add frontend visibility
|
||||
7. Add scheduled checks
|
||||
8. Expand tests
|
||||
|
||||
|
||||
## Minimal First Milestone
|
||||
|
||||
If we want the fastest useful slice, do this first:
|
||||
|
||||
1. health table
|
||||
2. deterministic check service
|
||||
3. manual per-datasource health check API
|
||||
4. latest health list API
|
||||
5. frontend status badge column
|
||||
|
||||
That is enough to start operating the system and will provide the input layer for Stage 3.
|
||||
|
||||
|
||||
## Dependency On Later Stages
|
||||
|
||||
Stage 2 outputs become direct inputs for Stage 3.
|
||||
|
||||
Specifically:
|
||||
|
||||
- failed or schema-changed health records become agent triggers
|
||||
- health history becomes repair context
|
||||
- endpoint_checked becomes proposal baseline
|
||||
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Stage 2 is done when:
|
||||
|
||||
- every active datasource can be health-checked deterministically
|
||||
- the latest health state is visible in API and UI
|
||||
- operators can manually trigger checks
|
||||
- failures are categorized into stable machine-readable statuses
|
||||
- no LLM is required for core health visibility
|
||||
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
309
docs/plans/agents-situational-awareness-foundation-plan.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Situational Awareness Foundation Plan
|
||||
|
||||
## 定位
|
||||
|
||||
当前这套 AI 能力应被视为 `态势感知服务底座`,而不是完整的态势感知产品。
|
||||
|
||||
也就是说,现阶段的目标不是:
|
||||
|
||||
- 做一个“什么都能分析”的万能 AI 页面
|
||||
- 让模型在证据不足时替代人工研判
|
||||
- 过早把页面做成完整指挥大屏
|
||||
|
||||
现阶段真正要做的是:
|
||||
|
||||
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
|
||||
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
|
||||
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
|
||||
|
||||
## 当前现实约束
|
||||
|
||||
### 1. 数据维度不足
|
||||
|
||||
目前系统能提供的主要证据仍集中在:
|
||||
|
||||
- BGP incidents / anomalies / events
|
||||
- collector coverage
|
||||
- datasource health / platform alerts
|
||||
- prefix geography 的部分归属信息
|
||||
|
||||
当前明显还缺:
|
||||
|
||||
- 流量异常与业务指标
|
||||
- 电商、支付、物流等业务侧指标
|
||||
- 更丰富的资产、链路、区域、行业画像
|
||||
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
|
||||
|
||||
这意味着:
|
||||
|
||||
- 模型现在可以做“基于现有证据的摘要与归纳”
|
||||
- 但还不能可靠地做“跨维度因果研判”
|
||||
|
||||
### 2. 维度之间联动还弱
|
||||
|
||||
目前不同模块之间更多是“并列展示”,还不是“强关联分析”:
|
||||
|
||||
- 系统告警和 BGP 事件还没有统一事件模型
|
||||
- collector bias 与真实区域热度还没有完全剥离
|
||||
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
|
||||
|
||||
这意味着:
|
||||
|
||||
- 当前更适合做 `brief / overview / operator notes`
|
||||
- 还不适合过度承诺“自动态势判断”
|
||||
|
||||
### 3. 结构化 assessment 还未成为主输出
|
||||
|
||||
虽然已经有 BGP brief、系统告警 brief、态势告警 brief,但目前主输出仍偏向:
|
||||
|
||||
- 文本摘要
|
||||
- facts/context 附带证据
|
||||
|
||||
后续真正要服务态势感知,需要更稳定的结构化输出,例如:
|
||||
|
||||
- summary
|
||||
- key risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing data
|
||||
|
||||
## 当前基座已经具备的能力
|
||||
|
||||
### 1. AI 调用边界已经明确
|
||||
|
||||
- `aiprovider` 负责模型协议与 provider 兼容
|
||||
- `backend` 负责业务 API、证据整合和鉴权
|
||||
- `frontend` 负责页面入口与结果展示
|
||||
|
||||
### 2. 页面级 AI 入口已经开始成型
|
||||
|
||||
当前已经有或正在收口的入口:
|
||||
|
||||
- `Playground`
|
||||
- 用于链路验证与 provider 诊断
|
||||
- `BGP AI 简报`
|
||||
- 用于 BGP 事实摘要和区域风险归纳
|
||||
- `Alerts`
|
||||
- 用于系统告警、BGP 告警、态势告警三类入口
|
||||
|
||||
### 3. 证据优先的方向已经建立
|
||||
|
||||
已经不再只依赖人工在 Playground 中手填 prompt,系统开始具备:
|
||||
|
||||
- 从真实业务数据生成事实输入
|
||||
- 保存 facts/context 快照
|
||||
- 回看 AI 输出时同时回看证据
|
||||
|
||||
这一步非常关键,因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
|
||||
|
||||
## 近期收尾建议
|
||||
|
||||
这些事情都属于“底座收口”,值得做,但不应该再继续重产品包装。
|
||||
|
||||
### 1. 统一 Alerts 页面
|
||||
|
||||
已采用:
|
||||
|
||||
- 一个 `Alerts` 页面
|
||||
- 三个 tab:
|
||||
- `系统告警`
|
||||
- `BGP 告警`
|
||||
- `态势告警`
|
||||
|
||||
收尾重点:
|
||||
|
||||
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
|
||||
- 不额外扩展成多个独立二级页面
|
||||
|
||||
### 2. 保持 Playground 为测试台
|
||||
|
||||
原则:
|
||||
|
||||
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
|
||||
- 不继续堆“万能业务分析器”式交互
|
||||
|
||||
### 3. 把 brief 能力当服务能力而不是页面特效
|
||||
|
||||
页面现在能看到按钮和结果,这很好,但更重要的是:
|
||||
|
||||
- 后端接口稳定
|
||||
- facts/context 可追踪
|
||||
- 输出结构后续可升级
|
||||
|
||||
### 4. 导航结构先收口,不继续平铺一级菜单
|
||||
|
||||
随着后续能力扩展,系统很可能继续新增:
|
||||
|
||||
- 海缆
|
||||
- 算力中心
|
||||
- 战争信息
|
||||
- 电商分析
|
||||
- 其他专题观测页
|
||||
|
||||
如果继续把这些入口全部平铺在左侧一级菜单中,会带来两个问题:
|
||||
|
||||
- 一级菜单过长,用户难以判断先进入哪个上下文
|
||||
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
|
||||
|
||||
因此近期应明确采用分组导航,而不是继续扩展平铺菜单。
|
||||
|
||||
推荐的导航分组如下:
|
||||
|
||||
- `总览`
|
||||
- 仪表盘
|
||||
- Earth
|
||||
- `专题观测`
|
||||
- BGP 观测
|
||||
- 采集数据
|
||||
- 后续可扩展:海缆、算力中心、战争信息、电商分析
|
||||
- `告警与研判`
|
||||
- Alerts
|
||||
- `运维与配置`
|
||||
- 数据源
|
||||
- AI Playground
|
||||
- 用户管理
|
||||
- 系统配置
|
||||
|
||||
这套结构的含义是:
|
||||
|
||||
- `专题观测` 页面负责看某个维度本身
|
||||
- `Alerts` 负责跨模块风险与值班工作台
|
||||
- `Playground` 保持为测试台,不挤占业务导航语义
|
||||
|
||||
短期收尾时,应优先重组现有入口,而不是继续增加新的一级菜单。
|
||||
|
||||
## 后续路线
|
||||
|
||||
## Phase 1:服务底座稳固
|
||||
|
||||
目标:
|
||||
|
||||
- 不追求“更炫的 AI 页面”
|
||||
- 先把当前接口、证据、存储和页面入口收稳
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一页面级 AI 入口模式
|
||||
- 统一 brief response schema
|
||||
- 保证 facts/context 在前后端都可回看
|
||||
- 继续清理 mock 和临时分支逻辑
|
||||
|
||||
完成标准:
|
||||
|
||||
- 每个 AI 入口都是真实链路
|
||||
- 每个 AI 结果都能追溯到证据输入
|
||||
|
||||
## Phase 2:Evidence-first Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 从“文本摘要”升级成“结构化 assessment”
|
||||
|
||||
工作项:
|
||||
|
||||
- 为 brief/assessment 定义统一 schema
|
||||
- 固化:
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- confidence
|
||||
- recommendations
|
||||
- missing_data
|
||||
- 页面以结构化区块展示,而不只是大段文本
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 输出可持久化、可比较、可审计
|
||||
|
||||
## Phase 3:多维证据接入
|
||||
|
||||
目标:
|
||||
|
||||
- 让“态势感知”真正拥有更多维度,而不是只靠 BGP 与系统告警
|
||||
|
||||
优先接入方向:
|
||||
|
||||
- datasource health findings
|
||||
- 流量或业务指标
|
||||
- 区域/资产/链路映射
|
||||
- 外部事件与公告
|
||||
- 业务垂直数据,例如电商分析相关指标
|
||||
|
||||
完成标准:
|
||||
|
||||
- AI 能基于多个维度做交叉说明
|
||||
- 不再只围绕单一模块自说自话
|
||||
|
||||
## Phase 4:Correlation Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不同来源的信号不再只是并列,而是形成统一的事件关联
|
||||
|
||||
工作项:
|
||||
|
||||
- 统一 signal/finding 模型
|
||||
- 跨模块事件聚合
|
||||
- 证据来源权重
|
||||
- collector bias 与真实热度分离
|
||||
|
||||
完成标准:
|
||||
|
||||
- 系统能回答“这些异常是不是同一件事”
|
||||
- 系统能回答“哪些结论只是观测偏差”
|
||||
|
||||
## Phase 5:Agent-assisted Situational Awareness
|
||||
|
||||
目标:
|
||||
|
||||
- 在证据足够的前提下,再让 agent 负责更复杂的推理与建议
|
||||
|
||||
工作项:
|
||||
|
||||
- 复用现有 agent runtime 规划
|
||||
- 引入 web search / docs fetch / repair proposal 等能力
|
||||
- 但始终坚持:
|
||||
- evidence first
|
||||
- proposal before action
|
||||
- no silent mutation of defaults
|
||||
|
||||
完成标准:
|
||||
|
||||
- agent 成为证据驱动的分析层
|
||||
- 而不是一个“万能猜测层”
|
||||
|
||||
## 设计原则
|
||||
|
||||
### 1. 先底座,后产品化
|
||||
|
||||
先把服务链路和证据模型做好,再做更大的页面表达。
|
||||
|
||||
### 2. 先证据,后判断
|
||||
|
||||
事实输入应先稳定,再让模型做归纳。
|
||||
|
||||
### 3. 先专用 brief,后统一态势层
|
||||
|
||||
先让各业务页有各自可信的 AI 入口,再考虑统一态势页。
|
||||
|
||||
### 4. 先 proposal,后自动动作
|
||||
|
||||
涉及修复、覆盖、写配置、调任务的动作,都应经过 proposal 和审计。
|
||||
|
||||
## 当前建议结论
|
||||
|
||||
对现在这个项目,最合理的定位是:
|
||||
|
||||
- `Playground` 是测试台
|
||||
- `BGP / Alerts` 是第一批业务 AI 入口
|
||||
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
|
||||
|
||||
现阶段不需要追求“已经具备完整态势感知能力”。
|
||||
|
||||
现阶段真正的成功标准是:
|
||||
|
||||
- 这套底座可用
|
||||
- 可回看
|
||||
- 可扩展
|
||||
- 不自欺欺人
|
||||
@@ -17,7 +17,7 @@ It is an aggregation/view-model layer:
|
||||
|
||||
## Why This Layer Exists
|
||||
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md):
|
||||
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
|
||||
|
||||
- incident density is naturally low
|
||||
- anomaly density is higher, but still not enough to keep the globe expressive all the time
|
||||
@@ -290,7 +290,7 @@ Each feature should include:
|
||||
|
||||
## Earth Rendering Plan
|
||||
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/bgp-earth-rendering-plan.md).
|
||||
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
|
||||
|
||||
### Layer Relationship
|
||||
|
||||
715
docs/plans/earth-celestial-background-plan.md
Normal file
715
docs/plans/earth-celestial-background-plan.md
Normal file
@@ -0,0 +1,715 @@
|
||||
# Earth 天球背景与日月位置实施方案
|
||||
|
||||
## 目标
|
||||
|
||||
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
|
||||
|
||||
1. 用真实天球背景替换当前随机星点
|
||||
2. 在当前时间下显示太阳与月亮的相对位置
|
||||
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
|
||||
|
||||
本方案优先追求:
|
||||
|
||||
- 与当前 Three.js Earth 架构兼容
|
||||
- 风险可控
|
||||
- 先落地一版真实感明显提升的 V1
|
||||
- 为后续更严格的天文参考系升级预留余地
|
||||
|
||||
## 当前现状
|
||||
|
||||
当前 Earth 的基础条件已经具备:
|
||||
|
||||
- 地球、云层、地形、网格都基于 Three.js,主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
|
||||
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
|
||||
|
||||
这意味着:
|
||||
|
||||
- 不需要重写 Earth
|
||||
- 可以在现有 scene/world 层新增一个 celestial layer
|
||||
- 第一阶段不必拆 Earth / satellite / cable 的参考系
|
||||
|
||||
## 总体策略
|
||||
|
||||
采用“两层现实”设计:
|
||||
|
||||
### 1. 世界层(world-space celestial layer)
|
||||
|
||||
用于放置:
|
||||
|
||||
- 天球背景
|
||||
- 太阳
|
||||
- 月亮
|
||||
- 太阳光方向
|
||||
|
||||
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
|
||||
|
||||
### 2. 地球层(earth-fixed layer)
|
||||
|
||||
继续保持当前结构:
|
||||
|
||||
- 海缆
|
||||
- 登陆点
|
||||
- 卫星点与轨迹
|
||||
- BGP 覆盖
|
||||
- 地球纹理、云层、地形
|
||||
|
||||
这些对象继续挂在 `earthObj` 下,不打断现有交互。
|
||||
|
||||
## 为什么先这样做
|
||||
|
||||
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
|
||||
如果现在直接做严格惯性参考系改造,会同时影响:
|
||||
|
||||
- `earthObj.rotation`
|
||||
- 卫星轨迹与锁定逻辑
|
||||
- 海缆与登陆点附着关系
|
||||
- resetView / autoRotate / hover / click 等交互链路
|
||||
|
||||
所以第一阶段只做:
|
||||
|
||||
- 真正的天空
|
||||
- 真正的日月方向
|
||||
- 不碰现有 Earth 附着对象的语义
|
||||
|
||||
## 推荐技术选型
|
||||
|
||||
### 天文计算库
|
||||
|
||||
推荐:
|
||||
|
||||
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
|
||||
原因:
|
||||
|
||||
- 有 JavaScript 版本
|
||||
- 支持 Sun / Moon 的矢量与坐标变换
|
||||
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
|
||||
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
|
||||
|
||||
不作为主选的库:
|
||||
|
||||
- [SunCalc](https://github.com/mourner/suncalc)
|
||||
|
||||
原因:
|
||||
|
||||
- 更偏本地观察者视角的太阳/月亮高度角
|
||||
- 用于“地面日出日落”很好
|
||||
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
|
||||
|
||||
### Three.js 表现层
|
||||
|
||||
推荐组合:
|
||||
|
||||
- 天球:内翻球壳 + 星图纹理
|
||||
- 太阳:`THREE.Sprite`
|
||||
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
|
||||
- 太阳光:`THREE.DirectionalLight`
|
||||
|
||||
参考:
|
||||
|
||||
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
|
||||
|
||||
## 天球背景资源与星体数据来源
|
||||
|
||||
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
|
||||
|
||||
### 1. 背景资源:全天星图贴图
|
||||
|
||||
用于 Phase 1 的“真实天空背景”。
|
||||
|
||||
推荐优先来源:
|
||||
|
||||
- NASA SVS 的 Tycho 全天星图
|
||||
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
|
||||
- NASA Deep Star Maps 2020
|
||||
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
|
||||
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
|
||||
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议要求:
|
||||
|
||||
- 使用官方来源或官方衍生可复用资源
|
||||
- 等距矩形投影(equirectangular)
|
||||
- 坐标定义尽量明确为赤道坐标展开
|
||||
- 分辨率建议至少 `4k`
|
||||
- 颜色不要过亮,避免压过 Earth HUD 前景
|
||||
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
|
||||
|
||||
建议本地资源目录:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
### 2. 位置数据:星表与天体计算
|
||||
|
||||
用于 Phase 2+ 的“位置正确的星体”。
|
||||
|
||||
推荐来源分两层:
|
||||
|
||||
- 太阳、月亮位置
|
||||
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
|
||||
- 恒星位置
|
||||
- 第一优先:Hipparcos / Tycho
|
||||
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
|
||||
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
|
||||
- 第二优先:Gaia
|
||||
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
|
||||
|
||||
建议策略:
|
||||
|
||||
- V1:背景球壳只用全天星图,不立即生成全量恒星点
|
||||
- V2:只挑选亮星(例如星等 `< 5.5`)生成恒星点层
|
||||
- V3:如果确实需要更丰富的星场,再逐步扩展到更深星等
|
||||
|
||||
这样做的原因:
|
||||
|
||||
- 背景球壳负责“天球真实感”
|
||||
- 亮星点负责“位置正确、可后续标注和高亮”
|
||||
- 不需要一开始就处理数十万甚至数百万颗星
|
||||
|
||||
### 3. 对外部成熟实现的参考结论
|
||||
|
||||
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
|
||||
|
||||
- 图形渲染使用 `TWGL.js`
|
||||
- 天文计算使用 `Skyfield` 与 `Astronomia`
|
||||
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
|
||||
|
||||
这给本项目的启发是:
|
||||
|
||||
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
|
||||
- “位置正确的动态天体”则应依赖单独的天文计算链路
|
||||
- 没有必要在第一版就直接渲染完整星表
|
||||
|
||||
因此本项目推荐继续坚持两层拆分:
|
||||
|
||||
- 背景层:官方深空图 / 全天星图
|
||||
- 计算层:太阳、月亮与后续亮星点
|
||||
|
||||
## 如何保证星体位置正确
|
||||
|
||||
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
|
||||
|
||||
### 1. 统一坐标基准
|
||||
|
||||
本方案推荐统一使用:
|
||||
|
||||
- `J2000` 赤道坐标系作为恒星位置基准
|
||||
|
||||
原因:
|
||||
|
||||
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
|
||||
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
|
||||
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
|
||||
|
||||
### 2. 背景贴图与点位必须使用同一展开逻辑
|
||||
|
||||
如果背景球壳使用赤道坐标全天图,那么:
|
||||
|
||||
- 亮星点也必须按赤道坐标贴到同一球面方向
|
||||
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
|
||||
|
||||
否则会出现:
|
||||
|
||||
- 背景银河带是对的
|
||||
- 但太阳/月亮或亮星点飘到不匹配的位置
|
||||
|
||||
### 3. RA / Dec 到 Three.js 坐标的落点方式
|
||||
|
||||
亮星点和日月方向最终都要转成单位球面向量。
|
||||
|
||||
概念步骤:
|
||||
|
||||
1. 读取赤经 `RA`
|
||||
2. 读取赤纬 `Dec`
|
||||
3. 转成弧度
|
||||
4. 映射到单位球面向量
|
||||
5. 再根据 Three.js 当前世界坐标定义做轴向映射
|
||||
|
||||
参考公式:
|
||||
|
||||
```text
|
||||
x = cos(dec) * cos(ra)
|
||||
y = sin(dec)
|
||||
z = cos(dec) * sin(ra)
|
||||
```
|
||||
|
||||
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
|
||||
|
||||
- 验证 `RA = 0h`
|
||||
- 验证 `RA = 6h`
|
||||
- 验证北天极
|
||||
- 验证银河带主方向
|
||||
|
||||
然后确定最终的:
|
||||
|
||||
- `x/y/z` 对应 Three.js 哪个轴
|
||||
- 是否需要 `z` 取反
|
||||
- 是否需要整体再做一个固定 `rotation`
|
||||
|
||||
建议把这层显式封装在:
|
||||
|
||||
```js
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
不要把轴映射散落在不同模块里。
|
||||
|
||||
### 4. 背景球壳与恒星点的关系
|
||||
|
||||
推荐最终组合:
|
||||
|
||||
- 背景层:全天星图球壳
|
||||
- 点位层:亮星点
|
||||
- 动态层:太阳 / 月亮
|
||||
|
||||
这样有三个好处:
|
||||
|
||||
- 背景层提供密集真实的天空纹理
|
||||
- 亮星点提供位置正确、可扩展的标注基础
|
||||
- 太阳/月亮提供与时间相关的真实动态对象
|
||||
|
||||
## 数据与资源建议清单
|
||||
|
||||
### 推荐首批引入资源
|
||||
|
||||
1. 全天星图
|
||||
- 来源:NASA Tycho all-sky map
|
||||
- 用途:背景球壳纹理
|
||||
|
||||
2. 月亮纹理
|
||||
- 用途:Phase 4 月相表现
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
|
||||
|
||||
3. 太阳 glow 贴图
|
||||
- 用途:太阳 sprite halo
|
||||
- 路径建议:
|
||||
- `frontend/public/earth/assets/celestial/sun_glow.png`
|
||||
|
||||
### 推荐首批数据文件
|
||||
|
||||
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/bright-stars.json`
|
||||
|
||||
建议字段:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32349,
|
||||
"name": "Sirius",
|
||||
"raDeg": 101.2875,
|
||||
"decDeg": -16.7161,
|
||||
"mag": -1.46,
|
||||
"colorIndex": 0.00
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
|
||||
|
||||
- 只保留亮星
|
||||
- 只保留渲染必需字段
|
||||
- JSON 或二进制轻量格式
|
||||
|
||||
## 资源与数据实施路线
|
||||
|
||||
### 路线 A:先做可用版本(推荐)
|
||||
|
||||
1. 引入 NASA Tycho 全天图
|
||||
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
|
||||
2. 实现背景球壳
|
||||
3. 用 Astronomy Engine 计算太阳/月亮方向
|
||||
4. 暂不做亮星点
|
||||
|
||||
优点:
|
||||
|
||||
- 最快见效
|
||||
- 风险最低
|
||||
- 就能明显提升天球真实感
|
||||
|
||||
### 路线 B:在 A 基础上增强
|
||||
|
||||
1. 离线生成 `bright-stars.json`
|
||||
2. 浏览器端渲染亮星点
|
||||
3. 后续可加:
|
||||
- 星座线
|
||||
- 亮星名称
|
||||
- 特定星体高亮
|
||||
|
||||
优点:
|
||||
|
||||
- 背景真实感和“位置正确的可交互星体”同时兼顾
|
||||
|
||||
## 代码模块建议细化
|
||||
|
||||
### 新增模块
|
||||
|
||||
- `frontend/public/earth/js/celestial.js`
|
||||
- 管理天球背景
|
||||
- 管理太阳/月亮
|
||||
- 管理亮星层(后续)
|
||||
|
||||
- `frontend/public/earth/js/celestial-data.js`
|
||||
- 资源路径
|
||||
- 星图方向配置
|
||||
- 亮星数据加载(后续)
|
||||
|
||||
### 建议函数设计
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date)
|
||||
export function setCelestialVisibility(visible)
|
||||
export function disposeCelestialLayer()
|
||||
|
||||
function loadStarMapTexture()
|
||||
function createSkySphere(texture)
|
||||
function createSunSprite()
|
||||
function createMoonSprite()
|
||||
function getSunEquatorialPosition(date)
|
||||
function getMoonEquatorialPosition(date)
|
||||
function equatorialToWorldVector(raRad, decRad)
|
||||
```
|
||||
|
||||
### 推荐后续预处理脚本
|
||||
|
||||
如要引入亮星层,建议单独做离线脚本:
|
||||
|
||||
- `scripts/build_bright_stars.py`
|
||||
|
||||
职责:
|
||||
|
||||
- 从 Hipparcos / Tycho 源数据读取
|
||||
- 过滤亮星
|
||||
- 生成 `bright-stars.json`
|
||||
|
||||
这样浏览器端只消费轻量结果,不承担大表解析成本。
|
||||
|
||||
## 分阶段实施
|
||||
|
||||
## Phase 1:真实天球背景
|
||||
|
||||
### 目标
|
||||
|
||||
用真实全天星图替换当前随机星点背景。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 新增一张全天星图纹理
|
||||
|
||||
建议路径:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
|
||||
|
||||
纹理要求:
|
||||
|
||||
- 等距矩形投影
|
||||
- 赤经/赤纬坐标展开
|
||||
- 无地平线、无地景遮挡
|
||||
- 尽量深色、弱干扰,适合大屏 HUD 叠加
|
||||
|
||||
2. 新增天球球壳
|
||||
|
||||
新增模块:
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
建议接口:
|
||||
|
||||
```js
|
||||
export function initCelestialLayer(scene)
|
||||
export function updateCelestialLayer(date, camera, earth)
|
||||
export function disposeCelestialLayer()
|
||||
```
|
||||
|
||||
3. 实现一个大半径内翻球体
|
||||
|
||||
建议参数:
|
||||
|
||||
- 半径:`600 ~ 900`
|
||||
- 材质:`MeshBasicMaterial`
|
||||
- `side: THREE.BackSide`
|
||||
- 不受场景光照影响
|
||||
- 始终围绕场景中心
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 初始加载后背景不再是随机星点
|
||||
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
|
||||
- 不明显干扰海缆/卫星/BGP 的前景识别
|
||||
|
||||
## Phase 2:太阳与月亮真实位置
|
||||
|
||||
### 目标
|
||||
|
||||
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 在 `celestial.js` 内封装天体位置计算
|
||||
|
||||
建议函数:
|
||||
|
||||
```js
|
||||
function getSunDirection(date)
|
||||
function getMoonDirection(date)
|
||||
```
|
||||
|
||||
输出统一为 world-space `THREE.Vector3`
|
||||
|
||||
2. 太阳显示
|
||||
|
||||
- 一个暖色发光 sprite
|
||||
- 比月亮更大、更亮
|
||||
- 可选添加柔和 halo
|
||||
|
||||
3. 月亮显示
|
||||
|
||||
- 一个较小 sprite 或 sphere
|
||||
- 灰白偏冷色
|
||||
- 后续 Phase 3 再做月相
|
||||
|
||||
4. 更新频率
|
||||
|
||||
不要每帧重新做完整天文计算,建议:
|
||||
|
||||
- 每 30 秒或 60 秒重算一次真实位置
|
||||
- 渲染帧内做平滑过渡
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 页面可见太阳与月亮两个对象
|
||||
- 时间变化时位置会更新
|
||||
- 日月不会跟随地球局部旋转而错误附着
|
||||
|
||||
## Phase 3:太阳驱动地球受光
|
||||
|
||||
### 目标
|
||||
|
||||
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
|
||||
|
||||
### 做法
|
||||
|
||||
1. 替换或接管当前主定向光
|
||||
|
||||
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`。
|
||||
|
||||
建议改为:
|
||||
|
||||
- 保留环境补光
|
||||
- 主太阳光方向由 `sunDirection` 决定
|
||||
|
||||
2. 太阳光参数建议
|
||||
|
||||
- `DirectionalLight` 颜色偏暖白
|
||||
- 强度略高于当前主光
|
||||
- 保留一个弱背光作为氛围补偿,避免背面过死黑
|
||||
|
||||
3. 先不做物理级大气散射
|
||||
|
||||
第一版只要求:
|
||||
|
||||
- 亮面与暗面方向真实
|
||||
- 云层和大气仍保持当前风格
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 地球明暗面会随太阳方向改变
|
||||
- 太阳 sprite 和地球亮面方向一致
|
||||
- 不破坏现有海缆、卫星、BGP 的可见性
|
||||
|
||||
## Phase 4:月相与天文细节增强
|
||||
|
||||
### 目标
|
||||
|
||||
在日月真实位置基础上增加更强的“天文可信度”。
|
||||
|
||||
### 可选项
|
||||
|
||||
1. 月相
|
||||
|
||||
- 根据日月夹角计算 illuminated fraction
|
||||
- 用月相纹理或 shader 表达盈亏
|
||||
|
||||
2. 赤道/黄道辅助线
|
||||
|
||||
- 可作为开发调试层,不默认显示
|
||||
|
||||
3. 太阳 terminator 增强
|
||||
|
||||
- 给地球夜面加入更自然的 night tint
|
||||
- 未来可叠加城市夜光纹理
|
||||
|
||||
4. 天文时间入口
|
||||
|
||||
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
|
||||
|
||||
### 验收标准
|
||||
|
||||
- 月亮不再只是一个静态圆点
|
||||
- 后续扩展行星或观测模式时无需推倒重来
|
||||
|
||||
## Phase 5:严格参考系升级(可选,不作为 V1 必做)
|
||||
|
||||
### 目标
|
||||
|
||||
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
|
||||
|
||||
### 需要处理的问题
|
||||
|
||||
- 地球自转角与 UTC 的一致性
|
||||
- 赤道坐标系、地固坐标系、相机交互层分离
|
||||
- 卫星轨道显示与 Earth 旋转同步关系
|
||||
- resetView 和 autoRotate 的语义重定
|
||||
|
||||
### 风险
|
||||
|
||||
这一步会影响:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
|
||||
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
|
||||
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
因此不建议与 V1 同时推进。
|
||||
|
||||
## 代码改造清单
|
||||
|
||||
## 1. 新增文件
|
||||
|
||||
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理天球背景、太阳、月亮
|
||||
- 对外暴露 init/update/dispose
|
||||
|
||||
## 2. 修改 `constants.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
|
||||
新增:
|
||||
|
||||
```js
|
||||
export const CELESTIAL_CONFIG = {
|
||||
sphereRadius: 800,
|
||||
updateIntervalMs: 60000,
|
||||
sunSpriteScale: 28,
|
||||
moonSpriteScale: 16,
|
||||
sunLightIntensity: 1.25,
|
||||
ambientIntensity: 0.28,
|
||||
backLightIntensity: 0.18,
|
||||
};
|
||||
```
|
||||
|
||||
## 3. 修改 `main.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
1. `init()` 中:
|
||||
- 初始化 celestial layer
|
||||
2. `addLights()` 中:
|
||||
- 把固定太阳光改成可更新的 celestial sun light
|
||||
3. `animate()` 中:
|
||||
- 每帧调 `updateCelestialLayer()`
|
||||
4. `destroy()` 中:
|
||||
- 清理 celestial 资源
|
||||
|
||||
## 4. 修改 `earth.js`
|
||||
|
||||
文件:
|
||||
|
||||
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
主要改动:
|
||||
|
||||
- `createStars()` 逐步退役
|
||||
- 第一阶段可先保留作为 fallback
|
||||
- 当真实星图加载成功后,不再显示随机星点
|
||||
|
||||
## 5. 新增资源
|
||||
|
||||
目录建议:
|
||||
|
||||
- `frontend/public/earth/assets/celestial/`
|
||||
|
||||
建议至少包含:
|
||||
|
||||
- `starmap_equatorial_4k.jpg`
|
||||
- `sun_glow.png`
|
||||
- `moon_albedo_2k.jpg`
|
||||
|
||||
## 数据流设计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
|
||||
B --> C["创建天球球壳"]
|
||||
B --> D["创建太阳 sprite + 主定向光"]
|
||||
B --> E["创建月亮 sprite"]
|
||||
|
||||
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
|
||||
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
|
||||
H --> I["更新 sun sprite / moon sprite 位置"]
|
||||
H --> J["更新太阳 DirectionalLight 方向"]
|
||||
J --> K["地球昼夜方向变化"]
|
||||
```
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
### 1. 星图投影方向容易反
|
||||
|
||||
这会表现为:
|
||||
|
||||
- 星图左右镜像
|
||||
- 赤经方向颠倒
|
||||
- 日月位置和背景对不上
|
||||
|
||||
建议:
|
||||
|
||||
- 先做一个开发调试模式
|
||||
- 显示赤经/赤纬参考点,快速校正纹理朝向
|
||||
|
||||
### 2. 不要让天球跟随 Earth 旋转
|
||||
|
||||
天球背景和日月必须属于 scene/world,而不是 `earthObj`。
|
||||
|
||||
### 3. 不要每帧做重型天文计算
|
||||
|
||||
真实位置更新应节流,否则会浪费 CPU。
|
||||
|
||||
### 4. 月亮先求“方向正确”,再求“月相精致”
|
||||
|
||||
月相属于第二步优化,不应阻塞 V1 上线。
|
||||
|
||||
## 推荐实施顺序
|
||||
|
||||
1. 新建 `celestial.js`
|
||||
2. 用星图球壳替换随机星点
|
||||
3. 接入 Astronomy Engine
|
||||
4. 加太阳/月亮 sprite
|
||||
5. 用太阳方向驱动主光
|
||||
6. 再决定要不要做月相和更严格参考系
|
||||
|
||||
## 最终建议
|
||||
|
||||
对于当前 Planet Earth,最稳妥的方案是:
|
||||
|
||||
- 先做真实天球背景
|
||||
- 再做真实太阳/月亮方向
|
||||
- 再让太阳驱动地球受光
|
||||
- 暂时不做 Earth 参考系重构
|
||||
|
||||
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。
|
||||
372
docs/plans/earth-compute-center-bgp-style-plan.md
Normal file
372
docs/plans/earth-compute-center-bgp-style-plan.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# Earth Compute Center BGP-Style Plan
|
||||
|
||||
## Goal
|
||||
|
||||
这份文档定义如何按照 BGP 模块的产品方式,把“算力中心”提升为 Earth 上的一级能力。
|
||||
|
||||
这里的“按 BGP 方式”指的是:
|
||||
|
||||
- 有独立的数据语义和接口入口
|
||||
- 有独立的 Earth 图层与图例
|
||||
- 有独立的 hover / click / 选中态 / 详情卡
|
||||
- 有独立的统计口径与后续专题页扩展空间
|
||||
|
||||
这里的“按 BGP 方式”不指:
|
||||
|
||||
- 机械复制 BGP 的 anomaly / incident / collector 三层事件模型
|
||||
- 为静态算力设施强行引入不必要的复杂告警语义
|
||||
|
||||
算力中心本质上更接近“长期基础设施分布层”,不是“高频动态异常层”。
|
||||
因此应该复用 BGP 的模块化方法,而不是照搬 BGP 的事件结构。
|
||||
|
||||
## Why
|
||||
|
||||
当前仓库里已经有算力相关基础:
|
||||
|
||||
- 后端已有 `top500` 和 `epoch_ai_gpu` 数据采集
|
||||
- 可视化接口已有 `/api/v1/visualization/geo/supercomputers` 和 `/api/v1/visualization/geo/gpu-clusters`
|
||||
- Earth 信息卡已对 `supercomputer` 和 `gpu_cluster` 做了基础类型兼容
|
||||
|
||||
但当前能力还停留在“数据可取到”的阶段,没有形成像 BGP 那样完整的可视化模块:
|
||||
|
||||
- Earth 缺少独立的算力图层加载模块
|
||||
- 缺少算力 marker 体系和视觉层级
|
||||
- 缺少算力图例、统计、开关和搜索接入
|
||||
- 缺少与海缆、BGP、卫星的关系表达
|
||||
- 缺少算力专题页和后续告警/研判扩展入口
|
||||
|
||||
所以当前真正的缺口不是“有没有数据”,而是“有没有产品级模块”。
|
||||
|
||||
## Core Principle
|
||||
|
||||
算力中心应当采用和 BGP 一致的模块化分层:
|
||||
|
||||
1. 数据层:稳定的数据契约和 GeoJSON 输出
|
||||
2. 渲染层:独立的 Earth 图层、marker 和视觉状态管理
|
||||
3. 交互层:hover、click、锁定态、详情卡、图例和统计
|
||||
4. 扩展层:后续专题页、关系分析、告警和 AI 研判
|
||||
|
||||
但语义上必须保持算力中心自身的特点:
|
||||
|
||||
- `site / center` 是主对象,不是事件
|
||||
- `capacity / rank / vendor / operator / status` 是主信息,不是异常严重度
|
||||
- `distribution / concentration / dependency` 是后续分析方向,不是第一阶段必须项
|
||||
|
||||
## Recommended Scope
|
||||
|
||||
第一版“算力中心”建议统一承载两类对象:
|
||||
|
||||
- `supercomputer`
|
||||
- `gpu_cluster`
|
||||
|
||||
并在 Earth 上收口为一个主题层:`compute_centers`
|
||||
|
||||
这样做有几个好处:
|
||||
|
||||
- 用户看到的是统一的“算力基础设施”语义,而不是零散数据源
|
||||
- 后端仍可保留 `top500` 和 `epoch_ai_gpu` 的来源差异
|
||||
- 前端可以在一个图层里再细分两种 marker 语言
|
||||
|
||||
## Current Gap
|
||||
|
||||
和 BGP 对比,当前差距主要在下面几层。
|
||||
|
||||
### 1. Data Contract Gap
|
||||
|
||||
现在的算力 GeoJSON 还是通用 `collected_data` 输出思路,字段较轻:
|
||||
|
||||
- `gpu_cluster` 只有基础名称和地点
|
||||
- `supercomputer` 只暴露一部分性能字段
|
||||
- 缺少统一的 `site_type / operator / capacity_band / source / updated_at / confidence`
|
||||
- 缺少统一的算力层聚合出口
|
||||
|
||||
### 2. Earth Rendering Gap
|
||||
|
||||
当前 Earth 里没有类似 `bgp.js` 的算力模块:
|
||||
|
||||
- `constants.js` 没有算力 API 路径和视觉配置
|
||||
- `main.js` 没有算力加载、拾取、状态同步和 HUD 更新
|
||||
- `controls.js` 没有算力图层开关和启动加载优先级
|
||||
- `layer-startup-tasks.js` 没有算力启动任务
|
||||
- `legend.js` / `ui.js` 没有算力统计与图例模式
|
||||
|
||||
### 3. Interaction Gap
|
||||
|
||||
虽然 `info-card.js` 支持基础字段,但还没有形成 BGP 那种完整交互链路:
|
||||
|
||||
- 没有 hover / selected / dimmed 的视觉状态
|
||||
- 没有算力对象专属 tooltip 与摘要文案
|
||||
- 没有锁定后与其他基础设施的联动高亮
|
||||
- 没有搜索、统计卡和详情组织方式
|
||||
|
||||
### 4. Product Expansion Gap
|
||||
|
||||
当前还没有“算力中心”专题页与分析语义:
|
||||
|
||||
- 没有全球分布/国家聚合/厂商聚合视图
|
||||
- 没有算力与海缆/BGP/区域的关系表达
|
||||
- 没有 AI brief / assessment 的后续落点
|
||||
|
||||
## Architecture Direction
|
||||
|
||||
推荐把算力中心做成“BGP 同级能力”,但采用更适合静态基础设施的结构。
|
||||
|
||||
### Backend
|
||||
|
||||
建议新增统一聚合接口,例如:
|
||||
|
||||
- `/api/v1/visualization/geo/compute-centers`
|
||||
|
||||
它的职责是把:
|
||||
|
||||
- `top500`
|
||||
- `epoch_ai_gpu`
|
||||
|
||||
统一转换成一个主题层输出,同时保留对象细分类型:
|
||||
|
||||
- `site_type: supercomputer | gpu_cluster`
|
||||
|
||||
建议统一字段至少包括:
|
||||
|
||||
- `id`
|
||||
- `name`
|
||||
- `site_type`
|
||||
- `country`
|
||||
- `city`
|
||||
- `latitude`
|
||||
- `longitude`
|
||||
- `operator`
|
||||
- `vendor`
|
||||
- `capacity_value`
|
||||
- `capacity_unit`
|
||||
- `capacity_band`
|
||||
- `rank`
|
||||
- `source`
|
||||
- `updated_at`
|
||||
- `location_precision`
|
||||
- `geography_mode`
|
||||
- `is_estimated`
|
||||
- `estimated_reason`
|
||||
- `metadata`
|
||||
|
||||
这里建议优先做“统一聚合出口”,而不是一开始就新增独立数据库表。
|
||||
|
||||
原因:
|
||||
|
||||
- 当前源数据更新频率低,先复用 `collected_data` 成本更低
|
||||
- 可以先把 Earth 产品体验做完整
|
||||
- 如果后续要做历史趋势、关系推断、告警,再评估是否拆成独立模型
|
||||
|
||||
### Frontend Earth
|
||||
|
||||
建议新增独立模块,例如:
|
||||
|
||||
- `frontend/public/earth/js/compute-centers.js`
|
||||
|
||||
职责参照 `bgp.js`:
|
||||
|
||||
- 拉取算力中心 GeoJSON
|
||||
- 创建 marker
|
||||
- 管理 hover / selected / dimmed 状态
|
||||
- 输出图例项
|
||||
- 输出统计摘要
|
||||
- 提供 overlay 和详情格式化辅助函数
|
||||
|
||||
推荐视觉分层:
|
||||
|
||||
1. `supercomputer` 用更稳定、更规整的设施型符号
|
||||
2. `gpu_cluster` 用更活跃、更现代的密度型符号
|
||||
3. 选中态通过 halo / ring / related infrastructure highlight 表达
|
||||
|
||||
视觉上应避免把算力中心做成“BGP 事件点”那种高频脉冲风格。
|
||||
它应该更像长期存在的高价值设施。
|
||||
|
||||
## Phases
|
||||
|
||||
## Phase 1: Unified Earth Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 先把算力中心做成 Earth 上可用、可点、可解释的一级图层
|
||||
|
||||
工作项:
|
||||
|
||||
- 新增统一算力 GeoJSON 接口
|
||||
- 新增 `compute-centers.js`
|
||||
- 在 `constants.js` 增加 API 路径和视觉配置
|
||||
- 在 `controls.js` 增加算力图层开关与启动元数据
|
||||
- 在 `layer-startup-tasks.js` 增加算力启动加载任务
|
||||
- 在 `main.js` 接入算力拾取、hover、click、锁定态和 HUD 统计
|
||||
- 在 `ui.js` / `legend.js` / `index.html` 增加算力统计与图例入口
|
||||
- 在 `info-card.js` 提升算力详情字段组织
|
||||
- 对无法精确定位、但可按国家或弱线索推测的大概位置,仍然生成地图点位
|
||||
- 这类对象必须带显式“估算位置”状态,例如图标问号角标与详情说明
|
||||
|
||||
完成标准:
|
||||
|
||||
- Earth 上能独立显示/隐藏算力中心
|
||||
- 两类对象有可区分的视觉表达
|
||||
- hover / click / 详情卡 / 图例 / 统计全部打通
|
||||
- 精确位置与估算位置在图标或文案上可区分,不会误导为同一精度
|
||||
- 不干扰现有海缆、卫星、BGP 的交互链路
|
||||
|
||||
## Phase 2: Relationship Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 让算力中心不只是“点”,而是和其他基础设施产生上下文关系
|
||||
|
||||
工作项:
|
||||
|
||||
- 建立算力中心与国家/区域聚合摘要
|
||||
- 增加与附近海缆登陆点的关系提示
|
||||
- 增加与 BGP 事件/观测范围的空间邻近提示
|
||||
- 增加与卫星覆盖或区域连通性的实验性提示
|
||||
|
||||
完成标准:
|
||||
|
||||
- 点击算力中心时,用户能看到“它和哪些基础设施相关”
|
||||
- 信息表达以辅助判断为主,不做夸张推断
|
||||
|
||||
## Phase 3: Compute Center Observatory
|
||||
|
||||
目标:
|
||||
|
||||
- 把算力中心从 Earth 图层扩展成独立专题观测能力
|
||||
|
||||
工作项:
|
||||
|
||||
- 新增算力中心专题页
|
||||
- 提供国家/厂商/类型/容量分布统计
|
||||
- 支持列表、筛选、详情和历史快照
|
||||
- 预留 AI brief / assessment 入口
|
||||
|
||||
完成标准:
|
||||
|
||||
- 算力中心不再只是 Earth 上的视觉点位
|
||||
- 能作为独立业务上下文进入日常观察与研判
|
||||
|
||||
## Phase 4: Alerts And Assessment
|
||||
|
||||
目标:
|
||||
|
||||
- 在不滥造“假动态告警”的前提下,引入真正有价值的变化感知
|
||||
|
||||
候选方向:
|
||||
|
||||
- 新增大规模算力中心
|
||||
- 既有中心容量显著变化
|
||||
- 国家/区域集中度显著变化
|
||||
- 高价值中心与关键网络基础设施关系变化
|
||||
|
||||
完成标准:
|
||||
|
||||
- 告警来自可解释的结构变化
|
||||
- 不把静态数据硬做成噪声式实时事件流
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
建议按下面顺序推进:
|
||||
|
||||
1. 先统一 GeoJSON 契约
|
||||
2. 再做 Earth 独立模块和图层开关
|
||||
3. 再补详情卡、图例和统计
|
||||
4. 最后才做关系层和专题页
|
||||
|
||||
这样可以避免一开始把范围摊得过大。
|
||||
|
||||
## Unknown Location Strategy
|
||||
|
||||
由于部分算力数据源不会直接提供经纬度,未知位置补全不能只依赖“继续找 API 字段”。
|
||||
更稳妥的方式是做成一条分层富化链路,而不是单一猜测规则。
|
||||
|
||||
推荐按下面优先级推进:
|
||||
|
||||
1. 直接源信息
|
||||
|
||||
- 源记录显式给出 `latitude / longitude`
|
||||
- 源记录给出 `city / region / facility / campus / operator`
|
||||
- 源页面详情、内嵌 JSON、结构化元数据、新闻稿链接里能抽出地点线索
|
||||
|
||||
2. 名称与机构归一化
|
||||
|
||||
- 建立 `canonical_name / aliases / operator / facility` 归一化表
|
||||
- 把 `cluster name`、`operator`、`campus name` 归一到同一个实体
|
||||
- 优先解决同一对象多写法导致的命中失败,而不是先扩大猜测范围
|
||||
|
||||
3. 本地位置注册表
|
||||
|
||||
- 用仓库内可维护的 registry 保存高价值对象的位置知识
|
||||
- 每条记录至少包含:`canonical_name`、`aliases`、`operator`、`country`、`region`、`city`、`lat`、`lon`、`confidence`、`source_note`
|
||||
- 转换层优先读取 registry,避免地点知识长期散落在转换代码里
|
||||
|
||||
4. 分层回退定位
|
||||
|
||||
- `precise`
|
||||
- `estimated_site`
|
||||
- `estimated_city`
|
||||
- `estimated_region`
|
||||
- `estimated_national_hub`
|
||||
- `estimated_country`
|
||||
|
||||
这里建议把“国家内主要算力城市”作为国家质心之前的一层。
|
||||
例如没有美国精确位置时,优先考虑已知的主要算力/数据中心城市候选,而不是直接落在几何质心。
|
||||
|
||||
5. 候选证据富化
|
||||
|
||||
- 如果源 API 无地点信息,可以允许采集链路读取公开辅助证据
|
||||
- 例如机构官网、数据中心介绍页、新闻稿、百科型页面、公开 PDF
|
||||
- 但只提取“地点线索”,不把外部页面上的经纬度当真值直接写回
|
||||
|
||||
6. 人工校验闭环
|
||||
|
||||
- 对高价值且仍然未知的对象输出待核验清单
|
||||
- 把人工确认结果回写到位置注册表
|
||||
- 后续采集继续优先复用这层人工确认结果
|
||||
|
||||
### Additional Solution Paths
|
||||
|
||||
除了静态映射表,还可以考虑下面这些办法:
|
||||
|
||||
- 基于国家和运营方建立“主要园区候选集”,用稳定散列把同国未知节点分散到若干可信城市,而不是全部压到一个点
|
||||
- 基于数据中心/云厂商公开 region 列表建立 `operator -> city set` 候选映射,用于云 GPU 集群类对象
|
||||
- 把“估算依据”结构化,例如 `matched_alias`、`matched_operator`、`matched_city_text`、`fallback_country_hub`
|
||||
- 给位置补全增加 `last_verified_at`,便于后续按时间重新校验老旧映射
|
||||
- 单独维护“不可可靠定位”状态;这类对象仍可在国家级聚合统计中出现,但可以允许用户在地图上过滤掉
|
||||
- 后续如果你们愿意投入更多,可把这条链路做成小型 enrichment pipeline,而不是仅在 API 转换时临时判断
|
||||
|
||||
## Non-Goals
|
||||
|
||||
第一阶段不建议做这些内容:
|
||||
|
||||
- 不复制 BGP 巡航模式到算力中心
|
||||
- 不先做复杂实时 websocket 推送
|
||||
- 不先引入独立 `compute_center_incident` 一类模型
|
||||
- 不先做全量 AI 分析面板
|
||||
|
||||
原因是算力中心的第一需求是“被看清楚”,不是“被实时播报”。
|
||||
但“被看清楚”不等于“只显示精确坐标对象”。
|
||||
对于没有精确经纬度、但能推测到国家或区域级位置的算力中心,应优先以上图并标注估算状态的方式处理,而不是直接在地图上消失。
|
||||
|
||||
## Acceptance Checklist
|
||||
|
||||
- 后端存在统一的算力中心 GeoJSON 出口
|
||||
- Earth 有独立算力图层模块,而不是散落在 `main.js`
|
||||
- 页面上有清晰的算力开关、图例和统计
|
||||
- `supercomputer` 和 `gpu_cluster` 在视觉和详情上都可区分
|
||||
- 估算位置对象在地图和详情中都有明确状态提示
|
||||
- 现有 BGP / 海缆 / 卫星功能无回归
|
||||
- 代码结构上为后续专题页和关系分析留出了明确扩展点
|
||||
|
||||
## Summary
|
||||
|
||||
这项工作的本质不是“再多画几个点”。
|
||||
|
||||
它应该把算力中心从已有数据源,升级成与 BGP 同级的 Earth 观测主题:
|
||||
|
||||
- 有独立语义
|
||||
- 有独立图层
|
||||
- 有独立交互
|
||||
- 有后续分析扩展能力
|
||||
|
||||
推荐先完成 Phase 1,把算力中心做成真正可用的 Earth 一级模块,再继续推进关系层和专题页。
|
||||
400
docs/plans/earth-mobile-drawer-ui-plan.md
Normal file
400
docs/plans/earth-mobile-drawer-ui-plan.md
Normal file
@@ -0,0 +1,400 @@
|
||||
# Earth Mobile Drawer UI Plan
|
||||
|
||||
## 背景
|
||||
|
||||
当前 Earth 移动端已经补上了基础触控能力,例如:
|
||||
|
||||
- 单指拖拽旋转地球
|
||||
- 双指缩放
|
||||
- 点击阈值和基础事件隔离
|
||||
|
||||
但移动端 UI 仍然存在一个根本问题:
|
||||
|
||||
它还在沿用桌面 HUD 的内容切分方式,只是把原来的 panel、modal、toolbar 改位置、改层级、改容器。这样虽然能快速复用旧代码,但手机端体验仍然是生硬的,因为:
|
||||
|
||||
- 信息密度和结构是按桌面设计的
|
||||
- 面板标题、关闭、折叠、开关项是桌面心智,不是手机心智
|
||||
- 很多内容只是“被塞进抽屉”,而不是为抽屉重新设计
|
||||
- 设置里仍然带有“显示/隐藏某些 panel”的思路,但移动端本来就不应该存在那些独立 panel
|
||||
|
||||
因此本计划进一步收紧:
|
||||
|
||||
移动端不只是“底部抽屉化”,而是**重新设计一套 fit 抽屉体系的 mobile-first UI**。
|
||||
|
||||
## 新目标
|
||||
|
||||
1. 手机端不再使用现有 `toolbar` 作为主入口。
|
||||
2. 手机端不再使用现有独立 `panel / modal / sheet` 作为直接 UI 单元。
|
||||
3. 手机端统一采用“底部抽屉 + 顶部标题 + tab 切换 + 卡片内容”的单前景模式。
|
||||
4. 抽屉内部每个 tab 页面都按移动端重新设计内容结构,而不是直接复用旧 panel 结构。
|
||||
5. 设置页移除“显示/隐藏 panel”的桌面遗留配置。
|
||||
6. 媒体页拆成两个移动端页面:`新闻` 与 `TV`,都归入抽屉体系。
|
||||
7. 桌面端保持现有 HUD 体系,不回退。
|
||||
|
||||
## 核心原则
|
||||
|
||||
### 1. 只复用数据和状态,不复用桌面 UI 结构
|
||||
|
||||
可复用:
|
||||
|
||||
- 图层注册表
|
||||
- 搜索结果数据
|
||||
- BGP / 海缆 / 卫星详情数据
|
||||
- 媒体数据
|
||||
- 旋转、缩放、选择、高亮等运行时状态
|
||||
|
||||
不直接复用:
|
||||
|
||||
- 桌面 panel DOM 结构
|
||||
- 桌面 panel header / close / collapse 交互
|
||||
- 桌面 settings 项里的“显示某 panel”逻辑
|
||||
- 桌面媒体面板布局
|
||||
|
||||
### 2. 抽屉是唯一主前景层
|
||||
|
||||
移动端同一时刻只有一个主前景层:底部抽屉。
|
||||
|
||||
抽屉内部切换内容页,而不是多个悬浮层互相覆盖。
|
||||
|
||||
### 3. 每个 tab 都是移动端页面,而不是 panel 容器
|
||||
|
||||
抽屉中的每一项都应视为一个移动端子页面:
|
||||
|
||||
- 有自己的标题
|
||||
- 有自己的内容层次
|
||||
- 有自己的滚动区域
|
||||
- 有自己的主操作
|
||||
|
||||
而不是简单挂一个旧面板进去。
|
||||
|
||||
### 4. 移动端状态提示不占据屏幕正中
|
||||
|
||||
桌面端当前很多通知、状态提示、胶囊消息更适合在屏幕上方居中出现,但移动端不应继续沿用这套布局。
|
||||
|
||||
移动端统一改为:
|
||||
|
||||
- 通知栏放在右上角安全区
|
||||
- 胶囊提示放在右上角堆叠
|
||||
- 不遮挡地球中心视野
|
||||
- 不与底部抽屉主交互区冲突
|
||||
|
||||
## 交互模型
|
||||
|
||||
### 默认态
|
||||
|
||||
移动端默认只显示:
|
||||
|
||||
- 地球主画布
|
||||
- 底部半露出的抽屉头部
|
||||
|
||||
不再单独显示上箭头按钮。
|
||||
|
||||
### 展开态
|
||||
|
||||
用户从底边直接上拉抽屉,或点击抽屉头部展开。
|
||||
|
||||
展开后显示:
|
||||
|
||||
- 当前页面标题
|
||||
- tab 导航
|
||||
- 当前页面内容
|
||||
|
||||
### 收起态
|
||||
|
||||
用户下拉抽屉头部收起,或点击背景收起。
|
||||
|
||||
## 信息架构
|
||||
|
||||
移动端抽屉内的一级页面重定为:
|
||||
|
||||
1. 图层
|
||||
2. 搜索
|
||||
3. 态势
|
||||
4. 新闻
|
||||
5. TV
|
||||
6. 设置
|
||||
7. 详情(按需出现,不固定常驻 tab)
|
||||
|
||||
其中 `新闻` 和 `TV` 不再共享同一个移动端媒体面板。
|
||||
|
||||
## 页面重设计要求
|
||||
|
||||
### 图层页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为移动端最核心的控制页
|
||||
- 强调快速开关,不强调桌面 panel 感
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部摘要:当前已启用图层数量
|
||||
- 图层列表卡片
|
||||
- 每个图层项只保留:
|
||||
- 图标
|
||||
- 中文名
|
||||
- 英文副标题
|
||||
- 开关
|
||||
- 去掉桌面式 header / collapse / close 结构
|
||||
|
||||
### 搜索页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为抽屉中的完整搜索页
|
||||
- 避免看起来像桌面 modal 被塞进抽屉
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部搜索输入框
|
||||
- 搜索提示文案
|
||||
- 结果列表
|
||||
- 结果项更适合手指点击
|
||||
- 结果点击后:
|
||||
- 聚焦地球对象
|
||||
- 自动切换到详情页
|
||||
|
||||
### 态势页
|
||||
|
||||
目标:
|
||||
|
||||
- 合并原来的 `stats + legend` 思路
|
||||
- 成为移动端全局态势页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部核心统计卡
|
||||
- 海缆数量
|
||||
- 登陆点数量
|
||||
- 卫星数量
|
||||
- BGP 事件数量
|
||||
- 当前关注层图例
|
||||
- BGP 状态摘要
|
||||
- 不再出现独立 legend 面板和独立 stats 面板
|
||||
|
||||
### 新闻页
|
||||
|
||||
目标:
|
||||
|
||||
- 从原媒体面板中拆出单独的移动端新闻页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 当前区域焦点
|
||||
- 新闻源数量
|
||||
- 新闻卡片列表
|
||||
- 卡片内显示标题、来源、时间、区域
|
||||
- 外链操作更清晰
|
||||
|
||||
### TV 页
|
||||
|
||||
目标:
|
||||
|
||||
- 从原媒体面板中拆出单独的移动端 TV 页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 顶部频道选择
|
||||
- 直播状态
|
||||
- 当前频道说明
|
||||
- 视频播放器区域
|
||||
- 刷新和外链按钮
|
||||
|
||||
不再保留桌面式“新闻/TV tab 共处一个 panel”的结构。
|
||||
|
||||
### 设置页
|
||||
|
||||
目标:
|
||||
|
||||
- 只保留对移动端仍有意义的系统配置
|
||||
|
||||
必须移除:
|
||||
|
||||
- 图层控制 panel 显示/隐藏
|
||||
- 图例 panel 显示/隐藏
|
||||
- 全球态势 panel 显示/隐藏
|
||||
- 媒体 panel 显示/隐藏
|
||||
|
||||
保留项建议:
|
||||
|
||||
- 旋转模式
|
||||
- 日夜模式
|
||||
- 地球默认大小
|
||||
- 地形透明度
|
||||
- 系统入口
|
||||
|
||||
原因:
|
||||
|
||||
移动端已经没有这些独立 panel 了,所以继续保留这些开关会制造错误心智。
|
||||
|
||||
### 详情页
|
||||
|
||||
目标:
|
||||
|
||||
- 成为海缆 / BGP / 卫星对象的统一移动端详情页
|
||||
|
||||
内容建议:
|
||||
|
||||
- 标题区
|
||||
- 类型标签
|
||||
- 关键属性列表
|
||||
- 相关对象摘要
|
||||
- 相关图层或态势提示
|
||||
|
||||
行为建议:
|
||||
|
||||
- 点击对象后自动切入详情页
|
||||
- 搜索结果点击后也切入详情页
|
||||
|
||||
## 阶段重定义
|
||||
|
||||
### 阶段 2:抽屉壳层
|
||||
|
||||
目标:
|
||||
|
||||
1. 实现底部抽屉基本壳层。
|
||||
2. 支持上拉展开、下拉收起、背景点击关闭。
|
||||
3. `mobile` 模式下隐藏旧 toolbar。
|
||||
4. `mobile` 模式下不再直接显示旧 panel。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 手机端只有地球主视图和抽屉。
|
||||
2. 抽屉开合稳定。
|
||||
|
||||
### 阶段 3:基础页面重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 重新设计并实现图层页。
|
||||
2. 重新设计并实现搜索页。
|
||||
3. 重新设计并实现设置页。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 这三个页面不再是旧 panel 原样移植。
|
||||
2. 设置页已移除 panel 可见性开关。
|
||||
|
||||
### 阶段 4:态势与详情重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 将 stats 和 legend 合并为新的态势页。
|
||||
2. 实现统一详情页。
|
||||
3. 对象点击与搜索结果点击都可切入详情页。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 不再存在移动端独立 legend / stats 面板。
|
||||
2. 详情页成为统一对象信息入口。
|
||||
|
||||
### 阶段 5:媒体拆分重做
|
||||
|
||||
目标:
|
||||
|
||||
1. 将原媒体面板拆成两个移动端页面:新闻页、TV 页。
|
||||
2. 分别重做这两个页面的布局。
|
||||
3. 保留各自必要操作,但不继续共享桌面 panel 结构。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 新闻与 TV 各自成为独立移动端页面。
|
||||
2. 不再使用桌面媒体 panel 的 tab 结构作为移动端主体。
|
||||
|
||||
### 阶段 6:手感与真机修正
|
||||
|
||||
目标:
|
||||
|
||||
1. 调整抽屉高度、节奏、手势阈值。
|
||||
2. 调整 tab 密度与文字层级。
|
||||
3. 优化 iPhone / Android 安全区。
|
||||
4. 优化抽屉滚动与地球拖拽边界。
|
||||
|
||||
完成标准:
|
||||
|
||||
1. 抽屉和地球不会抢手势。
|
||||
2. 手机端各页面信息层次清晰。
|
||||
3. 真机下无遮挡、无死层、无错误交互心智。
|
||||
|
||||
## 技术落点调整
|
||||
|
||||
### [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
|
||||
|
||||
职责:
|
||||
|
||||
- 只保留移动端抽屉壳层
|
||||
- 为各页面提供新的页面容器
|
||||
|
||||
不再把旧 panel 作为最终结构直接塞进抽屉。
|
||||
|
||||
### [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 管理抽屉开合
|
||||
- 管理 tab 切换
|
||||
- 管理详情页切入
|
||||
- 管理 mobile / desktop 分流
|
||||
|
||||
### [frontend/public/earth/js/search.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/search.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 保留搜索能力和结果逻辑
|
||||
- 输出给新的移动端搜索页
|
||||
|
||||
### [frontend/public/earth/js/info-card.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/info-card.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 从桌面 info-card 逻辑中提取可复用的数据层
|
||||
- 服务新的移动端详情页
|
||||
|
||||
### [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 为新的 TV 页面提供数据和状态
|
||||
- 不再直接主导移动端媒体 panel 壳层
|
||||
|
||||
### [frontend/public/earth/js/news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
|
||||
职责:
|
||||
|
||||
- 为新的新闻页面提供列表和区域焦点数据
|
||||
|
||||
### CSS
|
||||
|
||||
需要新增真正的移动端页面样式,而不是继续在旧 panel class 上堆条件分支:
|
||||
|
||||
- 图层页样式
|
||||
- 搜索页样式
|
||||
- 态势页样式
|
||||
- 新闻页样式
|
||||
- TV 页样式
|
||||
- 设置页样式
|
||||
- 详情页样式
|
||||
- 移动端右上角通知 / 胶囊提示样式
|
||||
|
||||
## 验收标准
|
||||
|
||||
1. `mobile` 模式下不再显示旧 toolbar。
|
||||
2. `mobile` 模式下不再把旧 panel 直接作为最终 UI。
|
||||
3. 图层、搜索、态势、新闻、TV、设置都是重新设计的移动端页面。
|
||||
4. 设置页不再包含移动端无意义的 panel 显示/隐藏项。
|
||||
5. 新闻与 TV 已拆分为两个移动端页面。
|
||||
6. legend / stats 已整合为态势页。
|
||||
7. 详情页成为统一对象详情入口。
|
||||
8. 移动端通知栏和胶囊提示已统一放到右上角安全区,而不是屏幕正中。
|
||||
|
||||
## 结论
|
||||
|
||||
本计划进一步明确:
|
||||
|
||||
移动端目标不是“把桌面 HUD 放进抽屉”,而是“以抽屉为载体,重做一套适合手机端的信息页面”。
|
||||
|
||||
后续开发必须以此为准:
|
||||
|
||||
- 复用数据
|
||||
- 重做界面
|
||||
- 清除桌面遗留心智
|
||||
156
docs/plans/earth-news-source-configuration-and-collector-plan.md
Normal file
156
docs/plans/earth-news-source-configuration-and-collector-plan.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Earth News Source Configuration And Collector Plan
|
||||
|
||||
## Why
|
||||
|
||||
当前 Earth 的“态势新闻”由 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 直接在请求时抓取 RSS / Google News feed,再按当前地球视角中心区域聚合返回。
|
||||
|
||||
这条链已经可用,但存在两个明显限制:
|
||||
|
||||
- 新闻源写死在代码里,不能像 TV 直播源一样从后台维护
|
||||
- 新闻并未进入统一采集体系,没有采集状态、失败监控、历史数据和后续 AI 复用能力
|
||||
|
||||
因此这块更合理的路线不是一步到位重写,而是分阶段推进:
|
||||
|
||||
1. 先做“新闻源配置化”
|
||||
2. 再做“新闻采集器化”
|
||||
|
||||
## Current State
|
||||
|
||||
当前实现分布在:
|
||||
|
||||
- 新闻接口
|
||||
- [news.py](/home/ray/dev/linkong/planet/backend/app/api/v1/news.py)
|
||||
- 实时聚合逻辑
|
||||
- [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py)
|
||||
- 前端消费
|
||||
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
|
||||
|
||||
当前新闻源包含:
|
||||
|
||||
- `BBC World` RSS
|
||||
- `DW Top Stories` RSS
|
||||
- 按区域关键词拼出来的 `Google News RSS`
|
||||
- `Global`
|
||||
- `Americas`
|
||||
- `Europe`
|
||||
- `Middle East / Africa`
|
||||
- `Asia Pacific`
|
||||
|
||||
当前不是采集器,也不落库,只做内存缓存。
|
||||
|
||||
## Phase 1: Source Configuration
|
||||
|
||||
### Goal
|
||||
|
||||
把 `NEWS_FEED_SOURCES` 从硬编码列表升级成可配置新闻源目录,但继续保留当前“实时聚合”的工作方式。
|
||||
|
||||
### Scope
|
||||
|
||||
- 为 Earth news 建立独立配置结构
|
||||
- 支持后台维护 feed 源
|
||||
- 支持启用/禁用、优先级、区域、源类型
|
||||
- 保持现有 `/api/v1/news/earth-feed` 输出协议不变
|
||||
|
||||
### Proposed Shape
|
||||
|
||||
建议配置字段至少包括:
|
||||
|
||||
- `id`
|
||||
- `name`
|
||||
- `region`
|
||||
- `feed_url`
|
||||
- `homepage_url`
|
||||
- `source_type`
|
||||
- `priority`
|
||||
- `is_enabled`
|
||||
- 可选 `query_profile`
|
||||
- 可选 `language`
|
||||
- 可选 `notes`
|
||||
|
||||
### Suggested Storage
|
||||
|
||||
优先走系统设置或单独的 news source settings payload,而不是先建复杂新表。
|
||||
|
||||
推荐原因:
|
||||
|
||||
- 改动小
|
||||
- 易上线
|
||||
- 和当前 TV settings 维护体验更接近
|
||||
- 先解决“写死在代码里”的问题
|
||||
|
||||
### Non-goals
|
||||
|
||||
这一阶段不做:
|
||||
|
||||
- 新闻入库
|
||||
- 新闻历史回看
|
||||
- 新闻采集任务监控
|
||||
- 新闻去重流水线
|
||||
|
||||
## Phase 2: News Collectorization
|
||||
|
||||
### Goal
|
||||
|
||||
把“态势新闻”升级为真正的采集器链路,使其进入采集系统和数据层。
|
||||
|
||||
### Scope
|
||||
|
||||
- 新增专用 news collector
|
||||
- 按配置源定时采集 RSS / feed
|
||||
- 做标题/链接级去重
|
||||
- 建立统一新闻记录模型
|
||||
- 为 Earth、控制台、AI 研判复用同一份新闻数据
|
||||
|
||||
### Benefits
|
||||
|
||||
- 有采集状态
|
||||
- 有失败监控
|
||||
- 有历史缓存
|
||||
- 可以做时间轴 / 区域新闻基线
|
||||
- 可以作为 AI 引用证据
|
||||
|
||||
### Required Design Work
|
||||
|
||||
需要提前明确:
|
||||
|
||||
- 新闻数据模型
|
||||
- 去重策略
|
||||
- 过期清理策略
|
||||
- 区域映射策略
|
||||
- 聚合排序策略
|
||||
- 新闻与 Earth 当前视角/区域的关联方式
|
||||
|
||||
### Candidate Output Model
|
||||
|
||||
至少应包含:
|
||||
|
||||
- `source_id`
|
||||
- `headline`
|
||||
- `summary`
|
||||
- `url`
|
||||
- `publisher`
|
||||
- `region`
|
||||
- `published_at`
|
||||
- `language`
|
||||
- `tags`
|
||||
- `raw_feed_source`
|
||||
- `reference_date`
|
||||
|
||||
## Recommended Order
|
||||
|
||||
推荐执行顺序:
|
||||
|
||||
1. 先完成 Phase 1 配置化
|
||||
2. 保持 Earth 继续实时聚合,但改为读取配置源
|
||||
3. 等新闻源稳定后,再设计 Phase 2 的 collector / storage / dedupe
|
||||
|
||||
## Decision
|
||||
|
||||
当前结论:
|
||||
|
||||
- TV 直播源:优先采集器化
|
||||
- 态势新闻:优先配置化,再采集器化
|
||||
|
||||
## Source Note
|
||||
|
||||
This plan is newly created for the Planet repo to separate the short-term "configurable source directory" work from the longer-term "collectorized news pipeline" work.
|
||||
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
98
docs/plans/earth-predicted-orbit-plan.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Earth Predicted Orbit Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
|
||||
|
||||
- 从当前时刻开始
|
||||
- 绕地球一圈
|
||||
- 当前点最亮
|
||||
- 向后沿轨道逐步衰减
|
||||
|
||||
## Current State
|
||||
|
||||
当前已经有:
|
||||
|
||||
- 卫星历史轨迹
|
||||
- 锁定卫星
|
||||
- 轨道高亮与相关联动
|
||||
|
||||
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
|
||||
|
||||
## Why It Is Valuable
|
||||
|
||||
预测轨道可以明显提升:
|
||||
|
||||
- 锁定卫星后的空间可读性
|
||||
- 轨道类型辨识
|
||||
- 演示解释力
|
||||
|
||||
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
|
||||
|
||||
## Scope
|
||||
|
||||
### Phase 1
|
||||
|
||||
- 锁定卫星时显示一整圈预测轨道
|
||||
- 解锁时隐藏
|
||||
- 不替代现有普通轨迹系统
|
||||
|
||||
### Phase 2
|
||||
|
||||
- 根据轨道类型调整采样率
|
||||
- GEO / MEO / LEO 不同密度
|
||||
- 进一步减少 fallback 轨迹的比例
|
||||
|
||||
## Implementation Direction
|
||||
|
||||
### 1. Orbit period
|
||||
|
||||
基于 `meanMotion` 估算轨道周期。
|
||||
|
||||
### 2. Predicted samples
|
||||
|
||||
以固定采样步长从 `now -> now + period` 推算轨迹点。
|
||||
|
||||
### 3. Render object lifecycle
|
||||
|
||||
预测轨道应是一个独立渲染对象:
|
||||
|
||||
- show
|
||||
- update
|
||||
- hide
|
||||
- dispose
|
||||
|
||||
### 4. Visual semantics
|
||||
|
||||
预测轨道不应与普通尾迹混淆:
|
||||
|
||||
- 更稳定
|
||||
- 更完整
|
||||
- 透明度沿轨道衰减
|
||||
- 当前点附近更亮
|
||||
|
||||
## Known Risks
|
||||
|
||||
### 1. TLE propagation gaps
|
||||
|
||||
部分卫星可能出现 SGP4 计算不足,需要 fallback。
|
||||
|
||||
### 2. Multiple orbit lines
|
||||
|
||||
必须确保:
|
||||
|
||||
- 锁定切换前先清旧轨道
|
||||
- 页面隐藏/销毁时清理
|
||||
|
||||
### 3. Performance
|
||||
|
||||
GEO 轨道点数高,采样率需要按轨道类型分层。
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 锁定单颗卫星时只显示一条预测轨道
|
||||
2. 解锁后轨道立即清除
|
||||
3. 不同轨道类型下点数可控
|
||||
4. 页面切换回来不会闪出旧轨道残留
|
||||
472
docs/plans/earth-real-terrain-plan.md
Normal file
472
docs/plans/earth-real-terrain-plan.md
Normal file
@@ -0,0 +1,472 @@
|
||||
# Earth Real Terrain Plan
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 页当前的“程序噪声假地形”替换成基于真实 DEM 的可用地形层,使 `地形 terrain` 开关真正显示全球海拔起伏,而不是占位效果。
|
||||
|
||||
当前占位实现位于:
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
|
||||
具体问题:
|
||||
|
||||
- `createTerrain()` 直接对球体顶点应用 `simplex noise`
|
||||
- 没有真实海拔数据来源
|
||||
- 没有分辨率分层
|
||||
- 没有和当前相机/视角配套的性能控制
|
||||
|
||||
## Constraints
|
||||
|
||||
本计划必须贴合当前 Earth 架构,而不是引入一套全新的地形引擎:
|
||||
|
||||
- 地球主体仍然是一个 Three.js sphere
|
||||
- 海缆、登陆点、卫星、BGP 都已经建立在当前球体坐标系之上
|
||||
- 不能为了地形把整页改成 Cesium/MapLibre Globe 之类的全栈替换
|
||||
- 第一阶段优先做“真实可用”,不是一步到位做摄影测量级地形
|
||||
|
||||
## Recommended Data Source
|
||||
|
||||
### Primary recommendation
|
||||
|
||||
使用公开的 Terrarium 编码高程瓦片作为浏览器端高度来源,第一阶段优先接入:
|
||||
|
||||
- Mapzen/AWS `Terrarium` elevation tiles
|
||||
参考:[Mapzen terrain tile format / Terrarium](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
|
||||
原因:
|
||||
|
||||
- 已经是全球瓦片化高程
|
||||
- 浏览器端按 tile 请求,最适合当前 Earth 这种在线 globe
|
||||
- 编码简单稳定:
|
||||
- `heightMeters = (R * 256 + G + B / 256) - 32768`
|
||||
- 不需要我们先离线拼整球 DEM
|
||||
|
||||
### Data quality upgrade path
|
||||
|
||||
如果后面第一阶段效果确认可用,再逐步升级到底层源:
|
||||
|
||||
- Copernicus DEM GLO-30
|
||||
参考:[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
- 或用 Copernicus / SRTM / ASTER 等离线切成我们自己的 terrain tiles
|
||||
|
||||
这条升级路径适合第二阶段,不建议一开始就直接自建全球瓦片服务。
|
||||
|
||||
## Why Not Replace the Engine
|
||||
|
||||
不建议为了地形直接切到 Cesium terrain / quantized mesh 引擎,原因:
|
||||
|
||||
- 现有 Earth 业务对象都依附当前球面坐标
|
||||
- 切引擎会同时波及:
|
||||
- 海缆绘制
|
||||
- 卫星/轨迹
|
||||
- BGP 标记
|
||||
- HUD 与交互
|
||||
- 这是“重做一页”,不是“给地形层接真实数据”
|
||||
|
||||
所以推荐路线是:
|
||||
|
||||
- 保持当前 sphere globe
|
||||
- 为 sphere 增加真实高度位移层
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
分三期推进。
|
||||
|
||||
### Phase 1 — Global Heightmap Terrain Overlay
|
||||
|
||||
目标:
|
||||
|
||||
- 地形层切换后显示真实海拔起伏
|
||||
- 全球范围可用
|
||||
- 性能可控
|
||||
|
||||
做法:
|
||||
|
||||
1. 新增 terrain 数据模块
|
||||
|
||||
建议文件:
|
||||
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
职责:
|
||||
|
||||
- 选择 DEM zoom level
|
||||
- 请求 Terrarium tiles
|
||||
- 解码 tile 高程
|
||||
- 将高程重采样到当前地形球体网格
|
||||
|
||||
2. 替换 `createTerrain()`
|
||||
|
||||
当前:
|
||||
|
||||
- 在 [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) 中同步生成噪声地形
|
||||
|
||||
调整后:
|
||||
|
||||
- `createTerrain()` 只负责创建 terrain mesh 骨架
|
||||
- 真正的顶点位移由 terrain 模块异步注入
|
||||
|
||||
3. 第一阶段采用“整球低分辨率位移”
|
||||
|
||||
不要一上来做动态 patch stitching。第一阶段更稳的办法是:
|
||||
|
||||
- 保留一张全球 terrain sphere
|
||||
- 使用较低分辨率几何
|
||||
- 例如 `SphereGeometry(radius, 192, 192)` 或 `256/256`
|
||||
- 运行时按一个固定地形 zoom(如 `z=4` 或 `z=5`)抓取覆盖全球的 Terrarium tiles
|
||||
- 将 tile 解码后重投影到经纬度采样网格
|
||||
- 将每个球面顶点按真实高度抬升
|
||||
|
||||
这样第一阶段就能做到:
|
||||
|
||||
- 有真实地形
|
||||
- 不需要复杂的局部 LOD
|
||||
- 不会让现有球体对象体系爆炸
|
||||
|
||||
### Phase 2 — View-Aware Refinement
|
||||
|
||||
目标:
|
||||
|
||||
- 正面可见区域更精细
|
||||
- 背面与远处维持低成本
|
||||
|
||||
做法:
|
||||
|
||||
- 引入“基础全球地形 + 当前视角高分局部补丁”
|
||||
- 正面区域额外抓更高 zoom 的高程 tile
|
||||
- 只替换局部顶点位移或局部 overlay mesh
|
||||
|
||||
这一阶段适合在第一阶段稳定后做。
|
||||
|
||||
### Phase 3 — Normals / Shading / Terrain UX
|
||||
|
||||
目标:
|
||||
|
||||
- 地形不仅有起伏,还更好看、更可读
|
||||
|
||||
包括:
|
||||
|
||||
- 根据高度生成更合理的 normals
|
||||
- 调整 terrain material,使山脉/高原更易读
|
||||
- 可选加入:
|
||||
- hillshade
|
||||
- contour lines
|
||||
- snowline / bathymetry tint
|
||||
|
||||
## Calibration Overlay Before More Terrain Tuning
|
||||
|
||||
在当前项目里,terrain 看起来“不像真地形”,不一定只是 DEM 或 exaggeration 不够,也可能是因为缺少稳定参照物。
|
||||
|
||||
没有清晰的海岸线、国界线和地表分层时,人眼很难判断:
|
||||
|
||||
- 山脉是不是在应该高的地方高
|
||||
- terrain 是否真的贴在正确的大陆位置上
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否存在偏移
|
||||
|
||||
这里要明确区分两件事:
|
||||
|
||||
- 国界线不会修好错误的 terrain
|
||||
- 但海岸线 / 国界线会让我们更容易判断 terrain 有没有贴准
|
||||
|
||||
所以在继续盲调 terrain 参数之前,建议先插入一个“校准参照层”阶段。
|
||||
|
||||
### Recommended order for the calibration layer
|
||||
|
||||
1. 海岸线
|
||||
2. 国界线
|
||||
3. 再继续调 terrain
|
||||
|
||||
原因:
|
||||
|
||||
- 海岸线比国界线更基础,也更接近真实地表边界
|
||||
- 判断 terrain 是否贴准,最重要的是大陆边缘和山脉/海岸关系
|
||||
- 国界线更多是政治边界,只能作为辅助参照
|
||||
|
||||
如果只加国界线,不加海岸线,效果仍然可能会怪,因为:
|
||||
|
||||
- 很多国界线本来就是人为直线
|
||||
- 它们并不总是跟真实地形走
|
||||
|
||||
### Suggested layer order during debugging
|
||||
|
||||
建议调试期临时把地球层次明确成:
|
||||
|
||||
1. base earth texture
|
||||
2. coastline / borders overlay
|
||||
3. terrain relief
|
||||
4. cables / landing points / bgp / satellites
|
||||
|
||||
这样会比现在更容易判断:
|
||||
|
||||
- 山脉是否位于正确区域
|
||||
- terrain 是否和地表对齐
|
||||
- 国界/海岸是否漂移
|
||||
|
||||
### Suggested data source for the calibration overlay
|
||||
|
||||
优先用 `Natural Earth` 的轻量全球矢量数据:
|
||||
|
||||
- 海岸线(coastline)
|
||||
- Admin 0 国界线(country borders)
|
||||
|
||||
优点:
|
||||
|
||||
- 全球一致
|
||||
- 轻量
|
||||
- 很适合当前 Three.js globe 做 overlay
|
||||
|
||||
### Recommended execution path
|
||||
|
||||
#### Phase A — Add reference overlays
|
||||
|
||||
先加两层可开关的参考线:
|
||||
|
||||
- 海岸线
|
||||
- 国界线
|
||||
|
||||
这两层的目标不是最终美术表现,而是调试 / 校准。
|
||||
|
||||
#### Phase B — Recalibrate terrain against coastline
|
||||
|
||||
有了海岸线以后,再重新看 terrain:
|
||||
|
||||
- terrain 是否和大陆边缘错位
|
||||
- 地球纹理、本初子午线、terrain 采样之间是否有固定偏移
|
||||
|
||||
#### Phase C — Decide whether to keep the current terrain path
|
||||
|
||||
这时再决定后面的路线:
|
||||
|
||||
- 如果发现真实高程整体是对的,只是缺少 shading / readability
|
||||
继续保留当前 DEM + terrain overlay 路线
|
||||
- 如果发现整球采样投影、本初子午线或 overlay 关系本身就很别扭
|
||||
再考虑重做 terrain pipeline
|
||||
|
||||
### Practical recommendation
|
||||
|
||||
当前阶段不建议“从头开始重做 terrain”。
|
||||
|
||||
更稳的策略是:
|
||||
|
||||
- 暂停继续盲调 terrain 参数
|
||||
- 先补海岸线 / 国界线作为校准参照层
|
||||
- 再基于参照层判断 terrain 是“参数没调好”,还是“整条实现路径有偏移”
|
||||
|
||||
## Recommended Geometry Model
|
||||
|
||||
### First usable model
|
||||
|
||||
保留一层独立 terrain sphere:
|
||||
|
||||
- base earth sphere:贴纹理、昼夜、海洋
|
||||
- terrain sphere:略高于地球半径,真实高程位移
|
||||
|
||||
建议:
|
||||
|
||||
- `terrainBaseRadius = CONFIG.earthRadius + 0.2`
|
||||
- 高度缩放使用真实米制换算,再乘一个可调 exaggeration
|
||||
|
||||
示例关系:
|
||||
|
||||
- `heightWorld = (elevationMeters / 6371000) * CONFIG.earthRadius * exaggeration`
|
||||
|
||||
建议第一阶段 `exaggeration = 1.3 ~ 1.8`
|
||||
|
||||
因为完全真实比例在全球球体上会太平,看不出来。
|
||||
|
||||
## Tile Decoding Plan
|
||||
|
||||
### Terrarium decode
|
||||
|
||||
对于每个高程 tile 像素:
|
||||
|
||||
```text
|
||||
heightMeters = (R * 256 + G + B / 256) - 32768
|
||||
```
|
||||
|
||||
### Sampling path
|
||||
|
||||
对于 terrain mesh 上每个顶点:
|
||||
|
||||
1. 将顶点方向转成经纬度
|
||||
2. 将经纬度映射到 Web Mercator tile 坐标
|
||||
3. 找到对应的 tile 和像素
|
||||
4. 解码高程
|
||||
5. 将顶点沿法线方向抬升
|
||||
|
||||
### Needed helpers
|
||||
|
||||
建议新增:
|
||||
|
||||
- `latLonToTileXY(lat, lon, z)`
|
||||
- `tilePixelFromLatLon(lat, lon, z, tileSize)`
|
||||
- `decodeTerrariumHeight(r, g, b)`
|
||||
|
||||
## Caching Strategy
|
||||
|
||||
为了不让地形开关每次重开都重新抓全量 tile:
|
||||
|
||||
- terrain tile 按 `z/x/y` 存到内存缓存
|
||||
- terrain mesh 结果也缓存一份
|
||||
- 当用户关闭/开启 terrain:
|
||||
- 直接复用已有位移结果
|
||||
|
||||
建议:
|
||||
|
||||
- `Map<string, Float32Array | ImageBitmap>`
|
||||
|
||||
## Material Strategy
|
||||
|
||||
第一阶段不要复杂化。
|
||||
|
||||
建议 terrain material:
|
||||
|
||||
- 半透明低饱和地形色
|
||||
- 比 base earth 稍亮或稍偏冷
|
||||
- 保留当前 HUD 风格下的可读性
|
||||
|
||||
第一阶段不需要:
|
||||
|
||||
- 真实土地覆被纹理
|
||||
- 独立卫星影像贴 terrain
|
||||
|
||||
因为那会和现有地球纹理、云层、昼夜 shader 打架。
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Files to change
|
||||
|
||||
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
|
||||
- 重写 `createTerrain()`
|
||||
- 删除 simplex noise 占位逻辑
|
||||
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
|
||||
- 初始化 terrain 数据加载
|
||||
- 控制 terrain readiness / loading message
|
||||
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
|
||||
- `toggleTerrain` 逻辑保持,但应能区分:
|
||||
- mesh 已就绪
|
||||
- 正在加载
|
||||
- [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
|
||||
- 新增 `TERRAIN_CONFIG`
|
||||
- 新文件:
|
||||
- `frontend/public/earth/js/terrain.js`
|
||||
|
||||
### Suggested new config
|
||||
|
||||
建议新增:
|
||||
|
||||
```js
|
||||
export const TERRAIN_CONFIG = {
|
||||
enabled: true,
|
||||
tileSize: 256,
|
||||
baseZoom: 4,
|
||||
baseRadiusOffset: 0.2,
|
||||
exaggeration: 1.5,
|
||||
opacity: 0.55,
|
||||
color: 0x6c876f,
|
||||
maxConcurrentRequests: 8,
|
||||
cacheEnabled: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Loading UX
|
||||
|
||||
地形第一次开启时,不能像现在一样瞬时切换。
|
||||
|
||||
建议:
|
||||
|
||||
- 如果地形数据尚未准备:
|
||||
- 顶部状态条显示:`正在加载真实地形数据...`
|
||||
- 完成后:
|
||||
- `真实地形已就绪`
|
||||
|
||||
如果加载失败:
|
||||
|
||||
- 保留 base earth
|
||||
- 显示轻量错误提示
|
||||
- 不要让 terrain 开关卡死在“开”状态
|
||||
|
||||
## Risks
|
||||
|
||||
### 1. Global tile count too high
|
||||
|
||||
即使 `z=5` 全球 tile 数也不少。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段限定低 zoom
|
||||
- 并发上限
|
||||
- 缓存
|
||||
|
||||
### 2. Mesh resolution too low
|
||||
|
||||
如果球面分段太低,山脉会被抹平。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段先选一个中等分辨率
|
||||
- 用 exaggeration 保证可见性
|
||||
|
||||
### 3. Existing overlays may z-fight with terrain
|
||||
|
||||
海缆、登陆点、BGP、卫星相关对象都假设地球半径固定。
|
||||
|
||||
缓解:
|
||||
|
||||
- terrain sphere 单独作为 overlay
|
||||
- overlay 保持略低或略高的固定 offset
|
||||
- 必要时局部调整 landing point / cable altitude offset
|
||||
|
||||
### 4. Mercator sampling distortion near poles
|
||||
|
||||
Web Mercator 在高纬会有失真。
|
||||
|
||||
缓解:
|
||||
|
||||
- 第一阶段接受
|
||||
- 后续若需要更严格极区质量,再上 geodetic reprojection pipeline
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
第一阶段完成后,应满足:
|
||||
|
||||
1. `地形 terrain` 开关开启时,地表起伏明显不再是随机噪声
|
||||
2. 喜马拉雅、安第斯、落基山、东非高原等全球大尺度地形可辨认
|
||||
3. 关闭/重新开启 terrain 不重复全量请求
|
||||
4. 不破坏:
|
||||
- 海缆
|
||||
- 卫星
|
||||
- BGP
|
||||
- 地球昼夜
|
||||
- 天球层
|
||||
|
||||
## Suggested Execution Order
|
||||
|
||||
1. 引入 `TERRAIN_CONFIG`
|
||||
2. 新建 `terrain.js`
|
||||
3. 实现 Terrarium tile 请求与 decode
|
||||
4. 用低 zoom 全球 tile 构建真实 terrain sphere
|
||||
5. 接管 `toggleTerrain()`
|
||||
6. 调整 terrain material 和高度 exaggeration
|
||||
7. 做缓存
|
||||
8. 再考虑第二阶段局部高分 refinement
|
||||
|
||||
## Source References
|
||||
|
||||
- Mapzen Terrarium / AWS terrain tiles
|
||||
[Mapzen Terrain Tile Service](https://www.mapzen.com/blog/terrain-tile-service/)
|
||||
- Terrarium tile experiments / format background
|
||||
[mapzen/terrarium](https://github.com/mapzen/terrarium)
|
||||
- Copernicus DEM overview
|
||||
[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
如果现在就要开始做,我建议直接按这条路线开工:
|
||||
|
||||
- 第一阶段接入 Terrarium 全球高程 tile
|
||||
- 替换掉当前 simplex 假地形
|
||||
- 先做一层真实可见的全球 terrain overlay
|
||||
- 等第一阶段稳定,再做视角高分 refinement
|
||||
|
||||
这是对当前项目风险最低、最贴合现有 Earth 架构的一条路。
|
||||
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
111
docs/plans/earth-renderer-architecture-separation-plan.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Earth Renderer / Logic Separation Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/earth-architecture-refactor.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
将 Earth 前端继续往“逻辑层 / 状态层 / 渲染层”分离推进,降低后续这几类工作的耦合成本:
|
||||
|
||||
- Three.js 渲染重构
|
||||
- 部分图层替换实现
|
||||
- 未来 UE / Cesium 客户端迁移
|
||||
- Earth 行为逻辑复用
|
||||
|
||||
## Why This Matters
|
||||
|
||||
当前 Earth 已经有一些良好分层,例如:
|
||||
|
||||
- 图层显隐入口
|
||||
- Cable state 枚举与状态 map
|
||||
- 交互逻辑与实际视觉效果的部分分离
|
||||
|
||||
但还没有形成一套更明确的统一规则。现在的风险是:
|
||||
|
||||
- 同一类对象的 hover / locked / hidden / loading 语义不一致
|
||||
- 状态和渲染更新散落在多个模块
|
||||
- 后续再加新图层时容易复制旧逻辑
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Earth 对每类对象都尽量拆成三层:
|
||||
|
||||
1. `state layer`
|
||||
- 保存对象状态
|
||||
- 例如:`normal / hovered / locked / hidden / loading`
|
||||
|
||||
2. `logic layer`
|
||||
- 处理点击、悬停、锁定、过滤、显隐切换
|
||||
- 不直接关心 Three.js 具体材质怎么改
|
||||
|
||||
3. `renderer layer`
|
||||
- 根据状态更新 Three.js / HUD 外观
|
||||
- 是最容易针对不同渲染引擎替换的一层
|
||||
|
||||
## Current Good Signals
|
||||
|
||||
当前已经接近这条方向的地方:
|
||||
|
||||
- cable 状态管理
|
||||
- 部分 landing point 状态同步
|
||||
- layer button 的统一状态入口
|
||||
- tooltip / legend / info-card 开始朝状态驱动靠拢
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Standardize object state enums
|
||||
|
||||
优先为这些对象建立更稳定的状态语义:
|
||||
|
||||
- cables
|
||||
- satellites
|
||||
- landing points
|
||||
- BGP markers
|
||||
- media / news 面板入口按钮
|
||||
|
||||
### 2. Unify state-to-visual adapters
|
||||
|
||||
为各模块建立更清晰的渲染适配函数,例如:
|
||||
|
||||
- `applyCableVisualState()`
|
||||
- `applySatelliteVisualState()`
|
||||
- `applyBGPVisualState()`
|
||||
|
||||
要求:
|
||||
|
||||
- 逻辑层只改状态
|
||||
- 视觉层负责把状态映射到材质、透明度、发光、尺寸、文字
|
||||
|
||||
### 3. Separate Earth UI state from render state
|
||||
|
||||
HUD / 面板 / 图层按钮状态也需要和渲染状态分离:
|
||||
|
||||
- `loading`
|
||||
- `active`
|
||||
- `locked`
|
||||
- `hidden`
|
||||
- `error`
|
||||
|
||||
不要再让 UI 通过“猜渲染结果”推导业务状态。
|
||||
|
||||
### 4. Prepare migration-safe boundaries
|
||||
|
||||
后续如果做 UE / Cesium 客户端,尽量保留:
|
||||
|
||||
- 状态枚举
|
||||
- 交互规则
|
||||
- 数据层接口
|
||||
|
||||
只替换:
|
||||
|
||||
- Three.js 具体渲染实现
|
||||
- HUD 展示实现
|
||||
|
||||
## Practical Rule
|
||||
|
||||
后续 Earth 新功能开发时,优先问三个问题:
|
||||
|
||||
1. 这个状态由谁持有?
|
||||
2. 这个交互逻辑在哪一层处理?
|
||||
3. 这个视觉变化是否能在不改逻辑的情况下单独替换?
|
||||
|
||||
如果答不上来,就说明还在把状态、逻辑、渲染揉在一起。
|
||||
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
82
docs/plans/earth-webgl-instancing-satellites-plan.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Earth WebGL Instancing Satellites Plan
|
||||
|
||||
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/webgl-instancing-satellites.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
把 Earth 卫星渲染从当前方案继续推进到更适合高数量卫星的 instancing 方向,目标是:
|
||||
|
||||
- 支持更多卫星
|
||||
- 降低渲染压力
|
||||
- 仍然保留当前数据层和交互层
|
||||
|
||||
## Why It Matters
|
||||
|
||||
当前卫星系统已经具备:
|
||||
|
||||
- 数据加载
|
||||
- 轨迹
|
||||
- 选择/锁定
|
||||
- 图例
|
||||
- 相关区域联动
|
||||
|
||||
但当卫星数量持续增加时,渲染层会越来越接近瓶颈。
|
||||
|
||||
## Recommended Direction
|
||||
|
||||
优先调研并原型验证:
|
||||
|
||||
- `InstancedBufferGeometry + custom shader`
|
||||
|
||||
而不是一开始就推倒重写成 raw WebGL。
|
||||
|
||||
原因:
|
||||
|
||||
- 仍能保留 Three.js 主架构
|
||||
- 更容易渐进迁移
|
||||
- 比继续堆普通点渲染更有上限
|
||||
|
||||
## What Should Stay
|
||||
|
||||
尽量保留这些层:
|
||||
|
||||
- 卫星数据获取
|
||||
- 位置计算
|
||||
- 锁定/悬停逻辑
|
||||
- legend / info-card / 相关联动
|
||||
|
||||
主要替换的是:
|
||||
|
||||
- 卫星点渲染实现
|
||||
- 颜色/大小等实例属性更新方式
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Prototype
|
||||
|
||||
- 用 instancing 做最小原型
|
||||
- 先只渲染卫星点
|
||||
- 不碰轨迹系统
|
||||
|
||||
### Phase 2: Integrate
|
||||
|
||||
- 接入当前 `satellites.js` 数据层
|
||||
- 保留当前选择和高亮语义
|
||||
|
||||
### Phase 3: Tune
|
||||
|
||||
- 调整可视大小
|
||||
- 调整选中高亮方式
|
||||
- 评估是否需要分层 LOD
|
||||
|
||||
## Risks
|
||||
|
||||
1. 透明度排序更复杂
|
||||
2. Shader 调试成本更高
|
||||
3. 选中态和 hover 态不能简单复用旧材质逻辑
|
||||
|
||||
## Acceptance
|
||||
|
||||
1. 在更高卫星数量下保持可接受帧率
|
||||
2. 不破坏现有锁定/高亮语义
|
||||
3. 图例、信息卡、相关卫星联动仍然成立
|
||||
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
361
docs/plans/frontend-ai-playground-development-plan.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# AI Playground Development Plan
|
||||
|
||||
## 目标
|
||||
|
||||
这份计划用于统一 `aiprovider`、`backend AI facade`、`Playground` 页面,以及后续 `BGP / 告警 / 数据源健康` 等 AI 入口的演进方向。
|
||||
|
||||
当前原则:
|
||||
|
||||
- `aiprovider` 继续作为独立模型网关
|
||||
- `backend` 继续作为稳定业务入口
|
||||
- `frontend` 负责测试台和业务 UI
|
||||
- 先做“可控、可验证、可解释”的 AI 能力,再逐步引入 agent/tool calling
|
||||
|
||||
## 当前已完成
|
||||
|
||||
### 1. AI 网关基础层
|
||||
|
||||
已完成:
|
||||
|
||||
- 独立 `aiprovider` 服务
|
||||
- `backend -> aiprovider -> model provider` 调用链
|
||||
- `provider/status` 与 `situational-awareness/analyze` 稳定接口
|
||||
- `X-Request-ID` 透传
|
||||
- 轻量超时与重试
|
||||
- MiniMax / Anthropic-compatible / OpenAI-compatible / Ollama 适配
|
||||
|
||||
相关文件:
|
||||
|
||||
- [backend/app/api/v1/ai.py](/home/ray/dev/linkong/planet/backend/app/api/v1/ai.py)
|
||||
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
|
||||
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
|
||||
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
|
||||
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
|
||||
|
||||
### 2. 本地运行与配置打通
|
||||
|
||||
已完成:
|
||||
|
||||
- `planet.sh` 启动链路纳入 `aiprovider`
|
||||
- `planet.sh` 启动完成后输出 Playground 入口
|
||||
- `docker-compose.yml` 为 `aiprovider` 加入 `env_file`
|
||||
- `backend/.env` 与 `aiprovider/.env` 两侧 service token 对齐
|
||||
- `Playground` 状态缓存,避免页面切换时每次都重新请求 provider 状态
|
||||
|
||||
相关文件:
|
||||
|
||||
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
|
||||
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
|
||||
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
|
||||
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
|
||||
|
||||
### 3. Playground UI 基础版
|
||||
|
||||
已完成:
|
||||
|
||||
- 新增前端路由 `/playground`
|
||||
- 左侧 `Provider 状态 + 测试说明`
|
||||
- 右侧 `请求 / 结果` Tabs
|
||||
- `Provider 状态` 支持手动刷新
|
||||
- `测试说明` 支持折叠
|
||||
- 内部区域采用细滚动条
|
||||
- 页面布局开始遵循“单屏工作区 + 模块内部滚动”规范
|
||||
|
||||
相关文件:
|
||||
|
||||
- [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
|
||||
- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
|
||||
- [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
|
||||
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
|
||||
|
||||
### 4. 前端布局规范沉淀
|
||||
|
||||
已完成:
|
||||
|
||||
- 把“一屏工作区、主模块优先、模块内部滚动”的规范文档化
|
||||
- 明确 `BGP` 页面为当前参考实现
|
||||
|
||||
相关文件:
|
||||
|
||||
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
|
||||
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
|
||||
|
||||
## 当前限制
|
||||
|
||||
### 1. Playground 还是 prompt playground,不是 agent playground
|
||||
|
||||
当前 `Playground` 的 `观察项 / 目标 / 约束条件` 都是人工输入。
|
||||
|
||||
模型现在拿到的是:
|
||||
|
||||
- 你手工输入的结构化字段
|
||||
- 后端传递的少量静态上下文
|
||||
|
||||
模型现在拿不到:
|
||||
|
||||
- 实时 BGP 事件
|
||||
- 真实告警列表
|
||||
- 数据源健康状态
|
||||
- 自动检索结果
|
||||
- tool calling / skills / 自主取数
|
||||
|
||||
### 2. `situational-awareness/analyze` 还是通用提示词接口
|
||||
|
||||
当前更适合:
|
||||
|
||||
- 测试链路
|
||||
- 测试模型输出风格
|
||||
- 验证不同 provider 是否正常返回
|
||||
|
||||
当前还不适合:
|
||||
|
||||
- 直接当真实态势系统主入口
|
||||
- 让用户手工维护长期分析模板
|
||||
- 代替专用业务研判接口
|
||||
|
||||
### 3. 还没有可验证的真实业务输入注入
|
||||
|
||||
目前最缺的是:
|
||||
|
||||
- 从业务系统自动整理“事实输入”
|
||||
- 再把这些事实喂给 AI
|
||||
|
||||
而不是继续让用户在 Playground 手工输入真实事件摘要。
|
||||
|
||||
## 短期计划
|
||||
|
||||
### Phase A: Playground 收敛为稳定测试台
|
||||
|
||||
目标:
|
||||
|
||||
- 保持 Playground 简洁可用
|
||||
- 不再继续堆“高级参数”
|
||||
|
||||
工作项:
|
||||
|
||||
- 继续微调左侧 `Provider 状态` 与 `测试说明` 的空间策略
|
||||
- 保持 `请求 / 结果` 为单一主工作区
|
||||
- 不引入盲填式高级字段
|
||||
- 统一滚动条、卡片、溢出行为
|
||||
|
||||
完成标准:
|
||||
|
||||
- 笔记本视口下依然可用
|
||||
- 各模块标题可见
|
||||
- 主要阅读区始终是右侧 Tabs
|
||||
|
||||
### Phase B: BGP AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 不再依赖手工填写“观察项”
|
||||
- 让系统自动把真实 BGP 数据注入 AI
|
||||
- 让 BGP 页面逐步从“摘要汇总”升级为“证据驱动的区域态势分析”
|
||||
|
||||
建议实现:
|
||||
|
||||
- 新增专用后端接口,例如:
|
||||
- `POST /api/v1/ai/bgp/brief`
|
||||
- 后端自动读取:
|
||||
- incidents summary
|
||||
- anomalies
|
||||
- recent events
|
||||
- collector coverage summary
|
||||
- 后端将结构化事实注入 `context / observations`
|
||||
- 前端在 BGP 页面增加“生成 AI 简报”
|
||||
|
||||
当前阶段说明:
|
||||
|
||||
- 第一版 `BGP AI 简报` 允许先落地为“值班摘要生成器”
|
||||
- 也就是先把 incidents / anomalies / events / collector coverage 自动注入
|
||||
- 允许模型先做事实摘要、风险归纳、建议动作
|
||||
|
||||
但这不应被视为 Phase B 的最终形态。
|
||||
|
||||
Phase B 后续还需要补齐:
|
||||
|
||||
- prefix geography 证据注入
|
||||
- `iptoasn`
|
||||
- `opengeofeed`
|
||||
- `nro_delegated`
|
||||
- 基于 `affected_regions` 与 prefix geography 的区域聚合
|
||||
- 区分“真实区域热度”与“collector coverage 偏差”
|
||||
- 对高风险 prefix / ASN 给出更明确的国家、城市、运营商归属线索
|
||||
- 让 AI 输出明确回答:
|
||||
- 哪些区域正在异常升温
|
||||
- 哪些结论只是观测站偏差
|
||||
- 当前还缺哪些区域证据
|
||||
|
||||
完成标准:
|
||||
|
||||
- 用户不需要手工录入 BGP 观察项
|
||||
- AI 输出能明确区分“事实”和“研判”
|
||||
- AI 不只是复述总量和最近几条事件,还能利用 prefix geography 与 affected regions 做区域态势判断
|
||||
- 输出中能明确指出:
|
||||
- 高风险区域
|
||||
- 区域证据来源
|
||||
- collector coverage 偏差对判断的影响
|
||||
|
||||
### Phase C: 告警 / 数据源健康 AI 简报
|
||||
|
||||
目标:
|
||||
|
||||
- 复用同样模式,扩展到其他模块
|
||||
|
||||
建议入口:
|
||||
|
||||
- `Alerts` 页面:异常与告警摘要
|
||||
- `DataSources` 页面:采集失败与健康状态总结
|
||||
|
||||
原则:
|
||||
|
||||
- 每个业务页优先做“专用 AI 简报”
|
||||
- 不优先做“万能大聊天框”
|
||||
|
||||
## 中期计划
|
||||
|
||||
### 1. Assessment Layer
|
||||
|
||||
目标:
|
||||
|
||||
- 不只返回自由文本
|
||||
- 返回结构化的 assessment
|
||||
|
||||
建议输出字段:
|
||||
|
||||
- summary
|
||||
- key_risks
|
||||
- evidence
|
||||
- recommendations
|
||||
- confidence
|
||||
- missing_data
|
||||
|
||||
这样后续才能:
|
||||
|
||||
- 持久化
|
||||
- 回看
|
||||
- 对比不同时间的 AI 结论
|
||||
- 在 Earth / Dashboard / BGP 页面稳定展示
|
||||
|
||||
### 2. Evidence-first Runtime
|
||||
|
||||
目标:
|
||||
|
||||
- 所有 AI 分析先取真实数据,再调模型
|
||||
|
||||
原则:
|
||||
|
||||
- 先 evidence
|
||||
- 再 prompt
|
||||
- 最后才是自由生成
|
||||
|
||||
优先要做的不是更强聊天,而是:
|
||||
|
||||
- 更稳定的数据注入
|
||||
- 更一致的事实模板
|
||||
- 更清晰的结果结构
|
||||
|
||||
### 3. 按页面提供专用入口
|
||||
|
||||
目标:
|
||||
|
||||
- 让 AI 成为业务视图的一部分,而不是孤立 playground
|
||||
|
||||
优先顺序建议:
|
||||
|
||||
1. `BGP` AI 简报
|
||||
2. `Alerts` AI 简报
|
||||
3. `DataSources` 健康研判
|
||||
4. `Dashboard` 总览总结
|
||||
|
||||
## 长期计划
|
||||
|
||||
### 1. Tool Calling / Agent Runtime
|
||||
|
||||
只有在以下基础稳定后再推进:
|
||||
|
||||
- 数据源健康信号稳定
|
||||
- BGP / Alerts / Datasource evidence 注入稳定
|
||||
- assessment 结构稳定
|
||||
|
||||
长期可做能力:
|
||||
|
||||
- AI 调用受控工具查询业务数据
|
||||
- AI 调用检索/web search 做外部验证
|
||||
- AI 生成建议而不是直接修改系统
|
||||
- 审核后触发受控动作
|
||||
|
||||
### 2. 受控动作与闭环
|
||||
|
||||
潜在方向:
|
||||
|
||||
- 根据健康异常生成修复建议
|
||||
- 根据态势变化生成处理建议
|
||||
- 进入 review queue
|
||||
- 审批后执行
|
||||
- 验证结果并形成闭环
|
||||
|
||||
### 3. 多模块统一 AI 体验
|
||||
|
||||
长期目标不是一个孤立 Playground,而是:
|
||||
|
||||
- 每个业务页都有自己的 AI 入口
|
||||
- 共享统一的 backend AI facade
|
||||
- 共享统一的 assessment 结构
|
||||
- 共享统一的 evidence 注入与审计链路
|
||||
|
||||
## 设计决策总结
|
||||
|
||||
### 为什么保留 `aiprovider`
|
||||
|
||||
因为它已经很好地承担了:
|
||||
|
||||
- provider 适配
|
||||
- 协议兼容
|
||||
- service token 边界
|
||||
- 独立重启与部署
|
||||
|
||||
因此短期内不建议把它并回 `backend`。
|
||||
|
||||
### 为什么 Playground 不做成万能聊天页
|
||||
|
||||
因为当前更需要的是:
|
||||
|
||||
- 稳定测试链路
|
||||
- 可验证业务输入
|
||||
- 专用分析入口
|
||||
|
||||
而不是一个泛化但没有真实数据支撑的聊天框。
|
||||
|
||||
### 为什么优先做专用 AI 简报
|
||||
|
||||
因为:
|
||||
|
||||
- 数据可以自动注入
|
||||
- 用户心智更清晰
|
||||
- 输出更容易结构化
|
||||
- 更容易校验事实与研判是否一致
|
||||
|
||||
## 下一步建议
|
||||
|
||||
按优先级建议接下来这样做:
|
||||
|
||||
1. 稳住 `Playground` 当前布局,不再大幅重做
|
||||
2. 在 `BGP` 页面新增专用 “AI 简报” 入口
|
||||
3. 后端新增 `BGP brief` 专用接口,自动注入真实数据
|
||||
4. 补齐 `BGP brief` 的区域态势证据层
|
||||
5. 把 AI 输出逐步从自由文本升级为结构化 assessment
|
||||
|
||||
### BGP Brief 后续子项
|
||||
|
||||
为避免把“已有 AI 简报”误判成“区域分析已完成”,这里单独记录 `BGP brief` 的后续 backlog:
|
||||
|
||||
1. 把高风险 prefix 命中的 `iptoasn / opengeofeed / nro_delegated` 结果注入 brief context
|
||||
2. 按国家/城市聚合 active incidents、anomalies、affected prefixes,生成区域热点事实层
|
||||
3. 把 collector coverage 与区域热点并排注入,避免模型把观测偏差误判成区域风险
|
||||
4. 对高风险 ASN / prefix 追加归属线索,如国家、城市、可能运营商或注册区域
|
||||
5. 在输出结构中单独增加:
|
||||
- 区域态势
|
||||
- 证据来源
|
||||
- 观测偏差说明
|
||||
- 缺失区域证据
|
||||
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
1015
docs/plans/ue5-mvp-fused-plan.md
Normal file
File diff suppressed because it is too large
Load Diff
26
docs/technical/README.md
Normal file
26
docs/technical/README.md
Normal file
@@ -0,0 +1,26 @@
|
||||
# Technical Docs
|
||||
|
||||
这里放“当前实现和当前结构”的文档,重点回答:
|
||||
|
||||
- 现在代码是怎么组织的
|
||||
- 当前入口在哪
|
||||
- 状态和组件如何工作
|
||||
- 后续改动应该沿着哪条实现边界继续走
|
||||
|
||||
适合放入这里的内容:
|
||||
|
||||
- 前端上下文
|
||||
- Earth 前端结构
|
||||
- 后端运行控制
|
||||
- collector 现状
|
||||
- 采集格式约定
|
||||
|
||||
不适合放入这里的内容:
|
||||
|
||||
- 尚未完成的 roadmap
|
||||
- 未来迭代方案
|
||||
- 大范围重构计划
|
||||
|
||||
这些应放入:
|
||||
|
||||
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)
|
||||
@@ -31,22 +31,40 @@ The recommended default is:
|
||||
- timeout and lightweight retry
|
||||
- request tracing via `X-Request-ID`
|
||||
|
||||
This now follows an OpenClaw-like seam:
|
||||
|
||||
- `AI_PROVIDER` identifies the vendor or logical provider
|
||||
- `AI_PROVIDER_API` identifies the wire adapter
|
||||
|
||||
That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field.
|
||||
|
||||
## Supported Providers
|
||||
|
||||
`aiprovider` currently supports:
|
||||
`aiprovider` currently supports these provider identities:
|
||||
|
||||
- `openai`
|
||||
- `openai_compatible`
|
||||
- `anthropic`
|
||||
- `minimax`
|
||||
- `ollama`
|
||||
|
||||
Supported request adapters:
|
||||
|
||||
- `openai-completions`
|
||||
- `anthropic-messages`
|
||||
- `ollama-generate`
|
||||
|
||||
Backward-compatible aliases still accepted:
|
||||
|
||||
- `openai_compatible`
|
||||
- `anthropic_compatible`
|
||||
- `claude_compatible`
|
||||
- `ollama`
|
||||
|
||||
Provider mapping:
|
||||
|
||||
- `vLLM`, `LM Studio`, `One API`: `openai_compatible`
|
||||
- `MiniMax`, Claude-compatible gateways: `claude_compatible`
|
||||
- `Ollama`: `ollama`
|
||||
- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions`
|
||||
- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages`
|
||||
- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate`
|
||||
|
||||
## API Surfaces
|
||||
|
||||
@@ -137,9 +155,13 @@ Both backend and `aiprovider` return the same payload shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "openai_compatible",
|
||||
"model": "gpt-4o-mini",
|
||||
"provider": "minimax",
|
||||
"api": "anthropic-messages",
|
||||
"model": "MiniMax-M2.7",
|
||||
"content": "1) 态势摘要 ...",
|
||||
"content_blocks": [],
|
||||
"text_blocks": [],
|
||||
"thinking_blocks": [],
|
||||
"raw_response": {}
|
||||
}
|
||||
```
|
||||
@@ -189,17 +211,37 @@ AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上
|
||||
### OpenAI-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=openai_compatible
|
||||
AI_PROVIDER=openai
|
||||
AI_PROVIDER_API=openai-completions
|
||||
AI_BASE_URL=http://127.0.0.1:8001/v1
|
||||
AI_API_KEY=local-key
|
||||
AI_MODEL=your-local-model
|
||||
```
|
||||
|
||||
### Claude-compatible example
|
||||
### MiniMax CN example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=claude_compatible
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
|
||||
AI_PROVIDER=minimax
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://api.minimaxi.com/anthropic
|
||||
AI_API_KEY=sk-cp-xxxxx
|
||||
AI_MODEL=MiniMax-M2.7
|
||||
AI_MAX_TOKENS=1200
|
||||
AI_ANTHROPIC_VERSION=2023-06-01
|
||||
```
|
||||
|
||||
MiniMax note:
|
||||
|
||||
- This follows the same Anthropic Messages request shape as the official MiniMax examples.
|
||||
- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object.
|
||||
- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior.
|
||||
|
||||
### Anthropic-compatible example
|
||||
|
||||
```env
|
||||
AI_PROVIDER=anthropic
|
||||
AI_PROVIDER_API=anthropic-messages
|
||||
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com/anthropic
|
||||
AI_API_KEY=your_api_key
|
||||
AI_MODEL=your-model
|
||||
AI_MAX_TOKENS=1200
|
||||
@@ -210,6 +252,7 @@ AI_ANTHROPIC_VERSION=2023-06-01
|
||||
|
||||
```env
|
||||
AI_PROVIDER=ollama
|
||||
AI_PROVIDER_API=ollama-generate
|
||||
AI_BASE_URL=http://127.0.0.1:11434
|
||||
AI_API_KEY=
|
||||
AI_MODEL=qwen2.5:7b
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user