Compare commits

..

67 Commits

Author SHA1 Message Date
opencode
8f67d8d108 feat: add ue_scene websocket integration 2026-05-09 15:51:46 +08:00
linkong
61baee00f6 Merge pull request 'dev' (#8) from dev into main
Reviewed-on: #8
2026-04-21 21:30:45 +00:00
rayd1o
0082cf3fbd release: bump version to 0.33.0 2026-04-22 05:28:54 +08:00
rayd1o
3ae4acdff8 release: bump version to 0.32.0 2026-04-22 04:41:39 +08:00
rayd1o
437efc848c release: bump version to 0.31.3 2026-04-22 03:52:09 +08:00
rayd1o
003a46ac30 release: bump version to 0.31.2 2026-04-21 23:50:35 +08:00
rayd1o
4b0be4cb76 release: bump version to 0.31.1 2026-04-21 22:49:39 +08:00
linkong
b7647379de release: bump version to 0.31.0 2026-04-21 18:35:40 +08:00
linkong
0f89372d71 release: bump version to 0.30.0 2026-04-21 12:28:04 +08:00
linkong
2b0d4cfc49 release: bump version to 0.29.2 2026-04-21 10:43:48 +08:00
rayd1o
e6d0332fba release: bump version to 0.29.1 2026-04-20 22:12:19 +08:00
linkong
fe45a99cbd release: bump version to 0.29.0 2026-04-20 17:43:59 +08:00
linkong
ae77b06c3c docs: expand earth celestial background implementation plan 2026-04-20 16:29:03 +08:00
linkong
b5dd4f12f8 release: bump version to 0.28.2 2026-04-20 16:00:54 +08:00
linkong
75cb214f23 release: bump version to 0.28.1 2026-04-20 15:14:53 +08:00
linkong
4c21973197 release: bump version to 0.28.0 2026-04-20 14:10:27 +08:00
linkong
51ae5e6ec9 release: bump version to 0.27.8 2026-04-20 12:03:09 +08:00
linkong
1cf1f32ddd feat: refine earth hud panel behaviors and news board 2026-04-20 11:23:05 +08:00
linkong
8f3ab88743 release: bump version to 0.27.7 2026-04-16 10:04:14 +08:00
rayd1o
f8b43a995b release: bump version to 0.27.6 2026-04-15 07:37:41 +08:00
linkong
d9adaf4134 release: bump version to 0.27.5 2026-04-14 17:34:04 +08:00
linkong
40e51d5b20 release: bump version to 0.27.4
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 13:11:39 +08:00
linkong
93c1c1e550 release: bump version to 0.27.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 12:52:05 +08:00
linkong
48eb13b993 release: bump version to 0.27.2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 10:55:04 +08:00
linkong
11179e7e67 release: bump version to 0.27.1 2026-04-14 10:22:05 +08:00
rayd1o
07e26d6d5a chore: add cleanup and release skills for claude and codex
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 07:46:19 +08:00
linkong
620190819b Merge pull request 'dev' (#5) from dev into main
Reviewed-on: #5
2026-04-13 23:41:04 +00:00
rayd1o
7cd29cf9c0 release: bump version to 0.27.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 07:37:45 +08:00
rayd1o
2ee4773f4f release: bump version to 0.26.2 2026-04-12 09:27:01 +08:00
rayd1o
b1d0624061 release: bump version to 0.26.1 2026-04-12 05:20:23 +08:00
rayd1o
812c825dc6 release: bump version to 0.26.0 2026-04-12 04:36:38 +08:00
rayd1o
a359d94127 refactor: clean up earth brand component wiring 2026-04-11 03:58:28 +08:00
rayd1o
c92be9c054 chore: remove unused earth icon assets 2026-04-11 03:46:59 +08:00
rayd1o
10e2bae8c2 release: bump version to 0.25.3 2026-04-11 03:42:35 +08:00
rayd1o
60ed88b609 release: bump version to 0.25.2 2026-04-10 23:43:56 +08:00
linkong
e85a9fc614 fix: refine earth settings modal and hud interactions 2026-04-10 22:53:09 +08:00
linkong
a2210f0f78 release: bump version to 0.25.1 2026-04-10 16:19:57 +08:00
linkong
62ad09e816 release: bump version to 0.25.0 2026-04-10 16:04:50 +08:00
linkong
89a71e6f29 feat: ship persistent ai playground and alerts foundation 2026-04-10 15:57:34 +08:00
rayd1o
60f5ff9bab fix: move bgp brief markdown into a modal workspace 2026-04-10 03:57:13 +08:00
rayd1o
fbb6adfbf5 docs: formalize release workflow and frontend layout guardrails 2026-04-10 03:08:39 +08:00
rayd1o
749e6e76b6 fix: optimize visualization hot paths and refine bgp brief workspace 2026-04-10 02:12:29 +08:00
rayd1o
83839b8b11 fix: ship persistent bgp ai briefs and optimize bgp queries 2026-04-10 00:33:01 +08:00
rayd1o
ed898aef9c docs: add docker compose and buildx upgrade guide
- add a standalone docs guide for reinstalling Docker, Compose, and Buildx from the official Ubuntu repository
- refine planet.sh compose/build diagnostics so docker compose is the primary wording and missing buildx / fallback cases report more accurately
- keep the startup failure output aligned with the existing terminal tone while reducing misleading compose-version messaging
2026-04-09 23:08:31 +08:00
linkong
abe0b5c11b fix: polish ai provider rebuild feedback 2026-04-09 16:52:21 +08:00
linkong
306ba7f850 fix: expand playground diagnostics and compose fallback 2026-04-09 16:38:55 +08:00
linkong
39f90bd575 fix: polish playground routing and bundle splits 2026-04-09 15:23:10 +08:00
rayd1o
c4ea918fac fix: refine earth hud structure and bun startup flow
- refactor the /earth HUD into class-first CSS layers with dedicated base, hud, and toolbar responsibilities
- clean up Earth markup and runtime DOM hooks so shared panel, toolbar, tooltip, and status classes stay consistent after dynamic updates
- keep HUD scaling configurable through extracted constants and remove leftover legacy panel/toolbar styling paths
- make planet.sh self-bootstrap Bun and uv by prepending local runtime bins to PATH and auto-installing missing tools in fresh environments
- document Bun as the frontend package manager of record and sync repository version metadata to 0.24.1
2026-04-09 02:12:22 +08:00
rayd1o
34d94a6b6b Redirect login success to admin 2026-04-09 00:37:02 +08:00
rayd1o
ef65acd49c Add AI playground and frontend layout guidelines 2026-04-09 00:35:38 +08:00
linkong
5639546990 fix: stabilize bgp overview table viewport 2026-04-08 17:48:32 +08:00
linkong
c8fe8cad59 fix: refine planet startup script UX 2026-04-08 16:54:45 +08:00
linkong
d395769df6 docs: add agent runtime planning docs 2026-04-08 12:49:49 +08:00
linkong
8bd9d34376 feat: configure collector endpoints and health plan 2026-04-08 10:15:33 +08:00
linkong
2d43263b9e fix: tighten datasource trigger guards 2026-04-08 09:33:48 +08:00
rayd1o
2da6ed166b Bump version to 0.23.2 2026-04-08 03:24:08 +08:00
rayd1o
da587398d9 Refine datasource trigger flow and CLI output 2026-04-08 03:22:08 +08:00
rayd1o
f5308340af Add force rerun recovery and polish CLI startup 2026-04-08 02:49:29 +08:00
rayd1o
981617ee80 Bump version to 0.23.1 2026-04-07 23:51:44 +08:00
rayd1o
7abf391c74 Harden startup retry and recovery flow 2026-04-07 23:25:13 +08:00
linkong
f12719914d feat: align aiprovider with adapter-based compatibility 2026-04-07 17:54:44 +08:00
linkong
f67d6bde60 Merge pull request 'codex/aiprovider-foundation' (#4) from codex/aiprovider-foundation into main
Reviewed-on: #4
2026-04-07 09:35:48 +00:00
linkong
bc90e00e25 Merge branch 'codex/aiprovider-foundation' into dev 2026-04-07 17:33:37 +08:00
linkong
36672e4c53 Merge pull request 'dev' (#3) from dev into main
Reviewed-on: #3
2026-03-25 09:25:38 +00:00
linkong
506402ce16 Merge pull request 'dev' (#2) from dev into main
Reviewed-on: #2
2026-03-20 21:17:30 +00:00
linkong
9d135bf2e1 revert 49a9c33836
revert feat(earth): toolbar and zoom improvements

- Add box-sizing/padding normalization to toolbar buttons
- Remove zoom slider, implement click/hold zoom behavior (+/- buttons)
- Add 10% step on click, 1% continuous on hold
- Fix satellite init: show satellite points immediately, delay trail visibility
- Fix breathing effect: faster pulse, wider opacity range
- Add toggle-cables functionality with visibility state
- Initialize satellites and cables as visible by default
2026-03-20 21:16:45 +00:00
linkong
49a9c33836 feat(earth): toolbar and zoom improvements
- Add box-sizing/padding normalization to toolbar buttons
- Remove zoom slider, implement click/hold zoom behavior (+/- buttons)
- Add 10% step on click, 1% continuous on hold
- Fix satellite init: show satellite points immediately, delay trail visibility
- Fix breathing effect: faster pulse, wider opacity range
- Add toggle-cables functionality with visibility state
- Initialize satellites and cables as visible by default
2026-03-20 17:13:02 +08:00
200 changed files with 36266 additions and 3871 deletions

120
.claude/commands/cleanup.md Normal file
View File

@@ -0,0 +1,120 @@
---
description: 审查当前工作区未提交代码中的垃圾代码,并在不影响逻辑的前提下自动清理
argument-hint: 可选:指定要检查的文件或目录(默认检查所有未提交修改)
allowed-tools: ["Read", "Edit", "Bash", "Grep", "Glob"]
---
# /cleanup — 垃圾代码审查与清理
分析当前工作区git diff中的未提交代码找出并修复常见垃圾代码**不得改变任何运行逻辑**。
## 检查范围
`$ARGUMENTS` 非空,则只检查指定文件/目录;否则检查所有未提交修改(`git diff HEAD`)。
## 审查清单
按优先级检查以下问题(只报告在本次 diff 中**新增或修改**的代码里存在的问题):
### 1. 重复逻辑 (Duplicate Logic)
- 完全相同或高度相似的代码块在多处出现
- 同一函数/方法被多个地方各自实现,已有公共版本未被复用
- 相同的 DOM 查询、正则、模板字符串在同一文件重复
### 2. Magic Numbers / Magic Strings
- 裸数字直接参与计算(如偏移量、时间、尺寸、阈值),没有命名常量
- 硬编码字符串(如 id 名、状态值、URL 片段)散落在逻辑中
- 例外:`0`, `1`, `-1`, `100`, `""` 等语义明确的惯用值不算
### 3. 命名问题
- 含义不明的缩写变量(如 `or_`, `tmp2`, `x2`
- 命名与实际用途不符
- 同一概念在不同地方用不同名字表达
### 4. 死代码 / 无效代码
- 注释掉的旧代码块3行以上
- 声明后从未使用的变量/参数/导入
- 永远不会执行的条件分支
### 5. 代码风格问题
- 尾部空白字符trailing whitespace
- 同一文件内风格不一致(如混用单双引号、缩进不统一)
- 空行使用不一致(连续多个空行等)
### 6. 其他常见问题
- 私有辅助函数应被 export 但没有,导致调用方重复实现
- 类型/接口重复定义
- 过于冗长的条件表达式可以简化(不改逻辑)
## 执行步骤
### Step 1 — 获取待检查文件列表
```bash
# 无参数时:获取所有未提交修改
git diff HEAD --name-only
# 有参数时:用 $ARGUMENTS 过滤
```
### Step 2 — 逐文件阅读并分析
- 用 Read 工具读取完整文件(不只读 diff
- 对照审查清单,记录每个问题:文件名、行号、问题类型、建议修复方式
### Step 3 — 报告问题清单
在修改前,先以列表形式输出所有发现的问题:
```
发现 N 个问题:
[文件] js/foo.js
· L34, L78: 重复逻辑 — 两处都实现了相同的 DOM 查询,可提取到 getPanel()
· L91: Magic number — 硬编码 14 作为偏移量,应命名为 TOOLTIP_OFFSET
[文件] js/bar.js
· L12: 命名问题 — 变量 `or_` 语义不明,应命名为 outerR/outerG/outerB
...
```
如果没有发现问题,直接输出"未发现垃圾代码,当前代码质量良好。"并停止。
### Step 4 — 执行修复
对每个问题,使用 Edit 工具进行**最小化修改**
- **重复逻辑**:提取为共享常量/函数,更新所有调用点
- **Magic number**:在文件顶部或逻辑附近声明 `const NAME = value`,替换所有引用
- **命名问题**:重命名变量,更新所有使用处
- **死代码**:直接删除
- **尾部空白/风格**:修正
- **未 export 的函数**:添加 `export`,在调用方改为导入(不重复实现)
**修复原则:**
- 只改在审查清单中发现的问题,不做额外优化
- 每次 Edit 只修改确实有问题的行,保持 diff 最小
- 改完后用 `grep` 验证旧的坏代码已消失
### Step 5 — 输出总结
```
清理完成:
修复了 N 个问题:
✓ earth.js — 提取重复 vertexShader 为 ATMOS_VERTEX_SHADER 常量
✓ main.js — 提取 TOOLTIP_CURSOR_OFFSET = 144处引用
✓ controls.js — export updateLayerButtonState移除 main.js 中的重复实现
...
未修改的问题(需人工确认):
! foo.js L45 — 注释代码块较长,建议手动确认是否可删除
```
## 约束
- **禁止**改变函数签名、接口定义、导出 API除非问题正是私有函数应被 export
- **禁止**添加新功能、新抽象、新参数
- **禁止**修改注释内容(只删除注释掉的死代码)
- **禁止**修改测试文件逻辑
- 如果一个 Magic number 的语义不完全确定,**跳过**,在总结中标记为"需人工确认"

146
.claude/commands/release.md Normal file
View File

@@ -0,0 +1,146 @@
---
description: 发版工作流:根据变更类型决定版本号,更新所有版本文件和 changelog运行验证commit 并 push
argument-hint: 可选feature | bugfix | 或直接描述本次发布内容
allowed-tools: ["Read", "Edit", "Bash", "Glob", "Grep"]
---
# /release — Planet 发版工作流
## 版本号规则
| 变更类型 | 版本跳动 | 适用场景 |
|---------|---------|---------|
| `feature` | `+0.1.0` | 纯新功能,无 bugfix |
| `improvement` | `+0.0.1` | UI 调整、小功能增强、bugfix 混合,或以 UI/体验改进为主的迭代 |
| `bugfix` | `+0.0.1` | 纯 bug 修复,无新功能 |
| `docs` / `maintenance` / `refactor` | 默认不发版,除非用户明确要求 |
意图混合时以用户明确描述为准bugfix + 小 feature 混合默认判定为 `improvement``+0.0.1`)。
## 必须同步更新的文件
使用 `git rev-parse --show-toplevel` 获取仓库根目录,以下路径均相对于根目录:
- `VERSION`
- `frontend/package.json``"version"` 字段)
- `pyproject.toml``version =` 字段)
- `uv.lock`**不要手动编辑**,通过 `uv lock` 重新生成)
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## 执行步骤
### Step 1 — 环境检查
```bash
git branch --show-current # 确认在 dev 分支
git status --short # 检查是否有无关的未暂存修改
cat VERSION # 读取当前版本
```
若当前**不在 `dev` 分支**,停下来告知用户,不要继续。
若存在无关的未暂存修改,列出并询问用户是否一并提交,或先 stash。
### Step 2 — 确定发版类型与新版本号
-`$ARGUMENTS` 提供了明确类型(`feature` / `bugfix`),直接使用
- 否则根据当前 `git diff HEAD``git log` 推断
- 计算新版本号(例:`0.26.2` → bugfix → `0.26.3`
- **先输出发版计划供用户确认**
```
发版计划:
类型bugfix
版本0.26.2 → 0.26.3
分支dev
将更新VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
### Step 3 — 更新版本号文件
按顺序更新(每步用 Edit 工具,精确替换,不要重写整个文件):
1. `VERSION` — 直接替换全部内容为新版本号
2. `frontend/package.json` — 替换 `"version": "x.x.x"`
3. `pyproject.toml` — 替换 `version = "x.x.x"`
4. 运行 `uv lock` 重新生成 `uv.lock`(在仓库根目录下执行)
### Step 4 — 更新 CHANGELOG.md
在文件顶部插入新条目,格式:
```markdown
## [x.x.x] — YYYY-MM-DD
### ✨ Features / 🐛 Fixes / 🔧 Improvements
- ...(只列高信号条目,最多 5 条)
- ...
---
```
日期使用 `date +%Y-%m-%d` 获取今天的日期。
### Step 5 — 更新 docs/version-history.md
- 更新文件头部的"当前开发版本"字段
- 在时间线表格顶部插入新行:`| vx.x.x | YYYY-MM-DD | 一句话摘要 |`
### Step 6 — 验证
针对本次变更范围做最小验证:
- Python 文件有修改:`python3 -m py_compile <changed_files>`
- Frontend 文件有修改:运行项目标准检查(若无则跳过并说明)
- 版本号一致性检查:用 grep 确认 VERSION、package.json、pyproject.toml 中的版本号完全一致
```bash
grep -h "version" VERSION frontend/package.json pyproject.toml
```
### Step 7 — 提交前预览
展示将要提交的文件列表:
```bash
git diff --stat HEAD
```
再次确认所有必须文件都在变更列表中,**不包含**非预期文件(如调试文件、.env 等)。
### Step 8 — Commit & Push用户确认后
```bash
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# 若有代码变更也一并 stage
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
commit message 固定格式:`release: bump version to x.x.x`
### Step 9 — 完成确认
输出摘要:
```
✓ 版本号已更新0.26.2 → 0.26.3
✓ CHANGELOG 已更新
✓ version-history 已更新
✓ uv.lock 已重新生成
✓ 验证通过
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ 已 push 到 origin/dev
```
## 注意事项
- `uv.lock` 只能通过 `uv lock` 生成,绝不手动编辑
- 发版 commit 只包含版本文件 + 本次功能代码,不混入无关改动
- 若环境中 `uv` 不可用,说明原因并跳过 lockfile 更新,提醒用户手动运行

View File

@@ -0,0 +1,124 @@
---
name: cleanup
description: Use when the user asks to clean up, lint, or review uncommitted code for common code smells — duplicate logic, magic numbers, unclear naming, dead code, style inconsistencies. Fixes issues without changing any runtime behavior.
---
# Cleanup
Review and fix code quality issues in the current working tree without altering any logic or behavior.
## When To Use
- The user asks to clean up, tidy, or lint uncommitted changes
- The user wants a code smell review before releasing or committing
- The user mentions magic numbers, duplicate logic, dead code, or naming issues
Do not refactor architecture, add features, or change behavior.
## Scope
If the user specifies a file or directory, check only that. Otherwise check all uncommitted changes (`git diff HEAD`).
Only report issues present in **newly added or modified** lines of this diff — do not audit unchanged code.
## Checklist
### 1. Duplicate Logic
- Identical or near-identical code blocks appearing in multiple places
- A function/helper that already exists but is re-implemented elsewhere instead of being reused
- Repeated DOM queries, regex literals, or template strings within the same file
### 2. Magic Numbers / Magic Strings
- Bare numeric literals used in calculations (offsets, timeouts, sizes, thresholds) without a named constant
- Hardcoded strings (IDs, status values, URL fragments) scattered through logic
- Exceptions: `0`, `1`, `-1`, `100`, `""` and other idiomatically clear values are fine
### 3. Naming Issues
- Cryptic abbreviations (`or_`, `tmp2`, `x2`)
- Names that do not match actual behavior
- The same concept referred to by different names in different places
### 4. Dead Code
- Commented-out code blocks (3+ lines)
- Variables, parameters, or imports declared but never used
- Branches that can never execute
### 5. Style Inconsistencies
- Trailing whitespace
- Mixed quote styles or indentation within the same file
- Inconsistent blank-line usage (multiple consecutive blank lines, etc.)
### 6. Other
- Private helper functions that should be exported but are not, causing callers to duplicate the implementation
- Overly verbose conditions that can be simplified without changing logic
## Steps
### Step 1 — Get the file list
```bash
git diff HEAD --name-only
```
Filter to the user-specified path if one was provided.
### Step 2 — Read and analyze each file
Read the full file (not just the diff) with the Read tool. For each file, record every issue found: filename, line number, category, and suggested fix.
### Step 3 — Report findings before touching anything
Print a structured list:
```
Found N issues:
[file] js/foo.js
· L34, L78: Duplicate logic — same DOM query implemented twice; extract to getPanel()
· L91: Magic number — bare 14 used as pixel offset; name it TOOLTIP_OFFSET
[file] js/bar.js
· L12: Naming — variable `or_` is unclear; rename to outerR, outerG, outerB
...
```
If no issues are found, output "No code smells detected. Code quality looks good." and stop.
### Step 4 — Fix each issue
Use the Edit tool for **minimal, targeted changes**:
- **Duplicate logic**: extract to a shared constant or function; update all call sites
- **Magic number/string**: declare `const NAME = value` near the top of the relevant scope; replace all usages
- **Naming**: rename the variable/function; update all references
- **Dead code**: delete it
- **Trailing whitespace / style**: fix in place
- **Unexported helper**: add `export`; update callers to import instead of re-implementing
Principles:
- Only fix issues identified in the checklist — no extra improvements
- Keep each Edit as small as possible
- After fixing, verify the old bad pattern is gone with grep
### Step 5 — Summary
```
Cleanup complete:
Fixed N issues:
✓ earth.js — extracted duplicate vertexShader into ATMOS_VERTEX_SHADER constant
✓ main.js — extracted TOOLTIP_CURSOR_OFFSET = 14 (4 references updated)
✓ controls.js — exported updateLayerButtonState; removed duplicate implementation in main.js
...
Skipped (needs manual review):
! foo.js L45 — large commented-out block; confirm it is safe to delete
```
## Constraints
- **Do not** change function signatures, exported interfaces, or public APIs (unless the issue is a missing export)
- **Do not** add new features, abstractions, or parameters
- **Do not** rewrite comments (only delete commented-out dead code)
- **Do not** touch test file logic
- If a magic number's intent is uncertain, skip it and flag it in the summary

View File

@@ -0,0 +1,157 @@
---
name: release
description: Use when the user asks to release, bump version, update changelog/version files, or commit/push a repository release for the Planet repo. Determines version bump type from changes, updates all required version-bearing files, updates changelog and version-history, runs minimal validation, then commits, tags, and pushes.
---
# Release Workflow
Use this skill for release-oriented work in this repository.
## When To Use
- The user asks to `发版`
- The user asks to bump a version
- The user asks to update `CHANGELOG`, `version-history`, or version files as part of a release
- The user asks to commit/push a release or a publishable bugfix/feature bundle
Do not use this skill for ordinary commits that are not being released.
## Versioning Rules
- `feature` -> bump `+0.1.0`
- `bugfix` -> bump `+0.0.1`
- `docs`, `maintenance`, and `refactor` do not bump by default unless the user explicitly wants a release
When intent is mixed, prefer the user's stated release intent.
## Required Files
Use `git rev-parse --show-toplevel` to get the repo root. All paths are relative to it:
- `VERSION`
- `frontend/package.json` (`"version"` field)
- `pyproject.toml` (`version =` field)
- `uv.lock` (**never edit manually** — regenerate by running `uv lock`)
- `docs/CHANGELOG.md`
- `docs/version-history.md`
## Workflow
### Step 1 — Environment check
```bash
git branch --show-current # must be on dev
git status --short # check for unrelated uncommitted changes
cat VERSION # read current version
```
If not on `dev`, stop and tell the user. Do not proceed.
If unrelated uncommitted changes exist, list them and ask the user whether to include them or stash first.
### Step 2 — Determine release type and next version
- If the user provided an explicit type (`feature` / `bugfix`), use it
- Otherwise infer from `git diff HEAD` and recent `git log`
- Compute the next version (e.g. `0.26.2` → bugfix → `0.26.3`)
- **Show the release plan before making any changes:**
```
Release plan:
Type: bugfix
Version: 0.26.2 → 0.26.3
Branch: dev
Will update: VERSION, frontend/package.json, pyproject.toml, uv.lock, CHANGELOG.md, version-history.md
```
### Step 3 — Update version files
Update in order (use Edit for precise replacement, never rewrite whole files):
1. `VERSION` — replace entire content with new version string
2. `frontend/package.json` — replace `"version": "x.x.x"` line
3. `pyproject.toml` — replace `version = "x.x.x"` line
4. Run `uv lock` at repo root to regenerate `uv.lock`
### Step 4 — Update CHANGELOG.md
Insert a new entry at the top of the file:
```markdown
## x.x.x
Released: YYYY-MM-DD
### Highlights
- ...
### Added / Fixed / Improved
- ... (high-signal items only, max 5)
---
```
Get today's date with `date +%Y-%m-%d`.
### Step 5 — Update docs/version-history.md
- Update the "current dev version" field in the file header
- Insert a new row at the top of the timeline table: `| vx.x.x | YYYY-MM-DD | one-line summary |`
### Step 6 — Validate
Run the smallest relevant validation for the changes in scope:
- Python files changed: `python3 -m py_compile <changed_files>`
- Frontend files changed: run the project-standard check if available; otherwise skip and say so
- Version consistency: confirm VERSION, package.json, pyproject.toml, and uv.lock all show the same version
```bash
grep -h "version" VERSION frontend/package.json pyproject.toml
```
### Step 7 — Pre-commit preview
Show what will be committed:
```bash
git diff --stat HEAD
```
Confirm all required files are present and no unexpected files (debug files, `.env`, etc.) are included.
### Step 8 — Commit, tag, and push
```bash
git add VERSION frontend/package.json pyproject.toml uv.lock docs/CHANGELOG.md docs/version-history.md
# also stage any code changes included in this release
git add <code_files>
git commit -m "release: bump version to x.x.x"
git tag vx.x.x
git push origin dev
git push origin vx.x.x
```
Commit message format is fixed: `release: bump version to x.x.x`
### Step 9 — Completion summary
```
✓ Version bumped: 0.26.2 → 0.26.3
✓ CHANGELOG updated
✓ version-history updated
✓ uv.lock regenerated
✓ Validation passed
✓ commit: release: bump version to 0.26.3
✓ tag: v0.26.3
✓ Pushed to origin/dev
```
## Notes
- `uv.lock` must only be updated by running `uv lock`, never manually
- The release commit should include only version files + the code for this release — no unrelated changes
- If `uv` is unavailable in the environment, say so explicitly and remind the user to run it manually

5
.gitignore vendored
View File

@@ -145,3 +145,8 @@ docs/.venv/
*.temp
tmp/
temp/
# ----------------------
# Runtime Data
# ----------------------
data/ai/bgp-briefs/

View File

@@ -102,6 +102,13 @@
| Axios | HTTP 客户端 |
| Socket.io-client | WebSocket 客户端 |
| ECharts | 统计图表 |
| Bun | 前端包管理与脚本运行 |
前端工程统一使用 Bun
- 安装依赖使用 `bun install`
- 运行脚本使用 `bun run <script>`
- 不使用 `npm``pnpm``yarn`
### 虚幻引擎客户端
@@ -205,10 +212,47 @@
./planet.sh health
```
前端命令约定:
```bash
cd frontend
bun install
bun run dev
bun run build
```
不要使用 `npm run ...`,避免在 WSL/Windows 混合环境里触发 `cmd.exe` 路径兼容问题。
## API 文档
启动服务后访问: `http://localhost:8000/docs`
## 启动容错参数
`planet.sh` 现在为依赖安装、数据库、AI Provider 启动加入了有限次重试,并会在数据库与 `aiprovider` 启动后额外等待 Docker healthcheck。
可通过环境变量临时调整:
```bash
# 例: 放宽 AI Provider 与数据库在网络抖动下的自愈次数
AI_PROVIDER_START_MAX_RETRIES=5 \
AI_PROVIDER_RETRY_INTERVAL=10 \
DATABASE_START_MAX_RETRIES=5 \
DATABASE_RETRY_INTERVAL=10 \
./planet.sh restart
```
常用参数:
- `DEPENDENCY_INSTALL_MAX_RETRIES` / `DEPENDENCY_INSTALL_RETRY_INTERVAL`: 控制 `uv sync``bun install` 的重试次数与间隔,默认 `3` 次、`5`
- `DATABASE_START_MAX_RETRIES` / `DATABASE_RETRY_INTERVAL`: 控制 `postgres``redis` 的启动/重启与健康检查自愈,默认 `3` 次、`5`
- `AI_PROVIDER_START_MAX_RETRIES` / `AI_PROVIDER_RETRY_INTERVAL`: 控制 `aiprovider` 的构建/启动与容器重启自愈,默认 `3` 次、`5`
- `BACKEND_MAX_RETRIES`: 控制后端进程启动重试次数,默认 `3`
- `FRONTEND_MAX_RETRIES`: 控制前端 dev server 启动重试次数,默认 `3`
- `BACKEND_HEALTH_CHECK_ATTEMPTS` / `BACKEND_HEALTH_CHECK_INTERVAL`: 控制后端 HTTP 健康检查等待次数与间隔,默认 `10` 次、`2`
- `FRONTEND_HEALTH_CHECK_ATTEMPTS` / `FRONTEND_HEALTH_CHECK_INTERVAL`: 控制前端 HTTP 可访问检查等待次数与间隔,默认 `10` 次、`2`
- `AI_PROVIDER_HEALTH_CHECK_ATTEMPTS` / `AI_PROVIDER_HEALTH_CHECK_INTERVAL`: 控制 `aiprovider` HTTP 健康检查等待次数与间隔,默认 `10` 次、`2`
## AI 接口预留
项目现在采用“两层”设计:
@@ -284,8 +328,25 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
详细文档:
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
- [aiprovider/README.md](/home/ray/dev/linkong/planet/aiprovider/README.md)
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
- [docs/plans/frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [docs/plans/agents-situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md)
## 前端页面布局规范
管理后台页面默认遵循“单屏工作区”原则:
- 页头、摘要区、主工作区应在一屏内形成稳定结构
- 主表格 / 主图表 / 主分析区应占据页面主要可视空间
- 模块内容超出时优先在卡片、表格、标签页内部滚动
- 不依赖整页纵向撑开来容纳主要工作区
当前推荐参考实现:
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
## License

View File

@@ -13,7 +13,14 @@
- [x] 接入 `IPtoASN / IPtoCountry` 作为 prefix-centric geography 的主数据源
- [x] 接入 `OpenGeoFeed` 作为 prefix geography 的高质量覆盖/override 数据源
- [x] 把 RIR delegated 设计成 prefix geography 的 fallback而不是主来源
- [ ]`aiprovider` 建立 `provider -> api adapter -> compat policy` 的配置中心,优先落成 `json``yaml` 文件,运行时按 `provider/model` 读取兼容设置,而不是把专项兼容继续散落在 Python 分支里
- [ ] 为市面上主流 AI 服务补专项兼容配置并固化到配置文件中,至少覆盖 `OpenAI / Anthropic / MiniMax / Ollama / Moonshot / DeepSeek / Qwen / GLM / Gemini / OpenRouter / vLLM / LM Studio / One API`
- [ ] 在兼容配置中补齐可声明项:`api adapter``base_url pattern``auth header``thinking default``reasoning block mapping``stream path``tool-call capability``multimodal capability``provider-specific request patch`
- [ ] 接入 `inetnum` / `inet6num` whois 作为比 RIR 更细粒度的后备层
- [x] 在 activity layer 之后继续补 `route leak``path instability / flap` detector
- [ ] 对 [frontend/public/earth/js/bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) 做按职责拆分的小重构,拆成 data / markers / overlays / animation降低后续维护复杂度
- [ ] 可选优化(非必做):将 BGP incident/collector 标点改为 HTML marker参考 worldmonitor 的 `htmlElementsData` 思路),实现近乎固定屏幕尺寸与更高密度可点击性
- [ ] 保持 Earth 当前这批纯个人偏好设置继续走本地持久化:`旋转模式`、HUD 面板显示/隐藏、`地形透明度` 暂不升级到后端系统设置,避免把设备级偏好过早做成全局配置
- [ ] 如果后续明确需要“账号级同步 Earth 偏好”,再单独设计 `Earth user preferences`:优先按用户维度而不是全局系统设置保存,并规划 `localStorage -> backend` 的平滑迁移策略
- [ ] 把 Earth 态势新闻源从 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 的硬编码列表抽成可配置目录,优先保持当前“实时聚合”链路不变,只先解决新闻源不可配置的问题
- [ ] 为 Earth 态势新闻设计后续采集器化方案:明确新闻数据模型、去重策略、区域映射、过期清理和 Earth/AI 复用方式,再决定何时把新闻从实时抓取升级成正式 collector

View File

@@ -1 +1 @@
0.23.0
0.33.0

View File

@@ -6,29 +6,51 @@ AI_TIMEOUT_SECONDS=60
AI_HTTP_RETRY_ATTEMPTS=2
AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上下文、观测与约束,输出结构化、克制、可执行的分析。
# Select one provider mode:
# - openai_compatible
# - claude_compatible
# Provider identity. Recommended values:
# - minimax
# - openai
# - ollama
AI_PROVIDER=ollama
# Compatibility aliases still accepted:
# - openai_compatible
# - anthropic_compatible
# - claude_compatible
AI_PROVIDER=minimax
# Request adapter style, following OpenClaw's API-seam pattern:
# - auto
# - openai-completions
# - anthropic-messages
# - ollama-generate
AI_PROVIDER_API=anthropic-messages
# Common model selection
AI_MODEL=qwen2.5:7b
AI_MODEL=MiniMax-M2.7
# MiniMax CN Anthropic-compatible example
AI_BASE_URL=https://api.minimaxi.com/anthropic
AI_API_KEY=sk-cp-change-me
AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
# OpenAI-compatible example (vLLM / LM Studio / One API / local gateway)
# AI_PROVIDER=openai_compatible
# AI_PROVIDER=openai
# AI_PROVIDER_API=openai-completions
# AI_BASE_URL=http://127.0.0.1:8001/v1
# AI_API_KEY=local-key
# AI_MODEL=your-local-model
# Claude-compatible example (Anthropic / MiniMax / Claude-compatible gateway)
# AI_PROVIDER=claude_compatible
# AI_BASE_URL=http://127.0.0.1:8002
# Anthropic-compatible example (Claude-compatible gateway)
# AI_PROVIDER=anthropic
# AI_PROVIDER_API=anthropic-messages
# AI_BASE_URL=http://127.0.0.1:8002/anthropic
# AI_API_KEY=local-key
# AI_MODEL=your-model
# AI_MAX_TOKENS=1200
# AI_ANTHROPIC_VERSION=2023-06-01
# Ollama native example
AI_BASE_URL=http://127.0.0.1:11434
AI_API_KEY=
AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
# AI_PROVIDER=ollama
# AI_PROVIDER_API=ollama-generate
# AI_BASE_URL=http://127.0.0.1:11434
# AI_API_KEY=
# AI_MODEL=qwen2.5:7b

View File

@@ -4,21 +4,31 @@
完整使用说明见:
- [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md)
- [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/agents/aiprovider.md)
当前支持:
- `AI_PROVIDER=openai`
- `AI_PROVIDER=openai_compatible`
- `AI_PROVIDER=anthropic`
- `AI_PROVIDER=anthropic_compatible`
- `AI_PROVIDER=claude_compatible`
- `AI_PROVIDER=ollama`
- provider identity:
- `AI_PROVIDER=openai`
- `AI_PROVIDER=anthropic`
- `AI_PROVIDER=minimax`
- `AI_PROVIDER=ollama`
- request adapter:
- `AI_PROVIDER_API=openai-completions`
- `AI_PROVIDER_API=anthropic-messages`
- `AI_PROVIDER_API=ollama-generate`
兼容别名仍然保留:
- `openai_compatible`
- `anthropic_compatible`
- `claude_compatible`
典型配置:
```env
AI_PROVIDER=openai_compatible
AI_PROVIDER=openai
AI_PROVIDER_API=openai-completions
AI_BASE_URL=https://api.openai.com/v1
AI_API_KEY=your_api_key
AI_MODEL=gpt-4o-mini
@@ -26,13 +36,14 @@ AI_TIMEOUT_SECONDS=60
AI_PROVIDER_SERVICE_TOKEN=change_me
```
Claude 兼容供应商示例:
MiniMax 中国大陆节点示例:
```env
AI_PROVIDER=claude_compatible
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
AI_API_KEY=your_api_key
AI_MODEL=your-claude-compatible-model
AI_PROVIDER=minimax
AI_PROVIDER_API=anthropic-messages
AI_BASE_URL=https://api.minimaxi.com/anthropic
AI_API_KEY=sk-cp-xxxxx
AI_MODEL=MiniMax-M2.7
AI_TIMEOUT_SECONDS=60
AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
@@ -43,12 +54,15 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
- Anthropic 官方 Claude API
- Claude 兼容网关
- MiniMax 等提供 Claude/Anthropic 风格消息接口的服务
- MiniMax 等提供 Anthropic Messages 风格接口的服务
这套命名方式参考了 OpenClaw 的接入模式: provider 负责标识供应商, `AI_PROVIDER_API` 负责标识协议适配层, 避免把“供应商”和“协议”绑死在一起。
Ollama 原生示例:
```env
AI_PROVIDER=ollama
AI_PROVIDER_API=ollama-generate
AI_BASE_URL=http://127.0.0.1:11434
AI_API_KEY=
AI_MODEL=qwen2.5:7b
@@ -58,8 +72,8 @@ AI_PROVIDER_SERVICE_TOKEN=change_me
本地模型接入建议:
- `vLLM``LM Studio``One API`优先使用 `openai_compatible`
- `MiniMax`、Claude 兼容网关:使用 `claude_compatible`
- `vLLM``LM Studio``One API``AI_PROVIDER=openai` + `AI_PROVIDER_API=openai-completions`
- `MiniMax`、Claude 兼容网关:`AI_PROVIDER=minimax|anthropic` + `AI_PROVIDER_API=anthropic-messages`
- `Ollama`:可直接使用 `ollama`
启动模板:

View File

@@ -9,6 +9,7 @@ class Settings(BaseSettings):
SERVICE_VERSION: str = "0.1.0"
AI_PROVIDER: str = "disabled"
AI_PROVIDER_API: str = "auto"
AI_BASE_URL: str = "https://api.openai.com/v1"
AI_API_KEY: str = ""
AI_MODEL: str = ""

View File

@@ -8,6 +8,7 @@ from fastapi import HTTPException, status
from aiprovider.config import settings
from aiprovider.schemas import (
AIContentBlock,
AIProviderStatusResponse,
SituationalAnalysisRequest,
SituationalAnalysisResponse,
@@ -18,9 +19,39 @@ def _normalize_provider(value: str) -> str:
return (value or "disabled").strip().lower()
def _normalize_provider_api(value: str) -> str:
return (value or "auto").strip().lower().replace("_", "-")
def _resolve_provider_api(provider: str, configured_api: str) -> str:
if configured_api and configured_api != "auto":
return configured_api
if provider in {"openai", "openai-compatible", "openai_compatible"}:
return "openai-completions"
if provider in {
"anthropic",
"anthropic-compatible",
"anthropic_compatible",
"claude-compatible",
"claude_compatible",
"minimax",
"kimi-coding",
"moonshot-anthropic",
}:
return "anthropic-messages"
if provider == "ollama":
return "ollama-generate"
return "disabled"
class ProviderService:
def __init__(self) -> None:
self.provider = _normalize_provider(settings.AI_PROVIDER)
self.provider_api = _resolve_provider_api(
self.provider,
_normalize_provider_api(settings.AI_PROVIDER_API),
)
self.base_url = settings.AI_BASE_URL.rstrip("/")
self.api_key = settings.AI_API_KEY
self.default_model = settings.AI_MODEL
@@ -32,9 +63,11 @@ class ProviderService:
def get_status(self) -> AIProviderStatusResponse:
enabled = self.provider != "disabled"
configured = enabled and bool(self.base_url and self.api_key and self.default_model)
has_credentials = bool(self.api_key) if self._requires_api_key() else True
configured = enabled and bool(self.base_url and has_credentials and self.default_model)
return AIProviderStatusResponse(
provider=self.provider,
api=self.provider_api if enabled else None,
enabled=enabled,
configured=configured,
model=self.default_model or None,
@@ -49,7 +82,8 @@ class ProviderService:
)
model = payload.preferred_model or self.default_model
if not self.base_url or not self.api_key or not model:
has_credentials = bool(self.api_key) if self._requires_api_key() else True
if not self.base_url or not has_credentials or not model:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="AI provider is not fully configured. Check AI_BASE_URL, AI_API_KEY, and AI_MODEL.",
@@ -57,28 +91,40 @@ class ProviderService:
prompt = self._build_prompt(payload)
if self.provider in {"openai", "openai_compatible"}:
if self.provider_api == "openai-completions":
data = await self._request_openai_compatible(model, prompt)
content = self._extract_openai_content(data)
elif self.provider in {"anthropic", "anthropic_compatible", "claude_compatible"}:
data = await self._request_anthropic_compatible(model, prompt)
content_blocks = self._extract_openai_blocks(data)
elif self.provider_api == "anthropic-messages":
data = await self._request_anthropic_messages(model, prompt, payload.thinking)
content = self._extract_anthropic_content(data)
elif self.provider == "ollama":
content_blocks = self._extract_anthropic_blocks(data)
elif self.provider_api == "ollama-generate":
data = await self._request_ollama(model, prompt)
content = self._extract_ollama_content(data)
content_blocks = self._extract_ollama_blocks(data)
else:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Unsupported AI provider: {self.provider}",
detail=f"Unsupported AI provider API: {self.provider_api}",
)
text_blocks = [block.text for block in content_blocks if block.text]
thinking_blocks = [block.thinking for block in content_blocks if block.thinking]
return SituationalAnalysisResponse(
provider=self.provider,
model=model,
content=content,
content_blocks=content_blocks,
text_blocks=text_blocks,
thinking_blocks=thinking_blocks,
raw_response=data,
)
def _requires_api_key(self) -> bool:
return self.provider_api != "ollama-generate"
def _build_prompt(self, payload: SituationalAnalysisRequest) -> str:
sections = [
f"任务标题:\n{payload.title}",
@@ -113,7 +159,12 @@ class ProviderService:
request_body=request_body,
)
async def _request_anthropic_compatible(self, model: str, prompt: str) -> dict[str, Any]:
async def _request_anthropic_messages(
self,
model: str,
prompt: str,
thinking: dict[str, Any] | None = None,
) -> dict[str, Any]:
request_body = {
"model": model,
"system": self.system_prompt,
@@ -131,8 +182,15 @@ class ProviderService:
"max_tokens": self.max_tokens,
"temperature": 0.2,
}
resolved_thinking = self._resolve_anthropic_thinking(thinking)
if resolved_thinking:
request_body["thinking"] = resolved_thinking
if self.provider == "minimax" and self.base_url.endswith("/anthropic"):
path = "/v1/messages"
else:
path = "/messages"
return await self._post(
path="/messages",
path=path,
headers={
"x-api-key": self.api_key,
"anthropic-version": self.anthropic_version,
@@ -141,6 +199,25 @@ class ProviderService:
request_body=request_body,
)
def _resolve_anthropic_thinking(self, thinking: dict[str, Any] | None) -> dict[str, Any] | None:
if thinking:
return thinking
# OpenClaw treats MiniMax's Anthropic-compatible path specially:
# disable thinking by default unless the caller explicitly opts in.
if self.provider == "minimax":
return {"type": "disabled"}
return None
async def _request_anthropic_compatible(
self,
model: str,
prompt: str,
thinking: dict[str, Any] | None = None,
) -> dict[str, Any]:
return await self._request_anthropic_messages(model, prompt, thinking)
async def _request_ollama(self, model: str, prompt: str) -> dict[str, Any]:
request_body = {
"model": model,
@@ -218,6 +295,30 @@ class ProviderService:
)
return ""
def _extract_openai_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
choices = payload.get("choices") or []
if not choices:
return []
message = choices[0].get("message") or {}
content = message.get("content")
if isinstance(content, str):
return [AIContentBlock(type="text", text=content)]
if not isinstance(content, list):
return []
blocks: list[AIContentBlock] = []
for item in content:
if not isinstance(item, dict):
continue
blocks.append(
AIContentBlock(
type=str(item.get("type", "text")),
text=item.get("text") if isinstance(item.get("text"), str) else None,
metadata={k: v for k, v in item.items() if k not in {"type", "text"}},
)
)
return blocks
def _extract_anthropic_content(self, payload: dict[str, Any]) -> str:
content = payload.get("content")
if isinstance(content, str):
@@ -233,8 +334,39 @@ class ProviderService:
fragments.append(item["text"])
return "".join(fragments)
def _extract_anthropic_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
content = payload.get("content")
if not isinstance(content, list):
return []
blocks: list[AIContentBlock] = []
for item in content:
if not isinstance(item, dict):
continue
blocks.append(
AIContentBlock(
type=str(item.get("type", "unknown")),
text=item.get("text") if isinstance(item.get("text"), str) else None,
thinking=item.get("thinking") if isinstance(item.get("thinking"), str) else None,
signature=item.get("signature") if isinstance(item.get("signature"), str) else None,
metadata={
k: v
for k, v in item.items()
if k not in {"type", "text", "thinking", "signature"}
},
)
)
return blocks
def _extract_ollama_content(self, payload: dict[str, Any]) -> str:
response = payload.get("response")
if isinstance(response, str):
return response
return ""
def _extract_ollama_blocks(self, payload: dict[str, Any]) -> list[AIContentBlock]:
response = payload.get("response")
if isinstance(response, str) and response:
return [AIContentBlock(type="text", text=response)]
return []

View File

@@ -3,6 +3,14 @@ from typing import Any
from pydantic import BaseModel, Field
class AIContentBlock(BaseModel):
type: str
text: str | None = None
thinking: str | None = None
signature: str | None = None
metadata: dict[str, Any] = Field(default_factory=dict)
class SituationalAnalysisRequest(BaseModel):
title: str = Field(..., min_length=1, max_length=200)
objective: str = Field(..., min_length=1, max_length=1000)
@@ -10,17 +18,22 @@ class SituationalAnalysisRequest(BaseModel):
observations: list[str] = Field(default_factory=list)
constraints: list[str] = Field(default_factory=list)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None
class SituationalAnalysisResponse(BaseModel):
provider: str
model: str
content: str
content_blocks: list[AIContentBlock] = Field(default_factory=list)
text_blocks: list[str] = Field(default_factory=list)
thinking_blocks: list[str] = Field(default_factory=list)
raw_response: dict[str, Any] = Field(default_factory=dict)
class AIProviderStatusResponse(BaseModel):
provider: str
api: str | None = None
enabled: bool
configured: bool
model: str | None = None

View File

@@ -13,7 +13,9 @@ from app.api.v1 import (
collected_data,
visualization,
bgp,
news,
system_control,
tv,
)
api_router = APIRouter()
@@ -33,3 +35,5 @@ api_router.include_router(settings.router, prefix="/settings", tags=["settings"]
api_router.include_router(system_control.router, prefix="/system", tags=["system"])
api_router.include_router(visualization.router, prefix="/visualization", tags=["visualization"])
api_router.include_router(bgp.router, prefix="/bgp", tags=["bgp"])
api_router.include_router(tv.router, prefix="/tv", tags=["tv"])
api_router.include_router(news.router, prefix="/news", tags=["news"])

View File

@@ -1,15 +1,52 @@
from uuid import uuid4
from fastapi import APIRouter, Depends, Request, Response
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.security import get_current_user
from app.db.session import get_db
from app.models.user import User
from app.schemas.ai import (
AIProviderStatusResponse,
AlertBriefRequest,
AlertBriefResponse,
BGPBriefRequest,
BGPBriefRecordResponse,
BGPBriefRecordSummary,
PlaygroundMessageActionResponse,
PlaygroundMessageCreateRequest,
PlaygroundMessageEditRequest,
PlaygroundMessageResendRequest,
PlaygroundMessageStopRequest,
PlaygroundSessionResponse,
PlaygroundSessionUpsertRequest,
PlaygroundThreadResponse,
SituationalAlertBriefRequest,
SituationalAlertBriefResponse,
SituationalAnalysisRequest,
SituationalAnalysisResponse,
)
from app.services.alert_ai_brief import build_alert_brief_request
from app.services.ai_client import AIProviderClient, get_ai_provider_client
from app.services.bgp_ai_brief import build_bgp_brief_request
from app.services.bgp_ai_brief_store import (
get_bgp_brief_record,
get_latest_bgp_brief_record,
list_bgp_brief_records,
save_bgp_brief_record,
)
from app.services.playground_session_store import (
get_playground_session,
upsert_playground_session,
)
from app.services.playground_chat_service import (
create_turn,
edit_user_message,
get_thread,
resend_turn,
stop_message,
)
from app.services.situational_alert_ai_brief import build_situational_alert_brief_request
router = APIRouter()
@@ -37,3 +74,208 @@ async def analyze_situational_awareness(
request_id = request.headers.get("X-Request-ID") or str(uuid4())
response.headers["X-Request-ID"] = request_id
return await provider_client.analyze(payload, request_id=request_id)
@router.get("/playground/thread", response_model=PlaygroundThreadResponse | None)
async def get_playground_thread(
session_key: str = "default",
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await get_thread(
db,
user_id=current_user.id,
session_key=session_key,
)
@router.get("/playground/session", response_model=PlaygroundSessionResponse | None)
async def get_saved_playground_session(
session_key: str = "default",
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await get_playground_session(
db,
user_id=current_user.id,
session_key=session_key,
)
@router.put("/playground/session", response_model=PlaygroundSessionResponse)
async def save_playground_session(
payload: PlaygroundSessionUpsertRequest,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await upsert_playground_session(
db,
user_id=current_user.id,
payload=payload,
)
@router.post("/playground/messages", response_model=PlaygroundMessageActionResponse)
async def create_playground_message(
payload: PlaygroundMessageCreateRequest,
current_user: User = Depends(get_current_user),
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
return await create_turn(
db,
user_id=current_user.id,
payload=payload,
provider_client=provider_client,
)
@router.post("/playground/messages/stop", response_model=PlaygroundMessageActionResponse)
async def stop_playground_message(
payload: PlaygroundMessageStopRequest,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await stop_message(
db,
user_id=current_user.id,
payload=payload,
)
@router.post("/playground/messages/resend", response_model=PlaygroundMessageActionResponse)
async def resend_playground_message(
payload: PlaygroundMessageResendRequest,
current_user: User = Depends(get_current_user),
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
return await resend_turn(
db,
user_id=current_user.id,
payload=payload,
provider_client=provider_client,
)
@router.post("/playground/messages/edit", response_model=PlaygroundMessageActionResponse)
async def edit_playground_message(
payload: PlaygroundMessageEditRequest,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return await edit_user_message(
db,
user_id=current_user.id,
payload=payload,
)
@router.get("/bgp/briefs", response_model=list[BGPBriefRecordSummary])
async def list_saved_bgp_briefs(
current_user: User = Depends(get_current_user),
):
return list_bgp_brief_records()
@router.get("/bgp/briefs/latest", response_model=BGPBriefRecordResponse | None)
async def get_latest_saved_bgp_brief(
current_user: User = Depends(get_current_user),
):
return get_latest_bgp_brief_record()
@router.get("/bgp/briefs/{brief_id}", response_model=BGPBriefRecordResponse)
async def get_saved_bgp_brief(
brief_id: str,
current_user: User = Depends(get_current_user),
):
record = get_bgp_brief_record(brief_id)
if record is None:
raise HTTPException(status_code=404, detail="BGP brief not found")
return record
@router.post("/bgp/brief", response_model=BGPBriefRecordResponse)
async def analyze_bgp_brief(
payload: BGPBriefRequest,
request: Request,
response: Response,
current_user: User = Depends(get_current_user),
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
request_id = request.headers.get("X-Request-ID") or str(uuid4())
response.headers["X-Request-ID"] = request_id
brief_request, facts, context = await build_bgp_brief_request(
db,
incident_limit=payload.incident_limit,
anomaly_limit=payload.anomaly_limit,
collector_limit=payload.collector_limit,
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return save_bgp_brief_record(
analysis,
request_id=request_id,
facts=facts,
context=context,
)
@router.post("/alerts/brief", response_model=AlertBriefResponse)
async def analyze_alert_brief(
payload: AlertBriefRequest,
request: Request,
response: Response,
current_user: User = Depends(get_current_user),
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
request_id = request.headers.get("X-Request-ID") or str(uuid4())
response.headers["X-Request-ID"] = request_id
brief_request, facts, context = await build_alert_brief_request(
db,
alert_limit=payload.alert_limit,
)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return AlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)
@router.post("/situational-alerts/brief", response_model=SituationalAlertBriefResponse)
async def analyze_situational_alert_brief(
payload: SituationalAlertBriefRequest,
request: Request,
response: Response,
current_user: User = Depends(get_current_user),
provider_client: AIProviderClient = Depends(get_ai_provider_client),
db: AsyncSession = Depends(get_db),
):
request_id = request.headers.get("X-Request-ID") or str(uuid4())
response.headers["X-Request-ID"] = request_id
brief_request, facts, context = await build_situational_alert_brief_request(db)
brief_request.preferred_model = payload.preferred_model
brief_request.thinking = payload.thinking
analysis = await provider_client.analyze(brief_request, request_id=request_id)
return SituationalAlertBriefResponse(
**analysis.model_dump(),
title=brief_request.title,
objective=brief_request.objective,
facts=facts,
context=context,
)

View File

@@ -1,7 +1,7 @@
from datetime import UTC, datetime
from typing import Optional
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import select, func, case
from sqlalchemy.ext.asyncio import AsyncSession
@@ -9,6 +9,7 @@ from app.db.session import get_db
from app.models.user import User
from app.core.security import get_current_user
from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.schemas.alert import AlertResolutionRequest
router = APIRouter()
@@ -77,7 +78,7 @@ async def acknowledge_alert(
@router.post("/{alert_id}/resolve")
async def resolve_alert(
alert_id: int,
resolution: str,
payload: AlertResolutionRequest,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -85,12 +86,12 @@ async def resolve_alert(
alert = result.scalar_one_or_none()
if not alert:
return {"error": "Alert not found"}
raise HTTPException(status_code=404, detail="Alert not found")
alert.status = AlertStatus.RESOLVED
alert.resolved_by = current_user.id
alert.resolved_at = datetime.now(UTC)
alert.resolution_notes = resolution
alert.resolution_notes = payload.resolution
await db.commit()
return {"message": "Alert resolved", "alert": alert.to_dict()}
@@ -101,25 +102,44 @@ async def get_alert_stats(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
critical_query = select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.CRITICAL,
Alert.status == AlertStatus.ACTIVE,
result = await db.execute(
select(
func.sum(
case(
(
(Alert.severity == AlertSeverity.CRITICAL)
& (Alert.status == AlertStatus.ACTIVE),
1,
),
else_=0,
)
).label("critical"),
func.sum(
case(
(
(Alert.severity == AlertSeverity.WARNING)
& (Alert.status == AlertStatus.ACTIVE),
1,
),
else_=0,
)
).label("warning"),
func.sum(
case(
(
(Alert.severity == AlertSeverity.INFO)
& (Alert.status == AlertStatus.ACTIVE),
1,
),
else_=0,
)
).label("info"),
)
)
warning_query = select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.WARNING,
Alert.status == AlertStatus.ACTIVE,
)
info_query = select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.INFO,
Alert.status == AlertStatus.ACTIVE,
)
critical_result = await db.execute(critical_query)
warning_result = await db.execute(warning_query)
info_result = await db.execute(info_query)
row = result.one()
return {
"critical": critical_result.scalar() or 0,
"warning": warning_result.scalar() or 0,
"info": info_result.scalar() or 0,
"critical": row.critical or 0,
"warning": row.warning or 0,
"info": row.info or 0,
}

View File

@@ -22,16 +22,161 @@ def _parse_dt(value: Optional[str]) -> Optional[datetime]:
if not value:
return None
return datetime.fromisoformat(value.replace("Z", "+00:00"))
def _event_filters(
*,
prefix: Optional[str],
origin_asn: Optional[int],
peer_asn: Optional[int],
collector: Optional[str],
event_type: Optional[str],
source: Optional[str],
time_from: Optional[datetime],
time_to: Optional[datetime],
):
filters = [BGPObservation.source.in_(BGP_SOURCES)]
if source:
filters.append(BGPObservation.source == source)
if prefix:
filters.append(BGPObservation.prefix == prefix)
if origin_asn is not None:
filters.append(BGPObservation.origin_asn == origin_asn)
if peer_asn is not None:
filters.append(BGPObservation.peer_asn == peer_asn)
if collector:
filters.append(BGPObservation.collector == collector)
if event_type:
filters.append(BGPObservation.event_type == event_type)
if time_from:
filters.append(BGPObservation.observed_at >= time_from)
if time_to:
filters.append(BGPObservation.observed_at <= time_to)
return filters
def _matches_time(value: Optional[datetime], time_from: Optional[datetime], time_to: Optional[datetime]) -> bool:
if value is None:
return False
if time_from and value < time_from:
return False
if time_to and value > time_to:
return False
return True
def _anomaly_filters(
*,
severity: Optional[str],
anomaly_type: Optional[str],
status: Optional[str],
prefix: Optional[str],
origin_asn: Optional[int],
time_from: Optional[datetime],
time_to: Optional[datetime],
):
filters = []
if severity:
filters.append(BGPAnomaly.severity == severity)
if anomaly_type:
filters.append(BGPAnomaly.anomaly_type == anomaly_type)
if status:
filters.append(BGPAnomaly.status == status)
if prefix:
filters.append(BGPAnomaly.prefix == prefix)
if origin_asn is not None:
filters.append(BGPAnomaly.origin_asn == origin_asn)
if time_from:
filters.append(BGPAnomaly.created_at >= time_from)
if time_to:
filters.append(BGPAnomaly.created_at <= time_to)
return filters
def _incident_filters(
*,
severity: Optional[str],
incident_type: Optional[str],
status: Optional[str],
):
filters = []
if severity:
filters.append(BGPIncident.severity == severity)
if incident_type:
filters.append(BGPIncident.incident_type == incident_type)
if status:
filters.append(BGPIncident.status == status)
return filters
async def _build_event_summary_payload(db: AsyncSession) -> dict:
base_filters = [BGPObservation.source.in_(BGP_SOURCES)]
total_result = await db.execute(
select(func.count(BGPObservation.id)).where(*base_filters)
)
collectors_result = await db.execute(
select(func.count(func.distinct(BGPObservation.collector))).where(
*base_filters, BGPObservation.collector.isnot(None)
)
)
prefixes_result = await db.execute(
select(func.count(func.distinct(BGPObservation.prefix))).where(
*base_filters, BGPObservation.prefix.isnot(None)
)
)
type_result = await db.execute(
select(BGPObservation.event_type, func.count(BGPObservation.id))
.where(*base_filters)
.group_by(BGPObservation.event_type)
)
return {
"total": total_result.scalar() or 0,
"collector_count": collectors_result.scalar() or 0,
"prefix_count": prefixes_result.scalar() or 0,
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
}
async def _build_anomaly_summary_payload(db: AsyncSession) -> dict:
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
type_result = await db.execute(
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.anomaly_type)
.order_by(func.count(BGPAnomaly.id).desc())
)
severity_result = await db.execute(
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.severity)
.order_by(func.count(BGPAnomaly.id).desc())
)
status_result = await db.execute(
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.status)
.order_by(func.count(BGPAnomaly.id).desc())
)
return {
"total": total_result.scalar() or 0,
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
}
async def _build_incident_summary_payload(db: AsyncSession) -> dict:
total_result = await db.execute(select(func.count(BGPIncident.id)))
type_result = await db.execute(
select(BGPIncident.incident_type, func.count(BGPIncident.id))
.group_by(BGPIncident.incident_type)
.order_by(func.count(BGPIncident.id).desc())
)
severity_result = await db.execute(
select(BGPIncident.severity, func.count(BGPIncident.id))
.group_by(BGPIncident.severity)
.order_by(func.count(BGPIncident.id).desc())
)
status_result = await db.execute(
select(BGPIncident.status, func.count(BGPIncident.id))
.group_by(BGPIncident.status)
.order_by(func.count(BGPIncident.id).desc())
)
return {
"total": total_result.scalar() or 0,
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
}
@router.get("/events")
@@ -49,41 +194,36 @@ async def list_bgp_events(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
stmt = (
select(BGPObservation)
.where(BGPObservation.source.in_(BGP_SOURCES))
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
)
if source:
stmt = stmt.where(BGPObservation.source == source)
result = await db.execute(stmt)
records = result.scalars().all()
dt_from = _parse_dt(time_from)
dt_to = _parse_dt(time_to)
filtered = []
for record in records:
if prefix and record.prefix != prefix:
continue
if origin_asn is not None and record.origin_asn != origin_asn:
continue
if peer_asn is not None and record.peer_asn != peer_asn:
continue
if collector and record.collector != collector:
continue
if event_type and record.event_type != event_type:
continue
if (dt_from or dt_to) and not _matches_time(record.observed_at, dt_from, dt_to):
continue
filtered.append(record)
filters = _event_filters(
prefix=prefix,
origin_asn=origin_asn,
peer_asn=peer_asn,
collector=collector,
event_type=event_type,
source=source,
time_from=dt_from,
time_to=dt_to,
)
offset = (page - 1) * page_size
count_result = await db.execute(
select(func.count(BGPObservation.id)).where(*filters)
)
data_result = await db.execute(
select(BGPObservation)
.where(*filters)
.order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
.offset(offset)
.limit(page_size)
)
records = data_result.scalars().all()
return {
"total": len(filtered),
"total": count_result.scalar() or 0,
"page": page,
"page_size": page_size,
"data": [record.to_dict() for record in filtered[offset : offset + page_size]],
"data": [record.to_dict() for record in records],
}
@@ -92,21 +232,7 @@ async def get_bgp_event_summary(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
result = await db.execute(select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES)))
records = result.scalars().all()
collectors = sorted({record.collector for record in records if record.collector})
prefixes = sorted({record.prefix for record in records if record.prefix})
by_type: dict[str, int] = {}
for record in records:
by_type[record.event_type] = by_type.get(record.event_type, 0) + 1
return {
"total": len(records),
"collector_count": len(collectors),
"prefix_count": len(prefixes),
"by_type": by_type,
}
return await _build_event_summary_payload(db)
@router.get("/collectors")
@@ -138,6 +264,32 @@ async def get_bgp_collector_summary(
}
@router.get("/overview/summary")
async def get_bgp_overview_summary(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
event_summary = await _build_event_summary_payload(db)
anomaly_summary = await _build_anomaly_summary_payload(db)
incident_summary = await _build_incident_summary_payload(db)
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
active_collectors = [item for item in collectors if item["observation_count"] > 0]
return {
"incidentSummary": incident_summary,
"anomalySummary": anomaly_summary,
"eventSummary": event_summary,
"collectorSummary": {
"total": len(collectors),
"active_collectors": len(active_collectors),
"observed_prefixes": sum(item["prefix_count"] for item in active_collectors),
"observed_origins": sum(item["origin_asn_count"] for item in active_collectors),
"recent_24h_events": sum(item["recent_24h_observation_count"] for item in active_collectors),
"recent_7d_events": sum(item["recent_7d_observation_count"] for item in active_collectors),
},
}
@router.get("/events/{event_id}")
async def get_bgp_event(
event_id: int,
@@ -164,31 +316,35 @@ async def list_bgp_anomalies(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
stmt = select(BGPAnomaly).order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
if severity:
stmt = stmt.where(BGPAnomaly.severity == severity)
if anomaly_type:
stmt = stmt.where(BGPAnomaly.anomaly_type == anomaly_type)
if status:
stmt = stmt.where(BGPAnomaly.status == status)
if prefix:
stmt = stmt.where(BGPAnomaly.prefix == prefix)
if origin_asn is not None:
stmt = stmt.where(BGPAnomaly.origin_asn == origin_asn)
result = await db.execute(stmt)
records = result.scalars().all()
dt_from = _parse_dt(time_from)
dt_to = _parse_dt(time_to)
if dt_from or dt_to:
records = [record for record in records if _matches_time(record.created_at, dt_from, dt_to)]
filters = _anomaly_filters(
severity=severity,
anomaly_type=anomaly_type,
status=status,
prefix=prefix,
origin_asn=origin_asn,
time_from=dt_from,
time_to=dt_to,
)
offset = (page - 1) * page_size
total_result = await db.execute(
select(func.count(BGPAnomaly.id)).where(*filters)
)
data_result = await db.execute(
select(BGPAnomaly)
.where(*filters)
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
.offset(offset)
.limit(page_size)
)
records = data_result.scalars().all()
return {
"total": len(records),
"total": total_result.scalar() or 0,
"page": page,
"page_size": page_size,
"data": [record.to_dict() for record in records[offset : offset + page_size]],
"data": [record.to_dict() for record in records],
}
@@ -197,29 +353,7 @@ async def get_bgp_anomaly_summary(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
total_result = await db.execute(select(func.count(BGPAnomaly.id)))
type_result = await db.execute(
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.anomaly_type)
.order_by(func.count(BGPAnomaly.id).desc())
)
severity_result = await db.execute(
select(BGPAnomaly.severity, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.severity)
.order_by(func.count(BGPAnomaly.id).desc())
)
status_result = await db.execute(
select(BGPAnomaly.status, func.count(BGPAnomaly.id))
.group_by(BGPAnomaly.status)
.order_by(func.count(BGPAnomaly.id).desc())
)
return {
"total": total_result.scalar() or 0,
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
}
return await _build_anomaly_summary_payload(db)
@router.get("/anomalies/{anomaly_id}")
@@ -244,22 +378,29 @@ async def list_bgp_incidents(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
stmt = select(BGPIncident).order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
if severity:
stmt = stmt.where(BGPIncident.severity == severity)
if incident_type:
stmt = stmt.where(BGPIncident.incident_type == incident_type)
if status:
stmt = stmt.where(BGPIncident.status == status)
result = await db.execute(stmt)
records = result.scalars().all()
filters = _incident_filters(
severity=severity,
incident_type=incident_type,
status=status,
)
offset = (page - 1) * page_size
total_result = await db.execute(
select(func.count(BGPIncident.id)).where(*filters)
)
data_result = await db.execute(
select(BGPIncident)
.where(*filters)
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
.offset(offset)
.limit(page_size)
)
records = data_result.scalars().all()
return {
"total": len(records),
"total": total_result.scalar() or 0,
"page": page,
"page_size": page_size,
"data": [record.to_dict() for record in records[offset : offset + page_size]],
"data": [record.to_dict() for record in records],
}
@@ -268,29 +409,7 @@ async def get_bgp_incident_summary(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
total_result = await db.execute(select(func.count(BGPIncident.id)))
type_result = await db.execute(
select(BGPIncident.incident_type, func.count(BGPIncident.id))
.group_by(BGPIncident.incident_type)
.order_by(func.count(BGPIncident.id).desc())
)
severity_result = await db.execute(
select(BGPIncident.severity, func.count(BGPIncident.id))
.group_by(BGPIncident.severity)
.order_by(func.count(BGPIncident.id).desc())
)
status_result = await db.execute(
select(BGPIncident.status, func.count(BGPIncident.id))
.group_by(BGPIncident.status)
.order_by(func.count(BGPIncident.id).desc())
)
return {
"total": total_result.scalar() or 0,
"by_type": {row[0]: row[1] for row in type_result.fetchall()},
"by_severity": {row[0]: row[1] for row in severity_result.fetchall()},
"by_status": {row[0]: row[1] for row in status_result.fetchall()},
}
return await _build_incident_summary_payload(db)
@router.get("/incidents/{incident_id}")

View File

@@ -2,7 +2,7 @@
from datetime import UTC, datetime, timedelta
from fastapi import APIRouter, Depends
from sqlalchemy import select, func, text
from sqlalchemy import case, select, func, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_db
@@ -118,58 +118,77 @@ async def get_stats(
built_in_count = len(COLLECTOR_INFO)
built_in_active = built_in_count # Built-in are always "active" for counting purposes
# Count custom configs from database
result = await db.execute(select(func.count(DataSourceConfig.id)))
custom_count = result.scalar() or 0
result = await db.execute(
select(func.count(DataSourceConfig.id)).where(DataSourceConfig.is_active == True)
select(
func.count(DataSourceConfig.id).label("custom_count"),
func.sum(
case((DataSourceConfig.is_active == True, 1), else_=0)
).label("custom_active"),
)
)
custom_active = result.scalar() or 0
datasource_stats = result.one()
custom_count = datasource_stats.custom_count or 0
custom_active = datasource_stats.custom_active or 0
# Total datasources
total_datasources = built_in_count + custom_count
active_datasources = built_in_active + custom_active
# Tasks today (from database)
result = await db.execute(
select(func.count(CollectionTask.id)).where(CollectionTask.started_at >= today_start)
)
tasks_today = result.scalar() or 0
result = await db.execute(
select(func.count(CollectionTask.id)).where(
CollectionTask.status == "success",
CollectionTask.started_at >= today_start,
select(
func.count(CollectionTask.id).label("tasks_today"),
func.sum(
case(
(CollectionTask.status == "success", 1),
else_=0,
)
).label("success_tasks"),
)
.where(CollectionTask.started_at >= today_start)
)
success_tasks = result.scalar() or 0
task_stats = result.one()
tasks_today = task_stats.tasks_today or 0
success_tasks = task_stats.success_tasks or 0
success_rate = (success_tasks / tasks_today * 100) if tasks_today > 0 else 0
# Alerts
result = await db.execute(
select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.CRITICAL,
Alert.status == "active",
select(
func.sum(
case(
(
(Alert.severity == AlertSeverity.CRITICAL)
& (Alert.status == "active"),
1,
),
else_=0,
)
).label("critical_alerts"),
func.sum(
case(
(
(Alert.severity == AlertSeverity.WARNING)
& (Alert.status == "active"),
1,
),
else_=0,
)
).label("warning_alerts"),
func.sum(
case(
(
(Alert.severity == AlertSeverity.INFO)
& (Alert.status == "active"),
1,
),
else_=0,
)
).label("info_alerts"),
)
)
critical_alerts = result.scalar() or 0
result = await db.execute(
select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.WARNING,
Alert.status == "active",
)
)
warning_alerts = result.scalar() or 0
result = await db.execute(
select(func.count(Alert.id)).where(
Alert.severity == AlertSeverity.INFO,
Alert.status == "active",
)
)
info_alerts = result.scalar() or 0
alert_stats = result.one()
critical_alerts = alert_stats.critical_alerts or 0
warning_alerts = alert_stats.warning_alerts or 0
info_alerts = alert_stats.info_alerts or 0
response = {
"total_datasources": total_datasources,

View File

@@ -3,7 +3,7 @@ from datetime import datetime, timedelta, timezone
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy import func, select
from sqlalchemy import func, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
@@ -11,10 +11,17 @@ from app.core.security import get_current_user
from app.core.data_sources import get_data_sources_config
from app.db.session import get_db
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
from app.models.datasource import DataSource
from app.models.datasource_config import DataSourceConfig
from app.models.task import CollectionTask
from app.models.user import User
from app.services.scheduler import get_latest_task_id_for_datasource, run_collector_now, sync_datasource_job
from app.services.scheduler import (
cancel_running_collector_now,
get_latest_task_id_for_datasource,
run_collector_now,
sync_datasource_job,
)
router = APIRouter()
STALE_RUNNING_TASK_TIMEOUT_MINUTES = 90
@@ -34,6 +41,156 @@ def is_due_for_collection(datasource: DataSource, now: datetime) -> bool:
return datasource.last_run_at + timedelta(minutes=datasource.frequency_minutes) <= now
def _task_rank_column(order_column):
return func.row_number().over(
partition_by=CollectionTask.datasource_id,
order_by=(order_column.desc().nullslast(), CollectionTask.id.desc()),
).label("row_num")
async def _load_latest_running_tasks(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, CollectionTask]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
_task_rank_column(CollectionTask.started_at),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.where(CollectionTask.status == "running")
.subquery()
)
result = await db.execute(
select(CollectionTask)
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_completed_tasks(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, CollectionTask]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
_task_rank_column(CollectionTask.completed_at),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.where(CollectionTask.completed_at.isnot(None))
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
.subquery()
)
result = await db.execute(
select(CollectionTask)
.join(ranked_tasks, CollectionTask.id == ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {task.datasource_id: task for task in result.scalars().all()}
async def _load_latest_task_ids(
db: AsyncSession,
datasource_ids: list[int],
) -> dict[int, int]:
if not datasource_ids:
return {}
ranked_tasks = (
select(
CollectionTask.id.label("task_id"),
CollectionTask.datasource_id.label("datasource_id"),
func.row_number().over(
partition_by=CollectionTask.datasource_id,
order_by=CollectionTask.id.desc(),
).label("row_num"),
)
.where(CollectionTask.datasource_id.in_(datasource_ids))
.subquery()
)
result = await db.execute(
select(ranked_tasks.c.datasource_id, ranked_tasks.c.task_id)
.where(ranked_tasks.c.row_num == 1)
)
return {datasource_id: task_id for datasource_id, task_id in result.all()}
async def _load_datasource_data_counts(
db: AsyncSession,
sources: list[str],
) -> dict[str, int]:
if not sources:
return {}
result = await db.execute(
select(CollectedData.source, func.count(CollectedData.id))
.where(CollectedData.source.in_(sources))
.group_by(CollectedData.source)
)
return {source: count for source, count in result.all()}
async def _load_datasource_endpoint_overrides(
db: AsyncSession,
sources: list[str],
) -> dict[str, str]:
if not sources:
return {}
result = await db.execute(
select(DataSourceConfig.name, DataSourceConfig.endpoint)
.where(DataSourceConfig.name.in_(sources))
.where(DataSourceConfig.is_active.is_(True))
.where(DataSourceConfig.endpoint.isnot(None))
)
return {
name: endpoint
for name, endpoint in result.all()
if endpoint
}
async def _load_datasource_list_context(
db: AsyncSession,
datasources: list[DataSource],
) -> tuple[dict[int, CollectionTask], dict[int, CollectionTask], dict[str, int], dict[str, str]]:
datasource_ids = [datasource.id for datasource in datasources]
sources = [datasource.source for datasource in datasources]
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
datasource_by_id = {datasource.id: datasource for datasource in datasources}
now = datetime.now(timezone.utc)
stale_datasource_ids: list[int] = []
for datasource_id, task in running_tasks.items():
started_at = task.started_at
if started_at is None:
continue
if started_at.tzinfo is None:
started_at = started_at.replace(tzinfo=timezone.utc)
if now - started_at > timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
datasource = datasource_by_id.get(datasource_id)
if datasource is not None:
await fail_and_rollback_stale_running_task(db, datasource, task)
stale_datasource_ids.append(datasource_id)
if stale_datasource_ids:
running_tasks = await _load_latest_running_tasks(db, datasource_ids)
completed_tasks = await _load_latest_completed_tasks(db, datasource_ids)
data_counts = await _load_datasource_data_counts(db, sources)
endpoint_overrides = await _load_datasource_endpoint_overrides(db, sources)
return running_tasks, completed_tasks, data_counts, endpoint_overrides
async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[DataSource]:
datasource = None
try:
@@ -52,18 +209,6 @@ async def get_datasource_record(db: AsyncSession, source_id: str) -> Optional[Da
return result.scalar_one_or_none()
async def get_last_completed_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
result = await db.execute(
select(CollectionTask)
.where(CollectionTask.datasource_id == datasource_id)
.where(CollectionTask.completed_at.isnot(None))
.where(CollectionTask.status.in_(("success", "failed", "cancelled")))
.order_by(CollectionTask.completed_at.desc())
.limit(1)
)
return result.scalar_one_or_none()
async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[CollectionTask]:
result = await db.execute(
select(CollectionTask)
@@ -87,17 +232,152 @@ async def get_running_task(db: AsyncSession, datasource_id: int) -> Optional[Col
if now - started_at <= timedelta(minutes=STALE_RUNNING_TASK_TIMEOUT_MINUTES):
return task
existing_error = (task.error_message or "").strip()
datasource = await db.get(DataSource, datasource_id)
if datasource is not None:
await fail_and_rollback_stale_running_task(db, datasource, task)
else:
existing_error = (task.error_message or "").strip()
stale_reason = (
f"Marked failed automatically after stale running timeout "
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
)
task.status = "failed"
task.phase = "failed"
task.completed_at = now
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
await db.commit()
return None
async def rollback_orphaned_running_task(
db: AsyncSession,
datasource: DataSource,
running_task: CollectionTask,
) -> None:
snapshot_result = await db.execute(
select(DataSnapshot)
.where(
DataSnapshot.datasource_id == datasource.id,
DataSnapshot.task_id == running_task.id,
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
snapshot = snapshot_result.scalar_one_or_none()
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = FALSE
WHERE source = :source
"""
),
{"source": datasource.source},
)
if snapshot is not None:
snapshot.status = "cancelled"
snapshot.is_current = False
snapshot.completed_at = datetime.now(timezone.utc)
summary = dict(snapshot.summary or {})
summary["rollback"] = True
summary["rollback_reason"] = "orphaned_running_task_after_backend_restart"
snapshot.summary = summary
if snapshot.parent_snapshot_id is not None:
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
if parent_snapshot:
parent_snapshot.is_current = True
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = TRUE
WHERE snapshot_id = :snapshot_id
"""
),
{"snapshot_id": snapshot.parent_snapshot_id},
)
running_task.status = "cancelled"
running_task.phase = "cancelled"
running_task.completed_at = datetime.now(timezone.utc)
existing_error = (running_task.error_message or "").strip()
cancel_reason = "Cancelled after backend restart because the running task handle was lost; incomplete writes rolled back"
running_task.error_message = f"{existing_error}\n{cancel_reason}".strip() if existing_error else cancel_reason
datasource.last_status = "cancelled"
datasource.last_run_at = datetime.now(timezone.utc)
await db.commit()
async def fail_and_rollback_stale_running_task(
db: AsyncSession,
datasource: DataSource,
running_task: CollectionTask,
) -> None:
snapshot_result = await db.execute(
select(DataSnapshot)
.where(
DataSnapshot.datasource_id == datasource.id,
DataSnapshot.task_id == running_task.id,
)
.order_by(DataSnapshot.id.desc())
.limit(1)
)
snapshot = snapshot_result.scalar_one_or_none()
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == running_task.id))
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = FALSE
WHERE source = :source
"""
),
{"source": datasource.source},
)
if snapshot is not None:
snapshot.status = "failed"
snapshot.is_current = False
snapshot.completed_at = datetime.now(timezone.utc)
summary = dict(snapshot.summary or {})
summary["rollback"] = True
summary["rollback_reason"] = "stale_running_task_timeout"
snapshot.summary = summary
if snapshot.parent_snapshot_id is not None:
parent_snapshot = await db.get(DataSnapshot, snapshot.parent_snapshot_id)
if parent_snapshot:
parent_snapshot.is_current = True
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = TRUE
WHERE snapshot_id = :snapshot_id
"""
),
{"snapshot_id": snapshot.parent_snapshot_id},
)
existing_error = (running_task.error_message or "").strip()
stale_reason = (
f"Marked failed automatically after stale running timeout "
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m)"
f"({STALE_RUNNING_TASK_TIMEOUT_MINUTES}m); incomplete writes rolled back"
)
task.status = "failed"
task.phase = "failed"
task.completed_at = now
task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
running_task.status = "failed"
running_task.phase = "failed"
running_task.completed_at = datetime.now(timezone.utc)
running_task.error_message = f"{existing_error}\n{stale_reason}".strip() if existing_error else stale_reason
datasource.last_status = "failed"
datasource.last_run_at = datetime.now(timezone.utc)
await db.commit()
return None
@router.get("")
@@ -121,14 +401,17 @@ async def list_datasources(
collector_list = []
config = get_data_sources_config()
running_tasks, completed_tasks, data_counts, endpoint_overrides = await _load_datasource_list_context(
db,
datasources,
)
for datasource in datasources:
running_task = await get_running_task(db, datasource.id)
last_task = await get_last_completed_task(db, datasource.id)
endpoint = await config.get_url(datasource.source, db)
data_count_result = await db.execute(
select(func.count(CollectedData.id)).where(CollectedData.source == datasource.source)
running_task = running_tasks.get(datasource.id)
last_task = completed_tasks.get(datasource.id)
endpoint = endpoint_overrides.get(datasource.source) or config.get_yaml_url(
datasource.source,
)
data_count = data_count_result.scalar() or 0
data_count = data_counts.get(datasource.source, 0)
last_run_at = datasource.last_run_at or (last_task.completed_at if last_task else None)
last_run = to_iso8601_utc(last_run_at)
@@ -137,6 +420,7 @@ async def list_datasources(
collector_list.append(
{
"id": datasource.id,
"source": datasource.source,
"name": datasource.name,
"module": datasource.module,
"priority": datasource.priority,
@@ -189,9 +473,13 @@ async def trigger_all_datasources(
skipped_sources: list[dict] = []
failed_sources: list[dict] = []
now = datetime.now(timezone.utc)
running_tasks = await _load_latest_running_tasks(
db,
[datasource.id for datasource in datasources],
)
for datasource in datasources:
running_task = await get_running_task(db, datasource.id)
running_task = running_tasks.get(datasource.id)
if running_task is not None:
skipped_sources.append(
{
@@ -219,7 +507,7 @@ async def trigger_all_datasources(
)
continue
previous_task_ids[datasource.id] = await get_latest_task_id_for_datasource(datasource.id)
previous_task_ids[datasource.id] = None
success = run_collector_now(datasource.source)
if not success:
failed_sources.append(
@@ -241,13 +529,24 @@ async def trigger_all_datasources(
}
)
latest_task_ids = await _load_latest_task_ids(
db,
[datasource.id for datasource in datasources],
)
for datasource_id in previous_task_ids:
previous_task_ids[datasource_id] = latest_task_ids.get(datasource_id)
for _ in range(20):
await asyncio.sleep(0.1)
pending = [item for item in triggered_sources if item["task_id"] is None]
if not pending:
break
latest_task_ids = await _load_latest_task_ids(
db,
[item["id"] for item in pending],
)
for item in pending:
task_id = await get_latest_task_id_for_datasource(item["id"])
task_id = latest_task_ids.get(item["id"])
if task_id is not None and task_id != previous_task_ids.get(item["id"]):
item["task_id"] = task_id
@@ -346,6 +645,7 @@ async def get_datasource_stats(
@router.post("/{source_id}/trigger")
async def trigger_datasource(
source_id: str,
force: bool = Query(False),
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
@@ -356,6 +656,26 @@ async def trigger_datasource(
if not datasource.is_active:
raise HTTPException(status_code=400, detail="Data source is disabled")
running_task = await get_running_task(db, datasource.id)
if running_task is not None and not force:
raise HTTPException(
status_code=409,
detail={
"reason": "running_task_in_progress",
"message": "当前采集任务尚未完成,重新触发会丢失本次未完成进度。是否强制重新采集?",
"task_id": running_task.id,
"phase": running_task.phase,
"progress": running_task.progress,
"records_processed": running_task.records_processed,
"total_records": running_task.total_records,
},
)
if running_task is not None and force:
cancelled = await cancel_running_collector_now(datasource.source)
if not cancelled:
await rollback_orphaned_running_task(db, datasource, running_task)
previous_task_id = await get_latest_task_id_for_datasource(datasource.id)
success = run_collector_now(datasource.source)
if not success:
@@ -375,6 +695,7 @@ async def trigger_datasource(
"source_id": datasource.id,
"task_id": task_id,
"collector_name": datasource.source,
"force": force,
"message": f"Collector '{datasource.source}' has been triggered",
}
@@ -412,6 +733,7 @@ async def clear_datasource_data(
async def get_task_status(
source_id: str,
task_id: Optional[int] = None,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
datasource = await get_datasource_record(db, source_id)

View File

@@ -0,0 +1,13 @@
from fastapi import APIRouter, Query
from app.services.earth_news import get_earth_news_payload
router = APIRouter()
@router.get("/earth-feed")
async def get_earth_feed(
lat: float | None = Query(None, description="Current Earth view center latitude"),
lon: float | None = Query(None, description="Current Earth view center longitude"),
):
return await get_earth_news_payload(lat=lat, lon=lon)

View File

@@ -1,3 +1,4 @@
from copy import deepcopy
from datetime import UTC, datetime
from typing import Optional
@@ -13,6 +14,7 @@ from app.models.datasource import DataSource
from app.models.system_setting import SystemSetting
from app.models.user import User
from app.services.scheduler import sync_datasource_job
from app.services.tv_streams import DEFAULT_TV_SETTINGS, get_tv_settings_payload, normalize_tv_settings
router = APIRouter()
@@ -36,6 +38,7 @@ DEFAULT_SETTINGS = {
"max_login_attempts": 5,
"password_policy": "medium",
},
"tv": DEFAULT_TV_SETTINGS,
}
@@ -67,8 +70,34 @@ class CollectorSettingsUpdate(BaseModel):
frequency_minutes: int = Field(default=60, ge=1, le=10080)
class TVStreamSourceUpdate(BaseModel):
id: str = Field(min_length=1, max_length=100)
name: str = Field(min_length=1, max_length=200)
provider: str = Field(default="Unknown", max_length=100)
region: str = Field(default="Global", max_length=100)
language: str = Field(default="und", max_length=32)
source_type: str = Field(default="iframe", pattern="^(iframe|hls|video|external|youtube)$")
embed_url: str = ""
stream_url: str = ""
homepage_url: str = ""
poster_url: str = ""
youtube_video_id: str = ""
youtube_channel: str = ""
is_enabled: bool = True
is_fallback: bool = False
sort_order: int = Field(default=10, ge=0, le=9999)
collector_source: Optional[str] = None
notes: str = ""
class TVSettingsUpdate(BaseModel):
default_source_id: str = Field(default=DEFAULT_TV_SETTINGS["default_source_id"], min_length=1)
auto_fallback: bool = True
sources: list[TVStreamSourceUpdate] = Field(default_factory=list)
def merge_with_defaults(category: str, payload: Optional[dict]) -> dict:
merged = DEFAULT_SETTINGS[category].copy()
merged = deepcopy(DEFAULT_SETTINGS[category])
if payload:
merged.update(payload)
return merged
@@ -79,6 +108,26 @@ async def get_setting_record(db: AsyncSession, category: str) -> Optional[System
return result.scalar_one_or_none()
async def get_setting_payloads(db: AsyncSession, categories: list[str]) -> dict[str, dict]:
if not categories:
return {}
result = await db.execute(
select(SystemSetting).where(SystemSetting.category.in_(categories))
)
records_by_category = {
record.category: record
for record in result.scalars().all()
}
return {
category: merge_with_defaults(
category,
records_by_category.get(category).payload if records_by_category.get(category) else None,
)
for category in categories
}
async def get_setting_payload(db: AsyncSession, category: str) -> dict:
record = await get_setting_record(db, category)
return merge_with_defaults(category, record.payload if record else None)
@@ -175,6 +224,25 @@ async def update_security_settings(
return {"status": "updated", "security": payload}
@router.get("/tv")
async def get_tv_settings(
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
return {"tv": await get_tv_settings_payload(db)}
@router.put("/tv")
async def update_tv_settings(
settings: TVSettingsUpdate,
current_user: User = Depends(get_current_user),
db: AsyncSession = Depends(get_db),
):
payload = normalize_tv_settings(settings.model_dump())
saved = await save_setting_payload(db, "tv", payload)
return {"status": "updated", "tv": normalize_tv_settings(saved)}
@router.get("/collectors")
async def get_collector_settings(
current_user: User = Depends(get_current_user),
@@ -212,10 +280,15 @@ async def get_all_settings(
):
result = await db.execute(select(DataSource).order_by(DataSource.module, DataSource.id))
datasources = result.scalars().all()
setting_payloads = await get_setting_payloads(
db,
["system", "notifications", "security"],
)
return {
"system": await get_setting_payload(db, "system"),
"notifications": await get_setting_payload(db, "notifications"),
"security": await get_setting_payload(db, "security"),
"system": setting_payloads["system"],
"notifications": setting_payloads["notifications"],
"security": setting_payloads["security"],
"tv": await get_tv_settings_payload(db),
"collectors": [serialize_collector(datasource) for datasource in datasources],
"generated_at": to_iso8601_utc(datetime.now(UTC)),
}

70
backend/app/api/v1/tv.py Normal file
View File

@@ -0,0 +1,70 @@
from urllib.parse import quote, urljoin
import httpx
from fastapi import APIRouter, Depends, HTTPException, Query
from fastapi.responses import Response
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_db
from app.services.tv_streams import get_public_tv_payload, is_allowed_tv_proxy_url
router = APIRouter()
@router.get("/streams")
async def list_public_tv_streams(
db: AsyncSession = Depends(get_db),
):
return await get_public_tv_payload(db)
@router.get("/proxy")
async def proxy_tv_stream(
url: str = Query(..., description="Upstream TV stream or manifest URL"),
db: AsyncSession = Depends(get_db),
):
payload = await get_public_tv_payload(db)
if not is_allowed_tv_proxy_url(url, payload.get("sources", [])):
raise HTTPException(status_code=403, detail="TV proxy target is not allowed")
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=20.0) as client:
upstream = await client.get(
url,
headers={
"User-Agent": "Mozilla/5.0",
"Referer": "https://tv.cctv.com/live/cctv4/",
},
)
upstream.raise_for_status()
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Failed to fetch TV stream: {exc}") from exc
content_type = upstream.headers.get("content-type", "application/octet-stream")
raw_content = upstream.content
response_url = str(upstream.url)
is_manifest = (
response_url.endswith(".m3u8")
or "mpegurl" in content_type.lower()
or raw_content.lstrip().startswith(b"#EXTM3U")
)
headers = {"Cache-Control": "no-store"}
if is_manifest:
manifest_text = upstream.text
rewritten_lines: list[str] = []
for line in manifest_text.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#"):
rewritten_lines.append(line)
continue
absolute_url = urljoin(response_url, stripped)
rewritten_lines.append(f"/api/v1/tv/proxy?url={quote(absolute_url, safe='')}")
return Response(
content="\n".join(rewritten_lines),
media_type="application/vnd.apple.mpegurl",
headers=headers,
)
return Response(content=raw_content, media_type=content_type, headers=headers)

View File

@@ -6,7 +6,8 @@ Returns GeoJSON format compatible with Three.js, CesiumJS, and Unreal Cesium.
from datetime import UTC, datetime
import math
from fastapi import APIRouter, HTTPException, Depends, Query
import httpx
from fastapi import APIRouter, HTTPException, Depends, Query, Response
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy import select, func
from typing import List, Dict, Any, Optional
@@ -23,6 +24,9 @@ from app.services.cable_graph import build_graph_from_data, CableGraph, haversin
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
router = APIRouter()
TERRAIN_TILE_URL_TEMPLATE = (
"https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png"
)
# ============== Converter Functions ==============
@@ -205,42 +209,84 @@ def convert_satellite_to_geojson(records: List[CollectedData]) -> Dict[str, Any]
return {"type": "FeatureCollection", "features": features}
def dedupe_satellite_records(records: List[CollectedData]) -> List[CollectedData]:
"""Keep only the newest record for each satellite identity."""
latest_by_key: Dict[str, CollectedData] = {}
for record in records:
metadata = record.extra_data or {}
norad_id = metadata.get("norad_cat_id")
dedupe_key = (
str(norad_id)
if norad_id not in (None, "")
else str(record.source_id or record.entity_key or record.name or record.id)
)
existing = latest_by_key.get(dedupe_key)
if existing is None or (record.id or 0) > (existing.id or 0):
latest_by_key[dedupe_key] = record
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
def _current_collected_data_stmt(source: str):
return (
select(CollectedData)
.where(CollectedData.source == source)
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.id.desc())
)
def dedupe_collected_records(records: List[CollectedData]) -> List[CollectedData]:
"""Keep only the newest record for each collected entity."""
latest_by_key: Dict[str, CollectedData] = {}
async def _load_current_collected_data(
db: AsyncSession,
source: str,
*,
exclude_unknown_name: bool = False,
limit: Optional[int] = None,
) -> List[CollectedData]:
stmt = _current_collected_data_stmt(source)
if exclude_unknown_name:
stmt = stmt.where(CollectedData.name != "Unknown")
if limit is not None:
stmt = stmt.limit(limit)
for record in records:
dedupe_key = str(
record.source_id
or record.entity_key
or record.name
or record.id
)
existing = latest_by_key.get(dedupe_key)
if existing is None or (record.id or 0) > (existing.id or 0):
latest_by_key[dedupe_key] = record
result = await db.execute(stmt)
return list(result.scalars().all())
return sorted(latest_by_key.values(), key=lambda item: item.id or 0, reverse=True)
async def _load_current_collected_data_by_sources(
db: AsyncSession,
sources: List[str],
) -> Dict[str, List[CollectedData]]:
if not sources:
return {}
stmt = (
select(CollectedData)
.where(CollectedData.source.in_(sources))
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
)
result = await db.execute(stmt)
grouped_records: Dict[str, List[CollectedData]] = {source: [] for source in sources}
for record in result.scalars().all():
grouped_records.setdefault(record.source, []).append(record)
return grouped_records
def _build_landing_point_cable_maps(
relation_records: List[CollectedData],
cable_records: List[CollectedData],
) -> tuple[Dict[int, List[int]], Dict[int, str]]:
city_to_cable_ids_map: Dict[int, List[int]] = {}
for relation_record in relation_records:
if not relation_record.extra_data:
continue
city_id = relation_record.extra_data.get("city_id")
cable_id = relation_record.extra_data.get("cable_id")
if city_id is None or cable_id is None:
continue
city_to_cable_ids_map.setdefault(city_id, [])
if cable_id not in city_to_cable_ids_map[city_id]:
city_to_cable_ids_map[city_id].append(cable_id)
cable_id_to_name_map: Dict[int, str] = {}
for cable_record in cable_records:
if not cable_record.extra_data:
continue
cable_id = cable_record.extra_data.get("cable_id")
cable_name = cable_record.name
if cable_id and cable_name:
cable_id_to_name_map[cable_id] = cable_name
return city_to_cable_ids_map, cable_id_to_name_map
def _filter_known_records(records: List[CollectedData]) -> List[CollectedData]:
return [record for record in records if record.name != "Unknown"]
def convert_supercomputer_to_geojson(records: List[CollectedData]) -> Dict[str, Any]:
@@ -722,9 +768,7 @@ def convert_bgp_incidents_to_geojson(
async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
"""获取海底电缆 GeoJSON 数据 (LineString)"""
try:
stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
result = await db.execute(stmt)
records = dedupe_collected_records(list(result.scalars().all()))
records = await _load_current_collected_data(db, "arcgis_cables")
if not records:
raise HTTPException(
@@ -742,36 +786,25 @@ async def get_cables_geojson(db: AsyncSession = Depends(get_db)):
@router.get("/geo/landing-points")
async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
try:
landing_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
landing_result = await db.execute(landing_stmt)
records = dedupe_collected_records(list(landing_result.scalars().all()))
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
relation_result = await db.execute(relation_stmt)
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
cable_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
cable_result = await db.execute(cable_stmt)
cable_records = dedupe_collected_records(list(cable_result.scalars().all()))
city_to_cable_ids_map = {}
for rel in relation_records:
if rel.extra_data:
city_id = rel.extra_data.get("city_id")
cable_id = rel.extra_data.get("cable_id")
if city_id is not None and cable_id is not None:
if city_id not in city_to_cable_ids_map:
city_to_cable_ids_map[city_id] = []
if cable_id not in city_to_cable_ids_map[city_id]:
city_to_cable_ids_map[city_id].append(cable_id)
cable_id_to_name_map = {}
for cable in cable_records:
if cable.extra_data:
cable_id = cable.extra_data.get("cable_id")
cable_name = cable.name
if cable_id and cable_name:
cable_id_to_name_map[cable_id] = cable_name
records_by_source = await _load_current_collected_data_by_sources(
db,
[
"arcgis_landing_points",
"arcgis_cable_landing_relation",
"arcgis_cables",
],
)
records = records_by_source.get("arcgis_landing_points", [])
relation_records = records_by_source.get(
"arcgis_cable_landing_relation",
[],
)
cable_records = records_by_source.get("arcgis_cables", [])
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
relation_records,
cable_records,
)
if not records:
raise HTTPException(
@@ -786,38 +819,67 @@ async def get_landing_points_geojson(db: AsyncSession = Depends(get_db)):
raise HTTPException(status_code=500, detail=f"Internal error: {str(e)}")
@router.get("/terrain/terrarium/{z}/{x}/{y}.png")
async def get_terrarium_tile(z: int, x: int, y: int):
"""Proxy Terrarium elevation tiles through the backend to avoid browser CORS issues."""
if z < 0 or x < 0 or y < 0:
raise HTTPException(status_code=400, detail="Invalid terrain tile coordinates")
url = TERRAIN_TILE_URL_TEMPLATE.format(z=z, x=x, y=y)
try:
async with httpx.AsyncClient(
timeout=20.0,
follow_redirects=True,
) as client:
upstream = await client.get(url)
upstream.raise_for_status()
except httpx.HTTPStatusError as exc:
raise HTTPException(
status_code=exc.response.status_code,
detail=f"Terrain tile upstream error: {exc.response.status_code}",
) from exc
except httpx.HTTPError as exc:
raise HTTPException(
status_code=502,
detail=f"Terrain tile fetch failed: {exc}",
) from exc
cache_control = upstream.headers.get("cache-control") or "public, max-age=86400"
etag = upstream.headers.get("etag")
last_modified = upstream.headers.get("last-modified")
headers = {
"Cache-Control": cache_control,
}
if etag:
headers["ETag"] = etag
if last_modified:
headers["Last-Modified"] = last_modified
return Response(
content=upstream.content,
media_type=upstream.headers.get("content-type", "image/png"),
headers=headers,
)
@router.get("/geo/all")
async def get_all_geojson(db: AsyncSession = Depends(get_db)):
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
cables_result = await db.execute(cables_stmt)
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
points_result = await db.execute(points_stmt)
points_records = dedupe_collected_records(list(points_result.scalars().all()))
relation_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
relation_result = await db.execute(relation_stmt)
relation_records = dedupe_collected_records(list(relation_result.scalars().all()))
city_to_cable_ids_map = {}
for rel in relation_records:
if rel.extra_data:
city_id = rel.extra_data.get("city_id")
cable_id = rel.extra_data.get("cable_id")
if city_id is not None and cable_id is not None:
if city_id not in city_to_cable_ids_map:
city_to_cable_ids_map[city_id] = []
if cable_id not in city_to_cable_ids_map[city_id]:
city_to_cable_ids_map[city_id].append(cable_id)
cable_id_to_name_map = {}
for cable in cables_records:
if cable.extra_data:
cable_id = cable.extra_data.get("cable_id")
cable_name = cable.name
if cable_id and cable_name:
cable_id_to_name_map[cable_id] = cable_name
records_by_source = await _load_current_collected_data_by_sources(
db,
[
"arcgis_cables",
"arcgis_landing_points",
"arcgis_cable_landing_relation",
],
)
cables_records = records_by_source.get("arcgis_cables", [])
points_records = records_by_source.get("arcgis_landing_points", [])
relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
relation_records,
cables_records,
)
cables = (
convert_cable_to_geojson(cables_records)
@@ -850,17 +912,12 @@ async def get_satellites_geojson(
db: AsyncSession = Depends(get_db),
):
"""获取卫星 TLE GeoJSON 数据"""
stmt = (
select(CollectedData)
.where(CollectedData.source == "celestrak_tle")
.where(CollectedData.name != "Unknown")
.order_by(CollectedData.id.desc())
records = await _load_current_collected_data(
db,
"celestrak_tle",
exclude_unknown_name=True,
limit=limit,
)
result = await db.execute(stmt)
records = dedupe_satellite_records(list(result.scalars().all()))
if limit is not None:
records = records[:limit]
if not records:
return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -878,15 +935,12 @@ async def get_supercomputers_geojson(
db: AsyncSession = Depends(get_db),
):
"""获取 TOP500 超算中心 GeoJSON 数据"""
stmt = (
select(CollectedData)
.where(CollectedData.source == "top500")
.where(CollectedData.name != "Unknown")
.order_by(CollectedData.id.desc())
records = await _load_current_collected_data(
db,
"top500",
exclude_unknown_name=True,
limit=limit,
)
result = await db.execute(stmt)
records = dedupe_collected_records(list(result.scalars().all()))
records = records[:limit]
if not records:
return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -904,15 +958,12 @@ async def get_gpu_clusters_geojson(
db: AsyncSession = Depends(get_db),
):
"""获取 GPU 集群 GeoJSON 数据"""
stmt = (
select(CollectedData)
.where(CollectedData.source == "epoch_ai_gpu")
.where(CollectedData.name != "Unknown")
.order_by(CollectedData.id.desc())
records = await _load_current_collected_data(
db,
"epoch_ai_gpu",
exclude_unknown_name=True,
limit=limit,
)
result = await db.execute(stmt)
records = dedupe_collected_records(list(result.scalars().all()))
records = records[:limit]
if not records:
return {"type": "FeatureCollection", "features": [], "count": 0}
@@ -990,37 +1041,27 @@ async def get_all_visualization_data(db: AsyncSession = Depends(get_db)):
- supercomputers: TOP500 超算
- gpu_clusters: GPU 集群
"""
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
cables_result = await db.execute(cables_stmt)
cables_records = dedupe_collected_records(list(cables_result.scalars().all()))
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
points_result = await db.execute(points_stmt)
points_records = dedupe_collected_records(list(points_result.scalars().all()))
satellites_stmt = (
select(CollectedData)
.where(CollectedData.source == "celestrak_tle")
.where(CollectedData.name != "Unknown")
records_by_source = await _load_current_collected_data_by_sources(
db,
[
"arcgis_cables",
"arcgis_landing_points",
"celestrak_tle",
"top500",
"epoch_ai_gpu",
],
)
satellites_result = await db.execute(satellites_stmt)
satellites_records = dedupe_satellite_records(list(satellites_result.scalars().all()))
supercomputers_stmt = (
select(CollectedData)
.where(CollectedData.source == "top500")
.where(CollectedData.name != "Unknown")
cables_records = records_by_source.get("arcgis_cables", [])
points_records = records_by_source.get("arcgis_landing_points", [])
satellites_records = _filter_known_records(
records_by_source.get("celestrak_tle", []),
)
supercomputers_result = await db.execute(supercomputers_stmt)
supercomputers_records = dedupe_collected_records(list(supercomputers_result.scalars().all()))
gpu_stmt = (
select(CollectedData)
.where(CollectedData.source == "epoch_ai_gpu")
.where(CollectedData.name != "Unknown")
supercomputers_records = _filter_known_records(
records_by_source.get("top500", []),
)
gpu_records = _filter_known_records(
records_by_source.get("epoch_ai_gpu", []),
)
gpu_result = await db.execute(gpu_stmt)
gpu_records = dedupe_collected_records(list(gpu_result.scalars().all()))
cables = (
convert_cable_to_geojson(cables_records)
@@ -1084,13 +1125,8 @@ async def get_cable_graph(db: AsyncSession) -> CableGraph:
global _cable_graph
if _cable_graph is None:
cables_stmt = select(CollectedData).where(CollectedData.source == "arcgis_cables")
cables_result = await db.execute(cables_stmt)
cables_records = list(cables_result.scalars().all())
points_stmt = select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
points_result = await db.execute(points_stmt)
points_records = list(points_result.scalars().all())
cables_records = await _load_current_collected_data(db, "arcgis_cables")
points_records = await _load_current_collected_data(db, "arcgis_landing_points")
cables_data = convert_cable_to_geojson(cables_records)
points_data = convert_landing_point_to_geojson(points_records)

View File

@@ -1,7 +1,6 @@
"""WebSocket API endpoints"""
import asyncio
import json
import logging
from datetime import UTC, datetime
from typing import Optional
@@ -12,9 +11,20 @@ from jose import jwt, JWTError
from app.core.config import settings
from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
from app.core.websocket.ue_scene import expand_scene_payload_for_transport, ue_scene_state_store
from app.db.session import async_session_factory
logger = logging.getLogger(__name__)
router = APIRouter()
SUPPORTED_CHANNELS = [
"gpu_clusters",
"submarine_cables",
"ixp_nodes",
"alerts",
"dashboard",
"datasource_tasks",
"ue_scene",
]
async def authenticate_token(token: str) -> Optional[dict]:
@@ -50,18 +60,12 @@ async def websocket_endpoint(
await websocket.send_json(
{
"type": "connection_established",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {
"connection_id": f"conn_{user_id}",
"server_version": settings.VERSION,
"heartbeat_interval": 30,
"supported_channels": [
"gpu_clusters",
"submarine_cables",
"ixp_nodes",
"alerts",
"dashboard",
"datasource_tasks",
],
"supported_channels": SUPPORTED_CHANNELS,
},
}
)
@@ -74,26 +78,79 @@ async def websocket_endpoint(
await websocket.send_json(
{
"type": "heartbeat",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {"action": "pong", "timestamp": to_iso8601_utc(datetime.now(UTC))},
}
)
elif data.get("type") == "subscribe":
channels = data.get("data", {}).get("channels", [])
requested_channels = data.get("data", {}).get("channels", [])
channels = manager.subscribe(websocket, requested_channels)
await websocket.send_json(
{
"type": "subscription_confirmed",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {"action": "subscribe", "channels": channels},
}
)
elif data.get("type") == "sync_request":
sync_data = data.get("data", {})
channel = sync_data.get("channel")
if channel == "ue_scene":
async with async_session_factory() as session:
scene_payloads = await ue_scene_state_store.get_sync_payloads(
session,
last_sequence=sync_data.get("last_sequence"),
reason=sync_data.get("reason"),
)
for scene_payload in scene_payloads:
for transport_payload in expand_scene_payload_for_transport(scene_payload):
await websocket.send_json(
{
"type": "data_frame",
"channel": "ue_scene",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": transport_payload,
}
)
else:
await websocket.send_json(
{
"type": "error",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {
"message": f"Unsupported sync channel: {channel}",
},
}
)
elif data.get("type") == "control_frame":
await websocket.send_json(
{"type": "control_acknowledged", "data": {"received": True}}
{
"type": "control_acknowledged",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {
"target": data.get("data", {}).get("target"),
"command": data.get("data", {}).get("command"),
"accepted": True,
},
}
)
else:
await websocket.send_json({"type": "ack", "data": {"received": True}})
await websocket.send_json(
{
"type": "ack",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {"received": True},
}
)
except asyncio.TimeoutError:
await websocket.send_json({"type": "heartbeat", "data": {"action": "ping"}})
await websocket.send_json(
{
"type": "heartbeat",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": {"action": "ping"},
}
)
except WebSocketDisconnect:
pass

View File

@@ -11,6 +11,7 @@ COLLECTOR_URL_KEYS = {
"fao_landing_points": "fao.landing_point_url",
"telegeography_cables": "telegeography.cable_url",
"telegeography_landing": "telegeography.landing_point_url",
"telegeography_systems": "telegeography.cable_url",
"huggingface_models": "huggingface.models_url",
"huggingface_datasets": "huggingface.datasets_url",
"huggingface_spaces": "huggingface.spaces_url",
@@ -23,11 +24,13 @@ COLLECTOR_URL_KEYS = {
"top500": "top500.url",
"epoch_ai_gpu": "epoch_ai.gpu_clusters_url",
"spacetrack_tle": "spacetrack.tle_query_url",
"celestrak_tle": "celestrak.base_url",
"ris_live_bgp": "ris_live.url",
"bgpstream_bgp": "bgpstream.url",
"iptoasn_prefix_geo": "iptoasn.combined_url",
"opengeofeed_prefix_geo": "opengeofeed.public_csv_url",
"nro_delegated_prefix_geo": "nro.delegated_stats_url",
"news_live_streams": "news_live_streams.channels_url",
}
@@ -41,18 +44,22 @@ class DataSourcesConfig:
with open(config_path, "r") as f:
self._yaml_config = yaml.safe_load(f) or {}
def get_yaml_url(self, collector_name: str) -> str:
key = COLLECTOR_URL_KEYS.get(collector_name, "")
def get_yaml_value(self, key: str):
if not key:
return ""
return None
parts = key.split(".")
value = self._yaml_config
for part in parts:
if isinstance(value, dict):
value = value.get(part, "")
value = value.get(part)
else:
return ""
return None
return value
def get_yaml_url(self, collector_name: str) -> str:
key = COLLECTOR_URL_KEYS.get(collector_name, "")
value = self.get_yaml_value(key)
return value if isinstance(value, str) else ""
async def get_url(self, collector_name: str, db) -> str:

View File

@@ -2,53 +2,95 @@
# All external data source URLs should be configured here
arcgis:
# ArcGIS 海缆 GeoJSON 查询接口
cable_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/2/query"
# ArcGIS 登陆点 GeoJSON 查询接口
landing_point_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/1/query"
# ArcGIS 海缆与登陆点关联关系查询接口
cable_landing_relation_url: "https://services.arcgis.com/6DIQcwlPy8knb6sg/ArcGIS/rest/services/SubmarineCables/FeatureServer/3/query"
fao:
# FAO 登陆点 CSV 下载地址
landing_point_url: "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
telegeography:
# TeleGeography 海缆/系统主数据源,当前使用 GitHub 镜像 JSON
cable_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
# TeleGeography 登陆点主数据源,当前使用 GitHub 镜像 JSON
landing_point_url: "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
# TeleGeography 历史 API 存档,用于 cable collector 的 fallback
archived_cable_url: "https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable"
# TeleGeography 官网页面,用于 cable collector 的最终 HTML 抓取 fallback
live_map_url: "https://www.submarinecablemap.com"
huggingface:
# Hugging Face 模型目录 API
models_url: "https://huggingface.co/api/models"
# Hugging Face 数据集目录 API
datasets_url: "https://huggingface.co/api/datasets"
# Hugging Face Spaces 目录 API
spaces_url: "https://huggingface.co/api/spaces"
cloudflare:
# Cloudflare Radar 设备类型摘要接口
radar_device_url: "https://api.cloudflare.com/client/v4/radar/http/summary/device_type"
# Cloudflare Radar 请求量时间序列接口
radar_traffic_url: "https://api.cloudflare.com/client/v4/radar/http/timeseries/requests"
# Cloudflare Radar 热点地理位置接口
radar_top_locations_url: "https://api.cloudflare.com/client/v4/radar/http/top/locations"
peeringdb:
# PeeringDB IXP API
ixp_url: "https://www.peeringdb.com/api/ix"
# PeeringDB Network API
network_url: "https://www.peeringdb.com/api/net"
# PeeringDB Facility API
facility_url: "https://www.peeringdb.com/api/fac"
top500:
# TOP500 榜单页面,用于主表抓取
url: "https://top500.org/lists/top500/list/2025/11/"
# TOP500 站点根地址,用于拼详情页链接
base_url: "https://top500.org"
epoch_ai:
# Epoch AI GPU Cluster 页面
gpu_clusters_url: "https://epoch.ai/data/gpu-clusters"
spacetrack:
# Space-Track 站点根地址,用于首页访问和登录地址推导
base_url: "https://www.space-track.org"
# Space-Track TLE 主查询接口
tle_query_url: "https://www.space-track.org/basicspacedata/query/class/gp/orderby/EPOCH%20desc/limit/1000/format/json"
celestrak:
# CelesTrak TLE 基础接口collector 会在其后拼接 GROUP / FORMAT 参数
base_url: "https://celestrak.org/NORAD/elements/gp.php"
ris_live:
# RIPE RIS Live 流式订阅地址
url: "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
bgpstream:
# CAIDA BGPStream Broker API
url: "https://broker.bgpstream.caida.org/v2"
iptoasn:
# IPtoASN prefix geography 合并数据下载地址
combined_url: "https://iptoasn.com/data/ip2asn-combined.tsv.gz"
opengeofeed:
# OpenGeoFeed 公共 geofeed CSV
public_csv_url: "https://opengeofeed.org/feed/public.csv"
nro:
# NRO delegated stats 下载地址
delegated_stats_url: "https://ftp.ripe.net/pub/stats/ripencc/nro-stats/latest/nro-delegated-stats"
news_live_streams:
# IPTV-org 频道元数据 JSON
channels_url: "https://iptv-org.github.io/api/channels.json"
# IPTV-org 频道播放流 JSON
streams_url: "https://iptv-org.github.io/api/streams.json"
# IPTV-org 台标 JSON
logos_url: "https://iptv-org.github.io/api/logos.json"

View File

@@ -155,6 +155,13 @@ DEFAULT_DATASOURCES = {
"priority": "P1",
"frequency_minutes": 1440,
},
"news_live_streams": {
"id": 26,
"name": "News Live Streams",
"module": "L4",
"priority": "P2",
"frequency_minutes": 720,
},
}
ID_TO_COLLECTOR = {info["id"]: name for name, info in DEFAULT_DATASOURCES.items()}

View File

@@ -1,11 +1,12 @@
"""Data broadcaster for WebSocket connections"""
"""Data broadcaster for WebSocket connections."""
import asyncio
from datetime import UTC, datetime
from typing import Dict, Any, Optional
from typing import Any, Dict
from app.core.time import to_iso8601_utc
from app.core.websocket.manager import manager
from app.db.session import async_session_factory
@@ -45,6 +46,30 @@ class DataBroadcaster:
pass
await asyncio.sleep(interval)
async def broadcast_ue_scene(self, interval: int = 5) -> None:
"""Broadcast UE scene updates for nDisplay primary nodes."""
from app.core.websocket.ue_scene import expand_scene_payload_for_transport, ue_scene_state_store
while self.running:
try:
async with async_session_factory() as session:
scene_payloads = await ue_scene_state_store.get_broadcast_payloads(session)
for scene_data in scene_payloads:
for transport_payload in expand_scene_payload_for_transport(scene_data):
await manager.broadcast(
{
"type": "data_frame",
"channel": "ue_scene",
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"data": transport_payload,
},
channel="ue_scene",
)
except Exception:
pass
await asyncio.sleep(interval)
async def broadcast_alert(self, alert: Dict[str, Any]):
"""Broadcast an alert to all connected clients"""
await manager.broadcast(
@@ -75,7 +100,7 @@ class DataBroadcaster:
"timestamp": to_iso8601_utc(datetime.now(UTC)),
"payload": data,
},
channel=channel if channel in manager.active_connections else "all",
channel=channel,
)
async def broadcast_datasource_task_update(self, data: Dict[str, Any]):
@@ -95,6 +120,7 @@ class DataBroadcaster:
if not self.running:
self.running = True
self.tasks["dashboard"] = asyncio.create_task(self.broadcast_stats(5))
self.tasks["ue_scene"] = asyncio.create_task(self.broadcast_ue_scene(5))
def stop(self):
"""Stop all broadcasters"""

View File

@@ -1,9 +1,7 @@
"""WebSocket Connection Manager"""
"""WebSocket connection manager with channel subscriptions."""
from typing import Dict, Optional, Set
import json
import asyncio
from typing import Dict, Set, Optional
from datetime import datetime
from fastapi import WebSocket
import redis.asyncio as redis
@@ -11,17 +9,25 @@ from app.core.config import settings
class ConnectionManager:
"""Manages WebSocket connections"""
"""Manage user connections and channel subscriptions."""
def __init__(self):
self.active_connections: Dict[str, Set[WebSocket]] = {} # user_id -> connections
def __init__(self) -> None:
self.user_connections: Dict[str, Set[WebSocket]] = {}
self.socket_users: Dict[WebSocket, str] = {}
self.channel_connections: Dict[str, Set[WebSocket]] = {}
self.socket_channels: Dict[WebSocket, Set[str]] = {}
self.redis_client: Optional[redis.Redis] = None
async def connect(self, websocket: WebSocket, user_id: str):
@property
def active_connections(self) -> Dict[str, Set[WebSocket]]:
"""Compatibility alias for existing callers."""
return self.user_connections
async def connect(self, websocket: WebSocket, user_id: str) -> None:
await websocket.accept()
if user_id not in self.active_connections:
self.active_connections[user_id] = set()
self.active_connections[user_id].add(websocket)
self.user_connections.setdefault(user_id, set()).add(websocket)
self.socket_users[websocket] = user_id
self.socket_channels.setdefault(websocket, set())
if self.redis_client is None:
redis_url = settings.REDIS_URL
@@ -35,32 +41,69 @@ class ConnectionManager:
decode_responses=True,
)
def disconnect(self, websocket: WebSocket, user_id: str):
if user_id in self.active_connections:
self.active_connections[user_id].discard(websocket)
if not self.active_connections[user_id]:
del self.active_connections[user_id]
def disconnect(self, websocket: WebSocket, user_id: str) -> None:
self.unsubscribe(websocket, list(self.socket_channels.get(websocket, set())))
async def send_personal_message(self, message: dict, user_id: str):
if user_id in self.active_connections:
for connection in self.active_connections[user_id]:
try:
await connection.send_json(message)
except Exception:
pass
if user_id in self.user_connections:
self.user_connections[user_id].discard(websocket)
if not self.user_connections[user_id]:
del self.user_connections[user_id]
async def broadcast(self, message: dict, channel: str = "all"):
self.socket_users.pop(websocket, None)
self.socket_channels.pop(websocket, None)
def subscribe(self, websocket: WebSocket, channels: list[str]) -> list[str]:
subscribed_channels: list[str] = []
socket_channel_set = self.socket_channels.setdefault(websocket, set())
for channel in channels:
normalized_channel = channel.strip()
if not normalized_channel:
continue
self.channel_connections.setdefault(normalized_channel, set()).add(websocket)
socket_channel_set.add(normalized_channel)
subscribed_channels.append(normalized_channel)
return subscribed_channels
def unsubscribe(self, websocket: WebSocket, channels: list[str]) -> None:
socket_channel_set = self.socket_channels.setdefault(websocket, set())
for channel in channels:
normalized_channel = channel.strip()
if not normalized_channel:
continue
if normalized_channel in self.channel_connections:
self.channel_connections[normalized_channel].discard(websocket)
if not self.channel_connections[normalized_channel]:
del self.channel_connections[normalized_channel]
socket_channel_set.discard(normalized_channel)
async def send_personal_message(self, message: dict, user_id: str) -> None:
for connection in list(self.user_connections.get(user_id, set())):
try:
await connection.send_json(message)
except Exception:
self.disconnect(connection, user_id)
async def broadcast(self, message: dict, channel: str = "all") -> None:
if channel == "all":
for user_id in self.active_connections:
await self.send_personal_message(message, user_id)
targets = list(self.socket_users.keys())
else:
await self.send_personal_message(message, channel)
targets = list(self.channel_connections.get(channel, set()))
async def close_all(self):
for user_id in self.active_connections:
for connection in self.active_connections[user_id]:
await connection.close()
self.active_connections.clear()
for connection in targets:
try:
await connection.send_json(message)
except Exception:
user_id = self.socket_users.get(connection)
if user_id is not None:
self.disconnect(connection, user_id)
async def close_all(self) -> None:
for websocket, user_id in list(self.socket_users.items()):
await websocket.close()
self.disconnect(websocket, user_id)
manager = ConnectionManager()

View File

@@ -0,0 +1,562 @@
"""UE scene state built from visualization aggregate output."""
import asyncio
import json
import zlib
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any, Dict, List
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
if TYPE_CHECKING:
from app.models.collected_data import CollectedData
DEFAULT_LAYER_ORDER = (
"satellites",
"supercomputers",
"gpu_clusters",
"submarine_cables",
"landing_points",
"bgp_anomalies",
"bgp_incidents",
"bgp_collectors",
"alerts",
)
FULL_RESYNC_REASONS = {"initial_connect", "manual_resync", "profile_changed"}
UE_SCENE_LAYER_CHUNK_ITEM_LIMITS = {
"satellites": 500,
"supercomputers": 200,
"gpu_clusters": 100,
"submarine_cables": 25,
"landing_points": 250,
"bgp_anomalies": 100,
"bgp_incidents": 100,
"bgp_collectors": 100,
"alerts": 100,
}
def _default_display_profile() -> Dict[str, Any]:
return {
"profile_id": "polarized-wall-a",
"stereo_mode": "polarized",
"screen_width_m": 3.0,
"screen_height_m": 2.0,
"target_refresh_hz": 120,
}
def _default_camera_state() -> Dict[str, Any]:
return {
"mode": "auto_cruise",
"path_id": "global_overview",
"fov": 42.0,
}
def _stable_json(value: Any) -> str:
return json.dumps(value, ensure_ascii=True, sort_keys=True, separators=(",", ":"))
def _entity_type_for_layer(layer_name: str) -> str:
return {
"satellites": "satellite",
"supercomputers": "supercomputer",
"gpu_clusters": "gpu_cluster",
"submarine_cables": "submarine_cable",
"landing_points": "landing_point",
"bgp_anomalies": "bgp_anomaly",
"bgp_incidents": "bgp_incident",
"bgp_collectors": "bgp_collector",
"alerts": "alert",
}[layer_name]
def _default_visual_for_layer(layer_name: str) -> Dict[str, Any]:
visuals = {
"satellites": {"style": "satellite_marker", "size": 0.7, "color": "#9BDBFF"},
"supercomputers": {"style": "supercomputer_marker", "size": 1.2, "color": "#FF6B6B"},
"gpu_clusters": {"style": "pulse_marker", "size": 1.0, "color": "#FF8C42"},
"submarine_cables": {"style": "cable_arc", "width": 2.0, "color": "#4ECDC4"},
"landing_points": {"style": "landing_point_marker", "size": 0.8, "color": "#45B7D1"},
"bgp_anomalies": {"style": "anomaly_marker", "size": 1.0, "color": "#FFB703"},
"bgp_incidents": {"style": "incident_marker", "size": 1.2, "color": "#E63946"},
"bgp_collectors": {"style": "collector_marker", "size": 0.9, "color": "#7B9ACC"},
"alerts": {"style": "alert_marker", "size": 1.0, "color": "#FFD166"},
}
return visuals[layer_name]
def _empty_scene_layers() -> Dict[str, Dict[str, Any]]:
return {
layer_name: {"revision": 0, "items": {}}
for layer_name in DEFAULT_LAYER_ORDER
}
def _empty_changes() -> Dict[str, Dict[str, List[Any]]]:
return {
layer_name: {"added": [], "updated": [], "removed": []}
for layer_name in DEFAULT_LAYER_ORDER
}
def _point_geo(coordinates: List[Any]) -> Dict[str, Any]:
lng = coordinates[0] if len(coordinates) > 0 else None
lat = coordinates[1] if len(coordinates) > 1 else None
alt = coordinates[2] if len(coordinates) > 2 else 0.0
return {"lat": lat, "lng": lng, "alt": alt}
def _path_geo(geometry_type: str, coordinates: Any) -> Dict[str, Any]:
if geometry_type == "LineString":
return {
"path": [
{"lat": point[1], "lng": point[0], "alt": point[2] if len(point) > 2 else 0.0}
for point in coordinates
if isinstance(point, list) and len(point) >= 2
]
}
if geometry_type == "MultiLineString":
return {
"segments": [
[
{"lat": point[1], "lng": point[0], "alt": point[2] if len(point) > 2 else 0.0}
for point in line
if isinstance(point, list) and len(point) >= 2
]
for line in coordinates
if isinstance(line, list)
]
}
return {}
def _item_identifier(layer_name: str, feature: Dict[str, Any], index: int) -> str:
properties = feature.get("properties") or {}
feature_id = feature.get("id") or properties.get("id") or properties.get("source_id")
return f"{layer_name}:{feature_id or index}"
def _item_revision(item: Dict[str, Any]) -> int:
return zlib.crc32(_stable_json(item).encode("utf-8")) & 0xFFFFFFFF
def _layer_revision(items: Dict[str, Dict[str, Any]]) -> int:
if not items:
return 0
joined = "|".join(
f"{item_id}:{items[item_id]['revision']}"
for item_id in sorted(items)
)
return zlib.crc32(joined.encode("utf-8")) & 0xFFFFFFFF
def _serialize_feature(layer_name: str, feature: Dict[str, Any], index: int) -> Dict[str, Any]:
properties = dict(feature.get("properties") or {})
geometry = feature.get("geometry") or {}
geometry_type = geometry.get("type", "")
coordinates = geometry.get("coordinates") or []
item_id = _item_identifier(layer_name, feature, index)
geo: Dict[str, Any] = {}
if geometry_type == "Point":
geo = _point_geo(coordinates)
elif geometry_type in {"LineString", "MultiLineString"}:
geo = _path_geo(geometry_type, coordinates)
title = (
properties.get("name")
or properties.get("Name")
or properties.get("title")
or item_id
)
subtitle_parts = [
properties.get("city"),
properties.get("country"),
properties.get("region"),
]
item = {
"id": item_id,
"entity_type": _entity_type_for_layer(layer_name),
"geo": geo,
"visual": {
**_default_visual_for_layer(layer_name),
**({"color": properties["color"]} if properties.get("color") else {}),
},
"metrics": properties,
"labels": {
"title": title,
"subtitle": ", ".join([part for part in subtitle_parts if part]),
},
"status": {
"health": properties.get("status", "normal"),
"alert_level": properties.get("severity", "none"),
},
}
item["revision"] = _item_revision(item)
return item
async def build_visualization_scene_state(db: AsyncSession) -> Dict[str, Any]:
from app.api.v1.visualization import (
_build_landing_point_cable_maps,
_filter_known_records,
_load_current_collected_data_by_sources,
build_anomaly_geography_hints,
build_incident_geography_hints,
convert_bgp_anomalies_to_geojson,
convert_bgp_collectors_to_geojson,
convert_bgp_incidents_to_geojson,
convert_cable_to_geojson,
convert_gpu_cluster_to_geojson,
convert_landing_point_to_geojson,
convert_satellite_to_geojson,
convert_supercomputer_to_geojson,
)
from app.services.bgp_collectors import build_bgp_collector_coverage
records_by_source = await _load_current_collected_data_by_sources(
db,
[
"arcgis_cables",
"arcgis_landing_points",
"arcgis_cable_landing_relation",
"celestrak_tle",
"top500",
"epoch_ai_gpu",
],
)
cables_records = records_by_source.get("arcgis_cables", [])
landing_point_records = records_by_source.get("arcgis_landing_points", [])
relation_records = records_by_source.get("arcgis_cable_landing_relation", [])
satellites_records = _filter_known_records(records_by_source.get("celestrak_tle", []))
supercomputer_records = _filter_known_records(records_by_source.get("top500", []))
gpu_records = _filter_known_records(records_by_source.get("epoch_ai_gpu", []))
bgp_anomalies_result = await db.execute(
select(BGPAnomaly)
.where(BGPAnomaly.status == "active")
.order_by(BGPAnomaly.created_at.desc())
.limit(200)
)
bgp_anomalies = list(bgp_anomalies_result.scalars().all())
bgp_anomaly_geography_hints = await build_anomaly_geography_hints(db, bgp_anomalies)
bgp_incidents_result = await db.execute(
select(BGPIncident)
.where(BGPIncident.status == "active")
.order_by(BGPIncident.created_at.desc())
.limit(100)
)
bgp_incidents = list(bgp_incidents_result.scalars().all())
bgp_incident_geography_hints = await build_incident_geography_hints(db, bgp_incidents)
bgp_collector_coverage = await build_bgp_collector_coverage(
db,
source_filter=("ris_live_bgp", "bgpstream_bgp"),
)
bgp_coverage_by_collector = {
item["collector"]: item
for item in bgp_collector_coverage
if item.get("collector")
}
city_to_cable_ids_map, cable_id_to_name_map = _build_landing_point_cable_maps(
relation_records,
cables_records,
)
aggregate = {
"satellites": convert_satellite_to_geojson(satellites_records),
"supercomputers": convert_supercomputer_to_geojson(supercomputer_records),
"gpu_clusters": convert_gpu_cluster_to_geojson(gpu_records),
"submarine_cables": convert_cable_to_geojson(cables_records),
"landing_points": convert_landing_point_to_geojson(
landing_point_records,
city_to_cable_ids_map,
cable_id_to_name_map,
),
"bgp_anomalies": convert_bgp_anomalies_to_geojson(
bgp_anomalies,
bgp_anomaly_geography_hints,
),
"bgp_incidents": convert_bgp_incidents_to_geojson(
bgp_incidents,
bgp_incident_geography_hints,
),
"bgp_collectors": convert_bgp_collectors_to_geojson(bgp_coverage_by_collector),
"alerts": {"type": "FeatureCollection", "features": []},
}
layers = _empty_scene_layers()
total_records = 0
for layer_name in DEFAULT_LAYER_ORDER:
feature_collection = aggregate.get(layer_name) or {"features": []}
items = {
item["id"]: item
for index, feature in enumerate(feature_collection.get("features", []), start=1)
for item in [_serialize_feature(layer_name, feature, index)]
}
layers[layer_name]["items"] = items
layers[layer_name]["revision"] = _layer_revision(items)
total_records += len(items)
timestamp = to_iso8601_utc(datetime.now(UTC))
state_hash = zlib.crc32(
_stable_json(
{
layer_name: {
item_id: item["revision"]
for item_id, item in layers[layer_name]["items"].items()
}
for layer_name in DEFAULT_LAYER_ORDER
}
).encode("utf-8")
) & 0xFFFFFFFF
return {
"generated_at": timestamp,
"state_hash": state_hash,
"total_records": total_records,
"layers": layers,
}
def _changes_exist(changes: Dict[str, Dict[str, List[Any]]]) -> bool:
return any(
layer_changes["added"] or layer_changes["updated"] or layer_changes["removed"]
for layer_changes in changes.values()
)
def build_incremental_changes(
previous_state: Dict[str, Any],
current_state: Dict[str, Any],
) -> Dict[str, Dict[str, List[Any]]]:
changes = _empty_changes()
for layer_name in DEFAULT_LAYER_ORDER:
previous_items = previous_state["layers"][layer_name]["items"]
current_items = current_state["layers"][layer_name]["items"]
previous_ids = set(previous_items)
current_ids = set(current_items)
for added_id in sorted(current_ids - previous_ids):
changes[layer_name]["added"].append(current_items[added_id])
for removed_id in sorted(previous_ids - current_ids):
changes[layer_name]["removed"].append(removed_id)
for common_id in sorted(previous_ids & current_ids):
if _stable_json(previous_items[common_id]) != _stable_json(current_items[common_id]):
changes[layer_name]["updated"].append(current_items[common_id])
return changes
def _full_payload_from_state(state: Dict[str, Any], sequence: int) -> Dict[str, Any]:
return {
"update_type": "full",
"sequence": sequence,
"cluster_time": state["generated_at"],
"display_profile": _default_display_profile(),
"camera_state": _default_camera_state(),
"payload": {
"meta": {
"generated_at": state["generated_at"],
"total_records": state["total_records"],
"state_hash": state["state_hash"],
},
"layers": {
layer_name: {
"revision": layer_data["revision"],
"items": [
layer_data["items"][item_id]
for item_id in sorted(layer_data["items"])
],
}
for layer_name, layer_data in state["layers"].items()
},
},
}
def _incremental_payload(
*,
current_state: Dict[str, Any],
sequence: int,
base_sequence: int,
changes: Dict[str, Dict[str, List[Any]]],
) -> Dict[str, Any]:
return {
"update_type": "incremental",
"sequence": sequence,
"base_sequence": base_sequence,
"cluster_time": current_state["generated_at"],
"changes": changes,
"meta": {
"generated_at": current_state["generated_at"],
"total_records": current_state["total_records"],
"state_hash": current_state["state_hash"],
},
}
def _noop_incremental_payload(state: Dict[str, Any], sequence: int) -> Dict[str, Any]:
return _incremental_payload(
current_state=state,
sequence=sequence,
base_sequence=sequence,
changes=_empty_changes(),
)
def expand_scene_payload_for_transport(scene_payload: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Split oversized scene payloads into smaller transport-safe chunks."""
update_type = scene_payload.get("update_type")
if update_type != "full":
return [scene_payload]
payload = scene_payload.get("payload") or {}
layers = payload.get("layers") or {}
if not layers:
return [scene_payload]
chunks: List[Dict[str, Any]] = []
for layer_name in DEFAULT_LAYER_ORDER:
layer_data = layers.get(layer_name) or {}
items = list(layer_data.get("items") or [])
if not items:
continue
chunk_size = UE_SCENE_LAYER_CHUNK_ITEM_LIMITS.get(layer_name, 100)
total_layer_chunks = max(1, (len(items) + chunk_size - 1) // chunk_size)
for chunk_index, offset in enumerate(range(0, len(items), chunk_size), start=1):
chunk_items = items[offset : offset + chunk_size]
chunks.append(
{
**scene_payload,
"payload": {
"meta": {
**(payload.get("meta") or {}),
"chunked": True,
"layer_name": layer_name,
"layer_chunk_index": chunk_index,
"layer_chunk_count": total_layer_chunks,
},
"layers": {
layer_name: {
"revision": layer_data.get("revision", 0),
"items": chunk_items,
}
},
},
}
)
if not chunks:
return [scene_payload]
total_chunks = len(chunks)
for transport_index, chunk in enumerate(chunks, start=1):
chunk_payload = chunk.setdefault("payload", {})
chunk_meta = chunk_payload.setdefault("meta", {})
chunk_meta["transport_chunk_index"] = transport_index
chunk_meta["transport_chunk_count"] = total_chunks
return chunks
class UeSceneStateStore:
"""Maintain cached ue_scene state and a bounded incremental replay history."""
def __init__(self, history_limit: int = 20) -> None:
self.history_limit = history_limit
self.sequence = 0
self.state: Dict[str, Any] | None = None
self.history: List[Dict[str, Any]] = []
self.lock = asyncio.Lock()
async def _refresh_locked(self, db: AsyncSession) -> List[Dict[str, Any]]:
current_state = await build_visualization_scene_state(db)
if self.state is None:
self.sequence = 1
self.state = current_state
return [_full_payload_from_state(self.state, self.sequence)]
if current_state["state_hash"] == self.state["state_hash"]:
self.state = current_state
return []
previous_sequence = self.sequence
previous_state = self.state
self.sequence += 1
self.state = current_state
changes = build_incremental_changes(previous_state, current_state)
incremental = _incremental_payload(
current_state=current_state,
sequence=self.sequence,
base_sequence=previous_sequence,
changes=changes,
)
self.history.append(incremental)
if len(self.history) > self.history_limit:
self.history = self.history[-self.history_limit :]
return [incremental]
async def get_broadcast_payloads(self, db: AsyncSession) -> List[Dict[str, Any]]:
async with self.lock:
payloads = await self._refresh_locked(db)
return [payload for payload in payloads if payload["update_type"] == "full" or _changes_exist(payload.get("changes", {}))]
async def get_sync_payloads(
self,
db: AsyncSession,
*,
last_sequence: int | None,
reason: str | None,
) -> List[Dict[str, Any]]:
async with self.lock:
await self._refresh_locked(db)
if self.state is None:
return []
normalized_reason = (reason or "").strip()
if last_sequence is None or normalized_reason in FULL_RESYNC_REASONS:
return [_full_payload_from_state(self.state, self.sequence)]
if last_sequence == self.sequence:
return [_noop_incremental_payload(self.state, self.sequence)]
if last_sequence > self.sequence:
return [_full_payload_from_state(self.state, self.sequence)]
replay_payloads = [
payload
for payload in self.history
if payload["base_sequence"] >= last_sequence
and payload["sequence"] > last_sequence
]
if replay_payloads:
expected_base = last_sequence
ordered_payloads: List[Dict[str, Any]] = []
for payload in replay_payloads:
if payload["base_sequence"] != expected_base:
return [_full_payload_from_state(self.state, self.sequence)]
ordered_payloads.append(payload)
expected_base = payload["sequence"]
if ordered_payloads and ordered_payloads[-1]["sequence"] == self.sequence:
return ordered_payloads
return [_full_payload_from_state(self.state, self.sequence)]
ue_scene_state_store = UeSceneStateStore()

View File

@@ -95,6 +95,8 @@ async def init_db():
import app.models.bgp_observation # noqa: F401
import app.models.collected_data # noqa: F401
import app.models.system_setting # noqa: F401
import app.models.playground_session # noqa: F401
import app.models.playground_message # noqa: F401
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)

View File

@@ -9,6 +9,8 @@ from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
from app.models.system_setting import SystemSetting
from app.models.playground_session import PlaygroundSession
from app.models.playground_message import PlaygroundMessage
__all__ = [
"User",

View File

@@ -0,0 +1,40 @@
from sqlalchemy import JSON, Boolean, Column, DateTime, ForeignKey, Integer, String, Text
from sqlalchemy.sql import func
from app.db.session import Base
class PlaygroundMessage(Base):
__tablename__ = "playground_messages"
id = Column(Integer, primary_key=True, autoincrement=True)
public_id = Column(String(64), unique=True, index=True, nullable=False)
session_id = Column(Integer, ForeignKey("playground_sessions.id", ondelete="CASCADE"), nullable=False, index=True)
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
parent_message_id = Column(Integer, ForeignKey("playground_messages.id", ondelete="SET NULL"), nullable=True)
role = Column(String(20), nullable=False)
kind = Column(String(20), nullable=False, default="message")
status = Column(String(20), nullable=False, default="done")
title = Column(String(255), nullable=True)
content = Column(Text, nullable=False, default="")
thinking_content = Column(Text, nullable=False, default="")
meta = Column(JSON, nullable=False, default=list)
provider = Column(String(100), nullable=True)
model = Column(String(200), nullable=True)
request_id = Column(String(100), nullable=True)
raw_response = Column(JSON, nullable=False, default=dict)
content_blocks = Column(JSON, nullable=False, default=list)
text_blocks = Column(JSON, nullable=False, default=list)
thinking_blocks = Column(JSON, nullable=False, default=list)
sort_order = Column(Integer, nullable=False, default=0, index=True)
is_visible = Column(Boolean, nullable=False, default=True)
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
updated_at = Column(
DateTime(timezone=True),
server_default=func.now(),
onupdate=func.now(),
nullable=False,
)
def __repr__(self):
return f"<PlaygroundMessage public_id={self.public_id} role={self.role} status={self.status}>"

View File

@@ -0,0 +1,27 @@
from sqlalchemy import JSON, Column, DateTime, ForeignKey, Integer, String, UniqueConstraint
from sqlalchemy.sql import func
from app.db.session import Base
class PlaygroundSession(Base):
__tablename__ = "playground_sessions"
__table_args__ = (
UniqueConstraint("user_id", "session_key", name="uq_playground_sessions_user_session_key"),
)
id = Column(Integer, primary_key=True, autoincrement=True)
user_id = Column(Integer, ForeignKey("users.id", ondelete="CASCADE"), nullable=False, index=True)
session_key = Column(String(100), nullable=False, default="default")
title = Column(String(200), nullable=False, default="Playground 会话")
state = Column(JSON, nullable=False, default={})
created_at = Column(DateTime(timezone=True), server_default=func.now(), nullable=False)
updated_at = Column(
DateTime(timezone=True),
server_default=func.now(),
onupdate=func.now(),
nullable=False,
)
def __repr__(self):
return f"<PlaygroundSession user_id={self.user_id} session_key={self.session_key}>"

View File

@@ -3,6 +3,14 @@ from typing import Any
from pydantic import BaseModel, Field
class AIContentBlock(BaseModel):
type: str
text: str | None = None
thinking: str | None = None
signature: str | None = None
metadata: dict[str, Any] = Field(default_factory=dict)
class SituationalAnalysisRequest(BaseModel):
title: str = Field(..., min_length=1, max_length=200)
objective: str = Field(..., min_length=1, max_length=1000)
@@ -10,18 +18,158 @@ class SituationalAnalysisRequest(BaseModel):
observations: list[str] = Field(default_factory=list)
constraints: list[str] = Field(default_factory=list)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None
class BGPBriefRequest(BaseModel):
incident_limit: int = Field(default=5, ge=1, le=10)
anomaly_limit: int = Field(default=6, ge=1, le=12)
collector_limit: int = Field(default=5, ge=1, le=10)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None
class AlertBriefRequest(BaseModel):
alert_limit: int = Field(default=8, ge=1, le=20)
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None
class SituationalAlertBriefRequest(BaseModel):
preferred_model: str | None = Field(default=None, max_length=200)
thinking: dict[str, Any] | None = None
class SituationalAnalysisResponse(BaseModel):
provider: str
model: str
content: str
content_blocks: list[AIContentBlock] = Field(default_factory=list)
text_blocks: list[str] = Field(default_factory=list)
thinking_blocks: list[str] = Field(default_factory=list)
raw_response: dict[str, Any] = Field(default_factory=dict)
class BGPBriefRecordSummary(BaseModel):
id: str
title: str
provider: str
model: str
request_id: str | None = None
generated_at: str
class BGPBriefRecordResponse(BGPBriefRecordSummary):
content_markdown: str
facts: list[str] = Field(default_factory=list)
context: dict[str, Any] = Field(default_factory=dict)
class AlertBriefResponse(SituationalAnalysisResponse):
title: str
objective: str
facts: list[str] = Field(default_factory=list)
context: dict[str, Any] = Field(default_factory=dict)
class SituationalAlertBriefResponse(SituationalAnalysisResponse):
title: str
objective: str
facts: list[str] = Field(default_factory=list)
context: dict[str, Any] = Field(default_factory=dict)
class AIProviderStatusResponse(BaseModel):
provider: str
api: str | None = None
enabled: bool
configured: bool
model: str | None = None
base_url: str | None = None
class PlaygroundSessionState(BaseModel):
messages: list[dict[str, Any]] = Field(default_factory=list)
selectedPresetKey: str = Field(default="bgp-brief", max_length=100)
title: str = Field(default="", max_length=200)
objective: str = Field(default="", max_length=1000)
constraints: str = Field(default="")
inputValue: str = Field(default="")
analysis: dict[str, Any] | None = None
latestAnalysisMessageId: str | None = Field(default=None, max_length=200)
analysisMeta: dict[str, Any] = Field(default_factory=dict)
helpExpanded: bool = True
class PlaygroundSessionUpsertRequest(BaseModel):
session_key: str = Field(default="default", min_length=1, max_length=100)
title: str | None = Field(default=None, max_length=200)
state: PlaygroundSessionState
class PlaygroundMessageRecord(BaseModel):
id: str
role: str
kind: str = "message"
status: str = "done"
title: str | None = None
content: str = ""
thinking_content: str = ""
meta: list[str] = Field(default_factory=list)
markdown: bool = True
provider: str | None = None
model: str | None = None
request_id: str | None = None
raw_response: dict[str, Any] = Field(default_factory=dict)
content_blocks: list[dict[str, Any]] = Field(default_factory=list)
text_blocks: list[str] = Field(default_factory=list)
thinking_blocks: list[str] = Field(default_factory=list)
parent_message_id: str | None = None
created_at: str
updated_at: str
class PlaygroundSessionResponse(BaseModel):
id: str
session_key: str
title: str
state: PlaygroundSessionState
created_at: str
updated_at: str
class PlaygroundThreadResponse(BaseModel):
session: PlaygroundSessionResponse
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
class PlaygroundMessageCreateRequest(BaseModel):
session_key: str = Field(default="default", min_length=1, max_length=100)
title: str = Field(..., min_length=1, max_length=200)
objective: str = Field(..., min_length=1, max_length=1000)
constraints: str = Field(default="")
input: str = Field(..., min_length=1)
selected_preset_key: str = Field(default="bgp-brief", max_length=100)
help_expanded: bool = True
class PlaygroundMessageActionResponse(BaseModel):
session: PlaygroundSessionResponse
messages: list[PlaygroundMessageRecord] = Field(default_factory=list)
active_message_id: str | None = None
class PlaygroundMessageStopRequest(BaseModel):
session_key: str = Field(default="default", min_length=1, max_length=100)
message_id: str = Field(..., min_length=1, max_length=64)
class PlaygroundMessageResendRequest(BaseModel):
session_key: str = Field(default="default", min_length=1, max_length=100)
user_message_id: str = Field(..., min_length=1, max_length=64)
class PlaygroundMessageEditRequest(BaseModel):
session_key: str = Field(default="default", min_length=1, max_length=100)
user_message_id: str = Field(..., min_length=1, max_length=64)
content: str = Field(..., min_length=1)

View File

@@ -0,0 +1,5 @@
from pydantic import BaseModel, Field
class AlertResolutionRequest(BaseModel):
resolution: str = Field(..., min_length=1, max_length=1000)

View File

@@ -0,0 +1,103 @@
from __future__ import annotations
from collections import Counter
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.schemas.ai import AlertBriefRequest, SituationalAnalysisRequest
def _format_counter(counter: Counter[str], empty_text: str = "") -> str:
if not counter:
return empty_text
return "".join(f"{key} {value}" for key, value in counter.items())
async def build_alert_brief_request(
db: AsyncSession,
*,
alert_limit: int = 8,
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
recent_alerts_result = await db.execute(
select(Alert)
.order_by(Alert.created_at.desc(), Alert.id.desc())
.limit(max(alert_limit, 1))
)
total_result = await db.execute(select(func.count(Alert.id)))
active_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE))
acknowledged_result = await db.execute(
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACKNOWLEDGED)
)
resolved_result = await db.execute(select(func.count(Alert.id)).where(Alert.status == AlertStatus.RESOLVED))
recent_alerts = recent_alerts_result.scalars().all()
total_alerts = total_result.scalar() or 0
active_alerts = active_result.scalar() or 0
acknowledged_alerts = acknowledged_result.scalar() or 0
resolved_alerts = resolved_result.scalar() or 0
severity_counts = Counter((item.severity.value if item.severity else "unknown") for item in recent_alerts)
status_counts = Counter((item.status.value if item.status else "unknown") for item in recent_alerts)
datasource_counts = Counter((item.datasource_name or "未命名数据源") for item in recent_alerts)
active_datasource_counts = Counter(
(item.datasource_name or "未命名数据源")
for item in recent_alerts
if item.status == AlertStatus.ACTIVE
)
facts = [
f"告警总量 {total_alerts} 条,其中 active {active_alerts} 条、acknowledged {acknowledged_alerts} 条、resolved {resolved_alerts} 条。",
f"最近告警严重度分布:{_format_counter(severity_counts)}",
f"最近告警状态分布:{_format_counter(status_counts)}",
f"最近告警数据源分布:{_format_counter(Counter(dict(datasource_counts.most_common(6))))}",
]
if active_datasource_counts:
facts.append(
"当前待处理告警主要集中在:"
+ _format_counter(Counter(dict(active_datasource_counts.most_common(5))))
+ ""
)
if recent_alerts:
facts.append(
"最近告警摘录:"
+ "".join(
[
f"{item.datasource_name or '未命名数据源'} / {item.severity.value if item.severity else '-'} / {item.status.value if item.status else '-'} / {item.message or '-'}"
for item in recent_alerts[:6]
]
)
)
context = {
"source": "alerts",
"total_alerts": total_alerts,
"active_alerts": active_alerts,
"acknowledged_alerts": acknowledged_alerts,
"resolved_alerts": resolved_alerts,
"severity_distribution": dict(severity_counts),
"status_distribution": dict(status_counts),
"top_datasources": dict(datasource_counts.most_common(6)),
"top_active_datasources": dict(active_datasource_counts.most_common(5)),
}
return (
SituationalAnalysisRequest(
title="告警态势 AI 简报",
objective="基于当前告警总量、严重度、状态、数据源分布与最近告警摘录,生成一份面向值班人员的简明告警态势简报,突出待处理风险、告警集中点和优先动作。",
observations=facts,
constraints=[
"明确区分事实、推断与建议。",
"优先指出仍处于 active 状态且高严重度的告警簇。",
"不要把 acknowledged 或 resolved 告警误判成当前仍在扩大。",
"如果证据不足,请明确指出缺失的上下文。",
],
context=context,
),
facts,
context,
)

View File

@@ -0,0 +1,259 @@
from __future__ import annotations
from collections import Counter
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.v1.bgp import BGP_SOURCES
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.models.bgp_observation import BGPObservation
from app.schemas.ai import SituationalAnalysisRequest
from app.services.bgp_collectors import build_bgp_collector_coverage
from app.services.bgp_enrichment import lookup_prefix_geography
def _format_counter(counter: dict[str, int], empty_text: str = "") -> str:
if not counter:
return empty_text
return "".join(f"{key} {value}" for key, value in counter.items())
def _severity_rank(value: str | None) -> int:
order = {
"critical": 0,
"high": 1,
"medium": 2,
"low": 3,
"info": 4,
}
return order.get((value or "").lower(), 99)
def _normalize_geo_key(country: str | None, city: str | None) -> str:
if city and country:
return f"{city}, {country}"
return city or country or "未知区域"
def _top_counter_items(counter: Counter[str], limit: int = 5) -> dict[str, int]:
return {name: count for name, count in counter.most_common(limit) if name}
def _collect_incident_regions(incidents: list[BGPIncident]) -> Counter[str]:
counter: Counter[str] = Counter()
for item in incidents:
for region in item.affected_regions or []:
if not isinstance(region, dict):
continue
counter[_normalize_geo_key(region.get("country"), region.get("city"))] += 1
return counter
def _collect_collector_regions(collectors: list[dict[str, Any]]) -> Counter[str]:
counter: Counter[str] = Counter()
for item in collectors:
counter[_normalize_geo_key(item.get("country"), item.get("city"))] += int(item.get("recent_24h_observation_count") or 0)
return counter
def _format_geo_evidence(prefix_geographies: dict[str, dict[str, Any]], limit: int = 6) -> str:
if not prefix_geographies:
return "没有命中 prefix geography 证据。"
rows = []
for prefix, item in list(prefix_geographies.items())[:limit]:
region = _normalize_geo_key(item.get("country"), item.get("city"))
source = item.get("source") or item.get("geography_mode") or "unknown"
as_hint = item.get("asn")
as_name = item.get("as_name")
as_text = ""
if as_hint:
as_text = f" / ASN AS{as_hint}"
if as_name:
as_text += f" ({as_name})"
rows.append(f"{prefix} -> {region} / 来源 {source}{as_text}")
return "".join(rows)
async def build_bgp_brief_request(
db: AsyncSession,
*,
incident_limit: int = 5,
anomaly_limit: int = 6,
collector_limit: int = 5,
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, int | str | dict[str, int]]]:
incidents_result = await db.execute(
select(BGPIncident)
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
.limit(max(incident_limit, 1))
)
anomalies_result = await db.execute(
select(BGPAnomaly)
.order_by(BGPAnomaly.created_at.desc(), BGPAnomaly.id.desc())
.limit(max(anomaly_limit, 1))
)
observations_result = await db.execute(
select(BGPObservation).where(BGPObservation.source.in_(BGP_SOURCES))
)
incident_count_result = await db.execute(select(func.count(BGPIncident.id)))
anomaly_count_result = await db.execute(select(func.count(BGPAnomaly.id)))
incidents = incidents_result.scalars().all()
anomalies = anomalies_result.scalars().all()
observations = observations_result.scalars().all()
collectors = await build_bgp_collector_coverage(db, source_filter=BGP_SOURCES)
total_incidents = incident_count_result.scalar() or 0
total_anomalies = anomaly_count_result.scalar() or 0
total_observations = len(observations)
active_collectors = [item for item in collectors if item["observation_count"] > 0]
incident_status_counts = Counter((item.status or "unknown") for item in incidents)
incident_severity_counts = Counter((item.severity or "unknown") for item in incidents)
incident_type_counts = Counter((item.incident_type or "unknown") for item in incidents)
anomaly_type_counts = Counter((item.anomaly_type or "unknown") for item in anomalies)
event_type_counts = Counter((item.event_type or "unknown") for item in observations)
incident_region_counts = _collect_incident_regions(incidents)
top_collectors = sorted(
active_collectors,
key=lambda item: (
-int(item["recent_24h_observation_count"]),
-int(item["observation_count"]),
str(item["collector"]),
),
)[: max(collector_limit, 1)]
collector_region_counts = _collect_collector_regions(top_collectors)
prefix_candidates = sorted(
{
prefix
for item in incidents
for prefix in (item.affected_prefixes or [])
if prefix
}
| {item.prefix for item in anomalies if item.prefix}
)
prefix_geographies = await lookup_prefix_geography(db, prefix_candidates) if prefix_candidates else {}
geography_region_counts = Counter(
_normalize_geo_key(item.get("country"), item.get("city"))
for item in prefix_geographies.values()
if item.get("country") or item.get("city")
)
hotspot_region_counts = geography_region_counts + incident_region_counts
collector_bias_regions = [
region
for region, count in collector_region_counts.most_common(3)
if count > hotspot_region_counts.get(region, 0)
]
observations_lines: list[str] = [
f"当前共有 {total_incidents} 起 BGP incidents、{total_anomalies} 条 anomalies、{total_observations} 条原始观测事件。",
f"活跃观测站 {len(active_collectors)} 个;近 24 小时事件数合计 {sum(int(item['recent_24h_observation_count']) for item in active_collectors)}",
f"最近 incidents 严重度分布:{_format_counter(dict(sorted(incident_severity_counts.items(), key=lambda item: _severity_rank(item[0]))))}",
f"最近 incidents 状态分布:{_format_counter(dict(incident_status_counts))}",
f"最近 incidents 类型分布:{_format_counter(dict(incident_type_counts.most_common(5)))}",
f"最近 anomalies 类型分布:{_format_counter(dict(anomaly_type_counts.most_common(6)))}",
f"观测事件类型分布:{_format_counter(dict(event_type_counts.most_common(6)))}",
]
if hotspot_region_counts:
observations_lines.append(
"区域热点事实层:"
+ _format_counter(_top_counter_items(hotspot_region_counts, limit=5), empty_text="无明显区域聚集")
+ ""
)
if prefix_geographies:
observations_lines.append("Prefix geography 证据:" + _format_geo_evidence(prefix_geographies))
if collector_bias_regions:
observations_lines.append(
"观测偏差提示:重点观测站最近 24h 活跃度更集中在 "
+ "".join(collector_bias_regions)
+ ",这些区域的事件升温结论需要结合 prefix geography 与 affected regions 交叉验证。"
)
elif top_collectors:
observations_lines.append(
"观测偏差提示:当前未发现明显高于区域热点事实层的单一观测站集中区域,但仍需区分 collector coverage 与真实区域风险。"
)
if incidents:
observations_lines.append(
"最近 incident 摘要:" + "".join(
[
f"{item.incident_type} / {item.severity} / {item.status}"
f" / 前缀 {', '.join(item.affected_prefixes[:2]) if item.affected_prefixes else '-'}"
f" / 观测站 {len(item.affected_collectors or [])}"
for item in incidents
]
)
)
if anomalies:
observations_lines.append(
"最近 anomaly 摘要:" + "".join(
[
f"{item.anomaly_type} / {item.severity}"
f" / 前缀 {item.prefix or '-'}"
f" / ASN {item.new_origin_asn or item.origin_asn or '-'}"
for item in anomalies
]
)
)
if top_collectors:
observations_lines.append(
"重点观测站:" + "".join(
[
f"{item['collector']} ({', '.join([part for part in [item.get('city'), item.get('country')] if part]) or '未知位置'})"
f" / 近24h {item['recent_24h_observation_count']}"
f" / 前缀 {item['prefix_count']}"
for item in top_collectors
]
)
)
context = {
"source": "bgp-overview",
"incident_total": total_incidents,
"anomaly_total": total_anomalies,
"observation_total": total_observations,
"active_collectors": len(active_collectors),
"top_incident_types": dict(incident_type_counts.most_common(5)),
"top_anomaly_types": dict(anomaly_type_counts.most_common(6)),
"top_event_types": dict(event_type_counts.most_common(6)),
"region_hotspots": _top_counter_items(hotspot_region_counts, limit=6),
"incident_regions": _top_counter_items(incident_region_counts, limit=6),
"collector_bias_regions": collector_bias_regions,
"prefix_geography_sources": dict(
Counter(str(item.get("source") or "unknown") for item in prefix_geographies.values()).most_common(5)
),
"prefix_geography_sample": {
prefix: {
"country": item.get("country"),
"city": item.get("city"),
"source": item.get("source"),
"asn": item.get("asn"),
"as_name": item.get("as_name"),
}
for prefix, item in list(prefix_geographies.items())[:8]
},
}
return SituationalAnalysisRequest(
title="BGP 态势 AI 简报",
objective="基于当前 BGP incidents、anomalies、原始观测事件、观测站覆盖与 prefix geography 证据,生成一份面向操作员的简明态势简报,突出区域热点、观测偏差、当前风险、证据和优先动作。",
observations=observations_lines,
constraints=[
"明确区分事实、推断与建议。",
"优先指出需要立即关注的高严重度 incident 或异常模式。",
"需要单独指出哪些区域结论来自 prefix geography / affected regions哪些可能受 collector coverage 偏差影响。",
"结论应服务值班排障,不要写成泛泛的模型演示文案。",
"如果证据不足,要明确指出缺失数据。",
],
context=context,
), observations_lines, context

View File

@@ -0,0 +1,160 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from uuid import uuid4
from app.core.config import ROOT_DIR
from app.schemas.ai import BGPBriefRecordResponse, BGPBriefRecordSummary, SituationalAnalysisResponse
_BRIEF_STORAGE_DIR = ROOT_DIR / "data" / "ai" / "bgp-briefs"
_METADATA_PREFIX = "<!-- planet-bgp-brief-meta "
_METADATA_SUFFIX = " -->"
_BRIEF_TITLE = "BGP AI 简报"
@dataclass(slots=True)
class _StoredBrief:
id: str
title: str
provider: str
model: str
request_id: str | None
generated_at: str
content_markdown: str
facts: list[str]
context: dict[str, Any]
path: Path
def _ensure_storage_dir() -> Path:
_BRIEF_STORAGE_DIR.mkdir(parents=True, exist_ok=True)
return _BRIEF_STORAGE_DIR
def _build_metadata_line(metadata: dict[str, Any]) -> str:
return f"{_METADATA_PREFIX}{json.dumps(metadata, ensure_ascii=False)}{_METADATA_SUFFIX}"
def _parse_brief_file(path: Path) -> _StoredBrief | None:
try:
raw_text = path.read_text(encoding="utf-8")
except OSError:
return None
first_line, separator, remainder = raw_text.partition("\n")
if not separator or not first_line.startswith(_METADATA_PREFIX) or not first_line.endswith(_METADATA_SUFFIX):
return None
metadata_payload = first_line[len(_METADATA_PREFIX) : -len(_METADATA_SUFFIX)]
try:
metadata = json.loads(metadata_payload)
except json.JSONDecodeError:
return None
return _StoredBrief(
id=str(metadata.get("id") or path.stem),
title=str(metadata.get("title") or _BRIEF_TITLE),
provider=str(metadata.get("provider") or "-"),
model=str(metadata.get("model") or "-"),
request_id=metadata.get("request_id"),
generated_at=str(metadata.get("generated_at") or datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat()),
content_markdown=remainder.lstrip("\n"),
facts=list(metadata.get("facts") or []),
context=dict(metadata.get("context") or {}),
path=path,
)
def list_bgp_brief_records(limit: int = 50) -> list[BGPBriefRecordSummary]:
storage_dir = _ensure_storage_dir()
records: list[_StoredBrief] = []
for path in storage_dir.glob("*.md"):
parsed = _parse_brief_file(path)
if parsed is not None:
records.append(parsed)
records.sort(key=lambda item: item.generated_at, reverse=True)
return [
BGPBriefRecordSummary(
id=item.id,
title=item.title,
provider=item.provider,
model=item.model,
request_id=item.request_id,
generated_at=item.generated_at,
)
for item in records[: max(limit, 1)]
]
def get_bgp_brief_record(brief_id: str) -> BGPBriefRecordResponse | None:
path = _ensure_storage_dir() / f"{brief_id}.md"
parsed = _parse_brief_file(path)
if parsed is None:
return None
return BGPBriefRecordResponse(
id=parsed.id,
title=parsed.title,
provider=parsed.provider,
model=parsed.model,
request_id=parsed.request_id,
generated_at=parsed.generated_at,
content_markdown=parsed.content_markdown,
facts=parsed.facts,
context=parsed.context,
)
def get_latest_bgp_brief_record() -> BGPBriefRecordResponse | None:
summaries = list_bgp_brief_records(limit=1)
if not summaries:
return None
return get_bgp_brief_record(summaries[0].id)
def save_bgp_brief_record(
analysis: SituationalAnalysisResponse,
*,
request_id: str | None,
facts: list[str] | None = None,
context: dict[str, Any] | None = None,
generated_at: datetime | None = None,
) -> BGPBriefRecordResponse:
created_at = generated_at or datetime.now(UTC)
brief_id = f"{created_at.strftime('%Y%m%dT%H%M%SZ')}-{uuid4().hex[:8]}"
path = _ensure_storage_dir() / f"{brief_id}.md"
metadata = {
"id": brief_id,
"title": _BRIEF_TITLE,
"provider": analysis.provider,
"model": analysis.model,
"request_id": request_id,
"generated_at": created_at.isoformat(),
"facts": facts or [],
"context": context or {},
}
markdown_text = f"{_build_metadata_line(metadata)}\n\n{analysis.content.rstrip()}\n"
path.write_text(markdown_text, encoding="utf-8")
return BGPBriefRecordResponse(
id=brief_id,
title=_BRIEF_TITLE,
provider=analysis.provider,
model=analysis.model,
request_id=request_id,
generated_at=created_at.isoformat(),
content_markdown=analysis.content,
facts=facts or [],
context=context or {},
)

View File

@@ -6,7 +6,7 @@ from collections import defaultdict
from datetime import UTC, datetime, timedelta
from typing import Any
from sqlalchemy import select
from sqlalchemy import case, distinct, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
@@ -14,6 +14,16 @@ from app.models.bgp_observation import BGPObservation
from app.services.collectors.bgp_common import RIPE_RIS_COLLECTOR_COORDS
def _collector_base_filters(source_filter: tuple[str, ...] | None) -> list[Any]:
filters: list[Any] = [
BGPObservation.collector.isnot(None),
func.length(func.btrim(BGPObservation.collector)) > 0,
]
if source_filter:
filters.append(BGPObservation.source.in_(source_filter))
return filters
async def build_bgp_collector_coverage(
db: AsyncSession,
*,
@@ -24,88 +34,148 @@ async def build_bgp_collector_coverage(
recent_24h_threshold = now - timedelta(hours=24)
recent_7d_threshold = now - timedelta(days=7)
stmt = select(BGPObservation).order_by(BGPObservation.observed_at.desc(), BGPObservation.id.desc())
if source_filter:
stmt = stmt.where(BGPObservation.source.in_(source_filter))
filters = _collector_base_filters(source_filter)
country_expr = func.nullif(BGPObservation.collector_geo["country"].as_string(), "")
city_expr = func.nullif(BGPObservation.collector_geo["city"].as_string(), "")
result = await db.execute(stmt)
records = list(result.scalars().all())
aggregate_stmt = (
select(
BGPObservation.collector.label("collector"),
func.count(BGPObservation.id).label("observation_count"),
func.count(distinct(BGPObservation.prefix)).label("prefix_count"),
func.count(distinct(BGPObservation.origin_asn)).label("origin_asn_count"),
func.count(distinct(BGPObservation.peer_asn)).label("peer_asn_count"),
func.sum(case((BGPObservation.observed_at >= recent_15m_threshold, 1), else_=0)).label("recent_15m_observation_count"),
func.sum(case((BGPObservation.observed_at >= recent_24h_threshold, 1), else_=0)).label("recent_24h_observation_count"),
func.sum(case((BGPObservation.observed_at >= recent_7d_threshold, 1), else_=0)).label("recent_7d_observation_count"),
func.count(distinct(case((BGPObservation.observed_at >= recent_15m_threshold, BGPObservation.prefix), else_=None))).label("recent_15m_prefix_count"),
func.count(distinct(case((BGPObservation.observed_at >= recent_24h_threshold, BGPObservation.prefix), else_=None))).label("recent_24h_prefix_count"),
func.count(distinct(case((BGPObservation.observed_at >= recent_7d_threshold, BGPObservation.prefix), else_=None))).label("recent_7d_prefix_count"),
func.max(BGPObservation.observed_at).label("latest_observed_at"),
)
.where(*filters)
.group_by(BGPObservation.collector)
)
aggregate_rows = (await db.execute(aggregate_stmt)).all()
latest_subquery = (
select(
BGPObservation.collector.label("collector"),
BGPObservation.event_type.label("latest_event_type"),
country_expr.label("country"),
city_expr.label("city"),
func.row_number()
.over(
partition_by=BGPObservation.collector,
order_by=(BGPObservation.observed_at.desc(), BGPObservation.id.desc()),
)
.label("rn"),
)
.where(*filters)
.subquery()
)
latest_rows = (
await db.execute(
select(
latest_subquery.c.collector,
latest_subquery.c.latest_event_type,
latest_subquery.c.country,
latest_subquery.c.city,
).where(latest_subquery.c.rn == 1)
)
).all()
event_counts_subquery = (
select(
BGPObservation.collector.label("collector"),
BGPObservation.event_type.label("event_type"),
func.count(BGPObservation.id).label("count"),
func.row_number()
.over(
partition_by=BGPObservation.collector,
order_by=(func.count(BGPObservation.id).desc(), BGPObservation.event_type.asc()),
)
.label("rn"),
)
.where(*filters)
.group_by(BGPObservation.collector, BGPObservation.event_type)
.subquery()
)
top_event_rows = (
await db.execute(
select(
event_counts_subquery.c.collector,
event_counts_subquery.c.event_type,
event_counts_subquery.c.count,
).where(event_counts_subquery.c.rn <= 3)
)
).all()
scope_rows = (
await db.execute(
select(
BGPObservation.collector.label("collector"),
country_expr.label("country"),
city_expr.label("city"),
)
.where(*filters)
.distinct()
)
).all()
latest_by_collector = {
row.collector: {
"latest_event_type": row.latest_event_type,
"country": row.country,
"city": row.city,
}
for row in latest_rows
}
scope_by_collector: dict[str, dict[str, set[str]]] = defaultdict(lambda: {"countries": set(), "cities": set()})
for row in scope_rows:
if row.country:
scope_by_collector[row.collector]["countries"].add(row.country)
if row.city:
scope_by_collector[row.collector]["cities"].add(row.city)
top_events_by_collector: dict[str, list[dict[str, Any]]] = defaultdict(list)
for row in top_event_rows:
top_events_by_collector[row.collector].append(
{"event_type": row.event_type, "count": row.count}
)
by_collector: dict[str, dict[str, Any]] = {}
for record in records:
collector = str(record.collector or "").strip()
if not collector:
continue
for row in aggregate_rows:
collector = row.collector
latest = latest_by_collector.get(collector, {})
fallback_location = RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
scope = scope_by_collector.get(collector, {"countries": set(), "cities": set()})
coverage = by_collector.get(collector)
if coverage is None:
location = record.collector_geo or RIPE_RIS_COLLECTOR_COORDS.get(collector, {})
coverage = {
"collector": collector,
"city": location.get("city"),
"country": location.get("country"),
"latitude": location.get("latitude"),
"longitude": location.get("longitude"),
"observation_count": 0,
"prefixes": set(),
"origin_asns": set(),
"peer_asns": set(),
"event_types": defaultdict(int),
"countries": set(),
"cities": set(),
"recent_15m_observation_count": 0,
"recent_24h_observation_count": 0,
"recent_7d_observation_count": 0,
"recent_15m_prefixes": set(),
"recent_24h_prefixes": set(),
"recent_7d_prefixes": set(),
"latest_observed_at": None,
"latest_event_type": None,
}
by_collector[collector] = coverage
coverage["observation_count"] += 1
if record.prefix:
coverage["prefixes"].add(record.prefix)
if record.origin_asn is not None:
coverage["origin_asns"].add(record.origin_asn)
if record.peer_asn is not None:
coverage["peer_asns"].add(record.peer_asn)
if record.event_type:
coverage["event_types"][record.event_type] += 1
observed_at = record.observed_at
if observed_at is not None:
aware_observed_at = (
observed_at.astimezone(UTC)
if observed_at.tzinfo
else observed_at.replace(tzinfo=UTC)
)
if aware_observed_at >= recent_15m_threshold:
coverage["recent_15m_observation_count"] += 1
if record.prefix:
coverage["recent_15m_prefixes"].add(record.prefix)
if aware_observed_at >= recent_24h_threshold:
coverage["recent_24h_observation_count"] += 1
if record.prefix:
coverage["recent_24h_prefixes"].add(record.prefix)
if aware_observed_at >= recent_7d_threshold:
coverage["recent_7d_observation_count"] += 1
if record.prefix:
coverage["recent_7d_prefixes"].add(record.prefix)
geo = record.collector_geo or {}
if geo.get("country"):
coverage["countries"].add(geo["country"])
if geo.get("city"):
coverage["cities"].add(geo["city"])
current_latest = coverage["latest_observed_at"]
if current_latest is None or (
record.observed_at is not None and record.observed_at > current_latest
):
coverage["latest_observed_at"] = record.observed_at
coverage["latest_event_type"] = record.event_type
by_collector[collector] = {
"collector": collector,
"city": latest.get("city") or fallback_location.get("city"),
"country": latest.get("country") or fallback_location.get("country"),
"latitude": fallback_location.get("latitude"),
"longitude": fallback_location.get("longitude"),
"observation_count": row.observation_count or 0,
"prefix_count": row.prefix_count or 0,
"origin_asn_count": row.origin_asn_count or 0,
"peer_asn_count": row.peer_asn_count or 0,
"recent_15m_observation_count": row.recent_15m_observation_count or 0,
"recent_24h_observation_count": row.recent_24h_observation_count or 0,
"recent_7d_observation_count": row.recent_7d_observation_count or 0,
"recent_15m_prefix_count": row.recent_15m_prefix_count or 0,
"recent_24h_prefix_count": row.recent_24h_prefix_count or 0,
"recent_7d_prefix_count": row.recent_7d_prefix_count or 0,
"top_event_types": top_events_by_collector.get(collector, []),
"latest_observed_at": to_iso8601_utc(row.latest_observed_at),
"latest_event_type": latest.get("latest_event_type"),
"baseline_scope": {
"countries": sorted(scope["countries"]),
"cities": sorted(scope["cities"]),
},
}
for collector, location in RIPE_RIS_COLLECTOR_COORDS.items():
if collector in by_collector:
@@ -117,57 +187,22 @@ async def build_bgp_collector_coverage(
"latitude": location.get("latitude"),
"longitude": location.get("longitude"),
"observation_count": 0,
"prefixes": set(),
"origin_asns": set(),
"peer_asns": set(),
"event_types": defaultdict(int),
"countries": {location.get("country")} if location.get("country") else set(),
"cities": {location.get("city")} if location.get("city") else set(),
"prefix_count": 0,
"origin_asn_count": 0,
"peer_asn_count": 0,
"recent_15m_observation_count": 0,
"recent_24h_observation_count": 0,
"recent_7d_observation_count": 0,
"recent_15m_prefixes": set(),
"recent_24h_prefixes": set(),
"recent_7d_prefixes": set(),
"recent_15m_prefix_count": 0,
"recent_24h_prefix_count": 0,
"recent_7d_prefix_count": 0,
"top_event_types": [],
"latest_observed_at": None,
"latest_event_type": None,
"baseline_scope": {
"countries": [location["country"]] if location.get("country") else [],
"cities": [location["city"]] if location.get("city") else [],
},
}
results: list[dict[str, Any]] = []
for collector in sorted(by_collector.keys()):
item = by_collector[collector]
top_event_types = sorted(
item["event_types"].items(),
key=lambda pair: (-pair[1], pair[0]),
)
results.append(
{
"collector": item["collector"],
"city": item["city"],
"country": item["country"],
"latitude": item["latitude"],
"longitude": item["longitude"],
"observation_count": item["observation_count"],
"prefix_count": len(item["prefixes"]),
"origin_asn_count": len(item["origin_asns"]),
"peer_asn_count": len(item["peer_asns"]),
"recent_15m_observation_count": item["recent_15m_observation_count"],
"recent_24h_observation_count": item["recent_24h_observation_count"],
"recent_7d_observation_count": item["recent_7d_observation_count"],
"recent_15m_prefix_count": len(item["recent_15m_prefixes"]),
"recent_24h_prefix_count": len(item["recent_24h_prefixes"]),
"recent_7d_prefix_count": len(item["recent_7d_prefixes"]),
"top_event_types": [
{"event_type": event_type, "count": count}
for event_type, count in top_event_types[:3]
],
"latest_observed_at": to_iso8601_utc(item["latest_observed_at"]),
"latest_event_type": item["latest_event_type"],
"baseline_scope": {
"countries": sorted(country for country in item["countries"] if country),
"cities": sorted(city for city in item["cities"] if city),
},
}
)
return results
return [by_collector[collector] for collector in sorted(by_collector.keys())]

View File

@@ -7,7 +7,7 @@ from collections import defaultdict
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import select, text
from sqlalchemy import Integer, cast, select, text
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.countries import get_country_centroid, normalize_country
@@ -231,6 +231,13 @@ async def _lookup_prefix_geography(
return results
async def lookup_prefix_geography(
db: AsyncSession,
prefix_values: list[str],
) -> dict[str, dict[str, Any]]:
return await _lookup_prefix_geography(db, prefix_values)
async def enrich_bgp_events_for_batch(
db: AsyncSession,
*,
@@ -261,29 +268,40 @@ async def enrich_bgp_events_for_batch(
historical_prefix_baseline: dict[str, dict[str, Any]] = {}
if prefix_values:
previous_result = await db.execute(
select(BGPObservation).where(
select(
BGPObservation.prefix,
BGPObservation.origin_asn,
BGPObservation.collector,
BGPObservation.collector_geo,
).where(
BGPObservation.source == source,
BGPObservation.prefix.in_(prefix_values),
)
)
by_prefix: defaultdict[str, list[BGPObservation]] = defaultdict(list)
for observation in previous_result.scalars().all():
if observation.prefix:
by_prefix[observation.prefix].append(observation)
by_prefix: defaultdict[str, list[dict[str, Any]]] = defaultdict(list)
for prefix, origin_asn, collector, collector_geo in previous_result.all():
if prefix:
by_prefix[str(prefix)].append(
{
"origin_asn": origin_asn,
"collector": collector,
"collector_geo": collector_geo or {},
}
)
for prefix, observations in by_prefix.items():
unique_origins = sorted(
{
observation.origin_asn
observation["origin_asn"]
for observation in observations
if observation.origin_asn is not None
if observation["origin_asn"] is not None
}
)
unique_collectors = sorted(
{
observation.collector
observation["collector"]
for observation in observations
if observation.collector
if observation["collector"]
}
)
historical_prefix_baseline[prefix] = {
@@ -292,9 +310,9 @@ async def enrich_bgp_events_for_batch(
"historical_observation_count": len(observations),
"historical_regions": _compact_locations(
[
observation.collector_geo or {}
observation["collector_geo"] or {}
for observation in observations
if observation.collector_geo
if observation["collector_geo"]
]
),
}
@@ -303,7 +321,13 @@ async def enrich_bgp_events_for_batch(
prefix_geographies = await _lookup_prefix_geography(db, prefix_values) if prefix_values else {}
if origin_asns:
peeringdb_result = await db.execute(
select(CollectedData).where(CollectedData.source == "peeringdb_network")
select(CollectedData)
.where(CollectedData.source == "peeringdb_network")
.where(CollectedData.is_current.is_(True))
.where(
cast(CollectedData.extra_data["asn"].as_string(), Integer).in_(origin_asns),
)
.order_by(CollectedData.id.desc())
)
for record in peeringdb_result.scalars().all():
metadata = record.extra_data or {}

View File

@@ -48,14 +48,36 @@ def _collector_regions_from_anomaly(anomaly: BGPAnomaly) -> list[dict]:
return collected
def _dedupe_collected_records(records: list[CollectedData]) -> list[CollectedData]:
latest_by_key: dict[str, CollectedData] = {}
for record in records:
dedupe_key = str(record.source_id or record.entity_key or record.name or record.id)
existing = latest_by_key.get(dedupe_key)
if existing is None or (record.id or 0) > (existing.id or 0):
latest_by_key[dedupe_key] = record
return list(latest_by_key.values())
async def _load_current_infrastructure_records(
db: AsyncSession,
) -> tuple[list[CollectedData], list[CollectedData], list[CollectedData]]:
result = await db.execute(
select(CollectedData)
.where(
CollectedData.source.in_(
(
"arcgis_landing_points",
"arcgis_cable_landing_relation",
"arcgis_cables",
)
)
)
.where(CollectedData.is_current.is_(True))
.order_by(CollectedData.source.asc(), CollectedData.id.desc())
)
grouped_records = {
"arcgis_landing_points": [],
"arcgis_cable_landing_relation": [],
"arcgis_cables": [],
}
for record in result.scalars().all():
grouped_records.setdefault(record.source, []).append(record)
return (
grouped_records["arcgis_landing_points"],
grouped_records["arcgis_cable_landing_relation"],
grouped_records["arcgis_cables"],
)
async def infer_related_infrastructure(
@@ -75,19 +97,9 @@ async def infer_related_infrastructure(
if not valid_regions:
return {"related_cables": [], "related_ixps": []}
landing_result = await db.execute(
select(CollectedData).where(CollectedData.source == "arcgis_landing_points")
landing_records, relation_records, cable_records = await _load_current_infrastructure_records(
db,
)
relation_result = await db.execute(
select(CollectedData).where(CollectedData.source == "arcgis_cable_landing_relation")
)
cable_result = await db.execute(
select(CollectedData).where(CollectedData.source == "arcgis_cables")
)
landing_records = _dedupe_collected_records(list(landing_result.scalars().all()))
relation_records = _dedupe_collected_records(list(relation_result.scalars().all()))
cable_records = _dedupe_collected_records(list(cable_result.scalars().all()))
city_to_cable_ids: dict[int, list[int]] = {}
for relation in relation_records:

View File

@@ -35,6 +35,7 @@ from app.services.collectors.bgpstream import BGPStreamBackfillCollector
from app.services.collectors.iptoasn import IPtoASNPrefixGeoCollector
from app.services.collectors.opengeofeed import OpenGeoFeedPrefixGeoCollector
from app.services.collectors.nro_delegated import NRODelegatedPrefixGeoCollector
from app.services.collectors.news_live_streams import NewsLiveStreamsCollector
collector_registry.register(TOP500Collector())
collector_registry.register(EpochAIGPUCollector())
@@ -61,3 +62,4 @@ collector_registry.register(BGPStreamBackfillCollector())
collector_registry.register(IPtoASNPrefixGeoCollector())
collector_registry.register(OpenGeoFeedPrefixGeoCollector())
collector_registry.register(NRODelegatedPrefixGeoCollector())
collector_registry.register(NewsLiveStreamsCollector())

View File

@@ -1,5 +1,6 @@
"""Base collector class for all data sources"""
import asyncio
from abc import ABC, abstractmethod
from typing import Dict, List, Any, Optional
from datetime import UTC, datetime
@@ -166,6 +167,59 @@ class BaseCollector(ABC):
await db.commit()
return snapshot.id
async def _rollback_incomplete_run(
self,
db: AsyncSession,
*,
task_id: int,
snapshot_id: Optional[int],
reason: str,
) -> None:
from app.models.collected_data import CollectedData
from app.models.data_snapshot import DataSnapshot
await db.execute(CollectedData.__table__.delete().where(CollectedData.task_id == task_id))
parent_snapshot_id: Optional[int] = None
if snapshot_id is not None:
snapshot = await db.get(DataSnapshot, snapshot_id)
if snapshot:
parent_snapshot_id = snapshot.parent_snapshot_id
snapshot.status = "cancelled"
snapshot.is_current = False
snapshot.completed_at = datetime.now(UTC)
summary = dict(snapshot.summary or {})
summary["rollback"] = True
summary["rollback_reason"] = reason
snapshot.summary = summary
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = FALSE
WHERE source = :source
"""
),
{"source": self.name},
)
if parent_snapshot_id is not None:
parent_snapshot = await db.get(DataSnapshot, parent_snapshot_id)
if parent_snapshot:
parent_snapshot.is_current = True
await db.execute(
text(
"""
UPDATE collected_data
SET is_current = TRUE
WHERE snapshot_id = :snapshot_id
"""
),
{"snapshot_id": parent_snapshot_id},
)
async def run(self, db: AsyncSession) -> Dict[str, Any]:
"""Full pipeline: fetch -> transform -> save"""
from app.services.collectors.registry import collector_registry
@@ -227,7 +281,24 @@ class BaseCollector(ABC):
"records_processed": records_count,
"execution_time_seconds": (datetime.now(UTC) - start_time).total_seconds(),
}
except asyncio.CancelledError:
await db.rollback()
task.status = "cancelled"
task.phase = "cancelled"
task.error_message = "Collection cancelled by operator and rolled back"
task.completed_at = datetime.now(UTC)
if snapshot_id is not None:
await self._rollback_incomplete_run(
db,
task_id=task_id,
snapshot_id=snapshot_id,
reason="cancelled_by_operator",
)
await db.commit()
await self._publish_task_update(force=True)
raise
except Exception as e:
await db.rollback()
task.status = "failed"
task.phase = "failed"
task.error_message = str(e)
@@ -276,20 +347,34 @@ class BaseCollector(ABC):
updated_count = 0
unchanged_count = 0
seen_entity_keys: set[str] = set()
previous_current_keys: set[str] = set()
progress_commit_interval = 1000
previous_current_result = await db.execute(
select(CollectedData.entity_key).where(
select(CollectedData)
.where(
CollectedData.source == self.name,
CollectedData.is_current == True,
)
.order_by(CollectedData.entity_key.asc(), CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
)
previous_current_keys = {row[0] for row in previous_current_result.fetchall() if row[0]}
previous_current_records = previous_current_result.scalars().all()
previous_current_keys = {record.entity_key for record in previous_current_records if record.entity_key}
previous_current_map: dict[str, CollectedData] = {}
stale_previous_records: list[CollectedData] = []
for existing_record in previous_current_records:
entity_key = existing_record.entity_key
if not entity_key:
continue
if entity_key not in previous_current_map:
previous_current_map[entity_key] = existing_record
continue
stale_previous_records.append(existing_record)
for stale_record in stale_previous_records:
stale_record.is_current = False
for i, item in enumerate(data):
print(
f"DEBUG: Saving item {i}: name={item.get('name')}, metadata={item.get('metadata', 'NOT FOUND')}"
)
raw_metadata = item.get("metadata", {})
extra_data = build_dynamic_metadata(
raw_metadata,
@@ -318,20 +403,9 @@ class BaseCollector(ABC):
previous_record = None
if entity_key and entity_key not in seen_entity_keys:
result = await db.execute(
select(CollectedData)
.where(
CollectedData.source == self.name,
CollectedData.entity_key == entity_key,
CollectedData.is_current == True,
)
.order_by(CollectedData.collected_at.desc().nullslast(), CollectedData.id.desc())
)
previous_records = result.scalars().all()
if previous_records:
previous_record = previous_records[0]
for old_record in previous_records:
old_record.is_current = False
previous_record = previous_current_map.get(entity_key)
if previous_record is not None:
previous_record.is_current = False
record = CollectedData(
snapshot_id=snapshot_id,
@@ -375,7 +449,7 @@ class BaseCollector(ABC):
seen_entity_keys.add(entity_key)
records_added += 1
if i % 100 == 0:
if (i + 1) % progress_commit_interval == 0:
await self.update_progress(i + 1, commit=True)
if snapshot_id is not None:

View File

@@ -21,7 +21,7 @@ class CelesTrakTLECollector(BaseCollector):
@property
def base_url(self) -> str:
return "https://celestrak.org/NORAD/elements/gp.php"
return self._resolved_url or ""
async def fetch(self) -> List[Dict[str, Any]]:
satellite_groups = [
@@ -40,7 +40,7 @@ class CelesTrakTLECollector(BaseCollector):
async with httpx.AsyncClient(timeout=120.0) as client:
for group in satellite_groups:
try:
url = f"https://celestrak.org/NORAD/elements/gp.php?GROUP={group}&FORMAT=json"
url = f"{self.base_url}?GROUP={group}&FORMAT=json"
response = await client.get(url)
if response.status_code == 200:

View File

@@ -39,6 +39,16 @@ class CloudflareRadarDeviceCollector(HTTPCollector):
if CLOUDFLARE_API_TOKEN:
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Cloudflare Radar device type response"""
data = []
@@ -87,6 +97,16 @@ class CloudflareRadarTrafficCollector(HTTPCollector):
if CLOUDFLARE_API_TOKEN:
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Cloudflare Radar traffic timeseries response"""
data = []
@@ -135,6 +155,16 @@ class CloudflareRadarTopASCollector(HTTPCollector):
if CLOUDFLARE_API_TOKEN:
self.headers["Authorization"] = f"Bearer {CLOUDFLARE_API_TOKEN}"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Cloudflare Radar top locations response"""
data = []

View File

@@ -23,7 +23,7 @@ class EpochAIGPUCollector(BaseCollector):
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch Epoch AI GPU clusters data from webpage"""
url = "https://epoch.ai/data/gpu-clusters"
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(url)

View File

@@ -18,11 +18,9 @@ class FAOLandingPointCollector(BaseCollector):
frequency_hours = 168
data_type = "landing_point"
csv_url = "https://data.apps.fao.org/catalog/dataset/1b75ff21-92f2-4b96-9b7b-98e8aa65ad5d/resource/b6071077-d1d4-4e97-aa00-42e902847c87/download/landing-point-geo.csv"
async def fetch(self) -> List[Dict[str, Any]]:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.csv_url)
response = await client.get(self._resolved_url or "")
response.raise_for_status()
return self.parse_csv(response.text)

View File

@@ -21,6 +21,18 @@ class HuggingFaceModelCollector(HTTPCollector):
data_type = "model"
base_url = "https://huggingface.co/api/models"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
from httpx import AsyncClient
async with AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Hugging Face models API response"""
data = []
@@ -63,6 +75,18 @@ class HuggingFaceDatasetCollector(HTTPCollector):
data_type = "dataset"
base_url = "https://huggingface.co/api/datasets"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
from httpx import AsyncClient
async with AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Hugging Face datasets API response"""
data = []
@@ -104,6 +128,18 @@ class HuggingFaceSpacesCollector(HTTPCollector):
data_type = "space"
base_url = "https://huggingface.co/api/spaces"
@property
def request_url(self) -> str:
return self._resolved_url or self.base_url
async def fetch(self) -> List[Dict[str, Any]]:
from httpx import AsyncClient
async with AsyncClient(timeout=60.0) as client:
response = await client.get(self.request_url, headers=self.headers)
response.raise_for_status()
return self.parse_response(response.json())
def parse_response(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Hugging Face Spaces API response"""
data = []

View File

@@ -0,0 +1,570 @@
from __future__ import annotations
import asyncio
import base64
from datetime import UTC, datetime
from typing import Any
from urllib.parse import urlparse
import httpx
from sqlalchemy import select
from app.core.data_sources import get_data_sources_config
from app.models.datasource_config import DataSourceConfig
from app.services.collectors.base import BaseCollector
class NewsLiveStreamsCollector(BaseCollector):
"""Collect normalized news live-stream sources from a JSON endpoint."""
name = "news_live_streams"
priority = "P2"
module = "L4"
frequency_hours = 12
data_type = "news_live_stream"
fail_on_empty = False
DEFAULT_TIMEOUT = 45.0
DEFAULT_HEADERS = {
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
RESPONSE_CANDIDATE_KEYS = ("sources", "streams", "channels", "items", "results", "data")
DEFAULT_ADAPTER = "iptv_org"
DEFAULT_IPTV_ORG_STREAMS_URL = "https://iptv-org.github.io/api/streams.json"
DEFAULT_IPTV_ORG_LOGOS_URL = "https://iptv-org.github.io/api/logos.json"
DEFAULT_IPTV_ORG_NEWS_CATEGORIES = ("news", "business", "weather")
DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES = ("music", "sports", "kids", "entertainment")
DEFAULT_IPTV_ORG_MAX_SOURCES = 120
async def fetch(self) -> list[dict[str, Any]]:
request_url = (self._resolved_url or "").strip()
if not request_url:
return []
datasource_config = await self._load_datasource_config()
effective_config = self._get_effective_config(datasource_config)
adapter = str(effective_config.get("adapter") or "").strip().lower()
if adapter == "iptv_org":
return await self._fetch_iptv_org(request_url, effective_config)
request_headers = self._build_request_headers(datasource_config)
request_config = self._get_request_config(datasource_config)
request_params = self._build_request_params(datasource_config)
request_json = self._build_request_json_body(datasource_config)
request_data = self._build_request_form_body(datasource_config)
timeout = self._get_timeout(datasource_config)
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
response = await client.request(
request_config["method"],
request_url,
headers=request_headers,
params=request_params or None,
json=request_json,
data=request_data,
)
response.raise_for_status()
return self.parse_response(
response.json(),
response_path=request_config["response_path"],
)
async def _load_datasource_config(self) -> DataSourceConfig | None:
if not self._db_session:
return None
result = await self._db_session.execute(
select(DataSourceConfig)
.where(DataSourceConfig.name == self.name)
.where(DataSourceConfig.is_active.is_(True))
.limit(1)
)
return result.scalar_one_or_none()
def _get_effective_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
payload = dict(datasource_config.config or {}) if datasource_config else {}
if payload:
return payload
yaml_config = get_data_sources_config()
return {
"adapter": self.DEFAULT_ADAPTER,
"streams_url": yaml_config.get_yaml_value("news_live_streams.streams_url")
or self.DEFAULT_IPTV_ORG_STREAMS_URL,
"logos_url": yaml_config.get_yaml_value("news_live_streams.logos_url")
or self.DEFAULT_IPTV_ORG_LOGOS_URL,
"news_categories": list(self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES),
"exclude_categories": list(self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES),
"max_sources": self.DEFAULT_IPTV_ORG_MAX_SOURCES,
}
def _get_request_config(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
payload = self._get_effective_config(datasource_config)
raw_method = payload.get("method") or payload.get("request_method") or "GET"
method = str(raw_method).strip().upper() or "GET"
if method not in {"GET", "POST"}:
method = "GET"
response_path = payload.get("response_path") or payload.get("payload_path") or payload.get("items_path")
if isinstance(response_path, str):
response_path = response_path.strip()
else:
response_path = None
return {
"method": method,
"response_path": response_path or None,
}
def _get_timeout(self, datasource_config: DataSourceConfig | None) -> float:
payload = self._get_effective_config(datasource_config)
try:
return float(payload.get("timeout", self.DEFAULT_TIMEOUT))
except (TypeError, ValueError):
return self.DEFAULT_TIMEOUT
def _build_request_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
headers = dict(self.DEFAULT_HEADERS)
if datasource_config:
headers.update(self._normalize_headers(datasource_config.headers))
headers.update(self._build_auth_headers(datasource_config))
return headers
def _build_request_params(self, datasource_config: DataSourceConfig | None) -> dict[str, Any]:
params: dict[str, Any] = {}
if not datasource_config:
return params
payload = datasource_config.config or {}
candidate = payload.get("params") or payload.get("query_params")
if isinstance(candidate, dict):
params.update(candidate)
if datasource_config.auth_type == "api_key":
auth_config = datasource_config.auth_config or {}
if str(auth_config.get("in") or auth_config.get("location") or "header").lower() == "query":
api_key = auth_config.get("api_key")
key_name = auth_config.get("key_name") or auth_config.get("param_name") or "api_key"
if api_key and key_name:
params[str(key_name)] = api_key
return params
def _build_request_json_body(self, datasource_config: DataSourceConfig | None) -> Any:
if not datasource_config:
return None
payload = datasource_config.config or {}
body = payload.get("json_body")
if body is None and str(payload.get("body_type") or "").lower() in {"json", ""}:
candidate = payload.get("body")
if isinstance(candidate, (dict, list)):
body = candidate
return body
def _build_request_form_body(self, datasource_config: DataSourceConfig | None) -> Any:
if not datasource_config:
return None
payload = datasource_config.config or {}
form_body = payload.get("form_body")
if form_body is not None:
return form_body
if str(payload.get("body_type") or "").lower() == "form":
candidate = payload.get("body")
if isinstance(candidate, dict):
return candidate
return None
def _normalize_headers(self, headers: Any) -> dict[str, str]:
if not isinstance(headers, dict):
return {}
normalized: dict[str, str] = {}
for key, value in headers.items():
header_name = str(key).strip()
if not header_name or value is None:
continue
normalized[header_name] = str(value)
return normalized
def _build_auth_headers(self, datasource_config: DataSourceConfig | None) -> dict[str, str]:
if not datasource_config:
return {}
auth_type = str(datasource_config.auth_type or "none").lower()
auth_config = datasource_config.auth_config or {}
if auth_type == "bearer" and auth_config.get("token"):
return {"Authorization": f"Bearer {auth_config['token']}"}
if auth_type == "api_key" and auth_config.get("api_key"):
location = str(auth_config.get("in") or auth_config.get("location") or "header").lower()
if location == "query":
return {}
key_name = auth_config.get("key_name") or "X-API-Key"
return {str(key_name): str(auth_config["api_key"])}
if auth_type == "basic":
username = str(auth_config.get("username") or "")
password = str(auth_config.get("password") or "")
encoded = base64.b64encode(f"{username}:{password}".encode()).decode()
return {"Authorization": f"Basic {encoded}"}
return {}
def _extract_candidates(self, response: Any, response_path: str | None) -> list[Any]:
if response_path:
extracted = self._extract_from_path(response, response_path)
if isinstance(extracted, list):
return extracted
if isinstance(extracted, dict):
for key in self.RESPONSE_CANDIDATE_KEYS:
nested = extracted.get(key)
if isinstance(nested, list):
return nested
return [extracted]
if isinstance(response, dict):
for key in self.RESPONSE_CANDIDATE_KEYS:
nested = response.get(key)
if isinstance(nested, list):
return nested
return []
if isinstance(response, list):
return response
return []
def _extract_from_path(self, payload: Any, path: str) -> Any:
current = payload
for segment in (part.strip() for part in path.split(".") if part.strip()):
if isinstance(current, dict):
current = current.get(segment)
continue
if isinstance(current, list):
try:
current = current[int(segment)]
except (TypeError, ValueError, IndexError):
return None
continue
return None
return current
def _infer_source_type(self, item: dict[str, Any]) -> str:
explicit = str(item.get("source_type") or item.get("type") or "").strip().lower()
if explicit in {"iframe", "hls", "video", "external", "youtube"}:
return explicit
youtube_video_id = self._clean_text(
item.get("youtube_video_id")
or item.get("video_id")
or item.get("youtubeVideoId")
)
youtube_channel = self._clean_text(item.get("youtube_channel") or item.get("channel_handle"))
embed_url = self._clean_url(item.get("embed_url") or item.get("embed") or item.get("page_url"))
stream_url = self._clean_url(item.get("stream_url") or item.get("stream") or item.get("playback_url") or item.get("hls_url"))
homepage_url = self._clean_url(item.get("homepage_url") or item.get("source_url") or item.get("website"))
if youtube_video_id or youtube_channel:
return "youtube"
if stream_url.endswith(".m3u8"):
return "hls"
if stream_url:
return "video"
if embed_url:
parsed = urlparse(embed_url)
if "youtube.com" in (parsed.netloc or "") or "youtu.be" in (parsed.netloc or ""):
return "youtube"
return "iframe"
if homepage_url:
return "external"
return "iframe"
def _parse_enabled(self, item: dict[str, Any]) -> bool:
if "is_enabled" in item:
return self._to_bool(item.get("is_enabled"), default=True)
if "enabled" in item:
return self._to_bool(item.get("enabled"), default=True)
if "active" in item:
return self._to_bool(item.get("active"), default=True)
if "status" in item:
status = str(item.get("status") or "").strip().lower()
if status in {"disabled", "inactive", "offline"}:
return False
if status in {"enabled", "active", "online", "live"}:
return True
return True
def _to_bool(self, value: Any, *, default: bool) -> bool:
if isinstance(value, bool):
return value
if value in (None, ""):
return default
if isinstance(value, str):
lowered = value.strip().lower()
if lowered in {"1", "true", "yes", "on", "enabled", "active", "online", "live"}:
return True
if lowered in {"0", "false", "no", "off", "disabled", "inactive", "offline"}:
return False
return bool(value)
def _clean_text(self, value: Any) -> str:
if value is None:
return ""
return str(value).strip()
def _clean_url(self, value: Any) -> str:
text = self._clean_text(value)
if not text:
return ""
parsed = urlparse(text)
if parsed.scheme and parsed.scheme not in {"http", "https"}:
return ""
if parsed.scheme and not parsed.netloc:
return ""
return text
async def _fetch_iptv_org(self, channels_url: str, collector_config: dict[str, Any]) -> list[dict[str, Any]]:
streams_url = self._clean_url(collector_config.get("streams_url")) or self.DEFAULT_IPTV_ORG_STREAMS_URL
logos_url = self._clean_url(collector_config.get("logos_url")) or self.DEFAULT_IPTV_ORG_LOGOS_URL
news_categories = {
self._clean_text(value).lower()
for value in (collector_config.get("news_categories") or self.DEFAULT_IPTV_ORG_NEWS_CATEGORIES)
if self._clean_text(value)
}
exclude_categories = {
self._clean_text(value).lower()
for value in (collector_config.get("exclude_categories") or self.DEFAULT_IPTV_ORG_EXCLUDE_CATEGORIES)
if self._clean_text(value)
}
try:
max_sources = int(collector_config.get("max_sources", self.DEFAULT_IPTV_ORG_MAX_SOURCES))
except (TypeError, ValueError):
max_sources = self.DEFAULT_IPTV_ORG_MAX_SOURCES
timeout = self.DEFAULT_TIMEOUT
try:
timeout = float(collector_config.get("timeout", self.DEFAULT_TIMEOUT))
except (TypeError, ValueError):
timeout = self.DEFAULT_TIMEOUT
async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as client:
channels_payload, streams_payload, logos_payload = await self._gather_iptv_org_payloads(
client,
channels_url,
streams_url,
logos_url,
)
channels = channels_payload if isinstance(channels_payload, list) else []
streams = streams_payload if isinstance(streams_payload, list) else []
logos = logos_payload if isinstance(logos_payload, list) else []
logo_by_channel = {
self._clean_text(item.get("channel")): self._clean_url(item.get("url"))
for item in logos
if isinstance(item, dict) and self._clean_text(item.get("channel")) and self._clean_url(item.get("url"))
}
streams_by_channel: dict[str, list[dict[str, Any]]] = {}
for stream in streams:
if not isinstance(stream, dict):
continue
channel_id = self._clean_text(stream.get("channel"))
if not channel_id:
continue
streams_by_channel.setdefault(channel_id, []).append(stream)
normalized: list[dict[str, Any]] = []
for channel in channels:
if not isinstance(channel, dict):
continue
categories = [
self._clean_text(value).lower()
for value in (channel.get("categories") or [])
if self._clean_text(value)
]
if news_categories and not any(category in news_categories for category in categories):
continue
if exclude_categories and any(category in exclude_categories for category in categories):
continue
if channel.get("is_nsfw") is True:
continue
if channel.get("closed"):
continue
channel_id = self._clean_text(channel.get("id"))
if not channel_id:
continue
stream = self._pick_iptv_org_stream(streams_by_channel.get(channel_id) or [])
if not stream:
continue
stream_url = self._clean_url(stream.get("url"))
if not stream_url:
continue
name = self._clean_text(channel.get("name")) or channel_id
notes_parts = [
f"Imported from IPTV-org catalog ({channel_id})",
f"Categories: {', '.join(categories)}" if categories else "",
f"Quality: {self._clean_text(stream.get('quality'))}" if self._clean_text(stream.get("quality")) else "",
]
metadata = {
"provider": self._clean_text(channel.get("network")) or "IPTV-org",
"region": self._clean_text(channel.get("country")) or "Global",
"language": "und",
"source_type": "hls" if stream_url.endswith(".m3u8") else "video",
"embed_url": "",
"stream_url": stream_url,
"homepage_url": self._clean_url(channel.get("website")),
"poster_url": logo_by_channel.get(channel_id, ""),
"youtube_video_id": "",
"youtube_channel": "",
"sort_order": 400 + len(normalized),
"notes": "; ".join(part for part in notes_parts if part),
"is_enabled": True,
"collector_adapter": "iptv_org",
"channel_id": channel_id,
"categories": categories,
"quality": self._clean_text(stream.get("quality")),
"stream_label": self._clean_text(stream.get("label") or stream.get("title")),
"stream_referrer": self._clean_text(stream.get("referrer")),
"stream_user_agent": self._clean_text(stream.get("user_agent")),
}
normalized.append(
{
"source_id": channel_id,
"name": name,
"description": metadata["notes"],
"metadata": metadata,
"reference_date": datetime.now(UTC).isoformat(),
}
)
if len(normalized) >= max_sources:
break
return normalized
async def _gather_iptv_org_payloads(
self,
client: httpx.AsyncClient,
channels_url: str,
streams_url: str,
logos_url: str,
) -> tuple[Any, Any, Any]:
headers = dict(self.DEFAULT_HEADERS)
channels_payload, streams_payload, logos_payload = await asyncio.gather(
client.get(channels_url, headers=headers),
client.get(streams_url, headers=headers),
client.get(logos_url, headers=headers),
)
channels_payload.raise_for_status()
streams_payload.raise_for_status()
logos_payload.raise_for_status()
return channels_payload.json(), streams_payload.json(), logos_payload.json()
def _pick_iptv_org_stream(self, streams: list[dict[str, Any]]) -> dict[str, Any] | None:
if not streams:
return None
def score(stream: dict[str, Any]) -> tuple[int, int]:
url = self._clean_url(stream.get("url"))
quality = self._clean_text(stream.get("quality")).lower()
quality_score = 0
if quality.endswith("p"):
try:
quality_score = int(quality[:-1])
except ValueError:
quality_score = 0
stream_score = 1000 if url.endswith(".m3u8") else 0
return stream_score, quality_score
sorted_streams = sorted(streams, key=score, reverse=True)
return sorted_streams[0]
def parse_response(self, response: Any, *, response_path: str | None = None) -> list[dict[str, Any]]:
candidates = self._extract_candidates(response, response_path)
normalized: list[dict[str, Any]] = []
for index, item in enumerate(candidates):
if not isinstance(item, dict):
continue
stream_id = (
item.get("id")
or item.get("source_id")
or item.get("slug")
or item.get("channel_id")
or item.get("code")
or f"news-live-{index + 1}"
)
name = self._clean_text(
item.get("name")
or item.get("title")
or item.get("channel")
or item.get("display_name")
or f"News Live {index + 1}"
)
if not name:
continue
source_type = self._infer_source_type(item)
stream_url = self._clean_url(
item.get("stream_url")
or item.get("stream")
or item.get("playback_url")
or item.get("hls_url")
or item.get("m3u8_url")
)
embed_url = self._clean_url(
item.get("embed_url")
or item.get("embed")
or item.get("page_url")
or (item.get("url") if source_type == "iframe" else "")
)
homepage_url = self._clean_url(
item.get("homepage_url")
or item.get("source_url")
or item.get("website")
or item.get("url")
)
metadata = {
"provider": self._clean_text(item.get("provider") or item.get("publisher") or item.get("network")) or "Collector",
"region": self._clean_text(item.get("region") or item.get("country") or item.get("market")) or "Global",
"language": self._clean_text(item.get("language") or item.get("lang") or item.get("locale")) or "und",
"source_type": source_type,
"embed_url": embed_url,
"stream_url": stream_url,
"homepage_url": homepage_url,
"poster_url": self._clean_url(item.get("poster_url") or item.get("thumbnail_url") or item.get("logo_url")),
"youtube_video_id": self._clean_text(
item.get("youtube_video_id")
or item.get("video_id")
or item.get("youtubeVideoId")
),
"youtube_channel": self._clean_text(
item.get("youtube_channel")
or item.get("channel_handle")
or item.get("youtubeChannel")
),
"sort_order": item.get("sort_order", 200 + index),
"notes": self._clean_text(item.get("notes") or item.get("description") or item.get("summary")),
"is_enabled": self._parse_enabled(item),
}
normalized.append(
{
"source_id": str(stream_id),
"name": name,
"description": metadata["notes"],
"metadata": metadata,
"reference_date": item.get("reference_date") or datetime.now(UTC).isoformat(),
}
)
return normalized

View File

@@ -16,6 +16,7 @@ from typing import Dict, Any, List
from datetime import UTC, datetime
import httpx
from urllib.parse import urlencode
from app.services.collectors.base import HTTPCollector
@@ -38,9 +39,13 @@ class PeeringDBIXPCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
# API key is added to URL as query parameter
if PEERINGDB_API_KEY:
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
if not PEERINGDB_API_KEY:
return base
separator = "&" if "?" in base else "?"
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
async def fetch_with_retry(
self, max_retries: int = 3, base_delay: float = 2.0
@@ -51,7 +56,7 @@ class PeeringDBIXPCollector(HTTPCollector):
for attempt in range(max_retries):
try:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.base_url, headers=self.headers)
response = await client.get(self.request_url, headers=self.headers)
if response.status_code == 429:
# Rate limited - wait and retry with exponential backoff
@@ -141,8 +146,13 @@ class PeeringDBNetworkCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
if PEERINGDB_API_KEY:
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
if not PEERINGDB_API_KEY:
return base
separator = "&" if "?" in base else "?"
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
async def fetch_with_retry(
self, max_retries: int = 3, base_delay: float = 2.0
@@ -153,7 +163,7 @@ class PeeringDBNetworkCollector(HTTPCollector):
for attempt in range(max_retries):
try:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.base_url, headers=self.headers)
response = await client.get(self.request_url, headers=self.headers)
if response.status_code == 429:
delay = base_delay * (2**attempt)
@@ -244,8 +254,13 @@ class PeeringDBFacilityCollector(HTTPCollector):
"User-Agent": "Planet-Intelligence-System/1.0 (Python/collector)",
"Accept": "application/json",
}
if PEERINGDB_API_KEY:
self.base_url = f"{self.base_url}?key={PEERINGDB_API_KEY}"
@property
def request_url(self) -> str:
base = self._resolved_url or self.base_url
if not PEERINGDB_API_KEY:
return base
separator = "&" if "?" in base else "?"
return f"{base}{separator}{urlencode({'key': PEERINGDB_API_KEY})}"
async def fetch_with_retry(
self, max_retries: int = 3, base_delay: float = 2.0
@@ -256,7 +271,7 @@ class PeeringDBFacilityCollector(HTTPCollector):
for attempt in range(max_retries):
try:
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(self.base_url, headers=self.headers)
response = await client.get(self.request_url, headers=self.headers)
if response.status_code == 429:
delay = base_delay * (2**attempt)

View File

@@ -33,7 +33,7 @@ class RISLiveCollector(BaseCollector):
def _fetch_via_stream(self) -> list[dict[str, Any]]:
events: list[dict[str, Any]] = []
stream_url = "https://ris-live.ripe.net/v1/stream/?format=json&client=planet-ris-live"
stream_url = self._resolved_url or ""
subscribe = json.dumps(
{
"host": "rrc00",

View File

@@ -7,6 +7,7 @@ API documentation: https://www.space-track.org/documentation
import json
from typing import Dict, Any, List
import httpx
from urllib.parse import urlparse
from app.services.collectors.base import BaseCollector
from app.core.data_sources import get_data_sources_config
@@ -21,12 +22,30 @@ class SpaceTrackTLECollector(BaseCollector):
data_type = "satellite_tle"
@property
def base_url(self) -> str:
def query_url(self) -> str:
config = get_data_sources_config()
if self._resolved_url:
return self._resolved_url
return config.get_yaml_url("spacetrack_tle")
@property
def site_root(self) -> str:
config = get_data_sources_config()
configured_root = config.get_yaml_value("spacetrack.base_url")
if isinstance(configured_root, str) and configured_root:
return configured_root.rstrip("/")
parsed = urlparse(self.query_url)
return f"{parsed.scheme}://{parsed.netloc}".rstrip("/")
@property
def login_url(self) -> str:
return f"{self.site_root}/ajaxauth/login"
@property
def probe_url(self) -> str:
return f"{self.site_root}/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
async def fetch(self) -> List[Dict[str, Any]]:
from app.core.config import settings
@@ -47,13 +66,13 @@ class SpaceTrackTLECollector(BaseCollector):
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "application/json, text/html, */*",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.space-track.org/",
"Referer": f"{self.site_root}/",
},
) as client:
await client.get("https://www.space-track.org/")
await client.get(f"{self.site_root}/")
login_response = await client.post(
"https://www.space-track.org/ajaxauth/login",
self.login_url,
data={
"identity": username,
"password": password,
@@ -69,7 +88,7 @@ class SpaceTrackTLECollector(BaseCollector):
timeout=120.0,
follow_redirects=True,
) as alt_client:
await alt_client.get("https://www.space-track.org/")
await alt_client.get(f"{self.site_root}/")
form_data = {
"username": username,
@@ -77,7 +96,7 @@ class SpaceTrackTLECollector(BaseCollector):
"query": "class/gp/NORAD_CAT_ID/25544/format/json",
}
alt_login = await alt_client.post(
"https://www.space-track.org/ajaxauth/login",
self.login_url,
data={
"identity": username,
"password": password,
@@ -86,9 +105,7 @@ class SpaceTrackTLECollector(BaseCollector):
print(f"SPACETRACK: Alt login status: {alt_login.status_code}")
if alt_login.status_code == 200:
tle_response = await alt_client.get(
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
)
tle_response = await alt_client.get(self.probe_url)
if tle_response.status_code == 200:
data = tle_response.json()
print(f"SPACETRACK: Received {len(data)} records via alt method")
@@ -98,9 +115,7 @@ class SpaceTrackTLECollector(BaseCollector):
print(f"SPACETRACK: Login failed, using sample data")
return self._get_sample_data()
tle_response = await client.get(
"https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/25544/format/json"
)
tle_response = await client.get(self.probe_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:
@@ -127,11 +142,11 @@ class SpaceTrackTLECollector(BaseCollector):
},
) as client:
# First, visit the main page to get any cookies
await client.get("https://www.space-track.org/")
await client.get(f"{self.site_root}/")
# Login to get session cookie
login_response = await client.post(
"https://www.space-track.org/ajaxauth/login",
self.login_url,
data={
"identity": username,
"password": password,
@@ -146,13 +161,7 @@ class SpaceTrackTLECollector(BaseCollector):
return self._get_sample_data()
# Query for TLE data (get first 1000 satellites)
tle_response = await client.get(
"https://www.space-track.org/basicspacedata/query"
"/class/gp"
"/orderby/EPOCH%20desc"
"/limit/1000"
"/format/json"
)
tle_response = await client.get(self.query_url)
print(f"SPACETRACK: TLE query status: {tle_response.status_code}")
if tle_response.status_code != 200:

View File

@@ -11,6 +11,7 @@ from datetime import UTC, datetime
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
@@ -24,15 +25,17 @@ class TeleGeographyCableCollector(BaseCollector):
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch submarine cable data from Wayback Machine"""
config = get_data_sources_config()
# Try multiple data sources
sources = [
# Wayback Machine archive of TeleGeography
"https://web.archive.org/web/2024/https://www.submarinecablemap.com/api/v3/cable",
# Alternative: Try scraping the page
"https://www.submarinecablemap.com",
self._resolved_url or "",
str(config.get_yaml_value("telegeography.archived_cable_url") or ""),
str(config.get_yaml_value("telegeography.live_map_url") or ""),
]
for url in sources:
if not url:
continue
try:
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
@@ -161,7 +164,7 @@ class TeleGeographyLandingPointCollector(BaseCollector):
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch landing point data from GitHub mirror"""
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/landing_point.json"
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(url)
@@ -225,7 +228,7 @@ class TeleGeographyCableSystemCollector(BaseCollector):
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch cable system data"""
url = "https://raw.githubusercontent.com/lintaojlu/submarine_cable_information/main/cable.json"
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0) as client:
response = await client.get(url)

View File

@@ -10,6 +10,7 @@ from typing import Dict, Any, List
from bs4 import BeautifulSoup
import httpx
from app.core.data_sources import get_data_sources_config
from app.services.collectors.base import BaseCollector
@@ -22,7 +23,7 @@ class TOP500Collector(BaseCollector):
async def fetch(self) -> List[Dict[str, Any]]:
"""Fetch TOP500 list data and enrich each row with detail-page metadata."""
url = "https://top500.org/lists/top500/list/2025/11/"
url = self._resolved_url or ""
async with httpx.AsyncClient(timeout=60.0, follow_redirects=True) as client:
response = await client.get(url)
@@ -48,11 +49,13 @@ class TOP500Collector(BaseCollector):
return await asyncio.gather(*(enrich(entry) for entry in entries))
def _extract_system_fields(self, system_cell) -> Dict[str, str]:
config = get_data_sources_config()
top500_base_url = config.get_yaml_value("top500.base_url") or "https://top500.org"
link = system_cell.find("a")
system_name = link.get_text(" ", strip=True) if link else system_cell.get_text(" ", strip=True)
detail_url = ""
if link and link.get("href"):
detail_url = f"https://top500.org{link.get('href')}"
detail_url = f"{str(top500_base_url).rstrip('/')}{link.get('href')}"
manufacturer = ""
if link and link.next_sibling:

View File

@@ -0,0 +1,490 @@
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
import hashlib
import html
import re
from typing import Any
from urllib.parse import quote
import xml.etree.ElementTree as ET
import httpx
from bs4 import BeautifulSoup
USER_AGENT = "PlanetEarthNewsBoard/1.0 (+https://planet.local)"
REQUEST_TIMEOUT = 12.0
MAX_ITEMS_PER_SOURCE = 6
MAX_ITEMS_TOTAL = 12
STALE_CACHE_MAX_AGE_SECONDS = 60 * 45
@dataclass(frozen=True)
class RegionProfile:
key: str
label: str
query: str
accent: str
@dataclass(frozen=True)
class NewsFeedSource:
id: str
name: str
region: str
feed_url: str
homepage_url: str
source_type: str = "rss"
priority: int = 100
@dataclass
class ParsedNewsItem:
id: str
title: str
summary: str
url: str
source: str
feed_name: str
feed_region: str
homepage_url: str
published_at: datetime | None
@dataclass
class CachedRegionFeed:
region: str
fetched_at: datetime
items: list[ParsedNewsItem]
sources: list[NewsFeedSource]
REGION_PROFILES: dict[str, RegionProfile] = {
"americas": RegionProfile(
key="americas",
label="美洲焦点",
query='Americas geopolitics OR Latin America OR "United States" OR Canada',
accent="#79d3ff",
),
"europe": RegionProfile(
key="europe",
label="欧洲焦点",
query='Europe geopolitics OR EU OR NATO OR "Eastern Europe"',
accent="#8fd4ff",
),
"middle-east-africa": RegionProfile(
key="middle-east-africa",
label="中东与非洲焦点",
query='"Middle East" OR Africa geopolitics OR Red Sea OR Gulf',
accent="#ffb56a",
),
"asia-pacific": RegionProfile(
key="asia-pacific",
label="亚太焦点",
query='"Asia Pacific" OR Indo-Pacific OR China OR Japan OR Korea OR ASEAN',
accent="#78f2cf",
),
"global": RegionProfile(
key="global",
label="全球焦点",
query='"world news" OR geopolitics OR "global affairs"',
accent="#d6e6ff",
),
}
def _google_news_feed(query: str, *, hl: str, gl: str, ceid: str) -> str:
return (
"https://news.google.com/rss/search?q="
+ quote(query, safe="")
+ f"&hl={hl}&gl={gl}&ceid={ceid}"
)
NEWS_FEED_SOURCES: tuple[NewsFeedSource, ...] = (
NewsFeedSource(
id="bbc-world",
name="BBC World",
region="global",
feed_url="https://feeds.bbci.co.uk/news/world/rss.xml",
homepage_url="https://www.bbc.com/news/world",
priority=10,
),
NewsFeedSource(
id="dw-top",
name="DW Top Stories",
region="europe",
feed_url="https://rss.dw.com/rdf/rss-en-top",
homepage_url="https://www.dw.com/en/top-stories/s-9097",
priority=20,
),
NewsFeedSource(
id="global-scan",
name="Global Monitor / World",
region="global",
feed_url=_google_news_feed(
REGION_PROFILES["global"].query,
hl="en-US",
gl="US",
ceid="US:en",
),
homepage_url="https://news.google.com/",
source_type="aggregated",
priority=30,
),
NewsFeedSource(
id="google-americas",
name="Global Monitor / Americas",
region="americas",
feed_url=_google_news_feed(
REGION_PROFILES["americas"].query,
hl="en-US",
gl="US",
ceid="US:en",
),
homepage_url="https://news.google.com/",
source_type="aggregated",
priority=40,
),
NewsFeedSource(
id="google-europe",
name="Global Monitor / Europe",
region="europe",
feed_url=_google_news_feed(
REGION_PROFILES["europe"].query,
hl="en-GB",
gl="GB",
ceid="GB:en",
),
homepage_url="https://news.google.com/",
source_type="aggregated",
priority=40,
),
NewsFeedSource(
id="google-mea",
name="Global Monitor / MEA",
region="middle-east-africa",
feed_url=_google_news_feed(
REGION_PROFILES["middle-east-africa"].query,
hl="en-US",
gl="US",
ceid="US:en",
),
homepage_url="https://news.google.com/",
source_type="aggregated",
priority=40,
),
NewsFeedSource(
id="google-apac",
name="Global Monitor / APAC",
region="asia-pacific",
feed_url=_google_news_feed(
REGION_PROFILES["asia-pacific"].query,
hl="en-SG",
gl="SG",
ceid="SG:en",
),
homepage_url="https://news.google.com/",
source_type="aggregated",
priority=40,
),
)
_REGION_CACHE: dict[str, CachedRegionFeed] = {}
def determine_focus_region(lat: float | None, lon: float | None) -> str:
if lat is None or lon is None:
return "global"
if -170 <= lon <= -30:
return "americas"
if -30 < lon <= 45:
return "europe" if lat >= 30 else "middle-east-africa"
if 45 < lon <= 150:
return "middle-east-africa" if lat < 10 else "asia-pacific"
return "asia-pacific"
def get_region_profile(region: str) -> RegionProfile:
return REGION_PROFILES.get(region, REGION_PROFILES["global"])
def get_sources_for_region(region: str) -> list[NewsFeedSource]:
return sorted(
[source for source in NEWS_FEED_SOURCES if source.region in {"global", region}],
key=lambda source: (source.priority, source.name),
)
def _strip_html(value: str) -> str:
if not value:
return ""
soup = BeautifulSoup(value, "html.parser")
return re.sub(r"\s+", " ", soup.get_text(" ", strip=True)).strip()
def _truncate(value: str, limit: int = 180) -> str:
text = value.strip()
if len(text) <= limit:
return text
return text[: limit - 1].rstrip() + ""
def _normalize_source_name(raw: str, fallback: str) -> str:
text = html.unescape((raw or "").strip())
if " - " in text:
return text.split(" - ")[-1].strip() or fallback
return text or fallback
def _parse_datetime(raw: str | None) -> datetime | None:
if not raw:
return None
text = raw.strip()
if not text:
return None
for parser in (
lambda value: parsedate_to_datetime(value),
lambda value: datetime.fromisoformat(value.replace("Z", "+00:00")),
):
try:
parsed = parser(text)
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
except Exception:
continue
return None
def _extract_item_text(element: ET.Element, *names: str) -> str:
for name in names:
node = element.find(name)
if node is not None and node.text:
return node.text.strip()
return ""
def _parse_feed_entries(xml_text: str, source: NewsFeedSource) -> list[ParsedNewsItem]:
root = ET.fromstring(xml_text)
items: list[ParsedNewsItem] = []
rss_items = root.findall("./channel/item")
atom_entries = root.findall("{http://www.w3.org/2005/Atom}entry")
nodes = rss_items or atom_entries
for node in nodes[:MAX_ITEMS_PER_SOURCE]:
if node.tag.endswith("entry"):
title = _extract_item_text(node, "{http://www.w3.org/2005/Atom}title")
summary = _extract_item_text(
node,
"{http://www.w3.org/2005/Atom}summary",
"{http://www.w3.org/2005/Atom}content",
)
link_node = node.find("{http://www.w3.org/2005/Atom}link")
link = link_node.get("href", "").strip() if link_node is not None else ""
published = _extract_item_text(
node,
"{http://www.w3.org/2005/Atom}updated",
"{http://www.w3.org/2005/Atom}published",
)
else:
title = _extract_item_text(node, "title")
summary = _extract_item_text(node, "description", "content")
link = _extract_item_text(node, "link")
published = _extract_item_text(node, "pubDate", "published", "updated")
clean_title = html.unescape(title).strip()
clean_summary = _truncate(_strip_html(summary), 180)
if not clean_title or not link:
continue
item_source = _normalize_source_name(clean_title, source.name)
display_title = clean_title
if source.source_type == "aggregated" and " - " in clean_title:
parts = clean_title.rsplit(" - ", 1)
display_title = parts[0].strip()
item_source = _normalize_source_name(parts[1], source.name)
items.append(
ParsedNewsItem(
id=f"{source.id}:{hashlib.sha1(link.encode('utf-8')).hexdigest()[:12]}",
title=display_title,
summary=clean_summary,
url=link,
source=item_source,
feed_name=source.name,
feed_region=source.region,
homepage_url=source.homepage_url,
published_at=_parse_datetime(published),
)
)
return items
def _serialize_sources(sources: list[NewsFeedSource]) -> list[dict[str, Any]]:
return [
{
"id": source.id,
"name": source.name,
"region": source.region,
"homepage_url": source.homepage_url,
}
for source in sources
]
def _serialize_item(item: ParsedNewsItem, *, active_region: str) -> dict[str, Any]:
published_at = item.published_at
return {
"id": item.id,
"title": item.title,
"summary": item.summary,
"url": item.url,
"source": item.source,
"feed_name": item.feed_name,
"region": item.feed_region,
"homepage_url": item.homepage_url,
"published_at": published_at.isoformat().replace("+00:00", "Z") if published_at else None,
"is_focus_match": item.feed_region == active_region,
}
def _build_payload(
*,
lat: float | None,
lon: float | None,
active_region: str,
items: list[ParsedNewsItem],
sources: list[NewsFeedSource],
errors: list[str],
stale: bool,
generated_at: datetime | None = None,
) -> dict[str, Any]:
profile = get_region_profile(active_region)
timestamp = generated_at or datetime.now(UTC)
return {
"generated_at": timestamp.isoformat().replace("+00:00", "Z"),
"focus": {
"lat": lat,
"lon": lon,
"region": active_region,
"label": profile.label,
"accent": profile.accent,
},
"sources": _serialize_sources(sources),
"items": [_serialize_item(item, active_region=active_region) for item in items],
"errors": errors,
"stale": stale,
}
def _rank_and_trim_items(items: list[ParsedNewsItem], *, active_region: str) -> list[ParsedNewsItem]:
deduped: dict[str, ParsedNewsItem] = {}
for item in items:
key = item.url.strip() or item.title.strip().lower()
if key not in deduped:
deduped[key] = item
return sorted(
deduped.values(),
key=lambda item: (
item.feed_region != active_region,
item.published_at is None,
-(item.published_at.timestamp() if item.published_at else 0),
item.feed_name,
),
)[:MAX_ITEMS_TOTAL]
def _get_cached_region_feed(region: str) -> CachedRegionFeed | None:
cached = _REGION_CACHE.get(region)
if not cached:
return None
age_seconds = (datetime.now(UTC) - cached.fetched_at).total_seconds()
if age_seconds > STALE_CACHE_MAX_AGE_SECONDS:
return None
return cached
def _store_region_cache(region: str, *, items: list[ParsedNewsItem], sources: list[NewsFeedSource]) -> None:
_REGION_CACHE[region] = CachedRegionFeed(
region=region,
fetched_at=datetime.now(UTC),
items=list(items),
sources=list(sources),
)
async def _fetch_source(
client: httpx.AsyncClient,
source: NewsFeedSource,
) -> tuple[NewsFeedSource, list[ParsedNewsItem], str | None]:
try:
response = await client.get(source.feed_url)
response.raise_for_status()
return source, _parse_feed_entries(response.text, source), None
except Exception as exc:
return source, [], str(exc)
async def get_earth_news_payload(lat: float | None = None, lon: float | None = None) -> dict[str, Any]:
active_region = determine_focus_region(lat, lon)
sources = get_sources_for_region(active_region)
errors: list[str] = []
async with httpx.AsyncClient(
timeout=REQUEST_TIMEOUT,
follow_redirects=True,
headers={"User-Agent": USER_AGENT},
) as client:
results = await asyncio.gather(*(_fetch_source(client, source) for source in sources))
fetched_items: list[ParsedNewsItem] = []
for source, items, error in results:
if error:
errors.append(f"{source.name}: {error}")
continue
fetched_items.extend(items)
ranked_items = _rank_and_trim_items(fetched_items, active_region=active_region)
if ranked_items:
_store_region_cache(active_region, items=ranked_items, sources=sources)
return _build_payload(
lat=lat,
lon=lon,
active_region=active_region,
items=ranked_items,
sources=sources,
errors=errors,
stale=False,
)
cached = _get_cached_region_feed(active_region)
if cached:
return _build_payload(
lat=lat,
lon=lon,
active_region=active_region,
items=cached.items,
sources=cached.sources,
errors=errors,
stale=True,
generated_at=cached.fetched_at,
)
return _build_payload(
lat=lat,
lon=lon,
active_region=active_region,
items=[],
sources=sources,
errors=errors,
stale=False,
)

View File

@@ -0,0 +1,694 @@
from __future__ import annotations
import asyncio
from collections.abc import Sequence
from datetime import UTC, datetime
from time import perf_counter
from uuid import uuid4
from fastapi import HTTPException, status
from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import async_session_factory
from app.models.playground_message import PlaygroundMessage
from app.models.playground_session import PlaygroundSession
from app.schemas.ai import (
PlaygroundMessageEditRequest,
PlaygroundMessageActionResponse,
PlaygroundMessageCreateRequest,
PlaygroundMessageRecord,
PlaygroundMessageResendRequest,
PlaygroundMessageStopRequest,
PlaygroundSessionResponse,
PlaygroundSessionState,
PlaygroundSessionUpsertRequest,
PlaygroundThreadResponse,
SituationalAnalysisRequest,
)
from app.services.ai_client import AIProviderClient
from app.services.playground_session_store import _to_response as session_to_response
from app.services.playground_session_store import upsert_playground_session
STREAM_CHUNK_SIZE = 24
STREAM_INTERVAL_SECONDS = 0.08
THINKING_PREVIEW_SECONDS = 2.6
class _ActiveRun:
def __init__(self, task: asyncio.Task[None]) -> None:
self.task = task
self.stop_requested = asyncio.Event()
_ACTIVE_RUNS: dict[str, _ActiveRun] = {}
async def _get_session_by_key(
db: AsyncSession,
*,
user_id: int,
session_key: str,
) -> PlaygroundSession | None:
result = await db.execute(
select(PlaygroundSession).where(
PlaygroundSession.user_id == user_id,
PlaygroundSession.session_key == session_key,
)
)
return result.scalar_one_or_none()
async def _require_session(
db: AsyncSession,
*,
user_id: int,
session_key: str,
) -> PlaygroundSession:
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
if session is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Playground session not found")
return session
async def _require_visible_message(
db: AsyncSession,
*,
user_id: int,
public_id: str,
role: str | None = None,
) -> PlaygroundMessage:
conditions = [
PlaygroundMessage.user_id == user_id,
PlaygroundMessage.public_id == public_id,
PlaygroundMessage.is_visible.is_(True),
]
if role is not None:
conditions.append(PlaygroundMessage.role == role)
result = await db.execute(select(PlaygroundMessage).where(*conditions))
message = result.scalar_one_or_none()
if message is None:
detail = "User message not found" if role == "user" else "Playground message not found"
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=detail)
return message
def _message_to_record(message: PlaygroundMessage, parent_public_id: str | None = None) -> PlaygroundMessageRecord:
return PlaygroundMessageRecord(
id=message.public_id,
role=message.role,
kind=message.kind,
status=message.status,
title=message.title,
content=message.content or "",
thinking_content=message.thinking_content or "",
meta=list(message.meta or []),
markdown=message.role != "system",
provider=message.provider,
model=message.model,
request_id=message.request_id,
raw_response=dict(message.raw_response or {}),
content_blocks=list(message.content_blocks or []),
text_blocks=list(message.text_blocks or []),
thinking_blocks=list(message.thinking_blocks or []),
parent_message_id=parent_public_id,
created_at=message.created_at.isoformat(),
updated_at=message.updated_at.isoformat(),
)
async def _ensure_session(
db: AsyncSession,
*,
user_id: int,
session_key: str,
title: str,
state: PlaygroundSessionState | None = None,
) -> PlaygroundSession:
result = await db.execute(
select(PlaygroundSession).where(
PlaygroundSession.user_id == user_id,
PlaygroundSession.session_key == session_key,
)
)
session = result.scalar_one_or_none()
if session is not None:
if title:
session.title = title[:200]
if state is not None:
session.state = state.model_dump(mode="json")
await db.flush()
await db.refresh(session)
return session
payload = PlaygroundSessionUpsertRequest(
session_key=session_key,
title=title[:200],
state=state or PlaygroundSessionState(title=title[:200]),
)
await upsert_playground_session(db, user_id=user_id, payload=payload)
result = await db.execute(
select(PlaygroundSession).where(
PlaygroundSession.user_id == user_id,
PlaygroundSession.session_key == session_key,
)
)
return result.scalar_one()
async def _list_visible_messages(
db: AsyncSession,
*,
session_id: int,
) -> list[PlaygroundMessage]:
result = await db.execute(
select(PlaygroundMessage)
.where(
PlaygroundMessage.session_id == session_id,
PlaygroundMessage.is_visible.is_(True),
)
.order_by(PlaygroundMessage.sort_order.asc(), PlaygroundMessage.id.asc())
)
return list(result.scalars().all())
async def _build_thread_response(
db: AsyncSession,
*,
session: PlaygroundSession,
) -> PlaygroundThreadResponse:
messages = await _list_visible_messages(db, session_id=session.id)
id_map = {item.id: item.public_id for item in messages}
return PlaygroundThreadResponse(
session=session_to_response(session),
messages=[_message_to_record(item, id_map.get(item.parent_message_id)) for item in messages],
)
async def get_thread(
db: AsyncSession,
*,
user_id: int,
session_key: str,
) -> PlaygroundThreadResponse | None:
session = await _get_session_by_key(db, user_id=user_id, session_key=session_key)
if session is None:
return None
return await _build_thread_response(db, session=session)
async def _build_action_response(
db: AsyncSession,
*,
session: PlaygroundSession,
active_message_id: str | None = None,
) -> PlaygroundMessageActionResponse:
thread = await _build_thread_response(db, session=session)
return PlaygroundMessageActionResponse(
session=thread.session,
messages=thread.messages,
active_message_id=active_message_id,
)
def _collect_constraints(raw_constraints: str) -> list[str]:
return [item.strip() for item in raw_constraints.split("\n") if item.strip()]
async def _next_sort_order(db: AsyncSession, session_id: int) -> int:
result = await db.execute(
select(func.max(PlaygroundMessage.sort_order)).where(PlaygroundMessage.session_id == session_id)
)
current = result.scalar_one_or_none()
return int(current or 0)
async def _set_session_state(
db: AsyncSession,
*,
session: PlaygroundSession,
payload: PlaygroundMessageCreateRequest,
) -> PlaygroundSession:
session.state = PlaygroundSessionState(
messages=[],
selectedPresetKey=payload.selected_preset_key,
title=payload.title,
objective=payload.objective,
constraints=payload.constraints,
inputValue="",
analysis=None,
latestAnalysisMessageId=None,
analysisMeta={},
helpExpanded=payload.help_expanded,
).model_dump(mode="json")
session.title = payload.title[:200]
await db.flush()
await db.refresh(session)
return session
def _spawn_assistant_run(
*,
user_id: int,
session_id: int,
session_key: str,
user_message_id: int,
assistant_message_id: int,
assistant_public_id: str,
payload: PlaygroundMessageCreateRequest,
provider_client: AIProviderClient,
) -> None:
task = asyncio.create_task(
_run_assistant_message(
user_id=user_id,
session_id=session_id,
session_key=session_key,
user_message_id=user_message_id,
assistant_message_id=assistant_message_id,
payload=payload,
provider_client=provider_client,
)
)
_ACTIVE_RUNS[assistant_public_id] = _ActiveRun(task)
async def create_turn(
db: AsyncSession,
*,
user_id: int,
payload: PlaygroundMessageCreateRequest,
provider_client: AIProviderClient,
) -> PlaygroundMessageActionResponse:
session = await _ensure_session(
db,
user_id=user_id,
session_key=payload.session_key,
title=payload.title,
state=PlaygroundSessionState(
selectedPresetKey=payload.selected_preset_key,
title=payload.title,
objective=payload.objective,
constraints=payload.constraints,
inputValue="",
helpExpanded=payload.help_expanded,
),
)
session = await _set_session_state(db, session=session, payload=payload)
base_order = await _next_sort_order(db, session.id)
user_message = PlaygroundMessage(
public_id=uuid4().hex,
session_id=session.id,
user_id=user_id,
role="user",
kind="message",
status="done",
title=payload.selected_preset_key,
content=payload.input,
meta=[payload.title],
sort_order=base_order + 10,
)
assistant_message = PlaygroundMessage(
public_id=uuid4().hex,
session_id=session.id,
user_id=user_id,
parent_message_id=None,
role="assistant",
kind="thinking",
status="pending",
title="AI 回应",
content="",
thinking_content="",
meta=[],
sort_order=base_order + 20,
)
db.add(user_message)
await db.flush()
assistant_message.parent_message_id = user_message.id
db.add(assistant_message)
await db.flush()
await db.refresh(user_message)
await db.refresh(assistant_message)
await db.commit()
await db.refresh(session)
await db.refresh(user_message)
await db.refresh(assistant_message)
_spawn_assistant_run(
user_id=user_id,
session_id=session.id,
session_key=payload.session_key,
user_message_id=user_message.id,
assistant_message_id=assistant_message.id,
assistant_public_id=assistant_message.public_id,
payload=payload,
provider_client=provider_client,
)
return await _build_action_response(
db,
session=session,
active_message_id=assistant_message.public_id,
)
async def _create_assistant_retry_turn(
db: AsyncSession,
*,
user_id: int,
session: PlaygroundSession,
user_message: PlaygroundMessage,
payload: PlaygroundMessageCreateRequest,
provider_client: AIProviderClient,
) -> PlaygroundMessageActionResponse:
base_order = await _next_sort_order(db, session.id)
assistant_message = PlaygroundMessage(
public_id=uuid4().hex,
session_id=session.id,
user_id=user_id,
parent_message_id=user_message.id,
role="assistant",
kind="thinking",
status="pending",
title="AI 回应",
content="",
thinking_content="",
meta=[],
sort_order=base_order + 10,
)
db.add(assistant_message)
await db.flush()
await db.commit()
await db.refresh(session)
await db.refresh(assistant_message)
_spawn_assistant_run(
user_id=user_id,
session_id=session.id,
session_key=payload.session_key,
user_message_id=user_message.id,
assistant_message_id=assistant_message.id,
assistant_public_id=assistant_message.public_id,
payload=payload,
provider_client=provider_client,
)
return await _build_action_response(
db,
session=session,
active_message_id=assistant_message.public_id,
)
async def stop_message(
db: AsyncSession,
*,
user_id: int,
payload: PlaygroundMessageStopRequest,
) -> PlaygroundMessageActionResponse:
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
message = await _require_visible_message(db, user_id=user_id, public_id=payload.message_id)
if message.status not in {"pending", "thinking", "answering"}:
return await _build_action_response(db, session=session)
active_run = _ACTIVE_RUNS.get(message.public_id)
if active_run is not None:
active_run.stop_requested.set()
active_run.task.cancel()
message.status = "stopped"
if "已手动停止生成" not in (message.meta or []):
message.meta = [*(message.meta or []), "已手动停止生成"]
await db.flush()
await db.commit()
await db.refresh(message)
return await _build_action_response(db, session=session)
async def resend_turn(
db: AsyncSession,
*,
user_id: int,
payload: PlaygroundMessageResendRequest,
provider_client: AIProviderClient,
) -> PlaygroundMessageActionResponse:
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
user_message = await _require_visible_message(
db,
user_id=user_id,
public_id=payload.user_message_id,
role="user",
)
later_messages = await db.execute(
select(PlaygroundMessage).where(
PlaygroundMessage.session_id == session.id,
PlaygroundMessage.sort_order > user_message.sort_order,
PlaygroundMessage.is_visible.is_(True),
)
)
for item in later_messages.scalars().all():
item.is_visible = False
if item.status in {"pending", "thinking", "answering"}:
active_run = _ACTIVE_RUNS.get(item.public_id)
if active_run is not None:
active_run.stop_requested.set()
active_run.task.cancel()
await db.flush()
await db.commit()
session_state = PlaygroundSessionState.model_validate(session.state or {})
create_payload = PlaygroundMessageCreateRequest(
session_key=payload.session_key,
title=session_state.title or session.title,
objective=session_state.objective or "继续当前对话",
constraints=session_state.constraints or "",
input=user_message.content,
selected_preset_key=session_state.selectedPresetKey or "bgp-brief",
help_expanded=session_state.helpExpanded,
)
return await _create_assistant_retry_turn(
db,
user_id=user_id,
session=session,
user_message=user_message,
payload=create_payload,
provider_client=provider_client,
)
async def edit_user_message(
db: AsyncSession,
*,
user_id: int,
payload: PlaygroundMessageEditRequest,
) -> PlaygroundMessageActionResponse:
session = await _require_session(db, user_id=user_id, session_key=payload.session_key)
user_message = await _require_visible_message(
db,
user_id=user_id,
public_id=payload.user_message_id,
role="user",
)
user_message.content = payload.content.strip()
await db.flush()
await db.commit()
await db.refresh(user_message)
return await _build_action_response(db, session=session)
async def _append_meta_if_missing(db: AsyncSession, message_id: int, meta_line: str) -> None:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
message = result.scalar_one_or_none()
if message is None:
return
if meta_line not in (message.meta or []):
message.meta = [*(message.meta or []), meta_line]
await db.flush()
async def _should_stop(message_public_id: str) -> bool:
active_run = _ACTIVE_RUNS.get(message_public_id)
return active_run.stop_requested.is_set() if active_run is not None else False
async def _mark_message_state(
db: AsyncSession,
*,
message_id: int,
**updates,
) -> PlaygroundMessage:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == message_id))
message = result.scalar_one()
for key, value in updates.items():
setattr(message, key, value)
await db.flush()
await db.refresh(message)
return message
def _build_conversation_history(messages: Sequence[PlaygroundMessage], current_user_message_id: int) -> list[dict]:
history: list[dict] = []
for item in messages:
if item.id >= current_user_message_id:
break
if item.role == "system":
continue
history.append(
{
"role": item.role,
"kind": item.kind or "message",
"title": item.title,
"content": item.content or "",
}
)
return history[-8:]
async def _run_assistant_message(
*,
user_id: int,
session_id: int,
session_key: str,
user_message_id: int,
assistant_message_id: int,
payload: PlaygroundMessageCreateRequest,
provider_client: AIProviderClient,
) -> None:
request_id = str(uuid4())
started_at = perf_counter()
assistant_public_id: str | None = None
try:
async with async_session_factory() as db:
session = await db.get(PlaygroundSession, session_id)
user_message = await db.get(PlaygroundMessage, user_message_id)
assistant_message = await db.get(PlaygroundMessage, assistant_message_id)
if session is None or user_message is None or assistant_message is None:
return
assistant_public_id = assistant_message.public_id
visible_messages = await _list_visible_messages(db, session_id=session_id)
conversation_history = _build_conversation_history(visible_messages, user_message_id)
request_payload = SituationalAnalysisRequest(
title=payload.title,
objective=payload.objective,
observations=[item.strip() for item in payload.input.split("\n") if item.strip()],
constraints=_collect_constraints(payload.constraints),
context={
"source": "playground",
"preset": payload.selected_preset_key,
"conversation_history": conversation_history,
"history_size": len(conversation_history),
},
thinking={"type": "enabled"},
)
analysis = await provider_client.analyze(request_payload, request_id=request_id)
async with async_session_factory() as db:
assistant_message = await _mark_message_state(
db,
message_id=assistant_message_id,
status="thinking" if analysis.thinking_blocks else "answering",
title=f"{analysis.provider} / {analysis.model}",
provider=analysis.provider,
model=analysis.model,
request_id=request_id,
raw_response=analysis.raw_response,
content_blocks=[item.model_dump(mode="json") for item in analysis.content_blocks],
text_blocks=analysis.text_blocks,
thinking_blocks=analysis.thinking_blocks,
thinking_content="\n\n".join(analysis.thinking_blocks).strip(),
)
session = await db.get(PlaygroundSession, session_id)
if session is not None:
session_state = PlaygroundSessionState.model_validate(session.state or {})
session.state = session_state.model_copy(
update={
"latestAnalysisMessageId": assistant_message.public_id,
"analysis": analysis.model_dump(mode="json"),
}
).model_dump(mode="json")
await db.flush()
await db.commit()
if assistant_public_id and analysis.thinking_blocks:
await asyncio.sleep(THINKING_PREVIEW_SECONDS)
if await _should_stop(assistant_public_id):
return
content = analysis.content or ""
cursor = 0
while cursor < len(content):
if assistant_public_id and await _should_stop(assistant_public_id):
return
cursor = min(len(content), cursor + STREAM_CHUNK_SIZE)
async with async_session_factory() as db:
await _mark_message_state(
db,
message_id=assistant_message_id,
status="answering",
content=content[:cursor],
)
await db.commit()
await asyncio.sleep(STREAM_INTERVAL_SECONDS)
duration_ms = round((perf_counter() - started_at) * 1000)
async with async_session_factory() as db:
assistant_message = await _mark_message_state(
db,
message_id=assistant_message_id,
status="done",
content=content,
meta=[
f"Request ID: {request_id}",
f"耗时: {duration_ms} ms",
f"完成时间: {datetime.now(UTC).astimezone().isoformat()}",
],
)
session = await db.get(PlaygroundSession, session_id)
if session is not None:
session_state = PlaygroundSessionState.model_validate(session.state or {})
session.state = session_state.model_copy(
update={
"latestAnalysisMessageId": assistant_message.public_id,
"analysis": analysis.model_dump(mode="json"),
"analysisMeta": {
"requestId": request_id,
"durationMs": duration_ms,
"completedAt": datetime.now().isoformat(),
},
}
).model_dump(mode="json")
await db.flush()
await db.commit()
except asyncio.CancelledError:
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
if message is not None and message.status in {"pending", "thinking", "answering"}:
message.status = "stopped"
if "已手动停止生成" not in (message.meta or []):
message.meta = [*(message.meta or []), "已手动停止生成"]
await db.flush()
await db.commit()
raise
except Exception as exc:
async with async_session_factory() as db:
result = await db.execute(select(PlaygroundMessage).where(PlaygroundMessage.id == assistant_message_id))
message = result.scalar_one_or_none()
if message is not None:
message.status = "error"
message.content = message.content or "分析失败,请检查 AI Provider 配置或稍后再试。"
message.meta = [*(message.meta or []), f"错误: {type(exc).__name__}"]
await db.flush()
await db.commit()
finally:
if assistant_public_id:
_ACTIVE_RUNS.pop(assistant_public_id, None)

View File

@@ -0,0 +1,72 @@
from __future__ import annotations
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.playground_session import PlaygroundSession
from app.schemas.ai import (
PlaygroundSessionResponse,
PlaygroundSessionState,
PlaygroundSessionUpsertRequest,
)
def _to_response(record: PlaygroundSession) -> PlaygroundSessionResponse:
return PlaygroundSessionResponse(
id=str(record.id),
session_key=record.session_key,
title=record.title,
state=PlaygroundSessionState.model_validate(record.state or {}),
created_at=record.created_at.isoformat(),
updated_at=record.updated_at.isoformat(),
)
async def get_playground_session(
db: AsyncSession,
*,
user_id: int,
session_key: str = "default",
) -> PlaygroundSessionResponse | None:
result = await db.execute(
select(PlaygroundSession).where(
PlaygroundSession.user_id == user_id,
PlaygroundSession.session_key == session_key,
)
)
record = result.scalar_one_or_none()
if record is None:
return None
return _to_response(record)
async def upsert_playground_session(
db: AsyncSession,
*,
user_id: int,
payload: PlaygroundSessionUpsertRequest,
) -> PlaygroundSessionResponse:
result = await db.execute(
select(PlaygroundSession).where(
PlaygroundSession.user_id == user_id,
PlaygroundSession.session_key == payload.session_key,
)
)
record = result.scalar_one_or_none()
title = (payload.title or payload.state.title or "Playground 会话").strip()[:200] or "Playground 会话"
if record is None:
record = PlaygroundSession(
user_id=user_id,
session_key=payload.session_key,
title=title,
state=payload.state.model_dump(mode="json"),
)
db.add(record)
else:
record.title = title
record.state = payload.state.model_dump(mode="json")
await db.flush()
await db.refresh(record)
return _to_response(record)

View File

@@ -19,6 +19,30 @@ logger = logging.getLogger(__name__)
scheduler = AsyncIOScheduler()
RUNNING_TASK_GUARD_TIMEOUT_MINUTES = 90
RUNNING_COLLECTOR_TASKS: dict[str, asyncio.Task[Any]] = {}
def _collector_task_name(collector_name: str) -> str:
return f"collector:{collector_name}"
def get_running_collector_task(collector_name: str) -> asyncio.Task[Any] | None:
task = RUNNING_COLLECTOR_TASKS.get(collector_name)
if task is not None and not task.done():
return task
if task is not None and task.done():
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
target_name = _collector_task_name(collector_name)
for candidate in asyncio.all_tasks():
if candidate.done():
continue
if candidate.get_name() == target_name:
RUNNING_COLLECTOR_TASKS[collector_name] = candidate
return candidate
return None
async def _update_next_run_at(datasource: DataSource, session) -> None:
@@ -133,6 +157,12 @@ async def run_collector_task(collector_name: str):
datasource.last_status = task_result.get("status")
await _update_next_run_at(datasource, db)
logger.info("Collector %s completed: %s", collector_name, task_result)
except asyncio.CancelledError:
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = "cancelled"
await db.commit()
logger.warning("Collector %s cancelled by operator", collector_name)
raise
except Exception as exc:
datasource.last_run_at = datetime.now(UTC)
datasource.last_status = "failed"
@@ -244,10 +274,37 @@ def run_collector_now(collector_name: str) -> bool:
logger.error("Collector not found: %s", collector_name)
return False
existing_task = get_running_collector_task(collector_name)
if existing_task is not None and not existing_task.done():
logger.warning("Collector %s is already running in-memory; skipping duplicate trigger", collector_name)
return False
try:
asyncio.create_task(run_collector_task(collector_name))
task = asyncio.create_task(run_collector_task(collector_name), name=_collector_task_name(collector_name))
RUNNING_COLLECTOR_TASKS[collector_name] = task
def _cleanup_task(done_task: asyncio.Task[Any]) -> None:
current = RUNNING_COLLECTOR_TASKS.get(collector_name)
if current is done_task:
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
task.add_done_callback(_cleanup_task)
logger.info("Triggered collector: %s", collector_name)
return True
except Exception as exc:
logger.error("Failed to trigger collector %s: %s", collector_name, exc)
return False
return False
async def cancel_running_collector_now(collector_name: str) -> bool:
task = get_running_collector_task(collector_name)
if task is None or task.done():
RUNNING_COLLECTOR_TASKS.pop(collector_name, None)
return False
task.cancel()
try:
await task
except asyncio.CancelledError:
return True
return task.cancelled()

View File

@@ -0,0 +1,174 @@
from __future__ import annotations
from collections import Counter
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.alert import Alert, AlertSeverity, AlertStatus
from app.models.bgp_anomaly import BGPAnomaly
from app.models.bgp_incident import BGPIncident
from app.schemas.ai import SituationalAnalysisRequest
from app.services.bgp_ai_brief_store import get_latest_bgp_brief_record
def _format_pairs(pairs: list[tuple[str, int]], empty_text: str = "") -> str:
if not pairs:
return empty_text
return "".join(f"{key} {value}" for key, value in pairs if key)
async def build_situational_alert_brief_request(
db: AsyncSession,
) -> tuple[SituationalAnalysisRequest, list[str], dict[str, Any]]:
total_alerts_result = await db.execute(select(func.count(Alert.id)))
active_alerts_result = await db.execute(
select(func.count(Alert.id)).where(Alert.status == AlertStatus.ACTIVE)
)
alert_severity_result = await db.execute(
select(Alert.severity, func.count(Alert.id))
.where(Alert.status == AlertStatus.ACTIVE)
.group_by(Alert.severity)
)
alert_source_result = await db.execute(
select(Alert.datasource_name, func.count(Alert.id))
.where(Alert.status == AlertStatus.ACTIVE)
.group_by(Alert.datasource_name)
.order_by(func.count(Alert.id).desc())
.limit(6)
)
recent_alerts_result = await db.execute(
select(Alert)
.order_by(Alert.created_at.desc(), Alert.id.desc())
.limit(6)
)
total_incidents_result = await db.execute(select(func.count(BGPIncident.id)))
active_incidents_result = await db.execute(
select(func.count(BGPIncident.id)).where(BGPIncident.status == "active")
)
bgp_severity_result = await db.execute(
select(BGPIncident.severity, func.count(BGPIncident.id))
.where(BGPIncident.status == "active")
.group_by(BGPIncident.severity)
)
bgp_region_counter: Counter[str] = Counter()
recent_incidents_result = await db.execute(
select(BGPIncident)
.order_by(BGPIncident.created_at.desc(), BGPIncident.id.desc())
.limit(5)
)
total_anomalies_result = await db.execute(select(func.count(BGPAnomaly.id)))
active_anomalies_result = await db.execute(
select(func.count(BGPAnomaly.id)).where(BGPAnomaly.status == "active")
)
anomaly_type_result = await db.execute(
select(BGPAnomaly.anomaly_type, func.count(BGPAnomaly.id))
.where(BGPAnomaly.status == "active")
.group_by(BGPAnomaly.anomaly_type)
.order_by(func.count(BGPAnomaly.id).desc())
.limit(6)
)
recent_incidents = recent_incidents_result.scalars().all()
for incident in recent_incidents:
for region in incident.affected_regions or []:
if not isinstance(region, dict):
continue
label = ", ".join(part for part in [region.get("city"), region.get("country")] if part) or "未知区域"
bgp_region_counter[label] += 1
latest_bgp_brief = get_latest_bgp_brief_record()
active_alert_severities = [
(item[0].value if isinstance(item[0], AlertSeverity) else str(item[0]), item[1])
for item in alert_severity_result.fetchall()
if item[0]
]
active_bgp_severities = [
(str(item[0]), item[1])
for item in bgp_severity_result.fetchall()
if item[0]
]
active_anomaly_types = [(str(item[0]), item[1]) for item in anomaly_type_result.fetchall() if item[0]]
active_alert_sources = [
(str(item[0] or "未命名数据源"), item[1])
for item in alert_source_result.fetchall()
]
facts = [
(
f"系统告警侧:总告警 {total_alerts_result.scalar() or 0}active {active_alerts_result.scalar() or 0} 条;"
f"活跃告警严重度分布为 {_format_pairs(active_alert_severities)}"
),
(
f"BGP态势侧累计 incidents {total_incidents_result.scalar() or 0}active incidents {active_incidents_result.scalar() or 0} 条;"
f"活跃 incidents 严重度分布为 {_format_pairs(active_bgp_severities)}"
),
(
f"BGP异常侧累计 anomalies {total_anomalies_result.scalar() or 0}active anomalies {active_anomalies_result.scalar() or 0} 条;"
f"活跃 anomaly 类型分布为 {_format_pairs(active_anomaly_types)}"
),
]
if active_alert_sources:
facts.append(f"当前系统告警主要集中在:{_format_pairs(active_alert_sources)}")
if bgp_region_counter:
facts.append(f"BGP近期高风险区域线索{_format_pairs(bgp_region_counter.most_common(5))}")
recent_alerts = recent_alerts_result.scalars().all()
if recent_alerts:
facts.append(
"最近系统告警摘录:"
+ "".join(
[
f"{alert.datasource_name or '未命名数据源'} / {alert.severity.value if alert.severity else '-'} / {alert.status.value if alert.status else '-'} / {alert.message or '-'}"
for alert in recent_alerts
]
)
)
if recent_incidents:
facts.append(
"最近BGP事件摘录"
+ "".join(
[
f"{incident.incident_type} / {incident.severity} / {incident.status} / {incident.summary}"
for incident in recent_incidents
]
)
)
if latest_bgp_brief:
facts.append(
f"最近一份 BGP AI 简报生成于 {latest_bgp_brief.generated_at},模型 {latest_bgp_brief.model},可作为当前态势的补充说明。"
)
context = {
"source": "situational-alerts",
"active_system_alerts": active_alerts_result.scalar() or 0,
"active_system_alert_severities": dict(active_alert_severities),
"top_system_alert_sources": dict(active_alert_sources),
"active_bgp_incidents": active_incidents_result.scalar() or 0,
"active_bgp_incident_severities": dict(active_bgp_severities),
"active_bgp_anomalies": active_anomalies_result.scalar() or 0,
"active_bgp_anomaly_types": dict(active_anomaly_types),
"bgp_hot_regions": dict(bgp_region_counter.most_common(5)),
"latest_bgp_brief_id": latest_bgp_brief.id if latest_bgp_brief else None,
"latest_bgp_brief_generated_at": latest_bgp_brief.generated_at if latest_bgp_brief else None,
}
request = SituationalAnalysisRequest(
title="态势告警 AI 简报",
objective="综合系统告警、BGP incidents、BGP anomalies 与近期 BGP AI 简报,生成一份面向值班人员的态势告警简报,指出当前最需要关注的风险域、跨模块联动迹象和优先动作。",
observations=facts,
constraints=[
"明确区分事实、推断与建议。",
"优先指出仍在 active 状态的系统告警与 BGP 风险是否存在联动。",
"不要把单一数据源的局部异常夸大成全局态势。",
"如果证据不足,请明确写出仍缺哪些模块或区域信息。",
],
context=context,
)
return request, facts, context

View File

@@ -0,0 +1,466 @@
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from urllib.parse import urlparse
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.time import to_iso8601_utc
from app.models.collected_data import CollectedData
from app.models.system_setting import SystemSetting
DEFAULT_TV_SOURCE_ID = "cgtn-en"
TV_SETTINGS_CATEGORY = "tv"
TV_LIVE_SOURCE_COLLECTOR = "news_live_streams"
TV_LIVE_SOURCE_DATA_TYPE = "news_live_stream"
DEFAULT_TV_SETTINGS = {
"default_source_id": DEFAULT_TV_SOURCE_ID,
"auto_fallback": True,
"sources": [
{
"id": "cctv4",
"name": "CCTV-4 中文国际",
"provider": "CCTV",
"region": "China",
"language": "zh-CN",
"source_type": "hls",
"embed_url": "https://tv.cctv.com/live/cctv4/",
"stream_url": "https://ldocctvwbcdtxy.liveplay.myqcloud.com/ldocctvwbcd/cdrmldcctv4_1_td.m3u8",
"homepage_url": "https://tv.cctv.com/live/cctv4/",
"poster_url": "",
"is_enabled": True,
"is_fallback": True,
"sort_order": 10,
"collector_source": None,
"notes": "默认兜底新闻直播源。优先尝试 CCTV-4 官方 HLS 播放流,若直播放失败则回退到央视官网直播页。",
},
{
"id": "reuters-tv",
"name": "Reuters TV",
"provider": "Reuters",
"region": "Global",
"language": "en",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://reuters-reutersnow-1-eu.rakuten.wurl.tv/playlist.m3u8",
"homepage_url": "https://www.reuters.com/video/live/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 20,
"collector_source": None,
"notes": "参考 worldmonitor 的默认新闻频道清单,优先作为全球英文新闻直播放源。",
},
{
"id": "cgtn-en",
"name": "CGTN English",
"provider": "CGTN",
"region": "Global",
"language": "en",
"source_type": "youtube",
"embed_url": "https://www.youtube.com/watch?v=BOy2xDU1LC8",
"stream_url": "https://news.cgtn.com/resource/live/english/cgtn-news.m3u8",
"youtube_video_id": "BOy2xDU1LC8",
"homepage_url": "https://news.cgtn.com/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 30,
"collector_source": None,
"notes": "优先使用官方 YouTube 直播源,保留 HLS 直播放流作为候选信息。",
},
{
"id": "cgtn-es",
"name": "CGTN Espanol",
"provider": "CGTN",
"region": "Latin America",
"language": "es",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://news.cgtn.com/resource/live/espanol/cgtn-e.m3u8",
"homepage_url": "https://news.cgtn.com/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 40,
"collector_source": None,
"notes": "西语国际新闻频道,覆盖拉美方向态势。",
},
{
"id": "dw-espanol",
"name": "DW Espanol",
"provider": "Deutsche Welle",
"region": "Europe",
"language": "es",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://dwamdstream104.akamaized.net/hls/live/2015530/dwstream104/stream04/streamPlaylist.m3u8",
"homepage_url": "https://www.dw.com/es/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 50,
"collector_source": None,
"notes": "来自 worldmonitor 可选频道清单的直播放源。",
},
{
"id": "dw-arabic",
"name": "DW Arabic",
"provider": "Deutsche Welle",
"region": "Middle East",
"language": "ar",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://dwamdstream103.akamaized.net/hls/live/2015526/dwstream103/index.m3u8",
"homepage_url": "https://www.dw.com/ar/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 60,
"collector_source": None,
"notes": "阿拉伯语新闻流,适合作为中东方向新闻补充源。",
},
{
"id": "aljazeera-mubasher",
"name": "Al Jazeera Mubasher",
"provider": "Al Jazeera",
"region": "Middle East",
"language": "ar",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://live-hls-web-ajm.getaj.net/AJM/index.m3u8",
"homepage_url": "https://www.aljazeera.net/live",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 70,
"collector_source": None,
"notes": "中东实时新闻流,来自 worldmonitor HLS 频道目录。",
},
{
"id": "arirang-news",
"name": "Arirang News",
"provider": "Arirang",
"region": "Korea",
"language": "en",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://amdlive-ch01-ctnd-com.akamaized.net/arirang_1ch/smil:arirang_1ch.smil/playlist.m3u8",
"homepage_url": "https://www.arirang.com/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 80,
"collector_source": None,
"notes": "东北亚英语新闻源,适合补充韩半岛与东亚视角。",
},
{
"id": "abp-news",
"name": "ABP News",
"provider": "ABP",
"region": "India",
"language": "hi",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://abplivetv.pc.cdn.bitgravity.com/httppush/abp_livetv/abp_abpnews/master.m3u8",
"homepage_url": "https://news.abplive.com/live-tv",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 90,
"collector_source": None,
"notes": "印度新闻直播放源,补充南亚区域视角。",
},
{
"id": "sabc-news",
"name": "SABC News",
"provider": "SABC",
"region": "Africa",
"language": "en",
"source_type": "hls",
"embed_url": "",
"stream_url": "https://sabconetanw.cdn.mangomolo.com/news/smil:news.stream.smil/playlist.m3u8",
"homepage_url": "https://www.sabcnews.com/sabcnews/",
"poster_url": "",
"is_enabled": True,
"is_fallback": False,
"sort_order": 100,
"collector_source": None,
"notes": "非洲英语新闻源,补充非洲区域新闻覆盖。",
},
],
}
def _clean_text(value: Any) -> str:
if value is None:
return ""
return str(value).strip()
def _clean_url(value: Any) -> str:
text = _clean_text(value)
if not text:
return ""
parsed = urlparse(text)
if parsed.scheme and parsed.scheme not in {"http", "https"}:
return ""
if parsed.scheme and not parsed.netloc:
return ""
return text
def _clean_bool(value: Any, *, default: bool) -> bool:
if isinstance(value, bool):
return value
if value in (None, ""):
return default
if isinstance(value, str):
lowered = value.strip().lower()
if lowered in {"1", "true", "yes", "on"}:
return True
if lowered in {"0", "false", "no", "off"}:
return False
return bool(value)
def _clean_int(value: Any, *, default: int) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def normalize_tv_source(source: dict[str, Any] | None, *, index: int = 0) -> dict[str, Any]:
payload = dict(source or {})
source_id = _clean_text(payload.get("id")) or f"tv-source-{index + 1}"
source_type = _clean_text(payload.get("source_type")).lower()
youtube_video_id = _clean_text(payload.get("youtube_video_id"))
youtube_channel = _clean_text(payload.get("youtube_channel"))
if source_type not in {"iframe", "hls", "video", "external", "youtube"}:
if youtube_video_id or youtube_channel:
source_type = "youtube"
else:
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
if source_type == "youtube" and not youtube_video_id and not youtube_channel:
source_type = "iframe" if _clean_text(payload.get("embed_url")) else "external"
return {
"id": source_id,
"name": _clean_text(payload.get("name")) or f"新闻直播源 {index + 1}",
"provider": _clean_text(payload.get("provider")) or "Unknown",
"region": _clean_text(payload.get("region")) or "Global",
"language": _clean_text(payload.get("language")) or "und",
"source_type": source_type,
"embed_url": _clean_url(payload.get("embed_url")),
"stream_url": _clean_url(payload.get("stream_url")),
"homepage_url": _clean_url(payload.get("homepage_url")),
"poster_url": _clean_url(payload.get("poster_url")),
"youtube_video_id": youtube_video_id,
"youtube_channel": youtube_channel,
"is_enabled": _clean_bool(payload.get("is_enabled"), default=True),
"is_fallback": _clean_bool(payload.get("is_fallback"), default=False),
"sort_order": _clean_int(payload.get("sort_order"), default=(index + 1) * 10),
"collector_source": payload.get("collector_source"),
"notes": _clean_text(payload.get("notes")),
"updated_at": _clean_text(payload.get("updated_at")),
}
def normalize_tv_settings(payload: dict[str, Any] | None) -> dict[str, Any]:
merged = {
"default_source_id": DEFAULT_TV_SETTINGS["default_source_id"],
"auto_fallback": DEFAULT_TV_SETTINGS["auto_fallback"],
"sources": [],
}
raw_sources = []
if isinstance(payload, dict):
merged["default_source_id"] = (
_clean_text(payload.get("default_source_id")) or merged["default_source_id"]
)
merged["auto_fallback"] = _clean_bool(
payload.get("auto_fallback"),
default=DEFAULT_TV_SETTINGS["auto_fallback"],
)
if isinstance(payload.get("sources"), list):
raw_sources = payload["sources"]
if not raw_sources:
raw_sources = DEFAULT_TV_SETTINGS["sources"]
normalized_sources = [
normalize_tv_source(source, index=index)
for index, source in enumerate(raw_sources)
]
if not any(source["id"] == DEFAULT_TV_SOURCE_ID for source in normalized_sources):
normalized_sources.append(
normalize_tv_source(DEFAULT_TV_SETTINGS["sources"][0], index=len(normalized_sources))
)
default_source_exists = any(
source["id"] == merged["default_source_id"] and source["is_enabled"]
for source in normalized_sources
)
if not default_source_exists:
fallback_source = next(
(source for source in normalized_sources if source["is_fallback"] and source["is_enabled"]),
None,
)
first_enabled_source = next(
(source for source in normalized_sources if source["is_enabled"]),
None,
)
merged["default_source_id"] = (
fallback_source["id"]
if fallback_source
else first_enabled_source["id"]
if first_enabled_source
else DEFAULT_TV_SOURCE_ID
)
merged["sources"] = sorted(
normalized_sources,
key=lambda item: (item["sort_order"], item["name"], item["id"]),
)
return merged
async def get_tv_settings_payload(db: AsyncSession) -> dict[str, Any]:
result = await db.execute(
select(SystemSetting).where(SystemSetting.category == TV_SETTINGS_CATEGORY)
)
record = result.scalar_one_or_none()
payload = record.payload if record else None
return normalize_tv_settings(payload)
def _build_collected_tv_source(record: CollectedData, index: int) -> dict[str, Any]:
metadata = dict(record.extra_data or {})
return normalize_tv_source(
{
"id": metadata.get("id") or record.source_id or record.entity_key,
"name": record.name or record.title or metadata.get("name") or f"采集直播源 {index + 1}",
"provider": metadata.get("provider") or metadata.get("publisher") or "Collector",
"region": metadata.get("region") or metadata.get("country") or "Global",
"language": metadata.get("language") or "und",
"source_type": metadata.get("source_type") or "iframe",
"embed_url": metadata.get("embed_url") or metadata.get("url") or "",
"stream_url": metadata.get("stream_url") or "",
"homepage_url": metadata.get("homepage_url") or metadata.get("source_url") or "",
"poster_url": metadata.get("poster_url") or "",
"youtube_video_id": metadata.get("youtube_video_id") or metadata.get("video_id") or "",
"youtube_channel": metadata.get("youtube_channel") or metadata.get("channel_handle") or "",
"is_enabled": metadata.get("is_enabled", True),
"is_fallback": False,
"sort_order": metadata.get("sort_order", 200 + index),
"collector_source": record.source,
"notes": record.description or metadata.get("notes") or "",
"updated_at": to_iso8601_utc(record.collected_at or record.reference_date or datetime.now(UTC)),
},
index=index,
)
async def get_collected_tv_sources(db: AsyncSession) -> list[dict[str, Any]]:
result = await db.execute(
select(CollectedData)
.where(CollectedData.source == TV_LIVE_SOURCE_COLLECTOR)
.where(CollectedData.data_type == TV_LIVE_SOURCE_DATA_TYPE)
.where(CollectedData.is_current.is_(True))
.where(CollectedData.is_valid == 1)
.order_by(CollectedData.reference_date.desc().nullslast(), CollectedData.id.desc())
)
rows = result.scalars().all()
return [_build_collected_tv_source(record, index) for index, record in enumerate(rows)]
def build_public_tv_payload(
settings_payload: dict[str, Any],
collected_sources: list[dict[str, Any]],
) -> dict[str, Any]:
configured_sources = [
source for source in settings_payload["sources"] if source["is_enabled"]
]
merged_by_id = {source["id"]: source for source in configured_sources}
for source in collected_sources:
if source["id"] in merged_by_id or not source["is_enabled"]:
continue
merged_by_id[source["id"]] = source
available_sources = sorted(
merged_by_id.values(),
key=lambda item: (item["sort_order"], item["name"], item["id"]),
)
default_source = next(
(
source
for source in available_sources
if source["id"] == settings_payload["default_source_id"]
),
None,
)
fallback_source = next(
(source for source in available_sources if source["is_fallback"]),
None,
)
resolved_source = default_source or fallback_source or (available_sources[0] if available_sources else None)
latest_updated_at = max(
(source.get("updated_at") or "" for source in available_sources),
default="",
)
return {
"default_source_id": settings_payload["default_source_id"],
"auto_fallback": settings_payload["auto_fallback"],
"selected_source": resolved_source,
"fallback_source": fallback_source,
"sources": available_sources,
"source_count": len(available_sources),
"latest_updated_at": latest_updated_at or to_iso8601_utc(datetime.now(UTC)),
"generated_at": to_iso8601_utc(datetime.now(UTC)),
}
async def get_public_tv_payload(db: AsyncSession) -> dict[str, Any]:
settings_payload = await get_tv_settings_payload(db)
collected_sources = await get_collected_tv_sources(db)
return build_public_tv_payload(settings_payload, collected_sources)
def _extract_allowed_tv_hosts(sources: list[dict[str, Any]]) -> set[str]:
hosts: set[str] = set()
for source in sources:
for field in ("stream_url", "embed_url", "homepage_url", "youtube_channel"):
value = _clean_url(source.get(field))
if not value:
continue
parsed = urlparse(value)
if parsed.hostname:
hosts.add(parsed.hostname.lower())
return hosts
def is_allowed_tv_proxy_url(url: str, sources: list[dict[str, Any]]) -> bool:
cleaned = _clean_url(url)
if not cleaned:
return False
parsed = urlparse(cleaned)
hostname = (parsed.hostname or "").lower()
if not hostname:
return False
allowed_hosts = _extract_allowed_tv_hosts(sources)
if hostname in allowed_hosts:
return True
return any(hostname.endswith(f".{allowed_host}") for allowed_host in allowed_hosts)

View File

@@ -10,7 +10,12 @@ from app.core.config import settings
from app.core.security import create_access_token
from app.db.session import get_db
from app.models.user import User
from app.schemas.ai import AIProviderStatusResponse, SituationalAnalysisResponse
from app.schemas.ai import (
AIProviderStatusResponse,
PlaygroundSessionResponse,
PlaygroundSessionState,
SituationalAnalysisResponse,
)
@pytest.fixture
@@ -90,6 +95,15 @@ async def test_alerts_without_auth():
assert response.status_code == 401
@pytest.mark.asyncio
async def test_datasource_task_status_without_auth():
"""Test datasource task-status endpoint requires authentication"""
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/datasources/1/task-status")
assert response.status_code == 401
@pytest.mark.asyncio
async def test_alerts_endpoint_with_auth(auth_headers):
"""Test alerts endpoint with authentication"""
@@ -165,7 +179,8 @@ async def test_ai_provider_status_with_auth(auth_headers):
class _FakeAIProviderClient:
async def get_status(self, request_id=None):
return AIProviderStatusResponse(
provider="openai_compatible",
provider="minimax",
api="anthropic-messages",
enabled=True,
configured=True,
model="test-model",
@@ -193,6 +208,7 @@ async def test_ai_provider_status_with_auth(auth_headers):
assert response.status_code == 200
data = response.json()
assert "provider" in data
assert "api" in data
assert "configured" in data
finally:
app.dependency_overrides.clear()
@@ -207,6 +223,9 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
provider="openai_compatible",
model="test-model",
content="1) 态势摘要: 测试返回",
content_blocks=[],
text_blocks=["1) 态势摘要: 测试返回"],
thinking_blocks=[],
raw_response={"id": "mock-response"},
)
@@ -241,5 +260,297 @@ async def test_ai_situational_analysis_returns_503_when_disabled(auth_headers):
data = response.json()
assert data["provider"] == "openai_compatible"
assert data["content"]
assert "content_blocks" in data
assert "text_blocks" in data
assert "thinking_blocks" in data
@pytest.mark.asyncio
async def test_get_playground_session_with_auth(auth_headers):
"""Test playground session restore endpoint."""
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
async def override_get_db():
yield AsyncMock()
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
get_db: override_get_db,
}
transport = ASGITransport(app=app)
try:
with patch(
"app.api.v1.ai.get_playground_session",
new=AsyncMock(
return_value=PlaygroundSessionResponse(
id="1",
session_key="default",
title="Playground 会话",
state=PlaygroundSessionState(
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
title="测试标题",
objective="测试目标",
),
created_at="2026-04-10T00:00:00+00:00",
updated_at="2026-04-10T00:00:00+00:00",
)
),
):
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.get("/api/v1/ai/playground/session", headers=auth_headers)
assert response.status_code == 200
data = response.json()
assert data["session_key"] == "default"
assert data["state"]["messages"][0]["content"] == "hello"
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_save_playground_session_with_auth(auth_headers):
"""Test playground session save endpoint."""
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
async def override_get_db():
yield AsyncMock()
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
get_db: override_get_db,
}
transport = ASGITransport(app=app)
try:
with patch(
"app.api.v1.ai.upsert_playground_session",
new=AsyncMock(
return_value=PlaygroundSessionResponse(
id="1",
session_key="default",
title="测试标题",
state=PlaygroundSessionState(
messages=[{"id": "msg-1", "role": "user", "content": "hello"}],
title="测试标题",
objective="测试目标",
),
created_at="2026-04-10T00:00:00+00:00",
updated_at="2026-04-10T00:00:00+00:00",
)
),
):
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.put(
"/api/v1/ai/playground/session",
headers=auth_headers,
json={
"session_key": "default",
"title": "测试标题",
"state": {
"messages": [{"id": "msg-1", "role": "user", "content": "hello"}],
"selectedPresetKey": "bgp-brief",
"title": "测试标题",
"objective": "测试目标",
"constraints": "",
"inputValue": "",
"analysis": None,
"latestAnalysisMessageId": None,
"analysisMeta": {},
"helpExpanded": True,
},
},
)
assert response.status_code == 200
data = response.json()
assert data["title"] == "测试标题"
assert data["state"]["objective"] == "测试目标"
finally:
app.dependency_overrides.clear()
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_ai_bgp_brief_endpoint_persists_fact_snapshot(auth_headers):
class _FakeAIProviderClient:
async def analyze(self, _payload, request_id=None):
return SituationalAnalysisResponse(
provider="minimax",
model="MiniMax-M2.5",
content="# BGP AI 简报\n\n事实摘要:测试",
content_blocks=[],
text_blocks=["# BGP AI 简报\n\n事实摘要:测试"],
thinking_blocks=[],
raw_response={"id": "mock-bgp-brief"},
)
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
async def override_get_db():
yield AsyncMock()
async def _fake_build_bgp_brief_request(_db, **_kwargs):
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
title="BGP 态势 AI 简报",
objective="生成值班简报",
observations=["事实A", "事实B"],
constraints=["不要编造"],
context={"incident_total": 2, "active_collectors": 3},
)
return request_payload, ["事实A", "事实B"], {"incident_total": 2, "active_collectors": 3}
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
get_db: override_get_db,
}
transport = ASGITransport(app=app)
try:
with patch("app.api.v1.ai.build_bgp_brief_request", side_effect=_fake_build_bgp_brief_request):
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post("/api/v1/ai/bgp/brief", headers=auth_headers, json={})
assert response.status_code == 200
data = response.json()
assert data["facts"] == ["事实A", "事实B"]
assert data["context"]["incident_total"] == 2
assert data["content_markdown"]
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_ai_alert_brief_endpoint_with_auth(auth_headers):
class _FakeAIProviderClient:
async def analyze(self, _payload, request_id=None):
return SituationalAnalysisResponse(
provider="minimax",
model="MiniMax-M2.7",
content="事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。",
content_blocks=[],
text_blocks=["事实摘要:告警测试。风险研判:告警测试。建议动作:告警测试。"],
thinking_blocks=[],
raw_response={"id": "mock-alert-brief"},
)
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
async def override_get_db():
yield AsyncMock()
async def _fake_build_alert_brief_request(_db, **_kwargs):
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
title="告警态势 AI 简报",
objective="输出告警简报",
observations=["告警事实A", "告警事实B"],
constraints=["不要编造"],
context={"active_alerts": 3, "top_datasources": {"bgp": 2}},
)
return request_payload, ["告警事实A", "告警事实B"], {"active_alerts": 3, "top_datasources": {"bgp": 2}}
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
get_db: override_get_db,
}
transport = ASGITransport(app=app)
try:
with patch("app.api.v1.ai.build_alert_brief_request", side_effect=_fake_build_alert_brief_request):
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post("/api/v1/ai/alerts/brief", headers=auth_headers, json={})
assert response.status_code == 200
data = response.json()
assert data["title"] == "告警态势 AI 简报"
assert data["facts"] == ["告警事实A", "告警事实B"]
assert data["context"]["active_alerts"] == 3
assert data["content"]
finally:
app.dependency_overrides.clear()
@pytest.mark.asyncio
async def test_ai_situational_alert_brief_endpoint_with_auth(auth_headers):
class _FakeAIProviderClient:
async def analyze(self, _payload, request_id=None):
return SituationalAnalysisResponse(
provider="minimax",
model="MiniMax-M2.7",
content="事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。",
content_blocks=[],
text_blocks=["事实摘要:态势测试。风险研判:态势测试。建议动作:态势测试。"],
thinking_blocks=[],
raw_response={"id": "mock-situational-brief"},
)
def override_get_current_user():
return User(
id=1,
username="testuser",
email="test@example.com",
password_hash="hashed",
role="admin",
is_active=True,
)
async def override_get_db():
yield AsyncMock()
async def _fake_build_situational_alert_brief_request(_db):
request_payload = __import__("app.schemas.ai", fromlist=["SituationalAnalysisRequest"]).SituationalAnalysisRequest(
title="态势告警 AI 简报",
objective="输出态势告警简报",
observations=["态势事实A", "态势事实B"],
constraints=["不要编造"],
context={"active_system_alerts": 2, "active_bgp_incidents": 1},
)
return request_payload, ["态势事实A", "态势事实B"], {"active_system_alerts": 2, "active_bgp_incidents": 1}
app.dependency_overrides = {
__import__("app.core.security", fromlist=["get_current_user"]).get_current_user: override_get_current_user,
__import__("app.services.ai_client", fromlist=["get_ai_provider_client"]).get_ai_provider_client: lambda: _FakeAIProviderClient(),
get_db: override_get_db,
}
transport = ASGITransport(app=app)
try:
with patch("app.api.v1.ai.build_situational_alert_brief_request", side_effect=_fake_build_situational_alert_brief_request):
async with AsyncClient(transport=transport, base_url="http://test") as client:
response = await client.post("/api/v1/ai/situational-alerts/brief", headers=auth_headers, json={})
assert response.status_code == 200
data = response.json()
assert data["title"] == "态势告警 AI 简报"
assert data["facts"] == ["态势事实A", "态势事实B"]
assert data["context"]["active_system_alerts"] == 2
assert data["content"]
finally:
app.dependency_overrides.clear()

View File

@@ -0,0 +1,150 @@
import pytest
from app.core.websocket.manager import ConnectionManager
from app.core.websocket.ue_scene import (
UeSceneStateStore,
_item_revision,
build_incremental_changes,
)
class _FakeSession:
pass
class _FakeWebSocket:
def __init__(self):
self.accepted = False
self.messages = []
async def accept(self):
self.accepted = True
async def send_json(self, message):
self.messages.append(message)
async def close(self):
return None
def _scene_item(item_id: str, title: str, *, lat: float = 0.0, lng: float = 0.0):
item = {
"id": item_id,
"entity_type": "gpu_cluster",
"geo": {"lat": lat, "lng": lng, "alt": 0.0},
"visual": {"style": "pulse_marker", "size": 1.0, "color": "#FF8C42"},
"metrics": {"name": title},
"labels": {"title": title, "subtitle": ""},
"status": {"health": "normal", "alert_level": "none"},
}
item["revision"] = _item_revision(item)
return item
def _scene_state(state_hash: int, items_by_layer: dict[str, dict[str, dict]]):
base_layers = {
"satellites": {"revision": 0, "items": {}},
"supercomputers": {"revision": 0, "items": {}},
"gpu_clusters": {"revision": 0, "items": {}},
"submarine_cables": {"revision": 0, "items": {}},
"landing_points": {"revision": 0, "items": {}},
"alerts": {"revision": 0, "items": {}},
}
for layer_name, items in items_by_layer.items():
base_layers[layer_name]["items"] = items
base_layers[layer_name]["revision"] = len(items)
return {
"generated_at": "2026-04-17T00:00:00Z",
"state_hash": state_hash,
"total_records": sum(len(items) for items in items_by_layer.values()),
"layers": base_layers,
}
def test_build_incremental_changes_detects_add_update_remove():
previous_state = _scene_state(
1,
{
"gpu_clusters": {
"gpu:a": _scene_item("gpu:a", "A"),
"gpu:b": _scene_item("gpu:b", "B"),
}
},
)
current_state = _scene_state(
2,
{
"gpu_clusters": {
"gpu:b": _scene_item("gpu:b", "B Updated"),
"gpu:c": _scene_item("gpu:c", "C"),
}
},
)
changes = build_incremental_changes(previous_state, current_state)
assert changes["gpu_clusters"]["added"][0]["id"] == "gpu:c"
assert changes["gpu_clusters"]["updated"][0]["id"] == "gpu:b"
assert changes["gpu_clusters"]["removed"] == ["gpu:a"]
@pytest.mark.asyncio
async def test_state_store_replays_incremental_history(monkeypatch):
state_list = [
_scene_state(1, {"gpu_clusters": {"gpu:a": _scene_item("gpu:a", "A")}}),
_scene_state(
2,
{"gpu_clusters": {"gpu:a": _scene_item("gpu:a", "A"), "gpu:b": _scene_item("gpu:b", "B")}},
),
_scene_state(
3,
{"gpu_clusters": {"gpu:a": _scene_item("gpu:a", "A Updated"), "gpu:b": _scene_item("gpu:b", "B")}},
),
]
state_index = {"value": 0}
async def _fake_build_visualization_scene_state(_db):
current_index = state_index["value"]
if current_index < len(state_list) - 1:
state_index["value"] += 1
return state_list[current_index]
monkeypatch.setattr(
"app.core.websocket.ue_scene.build_visualization_scene_state",
_fake_build_visualization_scene_state,
)
store = UeSceneStateStore(history_limit=5)
session = _FakeSession()
first_payloads = await store.get_broadcast_payloads(session)
second_payloads = await store.get_broadcast_payloads(session)
third_payloads = await store.get_broadcast_payloads(session)
replay_payloads = await store.get_sync_payloads(
session,
last_sequence=1,
reason="sequence_gap",
)
assert first_payloads[0]["update_type"] == "full"
assert second_payloads[0]["update_type"] == "incremental"
assert third_payloads[0]["sequence"] == 3
assert [payload["sequence"] for payload in replay_payloads] == [2, 3]
@pytest.mark.asyncio
async def test_connection_manager_broadcasts_to_subscribed_channel_only():
manager = ConnectionManager()
ws_dashboard = _FakeWebSocket()
ws_scene = _FakeWebSocket()
await manager.connect(ws_dashboard, "user-dashboard")
await manager.connect(ws_scene, "user-scene")
manager.subscribe(ws_dashboard, ["dashboard"])
manager.subscribe(ws_scene, ["ue_scene"])
await manager.broadcast({"type": "data_frame", "channel": "ue_scene"}, channel="ue_scene")
assert ws_dashboard.messages == []
assert ws_scene.messages == [{"type": "data_frame", "channel": "ue_scene"}]

View File

@@ -5,6 +5,8 @@ services:
build:
context: .
dockerfile: aiprovider/Dockerfile
env_file:
- ./aiprovider/.env
container_name: planet_aiprovider
ports:
- "8010:8010"

View File

@@ -5,8 +5,741 @@ All notable changes to `planet` are documented here.
This project follows the repository versioning rule:
- `feature` -> `+0.1.0`
- `improvement` -> `+0.0.1`bugfix + 小功能混合)
- `bugfix` -> `+0.0.1`
## [0.33.0] — 2026-04-22
### ✨ Highlights
- `news_live_streams` 采集器默认接入 `iptv-org` 频道目录,并将采集结果稳定并入 Earth TV 直播源列表
- 数据源页支持直接编辑内置数据源 override并为内置源提供一键恢复默认配置入口
### 🔧 Improvements
- `News Live Streams` 现在作为可直接触发的内置默认数据源提供,无需先手工补 override 才能采集
- TV 播放源菜单会直接区分 `[内置]``[采集]` 来源,频道来源信息也会同步展示
- 新增 [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md),正式规划 Earth 态势新闻源配置化与后续采集器化路线
### 🐛 Fixes
- 修复 `news_live_streams` 采集完成后 `/api/v1/tv/streams` 因读取不存在的 `updated_at` 字段而导致默认频道全部消失的问题
- 修复内置数据源操作列按钮显示不全,以及编辑抽屉中多个 `Collapse` 紧贴的问题
---
## [0.32.0] — 2026-04-22
### ✨ Highlights
- Earth 设置新增“地球默认大小”持久化项,重置视角、缩放百分比重置和 BGP 巡航视图现在统一复用这一份默认 zoom
- 卫星焦点层次继续收口:巡航进入 presentation 前不再过早 dim非焦点卫星改成“降亮度/尾迹/背板”而不是去饱和度
### 🔧 Improvements
- Earth 设置面板区块和左右留白进一步收紧,整体更贴近 HUD 面板的密度
- toolbar 展开边界缓存改为按需刷新,减少 document 级 mousemove 期间的重复布局读取
- Scrollbar 和 ScrollbarOverlay 收窄 observer 范围,减少大表格和动态菜单下的额外刷新成本
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md),补充默认视图大小已进入 Earth 设置持久化真源
### 🐛 Fixes
- 修复开启巡航后,尚未进入连线/presentation 时卫星已经整体变暗的问题
- 修复默认大小重置链路分散在多个入口、实际 reset/cruise/缩放提示不一致的问题
- 修复开启地形后卫星反馈层与地球背面可见性之间的一组表现问题,保留正面反馈同时恢复背面轨道遮挡
---
## [0.31.3] — 2026-04-22
### ✨ Highlights
- Earth 图层注册表和启动任务框架继续收口,启动顺序、启动模式、启动提示和任务注册现在都能从统一入口扩展
- 修复 Earth 普通旋转模式与巡航模式切换时的一组交互回归,同时让卫星/地形/昼夜模式的表现更稳定
### 🔧 Improvements
- 新增 [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js) 启动任务注册表,支持 `registerLayerStartupTask(id, taskFactory)`,并拆成海缆 / 卫星 / BGP 独立注册函数
- Earth 图层控制改成注册表驱动,统一承载 `startupPriority``startupMode``startupLabel``startupMessage` 与图层持久化元信息
- Earth 设置支持持久化图层开关、旋转模式、HUD 面板显示状态、地形透明度与日夜模式,并提供一键重置
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 记录图层注册表、启动任务、设置持久化与巡航适配边界
### 🐛 Fixes
- 修复普通旋转模式下点击海缆 / 卫星 / BGP 后卡片和选中表现会被异常清空的问题
- 修复巡航模式切回旋转再切回巡航后无法继续自动巡航的问题
- 修复开启地形后卫星选中反馈层被高海拔区域吞掉的问题,并恢复轨道只在地球前半侧可见
- 修复关闭日夜模式后地球照明仍沿真实昼夜切换、亮部过曝和偏色的问题,改成更中性的 inspection lighting
- 修复 toolbar 收起态仍挡住地球交互,以及首帧短暂展开闪现的问题
---
## [0.31.0] — 2026-04-21
## [0.31.2] — 2026-04-21
### ✨ Highlights
- Earth 巡航模式重构为“通用巡航队列 + 通用连线动画 + BGP 业务适配”三层结构,后续扩到海缆、卫星或新闻巡航时不必再复制一套 `main.js` 状态机
- 修复巡航重构后的交互回归:空白点击重新稳定切到下一项,连线按“起点 → 引导线 → 终点”顺序入场
### 🔧 Improvements
- 新增 [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js) 统一管理队列推进、停留时长、打断与恢复
- 新增 [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js) 统一管理 SVG 连线、折线路径与描边动画
- 新增 [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 收口 BGP 巡航目标排序、卡片落点、轮询去重与连线适配
- 更新 [earth-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-frontend-context.md) 说明新的巡航分层与复用边界
### 🐛 Fixes
- 修复巡航模式下点击空白处无法稳定跳转到下一项、切回旋转再切回巡航后直接卡住的问题
- 修复巡航连线被实时重定位覆盖导致“直接出现”而非绘制动画的问题
- 修复连线动画节点入场节奏不对的问题,改为先出现起点,再绘制连线,最后出现终点
---
## [0.31.1] — 2026-04-21
### ✨ Highlights
- Earth 图层开关状态统一成可复用的 `active / loading` 状态机,首次启用地形和卫星时不再像按钮失效
- 文档目录重构为 `docs/technical``docs/plans``docs/deprecated`,并吸收 `.sisyphus/plans` 中有价值的 Earth / 卫星 / UE5 草案
### 🔧 Improvements
- 新增 [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js),统一按钮 tooltip、`aria-busy`、禁用态和状态文本同步
- 地形图层支持 hover/focus 预热与空闲预热,首次点击等待前移,加载中状态持续可见
- 卫星图层启用前会立即切换为 `loading` 中间态,请求完成后再切回正常开关表现
### 🐛 Fixes
- 修复地形首次加载时通知过早消失、开关仍像关闭状态导致用户误判按钮损坏的问题
- 修复卫星接口较慢时按钮没有任何中间态反馈的问题
---
### ✨ Features
- Earth 新增"巡航展示"模式:自动轮播 BGP 异常事件逐帧追踪连接线位置支持外部交互立即中断序列cancel notifier 模式)
- 巡航目标事件点高亮显示hover 外观 + 锁定脉冲动画,并与点击行为统一展示周边受影响卫星与海缆
- BGP 事件图标新增填充 W 形波动符号flap 类型),替换原有难以辨认的贝塞尔细线
- 巡航/点击激活时其余卫星自动降饱和度 + 增加透明度以突出焦点;海缆未受影响时同步变暗
### 🔧 Improvements
- 修复巡航轮播期间 BGP 事件 polling 刷新导致标记闪烁消失的问题clearBGPData 延迟到请求完成后执行)
- 点击与巡航锁定颜色统一为 hover 色0.92, 0.98, 1.0 全透明),移除锁定态脉冲动画
- 巡航连接折线转折点从尖角调整为钝角linkElbowDropPx提升连线可读性
---
## [0.29.1] — 2026-04-20
## [0.30.0] — 2026-04-21
### ✨ Features
- Earth 新增真实地形图层:后端代理 Terrarium DEM 瓦片(`/api/v1/visualization/terrain/terrarium/{z}/{x}/{y}.png`),前端新增 `terrain.js` 负责瓦片拉取、顶点位移与按海拔着色
- 设置弹窗新增"地形"分组,支持通过滑块实时调整地形图层透明度
### 🔧 Improvements
- 地形按钮改为异步加载,首次点击显示进度提示并在失败时自动回退
- 启动阶段改用 `applyImmediateView` 直接应用初始视角,`showStatusMessage` / `queueStatusMessage` 区分即时与队列态状态消息,加载中不再被临时状态打断
- 控制面板抽取 `applyTerrainUiState` / `getViewRotation` 收敛地形切换与视角旋转的重复 UI 同步逻辑
---
## [0.29.2] — 2026-04-21
### ✨ Highlights
- Earth 继续收口 HUD 交互与设置面板表现,设置弹窗改成更接近从按钮展开的窗口感,同时加入系统级 admin 入口
- 修正天球太阳方向与地球受光解耦后的日照逻辑,地表昼夜判断改为按太阳直射点经纬度落到地球贴图坐标
### 🔧 Improvements
- toolbar 进一步收成更贴近 hub 的浅弓形排列,并统一成与 HUD panel 一致的液态玻璃配色与透明度
- 设置弹窗与各 HUD panel 继续统一样式、等比缩放和头部基线,设置列表补充系统分组与 admin 跳转
- 所有 HUD panel 增加更统一的液态玻璃高光与 hover / press 反馈
### 🐛 Fixes
- 修复设置弹窗仍像旧圆角矩形、标题文案重复和从底边直直飞出的动画问题
- 修复天球与太阳方向混用显示校准导致中国白天仍落在夜面的日照错误
---
## [0.29.1] — 2026-04-20
### ✨ Highlights
- Earth 加载状态条改成单一队列式通知面板,加载阶段不再因为步骤文案变化而回缩,也不会被其他通知打断
- 调整 brand panel 的呈现方式与昼夜/选中态可读性,让品牌区更自然、交互高亮在白天和黑夜里都更稳定
### 🔧 Improvements
- 移除旧的地球加载浮层结构,统一由 HUD 状态消息承载三点脉冲加载过程
- brand panel 改为无边框品牌层,仅保留轻微氛围光,不再因为非常规尺寸显得像第五块功能面板
- 温和收敛地球昼夜材质与主背光强度,保留昼夜辨识度的同时提升白天地表纹理和夜面交互可见性
### 🐛 Fixes
- 修复加载地球时通知条在步骤切换中反复缩短、其他状态消息抢占加载流程的问题
- 修复海缆、登陆点和 BGP 选中高亮在黑夜中过暗、在高光中过亮导致难以辨识的问题
---
## [0.29.0] — 2026-04-20
### ✨ Highlights
- Earth 新增天球层第一版:引入真实全天星图、亮星层与太阳/月亮位置计算,地球场景首次具备可校准的天文背景
- 地球昼夜分隔升级为更明显的日夜增强效果,夜面、晨昏带和太阳方向联动更容易直接读出来
### 🔧 Improvements
- 新增 `celestial.js` 模块和 `assets/celestial/` 资源目录,统一管理星图、亮星数据以及太阳/月亮与光照同步
- 卫星图例改为按倾角分组,严格固定为“赤道轨道 → 低倾角轨道 → 中倾角轨道 → 高倾角轨道 → 逆行轨道”顺序,并全部中文化
- 图层面板补齐关闭按钮,拖拽脱离左列后不再被流布局 margin 影响,能够真正贴到品牌面板下沿
### 🐛 Fixes
- 修复天球球壳放大后被相机 far plane 裁剪导致的外层黑环问题
- 修复图层面板在左侧上移时始终与 brand panel 保持额外间距的问题
---
## [0.28.2] — 2026-04-20
### ✨ Highlights
- 修正媒体情报面板在 `电视直播 / 态势聚合` tab 间切换时的尺寸记忆逻辑,切回原 tab 后可恢复各自大小状态
- 清理 `docs/` 根目录遗留的旧路径文档,只保留新的分组目录和归档目录,结束同一文档双路径并存状态
### 🔧 Improvements
- `media-panel` 切换逻辑改成按 tab 分别记忆尺寸状态,避免 `A -> B -> A` 时继续共用同一套外层尺寸
- 目录整理真正完成收尾:旧的 `docs/*.md` 平铺计划文档删除,继续以 `docs/agents / earth / backend / frontend / ops / ue5 / deprecated` 为唯一入口
### 🐛 Fixes
- 修复拉伸 `media-panel` 后切换 tab 时,`news-panel` 高度回退到旧默认值的问题
- 修复拉伸后切换 tab 导致面板视觉锚点异常的问题,切换时改为围绕当前卡片自身右下角进行尺寸恢复
---
## [0.28.1] — 2026-04-20
### ✨ Highlights
- 收口 Earth 媒体情报面板命名,明确外层 `media-panel` 与内部 `tv-panel / news-panel` 的职责边界
- 整理 `docs/` 目录分组,并将已完成或已废弃的计划文档归档到 `docs/deprecated`
### 🔧 Improvements
- 底部 tab 语义统一为 `media-panel-tabs / media-panel-tab`,并将文案更新为“电视直播 / 态势聚合”
- 补充媒体面板、聚合新闻模块的注释说明,减少 `tv-panel` 同时指代外层壳和内层直播 pane 的阅读歧义
- 更新 README、AI Provider README 与历史文档互链,适配新的 `docs/agents / docs/earth / docs/frontend / docs/backend / docs/ops / docs/ue5` 分组结构
### 🐛 Fixes
- 修复媒体情报面板标题组在头部撑出多余空白的问题,去掉 `hud-panel__title-group` 的无效弹性占位
- 修复聚合 tab 头部仍保留冗余固定标题的问题,现在仅显示区域标签
---
## [0.28.0] — 2026-04-20
### ✨ Features
- 将 Earth 的“新闻直播”和“全球态势聚合”合并为统一的“媒体情报”面板,支持底部 tab 切换与共享标题栏操作区
- 聚合新闻不再单独占据一个 HUD 面板,而是作为媒体情报面板内的第二视图与直播协同呈现
说明:
- 当前结构中,外层 HUD 壳为 `media-panel`,内部 tab 内容区分别为 `tv-panel``news-panel`
### 🔧 Improvements
- TV / News 面板切换加入底边锚定的 reform 动画,并继续保留拖拽、缩放和共享 HUD 行为收口
- 聚合新闻视图新增默认高度约束与内部滚动填充逻辑,避免初始高度过度膨胀
- `tv.js``news.js` 进一步清理共享 HUDPanel 迁移后的残留逻辑,收紧 tab / resize / reform 相关局部 helper
### 🐛 Fixes
- 修复媒体情报面板右下角缩放时高度异常抬升、标题栏被顶出视口的问题
- 修复直播/聚合 tab 切换时按钮高亮、内容切换和底边基准表现不一致的问题
---
## [0.27.7] — 2026-04-16
## [0.27.8] — 2026-04-20
### 🔧 Improvements
- Earth HUD 共享 `HUDPanel` 默认展开/收缩逻辑继续收口,图例与图层面板统一使用同一套边缘阈值与箭头状态机
- 保持新闻直播面板现有特例折叠行为不变,避免播放器区域被默认折叠逻辑影响
### 🐛 Fixes
- 修复图例与图层面板展开/收缩箭头方向和实际动作不一致的问题
- 修复拖动到屏幕底边附近时初始箭头、拖动中箭头和点击后动作不同步的问题
---
## [0.27.7] — 2026-04-16
### 🔧 Improvements
- 用户管理、数据源配置、电视直播源表格统一接入可折叠操作列,窄宽度下自动收起到下拉菜单,减少操作区挤压
- 电视直播设置改为表格总览 + 弹窗编辑模式,主表内容更紧凑,适合控制台一屏浏览
- Earth TV 面板新增失败源探测与自动回退恢复标记,便于值班时快速识别异常直播源
### 🐛 Fixes
- 修复 Settings 电视直播源新增后取消编辑会残留未保存草稿的问题
- 修复 Settings 删除直播源只改本地状态、刷新后恢复的问题,删除现在会立即持久化
- 修复 Users / DataSources / Settings 表格“备注/状态”和“操作”之间的空白占位列问题
- 修复 `useCollapsedActions` 未释放 `ResizeObserver` 导致的潜在内存泄漏与重复回调问题
---
## [0.27.6] — 2026-04-15
### 🔧 Improvements
- BGP 告警页表格纵向 overflow 修复:补全 flex 布局链tabs content-holder 正确撑满剩余高度
- 用户管理表格横向滚动修复:采用 flex-fill 方案替换 `height: auto !important`,自定义滚动条 X 轨道位置对齐表格底部
- Playground 宽布局隐藏"服务状态"按钮:侧边栏可见时不显示冗余入口
- AI Chatbox 输入框失焦收起为单行,聚焦或有内容时展开完整 composer
---
## [0.27.4] — 2026-04-14
### 🔧 Improvements
- info-card 改为懒加载动态挂载:页面初始 DOM 不再含隐藏的 `#info-panel` 节点,仅首次点击交互元素时创建
---
## [0.27.5] — 2026-04-14
### 🔧 Improvements
- 统一控制台多页面滚动体验BGP、alerts、采集数据、用户管理、任务、设置、Playground 等区域接入自定义滚动条与表格滚动容器
- 优化 BGP 与 alerts 页响应式布局:顶部概览卡在窄宽度下优先重排,必要时才启用横向滚动,避免卡片裁切和全局滚动条接管
- 调整 `situational alerts` 布局策略:统计卡按宽度在单行、两列和横滚之间切换,下方详情卡保持单行高度优先
- 实时采集进度优化:一键采集完成后在未刷新页面时保留 100% 完成态,不再错误归零
- 补充 UE5 MVP 融合方案文档,完善后续集成规划沉淀
### 🐛 Fixes
- 修复 BGP summary 与 alerts 顶部卡片在无真实溢出时误出现横向滚动的问题
- 修复 alerts 页面缩窄后外层全局竖向滚动被接管的问题,恢复“一屏内、内部滚动”的布局逻辑
- 修复 `situational alerts` 在两排布局下详情卡竖向溢出的问题,改为更稳定的分区响应式排版
- 修复自定义滚动条交互反馈,悬停、聚焦、拖拽时颜色加深但不再显示多余外圈
---
## [0.27.3] — 2026-04-14
### 🔧 Improvements
- TV panel meta 折叠展开方向稳定:底部锚定时向上生长,拖拽后(顶部锚定)通过 JS 补偿 top 保持播放器底部位置不变
- 修复 TV panel 展开/折叠时视频区域跳动问题:移除面板 min-height使播放器高度在两种状态下保持一致
- 修正 TV panel meta toggle 箭头方向:展开朝下,折叠朝上
- 修复图例面板折叠按钮失效legend-bar-btn 补充进拖拽排除列表)
- 调整图层搜索框图标尺寸为 20pxBR 缩放角标改为直角 L 形
---
## [0.27.2] — 2026-04-14
### 🔧 Improvements
- 修复 brand copy 宽度不随内容收缩的问题,现在与 title 图片宽度保持一致
- 提取 `--brand-copy-width` CSS 自定义属性,消除 160px / 172px 魔法数字重复
---
## [0.27.1] — 2026-04-14
### 🔧 Improvements
- 面板拖拽新增 L 形边界约束,其他面板无法覆盖 brand 面板区域,并从右侧/底部自然卡边
- brand 组件引入 `--brand-scale` 整体缩放变量padding 与内容尺寸独立控制
- 图层控制面板宽度收窄260px与 brand 面板错落排列,间距调大
### 🐛 Fixes
- 修复搜索框 `type="search"` 导致清除按钮重复显示的问题
- 修复 `[hidden]` 属性被组件 `display` 规则覆盖的问题
---
## 0.27.0
Released: 2026-04-14
### Highlights
- 全面重构 Earth HUD 布局:品牌面板、图层控制面板、信息详情卡片各自独立,支持拖拽与折叠,信息卡片改为跟随点击位置悬浮显示。
- 新增图层控制面板Layer Panel海缆、卫星、地形、BGP 等图层集中管理,支持关键字搜索过滤。
- Earth 大气层渲染升级,引入 Fresnel 着色器双层辉光效果和深度遮挡球体。
### ✨ Features
- 新增独立 Layer Panel`js/controls.js`, `css/layer-panel.css`),图层开关、搜索过滤、折叠收起,取代原工具栏弹出菜单
- 信息详情面板info-panel改为点击时定位到鼠标附近`js/info-card.js`),悬停改为轻量 tooltip降低视觉干扰
- Earth 材质重构(`js/earth.js`, `js/constants.js`):新增 `EARTH_MATERIAL_CONFIG`Fresnel 内外大气层辉光、深度遮挡球体,纹理加载独立为 `loadEarthTexture()`
### 🔧 Improvements
- Earth Stats 面板改为 2 列 KPI 网格布局,支持拖拽和关闭(`css/earth-stats.css`
- 数据加载改为分步串行(登陆点 → 海缆 → 卫星 → BGP → 纹理),每步之间 yield 帧,改善视觉渐现体验(`js/main.js`
- 图层按钮状态更新逻辑统一至 `updateLayerButtonState()`,消除重复实现
- 海缆状态识别新增 `active` 枚举值(兼容旧 `In Service`
---
## 0.26.1
Released: 2026-04-12
### Highlights
- Cleaned up the first TV follow-up and replaced the dashboard sidebar's brittle one-off scroll behavior with a reusable `Scrollbar` component, so the new live module code is easier to maintain and the console navigation can scroll without layout jitter.
### Improved
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by extracting the sidebar scrollbar into a dedicated component with explicit `x / y / both` axis support, hidden native scrollbars, compact account/version rows, and a sidebar-only vertical setup instead of the earlier patchwork CSS glued directly onto the layout.
### Fixed
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) by refactoring repeated iframe/video reset paths into shared helpers, so TV source switching, empty-state fallback, and playback retry handling no longer duplicate cleanup logic across multiple branches.
## 0.26.2
Released: 2026-04-12
### Highlights
- Stabilized the new reusable scrollbar work by restoring reliable sidebar visibility and extending the same floating scrollbar language to the Data Sources tables without letting scrollbars squeeze layout width or regress the console navigation.
### Added
- Added [frontend/src/components/Scrollbar/ScrollbarOverlay.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/ScrollbarOverlay.tsx) as an overlay variant that binds to existing scroll containers such as Ant Table bodies, so heavy data grids can adopt the new scrollbar visuals without replacing their built-in scrolling mechanics.
### Improved
- Improved [frontend/src/components/Scrollbar/Scrollbar.tsx](/home/ray/dev/linkong/planet/frontend/src/components/Scrollbar/Scrollbar.tsx), [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by reverting the sidebar to a reliable vertical-first scrollbar path, then reintroducing automatic dual-axis support with independent floating tracks that no longer hide the thumb when only the sidebar needs vertical scrolling.
### Fixed
- Fixed [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the built-in and custom datasource tables now use the new overlay scrollbar instead of native table scrollbars, keeping horizontal and vertical scrolling available without changing Ant Tables internal layout behavior.
## 0.26.0
Released: 2026-04-12
### Highlights
- Added an operator-facing TV live module to Earth, including backend-configurable live sources, a draggable/resizable live-news HUD window, default global news channels, and a dedicated settings workflow so the Earth page can open real news playback instead of only static telemetry.
### Added
- Added [backend/app/api/v1/tv.py](/home/ray/dev/linkong/planet/backend/app/api/v1/tv.py), [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py), and [backend/app/services/collectors/news_live_streams.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/news_live_streams.py) to provide TV source configuration, public stream payloads, a guarded HLS proxy path, and a collector entry point for future world-news live-source ingestion.
- Added the Earth TV HUD workspace through [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js), and [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css), including toolbar access, draggable/closable behavior, resize support, direct video/HLS playback, iframe fallback, and per-channel external-open handling.
- Added [docs/deprecated/earth-tv-live-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-tv-live-module-plan.md) and [docs/earth/technical/news-live-streams-collector-format.md](/home/ray/dev/linkong/planet/docs/technical/earth-news-live-streams-collector-format.md) to document the TV module rollout plan and the expected collector payload format for future curated live-channel ingestion.
### Improved
- Improved [frontend/src/pages/Settings/Settings.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Settings/Settings.tsx), [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py), and [backend/app/core/datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py) by adding TV source administration to system settings and registering the `news_live_streams` datasource as a first-class configurable collector.
- Improved [backend/app/services/tv_streams.py](/home/ray/dev/linkong/planet/backend/app/services/tv_streams.py) by seeding a curated first-pass news channel catalog that now defaults to `CGTN English` YouTube playback while keeping `CCTV-4` as a built-in fallback and exposing additional Reuters, CGTN, DW, Al Jazeera, Arirang, ABP, and SABC entries for operator testing.
### Fixed
- Fixed [frontend/public/earth/js/tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js) and [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) so HLS/video playback now actively attempts autoplay in the TV panel instead of only loading metadata and leaving the player visually idle.
- Fixed [frontend/public/earth/css/tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css) so the TV source selector and action controls better match the Earth HUD dark theme instead of falling back to a bright native dropdown presentation.
## 0.25.3
Released: 2026-04-11
### Highlights
- Refined the Earth HUD visual system into a calmer operator-facing style, turned the top-left Earth brand into a real reusable component with language-driven rendering, and cleaned up duplicated brand assets so the page now has a single source of truth for HUD branding.
### Added
- Added [frontend/public/earth/js/brand.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/brand.js) as a reusable Earth `brand` component that renders the top-left logo/title/subtitle block from a shared config instead of hardcoding the structure in HTML.
- Added [frontend/public/earth/assets/brand/earth-logo.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/earth-logo.svg), [frontend/public/earth/assets/brand/title-zh.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-zh.svg), and [frontend/public/earth/assets/brand/title-en.svg](/home/ray/dev/linkong/planet/frontend/public/earth/assets/brand/title-en.svg) as the canonical Earth brand asset set.
### Improved
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css), [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css), and [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css) by rebalancing the Earth HUD into a more restrained deep-blue control-room look instead of the earlier over-layered glass-and-neon mix.
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html), [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), and [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) by moving the Earth brand mount to a dedicated root and letting `HUD_CONFIG.brandLanguage` choose between `zh` and `en` without embedding the language decision in the DOM.
### Fixed
- Fixed [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css) so the Earth brand area now uses consistent `earth-brand` component selectors and English-specific typography hooks, avoiding the earlier one-off `brand-banner` naming drift and duplicated title styles.
## 0.25.2
Released: 2026-04-10
### Highlights
- Refined the Earth HUD operator polish so settings now behave like a true bounded menu, HUD panels render at the correct scale from the first frame, and dragged panels animate cleanly into and out of maximized layout targets without drifting to screen edges.
### Improved
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by precomputing the initial `--hud-scale` before Earth CSS loads, so HUD panels no longer flash at full size before shrinking to the target scale.
- Improved [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) by cleaning up duplicated settings-modal close-button styles, aligning the settings title with the shared HUD title system, and constraining the settings sheet to a stable centered width instead of viewport-relative modal sizing.
### Fixed
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) so dragged HUD panels now fly from their dragged positions into maximized layout targets, closed panels stay out of the transition, and restoring layout clears drag overrides back to the initial positions.
- Fixed [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) and [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css) so layout transitions no longer visibly stick to screen edges before landing; the FLIP motion now composes with the existing corner transforms instead of fighting them.
## 0.25.1
Released: 2026-04-10
### Highlights
- Cleaned up the first persistent Playground rollout, fixed sidebar submenu persistence to match the intended operator behavior, and added reusable code-hygiene rules to prevent this class of drift from accumulating again.
### Improved
- Improved [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) and [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by extracting repeated lookup, response, and request-action paths into shared helpers, reducing duplicated Playground flow code without changing behavior.
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding a new `Code Hygiene - MANDATORY` section covering single-source-of-truth state, transitional cleanup, repeated-logic extraction, layout debugging order, and post-feature cleanup expectations.
### Fixed
- Fixed [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) so first-level menu expansion now behaves correctly across in-app navigation: `采集与数据` remains the default expanded group after refresh, while manually expanded groups stay open when navigating to their own child routes and reset only on page reload.
## 0.25.0
Released: 2026-04-10
### Highlights
- Turned `AI Playground` into a persistent backend-backed chat workspace, added dedicated alert workspaces as a foundation for future situational analysis, and aligned the operator UI around a more structured AI + alerts workflow instead of one-off playground calls.
### Added
- Added [backend/app/models/playground_session.py](/home/ray/dev/linkong/planet/backend/app/models/playground_session.py), [backend/app/models/playground_message.py](/home/ray/dev/linkong/planet/backend/app/models/playground_message.py), and [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so Playground conversations, execution state, edits, retries, and stop/resume semantics are persisted in the backend database rather than living only in browser state.
- Added [backend/app/services/alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/alert_ai_brief.py), [backend/app/services/situational_alert_ai_brief.py](/home/ray/dev/linkong/planet/backend/app/services/situational_alert_ai_brief.py), and the dedicated alert pages [frontend/src/pages/Alerts/SystemAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SystemAlerts.tsx), [frontend/src/pages/Alerts/BGPAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/BGPAlerts.tsx), and [frontend/src/pages/Alerts/SituationalAlerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/SituationalAlerts.tsx) to establish the alert-analysis foundation for later situational awareness expansion.
### Improved
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by rebuilding Playground into a true chatbox workflow with persistent history, edit-and-resend behavior, grounded message actions, responsive composer behavior, bottom-stick scrolling, and tighter mobile layout handling.
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx), [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx), and [frontend/src/pages/Alerts/Alerts.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Alerts/Alerts.tsx) by reorganizing navigation around `采集与数据`, `专题观测`, and split alert entries so the app can scale to more observability and situational modules without turning the top-level UI into a single overloaded page.
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) and [docs/agents/situational-awareness-foundation-plan.md](/home/ray/dev/linkong/planet/docs/plans/agents-situational-awareness-foundation-plan.md) by documenting the current AI/alerts base, planned situational-awareness direction, and the new persistent Playground foundation.
### Fixed
- Fixed [backend/app/services/playground_chat_service.py](/home/ray/dev/linkong/planet/backend/app/services/playground_chat_service.py) so background Playground runs explicitly commit state transitions, allowing frontend polling to observe real pending/thinking/answering/done states instead of seeing stale empty threads.
- Fixed [frontend/src/services/situational-awareness/index.ts](/home/ray/dev/linkong/planet/frontend/src/services/situational-awareness/index.ts) by removing the temporary mock gateway path, so Playground and alert-related AI flows now reflect the real backend/provider chain instead of local fake responses.
## 0.24.8
Released: 2026-04-10
### Highlights
- Refined the BGP AI brief operator workflow so the tab now stays compact and metadata-focused, while the full Markdown brief opens in a bounded modal that respects the repos single-screen layout rules.
### Improved
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by turning the brief action row into a clearer `历史简报下拉框 + 查看 + 生成` flow, keeping inline metadata visible in the tab while moving full Markdown reading into a dedicated modal workspace.
### Fixed
- Fixed [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so the BGP brief modal no longer spills below the viewport; long brief content now scrolls inside the modal body instead of extending past the visible screen.
## 0.24.7
Released: 2026-04-10
### Highlights
- Formalized repository release hygiene and frontend layout guardrails so repeated versioning chores and recurring layout regressions now have explicit repo-level rules instead of living only in conversation context.
### Added
- Added [release-workflow/SKILL.md](/home/ray/dev/linkong/planet/.codex/skills/release-workflow/SKILL.md), defining the repository release workflow for version bumps, changelog/version-history updates, minimal validation, and commit/push sequencing.
### Improved
- Improved [rules.md](/home/ray/dev/linkong/planet/rules.md) by adding mandatory release-workflow requirements and a new frontend layout constraint section covering single-screen workspaces, overflow ownership, tab-pane behavior, compact-mode expectations, and readable-card fallbacks.
- Improved [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md) by summarizing the recurring Earth, Playground, BGP, and admin-layout regressions into concrete constraints for future frontend work, including “prefer scrollbars over unreadable compression” and “do not treat every tab as a table pane.”
## 0.24.6
Released: 2026-04-10
### Highlights
- Tightened several backend hot paths outside the original BGP page fixes, stabilized BGP collector coverage after the recent query refactors, and rebuilt the BGP AI brief tab so saved Markdown briefs render and scroll like a proper operator workspace instead of collapsing inside the shared table layout.
### Improved
- Improved [backend/app/api/v1/datasources.py](/home/ray/dev/linkong/planet/backend/app/api/v1/datasources.py) by replacing per-datasource task, count, and endpoint lookups with batched prefetch helpers, reducing the worst `1 + N` behavior on the datasource list and `trigger-all` flow.
- Improved [backend/app/api/v1/visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py) by switching the main Earth-facing `CollectedData` endpoints to `is_current` records, batching multi-source loads for aggregate endpoints, and removing stale Python-side dedupe paths from the hot route.
- Improved [backend/app/services/bgp_incidents.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_incidents.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) by avoiding historical full-table infrastructure scans, narrowing observation baseline payloads to required columns, and pushing more ASN filtering into the database.
- Improved [backend/app/api/v1/alerts.py](/home/ray/dev/linkong/planet/backend/app/api/v1/alerts.py), [backend/app/api/v1/dashboard.py](/home/ray/dev/linkong/planet/backend/app/api/v1/dashboard.py), and [backend/app/api/v1/settings.py](/home/ray/dev/linkong/planet/backend/app/api/v1/settings.py) by collapsing several repeated count and settings queries into fewer aggregate or batched reads.
- Improved [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx), [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css), and [frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx](/home/ray/dev/linkong/planet/frontend/src/components/MarkdownRenderer/MarkdownRenderer.tsx) by rebuilding the `AI 简报` tab layout, fixing saved brief scrolling behavior, and extending the renderer to handle tables, separators, and stored metadata comments more gracefully.
- Improved [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md) by explicitly recording that the current BGP brief is only the first-stage summary flow and that regional prefix-geography analysis remains a planned Phase B follow-up.
### Fixed
- Fixed [backend/app/services/bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [backend/app/services/bgp_enrichment.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_enrichment.py) so JSON field extraction no longer depends on the less portable `.astext` path that could break BGP collector endpoints in local environments.
- Fixed the BGP AI brief tab in [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) and [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) so long saved briefs are no longer compressed into a tiny clipped viewport by the shared tab/table overflow rules.
## 0.24.4
Released: 2026-04-09
### Highlights
- Refined the `planet.sh` AI Provider rebuild UX so image rebuilds now feel like first-class scripted tasks, with clearer stage boundaries and cleaner fallback messaging instead of leaking raw Compose output into the terminal.
### Improved
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by keeping AI Provider image build logs in a temporary file, surfacing stage-specific detail copy for `docker compose v2` and `docker-compose v1`, and showing an explicit success line when the image rebuild finishes before container health checks begin.
### Fixed
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the AI Provider image rebuild stage no longer dumps raw Compose build output into the main spinner flow during normal successful runs.
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so the `构建 AI Provider 镜像` phase now ends with an explicit completion signal instead of visually blending into the subsequent container health-check phase.
## 0.24.3
Released: 2026-04-09
### Highlights
- Extended `AI Playground` from a minimal prompt form into a more repeatable diagnostics workspace, and hardened `planet.sh` so AI Provider restarts can rebuild changed images and expose clearer Compose fallback behavior.
### Improved
- Improved [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) by adding preset scenarios, response metadata, `thinking` block visibility, raw JSON inspection, and copy actions so the page works more like a proper AI diagnostics console.
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by styling Playground presets, result metadata, raw response sections, and responsive action groups without breaking the single-screen workspace layout.
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so AI Provider restarts detect code/config changes, rebuild the image when needed, and surface which Compose path is being used during image and container operations.
### Fixed
- Fixed the AI Provider restart path in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so code changes inside `aiprovider/` no longer stay hidden behind an old container image after `restart -a`.
- Fixed Compose error reporting in [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by making the script explicitly show `docker compose v2` first, then `docker-compose v1`, and only fail after both execution paths are exhausted.
## 0.24.2
Released: 2026-04-09
### Highlights
- Fixed the public-entry and AI diagnostics regressions introduced during the recent frontend routing and playground work, while also reducing the main frontend bundle by switching to route-level lazy loading and more targeted vendor chunking.
### Improved
- Improved [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by lazy-loading admin pages and large workspaces through `React.lazy()` plus `Suspense`, so the initial frontend entry no longer pulls every route into the first bundle.
- Improved [frontend/vite.config.ts](/home/ray/dev/linkong/planet/frontend/vite.config.ts) by adding targeted manual chunking for React, icon, network, and Earth-related vendor dependencies instead of leaving everything in one monolithic application bundle.
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding a shared route-loading state and making the Playground help panel size to its content instead of stretching to fill the sidebar.
### Fixed
- Fixed [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) so anonymous visits to `/` once again reach the public Earth entry through the existing `/ -> /earth` redirect instead of being intercepted by the login screen.
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) so cached provider status is shown immediately but still refreshed from the backend, avoiding stale diagnostics after `.env` or provider changes within the same browser tab.
- Fixed [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx) and [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) so the `测试说明` card no longer over-expands and now fits its content more naturally in the sidebar.
## 0.24.1
Released: 2026-04-09
### Highlights
- Refined the `/earth` HUD into a cleaner class-first structure with responsive scaling, clearer CSS layer boundaries, and lower coupling between HTML, CSS, and runtime UI updates.
- Hardened local startup conventions around Bun so frontend tooling, docs, and `planet.sh` behave more predictably in fresh Ubuntu and mixed WSL environments.
### Added
- Added [frontend/public/earth/css/hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css), extracting shared HUD panel surfaces, shared HUD typography rows, status messaging, tooltip overlays, and layout-expanded panel transitions out of the old monolithic base layer.
- Added [frontend/public/earth/css/toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css), isolating Earth toolbar, popover, zoom dock, liquid-glass button, and toolbar-tooltip behavior into a dedicated toolbar layer.
- Added explicit Bun package-manager metadata to [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json) so the frontend package manager choice is declared instead of inferred from the lockfile alone.
### Improved
- Improved [frontend/public/earth/css/base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css) by reducing it to app-shell concerns only: global tokens, Earth app container, loading panel, and shared animation primitives.
- Improved [frontend/public/earth/index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html) by wiring the new CSS layer order, standardizing HUD utility classes on `hud-panel-*`, and replacing generic toolbar/tooltip hooks with more explicit Earth-specific classes.
- Improved [frontend/public/earth/css/info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css), [frontend/public/earth/css/coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css), [frontend/public/earth/css/legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css), and [frontend/public/earth/css/earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css) by leaving only panel-specific responsibilities in each file after the shared HUD and toolbar primitives moved out.
- Improved [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js), [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js), [frontend/public/earth/js/ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js), and [frontend/public/earth/js/legend.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/legend.js) by aligning runtime DOM queries and generated markup with the new class-first HUD and toolbar structure.
- Improved [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js) and [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) by keeping HUD scaling configurable through extracted constants instead of scattering scaling assumptions through the main Earth entrypoint.
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by prepending `~/.local/bin` and `~/.bun/bin` automatically, preferring direct `~/.bun/bin/bun` detection, and auto-installing missing `uv`/`bun` instead of requiring the user's interactive shell config to expose them first.
- Improved [README.md](/home/ray/dev/linkong/planet/README.md), [project_context.md](/home/ray/dev/linkong/planet/project_context.md), [rules.md](/home/ray/dev/linkong/planet/rules.md), and [scripts/bootstrap-dev.sh](/home/ray/dev/linkong/planet/scripts/bootstrap-dev.sh) by making the frontend Bun-only workflow explicit in both onboarding docs and command-line guidance.
### Fixed
- Fixed the Earth HUD cleanup path where splitting styles previously left the page with a missing `base.css` entry and mismatched utility class names during the refactor.
- Fixed several Earth UI update paths so toolbar tooltip text, legend re-rendering, status message classes, and mouse coordinate styling continue to work after removing legacy fallback selectors.
- Fixed the local developer bootstrap path where `planet.sh start` could fail in a clean Ubuntu or agent shell session simply because Bun or uv were installed outside the current shell's inherited `PATH`.
## 0.24.0
Released: 2026-04-09
### Highlights
- Added a dedicated `AI Playground` admin entry so operators can validate provider connectivity and run controlled situational-analysis prompts from the main frontend without introducing a separate UI service.
- Established a first explicit frontend layout rulebook centered on single-screen workspaces, internal module scrolling, and BGP-style page composition for future admin pages.
### Added
- Added [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx), introducing the first dedicated AI testing workspace with provider status visibility, prompt/result tabs, and collapsible operator guidance.
- Added [docs/frontend/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md), documenting the repository standard for one-screen admin workspaces and module-local overflow handling.
- Added [docs/frontend/plans/ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md), capturing the completed AI gateway/UI work and the next delivery phases for BGP briefs, evidence-first inputs, and future agent runtime expansion.
### Improved
- Improved [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx) and [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx) by wiring `AI Playground` into the main admin navigation and route tree instead of pointing operators to a nonexistent `aiprovider` chat page.
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by replacing the incorrect `localhost:8010/chat` closeout link with the frontend `AI Playground` entry.
- Improved [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml) by attaching `./aiprovider/.env` to the `aiprovider` service so provider identity, model, and credentials actually reach the running container.
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by linking the new frontend layout guidance and AI Playground development plan.
- Improved [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by refining the Playground workspace into a notebook-friendly left-sidebar plus right-tabbed layout, reusing thin scrollbars, and making provider/help/result regions degrade more gracefully under constrained height.
### Fixed
- Fixed the local AI status flow so provider configuration no longer appeared permanently `disabled / not configured` merely because `aiprovider/.env` was not mounted into the container.
- Fixed repeated `Provider 状态` refetching when switching away from and back to `/playground` by caching the last known provider status within the browser session until the operator explicitly refreshes it.
- Fixed several Playground layout regressions where auxiliary panels could push the result area out of view or clip provider details without exposing internal scrolling.
## 0.23.4
Released: 2026-04-08
### Highlights
- Fixed the BGP overview workspace so summary cards and tabular data now share viewport space more predictably, keeping the table header visible while preserving internal horizontal and vertical scrolling.
### Improved
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching BGP tables from frontend pagination to in-table scrolling, setting explicit horizontal scroll baselines per tab, and reshaping the summary cards into a desktop two-row layout with a compact single-row horizontal strip on tighter screens.
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by hardening the BGP table layout chain from card body to Ant Design table body, so width overflow stays inside the table region and compact summary cards no longer consume unnecessary vertical space above the workspace.
- Improved release consistency by aligning [VERSION](/home/ray/dev/linkong/planet/VERSION), [pyproject.toml](/home/ray/dev/linkong/planet/pyproject.toml), [frontend/package.json](/home/ray/dev/linkong/planet/frontend/package.json), and [uv.lock](/home/ray/dev/linkong/planet/uv.lock) on the same `0.23.4` bugfix version.
### Fixed
- Fixed the BGP collector/events tables so shrinking the browser window no longer clips the table card frame without exposing a usable horizontal scrollbar.
- Fixed the BGP table region so the Ant Design header row is no longer obscured by an overgrown table body after the page switched to full-height internal scrolling.
- Fixed the BGP summary area so medium and large screens no longer collapse the six KPI cards into overly narrow single-row tiles.
## 0.23.3
Released: 2026-04-08
### Highlights
- Refined `planet.sh` startup and restart presentation so service bring-up, health checks, and frontend dev-server readiness are easier to follow in real time.
### Improved
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by standardizing the script on `zsh`, tightening frontend startup stage rendering, and making spinner-driven subtask feedback show up step by step instead of bunching at the end.
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by clarifying health-check subtask copy, filtering noisy frontend startup lines, and aligning terminal output spacing across spinner and status rows.
### Fixed
- Fixed several `zsh` compatibility issues in [planet.sh](/home/ray/dev/linkong/planet/planet.sh), including reserved variable name collisions during `restart` and container health checks.
- Fixed [planet.sh](/home/ray/dev/linkong/planet/planet.sh) so frontend readiness animation no longer appears frozen while waiting for Vite startup and health probes.
## 0.23.2
Released: 2026-04-08
### Highlights
- Hardened datasource retrigger handling so operators can safely force reruns without losing control of task state visibility or rollback behavior.
### Improved
- Improved [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) by mapping phase-local task percentages into a continuous overall progress bar, pre-checking running task status before retriggering, and consolidating the trigger/force-confirm flow.
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by polishing startup CLI output, reducing duplicated spinner cleanup, and standardizing log/help output around the newer terminal presentation.
### Fixed
- Fixed forced datasource reruns in [backend/app/services/collectors/base.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/base.py) by rolling back invalid SQLAlchemy session state before marking cancelled or failed task cleanup, preventing `PendingRollbackError` during operator-triggered cancellation.
- Fixed datasource trigger UX in [frontend/src/pages/DataSources/DataSources.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/DataSources/DataSources.tsx) so batch and single-source progress no longer jump straight to `100%` on frontend-side status coercion while the backend is still reporting real progress.
## 0.23.1
Released: 2026-04-07
### Highlights
- Fixed the BGP overview page so high-DPI and lower-height screens can keep the observation workspace visible within a single viewport.
### Improved
- Improved [BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx) by switching the page to a height-aware shell, compact summary layout, and tabbed data workspace so observation tables retain the majority of the screen.
- Improved [index.css](/home/ray/dev/linkong/planet/frontend/src/index.css) by adding responsive BGP-specific compact spacing, denser table paddings, and an internal scroll region for collector coverage and event tables.
### Fixed
- Fixed the BGP observation page on small or high-scale displays where stacked stats and full-height blocks consumed too much vertical space, leaving only a couple of visible table rows.
## 0.23.0
Released: 2026-04-07
@@ -23,7 +756,7 @@ Released: 2026-04-07
- Added [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py), introducing an internal HTTP client for `backend -> aiprovider` calls with request-id propagation and lightweight retry.
- Added [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py), [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py), and related config/schema files to stand up the dedicated adapter service.
- Added [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example) and [docker-compose.local-model.yml](/home/ray/dev/linkong/planet/docker-compose.local-model.yml) as ready-to-edit local-model templates.
- Added [docs/aiprovider.md](/home/ray/dev/linkong/planet/docs/aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
- Added [docs/agents/aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md), documenting architecture, configuration, single-machine and multi-machine deployment, and cross-service calling patterns.
- Added a dedicated `重启 AI Provider` control path in [Dashboard.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Dashboard/Dashboard.tsx), [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py), and [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
### Improved
@@ -31,6 +764,8 @@ Released: 2026-04-07
- Improved backend-to-provider tracing by propagating `X-Request-ID` through the AI call chain and returning the same header from both backend and `aiprovider`.
- Improved resilience by adding lightweight retry handling to both `backend -> aiprovider` and `aiprovider -> model provider` HTTP calls.
- Improved operator workflow by folding `aiprovider` startup, health checks, restart support, and log viewing into [planet.sh](/home/ray/dev/linkong/planet/planet.sh).
- Improved [planet.sh](/home/ray/dev/linkong/planet/planet.sh) by adding bounded retry and container-health self-recovery for dependency installs, database startup, `aiprovider` startup, and interactive PostgreSQL boot paths.
- Improved [README.md](/home/ray/dev/linkong/planet/README.md) by documenting the new `planet.sh` retry and health-check tuning environment variables with concrete override examples.
- Improved container consistency by switching [backend/Dockerfile](/home/ray/dev/linkong/planet/backend/Dockerfile) and [aiprovider/Dockerfile](/home/ray/dev/linkong/planet/aiprovider/Dockerfile) to `uv sync` / `uv run`.
### Changed
@@ -147,7 +882,7 @@ Released: 2026-04-02
- Added a new `IPtoASN Prefix Geography` collector in [iptoasn.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/iptoasn.py) and registered it through [data_sources.yaml](/home/ray/dev/linkong/planet/backend/app/core/data_sources.yaml), [data_sources.py](/home/ray/dev/linkong/planet/backend/app/core/data_sources.py), [datasource_defaults.py](/home/ray/dev/linkong/planet/backend/app/core/datasource_defaults.py), and [collectors/__init__.py](/home/ray/dev/linkong/planet/backend/app/services/collectors/__init__.py).
- Added country centroid helpers in [countries.py](/home/ray/dev/linkong/planet/backend/app/core/countries.py) so country-level prefix geography can produce map coordinates instead of only labels.
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/prefix-geography-plan.md).
- Added a dedicated prefix-geography implementation note in [prefix-geography-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-prefix-geography-plan.md).
- Added recent `15m` collector activity dimensions to BGP coverage output in [bgp_collectors.py](/home/ray/dev/linkong/planet/backend/app/services/bgp_collectors.py) and [visualization.py](/home/ray/dev/linkong/planet/backend/app/api/v1/visualization.py).
- Added additional BGP detector coverage for `route_leak_candidate` and `path_flap` flows in [test_bgp.py](/home/ray/dev/linkong/planet/backend/tests/test_bgp.py).
- Added a local Earth cloud texture at [earth_clouds_1024.png](/home/ray/dev/linkong/planet/frontend/public/earth/assets/earth_clouds_1024.png) to avoid remote cloud-map dependency failures.
@@ -162,7 +897,7 @@ Released: 2026-04-02
- Improved Earth event animation semantics in [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js) by separating icon pulse from ring expansion so the center marker can breathe while the ring expands independently.
- Improved Earth texture reliability in [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) by switching clouds back to a local static asset under the restored `public/earth` runtime.
- Improved frontend boot noise in [frontend/index.html](/home/ray/dev/linkong/planet/frontend/index.html) by removing the default Vite favicon request that was generating irrelevant `vite.svg` timeouts during Earth debugging.
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
- Improved project planning docs in [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md) and [TODO.md](/home/ray/dev/linkong/planet/TODO.md) so the roadmap now explicitly prioritizes `activity layer`, `prefix-centric geography`, and follow-up geofeed/whois work.
### Fixed
@@ -320,7 +1055,7 @@ Released: 2026-03-31
- Added restart-task Redis helpers and whitelist command mapping in [system_control.py](/home/ray/dev/linkong/planet/backend/app/services/system_control.py).
- Added detached restart runner orchestration in [system_restart_runner.py](/home/ray/dev/linkong/planet/backend/scripts/system_restart_runner.py).
- Added `-d` / `--database` support to [planet.sh](/home/ray/dev/linkong/planet/planet.sh) for database-only restarts.
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/system-service-control.md).
- Added restart control documentation in [system-service-control.md](/home/ray/dev/linkong/planet/docs/technical/backend-system-service-control.md).
### Improved
@@ -523,7 +1258,7 @@ Released: 2026-03-26
### Added
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/earth-module-plan.md).
- Added a dedicated Earth module remediation plan in [earth-module-plan.md](/home/ray/dev/linkong/planet/docs/deprecated/earth-module-plan.md).
- Added backend TLE helpers in [satellite_tle.py](/home/ray/dev/linkong/planet/backend/app/core/satellite_tle.py).
- Added backend support for returning `tle_line1` and `tle_line2` from the satellite visualization API.

22
docs/deprecated/README.md Normal file
View File

@@ -0,0 +1,22 @@
# Deprecated Docs
这个目录用于存放两类文档:
1. 已经完成、主要保留为历史记录的实施计划
2. 已被现有实现或新方案替代的旧计划
放到这里并不代表这些文档“错误”,而是表示:
- 它们不再适合作为当前开发的主指导文档
- 如果要了解历史决策、演进路径或旧设计背景,仍然可以参考
当前归档原则:
- 明确写明“已完成”的计划,优先归档
- 已被正式实现替代、继续放在 `docs/` 根目录会误导后续开发的计划,归档
- 仍然指导未来开发、尚未完成或仍有明确执行价值的文档,继续保留在 `docs/`
补充说明:
- 一部分归档文档来自外部或临时工作流草案,例如 sisyphus 生成的初稿
- 这类文档如果有可用内容,应先吸收到 `docs/plans/``docs/technical/`,再归档保留来源记录

View File

@@ -0,0 +1,117 @@
# Earth 电视直播模块计划
## 目标
`Earth` 页面增加一个可配置、可扩展、可拖拽的电视直播模块:
- 后台可配置新闻直播源
- 默认兜底源为央视 `CCTV-4`
- 未来可通过采集器接入世界各地新闻直播源
- Earth 工具栏 `显示控制` 子菜单新增电视按钮
- 点击后打开一个与其他 HUD 一致的可拖拽/可关闭窗口
- 窗口内部可播放或承载新闻直播页面
## 设计原则
- 第一阶段先交付“后台可配 + Earth 可用 + 默认可回退”的版本
- 公开读取接口与后台管理接口分离
- 手工配置源与采集器源共用统一的前端消费结构
- Earth 里的电视窗口必须复用现有 HUD 拖拽、关闭、布局最大化逻辑
- 小屏下优先保证窗口完整显示,超出部分在窗口内部滚动
## 分阶段实现
### Phase 1后端配置与公开读取
- 在系统设置中新增 `tv` 分类
- 定义直播源配置结构:
- `default_source_id`
- `auto_fallback`
- `sources[]`
- 每个直播源至少包含:
- `id`
- `name`
- `provider`
- `region`
- `language`
- `source_type`
- `embed_url`
- `stream_url`
- `homepage_url`
- `is_enabled`
- `is_fallback`
- `sort_order`
- `collector_source`
- `notes`
- 默认兜底源使用央视官网 `CCTV-4` 直播页
- 新增公开读取接口,供 Earth 页面无登录态读取直播源配置
### Phase 2采集器扩展位
- 新增 `news_live_streams` collector 占位
- 规范采集器入库数据结构,使其能与后台手工配置源合并
- TV 公开接口支持合并:
- 后台手工配置源
- 采集器入库源
- 保持手工配置源优先级更高,避免采集器覆盖人工兜底配置
### Phase 3后台配置界面
- 在系统配置页新增 `电视直播` tab
- 支持:
- 查看当前默认源
- 开关自动回退
- 新增直播源
- 编辑直播源
- 删除直播源
- 启用/禁用直播源
- 将某个直播源设为默认源
- 明确区分:
- 手工配置源
- 采集器来源
### Phase 4Earth HUD 集成
-`显示控制` 子菜单加入电视按钮
- 新增 TV HUD 面板:
- 可拖拽
- 可关闭
- 支持显示/隐藏状态同步
- 参与布局最大化与恢复布局
- 面板内容至少包含:
- 当前频道标题
- 源切换下拉菜单
- 刷新按钮
- 打开官网按钮
- 播放区域
### Phase 5播放策略
- 第一版优先支持 `iframe`/嵌入页类直播源
- 为未来扩展保留:
- `hls`
- `video`
- `external`
- 如果默认源不可用:
- 优先回退到标记为 `is_fallback=true` 的源
- 若无明确回退源,则回退到第一个可用源
- 面板内要有清晰的加载、错误、回退提示
### Phase 6打磨与清理
- 统一 HUD 风格
- 小屏下限制窗口尺寸并启用内部滚动
- 避免窗口超出屏幕
- 补最小验证
- 清理临时代码、重复样式和无用资源
## 首版交付定义
当以下条件满足时,认为首版可用:
- 后台可以配置新闻直播源
- Earth 可以读取并显示默认直播源
- 工具栏可打开电视窗口
- 电视窗口可拖拽、可关闭
- 央视 `CCTV-4` 作为默认兜底源可被使用
- 代码结构已为后续采集器接入预留统一接口

View File

@@ -0,0 +1,165 @@
# HUD Panel Component Plan
## Goal
Unify Earth HUD panels into a reusable component layer so new panels can share:
- a consistent shell
- a consistent header
- a consistent action-button system
- a consistent body and collapse pattern
## Scope
Target panels:
- `tv-panel`
- `news-panel`
- `legend`
- `layer-panel`
- `earth-stats`
- `info-card`
- settings modal header/actions
## Component Model
### Base shell
- `.hud-panel`
- `.hud-panel--compact`
- `.hud-panel--media`
- `.hud-panel--collapsed`
- `.hud-panel-hidden`
- `.hud-panel.is-dragging`
- `.hud-panel.is-layout-animating`
### Header
- `.hud-panel__header`
- `.hud-panel__title-group`
- `.hud-panel__title`
- `.hud-panel__subtitle`
- `.hud-panel__chip`
- `.hud-panel__actions`
Header baseline rule:
- Header title styling is fixed by the component layer and should not drift per panel
- Title font size, font weight, letter spacing, line height, text color, and vertical alignment come from the shared header tokens and structure
- Header divider, border treatment, inner spacing, and title-to-actions alignment are part of the same shared baseline
- Panel-specific header differences should be limited to explicit variants such as `compact` or `media`, or token overrides with documented intent
- “Looks close enough” local header overrides should be treated as temporary compatibility code and removed during migration
### Actions
- `.hud-panel__action`
- `.hud-panel__action--icon`
- `.hud-panel__action--collapse`
- `.hud-panel__action--close`
- `.hud-panel__action--refresh`
- `.hud-panel__action--external`
Action-button baseline rule:
- Header action buttons must have one fixed default style baseline across all HUD panels
- Default width behavior, padding, icon size, radius, alignment, hover, and active feedback all come from `.hud-panel__action`
- Panel-specific differences must be expressed through explicit variants or token overrides, not ad-hoc local button rewrites
- `close` buttons are part of the same default action system and must not silently fall back to a separate legacy box model
### Body
- `.hud-panel__body`
- `.hud-panel__body--scroll`
- `.hud-panel__body--collapsible`
### Collapse behavior
- `.hud-panel--collapsed`
- `.hud-panel--expand-up`
- `.hud-panel--expand-down`
Adaptive collapse / expand rule:
- HUD panels support two expansion directions:
- top-to-bottom expansion
- bottom-to-top expansion
- Expansion direction should be decided at runtime from available viewport space rather than hardcoded per panel
- Use:
- `d` = available distance from the header anchor to the viewport bottom edge
- `h` = expected expanded panel height
- buffer = `20px`
- Collapsed-state direction rule:
- if `d > h + 20px`, the next action direction is `expand-up`
- if `d <= h + 20px`, the next action direction is `expand-down`
- To avoid jitter around the threshold, the shared controller should keep a small hysteresis band:
- if the current direction is already `up`, keep it until `d <= h`
- if the current direction is already `down`, keep it until `d > h + 20px`
- The opposite edge is still a safety guard:
- if the chosen side cannot fit at all, fall back to the other side if it can fit
- if neither side fully fits, choose the side with more space and let the body scroll
- If neither direction fully fits, choose the direction with more available space and let the body scroll
- Collapse icon direction must match the active expansion direction so the icon always describes the real open/close motion
- The collapse icon describes the next action, not the current state
- This mapping is fixed component behavior and must not drift per panel:
- collapsed + expand-down => `expand_more`
- expanded + expand-down => `expand_less`
- collapsed + expand-up => `expand_less`
- expanded + expand-up => `expand_more`
- Panels must not combine icon-name swapping with extra CSS rotation for the same collapse control
- Expansion direction and icon direction must come from one shared source of truth in the component controller
- The direction decision should be recomputed when opening, resizing the viewport, or restoring a dragged panel near another edge
## Tokens
Promote panel differences into CSS variables instead of duplicating selectors:
- `--hud-panel-padding`
- `--hud-header-padding`
- `--hud-header-gap`
- `--hud-action-padding`
- `--hud-action-gap`
- `--hud-action-icon-size`
- `--hud-body-gap`
- `--hud-body-max-height`
- `--hud-chip-radius`
- `--hud-title-font-size`
- `--hud-title-font-weight`
- `--hud-title-letter-spacing`
- `--hud-title-line-height`
- `--hud-title-color`
- `--hud-header-border-color`
- `--hud-header-divider-opacity`
- `--hud-expand-direction`
## Migration Order
1. Build the shared component layer in `frontend/public/earth/css/hud.css`
2. Migrate `tv-panel` and `news-panel` first as the reference implementation
3. Migrate `legend` and `layer-panel` into a compact variant
4. Migrate `earth-stats` and `info-card`
5. Align settings modal header/actions with the same action system
6. Remove legacy one-off button selectors after verification
## Guardrails
- Do not change panel behavior and data flow during the first pass
- Keep old class names temporarily as compatibility hooks
- Prefer variable overrides over per-panel reimplementation
- Treat header action-button default styling as fixed component API, not per-panel design space
- Treat header title typography, border, and divider styling as fixed component API, not per-panel design space
- Treat collapse direction as a component behavior contract, not a one-off panel trick
- Treat collapse icon semantics as a component behavior contract, not a per-panel visual preference
- Verify header alignment and drag/collapse behavior after each migration batch
## First Implementation Batch
Batch 1 should only do:
- shared header structure
- shared action-button system
- shared title typography and header border/divider baseline
- shared collapsible body pattern
- adaptive collapse direction logic and direction-aware collapse icons
- migration of `tv-panel` and `news-panel`
That keeps risk low while giving the rest of the HUD a stable target to migrate toward.

View File

@@ -1,3 +1,5 @@
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
# 地球3D可视化架构重构计划
## 背景

View File

@@ -1,3 +1,5 @@
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
# 卫星预测轨道显示功能
## TL;DR

View File

@@ -1,3 +1,5 @@
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
# UE5 3D 大屏客户端开发计划
## 项目概述

View File

@@ -1,3 +1,5 @@
> Archived note: this document was originally created by sisyphus and later reviewed against the main docs set. Useful content has been absorbed into `docs/plans/` where appropriate.
# WebGL Instancing 卫星渲染优化计划
## 背景

35
docs/plans/README.md Normal file
View File

@@ -0,0 +1,35 @@
# Plans Docs
这里放“未来实施方案和未完成计划”的文档,重点回答:
- 我们准备做什么
- 为什么要做
- 分几期做
- 当前差距和下一步是什么
适合放入这里的内容:
- Earth / BGP / 地形 / 天球实施方案
- AI Playground 发展计划
- backend / datasource / agent roadmap
- UE5 MVP 方案
当前重点入口:
- [earth-renderer-architecture-separation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-renderer-architecture-separation-plan.md)
- [earth-predicted-orbit-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-predicted-orbit-plan.md)
- [earth-webgl-instancing-satellites-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-webgl-instancing-satellites-plan.md)
- [earth-real-terrain-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-real-terrain-plan.md)
- [earth-news-source-configuration-and-collector-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-news-source-configuration-and-collector-plan.md)
- [frontend-ai-playground-development-plan.md](/home/ray/dev/linkong/planet/docs/plans/frontend-ai-playground-development-plan.md)
- [ue5-mvp-fused-plan.md](/home/ray/dev/linkong/planet/docs/plans/ue5-mvp-fused-plan.md)
不适合放入这里的内容:
- 当前代码结构说明
- 组件现状和实现入口
- 已经落地的技术上下文说明
这些应放入:
- [docs/technical/README.md](/home/ray/dev/linkong/planet/docs/technical/README.md)

View File

@@ -0,0 +1,647 @@
# Agent Architecture Plan
## Overview
This document defines the agent architecture for Planet.
The architecture is intentionally broader than datasource health checking.
It is designed to support both:
- datasource health governance
- future situational-awareness workflows
The core idea is to avoid building a one-off "repair broken API links" agent.
Instead, Planet should grow a reusable agent runtime that can:
- collect evidence
- evaluate signals
- reason over incomplete information
- generate proposals
- produce assessments
- execute limited actions under policy
## Design Goal
Build an agent foundation that can evolve in this order:
1. datasource health checks
2. datasource repair proposals
3. signal correlation
4. situational assessments
5. controlled runtime actions
This means the architecture should treat datasource health as one use case of the larger agent system, not as the whole system.
## Core Principles
1. Separate evidence from reasoning
- raw signals should be gathered first
- deterministic checks should run before LLM reasoning
2. Agents do not own the defaults
- repository defaults remain human-owned
- agents operate on runtime state, proposals, and overrides
3. Reasoning and action are different responsibilities
- many agents should be read-only or propose-only
- only tightly controlled flows may apply changes
4. Shared runtime, specialized roles
- multiple agent roles should share the same object model and orchestration patterns
- health and situational-awareness agents should not invent incompatible payloads
5. Auditability is mandatory
- every proposal, assessment, and applied action should be attributable
## System Layers
Planet agent architecture should be split into four layers.
### 1. Signal Layer
Purpose:
- gather raw evidence from internal and external systems
Example sources:
- collector outputs
- datasource health checks
- logs
- snapshots
- alerts
- web search results
- scraped pages
- external APIs
- operator inputs
Responsibilities:
- fetch
- normalize
- timestamp
- tag with source and trust level
This layer should not make high-level judgments.
### 2. Evaluation Layer
Purpose:
- perform deterministic analysis
Examples:
- reachability checks
- schema validation
- threshold checks
- time-window comparisons
- anomaly counters
- completeness checks
Responsibilities:
- classify signals into machine-readable findings
- attach deterministic evidence
This layer should avoid LLM dependency whenever possible.
### 3. Reasoning Layer
Purpose:
- use LLMs when semantic interpretation or incomplete-information reasoning is needed
Examples:
- endpoint migration inference
- multi-source event correlation
- causality hypotheses
- ambiguity reduction
- assessment narrative generation
- action recommendation generation
Responsibilities:
- synthesize evidence
- produce hypotheses
- rank confidence
- explain reasoning boundaries
This is the main place where `aiprovider` and web search are used.
### 4. Action Layer
Purpose:
- convert proposals or assessments into controlled system actions
Examples:
- create runtime override
- create proposal
- publish alert
- update operator task queue
- generate summary artifact
- trigger follow-up verification
Responsibilities:
- enforce policy
- enforce approval requirements
- verify post-action outcomes
- record audit trails
## Architecture Sketch
```mermaid
flowchart TD
A["Collectors / Logs / Snapshots / External APIs"] --> B["Signal Layer"]
W["Web Search / Page Fetch / Docs"] --> B
B --> C["Evaluation Layer"]
C --> D["Findings"]
D --> E["Reasoning Layer (LLM + Tools)"]
E --> F["Proposals"]
E --> G["Assessments"]
F --> H["Action Layer"]
H --> I["Runtime Overrides / Alerts / Tasks"]
H --> J["Verification Loop"]
J --> B
K["Policy Engine"] --> H
L["Audit / History Store"] --> H
L --> E
L --> C
```
## Agent Roles
The first version should define these logical roles.
### 1. Health Agent
Primary use case:
- datasource health governance
Inputs:
- datasource metadata
- current endpoint
- latest health records
- latest failures
- deterministic findings
Outputs:
- health interpretation
- repair proposal
- confidence
- evidence references
Typical action level:
- propose-only
### 2. Correlation Agent
Primary use case:
- identify whether multiple signals describe the same event or related events
Inputs:
- findings from multiple collectors
- time windows
- region / ASN / prefix / cable relationships
- prior incidents
Outputs:
- grouped event candidates
- correlation rationale
- confidence per relationship
Typical action level:
- read-only
### 3. Assessment Agent
Primary use case:
- produce situational-awareness outputs
Inputs:
- grouped events
- findings
- current context
- historical context
- operator constraints
Outputs:
- structured assessment
- risk summary
- evidence-backed recommendations
- missing-information list
Typical action level:
- read-only or propose-only
### 4. Recovery Agent
Primary use case:
- carry low-risk proposals into controlled runtime actions
Inputs:
- approved proposal
- policy constraints
- trusted-domain rules
- verification checks
Outputs:
- applied override
- failed application
- rollback request
Typical action level:
- apply-limited
## Shared Object Model
All agents should work on a shared object model.
That prevents the health subsystem and situational-awareness subsystem from drifting into incompatible payloads.
### Signal
Represents a raw observed fact.
Examples:
- a datasource returned HTTP 404
- a collector returned empty results
- BGP updates spiked in one region
- a known endpoint now redirects elsewhere
Suggested shape:
```json
{
"id": "sig_123",
"type": "datasource.http_failure",
"source": "ris_live_bgp",
"occurred_at": "2026-04-08T10:00:00Z",
"severity": "medium",
"payload": {},
"trust": 0.95
}
```
### Finding
Represents a deterministic or semi-deterministic interpretation of one or more signals.
Examples:
- `schema_changed`
- `endpoint_unreachable`
- `data_volume_abnormally_low`
- `event_cluster_detected`
Suggested shape:
```json
{
"id": "find_123",
"type": "datasource.schema_changed",
"source_ids": ["sig_123"],
"confidence": 0.92,
"evidence": [],
"details": {}
}
```
### Proposal
Represents a recommended action, not an already-applied action.
Examples:
- switch endpoint to new URL
- disable bad override
- escalate issue for manual review
Suggested shape:
```json
{
"id": "prop_123",
"kind": "endpoint_override",
"target": "telegeography_cables",
"confidence": 0.84,
"reason": "Official docs now point to a new API path",
"payload": {},
"evidence_urls": [],
"status": "proposed"
}
```
### Assessment
Represents a structured situational-awareness output for operators or downstream systems.
Examples:
- current network posture summary
- incident impact assessment
- risk and response recommendations
Suggested shape:
```json
{
"id": "assess_123",
"scope": "regional-network",
"risk_level": "high",
"summary": "Regional routing instability is increasing.",
"key_risks": [],
"evidence": [],
"recommendations": [],
"missing_data": []
}
```
## State Machine
The shared orchestration flow should look like this:
```mermaid
stateDiagram-v2
[*] --> Collect
Collect --> Validate
Validate --> Classify
Classify --> Reason
Reason --> Propose
Reason --> Assess
Propose --> Review
Review --> Apply
Apply --> Verify
Verify --> Archive
Assess --> Archive
Archive --> [*]
```
Definitions:
- `Collect`: gather signals
- `Validate`: run deterministic checks
- `Classify`: create findings
- `Reason`: invoke LLM reasoning when needed
- `Propose`: create change proposals
- `Review`: policy or human approval
- `Apply`: perform limited runtime action
- `Verify`: confirm action effect
- `Archive`: store artifacts and decisions
## Permission Model
Each agent role should be assigned one of these action levels.
### `read-only`
Allowed:
- read signals
- search web
- fetch pages
- read internal state
- generate findings and assessments
Not allowed:
- mutate config
- write overrides
- change live runtime behavior
### `propose-only`
Allowed:
- everything in `read-only`
- create proposals
- create review tasks
Not allowed:
- apply live changes
### `apply-limited`
Allowed:
- everything in `propose-only`
- write approved runtime overrides
- trigger verification checks
Not allowed:
- mutate repository defaults
- make destructive data changes
- bypass policy engine
## Runtime Components
The first durable architecture should introduce these components.
### 1. Signal Store
Stores normalized evidence and health outputs.
### 2. Finding Store
Stores deterministic classifications that can be reused by multiple agents.
### 3. Proposal Store
Stores recommended actions with evidence and confidence.
### 4. Assessment Store
Stores structured situational-awareness outputs.
### 5. Policy Engine
Decides:
- whether agent may run
- whether proposal requires review
- whether proposal may auto-apply
- whether post-apply verification passed
### 6. Override Store
Stores runtime-only configuration changes.
This is where endpoint repairs should live.
## Relation To `aiprovider`
`aiprovider` should remain the model gateway.
It should not become the full agent runtime.
Recommended split:
- `aiprovider`
- provider adaptation
- prompt transport
- model execution
- protocol compatibility
- agent runtime
- orchestration
- signal handling
- tool selection
- proposal generation
- policy and audit
This keeps provider concerns and agent behavior concerns separate.
## Relation To Datasource Health
Datasource health becomes one vertical slice of this architecture.
Mapping:
- signal:
- endpoint unreachable
- schema mismatch
- bad content type
- finding:
- `failed`
- `schema_changed`
- `moved_endpoint_suspected`
- proposal:
- runtime override suggestion
- assessment:
- datasource health summary for operators
## Relation To Situational Awareness
Future situational-awareness capabilities should reuse the same flow:
- raw telemetry becomes signals
- anomaly detection becomes findings
- LLM correlation becomes reasoning
- operator-facing output becomes assessments
- policy-approved mitigations become actions
This lets the platform evolve from operational health governance into broader cyber/network posture workflows without changing the architecture.
## Suggested Delivery Sequence
### Phase A
- finalize shared object model
- implement health-oriented signal and finding storage
### Phase B
- implement Health Agent
- generate proposals only
### Phase C
- implement Assessment Agent
- expose structured assessments via API
### Phase D
- implement Correlation Agent
- support multi-source incident grouping
### Phase E
- implement Recovery Agent with policy-gated runtime actions
## Recommended First Build
The first build should not try to implement every agent role.
Recommended initial slice:
- shared object model
- health signals
- health findings
- Health Agent
- proposal generation only
This gives immediate value while preserving the longer-term architecture.
## Non-Goals For The First Iteration
- repository YAML auto-rewrites
- unrestricted autonomous action
- full incident graph reasoning
- automatic large-scale remediation
- agent-owned configuration source of truth
## Summary
Planet should treat agents as a reusable runtime for evidence, reasoning, proposals, and assessments.
The datasource health use case is the first practical entrypoint, but the architecture should already assume future situational-awareness expansion.
The safest path is:
- deterministic checks first
- agent reasoning second
- proposals before actions
- runtime overrides instead of default mutation

View File

@@ -0,0 +1,346 @@
# Agent Runtime Roadmap
## Overview
This document connects three existing planning threads into one implementation roadmap:
- `aiprovider` as the model gateway
- datasource health governance as the first practical agent use case
- situational awareness as the broader long-term target
Related documents:
- [aiprovider](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
- [datasource-health-plan](/home/ray/dev/linkong/planet/docs/plans/agents-datasource-health-plan.md)
- [agent-architecture-plan](/home/ray/dev/linkong/planet/docs/plans/agents-agent-architecture-plan.md)
## Big Picture
Planet should evolve in layers:
1. stable model gateway
2. deterministic health and evidence collection
3. agent runtime for reasoning and proposal generation
4. situational-awareness assessments and controlled actions
This prevents the system from collapsing into a single giant "AI feature" with unclear boundaries.
## Architecture Overview
```mermaid
flowchart TD
U["Frontend / Backend APIs / Operators"] --> B["Planet Backend"]
B --> H["Datasource Health Services"]
B --> R["Agent Runtime"]
R --> P["aiprovider"]
P --> M["OpenAI / Anthropic / MiniMax / Ollama / Local Models"]
C["Collectors / Snapshots / Logs / Alerts / BGP Signals"] --> S["Signal Store"]
H --> S
S --> E["Evaluation Layer"]
E --> F["Findings"]
F --> R
W["Web Search / Page Fetch / Docs Fetch"] --> R
R --> PR["Proposals"]
R --> AS["Assessments"]
PR --> O["Runtime Overrides / Review Queue / Tasks"]
AS --> SA["Situational Awareness APIs / UI"]
O --> V["Verification Loop"]
V --> S
```
## Role Boundaries
### `aiprovider`
Responsibilities:
- provider compatibility
- protocol adaptation
- auth and model transport
- request/response normalization
Not responsible for:
- agent orchestration
- business workflows
- datasource repair policy
- situational-awareness domain logic
### Backend
Responsibilities:
- stable business APIs
- auth and permissions
- task orchestration
- health records
- proposal and override persistence
- assessment exposure
### Agent Runtime
Responsibilities:
- consume findings and context
- invoke LLMs via `aiprovider`
- invoke tools such as web search
- create proposals
- create assessments
- route to policy-controlled action paths
## Delivery Sequence
## Stage 1: Gateway Foundation
Status:
- already in place
Delivered by current work:
- `aiprovider`
- multi-provider compatibility
- backend AI facade
- MiniMax / Anthropic-compatible support
- request-id propagation
Primary outcome:
- the system already has a stable way to call models
## Stage 2: Datasource Health MVP
Goal:
- establish deterministic health observability
Key work:
- health check task runner
- health result table
- datasource health APIs
- UI visibility
- collector endpoint override precedence cleanup
Primary outcome:
- Planet knows which collectors are healthy before asking an LLM anything
## Stage 3: Health Agent
Goal:
- let the first agent role operate on health failures
Key work:
- convert health failures into signals/findings
- invoke agent only for failed or suspicious cases
- produce repair proposals with evidence and confidence
Primary outcome:
- Planet can suggest endpoint repairs without mutating defaults
## Stage 4: Runtime Repair Application
Goal:
- safely apply approved datasource repair proposals
Key work:
- override storage
- policy-gated apply flow
- verification after apply
- rollback path
Primary outcome:
- datasource repair becomes operationally useful without polluting repository defaults
## Stage 5: Situational Awareness Assessments
Goal:
- reuse the same runtime for broader operator-facing assessment
Key work:
- normalize telemetry and incident evidence into signals/findings
- build Assessment Agent
- expose structured assessments through backend APIs and UI
Primary outcome:
- LLM output becomes evidence-backed situational summary, not just ad hoc chat output
## Stage 6: Correlation and Controlled Actions
Goal:
- connect multiple sources into higher-level posture and event groupings
Key work:
- event correlation
- incident grouping
- recommendation scoring
- controlled action routing
Primary outcome:
- Planet becomes a true agent-assisted situational-awareness system
## Implementation Tracks
These tracks can progress in parallel, but they should stay loosely coupled.
### Track A: Config and Runtime Resolution
Scope:
- datasource defaults
- overrides
- runtime precedence
- audit trails
First milestone:
- health-safe override layer
### Track B: Health and Evidence
Scope:
- deterministic checks
- failure categorization
- signal and finding persistence
First milestone:
- datasource health record system
### Track C: Agent Runtime
Scope:
- shared object model
- orchestration flow
- prompt/tool pipeline
- policy integration
First milestone:
- Health Agent proposal pipeline
### Track D: Situational Awareness
Scope:
- assessment schema
- multi-source context assembly
- operator-facing outputs
First milestone:
- structured assessment API
## Shared Artifacts
To avoid fragmentation, these artifacts should be shared across all future agent work.
### Shared object model
- `Signal`
- `Finding`
- `Proposal`
- `Assessment`
### Shared orchestration flow
- collect
- validate
- classify
- reason
- propose or assess
- review or apply
- verify
- archive
### Shared policy model
- read-only
- propose-only
- apply-limited
## Recommended Next Concrete Steps
1. Build Stage 2 first
- datasource health records
- deterministic checks
- no automatic repair
2. Then build Stage 3
- Health Agent
- proposal generation only
3. Then Stage 4
- override apply flow
- rollback and verification
4. Only after that start Stage 5
- broader situational-awareness assessment workflows
## Why This Order
Because situational-awareness quality depends on reliable upstream data.
If datasource health is weak:
- agent reasoning quality will degrade
- false explanations will increase
- assessment trust will drop
So datasource health is not a side task.
It is the first operational foundation for the later situational-awareness system.
## Summary
Planet should be built as:
- `aiprovider` for model access
- backend services for orchestration and persistence
- datasource health as the first evidence-governance layer
- agent runtime as the reusable reasoning core
- situational awareness as the long-term application layer
That path keeps the architecture coherent and lets each phase produce useful functionality without forcing a rewrite later.

View File

@@ -0,0 +1,486 @@
# Datasource Health Plan
## Overview
This document defines a phased plan for datasource health governance.
The goal is to make collectors observable, diagnosable, and recoverable when upstream APIs change, while avoiding unsafe automatic mutation of repository defaults.
The key principle is:
- do not let runtime automation rewrite repository default config
Instead, split responsibilities across:
- default config
- runtime overrides
- health check records
- agent-generated repair proposals
## Problem Statement
Collectors currently depend on third-party APIs, data downloads, mirrored JSON files, archive links, and web pages.
These upstream dependencies can fail in several ways:
- endpoint becomes unreachable
- endpoint still responds but schema changes
- content-type changes
- website shuts down or moves
- mirror link disappears
- HTML structure changes and scraping fails
- endpoint requires a new path or new host
We want a system that can:
- detect datasource health degradation early
- identify likely cause
- search for updated endpoints when reasonable
- apply safe runtime fixes without polluting default repo config
- preserve auditability and rollback
## Design Principles
1. Default config is stable
- `backend/app/core/data_sources.yaml` remains the repository baseline.
- It should be changed intentionally through normal development flow, not by autonomous runtime agents.
2. Runtime fixes are isolated
- Emergency or adaptive fixes should live in a runtime override layer.
- Overrides should be reversible and auditable.
3. Deterministic checks come first
- Use normal programmatic health checks before using LLMs.
- Only call an agent when deterministic checks indicate a meaningful failure.
4. Agents suggest before they mutate
- Agents should produce proposals with evidence and confidence.
- Application of a proposal should be controlled by policy.
5. Every repair is attributable
- Store what changed, why, who or what suggested it, and when it was applied.
## Configuration Layers
Recommended runtime precedence:
1. datasource endpoint override
2. datasource DB endpoint override
3. repository default YAML
4. collector internal fallback logic
Definitions:
- repository default YAML:
- `backend/app/core/data_sources.yaml`
- versioned baseline
- datasource DB endpoint override:
- existing `DataSourceConfig.endpoint`
- current runtime override entrypoint
- datasource endpoint override:
- a dedicated new override table
- used for health-repair and proposal application
- collector internal fallback logic:
- final defensive fallback
- should be minimized over time
## Recommended Architecture
### 1. Deterministic Health Checks
Each collector gets a health profile with checks such as:
- endpoint resolves
- HTTP request succeeds
- status code is acceptable
- content-type is expected
- body parses successfully
- minimum structural fields exist
- sample item count is plausible
- latency is within threshold
Output states:
- `healthy`
- `degraded`
- `failed`
- `schema_changed`
- `rate_limited`
- `auth_required`
### 2. Agent-Assisted Repair Discovery
Only triggered when deterministic health checks fail or return suspicious structure.
Agent responsibilities:
- search for current official endpoint or replacement path
- inspect likely upstream documentation or landing pages
- compare candidate endpoint output to collector expectations
- produce a repair proposal with confidence and evidence
Agent should not directly modify repository defaults.
### 3. Safe Runtime Repair Application
Repair proposals can be:
- reviewed manually
- auto-applied only under strict low-risk policy
Auto-apply should be limited to cases like:
- same trusted domain
- highly similar response structure
- repeated successful verification
- confidence above threshold
## Phased Delivery Plan
## Phase 1: Deterministic Health MVP
Goal:
- build health observability without automated repair
Scope:
- datasource health check task runner
- datasource health result persistence
- endpoint reachability + parse checks
- dashboard or API visibility into health status
Deliverables:
- health check service
- health check record table
- status endpoint
- scheduled or manual check trigger
No agent usage yet.
## Phase 2: Agent Repair Proposals
Goal:
- let agent investigate failing sources and propose updated endpoints
Scope:
- invoke agent only when datasource health is `failed` or `schema_changed`
- web search + page inspection
- candidate endpoint extraction
- proposal persistence
Deliverables:
- repair proposal schema
- proposal generation pipeline
- confidence and evidence model
- operator review view or API
Still no automatic config mutation.
## Phase 3: Runtime Overrides
Goal:
- allow approved proposals to take effect safely at runtime
Scope:
- add dedicated override storage
- runtime resolution prefers override over default config
- proposal application writes override only
Deliverables:
- endpoint override table
- override-aware resolution logic
- apply/reject endpoints
- rollback endpoint
Repository default YAML remains untouched.
## Phase 4: Limited Auto-Apply
Goal:
- safely automate a narrow slice of low-risk repairs
Scope:
- policy engine for auto-apply
- same-domain or trusted-domain checks
- structure validation
- staged verification after apply
Deliverables:
- auto-apply rules
- audit logs
- automatic post-apply health verification
- auto-disable or rollback on regression
## Data Model Draft
### datasource_health_checks
Purpose:
- store each health evaluation result
Suggested fields:
- `id`
- `datasource_id`
- `collector_name`
- `endpoint_checked`
- `status`
- `http_status`
- `content_type`
- `latency_ms`
- `sample_count`
- `error_message`
- `details`
- `checked_at`
`details` can store structured diagnostic data such as:
- parsed fields
- schema mismatch summary
- retry count
- exception class
### datasource_repair_proposals
Purpose:
- store agent-generated repair suggestions
Suggested fields:
- `id`
- `datasource_id`
- `collector_name`
- `old_endpoint`
- `candidate_endpoint`
- `reason`
- `confidence`
- `evidence_urls`
- `evidence_summary`
- `status`
- `created_by`
- `created_at`
- `reviewed_at`
Suggested `status` values:
- `proposed`
- `approved`
- `rejected`
- `applied`
- `expired`
### datasource_endpoint_overrides
Purpose:
- runtime endpoint override layer
Suggested fields:
- `id`
- `datasource_id`
- `collector_name`
- `endpoint`
- `reason`
- `source`
- `proposal_id`
- `enabled`
- `created_at`
- `updated_at`
Suggested `source` values:
- `manual`
- `health-agent`
- `migration`
## API Draft
### Health
- `GET /api/v1/datasources/health`
- `GET /api/v1/datasources/{id}/health`
- `POST /api/v1/datasources/{id}/health-check`
- `POST /api/v1/datasources/health-check-all`
### Repair proposals
- `GET /api/v1/datasources/{id}/repair-proposals`
- `POST /api/v1/datasources/{id}/repair-proposals/generate`
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/approve`
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/reject`
- `POST /api/v1/datasources/{id}/repair-proposals/{proposal_id}/apply`
### Overrides
- `GET /api/v1/datasources/{id}/overrides`
- `POST /api/v1/datasources/{id}/overrides`
- `PUT /api/v1/datasources/{id}/overrides/{override_id}`
- `DELETE /api/v1/datasources/{id}/overrides/{override_id}`
## Agent Contract Draft
When deterministic health fails, the agent should receive:
- datasource name
- collector name
- current endpoint
- current failure mode
- expected response shape summary
- known trusted domains
Expected output:
```json
{
"status": "proposal",
"candidate_endpoint": "https://example.com/api/v2/data",
"confidence": 0.86,
"reason": "Official docs now point to v2 endpoint",
"evidence_urls": [
"https://example.com/docs/api",
"https://example.com/changelog"
],
"notes": "Response shape appears compatible after light field remapping"
}
```
The agent should never output "rewrite the default yaml" as its primary action.
## Risk Analysis
### Risk: wrong endpoint chosen by agent
Mitigation:
- use trusted-domain allowlists
- require evidence URLs
- require confidence threshold
- add manual review for medium-risk sources
### Risk: endpoint responds but schema silently changed
Mitigation:
- deterministic schema checks
- parse and sample validation
- content-type checks
- collector-specific required fields
### Risk: automatic runtime override causes hidden drift
Mitigation:
- store all overrides explicitly
- mark source of override
- keep default YAML unchanged
- expose active overrides in API/UI
### Risk: persistent bad override breaks data collection
Mitigation:
- allow rollback
- keep parent/default endpoint visible
- re-run verification after apply
- auto-disable override on repeated failure
## Operational Policy Recommendations
1. Do not auto-apply for high-value or high-fragility sources initially.
2. Use manual approval for:
- scraped HTML sources
- unofficial mirrors
- sources with auth or rate-limit complexity
- sources with legal or trust ambiguity
3. Allow auto-apply only for:
- same-domain version bumps
- obvious official migration paths
- repeated passing verification
4. Expose health + proposal + override state together in one operator view.
## Suggested Implementation Order
1. Phase 1
- health result table
- deterministic checks
- API and UI visibility
2. Phase 2
- proposal table
- agent prompt/output contract
- proposal generation job
3. Phase 3
- runtime override table
- resolver precedence update
- apply/reject endpoints
4. Phase 4
- auto-apply rules
- rollback policy
- operator automation
## Out Of Scope For The First Iteration
- direct automatic mutation of repository default YAML
- automatic git commits by repair agents
- unrestricted autonomous endpoint replacement
- fully generalized schema remapping engine
## Recommended First Milestone
The first milestone should be:
- deterministic datasource health checks
- persisted results
- manual visibility
- no automatic repair
This gives immediate operational value with low risk, and prepares clean inputs for the later agent phase.

View File

@@ -0,0 +1,478 @@
# Datasource Health Stage 2 Tasks
## Goal
Stage 2 focuses on the first practical operational layer:
- deterministic datasource health checks
- persisted health results
- health visibility through API and UI
- no agent-assisted repair yet
This stage should make Planet capable of answering:
- which collectors are healthy
- which collectors are degraded
- which collectors are failing
- why they are failing at a basic deterministic level
## Scope
Included:
- datasource health data model
- deterministic health check service
- manual and scheduled health check triggers
- health result APIs
- frontend visibility
Excluded:
- LLM reasoning
- web-search-based repair proposals
- automatic endpoint rewriting
- runtime override application
## Delivery Target
At the end of Stage 2, an operator should be able to:
1. see health status for each collector
2. trigger a health check manually
3. inspect the latest failure reason
4. inspect the last checked endpoint
5. understand whether the problem is:
- unreachable
- auth-related
- rate-limit-related
- schema-related
- empty-data-related
## Work Breakdown
## A. Data Model
### A1. Add datasource health record table
Create a new model, for example:
- `backend/app/models/datasource_health_check.py`
Suggested fields:
- `id`
- `datasource_id`
- `collector_name`
- `endpoint_checked`
- `status`
- `http_status`
- `content_type`
- `latency_ms`
- `sample_count`
- `error_message`
- `details`
- `checked_at`
Suggested status enum values:
- `healthy`
- `degraded`
- `failed`
- `schema_changed`
- `rate_limited`
- `auth_required`
- `empty_result`
### A2. Add datasource health summary fields
Option A:
- keep summary only in the health check table
Option B:
- also add summary fields on `data_sources`
Recommended first step:
- do not mutate `data_sources` schema yet
- derive summary from the latest health record
### A3. Migration task
Add migration for the health table.
Deliverables:
- migration file
- model registration
## B. Health Check Engine
### B1. Define health check service
Add a new service module, for example:
- `backend/app/services/datasource_health.py`
Responsibilities:
- resolve effective endpoint
- execute deterministic check
- classify result
- persist health record
### B2. Define shared result schema
Create a typed result object, for example:
- `HealthCheckResult`
Suggested fields:
- `status`
- `endpoint_checked`
- `http_status`
- `content_type`
- `latency_ms`
- `sample_count`
- `error_message`
- `details`
### B3. Implement base deterministic checks
Every datasource should go through a minimal baseline check:
1. resolve endpoint
2. perform request
3. measure latency
4. inspect status code
5. inspect content type
6. inspect body shape
Classification rules:
- network error -> `failed`
- HTTP 401/403 -> `auth_required`
- HTTP 429 -> `rate_limited`
- HTTP 404/410 -> `failed`
- parse failure -> `schema_changed`
- zero or suspiciously empty results -> `empty_result` or `degraded`
- valid parse -> `healthy`
### B4. Add collector-aware adapters
Some collectors do not use the same fetch semantics.
Add adapter profiles such as:
- `http_json`
- `http_csv`
- `html_scrape`
- `stream_probe`
- `auth_session_http`
Initial mapping suggestion:
- `huggingface`, `peeringdb`, `cloudflare` -> `http_json`
- `fao` -> `http_csv`
- `top500`, `epoch_ai`, `telegeography live_map` -> `html_scrape`
- `ris_live` -> `stream_probe`
- `spacetrack` -> `auth_session_http`
### B5. Add sample validation hooks
For each adapter, add a lightweight validation rule.
Examples:
- JSON array length > 0
- CSV rows > 1
- HTML page contains expected table or script patterns
- stream source yields at least one valid event within timeout
## C. Persistence and Query Layer
### C1. Save every check run
Each health check should insert a record.
Do not overwrite history in Stage 2.
### C2. Add latest-health query helpers
Add helper functions to fetch:
- latest health record by datasource
- latest failed health record
- recent health history
### C3. Optional retention policy
For Stage 2, retention can be deferred.
If desired, keep only:
- last N records per datasource
## D. API Layer
### D1. Add health list endpoint
Suggested endpoint:
- `GET /api/v1/datasources/health`
Returns:
- datasource id
- collector name
- current endpoint
- latest health status
- last checked time
- short reason
### D2. Add per-datasource health detail endpoint
Suggested endpoint:
- `GET /api/v1/datasources/{id}/health`
Returns:
- latest record
- recent history
- detailed classification fields
### D3. Add manual health trigger endpoint
Suggested endpoint:
- `POST /api/v1/datasources/{id}/health-check`
Behavior:
- run a health check now
- persist the result
- return the new record
### D4. Add bulk health trigger endpoint
Suggested endpoint:
- `POST /api/v1/datasources/health-check-all`
Behavior:
- enqueue or run health checks for all active datasources
## E. Scheduling
### E1. Add health scheduler task
Decide scheduling strategy.
Recommended first version:
- run collector jobs and health checks separately
- health checks run on a lower frequency
Suggested frequency:
- every 6h or 12h for most datasources
- optionally on-demand only in the very first cut
### E2. Prevent health check collision with collection
Rules:
- health checks should not disrupt active collection
- they should use light requests
- if a collector is currently running, health check may:
- skip
- or use a lightweight endpoint probe only
## F. Frontend
### F1. Add health columns to datasource list
Update:
- `frontend/src/pages/DataSources/DataSources.tsx`
Suggested new columns:
- health status
- last checked
- reason summary
### F2. Add manual health check action
Per datasource:
- button or dropdown action:
- `健康检查`
### F3. Add health detail drawer or modal
Show:
- endpoint checked
- status
- HTTP status
- content type
- sample count
- error message
- last few results
### F4. Add basic visual language
Suggested colors:
- green -> healthy
- yellow -> degraded
- orange -> rate-limited / auth-required
- red -> failed / schema-changed
## G. Observability
### G1. Structured logging
Every health check should log:
- datasource id
- collector name
- endpoint
- status
- latency
- failure class
### G2. Optional metrics
If metrics are added later, useful counters include:
- health checks total
- health checks failed
- schema changes detected
- rate limited checks
## H. Tests
### H1. Unit tests
Add tests for:
- status classification
- content type classification
- adapter behavior
- latest-health query helpers
### H2. API tests
Add tests for:
- health endpoints require auth
- manual trigger endpoint works
- list endpoint returns latest status
### H3. Failure-path tests
Add coverage for:
- HTTP 404
- HTTP 429
- invalid JSON
- empty response
- parse mismatch
## Suggested File Plan
Possible implementation files:
- `backend/app/models/datasource_health_check.py`
- `backend/app/services/datasource_health.py`
- `backend/app/schemas/datasource_health.py`
- `backend/app/api/v1/datasource_health.py`
- migration file under the project migration system
Likely touched existing files:
- `backend/app/api/main.py`
- `frontend/src/pages/DataSources/DataSources.tsx`
- `backend/tests/test_api.py`
## Suggested Execution Order
1. Add model and migration
2. Add service and result schema
3. Add deterministic adapters
4. Add manual trigger API
5. Add list/detail API
6. Add frontend visibility
7. Add scheduled checks
8. Expand tests
## Minimal First Milestone
If we want the fastest useful slice, do this first:
1. health table
2. deterministic check service
3. manual per-datasource health check API
4. latest health list API
5. frontend status badge column
That is enough to start operating the system and will provide the input layer for Stage 3.
## Dependency On Later Stages
Stage 2 outputs become direct inputs for Stage 3.
Specifically:
- failed or schema-changed health records become agent triggers
- health history becomes repair context
- endpoint_checked becomes proposal baseline
## Success Criteria
Stage 2 is done when:
- every active datasource can be health-checked deterministically
- the latest health state is visible in API and UI
- operators can manually trigger checks
- failures are categorized into stable machine-readable statuses
- no LLM is required for core health visibility

View File

@@ -0,0 +1,309 @@
# Situational Awareness Foundation Plan
## 定位
当前这套 AI 能力应被视为 `态势感知服务底座`,而不是完整的态势感知产品。
也就是说,现阶段的目标不是:
- 做一个“什么都能分析”的万能 AI 页面
- 让模型在证据不足时替代人工研判
- 过早把页面做成完整指挥大屏
现阶段真正要做的是:
- 先把 `model gateway / backend facade / evidence injection / page-specific brief` 这几层边界搭稳
- 让系统能够在已有证据上稳定地产出“可读、可回看、可扩展”的摘要
- 为后续更强的数据联动、agent 推理和 assessment 结构化输出预留好接口与数据模型
## 当前现实约束
### 1. 数据维度不足
目前系统能提供的主要证据仍集中在:
- BGP incidents / anomalies / events
- collector coverage
- datasource health / platform alerts
- prefix geography 的部分归属信息
当前明显还缺:
- 流量异常与业务指标
- 电商、支付、物流等业务侧指标
- 更丰富的资产、链路、区域、行业画像
- 外部舆情、公告、运营商状态、基础设施事件等背景信息
这意味着:
- 模型现在可以做“基于现有证据的摘要与归纳”
- 但还不能可靠地做“跨维度因果研判”
### 2. 维度之间联动还弱
目前不同模块之间更多是“并列展示”,还不是“强关联分析”:
- 系统告警和 BGP 事件还没有统一事件模型
- collector bias 与真实区域热度还没有完全剥离
- datasource health 与 BGP 风险、业务影响之间还没有稳定映射
这意味着:
- 当前更适合做 `brief / overview / operator notes`
- 还不适合过度承诺“自动态势判断”
### 3. 结构化 assessment 还未成为主输出
虽然已经有 BGP brief、系统告警 brief、态势告警 brief但目前主输出仍偏向
- 文本摘要
- facts/context 附带证据
后续真正要服务态势感知,需要更稳定的结构化输出,例如:
- summary
- key risks
- evidence
- confidence
- recommendations
- missing data
## 当前基座已经具备的能力
### 1. AI 调用边界已经明确
- `aiprovider` 负责模型协议与 provider 兼容
- `backend` 负责业务 API、证据整合和鉴权
- `frontend` 负责页面入口与结果展示
### 2. 页面级 AI 入口已经开始成型
当前已经有或正在收口的入口:
- `Playground`
- 用于链路验证与 provider 诊断
- `BGP AI 简报`
- 用于 BGP 事实摘要和区域风险归纳
- `Alerts`
- 用于系统告警、BGP 告警、态势告警三类入口
### 3. 证据优先的方向已经建立
已经不再只依赖人工在 Playground 中手填 prompt系统开始具备
- 从真实业务数据生成事实输入
- 保存 facts/context 快照
- 回看 AI 输出时同时回看证据
这一步非常关键,因为它决定后面能否从“玩具 demo”走向“有运维价值的系统”。
## 近期收尾建议
这些事情都属于“底座收口”,值得做,但不应该再继续重产品包装。
### 1. 统一 Alerts 页面
已采用:
- 一个 `Alerts` 页面
- 三个 tab
- `系统告警`
- `BGP 告警`
- `态势告警`
收尾重点:
- 保持 tab 的文案、摘要卡和 AI 简报交互一致
- 不额外扩展成多个独立二级页面
### 2. 保持 Playground 为测试台
原则:
- Playground 只承担链路验证、provider 状态诊断、请求结果观察
- 不继续堆“万能业务分析器”式交互
### 3. 把 brief 能力当服务能力而不是页面特效
页面现在能看到按钮和结果,这很好,但更重要的是:
- 后端接口稳定
- facts/context 可追踪
- 输出结构后续可升级
### 4. 导航结构先收口,不继续平铺一级菜单
随着后续能力扩展,系统很可能继续新增:
- 海缆
- 算力中心
- 战争信息
- 电商分析
- 其他专题观测页
如果继续把这些入口全部平铺在左侧一级菜单中,会带来两个问题:
- 一级菜单过长,用户难以判断先进入哪个上下文
- `观测页 / 告警页 / 研判页 / 运维页` 的职责边界会被混在一起
因此近期应明确采用分组导航,而不是继续扩展平铺菜单。
推荐的导航分组如下:
- `总览`
- 仪表盘
- Earth
- `专题观测`
- BGP 观测
- 采集数据
- 后续可扩展:海缆、算力中心、战争信息、电商分析
- `告警与研判`
- Alerts
- `运维与配置`
- 数据源
- AI Playground
- 用户管理
- 系统配置
这套结构的含义是:
- `专题观测` 页面负责看某个维度本身
- `Alerts` 负责跨模块风险与值班工作台
- `Playground` 保持为测试台,不挤占业务导航语义
短期收尾时,应优先重组现有入口,而不是继续增加新的一级菜单。
## 后续路线
## Phase 1服务底座稳固
目标:
- 不追求“更炫的 AI 页面”
- 先把当前接口、证据、存储和页面入口收稳
工作项:
- 统一页面级 AI 入口模式
- 统一 brief response schema
- 保证 facts/context 在前后端都可回看
- 继续清理 mock 和临时分支逻辑
完成标准:
- 每个 AI 入口都是真实链路
- 每个 AI 结果都能追溯到证据输入
## Phase 2Evidence-first Assessment
目标:
- 从“文本摘要”升级成“结构化 assessment”
工作项:
- 为 brief/assessment 定义统一 schema
- 固化:
- summary
- key_risks
- evidence
- confidence
- recommendations
- missing_data
- 页面以结构化区块展示,而不只是大段文本
完成标准:
- AI 输出可持久化、可比较、可审计
## Phase 3多维证据接入
目标:
- 让“态势感知”真正拥有更多维度,而不是只靠 BGP 与系统告警
优先接入方向:
- datasource health findings
- 流量或业务指标
- 区域/资产/链路映射
- 外部事件与公告
- 业务垂直数据,例如电商分析相关指标
完成标准:
- AI 能基于多个维度做交叉说明
- 不再只围绕单一模块自说自话
## Phase 4Correlation Layer
目标:
- 不同来源的信号不再只是并列,而是形成统一的事件关联
工作项:
- 统一 signal/finding 模型
- 跨模块事件聚合
- 证据来源权重
- collector bias 与真实热度分离
完成标准:
- 系统能回答“这些异常是不是同一件事”
- 系统能回答“哪些结论只是观测偏差”
## Phase 5Agent-assisted Situational Awareness
目标:
- 在证据足够的前提下,再让 agent 负责更复杂的推理与建议
工作项:
- 复用现有 agent runtime 规划
- 引入 web search / docs fetch / repair proposal 等能力
- 但始终坚持:
- evidence first
- proposal before action
- no silent mutation of defaults
完成标准:
- agent 成为证据驱动的分析层
- 而不是一个“万能猜测层”
## 设计原则
### 1. 先底座,后产品化
先把服务链路和证据模型做好,再做更大的页面表达。
### 2. 先证据,后判断
事实输入应先稳定,再让模型做归纳。
### 3. 先专用 brief后统一态势层
先让各业务页有各自可信的 AI 入口,再考虑统一态势页。
### 4. 先 proposal后自动动作
涉及修复、覆盖、写配置、调任务的动作,都应经过 proposal 和审计。
## 当前建议结论
对现在这个项目,最合理的定位是:
- `Playground` 是测试台
- `BGP / Alerts` 是第一批业务 AI 入口
- `aiprovider + backend AI facade + evidence snapshots` 是核心服务底座
现阶段不需要追求“已经具备完整态势感知能力”。
现阶段真正的成功标准是:
- 这套底座可用
- 可回看
- 可扩展
- 不自欺欺人

View File

@@ -17,7 +17,7 @@ It is an aggregation/view-model layer:
## Why This Layer Exists
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/bgp-context.md):
Current product gap from [bgp-context.md](/home/ray/dev/linkong/planet/docs/technical/earth-bgp-context.md):
- incident density is naturally low
- anomaly density is higher, but still not enough to keep the globe expressive all the time
@@ -290,7 +290,7 @@ Each feature should include:
## Earth Rendering Plan
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/bgp-earth-rendering-plan.md).
Detailed visual layering guidance is expanded in [bgp-earth-rendering-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-earth-rendering-plan.md).
### Layer Relationship

View File

@@ -0,0 +1,715 @@
# Earth 天球背景与日月位置实施方案
## 目标
为 Earth 大屏增加一套真正可用的天文背景层,覆盖三件事:
1. 用真实天球背景替换当前随机星点
2. 在当前时间下显示太阳与月亮的相对位置
3. 让太阳方向同时驱动地球受光,形成更可信的昼夜关系
本方案优先追求:
- 与当前 Three.js Earth 架构兼容
- 风险可控
- 先落地一版真实感明显提升的 V1
- 为后续更严格的天文参考系升级预留余地
## 当前现状
当前 Earth 的基础条件已经具备:
- 地球、云层、地形、网格都基于 Three.js主渲染入口在 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- 地球实体创建在 [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
- 当前所谓“宇宙背景”只是 `createStars()` 生成的随机星点,不是真实星图
- Earth 已有倾角常量 `EARTH_CONFIG.tiltRad`,位于 [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
- 主循环 `animate()` 已稳定运行,可在其中接入天体更新逻辑
这意味着:
- 不需要重写 Earth
- 可以在现有 scene/world 层新增一个 celestial layer
- 第一阶段不必拆 Earth / satellite / cable 的参考系
## 总体策略
采用“两层现实”设计:
### 1. 世界层world-space celestial layer
用于放置:
- 天球背景
- 太阳
- 月亮
- 太阳光方向
这些对象不挂在 `earthObj` 上,而是直接放在 `scene` 中。
### 2. 地球层earth-fixed layer
继续保持当前结构:
- 海缆
- 登陆点
- 卫星点与轨迹
- BGP 覆盖
- 地球纹理、云层、地形
这些对象继续挂在 `earthObj` 下,不打断现有交互。
## 为什么先这样做
当前用户交互是“拖动地球本体”,而不是“移动相机绕惯性系观测”。
如果现在直接做严格惯性参考系改造,会同时影响:
- `earthObj.rotation`
- 卫星轨迹与锁定逻辑
- 海缆与登陆点附着关系
- resetView / autoRotate / hover / click 等交互链路
所以第一阶段只做:
- 真正的天空
- 真正的日月方向
- 不碰现有 Earth 附着对象的语义
## 推荐技术选型
### 天文计算库
推荐:
- [Astronomy Engine](https://github.com/cosinekitty/astronomy)
原因:
- 有 JavaScript 版本
- 支持 Sun / Moon 的矢量与坐标变换
- 精度、可扩展性都比轻量太阳高度角库更适合本项目
- 后续若要加行星、月相、黄道、赤道网,也能继续沿用
不作为主选的库:
- [SunCalc](https://github.com/mourner/suncalc)
原因:
- 更偏本地观察者视角的太阳/月亮高度角
- 用于“地面日出日落”很好
- 但不如 Astronomy Engine 适合做真实天球与后续空间参考系扩展
### Three.js 表现层
推荐组合:
- 天球:内翻球壳 + 星图纹理
- 太阳:`THREE.Sprite`
- 月亮:`THREE.Sprite` 或小型 `THREE.Mesh`
- 太阳光:`THREE.DirectionalLight`
参考:
- [Three.js SpriteMaterial](https://threejs.org/docs/pages/SpriteMaterial.html)
## 天球背景资源与星体数据来源
为避免把“视觉背景”和“可计算天体位置”混为一谈,本方案明确分成两类资源:
### 1. 背景资源:全天星图贴图
用于 Phase 1 的“真实天空背景”。
推荐优先来源:
- NASA SVS 的 Tycho 全天星图
- [The Tycho Catalog Skymap - Version 2.0](https://svs.gsfc.nasa.gov/3572/)
- NASA Deep Star Maps 2020
- SatelliteMap.space 在 credits 中明确提到其使用了 `NASA Deep Star Maps 2020 - High-resolution star field (1.7 billion stars from Gaia DR2)` 作为星空视觉资源
- 这说明行业内成熟实现并不一定直接渲染全部星表点,而很可能先使用一张高质量官方深空星图作为背景层
- 如需后续替换,也可评估 ESA / Gaia 的全天 sky map 资源
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议要求:
- 使用官方来源或官方衍生可复用资源
- 等距矩形投影equirectangular
- 坐标定义尽量明确为赤道坐标展开
- 分辨率建议至少 `4k`
- 颜色不要过亮,避免压过 Earth HUD 前景
- 尽量优先选择官方天文机构已经生产好的深空图,而不是自行拼接低质量星空纹理
建议本地资源目录:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
### 2. 位置数据:星表与天体计算
用于 Phase 2+ 的“位置正确的星体”。
推荐来源分两层:
- 太阳、月亮位置
- 使用 [Astronomy Engine](https://github.com/cosinekitty/astronomy)
- 恒星位置
- 第一优先Hipparcos / Tycho
- [Hipparcos overview](https://www.cosmos.esa.int/web/Hipparcos)
- [Hipparcos catalogues](https://www.cosmos.esa.int/web/hipparcos/catalogues)
- 第二优先Gaia
- [Gaia DR3 stories](https://www.cosmos.esa.int/web/gaia/dr3-stories)
建议策略:
- V1背景球壳只用全天星图不立即生成全量恒星点
- V2只挑选亮星例如星等 `< 5.5`)生成恒星点层
- V3如果确实需要更丰富的星场再逐步扩展到更深星等
这样做的原因:
- 背景球壳负责“天球真实感”
- 亮星点负责“位置正确、可后续标注和高亮”
- 不需要一开始就处理数十万甚至数百万颗星
### 3. 对外部成熟实现的参考结论
`SatelliteMap.space` 的公开 credits 提供了一个很有价值的参考样板:
- 图形渲染使用 `TWGL.js`
- 天文计算使用 `Skyfield``Astronomia`
- 星空/天球视觉资源使用 `NASA Deep Star Maps 2020`
这给本项目的启发是:
- “真实感强的天球背景”完全可以先依赖官方高质量深空图
- “位置正确的动态天体”则应依赖单独的天文计算链路
- 没有必要在第一版就直接渲染完整星表
因此本项目推荐继续坚持两层拆分:
- 背景层:官方深空图 / 全天星图
- 计算层:太阳、月亮与后续亮星点
## 如何保证星体位置正确
位置正确不是只看“图看起来像”,而是要统一参考系和转换链路。
### 1. 统一坐标基准
本方案推荐统一使用:
- `J2000` 赤道坐标系作为恒星位置基准
原因:
- Hipparcos / Tycho 资料和大量天文可视化都容易映射到该基准
- 太阳、月亮也可以通过 Astronomy Engine 转到同一坐标系
- 这样背景、恒星点、太阳、月亮就能共用一套 sky orientation
### 2. 背景贴图与点位必须使用同一展开逻辑
如果背景球壳使用赤道坐标全天图,那么:
- 亮星点也必须按赤道坐标贴到同一球面方向
- 太阳/月亮 sprite 也必须按赤道坐标转换后落到同一 world-space
否则会出现:
- 背景银河带是对的
- 但太阳/月亮或亮星点飘到不匹配的位置
### 3. RA / Dec 到 Three.js 坐标的落点方式
亮星点和日月方向最终都要转成单位球面向量。
概念步骤:
1. 读取赤经 `RA`
2. 读取赤纬 `Dec`
3. 转成弧度
4. 映射到单位球面向量
5. 再根据 Three.js 当前世界坐标定义做轴向映射
参考公式:
```text
x = cos(dec) * cos(ra)
y = sin(dec)
z = cos(dec) * sin(ra)
```
实际接入 Three.js 时,需要做一次项目内坐标轴校准:
- 验证 `RA = 0h`
- 验证 `RA = 6h`
- 验证北天极
- 验证银河带主方向
然后确定最终的:
- `x/y/z` 对应 Three.js 哪个轴
- 是否需要 `z` 取反
- 是否需要整体再做一个固定 `rotation`
建议把这层显式封装在:
```js
function equatorialToWorldVector(raRad, decRad)
```
不要把轴映射散落在不同模块里。
### 4. 背景球壳与恒星点的关系
推荐最终组合:
- 背景层:全天星图球壳
- 点位层:亮星点
- 动态层:太阳 / 月亮
这样有三个好处:
- 背景层提供密集真实的天空纹理
- 亮星点提供位置正确、可扩展的标注基础
- 太阳/月亮提供与时间相关的真实动态对象
## 数据与资源建议清单
### 推荐首批引入资源
1. 全天星图
- 来源NASA Tycho all-sky map
- 用途:背景球壳纹理
2. 月亮纹理
- 用途Phase 4 月相表现
- 路径建议:
- `frontend/public/earth/assets/celestial/moon_albedo_2k.jpg`
3. 太阳 glow 贴图
- 用途:太阳 sprite halo
- 路径建议:
- `frontend/public/earth/assets/celestial/sun_glow.png`
### 推荐首批数据文件
如果要上亮星层,建议新增一个预处理后的轻量数据文件:
- `frontend/public/earth/assets/celestial/bright-stars.json`
建议字段:
```json
[
{
"id": 32349,
"name": "Sirius",
"raDeg": 101.2875,
"decDeg": -16.7161,
"mag": -1.46,
"colorIndex": 0.00
}
]
```
建议不要在浏览器里直接吞原始 Gaia 大表,而是先离线裁剪成:
- 只保留亮星
- 只保留渲染必需字段
- JSON 或二进制轻量格式
## 资源与数据实施路线
### 路线 A先做可用版本推荐
1. 引入 NASA Tycho 全天图
- 或评估替换为更接近 SatelliteMap.space 路线的 `NASA Deep Star Maps 2020`
2. 实现背景球壳
3. 用 Astronomy Engine 计算太阳/月亮方向
4. 暂不做亮星点
优点:
- 最快见效
- 风险最低
- 就能明显提升天球真实感
### 路线 B在 A 基础上增强
1. 离线生成 `bright-stars.json`
2. 浏览器端渲染亮星点
3. 后续可加:
- 星座线
- 亮星名称
- 特定星体高亮
优点:
- 背景真实感和“位置正确的可交互星体”同时兼顾
## 代码模块建议细化
### 新增模块
- `frontend/public/earth/js/celestial.js`
- 管理天球背景
- 管理太阳/月亮
- 管理亮星层(后续)
- `frontend/public/earth/js/celestial-data.js`
- 资源路径
- 星图方向配置
- 亮星数据加载(后续)
### 建议函数设计
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date)
export function setCelestialVisibility(visible)
export function disposeCelestialLayer()
function loadStarMapTexture()
function createSkySphere(texture)
function createSunSprite()
function createMoonSprite()
function getSunEquatorialPosition(date)
function getMoonEquatorialPosition(date)
function equatorialToWorldVector(raRad, decRad)
```
### 推荐后续预处理脚本
如要引入亮星层,建议单独做离线脚本:
- `scripts/build_bright_stars.py`
职责:
- 从 Hipparcos / Tycho 源数据读取
- 过滤亮星
- 生成 `bright-stars.json`
这样浏览器端只消费轻量结果,不承担大表解析成本。
## 分阶段实施
## Phase 1真实天球背景
### 目标
用真实全天星图替换当前随机星点背景。
### 做法
1. 新增一张全天星图纹理
建议路径:
- `frontend/public/earth/assets/celestial/starmap_equatorial_4k.jpg`
纹理要求:
- 等距矩形投影
- 赤经/赤纬坐标展开
- 无地平线、无地景遮挡
- 尽量深色、弱干扰,适合大屏 HUD 叠加
2. 新增天球球壳
新增模块:
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
建议接口:
```js
export function initCelestialLayer(scene)
export function updateCelestialLayer(date, camera, earth)
export function disposeCelestialLayer()
```
3. 实现一个大半径内翻球体
建议参数:
- 半径:`600 ~ 900`
- 材质:`MeshBasicMaterial`
- `side: THREE.BackSide`
- 不受场景光照影响
- 始终围绕场景中心
### 验收标准
- 初始加载后背景不再是随机星点
- 旋转地球时,背景保持为稳定天球而不是跟地球一起转
- 不明显干扰海缆/卫星/BGP 的前景识别
## Phase 2太阳与月亮真实位置
### 目标
在当前 UTC 时间下,计算太阳与月亮在天球中的方向,并显示出来。
### 做法
1.`celestial.js` 内封装天体位置计算
建议函数:
```js
function getSunDirection(date)
function getMoonDirection(date)
```
输出统一为 world-space `THREE.Vector3`
2. 太阳显示
- 一个暖色发光 sprite
- 比月亮更大、更亮
- 可选添加柔和 halo
3. 月亮显示
- 一个较小 sprite 或 sphere
- 灰白偏冷色
- 后续 Phase 3 再做月相
4. 更新频率
不要每帧重新做完整天文计算,建议:
- 每 30 秒或 60 秒重算一次真实位置
- 渲染帧内做平滑过渡
### 验收标准
- 页面可见太阳与月亮两个对象
- 时间变化时位置会更新
- 日月不会跟随地球局部旋转而错误附着
## Phase 3太阳驱动地球受光
### 目标
让地球光照方向与太阳方向一致,不再使用写死的固定主光。
### 做法
1. 替换或接管当前主定向光
当前 [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js) 中 `addLights()` 里用了固定方向的 `DirectionalLight`
建议改为:
- 保留环境补光
- 主太阳光方向由 `sunDirection` 决定
2. 太阳光参数建议
- `DirectionalLight` 颜色偏暖白
- 强度略高于当前主光
- 保留一个弱背光作为氛围补偿,避免背面过死黑
3. 先不做物理级大气散射
第一版只要求:
- 亮面与暗面方向真实
- 云层和大气仍保持当前风格
### 验收标准
- 地球明暗面会随太阳方向改变
- 太阳 sprite 和地球亮面方向一致
- 不破坏现有海缆、卫星、BGP 的可见性
## Phase 4月相与天文细节增强
### 目标
在日月真实位置基础上增加更强的“天文可信度”。
### 可选项
1. 月相
- 根据日月夹角计算 illuminated fraction
- 用月相纹理或 shader 表达盈亏
2. 赤道/黄道辅助线
- 可作为开发调试层,不默认显示
3. 太阳 terminator 增强
- 给地球夜面加入更自然的 night tint
- 未来可叠加城市夜光纹理
4. 天文时间入口
- 设置中加入“当前时刻 / 指定时刻 / 加速时间”模式
### 验收标准
- 月亮不再只是一个静态圆点
- 后续扩展行星或观测模式时无需推倒重来
## Phase 5严格参考系升级可选不作为 V1 必做)
### 目标
把 Earth 从“用户旋转球体”升级为“真实地球姿态 + 用户观察姿态”的双层模型。
### 需要处理的问题
- 地球自转角与 UTC 的一致性
- 赤道坐标系、地固坐标系、相机交互层分离
- 卫星轨道显示与 Earth 旋转同步关系
- resetView 和 autoRotate 的语义重定
### 风险
这一步会影响:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- [frontend/public/earth/js/satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [frontend/public/earth/js/cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [frontend/public/earth/js/controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
因此不建议与 V1 同时推进。
## 代码改造清单
## 1. 新增文件
- [frontend/public/earth/js/celestial.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/celestial.js)
职责:
- 管理天球背景、太阳、月亮
- 对外暴露 init/update/dispose
## 2. 修改 `constants.js`
文件:
- [frontend/public/earth/js/constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
新增:
```js
export const CELESTIAL_CONFIG = {
sphereRadius: 800,
updateIntervalMs: 60000,
sunSpriteScale: 28,
moonSpriteScale: 16,
sunLightIntensity: 1.25,
ambientIntensity: 0.28,
backLightIntensity: 0.18,
};
```
## 3. 修改 `main.js`
文件:
- [frontend/public/earth/js/main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
主要改动:
1. `init()` 中:
- 初始化 celestial layer
2. `addLights()` 中:
- 把固定太阳光改成可更新的 celestial sun light
3. `animate()` 中:
- 每帧调 `updateCelestialLayer()`
4. `destroy()` 中:
- 清理 celestial 资源
## 4. 修改 `earth.js`
文件:
- [frontend/public/earth/js/earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
主要改动:
- `createStars()` 逐步退役
- 第一阶段可先保留作为 fallback
- 当真实星图加载成功后,不再显示随机星点
## 5. 新增资源
目录建议:
- `frontend/public/earth/assets/celestial/`
建议至少包含:
- `starmap_equatorial_4k.jpg`
- `sun_glow.png`
- `moon_albedo_2k.jpg`
## 数据流设计
```mermaid
flowchart TD
A["main.js:init()"] --> B["initCelestialLayer(scene)"]
B --> C["创建天球球壳"]
B --> D["创建太阳 sprite + 主定向光"]
B --> E["创建月亮 sprite"]
F["animate()"] --> G["updateCelestialLayer(now, camera, earth)"]
G --> H["Astronomy Engine 计算 Sun/Moon 方向"]
H --> I["更新 sun sprite / moon sprite 位置"]
H --> J["更新太阳 DirectionalLight 方向"]
J --> K["地球昼夜方向变化"]
```
## 风险与注意事项
### 1. 星图投影方向容易反
这会表现为:
- 星图左右镜像
- 赤经方向颠倒
- 日月位置和背景对不上
建议:
- 先做一个开发调试模式
- 显示赤经/赤纬参考点,快速校正纹理朝向
### 2. 不要让天球跟随 Earth 旋转
天球背景和日月必须属于 scene/world而不是 `earthObj`
### 3. 不要每帧做重型天文计算
真实位置更新应节流,否则会浪费 CPU。
### 4. 月亮先求“方向正确”,再求“月相精致”
月相属于第二步优化,不应阻塞 V1 上线。
## 推荐实施顺序
1. 新建 `celestial.js`
2. 用星图球壳替换随机星点
3. 接入 Astronomy Engine
4. 加太阳/月亮 sprite
5. 用太阳方向驱动主光
6. 再决定要不要做月相和更严格参考系
## 最终建议
对于当前 Planet Earth最稳妥的方案是
- 先做真实天球背景
- 再做真实太阳/月亮方向
- 再让太阳驱动地球受光
- 暂时不做 Earth 参考系重构
这样可以在不破坏现有 Earth 交互和图层系统的前提下,显著提升空间感、真实感和演示说服力。

View File

@@ -0,0 +1,156 @@
# Earth News Source Configuration And Collector Plan
## Why
当前 Earth 的“态势新闻”由 [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py) 直接在请求时抓取 RSS / Google News feed再按当前地球视角中心区域聚合返回。
这条链已经可用,但存在两个明显限制:
- 新闻源写死在代码里,不能像 TV 直播源一样从后台维护
- 新闻并未进入统一采集体系,没有采集状态、失败监控、历史数据和后续 AI 复用能力
因此这块更合理的路线不是一步到位重写,而是分阶段推进:
1. 先做“新闻源配置化”
2. 再做“新闻采集器化”
## Current State
当前实现分布在:
- 新闻接口
- [news.py](/home/ray/dev/linkong/planet/backend/app/api/v1/news.py)
- 实时聚合逻辑
- [earth_news.py](/home/ray/dev/linkong/planet/backend/app/services/earth_news.py)
- 前端消费
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
当前新闻源包含:
- `BBC World` RSS
- `DW Top Stories` RSS
- 按区域关键词拼出来的 `Google News RSS`
- `Global`
- `Americas`
- `Europe`
- `Middle East / Africa`
- `Asia Pacific`
当前不是采集器,也不落库,只做内存缓存。
## Phase 1: Source Configuration
### Goal
`NEWS_FEED_SOURCES` 从硬编码列表升级成可配置新闻源目录,但继续保留当前“实时聚合”的工作方式。
### Scope
- 为 Earth news 建立独立配置结构
- 支持后台维护 feed 源
- 支持启用/禁用、优先级、区域、源类型
- 保持现有 `/api/v1/news/earth-feed` 输出协议不变
### Proposed Shape
建议配置字段至少包括:
- `id`
- `name`
- `region`
- `feed_url`
- `homepage_url`
- `source_type`
- `priority`
- `is_enabled`
- 可选 `query_profile`
- 可选 `language`
- 可选 `notes`
### Suggested Storage
优先走系统设置或单独的 news source settings payload而不是先建复杂新表。
推荐原因:
- 改动小
- 易上线
- 和当前 TV settings 维护体验更接近
- 先解决“写死在代码里”的问题
### Non-goals
这一阶段不做:
- 新闻入库
- 新闻历史回看
- 新闻采集任务监控
- 新闻去重流水线
## Phase 2: News Collectorization
### Goal
把“态势新闻”升级为真正的采集器链路,使其进入采集系统和数据层。
### Scope
- 新增专用 news collector
- 按配置源定时采集 RSS / feed
- 做标题/链接级去重
- 建立统一新闻记录模型
- 为 Earth、控制台、AI 研判复用同一份新闻数据
### Benefits
- 有采集状态
- 有失败监控
- 有历史缓存
- 可以做时间轴 / 区域新闻基线
- 可以作为 AI 引用证据
### Required Design Work
需要提前明确:
- 新闻数据模型
- 去重策略
- 过期清理策略
- 区域映射策略
- 聚合排序策略
- 新闻与 Earth 当前视角/区域的关联方式
### Candidate Output Model
至少应包含:
- `source_id`
- `headline`
- `summary`
- `url`
- `publisher`
- `region`
- `published_at`
- `language`
- `tags`
- `raw_feed_source`
- `reference_date`
## Recommended Order
推荐执行顺序:
1. 先完成 Phase 1 配置化
2. 保持 Earth 继续实时聚合,但改为读取配置源
3. 等新闻源稳定后,再设计 Phase 2 的 collector / storage / dedupe
## Decision
当前结论:
- TV 直播源:优先采集器化
- 态势新闻:优先配置化,再采集器化
## Source Note
This plan is newly created for the Planet repo to separate the short-term "configurable source directory" work from the longer-term "collectorized news pipeline" work.

View File

@@ -0,0 +1,98 @@
# Earth Predicted Orbit Plan
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/predicted-orbit.md`.
## Goal
在 Earth 中锁定卫星时,显示“预测轨道”而不是只有历史尾迹:
- 从当前时刻开始
- 绕地球一圈
- 当前点最亮
- 向后沿轨道逐步衰减
## Current State
当前已经有:
- 卫星历史轨迹
- 锁定卫星
- 轨道高亮与相关联动
但“预测轨道”仍然不是一套稳定、可验证的单独功能计划。
## Why It Is Valuable
预测轨道可以明显提升:
- 锁定卫星后的空间可读性
- 轨道类型辨识
- 演示解释力
相比短历史尾迹,预测轨道更符合用户对“这颗卫星接下来会怎么走”的预期。
## Scope
### Phase 1
- 锁定卫星时显示一整圈预测轨道
- 解锁时隐藏
- 不替代现有普通轨迹系统
### Phase 2
- 根据轨道类型调整采样率
- GEO / MEO / LEO 不同密度
- 进一步减少 fallback 轨迹的比例
## Implementation Direction
### 1. Orbit period
基于 `meanMotion` 估算轨道周期。
### 2. Predicted samples
以固定采样步长从 `now -> now + period` 推算轨迹点。
### 3. Render object lifecycle
预测轨道应是一个独立渲染对象:
- show
- update
- hide
- dispose
### 4. Visual semantics
预测轨道不应与普通尾迹混淆:
- 更稳定
- 更完整
- 透明度沿轨道衰减
- 当前点附近更亮
## Known Risks
### 1. TLE propagation gaps
部分卫星可能出现 SGP4 计算不足,需要 fallback。
### 2. Multiple orbit lines
必须确保:
- 锁定切换前先清旧轨道
- 页面隐藏/销毁时清理
### 3. Performance
GEO 轨道点数高,采样率需要按轨道类型分层。
## Acceptance
1. 锁定单颗卫星时只显示一条预测轨道
2. 解锁后轨道立即清除
3. 不同轨道类型下点数可控
4. 页面切换回来不会闪出旧轨道残留

View File

@@ -0,0 +1,472 @@
# Earth Real Terrain Plan
## Goal
将 Earth 页当前的“程序噪声假地形”替换成基于真实 DEM 的可用地形层,使 `地形 terrain` 开关真正显示全球海拔起伏,而不是占位效果。
当前占位实现位于:
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
具体问题:
- `createTerrain()` 直接对球体顶点应用 `simplex noise`
- 没有真实海拔数据来源
- 没有分辨率分层
- 没有和当前相机/视角配套的性能控制
## Constraints
本计划必须贴合当前 Earth 架构,而不是引入一套全新的地形引擎:
- 地球主体仍然是一个 Three.js sphere
- 海缆、登陆点、卫星、BGP 都已经建立在当前球体坐标系之上
- 不能为了地形把整页改成 Cesium/MapLibre Globe 之类的全栈替换
- 第一阶段优先做“真实可用”,不是一步到位做摄影测量级地形
## Recommended Data Source
### Primary recommendation
使用公开的 Terrarium 编码高程瓦片作为浏览器端高度来源,第一阶段优先接入:
- Mapzen/AWS `Terrarium` elevation tiles
参考:[Mapzen terrain tile format / Terrarium](https://www.mapzen.com/blog/terrain-tile-service/)
原因:
- 已经是全球瓦片化高程
- 浏览器端按 tile 请求,最适合当前 Earth 这种在线 globe
- 编码简单稳定:
- `heightMeters = (R * 256 + G + B / 256) - 32768`
- 不需要我们先离线拼整球 DEM
### Data quality upgrade path
如果后面第一阶段效果确认可用,再逐步升级到底层源:
- Copernicus DEM GLO-30
参考:[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
- 或用 Copernicus / SRTM / ASTER 等离线切成我们自己的 terrain tiles
这条升级路径适合第二阶段,不建议一开始就直接自建全球瓦片服务。
## Why Not Replace the Engine
不建议为了地形直接切到 Cesium terrain / quantized mesh 引擎,原因:
- 现有 Earth 业务对象都依附当前球面坐标
- 切引擎会同时波及:
- 海缆绘制
- 卫星/轨迹
- BGP 标记
- HUD 与交互
- 这是“重做一页”,不是“给地形层接真实数据”
所以推荐路线是:
- 保持当前 sphere globe
- 为 sphere 增加真实高度位移层
## Implementation Strategy
分三期推进。
### Phase 1 — Global Heightmap Terrain Overlay
目标:
- 地形层切换后显示真实海拔起伏
- 全球范围可用
- 性能可控
做法:
1. 新增 terrain 数据模块
建议文件:
- `frontend/public/earth/js/terrain.js`
职责:
- 选择 DEM zoom level
- 请求 Terrarium tiles
- 解码 tile 高程
- 将高程重采样到当前地形球体网格
2. 替换 `createTerrain()`
当前:
- 在 [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js) 中同步生成噪声地形
调整后:
- `createTerrain()` 只负责创建 terrain mesh 骨架
- 真正的顶点位移由 terrain 模块异步注入
3. 第一阶段采用“整球低分辨率位移”
不要一上来做动态 patch stitching。第一阶段更稳的办法是
- 保留一张全球 terrain sphere
- 使用较低分辨率几何
- 例如 `SphereGeometry(radius, 192, 192)``256/256`
- 运行时按一个固定地形 zoom`z=4``z=5`)抓取覆盖全球的 Terrarium tiles
- 将 tile 解码后重投影到经纬度采样网格
- 将每个球面顶点按真实高度抬升
这样第一阶段就能做到:
- 有真实地形
- 不需要复杂的局部 LOD
- 不会让现有球体对象体系爆炸
### Phase 2 — View-Aware Refinement
目标:
- 正面可见区域更精细
- 背面与远处维持低成本
做法:
- 引入“基础全球地形 + 当前视角高分局部补丁”
- 正面区域额外抓更高 zoom 的高程 tile
- 只替换局部顶点位移或局部 overlay mesh
这一阶段适合在第一阶段稳定后做。
### Phase 3 — Normals / Shading / Terrain UX
目标:
- 地形不仅有起伏,还更好看、更可读
包括:
- 根据高度生成更合理的 normals
- 调整 terrain material使山脉/高原更易读
- 可选加入:
- hillshade
- contour lines
- snowline / bathymetry tint
## Calibration Overlay Before More Terrain Tuning
在当前项目里terrain 看起来“不像真地形”,不一定只是 DEM 或 exaggeration 不够,也可能是因为缺少稳定参照物。
没有清晰的海岸线、国界线和地表分层时,人眼很难判断:
- 山脉是不是在应该高的地方高
- terrain 是否真的贴在正确的大陆位置上
- 地球纹理、本初子午线、terrain 采样之间是否存在偏移
这里要明确区分两件事:
- 国界线不会修好错误的 terrain
- 但海岸线 / 国界线会让我们更容易判断 terrain 有没有贴准
所以在继续盲调 terrain 参数之前,建议先插入一个“校准参照层”阶段。
### Recommended order for the calibration layer
1. 海岸线
2. 国界线
3. 再继续调 terrain
原因:
- 海岸线比国界线更基础,也更接近真实地表边界
- 判断 terrain 是否贴准,最重要的是大陆边缘和山脉/海岸关系
- 国界线更多是政治边界,只能作为辅助参照
如果只加国界线,不加海岸线,效果仍然可能会怪,因为:
- 很多国界线本来就是人为直线
- 它们并不总是跟真实地形走
### Suggested layer order during debugging
建议调试期临时把地球层次明确成:
1. base earth texture
2. coastline / borders overlay
3. terrain relief
4. cables / landing points / bgp / satellites
这样会比现在更容易判断:
- 山脉是否位于正确区域
- terrain 是否和地表对齐
- 国界/海岸是否漂移
### Suggested data source for the calibration overlay
优先用 `Natural Earth` 的轻量全球矢量数据:
- 海岸线coastline
- Admin 0 国界线country borders
优点:
- 全球一致
- 轻量
- 很适合当前 Three.js globe 做 overlay
### Recommended execution path
#### Phase A — Add reference overlays
先加两层可开关的参考线:
- 海岸线
- 国界线
这两层的目标不是最终美术表现,而是调试 / 校准。
#### Phase B — Recalibrate terrain against coastline
有了海岸线以后,再重新看 terrain
- terrain 是否和大陆边缘错位
- 地球纹理、本初子午线、terrain 采样之间是否有固定偏移
#### Phase C — Decide whether to keep the current terrain path
这时再决定后面的路线:
- 如果发现真实高程整体是对的,只是缺少 shading / readability
继续保留当前 DEM + terrain overlay 路线
- 如果发现整球采样投影、本初子午线或 overlay 关系本身就很别扭
再考虑重做 terrain pipeline
### Practical recommendation
当前阶段不建议“从头开始重做 terrain”。
更稳的策略是:
- 暂停继续盲调 terrain 参数
- 先补海岸线 / 国界线作为校准参照层
- 再基于参照层判断 terrain 是“参数没调好”,还是“整条实现路径有偏移”
## Recommended Geometry Model
### First usable model
保留一层独立 terrain sphere
- base earth sphere贴纹理、昼夜、海洋
- terrain sphere略高于地球半径真实高程位移
建议:
- `terrainBaseRadius = CONFIG.earthRadius + 0.2`
- 高度缩放使用真实米制换算,再乘一个可调 exaggeration
示例关系:
- `heightWorld = (elevationMeters / 6371000) * CONFIG.earthRadius * exaggeration`
建议第一阶段 `exaggeration = 1.3 ~ 1.8`
因为完全真实比例在全球球体上会太平,看不出来。
## Tile Decoding Plan
### Terrarium decode
对于每个高程 tile 像素:
```text
heightMeters = (R * 256 + G + B / 256) - 32768
```
### Sampling path
对于 terrain mesh 上每个顶点:
1. 将顶点方向转成经纬度
2. 将经纬度映射到 Web Mercator tile 坐标
3. 找到对应的 tile 和像素
4. 解码高程
5. 将顶点沿法线方向抬升
### Needed helpers
建议新增:
- `latLonToTileXY(lat, lon, z)`
- `tilePixelFromLatLon(lat, lon, z, tileSize)`
- `decodeTerrariumHeight(r, g, b)`
## Caching Strategy
为了不让地形开关每次重开都重新抓全量 tile
- terrain tile 按 `z/x/y` 存到内存缓存
- terrain mesh 结果也缓存一份
- 当用户关闭/开启 terrain
- 直接复用已有位移结果
建议:
- `Map<string, Float32Array | ImageBitmap>`
## Material Strategy
第一阶段不要复杂化。
建议 terrain material
- 半透明低饱和地形色
- 比 base earth 稍亮或稍偏冷
- 保留当前 HUD 风格下的可读性
第一阶段不需要:
- 真实土地覆被纹理
- 独立卫星影像贴 terrain
因为那会和现有地球纹理、云层、昼夜 shader 打架。
## Integration Points
### Files to change
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
- 重写 `createTerrain()`
- 删除 simplex noise 占位逻辑
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
- 初始化 terrain 数据加载
- 控制 terrain readiness / loading message
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- `toggleTerrain` 逻辑保持,但应能区分:
- mesh 已就绪
- 正在加载
- [constants.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/constants.js)
- 新增 `TERRAIN_CONFIG`
- 新文件:
- `frontend/public/earth/js/terrain.js`
### Suggested new config
建议新增:
```js
export const TERRAIN_CONFIG = {
enabled: true,
tileSize: 256,
baseZoom: 4,
baseRadiusOffset: 0.2,
exaggeration: 1.5,
opacity: 0.55,
color: 0x6c876f,
maxConcurrentRequests: 8,
cacheEnabled: true,
};
```
## Loading UX
地形第一次开启时,不能像现在一样瞬时切换。
建议:
- 如果地形数据尚未准备:
- 顶部状态条显示:`正在加载真实地形数据...`
- 完成后:
- `真实地形已就绪`
如果加载失败:
- 保留 base earth
- 显示轻量错误提示
- 不要让 terrain 开关卡死在“开”状态
## Risks
### 1. Global tile count too high
即使 `z=5` 全球 tile 数也不少。
缓解:
- 第一阶段限定低 zoom
- 并发上限
- 缓存
### 2. Mesh resolution too low
如果球面分段太低,山脉会被抹平。
缓解:
- 第一阶段先选一个中等分辨率
- 用 exaggeration 保证可见性
### 3. Existing overlays may z-fight with terrain
海缆、登陆点、BGP、卫星相关对象都假设地球半径固定。
缓解:
- terrain sphere 单独作为 overlay
- overlay 保持略低或略高的固定 offset
- 必要时局部调整 landing point / cable altitude offset
### 4. Mercator sampling distortion near poles
Web Mercator 在高纬会有失真。
缓解:
- 第一阶段接受
- 后续若需要更严格极区质量,再上 geodetic reprojection pipeline
## Acceptance Criteria
第一阶段完成后,应满足:
1. `地形 terrain` 开关开启时,地表起伏明显不再是随机噪声
2. 喜马拉雅、安第斯、落基山、东非高原等全球大尺度地形可辨认
3. 关闭/重新开启 terrain 不重复全量请求
4. 不破坏:
- 海缆
- 卫星
- BGP
- 地球昼夜
- 天球层
## Suggested Execution Order
1. 引入 `TERRAIN_CONFIG`
2. 新建 `terrain.js`
3. 实现 Terrarium tile 请求与 decode
4. 用低 zoom 全球 tile 构建真实 terrain sphere
5. 接管 `toggleTerrain()`
6. 调整 terrain material 和高度 exaggeration
7. 做缓存
8. 再考虑第二阶段局部高分 refinement
## Source References
- Mapzen Terrarium / AWS terrain tiles
[Mapzen Terrain Tile Service](https://www.mapzen.com/blog/terrain-tile-service/)
- Terrarium tile experiments / format background
[mapzen/terrarium](https://github.com/mapzen/terrarium)
- Copernicus DEM overview
[Copernicus DEM docs](https://documentation.dataspace.copernicus.eu/APIs/SentinelHub/Data/DEM.html)
## Recommendation Summary
如果现在就要开始做,我建议直接按这条路线开工:
- 第一阶段接入 Terrarium 全球高程 tile
- 替换掉当前 simplex 假地形
- 先做一层真实可见的全球 terrain overlay
- 等第一阶段稳定,再做视角高分 refinement
这是对当前项目风险最低、最贴合现有 Earth 架构的一条路。

View File

@@ -0,0 +1,111 @@
# Earth Renderer / Logic Separation Plan
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/earth-architecture-refactor.md`.
## Goal
将 Earth 前端继续往“逻辑层 / 状态层 / 渲染层”分离推进,降低后续这几类工作的耦合成本:
- Three.js 渲染重构
- 部分图层替换实现
- 未来 UE / Cesium 客户端迁移
- Earth 行为逻辑复用
## Why This Matters
当前 Earth 已经有一些良好分层,例如:
- 图层显隐入口
- Cable state 枚举与状态 map
- 交互逻辑与实际视觉效果的部分分离
但还没有形成一套更明确的统一规则。现在的风险是:
- 同一类对象的 hover / locked / hidden / loading 语义不一致
- 状态和渲染更新散落在多个模块
- 后续再加新图层时容易复制旧逻辑
## Target Architecture
Earth 对每类对象都尽量拆成三层:
1. `state layer`
- 保存对象状态
- 例如:`normal / hovered / locked / hidden / loading`
2. `logic layer`
- 处理点击、悬停、锁定、过滤、显隐切换
- 不直接关心 Three.js 具体材质怎么改
3. `renderer layer`
- 根据状态更新 Three.js / HUD 外观
- 是最容易针对不同渲染引擎替换的一层
## Current Good Signals
当前已经接近这条方向的地方:
- cable 状态管理
- 部分 landing point 状态同步
- layer button 的统一状态入口
- tooltip / legend / info-card 开始朝状态驱动靠拢
## Next Steps
### 1. Standardize object state enums
优先为这些对象建立更稳定的状态语义:
- cables
- satellites
- landing points
- BGP markers
- media / news 面板入口按钮
### 2. Unify state-to-visual adapters
为各模块建立更清晰的渲染适配函数,例如:
- `applyCableVisualState()`
- `applySatelliteVisualState()`
- `applyBGPVisualState()`
要求:
- 逻辑层只改状态
- 视觉层负责把状态映射到材质、透明度、发光、尺寸、文字
### 3. Separate Earth UI state from render state
HUD / 面板 / 图层按钮状态也需要和渲染状态分离:
- `loading`
- `active`
- `locked`
- `hidden`
- `error`
不要再让 UI 通过“猜渲染结果”推导业务状态。
### 4. Prepare migration-safe boundaries
后续如果做 UE / Cesium 客户端,尽量保留:
- 状态枚举
- 交互规则
- 数据层接口
只替换:
- Three.js 具体渲染实现
- HUD 展示实现
## Practical Rule
后续 Earth 新功能开发时,优先问三个问题:
1. 这个状态由谁持有?
2. 这个交互逻辑在哪一层处理?
3. 这个视觉变化是否能在不改逻辑的情况下单独替换?
如果答不上来,就说明还在把状态、逻辑、渲染揉在一起。

View File

@@ -0,0 +1,82 @@
# Earth WebGL Instancing Satellites Plan
> Source note: this plan absorbs useful ideas from a sisyphus-created draft formerly stored at `.sisyphus/plans/webgl-instancing-satellites.md`.
## Goal
把 Earth 卫星渲染从当前方案继续推进到更适合高数量卫星的 instancing 方向,目标是:
- 支持更多卫星
- 降低渲染压力
- 仍然保留当前数据层和交互层
## Why It Matters
当前卫星系统已经具备:
- 数据加载
- 轨迹
- 选择/锁定
- 图例
- 相关区域联动
但当卫星数量持续增加时,渲染层会越来越接近瓶颈。
## Recommended Direction
优先调研并原型验证:
- `InstancedBufferGeometry + custom shader`
而不是一开始就推倒重写成 raw WebGL。
原因:
- 仍能保留 Three.js 主架构
- 更容易渐进迁移
- 比继续堆普通点渲染更有上限
## What Should Stay
尽量保留这些层:
- 卫星数据获取
- 位置计算
- 锁定/悬停逻辑
- legend / info-card / 相关联动
主要替换的是:
- 卫星点渲染实现
- 颜色/大小等实例属性更新方式
## Phases
### Phase 1: Prototype
- 用 instancing 做最小原型
- 先只渲染卫星点
- 不碰轨迹系统
### Phase 2: Integrate
- 接入当前 `satellites.js` 数据层
- 保留当前选择和高亮语义
### Phase 3: Tune
- 调整可视大小
- 调整选中高亮方式
- 评估是否需要分层 LOD
## Risks
1. 透明度排序更复杂
2. Shader 调试成本更高
3. 选中态和 hover 态不能简单复用旧材质逻辑
## Acceptance
1. 在更高卫星数量下保持可接受帧率
2. 不破坏现有锁定/高亮语义
3. 图例、信息卡、相关卫星联动仍然成立

View File

@@ -0,0 +1,361 @@
# AI Playground Development Plan
## 目标
这份计划用于统一 `aiprovider``backend AI facade``Playground` 页面,以及后续 `BGP / 告警 / 数据源健康` 等 AI 入口的演进方向。
当前原则:
- `aiprovider` 继续作为独立模型网关
- `backend` 继续作为稳定业务入口
- `frontend` 负责测试台和业务 UI
- 先做“可控、可验证、可解释”的 AI 能力,再逐步引入 agent/tool calling
## 当前已完成
### 1. AI 网关基础层
已完成:
- 独立 `aiprovider` 服务
- `backend -> aiprovider -> model provider` 调用链
- `provider/status``situational-awareness/analyze` 稳定接口
- `X-Request-ID` 透传
- 轻量超时与重试
- MiniMax / Anthropic-compatible / OpenAI-compatible / Ollama 适配
相关文件:
- [backend/app/api/v1/ai.py](/home/ray/dev/linkong/planet/backend/app/api/v1/ai.py)
- [backend/app/services/ai_client.py](/home/ray/dev/linkong/planet/backend/app/services/ai_client.py)
- [aiprovider/main.py](/home/ray/dev/linkong/planet/aiprovider/main.py)
- [aiprovider/provider_service.py](/home/ray/dev/linkong/planet/aiprovider/provider_service.py)
- [docs/technical/agents-aiprovider.md](/home/ray/dev/linkong/planet/docs/technical/agents-aiprovider.md)
### 2. 本地运行与配置打通
已完成:
- `planet.sh` 启动链路纳入 `aiprovider`
- `planet.sh` 启动完成后输出 Playground 入口
- `docker-compose.yml``aiprovider` 加入 `env_file`
- `backend/.env``aiprovider/.env` 两侧 service token 对齐
- `Playground` 状态缓存,避免页面切换时每次都重新请求 provider 状态
相关文件:
- [planet.sh](/home/ray/dev/linkong/planet/planet.sh)
- [docker-compose.yml](/home/ray/dev/linkong/planet/docker-compose.yml)
- [backend/.env.example](/home/ray/dev/linkong/planet/backend/.env.example)
- [aiprovider/.env.example](/home/ray/dev/linkong/planet/aiprovider/.env.example)
### 3. Playground UI 基础版
已完成:
- 新增前端路由 `/playground`
- 左侧 `Provider 状态 + 测试说明`
- 右侧 `请求 / 结果` Tabs
- `Provider 状态` 支持手动刷新
- `测试说明` 支持折叠
- 内部区域采用细滚动条
- 页面布局开始遵循“单屏工作区 + 模块内部滚动”规范
相关文件:
- [frontend/src/pages/Playground/Playground.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Playground/Playground.tsx)
- [frontend/src/App.tsx](/home/ray/dev/linkong/planet/frontend/src/App.tsx)
- [frontend/src/components/AppLayout/AppLayout.tsx](/home/ray/dev/linkong/planet/frontend/src/components/AppLayout/AppLayout.tsx)
- [frontend/src/index.css](/home/ray/dev/linkong/planet/frontend/src/index.css)
### 4. 前端布局规范沉淀
已完成:
- 把“一屏工作区、主模块优先、模块内部滚动”的规范文档化
- 明确 `BGP` 页面为当前参考实现
相关文件:
- [docs/technical/frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
- [frontend/src/pages/BGP/BGP.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/BGP/BGP.tsx)
## 当前限制
### 1. Playground 还是 prompt playground不是 agent playground
当前 `Playground``观察项 / 目标 / 约束条件` 都是人工输入。
模型现在拿到的是:
- 你手工输入的结构化字段
- 后端传递的少量静态上下文
模型现在拿不到:
- 实时 BGP 事件
- 真实告警列表
- 数据源健康状态
- 自动检索结果
- tool calling / skills / 自主取数
### 2. `situational-awareness/analyze` 还是通用提示词接口
当前更适合:
- 测试链路
- 测试模型输出风格
- 验证不同 provider 是否正常返回
当前还不适合:
- 直接当真实态势系统主入口
- 让用户手工维护长期分析模板
- 代替专用业务研判接口
### 3. 还没有可验证的真实业务输入注入
目前最缺的是:
- 从业务系统自动整理“事实输入”
- 再把这些事实喂给 AI
而不是继续让用户在 Playground 手工输入真实事件摘要。
## 短期计划
### Phase A: Playground 收敛为稳定测试台
目标:
- 保持 Playground 简洁可用
- 不再继续堆“高级参数”
工作项:
- 继续微调左侧 `Provider 状态``测试说明` 的空间策略
- 保持 `请求 / 结果` 为单一主工作区
- 不引入盲填式高级字段
- 统一滚动条、卡片、溢出行为
完成标准:
- 笔记本视口下依然可用
- 各模块标题可见
- 主要阅读区始终是右侧 Tabs
### Phase B: BGP AI 简报
目标:
- 不再依赖手工填写“观察项”
- 让系统自动把真实 BGP 数据注入 AI
- 让 BGP 页面逐步从“摘要汇总”升级为“证据驱动的区域态势分析”
建议实现:
- 新增专用后端接口,例如:
- `POST /api/v1/ai/bgp/brief`
- 后端自动读取:
- incidents summary
- anomalies
- recent events
- collector coverage summary
- 后端将结构化事实注入 `context / observations`
- 前端在 BGP 页面增加“生成 AI 简报”
当前阶段说明:
- 第一版 `BGP AI 简报` 允许先落地为“值班摘要生成器”
- 也就是先把 incidents / anomalies / events / collector coverage 自动注入
- 允许模型先做事实摘要、风险归纳、建议动作
但这不应被视为 Phase B 的最终形态。
Phase B 后续还需要补齐:
- prefix geography 证据注入
- `iptoasn`
- `opengeofeed`
- `nro_delegated`
- 基于 `affected_regions` 与 prefix geography 的区域聚合
- 区分“真实区域热度”与“collector coverage 偏差”
- 对高风险 prefix / ASN 给出更明确的国家、城市、运营商归属线索
- 让 AI 输出明确回答:
- 哪些区域正在异常升温
- 哪些结论只是观测站偏差
- 当前还缺哪些区域证据
完成标准:
- 用户不需要手工录入 BGP 观察项
- AI 输出能明确区分“事实”和“研判”
- AI 不只是复述总量和最近几条事件,还能利用 prefix geography 与 affected regions 做区域态势判断
- 输出中能明确指出:
- 高风险区域
- 区域证据来源
- collector coverage 偏差对判断的影响
### Phase C: 告警 / 数据源健康 AI 简报
目标:
- 复用同样模式,扩展到其他模块
建议入口:
- `Alerts` 页面:异常与告警摘要
- `DataSources` 页面:采集失败与健康状态总结
原则:
- 每个业务页优先做“专用 AI 简报”
- 不优先做“万能大聊天框”
## 中期计划
### 1. Assessment Layer
目标:
- 不只返回自由文本
- 返回结构化的 assessment
建议输出字段:
- summary
- key_risks
- evidence
- recommendations
- confidence
- missing_data
这样后续才能:
- 持久化
- 回看
- 对比不同时间的 AI 结论
- 在 Earth / Dashboard / BGP 页面稳定展示
### 2. Evidence-first Runtime
目标:
- 所有 AI 分析先取真实数据,再调模型
原则:
- 先 evidence
- 再 prompt
- 最后才是自由生成
优先要做的不是更强聊天,而是:
- 更稳定的数据注入
- 更一致的事实模板
- 更清晰的结果结构
### 3. 按页面提供专用入口
目标:
- 让 AI 成为业务视图的一部分,而不是孤立 playground
优先顺序建议:
1. `BGP` AI 简报
2. `Alerts` AI 简报
3. `DataSources` 健康研判
4. `Dashboard` 总览总结
## 长期计划
### 1. Tool Calling / Agent Runtime
只有在以下基础稳定后再推进:
- 数据源健康信号稳定
- BGP / Alerts / Datasource evidence 注入稳定
- assessment 结构稳定
长期可做能力:
- AI 调用受控工具查询业务数据
- AI 调用检索/web search 做外部验证
- AI 生成建议而不是直接修改系统
- 审核后触发受控动作
### 2. 受控动作与闭环
潜在方向:
- 根据健康异常生成修复建议
- 根据态势变化生成处理建议
- 进入 review queue
- 审批后执行
- 验证结果并形成闭环
### 3. 多模块统一 AI 体验
长期目标不是一个孤立 Playground而是
- 每个业务页都有自己的 AI 入口
- 共享统一的 backend AI facade
- 共享统一的 assessment 结构
- 共享统一的 evidence 注入与审计链路
## 设计决策总结
### 为什么保留 `aiprovider`
因为它已经很好地承担了:
- provider 适配
- 协议兼容
- service token 边界
- 独立重启与部署
因此短期内不建议把它并回 `backend`
### 为什么 Playground 不做成万能聊天页
因为当前更需要的是:
- 稳定测试链路
- 可验证业务输入
- 专用分析入口
而不是一个泛化但没有真实数据支撑的聊天框。
### 为什么优先做专用 AI 简报
因为:
- 数据可以自动注入
- 用户心智更清晰
- 输出更容易结构化
- 更容易校验事实与研判是否一致
## 下一步建议
按优先级建议接下来这样做:
1. 稳住 `Playground` 当前布局,不再大幅重做
2.`BGP` 页面新增专用 “AI 简报” 入口
3. 后端新增 `BGP brief` 专用接口,自动注入真实数据
4. 补齐 `BGP brief` 的区域态势证据层
5. 把 AI 输出逐步从自由文本升级为结构化 assessment
### BGP Brief 后续子项
为避免把“已有 AI 简报”误判成“区域分析已完成”,这里单独记录 `BGP brief` 的后续 backlog
1. 把高风险 prefix 命中的 `iptoasn / opengeofeed / nro_delegated` 结果注入 brief context
2. 按国家/城市聚合 active incidents、anomalies、affected prefixes生成区域热点事实层
3. 把 collector coverage 与区域热点并排注入,避免模型把观测偏差误判成区域风险
4. 对高风险 ASN / prefix 追加归属线索,如国家、城市、可能运营商或注册区域
5. 在输出结构中单独增加:
- 区域态势
- 证据来源
- 观测偏差说明
- 缺失区域证据

File diff suppressed because it is too large Load Diff

26
docs/technical/README.md Normal file
View File

@@ -0,0 +1,26 @@
# Technical Docs
这里放“当前实现和当前结构”的文档,重点回答:
- 现在代码是怎么组织的
- 当前入口在哪
- 状态和组件如何工作
- 后续改动应该沿着哪条实现边界继续走
适合放入这里的内容:
- 前端上下文
- Earth 前端结构
- 后端运行控制
- collector 现状
- 采集格式约定
不适合放入这里的内容:
- 尚未完成的 roadmap
- 未来迭代方案
- 大范围重构计划
这些应放入:
- [docs/plans/README.md](/home/ray/dev/linkong/planet/docs/plans/README.md)

View File

@@ -31,22 +31,40 @@ The recommended default is:
- timeout and lightweight retry
- request tracing via `X-Request-ID`
This now follows an OpenClaw-like seam:
- `AI_PROVIDER` identifies the vendor or logical provider
- `AI_PROVIDER_API` identifies the wire adapter
That split makes MiniMax, Claude-compatible gateways, and self-hosted OpenAI-compatible services easier to model without overloading one config field.
## Supported Providers
`aiprovider` currently supports:
`aiprovider` currently supports these provider identities:
- `openai`
- `openai_compatible`
- `anthropic`
- `minimax`
- `ollama`
Supported request adapters:
- `openai-completions`
- `anthropic-messages`
- `ollama-generate`
Backward-compatible aliases still accepted:
- `openai_compatible`
- `anthropic_compatible`
- `claude_compatible`
- `ollama`
Provider mapping:
- `vLLM`, `LM Studio`, `One API`: `openai_compatible`
- `MiniMax`, Claude-compatible gateways: `claude_compatible`
- `Ollama`: `ollama`
- `vLLM`, `LM Studio`, `One API`: `AI_PROVIDER=openai`, `AI_PROVIDER_API=openai-completions`
- `MiniMax`: `AI_PROVIDER=minimax`, `AI_PROVIDER_API=anthropic-messages`
- Claude-compatible gateways: `AI_PROVIDER=anthropic`, `AI_PROVIDER_API=anthropic-messages`
- `Ollama`: `AI_PROVIDER=ollama`, `AI_PROVIDER_API=ollama-generate`
## API Surfaces
@@ -137,9 +155,13 @@ Both backend and `aiprovider` return the same payload shape:
```json
{
"provider": "openai_compatible",
"model": "gpt-4o-mini",
"provider": "minimax",
"api": "anthropic-messages",
"model": "MiniMax-M2.7",
"content": "1) 态势摘要 ...",
"content_blocks": [],
"text_blocks": [],
"thinking_blocks": [],
"raw_response": {}
}
```
@@ -189,17 +211,37 @@ AI_ANALYSIS_SYSTEM_PROMPT=你是态势感知分析助手。请基于输入的上
### OpenAI-compatible example
```env
AI_PROVIDER=openai_compatible
AI_PROVIDER=openai
AI_PROVIDER_API=openai-completions
AI_BASE_URL=http://127.0.0.1:8001/v1
AI_API_KEY=local-key
AI_MODEL=your-local-model
```
### Claude-compatible example
### MiniMax CN example
```env
AI_PROVIDER=claude_compatible
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com
AI_PROVIDER=minimax
AI_PROVIDER_API=anthropic-messages
AI_BASE_URL=https://api.minimaxi.com/anthropic
AI_API_KEY=sk-cp-xxxxx
AI_MODEL=MiniMax-M2.7
AI_MAX_TOKENS=1200
AI_ANTHROPIC_VERSION=2023-06-01
```
MiniMax note:
- This follows the same Anthropic Messages request shape as the official MiniMax examples.
- For MiniMax, `aiprovider` now disables `thinking` by default unless the caller explicitly passes a `thinking` object.
- This mirrors OpenClaw's caution around MiniMax Anthropic-compatible behavior.
### Anthropic-compatible example
```env
AI_PROVIDER=anthropic
AI_PROVIDER_API=anthropic-messages
AI_BASE_URL=https://your-claude-compatible-endpoint.example.com/anthropic
AI_API_KEY=your_api_key
AI_MODEL=your-model
AI_MAX_TOKENS=1200
@@ -210,6 +252,7 @@ AI_ANTHROPIC_VERSION=2023-06-01
```env
AI_PROVIDER=ollama
AI_PROVIDER_API=ollama-generate
AI_BASE_URL=http://127.0.0.1:11434
AI_API_KEY=
AI_MODEL=qwen2.5:7b

View File

@@ -187,7 +187,7 @@ Current reality:
- that is expected, because incidents are aggregated and de-noised
- but incident-first rendering makes the Earth view look too quiet unless there is another always-available activity layer
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/bgp-region-aggregation-plan.md).
Implementation detail for the recommended `activity layer` is expanded in [bgp-region-aggregation-plan.md](/home/ray/dev/linkong/planet/docs/plans/earth-bgp-region-aggregation-plan.md).
So the immediate next milestone is:

View File

@@ -0,0 +1,381 @@
# Earth Frontend Context
本文件描述当前 Earth 大屏前端的真实结构,重点是帮助后续继续改 HUD、图层、媒体面板、真实地形、BGP 可视化时,不再重复踩结构和状态同步上的坑。
相关规则建议一起参考:
- [rules.md](/home/ray/dev/linkong/planet/rules.md)
- [frontend-layout-guidelines.md](/home/ray/dev/linkong/planet/docs/technical/frontend-layout-guidelines.md)
## 当前目标
Earth 前端不是普通管理页,它是独立的大屏展示前端。当前产品目标是:
- 维持地球视图的空间感和可读性
- 让 HUD、图层、媒体面板、BGP、卫星、海缆等保持统一交互
- 把加载中、已启用、已隐藏、锁定中这类状态做清楚
## 当前入口
React 路由入口:
- [Earth.tsx](/home/ray/dev/linkong/planet/frontend/src/pages/Earth/Earth.tsx)
当前做法很简单:
- React 页面只负责提供一个全屏 `iframe`
- 真正的 Earth 应用运行在:
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
所以 Earth 前端本质上是 `public/earth` 下的一套独立静态应用。
## 当前文件分层
### 1. 页面入口与结构
- [index.html](/home/ray/dev/linkong/planet/frontend/public/earth/index.html)
职责:
- HUD 基础 DOM
- 图层面板
- 媒体面板
- 工具栏
- 设置弹窗
- 兼容旧元素 id
### 2. 主运行时
- [main.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/main.js)
职责:
- 地球初始化
- Three.js 场景组装
- 数据加载与刷新
- 各图层集成
- Earth 级别状态同步
### 3. 地球控制层
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
职责:
- 工具栏交互
- 图层面板交互
- 旋转/缩放/布局
- HUD 面板拖拽
- 图层开关状态机
- Earth 设置读取、持久化与重置
这份文件是 Earth 前端当前最核心的 UI 控制入口。
### 4. UI 与状态消息
- [ui.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/ui.js)
职责:
- loading 面板
- status message
- tooltip / error / 清理逻辑
### 5. 地球与地形
- [earth.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/earth.js)
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
职责:
- 地球球体、云层、大气
- 真实地形 mesh
- terrain tile 拉取、解码、位移、着色
### 6. 图层模块
- [satellites.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/satellites.js)
- [cables.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cables.js)
- [bgp.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js)
- [news.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/news.js)
- [tv.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/tv.js)
- [layer-startup-tasks.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-startup-tasks.js)
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
职责:
- 各自的数据层
- 开关行为
- 面板内容
- hover/lock/selection 语义
其中 Earth 启动加载链现在也拆成了两层:
- `controls.js`
- 提供图层注册表与启动元信息
- `layer-startup-tasks.js`
- 提供图层启动任务注册表
- 通过 `registerLayerStartupTask(id, taskFactory)` 扩展启动任务
- `main.js`
- 只负责读取排序后的启动图层,再按映射执行队列
其中巡航模式现在已经拆成两层:
- `cruise-sequencer.js`
- 负责目标队列顺序、停留时长、切换节奏、打断与恢复
- `callout-connector.js`
- 负责卡片连线 SVG、路径计算与绘制动画
- `bgp-cruise-adapter.js`
- 负责 BGP 巡航展示适配目标排序、卡片落点、连线路径、focus/overlay/info-card 时序
当前 BGP 巡航只是这套能力的一个调用方不应再把“按队列巡航”和“BGP 事件展示”混写在同一个状态机里。
## 当前样式分层
Earth 的 CSS 不是一份大样式表,而是分层管理:
- [base.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/base.css)
- [hud.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/hud.css)
- [toolbar.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/toolbar.css)
- [layer-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/layer-panel.css)
- [info-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/info-panel.css)
- [legend.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/legend.css)
- [earth-stats.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/earth-stats.css)
- [coordinates-display.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/coordinates-display.css)
- [tv-panel.css](/home/ray/dev/linkong/planet/frontend/public/earth/css/tv-panel.css)
当前建议:
- 通用 HUD 壳层写进 `hud.css`
- 单一面板特性写进各自子文件
- 不要把业务状态样式再散回 `index.html`
## 当前图层开关状态语义
Earth 图层按钮现在不应再只有“开/关”两态,而应支持:
- `inactive`
- `active`
- `loading`
当前入口在:
- [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)
- [layer-button-state.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/layer-button-state.js)
关键函数:
- `updateLayerButtonState(button, isActive)`
- `setLayerButtonState(button, options)`
`setLayerButtonState` 负责:
- `loading` 样式
- `aria-busy`
- 按钮禁用
- tooltip 更新
- 绑定状态文本更新
- 可选同步 `active`
因此后续如果别的图层也需要异步启用,应该直接走这套状态机,而不是再手写一套临时 loading class。
另外Earth 图层控制现在已经收成“注册表驱动”:
- 图层元数据
- `id`
- `icon`
- `label`
- `meta`
- `buttonId`
- `persist`
- `startupPriority`
- `startupMode`
- `startupLabel`
- `startupMessage`
- 图层行为
- `getVisible()`
- `setVisible(next, options)`
当前入口仍在 [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js)。
这意味着后续新增图层时,优先应补一条图层注册定义,而不是同时去改:
- 图层面板 HTML
- 持久化快照
- 初始化恢复
- click 绑定
这四处现在都应该由注册表派生。
其中:
- `startupPriority`
- 描述图层参与启动加载时的顺序
- `startupMode`
- `visible`
- 仅当前图层处于启用/可见状态时,才加入启动加载队列
- `preload`
- 即使当前图层未显示,也会参与启动预加载
当前 `main.js` 会通过注册表读取排序后的启动图层列表,再动态拼装启动加载队列,而不是手写一串固定步骤。像 BGP 这类需要尽早准备数据、但不一定默认显示的图层,应该优先走 `startupMode: "preload"`,而不是在启动流程里写隐式特判。
此外,启动阶段给用户看的提示文案也应尽量从注册表派生:
- `startupLabel`
- 用于描述当前启动任务的业务名称
- `startupMessage`
- 用于描述启动中的提示文案
- 可以是字符串
- 也可以是对象,用于像海缆这种“准备阶段 / 主加载阶段”两段式文案
这样后续新增会参与启动加载的图层时,顺序、模式和提示文案都在同一处定义,不需要再去 `main.js` 里补第二套常量。
### `data-status-target`
图层按钮可以通过:
- `data-status-target`
指向一个状态文本节点。当前 terrain 已接入:
- 按钮:`#toggle-terrain`
- 状态节点:`#terrain-status`
以后别的异步图层也可以沿用这套约定。
## 当前设置持久化
Earth 设置面板当前由 [controls.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/controls.js) 统一负责:
- 捕获默认值
-`localStorage` 读取上次设置
- 初始化应用当前设置
- 用户变更后即时持久化
- 一键重置回默认值
当前持久化的范围是:
- 旋转模式
- 地球默认大小(作为重置视角、缩放重置和巡航视图的默认 zoom 真源)
- HUD 面板显示/隐藏
- 图层控制开关:`地形 / 卫星 / 轨迹 / 海缆 / BGP`
- 地形透明度
也就是说Earth 设置不是一次性 UI 状态了,而是本地设备级偏好。后续如果再加入新的设置项,应优先接入同一条持久化链,而不是各自散着写 `localStorage`
## 当前地形链路
真实地形首次启用会慢,原因不只是一个:
1. 需要拉取 Terrarium 瓦片
2. 需要解码图片
3. 需要按顶点采样高程
4. 需要重新写入 geometry 和 color
5. 需要重新计算法线与包围体
当前入口在:
- [terrain.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/terrain.js)
当前已经做了两层体验优化:
1. 图层开关 loading 状态持续可见
2. 页面空闲时会预热 `ensureTerrainReady()`
也就是说,后续再继续优化 terrain 时,优先顺序应该是:
1. 先保证用户感知正确
2. 再压缩首次等待
3. 最后才做更激进的几何/瓦片优化
## 当前高频风险点
### 1. 视觉状态和业务状态不同步
Earth 里最常见的 bug 不是“没渲染”,而是:
- 图层关了tooltip 还在
- 锁定对象隐藏了info card 还在
- legend 没跟图层切换
- loading 已结束,但按钮还像没开
后续改动必须优先检查状态同步。
### 2. HUD 布局问题先查结构,不要先打 CSS 补丁
Earth HUD 历史上反复出现:
- 面板只剩一条缝
- markdown 被裁掉
- tabs/iframe 被 `overflow: hidden` 吃掉
优先检查:
1. 谁负责高度
2. 谁负责滚动
3. 哪一层在裁剪
不要上来先加 `overflow: hidden` 或额外包装层。
### 3. Transitional path 必须收口
Earth 已经经历过多轮 HUD、toolbar、media panel 重构,所以最容易积累:
- 旧 helper
- 旧 class
- 旧 fallback 逻辑
- 已废弃变体
每次大功能完成后,都要做一次 cleanup pass。
### 4. 巡航与业务事件不要再深度耦合
当前正确边界应该是:
- 通用巡航层只知道:
- 当前目标
- 队列顺序
- 相机 focus
- 停留 / 隐藏 / 切换
- 业务模块只负责:
- 提供目标队列
- 提供 focus 坐标
- 提供卡片内容
- 提供高亮/图层副作用
如果以后再给海缆、卫星或新闻做巡航,不应复制一套新的 `main.js` 状态变量,而应复用:
- [cruise-sequencer.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/cruise-sequencer.js)
- [callout-connector.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/callout-connector.js)
- [bgp-cruise-adapter.js](/home/ray/dev/linkong/planet/frontend/public/earth/js/bgp-cruise-adapter.js) 这种业务适配层模式
## 当前推荐改动方式
如果后续继续改 Earth建议按这个顺序
1. 先确认改的是:
- Three.js 渲染层
- HUD 结构层
- 图层状态层
- 面板内容层
2. 如果涉及图层按钮,优先接入统一状态机
3. 如果涉及可见性切换,检查 tooltip / legend / info-card / lock 是否一起收口
4. 如果涉及面板布局,先查结构再动 CSS
## 当前与控制台前端的边界
Earth 前端和控制台前端不是同一套 UI 系统:
- 控制台前端React + Ant Design 工作台
- Earth 前端:`public/earth` 原生 HUD + Three.js 展示面
因此:
- Earth 不应该直接复用 Ant Table / AppLayout 语义
- 控制台也不应该照搬 Earth HUD 动画和玻璃层语言
控制台相关结构见:
- [admin-frontend-context.md](/home/ray/dev/linkong/planet/docs/technical/frontend-admin-frontend-context.md)

Some files were not shown because too many files have changed in this diff Show More